feat(sdk,core,webapp,run-engine): runtime overrides for total and per-key queue limits - #4829
feat(sdk,core,webapp,run-engine): runtime overrides for total and per-key queue limits#4829matt-aitken wants to merge 6 commits into
Conversation
Sparse ckLimits HASH at the base queue whose fields are the exact ck-variant queue names, plus engine methods to set (atomic cardinality cap, default 1000 per queue), remove, and read the overrides. The admit-path gate wiring follows.
The ck dequeue admit and both enqueue fast paths read the queue's ckLimits HASH for the variant being admitted and use the env-clamped override in place of the queue's per-key limit, behind the totalConcurrencyEnabled flag. Covered by tests for lowered and raised keys, removal, the cardinality cap, and flag-off behavior.
Three nullable TaskQueue columns record when, by whom, and from what declared base the total concurrency limit was overridden, and a new TaskQueueConcurrencyKeyOverride child table stores per-key limit overrides, unique per queue and key and cascading with the queue.
queues.overrideConcurrencyLimit accepts a named concurrencyKey to adjust one key's limit independently, and new overrideTotalConcurrencyLimit and resetTotalConcurrencyLimit calls adjust the cap across all keys. Four API routes back them; the concurrency system validates against the environment limit, captures the declared base on first override, persists per-key overrides in the child table alongside the engine hash, and deploys keep an overridden total instead of clobbering it from the manifest.
🦋 Changeset detectedLatest commit: e7844da The changes in this PR will be included in the next version bump. This PR includes changesets to release 27 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Observability mapAs of 20/100 over 454 measured of 472 entry points (base 20, no change) What this PR changed
FIX FIRST
AUDIT 3 of 50 sensitive mutations record an actor. 47 without one. What the score is made ofThe score and findings here are report-only and never gate the merge. Separately, a required test suite keeps this tool's symbol and route lists in sync with the code they name, and can fail a pull request that renames or removes a symbol they reference, or that adds the first route with a segment they anticipate. Each failure names the list to edit. The rules and their reasons: internal-packages/observability-map/README.md. |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
…ked keys A variant blocked at its per-key limit or by a gate now backs off in the candidate index instead of pinning the bounded window, so zero-limit keys can never starve runnable keys behind them; acks and nacks rebalance the score back the moment capacity frees. Override writes persist before enforcing and resets enforce before clearing, so a failure on either side leaves a state a retry converges from, with the cap-rejected row compensated away. Deploys restore per-key overrides from their durable rows into the engine, and an empty concurrencyKey in the SDK no longer falls through to the queue-wide endpoint.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
…en races Gate capacity now reads the gate queue's ckLimits hash, so an override on a key applies whether runs meet it as their own queue or as a gate. A reset deletes only the exact row generation it read, so a concurrent override's newer record survives, and a cap rejection deletes its row unconditionally since the cap can only reject keys absent from the engine hash.
| if (error.type === "too_many_key_overrides") { | ||
| return fromPromise( | ||
| db.taskQueueConcurrencyKeyOverride.deleteMany({ | ||
| where: { taskQueueId: queue.id, concurrencyKey }, | ||
| }), |
There was a problem hiding this comment.
🟡 Rejected request deletes successful override
If capacity frees after one request fails, a concurrent retry can save the same key before deleteMany runs. The failed request then deletes the retry’s durable record.
Prompt for agents
Make cap-rejection cleanup generation-safe in apps/webapp/app/v3/services/concurrencySystem.server.ts. The cleanup in overrideQueueConcurrencyKeyLimit must delete only the row generation written by the rejected request. Capture a generation value from the upsert, such as overriddenAt or another compare-and-delete token, and include it in deleteMany. Ensure a concurrent successful override of the same queue and concurrency key cannot have its database row removed by an older rejected request.
Was this helpful? React with 👍 or 👎 to provide feedback.
| if gateKey and gateKey ~= '' then | ||
| local gateOverride = redis.call('HGET', base .. ':ckLimits', string.sub(variant, #gatesKeyPrefix + 1)) | ||
| if gateOverride then | ||
| perKeyLimit = math.min(tonumber(gateOverride), envLimit) | ||
| end |
There was a problem hiding this comment.
🟡 Gate overrides bypass rollout flag
When gates are enabled but total-limit enforcement is disabled, __gatesHaveCapacity still applies per-key overrides. Gate queues enforce settings disabled for normal queues.
Prompt for agents
Thread the totalConcurrencyEnabled state into __gatesHaveCapacity and guard the ckLimits lookup with it, matching the normal queue admission paths. Update every Lua call site in internal-packages/run-engine/src/run-queue/index.ts and add a test with gatesEnabled=true and totalConcurrencyEnabled=false that confirms gate per-key overrides are ignored.
Was this helpful? React with 👍 or 👎 to provide feedback.
1 participant
Summary
Stacked on #4827. Makes queue concurrency adjustable at runtime, per key and in total:
Today
overrideConcurrencyLimitmoves every key's limit at once, which is the structural blocker for "raise one tenant". The namedconcurrencyKeyparameter routes to a per-key override that leaves the rest of the queue alone, and the total override adjusts the cross-key cap that #4823 introduced.Design
Per-key overrides live in a sparse Redis hash on the base queue whose fields are the exact key-variant names, so the dequeue admit reads them with a single lookup and no parsing; a new child table mirrors them for durability, unique per queue and key, cascading with the queue. Writes go through an atomic script that caps the number of overrides per queue (default 1000) and rejects new keys past the cap while always allowing updates to existing ones. Resetting deletes the row and the hash field, falling back to the queue's limit.
Total overrides follow the existing override bookkeeping exactly: validated against the environment limit, the declared base captured on first override, reset restoring it, and deploys preserving an active override instead of clobbering it from the code manifest (the declared value lands in the base). Both read paths clamp to the environment limit at admit time, and per-key enforcement sits behind the same server flag as the total cap.
Not included, by design: dashboard surfacing for overrides (rides the metrics work) and automatic redistribution between keys.