feat(run-engine,sdk,webapp): queue total concurrency limit across keys

[matt-aitken]

Summary

Adds a totalConcurrencyLimit option to queues. On a queue used with concurrencyKey, concurrencyLimit applies to each key value independently, so ten active keys with a limit of 5 can run 50 at once and nothing bounds the queue as a whole short of the environment limit. totalConcurrencyLimit caps in-flight runs across all keys while each key still gets at most concurrencyLimit:

export const perUserQueue = queue({
  name: "per-user-queue",
  concurrencyLimit: 1,
  totalConcurrencyLimit: 10,
});

Enforcement is gated behind RUN_ENGINE_TOTAL_CONCURRENCY_LIMITS_ENABLED (default off) and applies to runs triggered with a concurrencyKey. With the gate off, admit paths are unchanged.

Design

The engine keeps a per-base-queue groupConcurrency set shared by every concurrency-key variant; its cardinality is the queue's total in-flight count. The concurrency-key dequeue script bounds each batch by min(totalLimit, envLimit) - SCARD(group) and adds admitted runs to the set. The enqueue fast path checks the same gate and falls back to a normal enqueue when the queue is at its total.

Every release path (ack, nack, dead-letter, concurrency release, TTL expiry, sweeper clear) removes a run from the group set whenever it removes it from the per-key currentConcurrency set. Those removals run regardless of the gate, so the set stays correct if the gate is later turned off, and the existing reconciliation sweep self-heals the group set because it is always a subset of the per-key sets it acks against.

The stored limit is the raw declared value; readers clamp to the environment concurrency limit. The option flows through the queue manifest into a new nullable TaskQueue.totalConcurrencyLimit column (additive migration) and syncs to the engine on deploy.

The group set self-heals against release paths that miss the removal (an instance on an older build during a rollout, or a future release script such as the one #4398 adds). Every terminal release deletes the run's message key, so when a queue sits at its total the dequeue gate prunes members whose message key no longer exists, throttled to one pass per interval per queue. Runs in flight before the gate is enabled are the opposite case: they are absent from the set, so a queue can transiently exceed its total by at most that count, converging as each one completes.

Not included here, planned as follow-ups: a runtime override API for the total limit, dashboard and metrics surfacing, and skipping queues at their total during fair-queue selection.

[changeset-bot]

🦋 Changeset detected

Latest commit: 6290352

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 27 packages

Name	Type
@trigger.dev/sdk	Patch
@trigger.dev/core	Patch
@trigger.dev/python	Patch
@internal/dashboard-agent	Patch
@trigger.dev/build	Patch
trigger.dev	Patch
@trigger.dev/redis-worker	Patch
@trigger.dev/schema-to-json	Patch
@internal/clickhouse	Patch
@internal/llm-model-catalog	Patch
@internal/metrics-pipeline	Patch
@trigger.dev/rbac	Patch
@internal/redis	Patch
@internal/replication	Patch
@internal/run-engine	Patch
@internal/run-store	Patch
@internal/schedule-engine	Patch
@internal/tracing	Patch
@internal/webhook-engine	Patch
@internal/webhook-sources	Patch
@internal/cache	Patch
@trigger.dev/react-hooks	Patch
@trigger.dev/rsc	Patch
@trigger.dev/database	Patch
@trigger.dev/otlp-importer	Patch
@trigger.dev/sso	Patch
@internal/testcontainers	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

Walkthrough

Adds an optional totalConcurrencyLimit queue setting for keyed runs. The SDK and queue schemas expose and register the setting. The database stores it, and the webapp synchronizes it with RunQueue. RunQueue tracks concurrency across key variants with Redis sets and enforces the limit during enqueue and dequeue operations. Release, retry, dead-letter, clear, and expiry paths update aggregate state. Integration tests cover enabled, disabled, per-key, fallback, acknowledgment, nack, and leaked-member reconciliation behavior.

Merge Risk: 🟡 Moderate · up to 588ac

The change adds a gated queue-wide concurrency cap, but saturated queues may currently perform large repeated cleanup operations that delay processing, while rollout or missed cleanup can strand capacity and prevent runs from starting; enabling the cap with existing runs can also temporarily exceed it. The feature is disabled by default, but these bounded availability and rollout risks require explicit owner follow-up before broad enablement.

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 30.77% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 13 functions across 14 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.
Description check	⚠️ Warning	The description provides a detailed and relevant summary of the feature and design, but it omits the template sections for issue closure, checklist completion, testing steps, changelog, and screenshot…	Add the required template sections. Include the linked issue, checklist status, exact testing performed, a short changelog entry, and screenshots or an explicit indication that screenshots are not applicable.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly identifies the main change: adding queue-wide total concurrency limits across concurrency keys.

Full details: Description check

Explanation

The description provides a detailed and relevant summary of the feature and design, but it omits the template sections for issue closure, checklist completion, testing steps, changelog, and screenshots.

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/queue-total-concurrency-limit

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[devin-ai-integration]

Note

This report is out of date. Scroll down for Devin Review's latest report on this PR.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

[pkg-pr-new]

Open in StackBlitz

@trigger.dev/build

npm i https://pkg.pr.new/@trigger.dev/build@3efb50b

trigger.dev

npm i https://pkg.pr.new/trigger.dev@3efb50b

@trigger.dev/core

npm i https://pkg.pr.new/@trigger.dev/core@3efb50b

@trigger.dev/python

npm i https://pkg.pr.new/@trigger.dev/python@3efb50b

@trigger.dev/react-hooks

npm i https://pkg.pr.new/@trigger.dev/react-hooks@3efb50b

@trigger.dev/redis-worker

npm i https://pkg.pr.new/@trigger.dev/redis-worker@3efb50b

@trigger.dev/rsc

npm i https://pkg.pr.new/@trigger.dev/rsc@3efb50b

@trigger.dev/schema-to-json

npm i https://pkg.pr.new/@trigger.dev/schema-to-json@3efb50b

@trigger.dev/sdk

npm i https://pkg.pr.new/@trigger.dev/sdk@3efb50b

commit: 3efb50b

[coderabbitai]

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

internal-packages/run-engine/src/run-queue/index.ts (1)

3827-3865: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Add crumbs for aggregate-concurrency state changes.

Add // @Crumbs markers or `// `#region` `@crumbs blocks around the new total-cap admission and group-set release paths. This includes the tracked CK enqueue, dequeue, acknowledgment, nack, dead-letter, release, and clear commands.

As per coding guidelines, “Add crumbs as you write code — not just when debugging.”

Also applies to: 4639-4661, 5249-5725

Source: Coding guidelines

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: e99334b8-fb48-462b-a79f-b9683eac6e60

📥 Commits

Reviewing files that changed from the base of the PR and between 468323d and 023c282.

📒 Files selected for processing (1)

• internal-packages/run-engine/src/run-queue/index.ts

Included review availability: Your plan provides up to 10 included reviews per hour; 7 remain after this review.

📜 Review details

⏰ Context from checks skipped due to timeout. (2)

• GitHub Check: webapp / 🧪 Unit Tests: Webapp (13, 24)
• GitHub Check: internal / 🧪 Unit Tests: Internal

🧰 Additional context used

📓 Path-based instructions (5)

**Prefer static imports over dynamic imports.** Only use dynamic `import()` when:

📄 CodeRabbit inference engine (AGENTS.md)

Files:

• internal-packages/run-engine/src/run-queue/index.ts

Add crumbs as you write code — not just when debugging. Mark lines with

📄 CodeRabbit inference engine (AGENTS.md)

Files:

• internal-packages/run-engine/src/run-queue/index.ts

Use function declarations instead of default exports

📄 CodeRabbit inference engine (.github/copilot-instructions.md)

Files:

• internal-packages/run-engine/src/run-queue/index.ts

Use types over interfaces for TypeScript

📄 CodeRabbit inference engine (.github/copilot-instructions.md)

Files:

• internal-packages/run-engine/src/run-queue/index.ts

When creating or editing OTEL metrics (counters, histograms, gauges), ensure metric attributes have low cardinality by using only enums, booleans, bounded error codes, or bounded shard IDs

📄 CodeRabbit inference engine (.cursor/rules/otel-metrics.mdc)

Files:

• internal-packages/run-engine/src/run-queue/index.ts

[devin-ai-integration]

Note

This report is out of date. Scroll down for Devin Review's latest report on this PR.

Devin Review found 0 new potential issues.

[devin-ai-integration]

Devin Review found 0 new potential issues.