feat(sdk,core,webapp,run-engine): runtime overrides for queue total concurrency limits - #4829
feat(sdk,core,webapp,run-engine): runtime overrides for queue total concurrency limits#4829matt-aitken wants to merge 30 commits into
Conversation
|
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Team Run ID: 📒 Files selected for processing (1)
Included review availability: Your plan provides up to 10 included reviews per hour; 8 remain after this review. 📜 Recent review details⏰ Context from checks skipped due to timeout. (9)
🧰 Additional context used📓 Path-based instructions (10)New code must target Run Engine V2 through the singleton in `app/v3/runEngine.server.ts`; do not reintroduce V1 execution paths.📄 CodeRabbit inference engine (apps/webapp/CLAUDE.md) Files:
Never use `request.signal` to detect client disconnects.📄 CodeRabbit inference engine (apps/webapp/CLAUDE.md) Files:
For dashboard changes, visually verify the running Remix app with Chrome DevTools MCP, using snapshots, screenshots, interaction, and console-message checks as appropriate.📄 CodeRabbit inference engine (apps/webapp/CLAUDE.md) Files:
**Prefer static imports over dynamic imports.**📄 CodeRabbit inference engine (AGENTS.md) Files:
Add crumbs as you write code — not just when debugging.📄 CodeRabbit inference engine (AGENTS.md) Files:
Use zod for validation in packages/core and apps/webapp📄 CodeRabbit inference engine (.github/copilot-instructions.md) Files:
Access environment variables through the `env` export of `env.server.ts` instead of directly accessing `process.env` Use subpath exports from `@trigger.dev/core` package instead of importing from the root `@trigger.dev/core` path📄 CodeRabbit inference engine (.cursor/rules/webapp.mdc) Files:
Use function declarations instead of default exports📄 CodeRabbit inference engine (.github/copilot-instructions.md) Files:
Use types over interfaces for TypeScript Avoid using enums; prefer string unions or const objects instead📄 CodeRabbit inference engine (.github/copilot-instructions.md) Files:
When creating or editing OTEL metrics (counters, histograms, gauges), ensure metric attributes have low cardinality by using only enums, booleans, bounded error codes, or bounded shard IDs Do not use high-cardinality attributes in OTEL metr...📄 CodeRabbit inference engine (.cursor/rules/otel-metrics.mdc) Files:
🔇 Additional comments (3)
WalkthroughAdds queue roles and concurrency-version metadata, materializes named and inline concurrency limits as LIMIT queues, and compiles task concurrency gates. Adds total concurrency override and reset routes with persistence and run-engine synchronization. Adds per-concurrency-key override storage and enforcement across enqueue, dequeue, gate checks, and candidate backoff. Restricts queue retrieval to QUEUE records. Merge Risk: 🟡 Moderate · up to The concurrency override feature can still report incorrect state, lose concurrent updates, target internal limit queues, or enforce limits under inconsistent semantics. These material runtime and persistence issues should be resolved before merge. 🚥 Pre-merge checks | ✅ 3 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (3 passed)
Full details: Description checkExplanation The description clearly explains the feature, design, API usage, and deferred per-key override scope. It does not include the required issue reference, checklist, testing details, changelog, or screenshots sections from the repository template. Resolution Add the required template sections. Include a valid ✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Observability mapAs of 19/100 over 456 measured of 474 entry points (base 20, down 1) What this PR changed
FIX FIRST
AUDIT 3 of 50 sensitive mutations record an actor. 47 without one. What the score is made ofThe score and findings here are report-only and never gate the merge. Separately, a required test suite keeps this tool's symbol and route lists in sync with the code they name, and can fail a pull request that renames or removes a symbol they reference, or that adds the first route with a segment they anticipate. Each failure names the list to edit. The rules and their reasons: internal-packages/observability-map/README.md. |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
fbd79d2 to
5110870
Compare
@trigger.dev/build
trigger.dev
@trigger.dev/core
@trigger.dev/python
@trigger.dev/react-hooks
@trigger.dev/redis-worker
@trigger.dev/rsc
@trigger.dev/schema-to-json
@trigger.dev/sdk
commit: |
5110870 to
fe1f3a4
Compare
2e8db3a to
eba7d2a
Compare
matt-aitken
changed the title
eba7d2a to
0f83715
Compare
0f83715 to
0bef7b3
Compare
0bef7b3 to
f103a5e
Compare
matt-aitken
changed the title
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
8ba16f9 to
9800b6a
Compare
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
da92e21 to
6b14574
Compare
Three nullable TaskQueue columns record when, by whom, and from what declared base the total concurrency limit was overridden, and a new TaskQueueConcurrencyKeyOverride child table stores per-key limit overrides, unique per queue and key and cascading with the queue.
queues.overrideConcurrencyLimit accepts a named concurrencyKey to adjust one key's limit independently, and new overrideTotalConcurrencyLimit and resetTotalConcurrencyLimit calls adjust the cap across all keys. Four API routes back them; the concurrency system validates against the environment limit, captures the declared base on first override, persists per-key overrides in the child table alongside the engine hash, and deploys keep an overridden total instead of clobbering it from the manifest.
…ked keys A variant blocked at its per-key limit or by a gate now backs off in the candidate index instead of pinning the bounded window, so zero-limit keys can never starve runnable keys behind them; acks and nacks rebalance the score back the moment capacity frees. Override writes persist before enforcing and resets enforce before clearing, so a failure on either side leaves a state a retry converges from, with the cap-rejected row compensated away. Deploys restore per-key overrides from their durable rows into the engine, and an empty concurrencyKey in the SDK no longer falls through to the queue-wide endpoint.
…en races Gate capacity now reads the gate queue's ckLimits hash, so an override on a key applies whether runs meet it as their own queue or as a gate. A reset deletes only the exact row generation it read, so a concurrent override's newer record survives, and a cap rejection deletes its row unconditionally since the cap can only reject keys absent from the engine hash.
…fe cap cleanup Gate admission reads a per-key override only when concurrency limit enforcement is enabled, matching the primary admit paths (the flag now threads through the unkeyed enqueue and dequeue scripts too). The cap-rejection cleanup deletes only the exact row generation the rejected request wrote, so a concurrent request that succeeded after capacity freed keeps its durable record.
The client build's export analyzer cannot statically resolve destructured route exports, so it treats the module's exports as depending on server-only code and the build fails. Also export the builder's loader so non-POST methods get a 405, matching the other concurrency routes.
The override and reset endpoints, client methods and SDK functions now say combinedConcurrencyLimit, matching the queue option.
…limit overrides Per-key runtime overrides are deferred: composing a per-key hash with pause semantics and keeping it convergent with durable rows across deploys and resets needs its own design. Combined-limit overrides stay. Declared per-key behavior is unchanged (concurrencyLimit applies per key as before).
Removes the per-key override endpoints, the override-aware admit and gauge reads, and the per-key limit column, following the removal of runtime per-key overrides from this stack.
…remain The per-key admit reads are removed in the PR above alongside the rest of the engine-internal plumbing; the key builders they reference stay until then so every level of the stack compiles.
concurrencyVersion records how a queue's concurrency was declared (V1 = legacy contextual concurrencyLimit, V2 = the explicit perKey/total shape) so display never has to guess what a stored number meant. role distinguishes real queues from the rows that back named concurrency limits, which every queue-facing surface filters out. Defaults cover all existing rows with no backfill.
The combined override and reset never ship; runtime management of the new named limits arrives with its own management API.
Named limits (declared or merely referenced) materialize as LIMIT-role queue rows under the reserved limit/ prefix, with a total-only limit storing its total as the per-key limit too so it truly caps keyless runs. A task's inline limit maps onto its own default queue (stamped concurrencyVersion V2) or, on a shared queue, onto an anonymous limit/task row holding one of the two gate slots. Declaring both the legacy queue concurrencyLimit and concurrency is a deploy error. Trigger-time concurrency names resolve to limit gates, replacing the task's declared set, and every queue read filters to QUEUE-role rows. Drops the release note for the combined override methods that no longer ship.
Validation errors now fail the deploy instead of being swallowed into a silently incomplete worker, user queue names may not claim the reserved limit/ prefix, and a limit whose prefixed row name would exceed the 128-character queue maximum is rejected rather than truncated into the wrong row.
… exist A rejected deploy previously left a partial worker behind, which a same-content retry then returned as if the deploy had succeeded. The whole manifest's concurrency declarations are now validated up front.
…us limit rows Deploy validation now enforces the SDK's limit-name charset, so sanitization can never merge two declared limits onto one row. A task's anonymous inline limit row hashes the raw task id whenever sanitizing it would be lossy or overflow the queue-name length, so distinct task ids always keep distinct rows.
The queue concurrency override/reset endpoints no longer resolve the TaskQueue rows that back named concurrency limits, matching the list and retrieve surfaces. Limits get their own management API instead of being mutable through a surface that cannot read them back.
Passing concurrency at trigger time replaces only the task's declared named limits. The anonymous limit/task/ gate that carries an inline limit on a shared queue is preserved, and an empty array clears named limits without dropping it. Trigger-time limit names are also validated against the queue-safe charset so lossy sanitization can never remap them onto another limit's row.
…ly for inline default queues The queue upsert's update carries the override markers in its where clause, so a concurrent override or reset makes the update miss and retry against a fresh read instead of silently overwriting what the operator just set. A queue row is marked concurrencyVersion V2 only when an inline limit compiles into the task's own default queue; shared queues stay V1 whichever code path materializes them.
…ploy The optimistic markers only guard the Postgres write; an override or reset landing between that write and the engine sync would leave the engine holding the deploy's stale values. The deploy now re-reads the markers after syncing and re-syncs once from the fresh row when they moved, so the last engine write always carries the freshest limits.
…p moving The single re-read left a window where a second override during the same deploy could be overwritten by a stale re-sync. The deploy now re-reads and re-syncs until the markers stop moving (bounded), and any marker moving after the final read is healed by that actor's own engine sync.
…alidate first The low-level gates option now carries the task's anonymous inline limit gate over the same way the concurrency option does (a replay resending stored gates collapses back through dedupe). The V4 deploy path rejects invalid concurrency declarations before any worker rows or engine keys are written, so a failed deploy can no longer leave the running version's limits mutated.
Prepending the task's inline gate to a full raw gates array could push the set past three; the resolver now throws a clear error instead of silently dropping the last gate. Replays are unaffected: resent stored gates collapse through the dedupe before the check.
…our-gate backstop A deploy syncing a paused queue writes the paused zero limit instead of skipping, and the post-deploy convergence loop also re-syncs when the paused flag moved, so a pause or resume landing mid-deploy can no longer leave the engine enforcing the wrong limit. Client-supplied task gates may not claim the reserved limit/ namespace. The gate overflow backstop moves to four to match the engine capacity.
Resume writes the engine limit from the row the update returned, so a concurrent deploy's change is not overwritten with a stale snapshot, and a declared limit of zero is written as zero instead of removing the limit and leaving the queue uncapped.
7c6f1be to
bcee429
Compare
matt-aitken
changed the title
1 participant
Summary
Adjust a queue's combined concurrency limit at runtime without a deploy.
queues.overrideCombinedConcurrencyLimitraises or lowers the cap on concurrent runs across all of a queue'sconcurrencyKeyvalues, andqueues.resetCombinedConcurrencyLimitreverts to the declared configuration. Stacked on #4827.Design
Combined overrides follow the existing override bookkeeping exactly: validated against the environment limit, the declared base captured on first override, reset restoring it, and deploys preserving an active override instead of clobbering it from the code manifest (the declared value lands in the base). The read path clamps to the environment limit at admit time.
Per-key runtime overrides were built in earlier revisions of this PR and then deliberately removed: composing a per-key override hash with pause semantics and keeping it convergent with durable rows across deploys, resets, and races needs its own design, so that feature is deferred to a future PR. Declared per-key behavior is unchanged (
concurrencyLimitapplies per key as before).