uber · mnoah1 · Sep 15, 2026
@@ -10,7 +10,7 @@ It handles only the trigger: it does not poll for completion, record greenness,
 
 `build` consumes a request id, published by `process` in Phase 1 (see [workflow.md](doc/rfc/stovepipe/workflow.md#workflow)). Both phases drive the same `build` → `buildsignal` machinery against the same `Request` row. The `process`/`analyze` → `build` topic is partitioned by **request id**; see [Partitioning](#partitioning) for the full rationale, including why `build` → `buildsignal` partitions finer (by build id) than SubmitQueue's equivalent topic.
 
-**`build` is not the sole writer of the rows it touches, and its own writes are narrow.** On `Request`, `build` never writes at all — it only reads the `BuildStrategy`/`BaseURI`/`URI` fields `process` set at admit, and it leaves `Request.State` untouched at `processing` throughout (`process`, `buildsignal`, and the DLQ reconciler are `Request.State`'s only writers). On `Build`, `build` is the sole creator — it calls `BuildStore.Create` exactly once, at step 6 — and never mutates the row again; `buildsignal` is the sole writer of `Build.Status`/`Build.Version` afterward (see [buildsignal.md](doc/rfc/stovepipe/steps/buildsignal.md#input-and-re-entrancy)). This division is why `build` never needs a CAS/version write of its own: `Create` is the only storage mutation in its algorithm.
+**`build` is not the sole writer of the rows it touches, and its own writes are narrow.** On `Request`, `build` never writes at all — it only reads the `BuildStrategy`/`BaseURI`/`URI` fields `process` set at admit, and it leaves `Request.State` untouched at `processing` throughout (`process`, `buildsignal`, and the DLQ reconciler are `Request.State`'s only writers). On `Build`, `build` is the sole creator — it calls `BuildStore.Create` exactly once, at step 6 — and never mutates the row again; `buildsignal` is the sole writer of `Build.Status`/`Build.TerminalAtMs`/`Build.Version` afterward (see [buildsignal.md](doc/rfc/stovepipe/steps/buildsignal.md#input-and-re-entrancy)). This division is why `build` never needs a CAS/version write of its own: `Create` is the only storage mutation in its algorithm.
 
 `build` is phase-agnostic: it never asks "which phase is this?" It reads whatever scope is already persisted and immutable on the `Request` and acts on it. Phase 2's project-scoped invocation is expected to read project-scoped equivalents of the scope fields off the same `Request`; the exact shape of a project-scoped trigger is left to the `analyze` design, consistent with `workflow.md`'s "project mapping contract" open question — see [Project-scoped `Trigger`: reserved, not yet designed](#project-scoped-trigger-reserved-not-yet-designed) for the resulting gap in the `BuildRunner` contract itself.
 
@@ -52,7 +52,7 @@ For a delivery carrying request id `R`:
      either domain — the shape is deferred until then, not decided here.
    - failure -> return raw; classifier decides (transient runner blip retryable, bad URI not).
 
-6. Persist Build{ID: buildID.ID, RequestID: R.ID, Status: accepted, Version: 1}
+6. Persist Build{ID: buildID.ID, RequestID: R.ID, Status: accepted, TerminalAtMs: 0, Version: 1}
    via BuildStore.Create.
    - the row carries no scope; it is recoverable from the Request's immutable fields
      (see the entity table).

@@ -12,7 +12,7 @@ It handles only the poll loop: it does not decide build strategy, write greennes
 
 Its logic does not branch on phase: it loads the `Build`, polls it toward terminal, persists the result, and publishes the request id onward to `record`. What differs between phases is what `record` does with that publish (whole-repo vs. per-project greenness) — not anything `buildsignal` decides.
 
-`buildsignal` is the sole writer of `Build.Status`/`Build.Version` after `build` creates the row (see [build.md](doc/rfc/stovepipe/steps/build.md#input-partitioning-and-the-single-writer-property)). It reads `Request` via `RequestStore.Get` (for `R.Queue`, to resolve the build-runner) and writes it exactly once, at the terminal transition, to record the build's outcome — the one `Request.State` write outside `process` and the DLQ reconciler.
+`buildsignal` is the sole writer of `Build.Status`/`Build.TerminalAtMs`/`Build.Version` after `build` creates the row (see [build.md](doc/rfc/stovepipe/steps/build.md#input-partitioning-and-the-single-writer-property)). It reads `Request` via `RequestStore.Get` (for `R.Queue`, to resolve the build-runner) and writes it exactly once, at the terminal transition, to record the build's outcome — the one `Request.State` write outside `process` and the DLQ reconciler.
 
 Its early-exit guard is deliberately narrower than `State.IsTerminal()`: it proceeds when the request is `processing` **or** already carries a build outcome. The second case matters because a redelivery after the outcome was stamped but before the `record` publish landed must re-publish rather than drop the signal; everything it re-runs is a no-op (the status is unchanged, the outcome is already recorded, the slot is not released twice) and the `record` publish is idempotent.
 
@@ -55,15 +55,20 @@ For a delivery carrying build id `B`:
    - Build.Status already terminal, status differs -> terminal is WRITE-ONCE: do not overwrite;
      continue with the STORED status as authoritative (see Edge cases).
    - otherwise persist via BuildStore.Update(ctx, Build{...Status: status}, oldVersion, newVersion):
+     - when status is terminal, stamp TerminalAtMs with the observation time in the same update;
      - newVersion = Build.Version + 1; assign Build.Version = newVersion only on success.
      - ErrVersionMismatch -> return with its declaration-level retryable classification (a concurrent writer moved the row; reload and re-check).
      - with the write-once rule, accepted -> running -> {succeeded|failed|cancelled} is monotonic
        by mechanism, not by assumption about the backend.
 
 7. If the stored status is terminal, and R does not already carry an outcome:
-   a. Release the queue's build slot: CAS-decrement Queue.in_flight_count, clamped at zero.
+   a. Load the queue config only when the runner-reported status is failed. Apply
+      failure_cooldown_ms only when it is positive; non-positive values disable the policy.
+   b. Release the queue's build slot in one Queue CAS: decrement in_flight_count, clamped at zero,
+      and for failed status advance build_admission_not_before_ms to at least
+      Build.TerminalAtMs + failure_cooldown_ms.
       - failure here aborts the step: R must not go terminal while still holding a slot.
-   b. CAS R from processing to the outcome the stored status projects onto it:
+   c. CAS R from processing to the outcome the stored status projects onto it:
       succeeded -> succeeded, failed -> failed, cancelled -> cancelled. First writer wins.
    Then publish R.ID to the record topic, partitioned by request id; ack, return.
    No re-publish to buildsignal.
@@ -86,6 +91,8 @@ For a delivery carrying build id `B`:
 
 **Why `buildsignal` releases the slot rather than `record`**: the gate `process` claims is a *build* slot — it exists to bound concurrent builds per Queue — and once the build is terminal the build is over. Releasing here also keeps the invariant *a terminal `Request` has already released its slot*, which is what makes the DLQ reconciler's early-return on terminal requests safe.
 
+**Why failure cooldown is part of the slot-release CAS**: both fields govern the next logical admission, so updating them from the same freshly loaded Queue snapshot prevents a process admission from slipping between capacity release and deadline advancement. A version conflict reloads the row and recomputes the maximum, preserving a later deadline written by the minimum-interval policy or another terminal result. The deadline is based on the persisted `Build.TerminalAtMs`, not redelivery time, so retries cannot slide the cooldown forward. Only an actual runner-reported `failed` status applies this policy; `cancelled` and DLQ-forced request failure do not.
+
 **Why `record` hears only terminal signals**: `record` has no non-terminal work — by its own contract a non-terminal signal would be a pure no-op — and step 7 already branches on terminality to decide whether to keep polling, so gating the publish costs nothing and spares `record` a no-op delivery on every poll tick of every running build. Crash-safety is unaffected: a crash between the terminal `Update` and the publish redelivers the message; step 5 re-polls (the runner reports the same terminal status), step 6 no-ops, step 7 publishes. This is a deliberate divergence from SubmitQueue, whose buildsignal republishes to `speculate` on every tick — sound there because speculate is a state machine that may act on any signal; stovepipe has no such consumer.
 
 **Why step 6 guards on status and makes terminal write-once**: an unchanged status skips the CAS write entirely, so a long build being polled every couple of seconds doesn't churn `Build.Version` on every tick — the version only advances on a real state transition. The write-once rule exists because CAS alone cannot provide it: optimistic locking defends against *concurrent* writers, but a later delivery that polls a flaky backend and sees a different terminal status would CAS cleanly against the current version and overwrite (see Edge cases). A given `Build` has a single poll partition (see [Partitioning](doc/rfc/stovepipe/steps/build.md#partitioning)), so the only writer racing the CAS is a redelivery of the same message (e.g. after a lapsed visibility lease); `ErrVersionMismatch` carries a retryable classification and converges on redelivery.

@@ -64,6 +64,7 @@ Validation is expensive and shares a baseline, so heads arriving while an earlie
 | Queue row | `build_admission_not_before_ms` | Durable earliest time for the next logical admission; zero until a time policy advances it. |
 | Queue config | `max_concurrent` | Cap on concurrent in-flight validations. **Default 1**; the YAML store supports per-queue overrides. |
 | Queue config | `minimum_build_admission_interval_ms` | Minimum start-to-start spacing between logical admissions. Positive values enable general throttling; non-positive values disable it. **Default 0**. |
+| Queue config | `failure_cooldown_ms` | Delay after a runner-reported failed build before the next logical admission. Positive values enable the cooldown; non-positive values disable it. **Default 0**. |
 
 A slot is held from admit until the build goes terminal (`process → build → buildsignal`), not just while `process` runs. It is released when the Request reaches **any** terminal state and `in_flight_count` is decremented — `buildsignal` recording the build's outcome, success *or* failure, or the DLQ reconciler forcing a terminal `failed` (see [integrity](#in_flight_count-integrity)). A build *failure* frees the slot just like a success; only a Request that never terminates keeps its slot.
 

@@ -24,7 +24,7 @@ Stovepipe therefore needs two MySQL databases: a **storage** database (the `queu
 - **`inMemoryCounter`** — a process-local `counter.Counter` for sequence numbers; not durable. A real deployment uses a persistent implementation (e.g. `platform/extension/counter/mysql`).
 - **`fakeSourceControlFactory`** — seeds each queue with a deterministic single-commit history so ingest resolves a stable head URI (and re-ingesting the same queue exercises the dedup path). A real deployment supplies a VCS-backed `sourcecontrol.Factory`, which is also where a queue's promotion ref is resolved. The fake has no ref to move, so a promotion locally shows up only in the record consumer's logs.
 
-`QUEUE_CONFIG_PATH` selects the YAML-backed queue policy store. When it is unset, the server uses the built-in defaults (`max_concurrent: 1`, `gate_wait_delay_ms: 5000`, and time-based admission throttling disabled). When it is set, the configured queue names must exactly match `MQ_TENANTS`. Each configured queue can set `minimum_build_admission_interval_ms` for general start-to-start throttling. For example, `minimum_build_admission_interval_ms: 3600000` permits at most one logical admission per hour.
+`QUEUE_CONFIG_PATH` selects the YAML-backed queue policy store. When it is unset, the server uses the built-in defaults (`max_concurrent: 1`, `gate_wait_delay_ms: 5000`, and both time-based admission policies disabled). When it is set, the configured queue names must exactly match `MQ_TENANTS`. Each configured queue can set `minimum_build_admission_interval_ms` for general start-to-start throttling and `failure_cooldown_ms` for a delay after a runner-reported failed build. For example, `minimum_build_admission_interval_ms: 3600000` permits at most one logical admission per hour.
 
 ## Layout
 

@@ -472,7 +472,7 @@ func registerPrimaryControllers(
 		scope,
 		store,
 		materializer,
-		queueconfigdefault.NewStore(),
+		queueConfigs,
 		sourceControl,
 		registry,
 		stovepipemq.TopicKeyProcess,
@@ -489,7 +489,7 @@ func registerPrimaryControllers(
 	}
 	count++
 
-	buildSignalController := buildsignal.NewController(logger, scope, store, materializer, brf, registry, stovepipemq.TopicKeyBuildSignal, "stovepipe-buildsignal")
+	buildSignalController := buildsignal.NewController(logger, scope, store, materializer, queueConfigs, brf, registry, stovepipemq.TopicKeyBuildSignal, "stovepipe-buildsignal")
 	if err := c.Register(buildSignalController); err != nil {
 		return count, fmt.Errorf("failed to register buildsignal controller: %w", err)
 	}

@@ -240,6 +240,7 @@ func TestLoadQueueConfigs(t *testing.T) {
 		require.NoError(t, err)
 		assert.EqualValues(t, 1, cfg.MaxConcurrent)
 		assert.Zero(t, cfg.MinimumBuildAdmissionIntervalMs)
+		assert.Zero(t, cfg.FailureCooldownMs)
 	})
 
 	t.Run("path loads per-queue policies", func(t *testing.T) {
@@ -249,6 +250,7 @@ func TestLoadQueueConfigs(t *testing.T) {
     max_concurrent: 2
     gate_wait_delay_ms: 5000
     minimum_build_admission_interval_ms: 3600000
+    failure_cooldown_ms: 900000
 `), 0o600))
 
 		store, err := loadQueueConfigs(path)
@@ -257,6 +259,7 @@ func TestLoadQueueConfigs(t *testing.T) {
 		require.NoError(t, err)
 		assert.EqualValues(t, 2, cfg.MaxConcurrent)
 		assert.EqualValues(t, 3_600_000, cfg.MinimumBuildAdmissionIntervalMs)
+		assert.EqualValues(t, 900_000, cfg.FailureCooldownMs)
 		require.NoError(t, validateQueueConfigTenants(context.Background(), path, store, []string{"monorepo/main"}))
 		require.Error(t, validateQueueConfigTenants(context.Background(), path, store, []string{"other"}))
 	})

@@ -5,11 +5,14 @@ queues:
     gate_wait_delay_ms: 5000
     # Set to 3600000 to admit at most one build per hour.
     minimum_build_admission_interval_ms: 0
+    failure_cooldown_ms: 0
   - name: monorepo/release
     max_concurrent: 1
     gate_wait_delay_ms: 5000
     minimum_build_admission_interval_ms: 0
+    failure_cooldown_ms: 0
   - name: "monorepo/slow?buildrunner-fake=build-slow"
     max_concurrent: 1
     gate_wait_delay_ms: 5000
     minimum_build_admission_interval_ms: 0
+    failure_cooldown_ms: 0
@@ -15,6 +15,7 @@ go_library(
         "//stovepipe/core/requestlog:go_default_library",
         "//stovepipe/entity:go_default_library",
         "//stovepipe/extension/buildrunner:go_default_library",
+        "//stovepipe/extension/queueconfig:go_default_library",
         "//stovepipe/extension/storage:go_default_library",
         "@com_github_uber_go_tally//:go_default_library",
         "@org_uber_go_zap//:go_default_library",
@@ -38,6 +39,7 @@ go_test(
         "//stovepipe/entity:go_default_library",
         "//stovepipe/extension/buildrunner:go_default_library",
         "//stovepipe/extension/buildrunner/mock:go_default_library",
+        "//stovepipe/extension/queueconfig/mock:go_default_library",
         "//stovepipe/extension/storage:go_default_library",
         "//stovepipe/extension/storage/mock:go_default_library",
         "@com_github_stretchr_testify//assert:go_default_library",