From 29c148b8125182127409d0c6e6969d9b9a3913b3 Mon Sep 17 00:00:00 2001 From: Preetam Dwivedi Date: Mon, 7 Sep 2026 16:11:19 -0700 Subject: [PATCH 1/6] docs(submitqueue): add outcome predictor RFC MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Summary ### Why? Speculation ranks paths on the scorer's price, which only reflects the change (lines, files) and ignores what the run has since observed — a passed build, a failed build, batch state. The predictor design needs a written contract before the implementation lands. ### What? Adds doc/rfc/submitqueue/outcome-predictor.md: scorer vs predictor, the four YAML factors and what their values mean, which path results count as evidence, and rejected alternatives. Links it from doc/rfc/index.md. ## Test Plan Docs-only change. --- doc/rfc/index.md | 1 + doc/rfc/submitqueue/outcome-predictor.md | 103 +++++++++++++++++++++++ 2 files changed, 104 insertions(+) create mode 100644 doc/rfc/submitqueue/outcome-predictor.md diff --git a/doc/rfc/index.md b/doc/rfc/index.md index 67719f0a..4b92d701 100644 --- a/doc/rfc/index.md +++ b/doc/rfc/index.md @@ -20,6 +20,7 @@ Design documents and technical proposals, grouped by scope. Shared/cross-cutting - [Extension Contract](submitqueue/extension-contract.md) - When extensions take orchestrator identity (request/batch) and resolve granular content themselves vs. take controller-resolved data; revises the BuildRunner base/head contract - [Gateway Status and List APIs](submitqueue/status-list-api.md) - Gateway-owned request context, materialized current status, sqid or change-URI status lookup, and queue admission listing - [Speculation](submitqueue/speculation.md) - Why SubmitQueue speculates, the path/tree model, and the two pluggable seams: speculation-tree enumeration and path selection +- [Outcome Predictor](submitqueue/outcome-predictor.md) - How likely a batch is to reach Succeeded: the scorer prices the change, the predictor revises that price from the path set and batch state (`pathPassed`, `pathFailed`, `merging`, `cancelling`) - [Best-First Speculation Path Generation](submitqueue/speculation-generator-best-first.md) - The default Generator: per-head lazy streams of flip subsets merged best-first across heads, log-probability ranking, and the strict snapshot contract - [Modular Queue Wiring](submitqueue/modular-queue-wiring.md) - Declare-don't-assemble engine (`pipeline.Construct`) that unifies topic registry, controller registration, DLQ pairing, and lifecycle ordering into one typed call; services self-declare via Deps struct + Stages slice, hosts own per-queue profiles and transport diff --git a/doc/rfc/submitqueue/outcome-predictor.md b/doc/rfc/submitqueue/outcome-predictor.md new file mode 100644 index 00000000..a408810e --- /dev/null +++ b/doc/rfc/submitqueue/outcome-predictor.md @@ -0,0 +1,103 @@ +# Outcome Predictor + +How likely a batch is to reach Succeeded, given the scorer's price for the change plus what this speculate run has already observed. + +See [speculation.md](speculation.md) for batches, paths, heads, and the Speculator. This document is the price the default Generator ranks on. + +## The idea + +The **scorer** prices the change (lines, files, who wrote it). That number does not move after the batch is admitted. + +The **predictor** prices the situation. It starts from the scorer's price and revises it with facts the speculate run already holds: a path *passed*, a path *failed*, the batch is *merging*, the batch is *cancelling*. + +`bestfirst` ranks a path by the probability that every unresolved assumption holds. It now asks the predictor for that probability, not the scorer. Two heads whose changes score the same can rank differently once one of them has a *passed* build. + +They are two contracts because they answer different questions. Putting path-set evidence on `Score` was tried: every content scorer took a parameter it discarded. + +**Default is a no-op.** Every factor starts at `1`, so the predictor returns the scorer's price until someone sets a factor. + +## What a factor is + +A factor revises the scorer's price. It is not itself a probability: `10` does not mean `0.10`, and `0.3` does not mean the batch is 30% likely to succeed. + +| Value | Meaning | +| --- | --- | +| `1` | Leave the scorer's price alone (the default if the key is omitted) | +| greater than `1` | More likely to reach Succeeded | +| between `0` and `1` | Less likely to reach Succeeded | + +Config rejects `0` and negatives. There is no upper cap. + +The unconfigured scorer prices every batch at `0.5`. From that price, one factor `f` produces: + +| Factor | Price | +| --- | --- | +| `1` | 0.50 | +| `10` | ~0.91 | +| `12` | ~0.92 | +| `0.3` | ~0.23 | +| `0.25` | 0.20 | + +A scorer price of `0.6` with `pathPassed: 10` becomes about `0.94`. `merging: 12` on top of that becomes about `0.995`. + +`pathFailed: 0.3` from `0.5` becomes about `0.23`. A second *failed* path of the same kind multiplies again. `0` is rejected: it would pin the batch at probability 0 for the rest of the run. + +The arithmetic multiplies odds (`p / (1-p)`), then converts back, so the result stays in `(0, 1)` and the same factor means the same thing at any scorer price. Adding to the probability does neither. + +YAML: + +```yaml +predictor: + type: evidence + factors: + pathPassed: 10 + pathFailed: 0.3 + merging: 12 + cancelling: 0.1 +``` + +The example values above are guesses, for reading the tables. The shipped default is to omit `factors` (every factor `1`). + +## Evidence + +| YAML key | When it applies | Typical direction | +| --- | --- | --- | +| `pathPassed` | Once, if a path that assumes every dependency *succeeds* has *passed* | Up | +| `pathFailed` | Once per *failed* path that assumes every dependency *succeeds* | Down | +| `merging` | While the batch is *merging* | Up | +| `cancelling` | While the batch is *cancelling* | Down | + +`bestfirst` already treats a terminal batch as a fact (*Succeeded*, *Failed*, *Cancelled*). The predictor is not asked. *Merging* is not terminal: a merge can still fail, so how much it is worth stays a price. + +### Only the *succeeds* path counts + +The batch being priced is itself a head, so the run may have built it more than once under different assumptions about *its* dependencies. Only one of those builds is evidence. + +Take `C` depending on `B`, and `B` depending on `A`. Ranking `C`'s candidates needs the probability that `B` reaches Succeeded, so the Generator calls `Predict` with `B` and `B`'s path set. That set can hold two finished builds: + +| `B`'s path | What was compiled | +| --- | --- | +| `B` with `A` *succeeds* | `B` on top of `A`'s changes | +| `B` with `A` *fails* | `B` without them | + +`B` merges after `A` does, so the first build is a build of the code that will actually land: if it *passed*, `B` is likely to merge, and `pathPassed` applies. + +The second is a different set of changes. `B` may call something `A` introduces and fail to compile on its own — a *failed* result that says nothing about `B` merging in the normal case. Counting it would push `B` down the ranking over a build it was never going to need, while a green build of the real combination sits in the same set. + +So `pathPassed` and `pathFailed` both look only at paths that assume every dependency *succeeds*. Results on any other path are skipped. This is a filter on which results are evidence, not a check on whether an assumption came true — nothing here revisits that. + +## Rejected + +**One contract, with the evidence on `Score`.** Tried: every scorer that prices content took a parameter it discarded, and the composite forwarded one it never read. + +**One estimate over content and evidence together.** They change at different rates and need different amounts of data, and it would force every queue onto the same content scorer. + +**Adding to the probability instead of multiplying odds.** Leaves the range, needs clamping, and the same increment means different things at different prices. + +**More dimensions on the bucket table.** A second dimension squares it, a third makes it unwritable, and every cell is still a guess. + +**Putting *merging* / *cancelling* in the Generator.** Tried and reverted: a merge can fail, so nothing is settled, and how much a state is worth is a price. + +**A scoring stage.** Prices only mean anything inside the run that produced them; storing them would make them stale by construction. + +**The predictor reads the path-set store.** Cheaper plumbing, stale or split-brain snapshot. The run reads once. From 4129a5b87963ce12126cdb134b2ea35e37819dfa Mon Sep 17 00:00:00 2001 From: Preetam Dwivedi Date: Mon, 7 Sep 2026 16:53:52 -0700 Subject: [PATCH 2/6] docs(submitqueue): clarify rejected predictor alternatives ## Summary ### Why? The RFC's rejected-options list used implementation shorthand and included alternatives already explained elsewhere, making the design tradeoffs difficult to evaluate without branch context. ### What? Keep the four alternatives reviewers are likely to raise and state each rejected option, why it fails, and the selected design. Remove redundant and scorer-specific entries. --- doc/rfc/submitqueue/outcome-predictor.md | 20 ++++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-) diff --git a/doc/rfc/submitqueue/outcome-predictor.md b/doc/rfc/submitqueue/outcome-predictor.md index a408810e..1feb4163 100644 --- a/doc/rfc/submitqueue/outcome-predictor.md +++ b/doc/rfc/submitqueue/outcome-predictor.md @@ -86,18 +86,22 @@ The second is a different set of changes. `B` may call something `A` introduces So `pathPassed` and `pathFailed` both look only at paths that assume every dependency *succeeds*. Results on any other path are skipped. This is a filter on which results are evidence, not a check on whether an assumption came true — nothing here revisits that. -## Rejected +## Rejected alternatives -**One contract, with the evidence on `Score`.** Tried: every scorer that prices content took a parameter it discarded, and the composite forwarded one it never read. +Design choices a reader might suggest after the sections above. Each names the alternative, why it fails here, and what this RFC does instead. -**One estimate over content and evidence together.** They change at different rates and need different amounts of data, and it would force every queue onto the same content scorer. +### Fold path evidence into `Score` -**Adding to the probability instead of multiplying odds.** Leaves the range, needs clamping, and the same increment means different things at different prices. +Give `Score` the speculate run's path sets so one call returns a situation-aware price. We tried it: content scorers took the parameter and discarded it; the composite forwarded evidence it never read. **Instead:** keep `Score` for the change; add `Predictor` for the situation (see [The idea](#the-idea)). -**More dimensions on the bucket table.** A second dimension squares it, a third makes it unwritable, and every cell is still a guess. +### One model for content and evidence -**Putting *merging* / *cancelling* in the Generator.** Tried and reverted: a merge can fail, so nothing is settled, and how much a state is worth is a price. +Train or tune a single estimate over diff shape and build outcomes together. Content signals and situation signals change at different rates, need different amounts of data, and would force every queue onto the same content scorer. **Instead:** scorer stays per-queue; evidence factors layer on in YAML. -**A scoring stage.** Prices only mean anything inside the run that produced them; storing them would make them stale by construction. +### Treat *merging* and *cancelling* as settled in the Generator -**The predictor reads the path-set store.** Cheaper plumbing, stale or split-brain snapshot. The run reads once. +Rank a *merging* batch like Succeeded and a *cancelling* batch like Cancelled. We tried and reverted: a merge can still fail, so the rank was wrong once outcomes diverged. **Instead:** only terminal states short-circuit in the Generator; *merging* and *cancelling* are predictor factors (see [Evidence](#evidence)). + +### Let the predictor read the path-set store + +`Predict` loads path sets from storage on each call — smaller API, fewer parameters. Each call can see a different snapshot mid-run (stale or split-brain relative to the rank the Generator is building). **Instead:** the speculate run reads path sets once per dependency and passes them in. From ac6684ba837ab34023646adff0bedb1098da2a5d Mon Sep 17 00:00:00 2001 From: Preetam Dwivedi Date: Mon, 7 Sep 2026 17:26:37 -0700 Subject: [PATCH 3/6] docs(submitqueue): reconcile predictor RFCs ## Summary ### Why? The outcome predictor RFC described failed-path evidence and ranking differently from the path-set identity and best-first behavior, while older speculation RFCs still assigned dynamic pricing to the scorer. ### What? Document that prediction applies to unresolved dependencies, failed all-succeeds evidence applies once, neutral factors preserve exact scorer prices, and factors must be finite. Update the existing speculation RFCs to consistently describe predictor-based ranking and state-aware pricing. # Conflicts: # doc/rfc/submitqueue/speculation-generator-best-first.md # Please enter the commit message for your changes. Lines starting # with '#' will be kept; you may remove them yourself if you want to. # An empty message aborts the commit. # # interactive rebase in progress; onto 91ddcce1 # Last commands done (3 commands done): # pick 32271325 # docs(submitqueue): clarify rejected predictor alternatives # pick 7d17f2dd # docs(submitqueue): reconcile predictor RFCs # Next commands to do (2 remaining commands): # pick 9696e219 # docs(submitqueue): complete outcome predictor RFC # pick f98b8e87 # docs(submitqueue): combine predictor into Scorer # You are currently rebasing branch 'preetam/outcome-predictor-rfc' on '91ddcce1'. # # Changes to be committed: # modified: doc/rfc/submitqueue/outcome-predictor.md # modified: doc/rfc/submitqueue/speculation-generator-best-first.md # modified: doc/rfc/submitqueue/speculation.md # --- doc/rfc/submitqueue/outcome-predictor.md | 18 +++++++++--------- .../speculation-generator-best-first.md | 10 +++++----- doc/rfc/submitqueue/speculation.md | 2 +- 3 files changed, 15 insertions(+), 15 deletions(-) diff --git a/doc/rfc/submitqueue/outcome-predictor.md b/doc/rfc/submitqueue/outcome-predictor.md index 1feb4163..d19ed57d 100644 --- a/doc/rfc/submitqueue/outcome-predictor.md +++ b/doc/rfc/submitqueue/outcome-predictor.md @@ -2,15 +2,15 @@ How likely a batch is to reach Succeeded, given the scorer's price for the change plus what this speculate run has already observed. -See [speculation.md](speculation.md) for batches, paths, heads, and the Speculator. This document is the price the default Generator ranks on. +See [speculation.md](speculation.md) for batches, paths, heads, and the Speculator. This document specifies the dependency probability the default Generator uses to rank paths. ## The idea -The **scorer** prices the change (lines, files, who wrote it). That number does not move after the batch is admitted. +The **scorer** prices the change from content signals such as its size. That number does not move after the batch is admitted. The **predictor** prices the situation. It starts from the scorer's price and revises it with facts the speculate run already holds: a path *passed*, a path *failed*, the batch is *merging*, the batch is *cancelling*. -`bestfirst` ranks a path by the probability that every unresolved assumption holds. It now asks the predictor for that probability, not the scorer. Two heads whose changes score the same can rank differently once one of them has a *passed* build. +`bestfirst` ranks a path by the probability that every unresolved dependency assumption holds. For each dependency, the predictor returns the probability of *succeeds* and `bestfirst` uses its complement for *fails*. Equal scorer prices can therefore produce different rankings for later heads once one dependency has a *passed* build. They are two contracts because they answer different questions. Putting path-set evidence on `Score` was tried: every content scorer took a parameter it discarded. @@ -24,9 +24,9 @@ A factor revises the scorer's price. It is not itself a probability: `10` does n | --- | --- | | `1` | Leave the scorer's price alone (the default if the key is omitted) | | greater than `1` | More likely to reach Succeeded | -| between `0` and `1` | Less likely to reach Succeeded | +| between `0` and `1` (exclusive) | Less likely to reach Succeeded | -Config rejects `0` and negatives. There is no upper cap. +Config accepts any finite value greater than `0`; there is no finite upper cap. The unconfigured scorer prices every batch at `0.5`. From that price, one factor `f` produces: @@ -40,9 +40,9 @@ The unconfigured scorer prices every batch at `0.5`. From that price, one factor A scorer price of `0.6` with `pathPassed: 10` becomes about `0.94`. `merging: 12` on top of that becomes about `0.995`. -`pathFailed: 0.3` from `0.5` becomes about `0.23`. A second *failed* path of the same kind multiplies again. `0` is rejected: it would pin the batch at probability 0 for the rest of the run. +`pathFailed: 0.3` from `0.5` becomes about `0.23`. It applies at most once because the path set has one current entry for the all-*succeeds* path; retry attempts replace that entry rather than adding evidence. `0` is rejected because it would pin matching batches at probability 0. -The arithmetic multiplies odds (`p / (1-p)`), then converts back, so the result stays in `(0, 1)` and the same factor means the same thing at any scorer price. Adding to the probability does neither. +When a factor revises the price, the arithmetic multiplies odds (`p / (1-p)`) and converts back, so the result stays in `(0, 1)` and the same factor means the same thing at any scorer price. Neutral factors return the scorer's price unchanged, including `0` or `1`. Adding to the probability provides neither property. YAML: @@ -63,7 +63,7 @@ The example values above are guesses, for reading the tables. The shipped defaul | YAML key | When it applies | Typical direction | | --- | --- | --- | | `pathPassed` | Once, if a path that assumes every dependency *succeeds* has *passed* | Up | -| `pathFailed` | Once per *failed* path that assumes every dependency *succeeds* | Down | +| `pathFailed` | Once, if the path that assumes every dependency *succeeds* has *failed* | Down | | `merging` | While the batch is *merging* | Up | | `cancelling` | While the batch is *cancelling* | Down | @@ -104,4 +104,4 @@ Rank a *merging* batch like Succeeded and a *cancelling* batch like Cancelled. W ### Let the predictor read the path-set store -`Predict` loads path sets from storage on each call — smaller API, fewer parameters. Each call can see a different snapshot mid-run (stale or split-brain relative to the rank the Generator is building). **Instead:** the speculate run reads path sets once per dependency and passes them in. +`Predict` loads path sets from storage on each call — smaller API, fewer parameters. Each call can see a different snapshot mid-run (stale or split-brain relative to the rank the Generator is building). **Instead:** the speculate run reads all path sets as one snapshot and passes the matching set for each dependency. diff --git a/doc/rfc/submitqueue/speculation-generator-best-first.md b/doc/rfc/submitqueue/speculation-generator-best-first.md index 5558109f..fd36bb14 100644 --- a/doc/rfc/submitqueue/speculation-generator-best-first.md +++ b/doc/rfc/submitqueue/speculation-generator-best-first.md @@ -36,7 +36,7 @@ D.Dependencies = [] C contains A because C directly conflicts with A, not because B depends on A. The generator uses these stored direct dependencies and does not compute a transitive closure. -The scorer estimates: +The predictor estimates: | Dependency | Success | Failure | | --- | ---: | ---: | @@ -49,11 +49,11 @@ The batch being built is written before its assumptions. For example, `C [A succ `Generate` receives the queue's live batches as a snapshot and takes it as given. A well-formed snapshot carries unique, non-empty batch IDs, includes every batch a head's direct dependencies reference, and gives no head an empty, duplicate, or self dependency. Those are preconditions the caller owns, established where the snapshot is assembled. The generator does not re-check them: it is on the hot path of every run, the checks it could make are the ones an assembled-correctly snapshot can never fail, and paying for them here only spreads the same contract across two places. A malformed snapshot yields undefined candidates rather than an error. -A dependency the generator cannot price is the one bad input it absorbs, because it arrives from the injected scorer rather than from the caller and there is no earlier point that could catch it. Three cases take the same 0.95 default: a score outside `[0, 1]` or `NaN`, a scorer call that returned an error, and a dependency the snapshot never carried. The default is optimistic on purpose, so a dependency nobody could estimate keeps its head's preferred path near the front instead of burying it or failing the whole run on one number — and failing the whole run is the real hazard, because `Generate` seeds the heap for every head at once, so one unpriceable dependency would otherwise cost the queue every candidate it had. A batch absent from the snapshot is never passed to the scorer at all: it would resolve to a zero-valued batch belonging to no queue, so scoring it would price some other batch entirely or fail on the empty queue name. Any deliberate defaulting still belongs to the scorer implementation, which knows what information it does and does not have; this is only the floor under it. +A dependency the generator cannot price is the one bad input it absorbs, because it arrives from the injected predictor rather than from the caller and there is no earlier point that could catch it. Three cases take the same 0.95 default: a probability outside `[0, 1]` or `NaN`, a predictor call that returned an error, and a dependency the snapshot never carried. The default is optimistic on purpose, so a dependency nobody could estimate keeps its head's preferred path near the front instead of burying it or failing the whole run on one number — and failing the whole run is the real hazard, because `Generate` seeds the heap for every head at once, so one unpriceable dependency would otherwise cost the queue every candidate it had. A batch absent from the snapshot is never passed to the predictor at all: it would resolve to a zero-valued batch belonging to no queue, so predicting it would price some other batch entirely or fail on the empty queue name. Any deliberate defaulting still belongs to the predictor implementation, which knows what information it does and does not have; this is only the floor under it. ## Step 1: `Generate` prepares each head -`Generate` scores each unique unresolved dependency once, however many heads wait on it. If A appears in both B's and C's dependency lists, the scorer is still called for A only once. +`Generate` predicts each unique unresolved dependency once, however many heads wait on it. If A appears in both B's and C's dependency lists, the predictor is still called for A only once. For each unresolved direct dependency, `Generate` records: @@ -420,7 +420,7 @@ A and D tie at 1.0, so batch ID puts A first. Other exact ties prefer fewer flip `Generate` must eagerly: -- score every unique unresolved direct dependency needed by an eligible head, substituting the default for any score that is not a probability; +- predict every unique unresolved direct dependency needed by an eligible head, substituting the default for an unusable probability; - choose each unresolved dependency's preferred assumption and calculate its `flipCost`; and - total the best score for every head. @@ -449,7 +449,7 @@ The ordering stays the same. `CandidatePath.RankingScore` contains this logarith - `Cancelling` remains undecided because cancellation may lose a race with completion. - `Landing` also remains undecided, because a land can fail. It is tempting to treat it as committed to landing and skip the scorer call, but that puts a state-specific policy inside the search: whether a path betting against a landing batch is worth funding is a question of price, and price belongs to the scorer. The allocator draws the same line — "no batch state enters this decision" — and the generator holds it too. Nothing is lost by staying open: a single passed path still waits for the land result, while passed paths covering every outcome let the controller bypass the dependency (see [speculation.md](speculation.md)). Funding the unlikely side spends budget, which is the allocator's to ration. - A fixed assumption stays in the returned path but contributes probability 1 and has no flip. -- A shared dependency is scored once per run. +- A shared dependency is predicted once per run. ## Why the algorithm works diff --git a/doc/rfc/submitqueue/speculation.md b/doc/rfc/submitqueue/speculation.md index 526bfa9c..9b7622ea 100644 --- a/doc/rfc/submitqueue/speculation.md +++ b/doc/rfc/submitqueue/speculation.md @@ -107,7 +107,7 @@ The one extension. It decides *which paths to build and which running ones to ca The default Speculator is composed from two swappable interfaces — a **Generator** and an **Allocator** — so scoring and preemption policy can vary independently. They are composition points inside the default implementation, not controller-facing extensions: the controller depends only on the Speculator contract, and an alternate Speculator need not use or expose this split. The default opens the Generator's candidate stream over the batches, then hands that stream and the path sets to the Allocator. -- **Generator** — yields the queue's candidate paths as one iterator across heads in `BatchStateSpeculating`. *Contract:* every candidate has a Speculating head and is coherent; none repeats or contradicts a resolved fact. Ranking is implementation-defined — the Generator may compute it directly, call an injected scorer extension, or use other injected data — and the score it carries is meaningful only within the run. The Allocator consumes the iterator in the order the Generator yields it and does not interpret the score. *Default:* `bestfirst` ranks best-first by the probability that a path's assumptions all hold. +- **Generator** — yields the queue's candidate paths as one iterator across heads in `BatchStateSpeculating`. *Contract:* every candidate has a Speculating head and is coherent; none repeats or contradicts a resolved fact. Ranking is implementation-defined — the Generator may compute it directly, call an injected pricing extension, or use other injected data — and the score it carries is meaningful only within the run. The Allocator consumes the iterator in the order the Generator yields it and does not interpret the score. *Default:* `bestfirst` asks the predictor for each unresolved dependency's probability of reaching Succeeded, then ranks paths by the probability that all their assumptions hold. - **Allocator** — spends the build budget (the queue's cap on concurrent builds) over the iterator. *Contract:* it pulls in order until the budget fills and matches candidates to existing paths by ID, so a pending or building path keeps the slot it already holds rather than starting a second attempt, and a candidate whose path is already terminal in the path sets is skipped rather than rebuilt; pending dispatches are replayed by the controller as described above. Pending, building, and cancelling paths charge the budget (a cancelling build holds CI until terminal), while terminal ones charge none. Cancellation is best-effort, so the Allocator does not spend capacity it merely expects a cancel to release and risk exceeding the hard CI cap. *Default:* the sticky policy fills only free slots and leaves in-flight builds running; a preempting policy cancels in-flight paths below the funded set. Budget is the only rationing lever — there is no ranking-score floor. A build cancelled to make room still charges budget until its cancel reaches terminal and publishes dirty, so the next run funds the released slot — the queue converges over successive ticks rather than oversubscribing in a single pass. ### Extension APIs From 640875c16f7137ecc5a3078971e02acf52356f43 Mon Sep 17 00:00:00 2001 From: Preetam Dwivedi Date: Mon, 7 Sep 2026 17:56:02 -0700 Subject: [PATCH 4/6] docs(submitqueue): complete outcome predictor RFC Document per-queue factor inheritance and the shared batch/path-set snapshot consumed by best-first ranking. --- doc/rfc/submitqueue/outcome-predictor.md | 2 ++ doc/rfc/submitqueue/speculation-generator-best-first.md | 2 +- doc/rfc/submitqueue/speculation.md | 8 ++++++-- 3 files changed, 9 insertions(+), 3 deletions(-) diff --git a/doc/rfc/submitqueue/outcome-predictor.md b/doc/rfc/submitqueue/outcome-predictor.md index d19ed57d..d4402a10 100644 --- a/doc/rfc/submitqueue/outcome-predictor.md +++ b/doc/rfc/submitqueue/outcome-predictor.md @@ -58,6 +58,8 @@ predictor: The example values above are guesses, for reading the tables. The shipped default is to omit `factors` (every factor `1`). +Profiles may set factors under `defaults.predictor` and revise them per queue. An omitted key keeps the inherited value — from defaults, or `1` when neither side named it. A queue `predictor` block overlays only the keys it names; it does not replace the whole map. An omitted `predictor` block on a queue inherits the defaults entirely, so every factor stays `1` until someone sets one. + ## Evidence | YAML key | When it applies | Typical direction | diff --git a/doc/rfc/submitqueue/speculation-generator-best-first.md b/doc/rfc/submitqueue/speculation-generator-best-first.md index fd36bb14..6b10574f 100644 --- a/doc/rfc/submitqueue/speculation-generator-best-first.md +++ b/doc/rfc/submitqueue/speculation-generator-best-first.md @@ -47,7 +47,7 @@ The batch being built is written before its assumptions. For example, `C [A succ ## The snapshot is a caller precondition -`Generate` receives the queue's live batches as a snapshot and takes it as given. A well-formed snapshot carries unique, non-empty batch IDs, includes every batch a head's direct dependencies reference, and gives no head an empty, duplicate, or self dependency. Those are preconditions the caller owns, established where the snapshot is assembled. The generator does not re-check them: it is on the hot path of every run, the checks it could make are the ones an assembled-correctly snapshot can never fail, and paying for them here only spreads the same contract across two places. A malformed snapshot yields undefined candidates rather than an error. +`Generate` receives the queue's live batches and path sets as one snapshot and takes it as given. Path sets are what each batch's builds have done so far — at most one per head, none for a batch nothing has speculated on. The speculate controller assembles both halves once per run and never re-reads mid-run; see [outcome-predictor.md](outcome-predictor.md) for why the predictor does not load them itself. A well-formed snapshot carries unique, non-empty batch IDs, includes every batch a head's direct dependencies reference, and gives no head an empty, duplicate, or self dependency. Those are preconditions the caller owns, established where the snapshot is assembled. The generator does not re-check them: it is on the hot path of every run, the checks it could make are the ones an assembled-correctly snapshot can never fail, and paying for them here only spreads the same contract across two places. A malformed snapshot yields undefined candidates rather than an error. A dependency the generator cannot price is the one bad input it absorbs, because it arrives from the injected predictor rather than from the caller and there is no earlier point that could catch it. Three cases take the same 0.95 default: a probability outside `[0, 1]` or `NaN`, a predictor call that returned an error, and a dependency the snapshot never carried. The default is optimistic on purpose, so a dependency nobody could estimate keeps its head's preferred path near the front instead of burying it or failing the whole run on one number — and failing the whole run is the real hazard, because `Generate` seeds the heap for every head at once, so one unpriceable dependency would otherwise cost the queue every candidate it had. A batch absent from the snapshot is never passed to the predictor at all: it would resolve to a zero-valued batch belonging to no queue, so predicting it would price some other batch entirely or fail on the empty queue name. Any deliberate defaulting still belongs to the predictor implementation, which knows what information it does and does not have; this is only the floor under it. diff --git a/doc/rfc/submitqueue/speculation.md b/doc/rfc/submitqueue/speculation.md index 9b7622ea..a9f1ed27 100644 --- a/doc/rfc/submitqueue/speculation.md +++ b/doc/rfc/submitqueue/speculation.md @@ -105,7 +105,7 @@ The one extension. It decides *which paths to build and which running ones to ca ### The default Speculator -The default Speculator is composed from two swappable interfaces — a **Generator** and an **Allocator** — so scoring and preemption policy can vary independently. They are composition points inside the default implementation, not controller-facing extensions: the controller depends only on the Speculator contract, and an alternate Speculator need not use or expose this split. The default opens the Generator's candidate stream over the batches, then hands that stream and the path sets to the Allocator. +The default Speculator is composed from two swappable interfaces — a **Generator** and an **Allocator** — so ranking and preemption policy can vary independently. They are composition points inside the default implementation, not controller-facing extensions: the controller depends only on the Speculator contract, and an alternate Speculator need not use or expose this split. The default opens the Generator's candidate stream over the batches, then hands that stream and the path sets to the Allocator. - **Generator** — yields the queue's candidate paths as one iterator across heads in `BatchStateSpeculating`. *Contract:* every candidate has a Speculating head and is coherent; none repeats or contradicts a resolved fact. Ranking is implementation-defined — the Generator may compute it directly, call an injected pricing extension, or use other injected data — and the score it carries is meaningful only within the run. The Allocator consumes the iterator in the order the Generator yields it and does not interpret the score. *Default:* `bestfirst` asks the predictor for each unresolved dependency's probability of reaching Succeeded, then ranks paths by the probability that all their assumptions hold. - **Allocator** — spends the build budget (the queue's cap on concurrent builds) over the iterator. *Contract:* it pulls in order until the budget fills and matches candidates to existing paths by ID, so a pending or building path keeps the slot it already holds rather than starting a second attempt, and a candidate whose path is already terminal in the path sets is skipped rather than rebuilt; pending dispatches are replayed by the controller as described above. Pending, building, and cancelling paths charge the budget (a cancelling build holds CI until terminal), while terminal ones charge none. Cancellation is best-effort, so the Allocator does not spend capacity it merely expects a cancel to release and risk exceeding the hard CI cap. *Default:* the sticky policy fills only free slots and leaves in-flight builds running; a preempting policy cancels in-flight paths below the funded set. Budget is the only rationing lever — there is no ranking-score floor. A build cancelled to make room still charges budget until its cancel reaches terminal and publishes dirty, so the next run funds the released slot — the queue converges over successive ticks rather than oversubscribing in a single pass. @@ -118,4 +118,8 @@ Signatures live in code and are not copied here, so they cannot drift. This sect **Speculator** — [`submitqueue/extension/speculation/speculator`](../../../submitqueue/extension/speculation/speculator/README.md). `Speculate` takes one queue snapshot (the batches and their path sets) and returns the build and cancel actions it proposes; a path it wants left alone has no entry. Actions must target Speculating heads. Verdicts stay controller-owned, so there is no land or fail action. -**Generator and Allocator** — [`generator`](../../../submitqueue/extension/speculation/generator/README.md) and [`allocator`](../../../submitqueue/extension/speculation/allocator/README.md), the two composition points inside the default Speculator. The Generator opens a pull-based stream of candidate paths over the batches; the Allocator spends the build budget over that stream, reconciling it against the path sets. Both abort on a cancelled context. +**Scorer** — [`submitqueue/extension/speculation/scorer`](../../../submitqueue/extension/speculation/scorer/README.md). Prices a batch's change from content signals. The default pipeline does not rank on it directly; the queue's predictor is built over it. + +**Predictor** — [`submitqueue/extension/speculation/predictor`](../../../submitqueue/extension/speculation/predictor/README.md). Revises the scorer's price with path-set evidence and batch state. The default `bestfirst` generator is built over it. See [outcome-predictor.md](outcome-predictor.md). + +**Generator and Allocator** — [`generator`](../../../submitqueue/extension/speculation/generator/README.md) and [`allocator`](../../../submitqueue/extension/speculation/allocator/README.md), the two composition points inside the default Speculator. The Generator opens a pull-based stream of candidate paths over the batches and path sets; the Allocator spends the build budget over that stream, reconciling it against the path sets. Both abort on a cancelled context. From f713d9a81429fd72a7ecad3ec80b6d8b6842653b Mon Sep 17 00:00:00 2001 From: Preetam Dwivedi Date: Mon, 14 Sep 2026 12:26:10 -0700 Subject: [PATCH 5/6] docs(submitqueue): combine predictor into Scorer Fold ranking into one Score(ctx, batch, paths). Evidence is the scorer; heuristic/composite are the nested base. Frame the revision as a configured logit-linear model. --- doc/rfc/index.md | 2 +- doc/rfc/submitqueue/outcome-predictor.md | 75 ++++++++++++------- .../speculation-generator-best-first.md | 12 +-- doc/rfc/submitqueue/speculation.md | 6 +- 4 files changed, 55 insertions(+), 40 deletions(-) diff --git a/doc/rfc/index.md b/doc/rfc/index.md index 4b92d701..1f0a5e95 100644 --- a/doc/rfc/index.md +++ b/doc/rfc/index.md @@ -20,7 +20,7 @@ Design documents and technical proposals, grouped by scope. Shared/cross-cutting - [Extension Contract](submitqueue/extension-contract.md) - When extensions take orchestrator identity (request/batch) and resolve granular content themselves vs. take controller-resolved data; revises the BuildRunner base/head contract - [Gateway Status and List APIs](submitqueue/status-list-api.md) - Gateway-owned request context, materialized current status, sqid or change-URI status lookup, and queue admission listing - [Speculation](submitqueue/speculation.md) - Why SubmitQueue speculates, the path/tree model, and the two pluggable seams: speculation-tree enumeration and path selection -- [Outcome Predictor](submitqueue/outcome-predictor.md) - How likely a batch is to reach Succeeded: the scorer prices the change, the predictor revises that price from the path set and batch state (`pathPassed`, `pathFailed`, `merging`, `cancelling`) +- [Outcome Scorer](submitqueue/outcome-predictor.md) - How likely a batch is to reach Succeeded: `Score(ctx, batch, paths)` as a logit-linear model — a base content price plus YAML weights on path and batch evidence (`pathPassed`, `pathFailed`, `merging`, `cancelling`) - [Best-First Speculation Path Generation](submitqueue/speculation-generator-best-first.md) - The default Generator: per-head lazy streams of flip subsets merged best-first across heads, log-probability ranking, and the strict snapshot contract - [Modular Queue Wiring](submitqueue/modular-queue-wiring.md) - Declare-don't-assemble engine (`pipeline.Construct`) that unifies topic registry, controller registration, DLQ pairing, and lifecycle ordering into one typed call; services self-declare via Deps struct + Stages slice, hosts own per-queue profiles and transport diff --git a/doc/rfc/submitqueue/outcome-predictor.md b/doc/rfc/submitqueue/outcome-predictor.md index d4402a10..f3d50b6c 100644 --- a/doc/rfc/submitqueue/outcome-predictor.md +++ b/doc/rfc/submitqueue/outcome-predictor.md @@ -1,34 +1,43 @@ -# Outcome Predictor +# Outcome Scorer -How likely a batch is to reach Succeeded, given the scorer's price for the change plus what this speculate run has already observed. +How likely a batch is to reach Succeeded, given the change's content plus what this speculate run has already observed. See [speculation.md](speculation.md) for batches, paths, heads, and the Speculator. This document specifies the dependency probability the default Generator uses to rank paths. ## The idea -The **scorer** prices the change from content signals such as its size. That number does not move after the batch is admitted. +`bestfirst` ranks a path by the probability that every unresolved dependency assumption holds. For each dependency it calls **one** extension: `Scorer.Score(ctx, batch, paths)` — the probability of *succeeds*; it uses the complement for *fails*. -The **predictor** prices the situation. It starts from the scorer's price and revises it with facts the speculate run already holds: a path *passed*, a path *failed*, the batch is *merging*, the batch is *cancelling*. +That number has two parts, composed as one scorer: -`bestfirst` ranks a path by the probability that every unresolved dependency assumption holds. For each dependency, the predictor returns the probability of *succeeds* and `bestfirst` uses its complement for *fails*. Equal scorer prices can therefore produce different rankings for later heads once one dependency has a *passed* build. +1. A **base** price for the change from content signals such as its size. Heuristic and composite supply this. They implement the same `Score` and ignore `paths`. +2. An **evidence** layer that revises the base with facts the speculate run already holds: a path *passed*, a path *failed*, the batch is *merging*, the batch is *cancelling*. -They are two contracts because they answer different questions. Putting path-set evidence on `Score` was tried: every content scorer took a parameter it discarded. +Evidence is the scorer the queue exposes. The base sits under it. There is no sibling Predictor factory. -**Default is a no-op.** Every factor starts at `1`, so the predictor returns the scorer's price until someone sets a factor. +This is a logit-linear model (a GLM with a logit link) with configured weights: + +``` +logit(p') = logit(p_base) + sum_i w_i x_i +``` + +`p_base` is the heuristic or composite price (the offset). `x_i` are binary features from the path set and batch state. `w_i = log(factor_i)` are YAML weights, not a fitted likelihood. + +**Default is a no-op.** Every factor starts at `1` (`w = 0`), so `Score` returns the base price until someone sets a factor. ## What a factor is -A factor revises the scorer's price. It is not itself a probability: `10` does not mean `0.10`, and `0.3` does not mean the batch is 30% likely to succeed. +A factor is an odds multiplier for one feature. It is not itself a probability: `10` does not mean `0.10`, and `0.3` does not mean the batch is 30% likely to succeed. Equivalently `w = log(factor)` on the logit. -| Value | Meaning | -| --- | --- | -| `1` | Leave the scorer's price alone (the default if the key is omitted) | -| greater than `1` | More likely to reach Succeeded | -| between `0` and `1` (exclusive) | Less likely to reach Succeeded | +| Value | Weight | Meaning | +| --- | --- | --- | +| `1` | `0` | Leave the base price alone (the default if the key is omitted) | +| greater than `1` | positive | More likely to reach Succeeded | +| between `0` and `1` (exclusive) | negative | Less likely to reach Succeeded | Config accepts any finite value greater than `0`; there is no finite upper cap. -The unconfigured scorer prices every batch at `0.5`. From that price, one factor `f` produces: +The unconfigured base prices every batch at `0.5`. From that price, one factor `f` produces `sigmoid(logit(0.5) + log(f)) = f / (1 + f)`: | Factor | Price | | --- | --- | @@ -38,27 +47,31 @@ The unconfigured scorer prices every batch at `0.5`. From that price, one factor | `0.3` | ~0.23 | | `0.25` | 0.20 | -A scorer price of `0.6` with `pathPassed: 10` becomes about `0.94`. `merging: 12` on top of that becomes about `0.995`. +A base price of `0.6` with `pathPassed: 10` becomes about `0.94`. `merging: 12` on top of that becomes about `0.995`. `pathFailed: 0.3` from `0.5` becomes about `0.23`. It applies at most once because the path set has one current entry for the all-*succeeds* path; retry attempts replace that entry rather than adding evidence. `0` is rejected because it would pin matching batches at probability 0. -When a factor revises the price, the arithmetic multiplies odds (`p / (1-p)`) and converts back, so the result stays in `(0, 1)` and the same factor means the same thing at any scorer price. Neutral factors return the scorer's price unchanged, including `0` or `1`. Adding to the probability provides neither property. +Odds revision keeps the result in `(0, 1)` and makes the same factor mean the same thing at any base price. Neutral factors return the base unchanged, including `0` or `1`. Adding to the probability provides neither property. A clamp at a small epsilon is a numerical guard around those endpoints, not part of the linear predictor. + +This is **not** a fitted GLM: weights are configured, not trained; there is no extra intercept (`p_base` is the offset); features are hand-defined, not learned. -YAML: +YAML. Evidence is the scorer; the content provider is `base`: ```yaml -predictor: +scorer: type: evidence factors: pathPassed: 10 pathFailed: 0.3 merging: 12 cancelling: 0.1 + base: + type: heuristic ``` -The example values above are guesses, for reading the tables. The shipped default is to omit `factors` (every factor `1`). +The example values above are guesses, for reading the tables. The shipped default is to omit `factors` (every factor `1`). Omitted `type` is `evidence`. Omitted `base` is the default heuristic. `type: heuristic` and `type: composite` belong on `base` (and on composite `components`), not at the top level. -Profiles may set factors under `defaults.predictor` and revise them per queue. An omitted key keeps the inherited value — from defaults, or `1` when neither side named it. A queue `predictor` block overlays only the keys it names; it does not replace the whole map. An omitted `predictor` block on a queue inherits the defaults entirely, so every factor stays `1` until someone sets one. +Profiles may set `factors` under `defaults.scorer` and revise them per queue. An omitted key keeps the inherited value — from defaults, or `1` when neither side named it. A queue `scorer` block overlays named factor keys; a present `base` replaces the default base wholesale. ## Evidence @@ -69,13 +82,13 @@ Profiles may set factors under `defaults.predictor` and revise them per queue. A | `merging` | While the batch is *merging* | Up | | `cancelling` | While the batch is *cancelling* | Down | -`bestfirst` already treats a terminal batch as a fact (*Succeeded*, *Failed*, *Cancelled*). The predictor is not asked. *Merging* is not terminal: a merge can still fail, so how much it is worth stays a price. +`bestfirst` already treats a terminal batch as a fact (*Succeeded*, *Failed*, *Cancelled*). The scorer is not asked. *Merging* is not terminal: a merge can still fail, so how much it is worth stays a price. ### Only the *succeeds* path counts The batch being priced is itself a head, so the run may have built it more than once under different assumptions about *its* dependencies. Only one of those builds is evidence. -Take `C` depending on `B`, and `B` depending on `A`. Ranking `C`'s candidates needs the probability that `B` reaches Succeeded, so the Generator calls `Predict` with `B` and `B`'s path set. That set can hold two finished builds: +Take `C` depending on `B`, and `B` depending on `A`. Ranking `C`'s candidates needs the probability that `B` reaches Succeeded, so the Generator calls `Score` with `B` and `B`'s path set. That set can hold two finished builds: | `B`'s path | What was compiled | | --- | --- | @@ -92,18 +105,22 @@ So `pathPassed` and `pathFailed` both look only at paths that assume every depen Design choices a reader might suggest after the sections above. Each names the alternative, why it fails here, and what this RFC does instead. -### Fold path evidence into `Score` +### A sibling Predictor factory + +Keep `Score(ctx, batch)` for content and add `Predict(ctx, batch, paths)` as a second extension. Ranking only needs one number per unresolved batch; two factories duplicate the per-queue seam. **Instead:** one `Scorer.Score(ctx, batch, paths)`. Evidence is a scorer implementation that wraps a base. + +### Put `paths` only on heuristic and composite -Give `Score` the speculate run's path sets so one call returns a situation-aware price. We tried it: content scorers took the parameter and discarded it; the composite forwarded evidence it never read. **Instead:** keep `Score` for the change; add `Predictor` for the situation (see [The idea](#the-idea)). +Every content backend reads the path set. We tried forwarding path sets through composite: components discarded them. **Instead:** heuristic and composite implement the same `Score` and ignore `paths`. Evidence is the layer that reads them. -### One model for content and evidence +### One fitted model for content and evidence -Train or tune a single estimate over diff shape and build outcomes together. Content signals and situation signals change at different rates, need different amounts of data, and would force every queue onto the same content scorer. **Instead:** scorer stays per-queue; evidence factors layer on in YAML. +Train a single estimate over diff shape and build outcomes together. Content signals and situation signals change at different rates, need different amounts of data, and would force every queue onto the same content scorer. **Instead:** the base stays per-queue; evidence weights layer on in YAML. `p_base` remains the GLM offset if someone later fits `w`. ### Treat *merging* and *cancelling* as settled in the Generator -Rank a *merging* batch like Succeeded and a *cancelling* batch like Cancelled. We tried and reverted: a merge can still fail, so the rank was wrong once outcomes diverged. **Instead:** only terminal states short-circuit in the Generator; *merging* and *cancelling* are predictor factors (see [Evidence](#evidence)). +Rank a *merging* batch like Succeeded and a *cancelling* batch like Cancelled. We tried and reverted: a merge can still fail, so the rank was wrong once outcomes diverged. **Instead:** only terminal states short-circuit in the Generator; *merging* and *cancelling* are scorer features (see [Evidence](#evidence)). -### Let the predictor read the path-set store +### Let the scorer read the path-set store -`Predict` loads path sets from storage on each call — smaller API, fewer parameters. Each call can see a different snapshot mid-run (stale or split-brain relative to the rank the Generator is building). **Instead:** the speculate run reads all path sets as one snapshot and passes the matching set for each dependency. +`Score` loads path sets from storage on each call — smaller API, fewer parameters. Each call can see a different snapshot mid-run (stale or split-brain relative to the rank the Generator is building). **Instead:** the speculate run reads all path sets as one snapshot and passes the matching set for each dependency. diff --git a/doc/rfc/submitqueue/speculation-generator-best-first.md b/doc/rfc/submitqueue/speculation-generator-best-first.md index 6b10574f..6b81105f 100644 --- a/doc/rfc/submitqueue/speculation-generator-best-first.md +++ b/doc/rfc/submitqueue/speculation-generator-best-first.md @@ -36,7 +36,7 @@ D.Dependencies = [] C contains A because C directly conflicts with A, not because B depends on A. The generator uses these stored direct dependencies and does not compute a transitive closure. -The predictor estimates: +The scorer estimates: | Dependency | Success | Failure | | --- | ---: | ---: | @@ -47,13 +47,13 @@ The batch being built is written before its assumptions. For example, `C [A succ ## The snapshot is a caller precondition -`Generate` receives the queue's live batches and path sets as one snapshot and takes it as given. Path sets are what each batch's builds have done so far — at most one per head, none for a batch nothing has speculated on. The speculate controller assembles both halves once per run and never re-reads mid-run; see [outcome-predictor.md](outcome-predictor.md) for why the predictor does not load them itself. A well-formed snapshot carries unique, non-empty batch IDs, includes every batch a head's direct dependencies reference, and gives no head an empty, duplicate, or self dependency. Those are preconditions the caller owns, established where the snapshot is assembled. The generator does not re-check them: it is on the hot path of every run, the checks it could make are the ones an assembled-correctly snapshot can never fail, and paying for them here only spreads the same contract across two places. A malformed snapshot yields undefined candidates rather than an error. +`Generate` receives the queue's live batches and path sets as one snapshot and takes it as given. Path sets are what each batch's builds have done so far — at most one per head, none for a batch nothing has speculated on. The speculate controller assembles both halves once per run and never re-reads mid-run; see [outcome-predictor.md](outcome-predictor.md) for why the scorer does not load them itself. A well-formed snapshot carries unique, non-empty batch IDs, includes every batch a head's direct dependencies reference, and gives no head an empty, duplicate, or self dependency. Those are preconditions the caller owns, established where the snapshot is assembled. The generator does not re-check them: it is on the hot path of every run, the checks it could make are the ones an assembled-correctly snapshot can never fail, and paying for them here only spreads the same contract across two places. A malformed snapshot yields undefined candidates rather than an error. -A dependency the generator cannot price is the one bad input it absorbs, because it arrives from the injected predictor rather than from the caller and there is no earlier point that could catch it. Three cases take the same 0.95 default: a probability outside `[0, 1]` or `NaN`, a predictor call that returned an error, and a dependency the snapshot never carried. The default is optimistic on purpose, so a dependency nobody could estimate keeps its head's preferred path near the front instead of burying it or failing the whole run on one number — and failing the whole run is the real hazard, because `Generate` seeds the heap for every head at once, so one unpriceable dependency would otherwise cost the queue every candidate it had. A batch absent from the snapshot is never passed to the predictor at all: it would resolve to a zero-valued batch belonging to no queue, so predicting it would price some other batch entirely or fail on the empty queue name. Any deliberate defaulting still belongs to the predictor implementation, which knows what information it does and does not have; this is only the floor under it. +A dependency the generator cannot price is the one bad input it absorbs, because it arrives from the injected scorer rather than from the caller and there is no earlier point that could catch it. Three cases take the same 0.95 default: a probability outside `[0, 1]` or `NaN`, a scorer call that returned an error, and a dependency the snapshot never carried. The default is optimistic on purpose, so a dependency nobody could estimate keeps its head's preferred path near the front instead of burying it or failing the whole run on one number — and failing the whole run is the real hazard, because `Generate` seeds the heap for every head at once, so one unpriceable dependency would otherwise cost the queue every candidate it had. A batch absent from the snapshot is never passed to the scorer at all: it would resolve to a zero-valued batch belonging to no queue, so scoring it would price some other batch entirely or fail on the empty queue name. Any deliberate defaulting still belongs to the scorer implementation, which knows what information it does and does not have; this is only the floor under it. ## Step 1: `Generate` prepares each head -`Generate` predicts each unique unresolved dependency once, however many heads wait on it. If A appears in both B's and C's dependency lists, the predictor is still called for A only once. +`Generate` scores each unique unresolved dependency once, however many heads wait on it. If A appears in both B's and C's dependency lists, the scorer is still called for A only once. For each unresolved direct dependency, `Generate` records: @@ -447,9 +447,9 @@ The ordering stays the same. `CandidatePath.RankingScore` contains this logarith - `Succeeded` fixes an assumption to succeeds. - `Failed` or `Cancelled` fixes an assumption to fails. - `Cancelling` remains undecided because cancellation may lose a race with completion. -- `Landing` also remains undecided, because a land can fail. It is tempting to treat it as committed to landing and skip the scorer call, but that puts a state-specific policy inside the search: whether a path betting against a landing batch is worth funding is a question of price, and price belongs to the scorer. The allocator draws the same line — "no batch state enters this decision" — and the generator holds it too. Nothing is lost by staying open: a single passed path still waits for the land result, while passed paths covering every outcome let the controller bypass the dependency (see [speculation.md](speculation.md)). Funding the unlikely side spends budget, which is the allocator's to ration. +- `Landing` also remains undecided, because a land can fail. It is tempting to treat it as committed to landing and skip scoring, but that would turn an uncertain state into a fact inside the search. How much *landing* changes the probability belongs to the scorer; the Generator only consumes that price, and the Allocator still does not interpret batch state. Nothing is lost by staying open: a single passed path still waits for the land result, while passed paths covering every outcome let the controller bypass the dependency (see [speculation.md](speculation.md)). Funding the unlikely side spends budget, which is the Allocator's to ration. - A fixed assumption stays in the returned path but contributes probability 1 and has no flip. -- A shared dependency is predicted once per run. +- A shared dependency is scored once per run. ## Why the algorithm works diff --git a/doc/rfc/submitqueue/speculation.md b/doc/rfc/submitqueue/speculation.md index a9f1ed27..a6878750 100644 --- a/doc/rfc/submitqueue/speculation.md +++ b/doc/rfc/submitqueue/speculation.md @@ -107,7 +107,7 @@ The one extension. It decides *which paths to build and which running ones to ca The default Speculator is composed from two swappable interfaces — a **Generator** and an **Allocator** — so ranking and preemption policy can vary independently. They are composition points inside the default implementation, not controller-facing extensions: the controller depends only on the Speculator contract, and an alternate Speculator need not use or expose this split. The default opens the Generator's candidate stream over the batches, then hands that stream and the path sets to the Allocator. -- **Generator** — yields the queue's candidate paths as one iterator across heads in `BatchStateSpeculating`. *Contract:* every candidate has a Speculating head and is coherent; none repeats or contradicts a resolved fact. Ranking is implementation-defined — the Generator may compute it directly, call an injected pricing extension, or use other injected data — and the score it carries is meaningful only within the run. The Allocator consumes the iterator in the order the Generator yields it and does not interpret the score. *Default:* `bestfirst` asks the predictor for each unresolved dependency's probability of reaching Succeeded, then ranks paths by the probability that all their assumptions hold. +- **Generator** — yields the queue's candidate paths as one iterator across heads in `BatchStateSpeculating`. *Contract:* every candidate has a Speculating head and is coherent; none repeats or contradicts a resolved fact. Ranking is implementation-defined — the Generator may compute it directly, call an injected pricing extension, or use other injected data — and the score it carries is meaningful only within the run. The Allocator consumes the iterator in the order the Generator yields it and does not interpret the score. *Default:* `bestfirst` asks the scorer for each unresolved dependency's probability of reaching Succeeded, then ranks paths by the probability that all their assumptions hold. - **Allocator** — spends the build budget (the queue's cap on concurrent builds) over the iterator. *Contract:* it pulls in order until the budget fills and matches candidates to existing paths by ID, so a pending or building path keeps the slot it already holds rather than starting a second attempt, and a candidate whose path is already terminal in the path sets is skipped rather than rebuilt; pending dispatches are replayed by the controller as described above. Pending, building, and cancelling paths charge the budget (a cancelling build holds CI until terminal), while terminal ones charge none. Cancellation is best-effort, so the Allocator does not spend capacity it merely expects a cancel to release and risk exceeding the hard CI cap. *Default:* the sticky policy fills only free slots and leaves in-flight builds running; a preempting policy cancels in-flight paths below the funded set. Budget is the only rationing lever — there is no ranking-score floor. A build cancelled to make room still charges budget until its cancel reaches terminal and publishes dirty, so the next run funds the released slot — the queue converges over successive ticks rather than oversubscribing in a single pass. ### Extension APIs @@ -118,8 +118,6 @@ Signatures live in code and are not copied here, so they cannot drift. This sect **Speculator** — [`submitqueue/extension/speculation/speculator`](../../../submitqueue/extension/speculation/speculator/README.md). `Speculate` takes one queue snapshot (the batches and their path sets) and returns the build and cancel actions it proposes; a path it wants left alone has no entry. Actions must target Speculating heads. Verdicts stay controller-owned, so there is no land or fail action. -**Scorer** — [`submitqueue/extension/speculation/scorer`](../../../submitqueue/extension/speculation/scorer/README.md). Prices a batch's change from content signals. The default pipeline does not rank on it directly; the queue's predictor is built over it. - -**Predictor** — [`submitqueue/extension/speculation/predictor`](../../../submitqueue/extension/speculation/predictor/README.md). Revises the scorer's price with path-set evidence and batch state. The default `bestfirst` generator is built over it. See [outcome-predictor.md](outcome-predictor.md). +**Scorer** — [`submitqueue/extension/speculation/scorer`](../../../submitqueue/extension/speculation/scorer/README.md). `Score(ctx, batch, paths)` is how likely the batch is to reach Succeeded. The default impl is evidence wrapping a base (heuristic or composite): the base prices the change and ignores `paths`; evidence revises that price from the path set and batch state. See [outcome-predictor.md](outcome-predictor.md). **Generator and Allocator** — [`generator`](../../../submitqueue/extension/speculation/generator/README.md) and [`allocator`](../../../submitqueue/extension/speculation/allocator/README.md), the two composition points inside the default Speculator. The Generator opens a pull-based stream of candidate paths over the batches and path sets; the Allocator spends the build budget over that stream, reconciling it against the path sets. Both abort on a cancelled context. From 2de54998713c985dffce996f70812d9962a452a7 Mon Sep 17 00:00:00 2001 From: Preetam Dwivedi Date: Mon, 14 Sep 2026 12:27:13 -0700 Subject: [PATCH 6/6] docs(submitqueue): rename outcome scorer RFC file ## Summary ### Why? The RFC dropped Predictor; the filename still said outcome-predictor.md. ### What? Move it to outcome-scorer.md and retarget links. Name the landing evidence factor after BatchStateLanding. --- doc/rfc/index.md | 2 +- ...{outcome-predictor.md => outcome-scorer.md} | 18 +++++++++--------- .../speculation-generator-best-first.md | 2 +- doc/rfc/submitqueue/speculation.md | 2 +- 4 files changed, 12 insertions(+), 12 deletions(-) rename doc/rfc/submitqueue/{outcome-predictor.md => outcome-scorer.md} (88%) diff --git a/doc/rfc/index.md b/doc/rfc/index.md index 1f0a5e95..11dcb60d 100644 --- a/doc/rfc/index.md +++ b/doc/rfc/index.md @@ -20,7 +20,7 @@ Design documents and technical proposals, grouped by scope. Shared/cross-cutting - [Extension Contract](submitqueue/extension-contract.md) - When extensions take orchestrator identity (request/batch) and resolve granular content themselves vs. take controller-resolved data; revises the BuildRunner base/head contract - [Gateway Status and List APIs](submitqueue/status-list-api.md) - Gateway-owned request context, materialized current status, sqid or change-URI status lookup, and queue admission listing - [Speculation](submitqueue/speculation.md) - Why SubmitQueue speculates, the path/tree model, and the two pluggable seams: speculation-tree enumeration and path selection -- [Outcome Scorer](submitqueue/outcome-predictor.md) - How likely a batch is to reach Succeeded: `Score(ctx, batch, paths)` as a logit-linear model — a base content price plus YAML weights on path and batch evidence (`pathPassed`, `pathFailed`, `merging`, `cancelling`) +- [Outcome Scorer](submitqueue/outcome-scorer.md) - How likely a batch is to reach Succeeded: `Score(ctx, batch, paths)` as a logit-linear model — a base content price plus YAML weights on path and batch evidence (`pathPassed`, `pathFailed`, `landing`, `cancelling`) - [Best-First Speculation Path Generation](submitqueue/speculation-generator-best-first.md) - The default Generator: per-head lazy streams of flip subsets merged best-first across heads, log-probability ranking, and the strict snapshot contract - [Modular Queue Wiring](submitqueue/modular-queue-wiring.md) - Declare-don't-assemble engine (`pipeline.Construct`) that unifies topic registry, controller registration, DLQ pairing, and lifecycle ordering into one typed call; services self-declare via Deps struct + Stages slice, hosts own per-queue profiles and transport diff --git a/doc/rfc/submitqueue/outcome-predictor.md b/doc/rfc/submitqueue/outcome-scorer.md similarity index 88% rename from doc/rfc/submitqueue/outcome-predictor.md rename to doc/rfc/submitqueue/outcome-scorer.md index f3d50b6c..5dd70ff8 100644 --- a/doc/rfc/submitqueue/outcome-predictor.md +++ b/doc/rfc/submitqueue/outcome-scorer.md @@ -11,7 +11,7 @@ See [speculation.md](speculation.md) for batches, paths, heads, and the Speculat That number has two parts, composed as one scorer: 1. A **base** price for the change from content signals such as its size. Heuristic and composite supply this. They implement the same `Score` and ignore `paths`. -2. An **evidence** layer that revises the base with facts the speculate run already holds: a path *passed*, a path *failed*, the batch is *merging*, the batch is *cancelling*. +2. An **evidence** layer that revises the base with facts the speculate run already holds: a path *passed*, a path *failed*, the batch is *landing*, the batch is *cancelling*. Evidence is the scorer the queue exposes. The base sits under it. There is no sibling Predictor factory. @@ -47,7 +47,7 @@ The unconfigured base prices every batch at `0.5`. From that price, one factor ` | `0.3` | ~0.23 | | `0.25` | 0.20 | -A base price of `0.6` with `pathPassed: 10` becomes about `0.94`. `merging: 12` on top of that becomes about `0.995`. +A base price of `0.6` with `pathPassed: 10` becomes about `0.94`. `landing: 12` on top of that becomes about `0.995`. `pathFailed: 0.3` from `0.5` becomes about `0.23`. It applies at most once because the path set has one current entry for the all-*succeeds* path; retry attempts replace that entry rather than adding evidence. `0` is rejected because it would pin matching batches at probability 0. @@ -63,7 +63,7 @@ scorer: factors: pathPassed: 10 pathFailed: 0.3 - merging: 12 + landing: 12 cancelling: 0.1 base: type: heuristic @@ -79,10 +79,10 @@ Profiles may set `factors` under `defaults.scorer` and revise them per queue. An | --- | --- | --- | | `pathPassed` | Once, if a path that assumes every dependency *succeeds* has *passed* | Up | | `pathFailed` | Once, if the path that assumes every dependency *succeeds* has *failed* | Down | -| `merging` | While the batch is *merging* | Up | +| `landing` | While the batch is *landing* | Up | | `cancelling` | While the batch is *cancelling* | Down | -`bestfirst` already treats a terminal batch as a fact (*Succeeded*, *Failed*, *Cancelled*). The scorer is not asked. *Merging* is not terminal: a merge can still fail, so how much it is worth stays a price. +`bestfirst` already treats a terminal batch as a fact (*Succeeded*, *Failed*, *Cancelled*). The scorer is not asked. *Landing* is not terminal: a land can still fail, so how much it is worth stays a price. ### Only the *succeeds* path counts @@ -95,9 +95,9 @@ Take `C` depending on `B`, and `B` depending on `A`. Ranking `C`'s candidates ne | `B` with `A` *succeeds* | `B` on top of `A`'s changes | | `B` with `A` *fails* | `B` without them | -`B` merges after `A` does, so the first build is a build of the code that will actually land: if it *passed*, `B` is likely to merge, and `pathPassed` applies. +`B` lands after `A` does, so the first build is a build of the code that will actually land: if it *passed*, `B` is likely to land, and `pathPassed` applies. -The second is a different set of changes. `B` may call something `A` introduces and fail to compile on its own — a *failed* result that says nothing about `B` merging in the normal case. Counting it would push `B` down the ranking over a build it was never going to need, while a green build of the real combination sits in the same set. +The second is a different set of changes. `B` may call something `A` introduces and fail to compile on its own — a *failed* result that says nothing about `B` landing in the normal case. Counting it would push `B` down the ranking over a build it was never going to need, while a green build of the real combination sits in the same set. So `pathPassed` and `pathFailed` both look only at paths that assume every dependency *succeeds*. Results on any other path are skipped. This is a filter on which results are evidence, not a check on whether an assumption came true — nothing here revisits that. @@ -117,9 +117,9 @@ Every content backend reads the path set. We tried forwarding path sets through Train a single estimate over diff shape and build outcomes together. Content signals and situation signals change at different rates, need different amounts of data, and would force every queue onto the same content scorer. **Instead:** the base stays per-queue; evidence weights layer on in YAML. `p_base` remains the GLM offset if someone later fits `w`. -### Treat *merging* and *cancelling* as settled in the Generator +### Treat *landing* and *cancelling* as settled in the Generator -Rank a *merging* batch like Succeeded and a *cancelling* batch like Cancelled. We tried and reverted: a merge can still fail, so the rank was wrong once outcomes diverged. **Instead:** only terminal states short-circuit in the Generator; *merging* and *cancelling* are scorer features (see [Evidence](#evidence)). +Rank a *landing* batch like Succeeded and a *cancelling* batch like Cancelled. We tried and reverted: a land can still fail, so the rank was wrong once outcomes diverged. **Instead:** only terminal states short-circuit in the Generator; *landing* and *cancelling* are scorer features (see [Evidence](#evidence)). ### Let the scorer read the path-set store diff --git a/doc/rfc/submitqueue/speculation-generator-best-first.md b/doc/rfc/submitqueue/speculation-generator-best-first.md index 6b81105f..27fd3bd2 100644 --- a/doc/rfc/submitqueue/speculation-generator-best-first.md +++ b/doc/rfc/submitqueue/speculation-generator-best-first.md @@ -47,7 +47,7 @@ The batch being built is written before its assumptions. For example, `C [A succ ## The snapshot is a caller precondition -`Generate` receives the queue's live batches and path sets as one snapshot and takes it as given. Path sets are what each batch's builds have done so far — at most one per head, none for a batch nothing has speculated on. The speculate controller assembles both halves once per run and never re-reads mid-run; see [outcome-predictor.md](outcome-predictor.md) for why the scorer does not load them itself. A well-formed snapshot carries unique, non-empty batch IDs, includes every batch a head's direct dependencies reference, and gives no head an empty, duplicate, or self dependency. Those are preconditions the caller owns, established where the snapshot is assembled. The generator does not re-check them: it is on the hot path of every run, the checks it could make are the ones an assembled-correctly snapshot can never fail, and paying for them here only spreads the same contract across two places. A malformed snapshot yields undefined candidates rather than an error. +`Generate` receives the queue's live batches and path sets as one snapshot and takes it as given. Path sets are what each batch's builds have done so far — at most one per head, none for a batch nothing has speculated on. The speculate controller assembles both halves once per run and never re-reads mid-run; see [outcome-scorer.md](outcome-scorer.md) for why the scorer does not load them itself. A well-formed snapshot carries unique, non-empty batch IDs, includes every batch a head's direct dependencies reference, and gives no head an empty, duplicate, or self dependency. Those are preconditions the caller owns, established where the snapshot is assembled. The generator does not re-check them: it is on the hot path of every run, the checks it could make are the ones an assembled-correctly snapshot can never fail, and paying for them here only spreads the same contract across two places. A malformed snapshot yields undefined candidates rather than an error. A dependency the generator cannot price is the one bad input it absorbs, because it arrives from the injected scorer rather than from the caller and there is no earlier point that could catch it. Three cases take the same 0.95 default: a probability outside `[0, 1]` or `NaN`, a scorer call that returned an error, and a dependency the snapshot never carried. The default is optimistic on purpose, so a dependency nobody could estimate keeps its head's preferred path near the front instead of burying it or failing the whole run on one number — and failing the whole run is the real hazard, because `Generate` seeds the heap for every head at once, so one unpriceable dependency would otherwise cost the queue every candidate it had. A batch absent from the snapshot is never passed to the scorer at all: it would resolve to a zero-valued batch belonging to no queue, so scoring it would price some other batch entirely or fail on the empty queue name. Any deliberate defaulting still belongs to the scorer implementation, which knows what information it does and does not have; this is only the floor under it. diff --git a/doc/rfc/submitqueue/speculation.md b/doc/rfc/submitqueue/speculation.md index a6878750..ceeabc11 100644 --- a/doc/rfc/submitqueue/speculation.md +++ b/doc/rfc/submitqueue/speculation.md @@ -118,6 +118,6 @@ Signatures live in code and are not copied here, so they cannot drift. This sect **Speculator** — [`submitqueue/extension/speculation/speculator`](../../../submitqueue/extension/speculation/speculator/README.md). `Speculate` takes one queue snapshot (the batches and their path sets) and returns the build and cancel actions it proposes; a path it wants left alone has no entry. Actions must target Speculating heads. Verdicts stay controller-owned, so there is no land or fail action. -**Scorer** — [`submitqueue/extension/speculation/scorer`](../../../submitqueue/extension/speculation/scorer/README.md). `Score(ctx, batch, paths)` is how likely the batch is to reach Succeeded. The default impl is evidence wrapping a base (heuristic or composite): the base prices the change and ignores `paths`; evidence revises that price from the path set and batch state. See [outcome-predictor.md](outcome-predictor.md). +**Scorer** — [`submitqueue/extension/speculation/scorer`](../../../submitqueue/extension/speculation/scorer/README.md). `Score(ctx, batch, paths)` is how likely the batch is to reach Succeeded. The default impl is evidence wrapping a base (heuristic or composite): the base prices the change and ignores `paths`; evidence revises that price from the path set and batch state. See [outcome-scorer.md](outcome-scorer.md). **Generator and Allocator** — [`generator`](../../../submitqueue/extension/speculation/generator/README.md) and [`allocator`](../../../submitqueue/extension/speculation/allocator/README.md), the two composition points inside the default Speculator. The Generator opens a pull-based stream of candidate paths over the batches and path sets; the Allocator spends the build budget over that stream, reconciling it against the path sets. Both abort on a cancelled context.