A gateway dispatch is a probe: workers record lease.declined, the gateway records every fleet outcome #431

Part of #329.

Scope

After this PR, every fleet request that ends while its gateway runs has one ending in the gateway's own history, and a worker's history stops calling gateway traffic rejected. A worker records each gateway dispatch it refuses or fails as lease.declined, and names the gateway's request on the lease events of that dispatch. The gateway records request.granted for every grant it settles, and its own lease.rejected for every other ending. Usage figures (#347) then join a fleet request to its outcome by id. ADR 0021 decides this.

In short

Today, when a gateway asks worker A for a device and A has no room, A records lease.rejected (no-wait), and the gateway asks worker B. The history says the request was rejected, although B granted it. When a request fails on a worker, the gateway records nothing of its own. After this PR, A records lease.declined carrying the gateway's request id as fleetRequestId. B's lease.granted carries the same id. The gateway records request.granted for B's grant, and a request that fails anywhere ends in one gateway lease.rejected.

sequenceDiagram
  participant G as Gateway
  participant A as Worker A
  participant B as Worker B
  G->>A: lease.request (noWait, fleetRequestId=req_1)
  A-->>G: NO_CAPACITY
  Note over A: lease.declined {fleetRequestId: req_1} (changed)
  G->>B: lease.request (noWait, fleetRequestId=req_1)
  B-->>G: grant
  Note over B: lease.granted {fleetRequestId: req_1} (changed)
  Note over G: request.granted {requestId: req_1, worker: B} (new)

# worker A, `simlock events`, before
lease.requested  requester=gw:gw1:alice  waitPolicy=no-wait
lease.rejected   requester=gw:gw1:alice  reason=no-wait
# after
lease.requested  requester=gw:gw1:alice  waitPolicy=no-wait  fleetRequestId=req_1
lease.declined   requester=gw:gw1:alice  reason=no-wait      fleetRequestId=req_1

If this goes wrong, an operator would see a gateway stop granting (a worker refusing the new field), a fleet request with neither request.granted nor lease.rejected on the gateway, or a worker's history still showing rejections for requests the gateway placed elsewhere.

Technical spec

Modules touched

flowchart LR
  FC[gateway/fleet-coordinator *] --> C[contract: operations, protocol *]
  D[daemon/dispatcher *] --> C
  D --> LAC[leasing/lease-acquisition-coordinator *]
  LAC --> RB[leasing/lease-request-book *]
  LAC --> LL[leasing/lease-lifecycle *]
  LS[leasing/lease-startup *] --> RB
  LAC --> BUS[bus/index: EventMap *]
  FC --> BUS
  FC --> MCP[mcp/contracts, http/app: field not offered]

• src/contract/operations.ts: the lease.request input gains fleetRequestId?: string, 1 to 200 characters (the idempotencyKey bound). The schema does not tie it to noWait; the handler does, after its session check, so a caller that may not send a well-formed field gets FORBIDDEN whatever noWait says; input the schema refuses is BAD_REQUEST first, as for any request.
• src/contract/protocol.ts: protocol +1 at merge. The doc comment gains the reason: a worker on the previous version is incompatible.
• src/mcp/contracts.ts: the lease tool's input omits fleetRequestId, as it omits requesterId. src/http/app.ts: the strict lease body does not declare it, so an HTTP body that carries it is a 400 as for any unknown field.
• src/gateway/dispatcher.ts lease.request handler: refuses fleetRequestId from every session with FORBIDDEN (gateways do not chain), beside its owner check.
• src/gateway/worker-link.ts: request.granted joins the gateway-own event names a worker's push is refused under.
• src/daemon/dispatcher.ts #leaseRequest: fleetRequestId from a session that is not the gateway uplink is refused with FORBIDDEN. This is the same check as owner, and the refusal comes before the request is stored. Then a fleetRequestId without noWait: true is refused with BAD_REQUEST. The dispatcher passes the field on in the lease request options.
• The stored request keeps fleetRequestId across a restart: LeaseRequestRecord (src/core/domain.ts), newLeaseRequestRecord (src/core/lease-request-store.ts), leaseRequestRecordKeys and parseLeaseRequest (src/core/registry.ts), and src/leasing/lease-request-book.ts. LeaseRequestOptions (src/leasing/wait-queue.ts, shared with the gateway's queue) carries it to the waiter.
• src/leasing/lease-acquisition-coordinator.ts: one emit helper decides between lease.rejected and lease.declined, on one condition only: the request carries fleetRequestId. Every worker rejection site goes through it or applies the same condition: killed and already-leased at admission (:239, :258), lease-id-taken (:317), killed from beginMaintenance (:336), the waiter's terminal rejection (:1033), the queue timeout in src/leasing/create-leasing.ts:155 (unreachable for a probe, which is never queued, but on the same rule), and startup below. lease.declined carries requestSpec. A probe is never enqueued: #defer already rejects a noWait waiter, and the second-failure path (:728-735) declines a probe with reason no-wait and rejects its caller with NoCapacityError instead of #enqueue; a local request still queues there as today. lease.requested carries the field.
• src/leasing/lease-startup.ts: daemon-restarted for a stored probe is lease.declined with its fleetRequestId, through the same helper or the same condition.
• src/leasing/lease-lifecycle.ts: lease.granted carries fleetRequestId when the grant served a probe.
• src/bus/index.ts: adds lease.declined and request.granted to EventMap. fleetRequestId? goes on lease.requested and lease.granted. lease.rejected gains the reason worker-failed, code? and worker?. The field is worker, never workerId: payload.workerId marks a relayed event (ADR 0014 §6).
• src/gateway/fleet-coordinator.ts:
  • Every dispatch sends fleetRequestId: waiter.id.
  • Every path that ends a waiter without a grant emits exactly one lease.rejected:
    • #attempt's terminal branch (:1014) emits worker-failed with code from the DispatchError it already builds;
    • #rejectUnservable with refusals emits unresolvable-spec whether or not the request was queued.
  • #settleGrant emits request.granted { requestId, worker, leaseId, workerLeaseId } only when the lease index accepted the grant and queue.resolve settled the waiter. A grant given back (retry), one the index refused (lease-id-taken, already rejected), and one landing after the waiter was settled (timeout, cancel, dispose()) emit nothing.
  • worker-failed rejections carry worker.
  • dispose() emits nothing (ADR 0021 §4: usage closes those requests at the next daemon.started).
• docs/EVENTS.md and docs/internal/EVENTS.md: the new event, the new fields and reason, and the emission changes on worker and gateway.

Contract and event changes

• lease.request input: fleetRequestId?: string, 1 to 200 characters, only with noWait: true, accepted only on the gateway uplink session. Protocol +1 at merge.
• New event lease.declined { requestId, fleetRequestId, requester, requestSpec, reason }; reason uses lease.rejected's values. Emitter: LeaseAcquisitionCoordinator and lease startup (worker). Both EVENTS.md files.
• New event request.granted { requestId, worker, leaseId, workerLeaseId }. Emitter: FleetLeaseCoordinator (gateway), when the grant is handed to the caller. Both EVENTS.md files.
• lease.requested and lease.granted gain optional fleetRequestId on a worker. lease.rejected gains the reason worker-failed and optional code and worker. All additive (events rule 6). Both EVENTS.md files.
• Emission changes (ADR 0021 also narrows ADR 0014 §6: a worker's events for a probe change; its events for local requests do not):
  • a worker emits no lease.rejected for a probe;
  • a gateway emits request.granted for every grant it settles, and lease.rejected for every other ending while it runs;
  • a worker never queues a probe.

Other code on the same state

• src/gateway/fleet-coordinator.ts #attempt and #rejectUnservable: the gateway's retry rule is unchanged. The worker no longer mirrors it: it declines every probe refusal, so no second copy of the rule exists (architecture rule 10).
• src/gateway/queue.ts: reject returns whether it settled the waiter. The new gateway emits go through #reject, which emits only when reject settled it, so a waiter settled twice still gives one event.
• src/gateway/routing/serviceability.ts: reads the gateway's routing verdicts, not events. Unchanged.
• src/leasing/wait-queue.ts: after this PR no probe enters it (the second-failure path declines a probe instead), so it emits nothing for them. Today one can, after a second failed provision.
• src/core/warm-pool/converger.ts: triggers on lease.granted. The new field does not change it.
• FleetLeaseIndex rebuild from a worker's lease list: reads leases, not events. Unchanged.
• The usage reader (usage.get and simlock stats: the figures from the event history, on a worker and a gateway #347, not merged): its fleet join moves to these ids in usage.get and simlock stats: the figures from the event history, on a worker and a gateway #347, not here.

Rules in play

• architecture.md rule 10: the decline condition exists once, in the coordinator's emit helper, and startup uses the same condition.
• events.md rules 1, 3, 6, 8: the name, post-commit emission, additive fields, both EVENTS.md files in this PR.
• safety.md rule 10: fleetRequestId is bounded, and accepted only from the uplink with noWait.
• testing.md rule 1.

Tests

Worker seam: createLeasing with the fake driver, modelled on src/leasing/lease-acquisition-coordinator.test.ts. Gateway seam: src/gateway/fleet-coordinator.test.ts with its fake workers. Uplink check: src/daemon/dispatcher.test.ts.

• (worker) a probe refused for no capacity emits lease.declined with reason no-wait and its fleetRequestId, and no lease.rejected
• (worker) a probe naming a model the catalog lacks emits lease.declined with reason unresolvable-spec
• (worker) a probe refused already-leased at admission emits lease.declined
• (worker) a probe refused lease-id-taken emits lease.declined
• (worker) a probe refused killed at admission, and one in progress when beginMaintenance runs, each emit lease.declined
• (worker) a probe whose boot times out after a progress push emits lease.declined with reason boot-timeout
• (worker) a probe still open when the daemon stops emits lease.declined daemon-restarted with its fleetRequestId at the next start
• (worker) a local noWait request with no room still emits lease.rejected no-wait and no lease.declined
• (worker) a probe's lease.requested and lease.granted carry its fleetRequestId; a local request's carry none
• (worker) the caller of a declined probe gets the same error code as today, except a probe whose provision fails twice, which today is queued and now gets NO_CAPACITY
• (dispatcher) lease.request with fleetRequestId from a session that is not the gateway uplink is refused with FORBIDDEN and stores nothing, with and without noWait: true
• (dispatcher) lease.request with fleetRequestId and without noWait: true is refused with BAD_REQUEST
• (gateway) every dispatch sends the gateway's request id as fleetRequestId
• (gateway) a request whose dispatch fails after a progress push emits one lease.rejected worker-failed with the caller's error code
• (gateway) a request whose worker is unreachable emits one lease.rejected worker-failed with code WORKER_UNREACHABLE
• (gateway) a request never queued that ends on a worker's cannot-serve refusal emits one lease.rejected unresolvable-spec
• (gateway) a request one worker refuses NO_CAPACITY and another grants emits no lease.rejected
• (gateway) a settled grant emits one request.granted with worker and both lease ids, and no workerId in its payload
• (gateway) a grant the lease index refuses emits lease.rejected lease-id-taken and no request.granted
• (gateway) a grant that lands after dispose() or after the dispatch timed out emits no request.granted
• (gateway) lease.request with fleetRequestId sent to a gateway is refused with FORBIDDEN, with and without noWait: true
• (gateway) dispose() with a queued and an in-flight request emits no lease.rejected and no request.granted
• (gateway) the worker link refuses a request.granted pushed by a worker
• (gateway) a grant given back for a mismatched leaseId emits no request.granted, and the retried grant emits one
• (gateway) a worker-failed rejection carries the failing worker's id as worker
• (gateway) a worker's ALREADY_LEASED or LEASE_ID_TAKEN before any progress push ends the request in lease.rejected worker-failed with that code, not the gateway's own lease-id-taken
• (worker) a probe whose provision fails twice is declined no-wait, its caller gets NO_CAPACITY, and it emits no lease.queued; a local request in the same case still queues
• (worker) a probe open at a restart is declined daemon-restarted with its fleetRequestId after the registry is reloaded from disk
• (worker) lease.declined for a probe refused at admission carries its requestSpec
• (mcp) the lease tool's input schema in tools/list has no fleetRequestId
• (http) a lease body carrying fleetRequestId is answered 400
• (e2e) on a gateway with one worker, a granted request leaves one gateway request.granted and a relayed lease.requested and lease.granted whose fleetRequestId is the gateway's own lease.requested id
• (e2e) on a gateway with one worker, a gateway request whose chosen leaseId a local client already holds on the worker ends in one gateway lease.rejected worker-failed (code LEASE_ID_TAKEN) and one relayed lease.declined (lease-id-taken) with the same fleetRequestId, and no relayed lease.rejected

Done when

• On a gateway, a granted request shows one gateway request.granted, and the relayed lease.granted carries the gateway's request id as fleetRequestId.
• On a gateway, a request whose chosen leaseId a local client already holds on the worker shows one gateway lease.rejected worker-failed with code LEASE_ID_TAKEN and worker, one relayed lease.declined with the same fleetRequestId, and no relayed lease.rejected.
• A worker that refuses a probe for lack of room records lease.declined, not lease.rejected (worker test).
• A local simlock lease --no-wait with no room on a worker still prints and records lease.rejected no-wait.
• A gateway and a worker one protocol version apart report the worker as incompatible.
• Both EVENTS.md files list lease.declined, request.granted, fleetRequestId, and worker-failed with code and worker.

Out of scope

• Usage figures and the fleet join (usage.get and simlock stats: the figures from the event history, on a worker and a gateway #347 uses this).
• A user-facing flag for probes (ADR 0021, alternatives).

Depends on

Approval

• Approved for delivery

Written by an agent.