Say on the Routines page when nothing is there to fire them - #460
Open
zopeVaibhav wants to merge 1 commit into
Open
Say on the Routines page when nothing is there to fire them#460zopeVaibhav wants to merge 1 commit into
zopeVaibhav wants to merge 1 commit into
Conversation
zopeVaibhav
requested review from
MikeRyanDev,
davidmckayv,
guidovizoso and
tylerslaton
as code owners
September 9, 2026 07:32
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
A routine needs a second process to fire it, and a deployment that never started one looked exactly
like a deployment that had. Every sweep now records that it happened, and the Routines page reads
that record: somebody with standing routines and nothing sweeping is told so, instead of being shown
a page that looks correct.
routine_sweeps, one row keyed"routines", holdingswept_atand theownerthat last swept(
server/drizzle/0029_routine_sweeps.sql, schema atserver/src/db/schema/coworker.ts:133-137).routineStore.recordSweep/lastSweptAt(server/src/routines/store.ts:797-814), an upsert onthe primary key so the table stays exactly one row however many passes run.
offerDueRoutinesrecords the pass before it reads what is due(
server/src/routines/sweep.ts:142), so the Kubernetes CronJob and the laptop loop both reportthrough the one funnel they already share.
GET /api/routinesgainssweep { lastSweptAt, working }(server/src/routines/routes.ts:39-47).was or saying none ever happened (
app/src/components/routines/routines-list.tsx:151).docs/routines.mdanddocs/configuration.mdasserted the old behaviour and now describe this one.Why the window is fifteen minutes, and fixed
workingmeans a sweep insideSWEEP_SILENCE_MS, which isMINIMUM_INTERVAL_MS— the floor aroutine's own schedule already has (
server/src/routines/schedule.ts:4), so a longer silence is oneno routine could have wanted.
It is a constant rather than something derived from the deployment's own cadence, and the chart is
why that holds:
routines.scheduledefaults to*/5 * * * *(charts/openbot/values.yaml:268),chosen, in that file's own words, to sit "inside the 15-minute floor the tools enforce, so a firing
waits at most one tick". A deployment on the defaults sweeps three times inside the window. Setting a
schedule looser than fifteen minutes would make this page read as quiet between runs — but such a
schedule is already outside what the floor is built for, and the docs now say so.
Deriving the window instead (from observed sweep intervals, or from whether any routine is actually
overdue) would remove that edge, at the cost of history to read and of detection latency that follows
each routine's own cadence — a weekly routine would take a week to notice a dead worker. Fifteen
minutes notices in fifteen minutes, whatever is standing.
Where it runs
routine_sweeps, in Postgres. Not amodule-level
Map.is the intended semantic — the question the page asks is "did anything sweep recently", not
"did this process sweep".
ownerrecords which process wrote last, so a stuck deployment stilltraces back to a claimant.
insert ... on conflict (id) do update, a singleconditional statement on the primary key, not a check-then-write. Concurrent sweepers cannot
produce two rows or a lost table.
query; no socket, no push.
hundred copies of it write one row.
Boundary and audit
recordSweepisbookkeeping on the dispatch path and takes no decision.
routine-sweep-heartbeat-failedrather than writing an audit row, deliberately — it is not anacting call, and a broken heartbeat must not be able to stop a sweep that would otherwise have
worked.
workingiscomputed server-side from the stored timestamp; the client is handed a boolean, not a rule.
This widens one response rather than a permission:
GET /api/routinesnow tells a signed-in callerwhen the deployment last swept. It is visible only to somebody already entitled to the routines on
that page, and it carries a timestamp and a lease name, not a hostname or a secret.
One skew to know about
GET /api/routineschanged shape from{ routines }to{ routines, sweep }, and the app defaultsa missing
sweepto{ lastSweptAt: null, working: false }. A new app served by an old server wouldtherefore show the warning when nothing is wrong. The app and the server ship from this repository
together, so there is no supported configuration where that happens — but it is the failure mode to
expect if anyone ever serves them from different builds, and the default was chosen that way on
purpose: a false warning is recoverable, a silently swallowed one is the bug this PR exists to fix.
Changelog
Unreleased: "The Routines page says when nothing is there to run them".Proof
Reproduced against the bug before fixing it. On
main,GET /api/routinesanswers with keys[ "routines" ]and there is no field a page could read to tell a dead sweeper from a live one; thesame assertion on this branch answers
[ "routines", "sweep" ]. The new tests fail onmainfor thesame reason and pass here.
Driven in a browser against a live deployment, all four states:
last checked 22 minutes ago. Until it is running again, none of these will fire."
will fire. A deployment needs one running to carry them out."
that process's lease name
Gates: typecheck clean across app, server and worker; lint 597 files, no warnings; format 593 files,
no diff.
drizzle-kit generatereports no schema changes, so the snapshot and the schema agree.Migration
0029_routine_sweepsapplied to a live database and confirmed byinformation_schema:idNOT NULL,swept_atNOT NULL,ownernullable.Suite, measured this session against
06633a4with the twoagent-handoff-*.integrationfilesexcluded — they fail 4-5 of their 5 tests on bare
mainwith no branch applied, and including themmakes any comparison meaningless: branch 2544 pass / 2 fail,
main2533 pass / 2 fail, eachidentical across three consecutive runs. The two failures are the same on both sides, and neither
belongs to this change. The +11 tests are exactly this PR's: 5 in
app/tests/routines-no-worker.test.ts,4 in
server/tests/routine-sweep-liveness.test.ts, and 2 added toserver/tests/routine-sweep.integration.test.ts.Closes #459