A routine sitting on the Routines page with "Next in 3 days" next to it says nothing about whether
anything will actually fire it. The page renders the same whether a worker is sweeping every thirty
seconds, or the CronJob was never enabled, or the worker died in the night. The first sign anything is
wrong is a report that never arrives, and by then the page has been quietly lying for days.
This is not a suspicion about the code. docs/routines.md:70-73 states it outright: "This fails
silently... a deployment with no worker looks identical to one running normally, right up until
nobody's standup notes ever arrive." The behaviour is known and documented; what is missing is
anything on the screen that acts on it.
Why the screen cannot tell
GET /api/routines asks one question and answers it: routineStore.listFor(actor.id), mapped to a
dto (server/src/routines/routes.ts:38-40). Everything it returns is derived from rows the routine
itself owns — its schedule, its next run stamp, its last outcome. None of those change when the
sweeper stops. nextRunAt in particular is a stored stamp rather than a queue position, so it keeps
reading as a promise nobody is keeping.
The app has the same blind spot by construction: app/src/lib/routines/queries.ts:48-51 unwraps the
"routines" key and hands the list to the page, which renders it
(app/src/components/routines/routines-list.tsx:141). There is no second source of truth to consult,
because nothing anywhere records that a sweep happened.
Reproduction
- Bring up the stack and create a routine through a Bot, so it is stored with a schedule.
- Confirm the Routines page shows it with a next run time.
- Stop the routines worker — on a laptop kill the process, on Kubernetes set
routines.enabled to
false.
- Reload the Routines page.
The page is byte-for-byte what it was in step 2. Wait past the routine's schedule and it still is;
nextRunAt only advances when a sweep advances it, so the stamp goes stale rather than visibly
wrong. Nothing in the UI, and nothing in the API response, distinguishes step 4 from step 2.
Why a log line is not enough
The worker logs each pass, so an operator watching stdout can see the gap. That is the wrong
audience. The person who created the routine in a chat window is not reading pod logs, and is the
only person who knows the report never arrived. The signal has to reach the screen where the promise
was made.
What a fix probably has to do
Record that a sweep happened somewhere both sweep callers already pass through — offerDueRoutines
(server/src/routines/sweep.ts:135) is the single funnel for the Kubernetes CronJob and the laptop
loop alike — and let the routines response carry it, so the page can say so.
Two decisions are worth making deliberately. How long is too quiet? Routines already have a floor
of fifteen minutes between runs (MINIMUM_INTERVAL_MS, server/src/routines/schedule.ts:4), and the
chart's own routines.schedule defaults to */5 * * * * to sit inside it
(charts/openbot/values.yaml:268), so a fixed fifteen-minute window fits what a deployment on the
defaults already does three times over; a CronJob scheduled looser than that would read as quiet
between runs and should be documented as such. Who is told? Somebody with no routines standing
has nothing to be warned about, so the warning belongs to the page only when there is something on it
that will not fire.
The regression to guard against is the recording itself failing: it runs on the dispatch path, so it
must not be able to stop a sweep that would otherwise have worked.
Severity
Low blast radius, high time-to-detection. Nothing breaks that was working, and no data is at risk — a
deployment with a live worker is unaffected. What it costs is trust in the screen: the failure it
hides is silent, unbounded in duration, and discovered by a person noticing an absence rather than by
anything reporting a fault.
A routine sitting on the Routines page with "Next in 3 days" next to it says nothing about whether
anything will actually fire it. The page renders the same whether a worker is sweeping every thirty
seconds, or the CronJob was never enabled, or the worker died in the night. The first sign anything is
wrong is a report that never arrives, and by then the page has been quietly lying for days.
This is not a suspicion about the code.
docs/routines.md:70-73states it outright: "This failssilently... a deployment with no worker looks identical to one running normally, right up until
nobody's standup notes ever arrive." The behaviour is known and documented; what is missing is
anything on the screen that acts on it.
Why the screen cannot tell
GET /api/routinesasks one question and answers it:routineStore.listFor(actor.id), mapped to adto (
server/src/routines/routes.ts:38-40). Everything it returns is derived from rows the routineitself owns — its schedule, its next run stamp, its last outcome. None of those change when the
sweeper stops.
nextRunAtin particular is a stored stamp rather than a queue position, so it keepsreading as a promise nobody is keeping.
The app has the same blind spot by construction:
app/src/lib/routines/queries.ts:48-51unwraps the"routines"key and hands the list to the page, which renders it(
app/src/components/routines/routines-list.tsx:141). There is no second source of truth to consult,because nothing anywhere records that a sweep happened.
Reproduction
routines.enabledtofalse.
The page is byte-for-byte what it was in step 2. Wait past the routine's schedule and it still is;
nextRunAtonly advances when a sweep advances it, so the stamp goes stale rather than visiblywrong. Nothing in the UI, and nothing in the API response, distinguishes step 4 from step 2.
Why a log line is not enough
The worker logs each pass, so an operator watching stdout can see the gap. That is the wrong
audience. The person who created the routine in a chat window is not reading pod logs, and is the
only person who knows the report never arrived. The signal has to reach the screen where the promise
was made.
What a fix probably has to do
Record that a sweep happened somewhere both sweep callers already pass through —
offerDueRoutines(
server/src/routines/sweep.ts:135) is the single funnel for the Kubernetes CronJob and the laptoploop alike — and let the routines response carry it, so the page can say so.
Two decisions are worth making deliberately. How long is too quiet? Routines already have a floor
of fifteen minutes between runs (
MINIMUM_INTERVAL_MS,server/src/routines/schedule.ts:4), and thechart's own
routines.scheduledefaults to*/5 * * * *to sit inside it(
charts/openbot/values.yaml:268), so a fixed fifteen-minute window fits what a deployment on thedefaults already does three times over; a CronJob scheduled looser than that would read as quiet
between runs and should be documented as such. Who is told? Somebody with no routines standing
has nothing to be warned about, so the warning belongs to the page only when there is something on it
that will not fire.
The regression to guard against is the recording itself failing: it runs on the dispatch path, so it
must not be able to stop a sweep that would otherwise have worked.
Severity
Low blast radius, high time-to-detection. Nothing breaks that was working, and no data is at risk — a
deployment with a live worker is unaffected. What it costs is trust in the screen: the failure it
hides is silent, unbounded in duration, and discovered by a person noticing an absence rather than by
anything reporting a fault.