fix(health): move the probe's schedule off GitHub cron to a systemd timer

.github/workflows/health-probe.yml

@@ -1,18 +1,17 @@
 name: Health probe
 
-# Hourly is the interval: six silent days was the failure mode, and catching
-# it the same day is the acceptance bar. The 6h ingest-age threshold is
-# independent — a quiet night is not an alert; six hours without a span is.
+# This workflow no longer schedules anything. `cron: "17 * * * *"` sat on the
+# default branch for nine hours and produced one run: GitHub schedules public
+# repositories on a best-effort basis and *drops* ticks rather than delaying
+# them, so the detection delay it bought was unbounded — the same silence the
+# probe exists to close. The scheduled probe is now a systemd timer on the Pi
+# (`~/ops/cotel-healthz.sh`, every 10 minutes), which owns the production alert
+# marker. See docs/operations/health-probe.md and ADR-0022.
 #
-# Two halves, neither sufficient alone. The loopback half runs on the deploy
-# host and sees the application; it is blind to its own host being down. The
-# edge half runs on a GitHub runner and sees host, tunnel, DNS and Access; it
-# cannot see /healthz at all until the Access service token is allowed on the
-# cotel.aignite.pl application, so it reports that as a warning rather than an
-# outage.
+# What is left here is dispatch-only: a drill of the pager, and the only
+# vantage point outside the LAN. Both jobs therefore page the *drill* markers,
+# never production.
 on:
-  schedule:
-    - cron: "17 * * * *"
   workflow_dispatch:
     inputs:
       url:
@@ -54,15 +53,15 @@ jobs:
 
   probe-loopback:
     name: Probe /healthz on the deploy host
-    if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
+    if: github.event_name == 'workflow_dispatch'
     runs-on: [self-hosted, flopsstuff, docker]
     # This job cannot report its own host being down: with the runner offline
     # the run sits in the queue, and timeout-minutes does not bound queue time
     # — it starts counting once a runner picks the job up. So an absent host
-    # produces no colour at all, which is what the edge half is for. The cap
-    # only bounds a job that is running and stuck (6h by default), and the
-    # concurrency group keeps each hour superseding the last queued attempt
-    # instead of piling up a day's worth of them.
+    # produces no colour at all, which is why the Pi timer probes this host
+    # over the LAN instead. The cap only bounds a job that is running and stuck
+    # (6h by default), and the concurrency group keeps a re-dispatch superseding
+    # the last queued attempt instead of stacking them.
     timeout-minutes: 5
     concurrency:
       group: health-probe-loopback
@@ -78,20 +77,20 @@ jobs:
       # drill can neither mint an alert that reads like an outage nor act on a
       # standing real one. Set explicitly rather than left to the script
       # default: inheriting the default is what made the drills invisible.
-      PC_ORIGIN_ID: ${{ github.event_name == 'workflow_dispatch' && 'cotel-health-probe-drill' || 'cotel-health-probe' }}
+      PC_ORIGIN_ID: cotel-health-probe-drill
       GITHUB_RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
     steps:
       - uses: actions/checkout@v5
 
       # The pager is the only thing here that wakes anyone, and this runner is
       # a developer Mac rather than a managed image. Check its interpreters on
-      # every green hour, so a missing one surfaces before an incident needs
-      # them.
+      # every run, so a missing one surfaces during a drill rather than when an
+      # incident needs them.
       - name: Check pager prerequisites
         run: |
           for bin in jq python3; do
             command -v "$bin" >/dev/null \
-              || echo "::warning title=Pager prerequisite missing::${bin} is not on this runner's PATH; a red hour would not page"
+              || echo "::warning title=Pager prerequisite missing::${bin} is not on this runner's PATH; a red probe would not page"
           done
 
       # No CF_ACCESS_* here: loopback has no Cloudflare in front of it, and the
@@ -114,22 +113,22 @@ jobs:
       # Paging failures are a warning, never the job's verdict: this job's
       # red/green must mean "production is red/green" and nothing else.
       - name: Page Paperclip on red
-        if: env.PROBE_RC != '0' && (github.event_name == 'schedule' || inputs.page)
+        if: env.PROBE_RC != '0' && inputs.page
         env:
           CF_ACCESS_CLIENT_ID: ${{ secrets.CF_ACCESS_CLIENT_ID }}
           CF_ACCESS_CLIENT_SECRET: ${{ secrets.CF_ACCESS_CLIENT_SECRET }}
         run: |
           scripts/page-cotel-health.sh raise "$PROBE_OUT" \
-            || echo "::warning title=Pager failed::could not raise the Paperclip alert; nobody was woken for this red hour"
+            || echo "::warning title=Pager failed::could not raise the Paperclip alert; nobody was woken for this red probe"
 
       - name: Wake the alert's assignee on green
-        if: env.PROBE_RC == '0' && (github.event_name == 'schedule' || inputs.page)
+        if: env.PROBE_RC == '0' && inputs.page
         env:
           CF_ACCESS_CLIENT_ID: ${{ secrets.CF_ACCESS_CLIENT_ID }}
           CF_ACCESS_CLIENT_SECRET: ${{ secrets.CF_ACCESS_CLIENT_SECRET }}
         run: |
           scripts/page-cotel-health.sh resolve "$PROBE_OUT" \
-            || echo "::warning title=Pager failed::could not wake the alert's assignee; the standing Paperclip alert will keep deduping later red hours"
+            || echo "::warning title=Pager failed::could not wake the alert's assignee; the standing Paperclip alert will keep deduping later red probes"
 
       - name: Fail the job when the probe is red
         if: env.PROBE_RC != '0'
@@ -140,7 +139,7 @@ jobs:
 
   probe-edge:
     name: Probe /healthz through Cloudflare
-    if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
+    if: github.event_name == 'workflow_dispatch'
     runs-on: ubuntu-latest
     env:
       HEALTHZ_URL: ${{ inputs.url || 'https://cotel.aignite.pl/healthz' }}
@@ -155,7 +154,7 @@ jobs:
       # halves would fight: a green loopback hour would resolve the alert the
       # edge half had just raised. A dispatched run is an exercise and gets a
       # separate marker again, for the same reason the loopback half does.
-      PC_ORIGIN_ID: ${{ github.event_name == 'workflow_dispatch' && 'cotel-health-probe-drill-edge' || 'cotel-health-probe-edge' }}
+      PC_ORIGIN_ID: cotel-health-probe-drill-edge
       GITHUB_RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
     steps:
       - uses: actions/checkout@v5
@@ -173,19 +172,19 @@ jobs:
           echo "PROBE_RC=$rc" >> "$GITHUB_ENV"
 
       - name: Page Paperclip on red
-        if: env.PROBE_RC != '0' && env.PROBE_RC != '4' && (github.event_name == 'schedule' || inputs.page)
+        if: env.PROBE_RC != '0' && env.PROBE_RC != '4' && inputs.page
         run: |
           scripts/page-cotel-health.sh raise "$PROBE_OUT" \
-            || echo "::warning title=Pager failed::could not raise the Paperclip alert; nobody was woken for this red hour"
+            || echo "::warning title=Pager failed::could not raise the Paperclip alert; nobody was woken for this red probe"
 
       - name: Wake the alert's assignee on green
-        if: env.PROBE_RC == '0' && (github.event_name == 'schedule' || inputs.page)
+        if: env.PROBE_RC == '0' && inputs.page
         run: |
           scripts/page-cotel-health.sh resolve "$PROBE_OUT" \
-            || echo "::warning title=Pager failed::could not wake the alert's assignee; the standing Paperclip alert will keep deduping later red hours"
+            || echo "::warning title=Pager failed::could not wake the alert's assignee; the standing Paperclip alert will keep deduping later red probes"
 
       # Exit 4 is "cannot see /healthz", not "cotel is down". Failing on it
-      # would paint this job red every hour until the Access service token is
+      # would paint this job red on every run until the Access service token is
       # allowed on the application, and an alarm that is always on is one
       # nobody reads. It stays a warning until it can actually observe.
       - name: Report the probe verdict

docs/.vitepress/config.js

@@ -61,6 +61,7 @@ export default defineConfig({
             { text: 'ADR-0019 — CI Never Mutates an Issue (superseded)', link: '/decisions/0019-ci-never-mutates-an-issue' },
             { text: 'ADR-0020 — Recovery Arrives as a New Issue (superseded)', link: '/decisions/0020-recovery-arrives-as-a-new-issue' },
             { text: "ADR-0021 — Recovery Wakes the Alert's Assignee", link: '/decisions/0021-recovery-wakes-the-alerts-assignee' },
+            { text: 'ADR-0022 — Health Probe Scheduler Outside This Repo', link: '/decisions/0022-health-probe-scheduler-outside-github' },
           ],
         },
       ],

docs/decisions/0022-health-probe-scheduler-outside-github.md

@@ -0,0 +1,95 @@
+# ADR 0022 — The health probe's scheduler lives outside this repo
+
+**Date:** 2026-10-05
+**Status:** Accepted
+**Deciders:** Prospero (CEO, mechanism choice), Daedalus (CTO, implementation)
+
+---
+
+## Context
+
+The probe and the pager work. Both halves were exercised live: a red probe
+creates the alert, the assignment wakes its assignee, and the recovery routes
+through a wake that the assignee acts on
+([ADR-0021](./0021-recovery-wakes-the-alerts-assignee)). What did not work was
+the leg nobody had measured — the schedule.
+
+`cron: "17 * * * *"` landed on `main` at 00:44Z on 2026-10-05. By 09:46Z it had
+produced **one** run instead of about nine, and `gh run list
+--workflow=health-probe.yml --limit 200` showed exactly one `event=schedule` run
+in the workflow's whole history. Two innocent explanations were checked and
+excluded: a cancelled run would still be listed as `cancelled`, and the 60-day
+deactivation of schedules on idle public repositories had not fired (the
+repository was active that day). GitHub schedules public repositories on a
+best-effort basis and **drops** ticks rather than delaying them.
+
+That matters more here than it would elsewhere. This probe exists because
+production sat dead for six days with nobody looking; a watcher whose detection
+delay is unbounded is a quieter version of the same silence, and nothing watches
+whether the watcher ran.
+
+A second gap compounded it. The loopback half runs on a self-hosted runner on
+the deploy host itself, so when that host is off the job does not go red — it
+queues, and `timeout-minutes` does not bound queue time. The failure mode most
+worth catching produced no colour at all.
+
+## Options considered
+
+1. **A systemd timer on the Pi** (`~/ops`), probing `robmini` over the LAN and
+   calling this repo's existing pager.
+2. **A scheduled Paperclip routine** — a reliable scheduler that wakes an agent
+   by construction.
+3. **Leave the GitHub schedule as it is** and accept best-effort detection.
+
+## Decision
+
+**Option 1.** `cotel-healthz.timer` on the Pi, every 10 minutes, paging on the
+second consecutive failure. It owns the production alert marker
+`[cotel-health-probe]`. The GitHub workflow keeps `workflow_dispatch` and its
+pull-request tests but no longer schedules anything; both of its probe jobs page
+drill markers only.
+
+Reasons, in the order they decided it:
+
+- **systemd does not drop ticks**, and `Persistent=true` makes up a tick missed
+  across a reboot. That is the entire defect being fixed.
+- **Probing from the Pi over the LAN catches the host being off**, which the job
+  on that host structurally cannot report. One prober now covers both failure
+  classes: dead process (non-200, refused) and dead host (timeout, refused).
+- **It costs nothing recurring.** Option 2 was rejected on exactly that: ~24
+  full agent runs a day to perform a check that is one `curl`, right after this
+  company cut idle heartbeat burn by moving the interval from 1h to 6h. A probe
+  should be cheap enough that its cadence is a free variable — at 10 minutes
+  this one spends a loopback HTTP request and nothing else while production is
+  healthy.
+- **Option 3 was rejected** because unbounded detection delay is the thing the
+  parent ticket exists to remove.
+
+Consequences accepted with it:
+
+- **The scheduler is not in this repo, and cannot be.** The repo documents where
+  it is (`docs/operations/health-probe.md` → `~/ops/README.md`) and the timer
+  materializes this repo's scripts from `origin/main` at each tick, so the code
+  path stays reviewed here and reaches the timer with no sync step. A reader of
+  this repo alone cannot see the schedule; a reader of either runbook can.
+- **One machine, one prober.** If the Pi is off, nothing probes — the same shape
+  as before, with a different single point. The Pi is where the Paperclip
+  instance and every agent already run, so an off Pi has no one to wake anyway.
+- **Edge coverage is now unscheduled.** Tunnel, DNS and Cloudflare Access are
+  outside the LAN and therefore outside what the Pi can see. This is a real loss
+  only on paper: the edge probe cannot observe `/healthz` at all today, because
+  its Access service token is not allowed on the application, so it returns exit
+  4 and watches a login redirect. Allowing that token is the prerequisite for an
+  edge schedule being worth anything, and it is a board action, not a code
+  change.
+- **The pager's defaults are GitHub's.** The alert description tells the woken
+  agent how to re-probe, and the default text names a workflow dispatch — which
+  for a Pi-raised alert would point at a runner on the possibly-dead host. The
+  pager therefore takes `PC_SOURCE_LINE` and `PC_REPROBE_HINT` overrides, and
+  the timer sets them to the local command. Any future non-Actions caller must
+  do the same.
+- **The credential is wider than ideal.** The timer uses the board ops key
+  already on that machine, because minting a narrow agent key is board-only.
+  Dropping an agent key at `~/.secrets/cotel-health-pager.token` narrows it with
+  no code change. ADR-0021's refusal of a board key stands where it was aimed —
+  a public repository's CI — and does not extend to the owner's own host.

docs/decisions/index.md

@@ -29,3 +29,4 @@ New ADRs go in this directory as `NNNN-short-title.md`, numbered sequentially.
 | [ADR-0019](./0019-ci-never-mutates-an-issue) | CI never mutates an issue: recovery wakes the alert's assignee | Superseded by ADR-0021 |
 | [ADR-0020](./0020-recovery-arrives-as-a-new-issue) | Recovery arrives as a new issue, and dedup is time-bounded | Superseded by ADR-0021 |
 | [ADR-0021](./0021-recovery-wakes-the-alerts-assignee) | Recovery wakes the alert's assignee, and dedup is time-bounded | Accepted |
+| [ADR-0022](./0022-health-probe-scheduler-outside-github) | The health probe's scheduler lives outside this repo | Accepted |

docs/index.md

@@ -100,4 +100,4 @@ Single Go binary. Single DuckDB file. Single named volume. No sidecars.
 
 - [Architecture Decisions](/decisions/) — ADRs explaining key technical choices
 - [Design Docs](/design/) — UI information architecture, page specs, components, tokens
-- [Production /healthz probe](/operations/health-probe) — hourly check of production `/healthz` and who is woken when it is red
+- [Production /healthz probe](/operations/health-probe) — the 10-minute check of production `/healthz`, where its scheduler lives, and who is woken when it is red