fix(health): give a dispatched probe run its own alert marker

[Fl0p]

What

A workflow_dispatch drill of the health probe paged the production dedup
marker from both halves — the loopback job set no PC_ORIGIN_ID and inherited
the script default, the edge job hardcoded the scheduled one. So a red drill
minted an alert indistinguishable from an outage (five of them this morning),
and a green drill could clear or file recovery against a standing real alert
nobody had read yet. That second half is the dangerous one.

• .github/workflows/health-probe.yml: PC_ORIGIN_ID is derived from
  github.event_name on both jobs. schedule keeps today's markers;
  workflow_dispatch gets cotel-health-probe-drill /
  cotel-health-probe-drill-edge. Set explicitly on the loopback job rather
  than left to the script default — inheriting it is what made the drills
  invisible.
• HEALTHZ_URL moves from the loopback probe step to the job, so the pager
  steps can read it.
• scripts/page-cotel-health.sh: a new alert's body quotes the triggering
  event, github.actor and the effective probe URL. The probe text alone
  cannot say — a closed-port drill and a dead process produce the same line,
  and the only clue used to be port 9 inside a curl error string.
• scripts/page-cotel-health_test.sh: the drill and production markers are
  now asserted to be different alerts in both directions, and the create
  body is asserted to carry the trigger context.
• docs/operations/health-probe.md: a "Running a drill" section — which
  marker a dispatched run uses, and that the dispatcher closes their own drill
  artifacts.

No behaviour change on a scheduled hour.

Verification

Filled in from a real dispatch on this branch — see the comment below.

Summary by CodeRabbit

• Improvements
  • Health-probe alerts now include the probe URL and, when available, the event type and actor. Alert instructions remind responders to re-probe before acting.
  • Drill alerts use markers distinct from scheduled probe alerts, keeping the two modes separate when raising or resolving issues.
• Documentation
  • Updated health-probe guidance explains scheduled workflow inactivity, drill-specific alert context, and the need to close issues created by drill runs.

[coderabbitai]

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

Note

Currently processing new changes in this PR. This may take a few minutes, please wait...

⚙️ Run configuration

• Configuration used: defaults
• Review profile: CHILL
• Plan: Advanced
• Run ID: aecc1970-ed3f-4a04-aace-9b6a6eb561b7

📥 Commits

Reviewing files that changed from the base of the PR and between d8f4e37 and aeba2d3.

📒 Files selected for processing (4)

• .github/workflows/health-probe.yml
• docs/operations/health-probe.md
• scripts/page-cotel-health.sh
• scripts/page-cotel-health_test.sh

 _________________________________________________
< Because, even your code needs a second opinion. >
 -------------------------------------------------
  \
   \   (\__/)
       (•ㅅ•)
       / 　 づ

✨ Finishing Touches

📝 Generate docstrings

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[Fl0p]

Drill run 37287230496 (this branch, 09:00Z) confirms the fix live, end to end:

• loopback half: PC_ORIGIN_ID: cotel-health-probe-drill, edge half cotel-health-probe-drill-edge — both derived, neither inherited the scheduled marker.
• the minted alert is FLO-995 [cotel-health-probe-drill], a separate dedup lane from the production [cotel-health-probe] one, and its body carries Triggered by: workflow_dispatch (Fl0p) + Probe URL: http://127.0.0.1:9/healthz. That is enough to triage it as a drill without opening the Actions log, which is what the five earlier drills lacked.
• independent verdict on main with page=false (37287320520): probe-healthz: OK — HTTP 200 ingest age 3s on 127.0.0.1:8080. Production was never down; FLO-995 closed as drill litter.

One blocker before merge: the branch conflicts with main — docs/operations/health-probe.md, content conflict against #130 (ADR-0020). mergeStateStatus: DIRTY, so the PR will not go green from a merge of main alone; the doc needs a manual resolve. Leaving that to you since this is a board-authored PR — say the word and I will rebase it.

[Fl0p]

Verification

Rebased onto main after ADR-0020 landed (#130); the one conflict was the
Schedule-section drill paragraph, now folded into the new Running a drill
section with ADR-0020's "the green half does not close the alert" caveat kept.

Unit tests (ubuntu-latest in this PR's Probe script tests job, and locally):

bash scripts/page-cotel-health_test.sh   ->  passed=36 failed=0
bash scripts/probe-healthz_test.sh       ->  passed=11 failed=0

Six of those 36 are new: the create body carries Triggered by / Probe URL /
Dedup marker, and the drill and production markers are asserted to be
different alerts in both directions (a drill raise does not comment on an
open production alert, a drill resolve does not PATCH it, and a production
resolve does not PATCH a drill alert).

Red drill — dispatch on this branch, page=true,
loopback_url=http://127.0.0.1:9/healthz
(run 37287230496):

Probe /healthz on the deploy host    PC_ORIGIN_ID: cotel-health-probe-drill
probe-healthz: FAILED — unreachable (connection refused; curl: (7) Failed to connect to 127.0.0.1 port 9 ...)
page-cotel-health: opened FLO-995 6a871c35-8850-48a8-b7ac-1b96f27b7575

The alert it opened reads:

Title: cotel prod /healthz is red [cotel-health-probe-drill]

Dedup marker: [cotel-health-probe-drill]
Triggered by: workflow_dispatch (Fl0p)
Probe URL: http://127.0.0.1:9/healthz

— the drill marker, not [cotel-health-probe], and the three facts a woken
reader needs without opening Actions. The edge half exited 4 (Access blocked),
so it warned and did not page, as designed.

Green drill — dispatch on this branch, page=true, default URLs
(run 37287330293,
conclusion success):

Resolve Paperclip alert on green
  HEALTHZ_URL: http://127.0.0.1:8080/healthz
  PC_ORIGIN_ID: cotel-health-probe-drill
probe-healthz: OK — HTTP 200 ingest age 8s (threshold 21600s) url=http://127.0.0.1:8080/healthz
page-cotel-health: FAILED — alert resolve: HTTP 401
page-cotel-health: this credential cannot close an alert from CI, ...

Two things to read there. The resolve ran against cotel-health-probe-drill,
and it found an open alert — it got as far as the PATCH, which only happens
when the search matched an open title-marker alert. At that moment the
production marker had nothing open (every [cotel-health-probe] issue was
already done/cancelled), so a production-marker resolve would have printed
no open alert and exited 0. Reaching the PATCH is therefore positive evidence
it acted on the drill alert and not on production. No new issue carrying
[cotel-health-probe] exists from either dispatch.

The HTTP 401 is the pre-existing gate ADR-0020 describes, not a regression
from this PR — it is what the next ticket replaces with a recovery-notice
create. Per this PR's own rule the drill artifact FLO-995 is closed, with a
comment saying it was a drill.

Scheduled hour unchanged — the workflow, grepped:

81:      PC_ORIGIN_ID: ${{ github.event_name == 'workflow_dispatch' && 'cotel-health-probe-drill' || 'cotel-health-probe' }}
155:     PC_ORIGIN_ID: ${{ github.event_name == 'workflow_dispatch' && 'cotel-health-probe-drill-edge' || 'cotel-health-probe-edge' }}

bash 3.2 (the loopback half runs on the macOS runner): no mapfile,
readarray, ${x,,}, ${x^^} or declare -A anywhere in
scripts/page-cotel-health.sh. The only array use is AUTH_HEADERS+=(...),
which predates this PR and is 3.1+.

No tracker IDs in the diff (git diff | grep -E '^\+.*FLO-[0-9]+' is empty).

[Fl0p]

Two requirements carried in from FLO-994, which I cancelled as a duplicate of this ticket. Both belong here before merge:

1. DRILL in the alert title, not only a separate dedup marker. A distinct marker splits the dedup state, which is what this PR is about - but the agent triaging the wake reads the title, and today a drill alert's title is byte-identical to an outage's.
2. The actual HEALTHZ_URL and the run's event in the body. That is what identifies a drill in one second, instead of inferring it from the port buried in a curl error string.

The cancelled ticket's reasoning for a marker rather than suppressing the page stands and is worth putting in the PR description: a drill must run the whole path through issue-create and the assignee wake, or it proves nothing about delivery. Only the distinguishability changes.

Posting here rather than on the ticket: a live run holds FLO-990's lock, and this is where the work is anyway. Checks read UNSTABLE as of 09:38 - worth a look alongside this.