feat(health): watch the public ingest path from the Pi tick, without an Access token

[Fl0p]

What this fixes

The scheduled probe only asked the dashboard /healthz on robmini over the LAN, so everything Cloudflare adds was unwatched. That is not theoretical: docker compose up -d without the deploy environment blanks CLOUDFLARE_TUNNEL_TOKEN and drops public ingest while the LAN /healthz stays green. Every agent then exports spans into a hole and keeps working silently — the exact silence the probe was built to end.

The runbook said a meaningful check from outside the LAN had to wait for an Access service token to be allowed on cotel.aignite.pl. It does not. Measured:

https://otlp.aignite.pl/           -> 404   (the application's own mux)
https://otlp.aignite.pl/v1/traces  -> 401   (the ingest handler's auth middleware)

The ingest host is not behind Access, and that 401 proves the whole chain: DNS, the tunnel, the process, and /v1/traces being routed.

What changed

• scripts/probe-edge-ingest.sh (new) — asks https://otlp.aignite.pl/v1/traces and expects exactly 401, not 200 and not "any answer" (a Cloudflare interstitial would otherwise pass as health). It sends a deliberately invalid cotel_ bearer rather than no bearer, so the expectation holds whichever way allow_anonymous is set; with anonymous ingest allowed a tokenless request would reach the handler and answer 405. GET, so a bypassed auth check could not write anything. Exit codes: 0 handler 401, 1 unreachable, 2 answered but not the handler's 401, 4 Access now fronts the ingest host.
• scripts/page-cotel-health.sh — learns PC_ALERT_SUBJECT and PC_ALERT_LEAD, both defaulting to the strings it already sent. The dedup marker is machine-facing; without its own subject a second watcher mints an alert whose title and wake reason claim production /healthz is red, sending the reader to the wrong half of the system.
• scripts/probe-edge-ingest_test.sh (new) + 4 cases in the pager test, wired into the test job and the push path filter.
• docs/operations/health-probe.md — the vantage-point table now carries both scheduled halves, the "needs an Access token to be worth having" claim is corrected, and the new half, its exit codes, the apart-classification rule and its drill recipe are documented.

Not in this repo (host side, sanctioned under this probe): ~/ops/cotel-healthz.sh runs both halves per tick with separate streak files (state, state-edge), separate dedup markers ([cotel-health-probe], [cotel-ingest-edge]), its own --half local|edge selector, and the alert strings above. No GitHub Actions schedule is revived — GitHub dropped eight of nine ticks, systemd does not.

Two deliberate asymmetries. While the LAN half is red the edge half does not page: a dead process makes the public path unreachable as a consequence, and a second alert would send its reader hunting Cloudflare for a dead container. The edge streak keeps counting through the suppression, so it pages on the next tick if the process returns and the public path does not. And a probe script that is not at the materialization ref is reported as "that half is not deployed" rather than failing the unit — the LAN half must not go dark because the other half has not landed, and the new half starts running by itself on the first tick after this merges.

How it was verified

All on the Pi, against origin/flo-1005-edge-ingest-probe as the materialization ref, so the real git show path was exercised:

bash scripts/probe-edge-ingest_test.sh    # passed=10 failed=0
bash scripts/probe-healthz_test.sh        # passed=11 failed=0
bash scripts/page-cotel-health_test.sh    # passed=83 failed=0

• live endpoint, both halves green — probe-healthz: OK — HTTP 200 ingest age 43s and probe-edge-ingest: OK — HTTP 401 from the ingest handler, --probe-only exit 0.
• unreachable endpoint (closed port 9, production untouched) — probe-edge-ingest: FAILED — public ingest unreachable (connection refused), exit 1.
• wrong answers, from the mock: an HTML 401 → "not from the ingest handler"; a 530 → "cloudflare reached, the origin did not answer"; a 404 → "/v1/traces is not routed"; a 200 → "accepting unauthenticated spans"; an Access redirect → exit 4.
• both halves red — the edge half logged not paging: the LAN /healthz half is red too, and its alert covers this and created nothing; the streak still advanced.
• red path end to end under its own drill marker — two red edge ticks opened a drill alert titled cotel public ingest at … is red [cotel-ingest-edge-drill], whose body leads with "The application is alive; its public ingest path is not" and points at the tunnel and DNS. Then the green half routed its close. Drill streaks were left at their true value (0).

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features
  • Production monitoring now checks the public telemetry ingest endpoint every 10 minutes, alongside the existing health check.
  • Ingest alerts have their own subject and description. Ingest failures are tracked separately and do not trigger paging while the LAN health check is already failing.
• Documentation
  • Added guidance for ingest-probe results, alert behavior, limitations, and local testing.

[coderabbitai]

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: defaults
• Review profile: CHILL
• Plan: Advanced
• Run ID: b33b26de-f7a8-48ef-947b-4b41e87a9d66

📥 Commits

Reviewing files that changed from the base of the PR and between ace2a50 and 79f1170.

📒 Files selected for processing (6)

• .github/workflows/health-probe.yml
• docs/operations/health-probe.md
• scripts/page-cotel-health.sh
• scripts/page-cotel-health_test.sh
• scripts/probe-edge-ingest.sh
• scripts/probe-edge-ingest_test.sh

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

---

📝 Walkthrough

Walkthrough

The pull request adds a probe for public OTLP ingest, enables configurable alert wording, and updates the health-probe documentation to describe scheduled LAN and edge checks.

Changes

Health probe and alerting

Layer / File(s)	Summary
Public ingest probe and validation scripts/probe-edge-ingest.sh, scripts/probe-edge-ingest_test.sh, .github/workflows/health-probe.yml, docs/operations/health-probe.md	The new probe sends a GET with an invalid bearer token and treats only an application-style JSON HTTP 401 as healthy. Its tests cover response classifications, connection failures, and environment overrides. The workflow runs the test, and the documentation describes probe behavior and exit codes.
Configurable alert wording scripts/page-cotel-health.sh, scripts/page-cotel-health_test.sh, docs/operations/health-probe.md	The pager accepts PC_ALERT_SUBJECT and PC_ALERT_LEAD values for alert and wake text. Tests cover edge-ingest alert creation and raise and resolve wake paths. The documentation describes the overrides.
Two-half timer operations docs/operations/health-probe.md	The documentation describes separate LAN and edge checks, failure streaks, alert markers, commands, deployment status, and drill procedures.

Priority: ⬇️ Low

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Feature

Merge Risk: ⚪ Minimal · up to 79f11

This change adds a public-ingest health check and lets alerts name the service being watched. Existing alert wording stays the same by default. No established defect blocks merging.

Security Architecture Review

Security architecture risk: 🔵 Low · up to 79f11

The new check uses no real credential by default and cannot write spans. Risk is limited to monitoring assurance: its accepted response does not uniquely identify the intended application, and the production watcher’s deployment and recovery behavior remain unverified.

Retained concerns

• Low · security · inferred: The new healthy verdict does not uniquely identify application authentication. An upstream or misrouted endpoint returning a JSON-like 401 containing an error key, without the recognized Access header patterns, could report healthy while the intended origin is unavailable. This is a conditional monitoring-control risk, not evidence of a deployed bypass.

Security review details

Security Blast Radius

• inferred — The new request is scoped to one configured ingest URL and carries no real credential by default. A misleading verdict could affect outage detection for exporters sharing that endpoint, but does not itself grant tenant, storage or deployment privileges. The deployed exporter population and environment scope are unknown.

Security Findings and Attack Paths

• inferred — The conditional false-green path requires the configured endpoint or an upstream responder to return an accepted JSON-like 401 while the intended application path is unusable. Ordinary unauthenticated ingest clients are not shown to have that capability, and no deployed matching challenge or verified exploitation is established.

Trust Boundaries and Controls

• observed — The probe preserves curl’s default TLS verification, does not request redirect following, and rejects recognized Access challenges and HTML 401 bodies. These controls strengthen classification, but the accepted body pattern is broader than the application’s specific unauthorized response.

Resilience and Maintainability Implications

• observed — The pager change substitutes human-facing wording while preserving alert branching and recovery identifiers. The new strings are passed as data through printf and jq arguments, not evaluated as shell code. Cross-watcher ownership and concurrency still depend on the unavailable host caller.

Hardening Proposals

• proposed — Validate the specific application error contract rather than accepting any error-key body, and describe success as evidence consistent with routing and authentication rejection rather than unique origin proof. Validate the external caller’s distinct state and marker ownership through failure, concurrent ticks, partial deployment and recovery.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 3 functions across 4 files. (2 skipped: 2 …	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly describes the main change: adding a public ingest probe that runs from the Pi without an Access token.

Full details: Docstring Coverage

Explanation

Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 3 functions across 4 files. (2 skipped: 2 unsupported.)

• Fix all pre-merge checks with AI

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}