Conversation
A daily pg_cron job (pgflow_telemetry.report) aggregates yesterday's pgflow.* activity into coarse, identifier-free buckets (active-day, worker versions, run/worker counts, flow shapes, feature adoption, durations) and posts one small JSON payload via pg_net to a Cloudflare Worker backed by Workers Analytics Engine. Edge workers stamp their package version into pgflow.workers at registration.
Design decisions: the cron.job row is the only switch — the migration schedules it exactly once and nothing re-schedules it, so disable() stays permanent across upgrades. pgflow_telemetry.sent_reports doubles as user-auditable payload log and per-day dedup marker. report() gates run in order: inactive day, already reported, is_local(); one pg_net attempt, 5s statement timeout, no retries, no response inspection. Metric names, bucket strings, and count buckets are closed allowlists identical across SQL (0133), the ingest worker (apps/telemetry-worker), and docs; anything else never leaves the database. Local supabase-start stacks never send.
Compatibility fixes found during implementation: Deno 2.1.4 requires the JSON import attribute ('with { type: json }') and a default import for package.json version lookup, so jsr.json publish.includes package.json and the e2e vendor script copies it; bucket_steps takes bigint because steps_per_flow passes count(*); 0136 uses $disable$/$enable$ dollar-quote tags because the plan's nested $$ would terminate enable()'s body early. Plan test bugs corrected with rationale in the tests: 90s lands in 1-4.9m not 1-9.9s; bucket_count(2) is '2-3'; the 65-contribution body always exceeds the 2KB cap so 413 fires before the count guard.
Verification: 4 new pgTAP files (red/green, migration replay with deps), telemetry-worker vitest 10, full pgTAP suite 314 files/1606 tests, nx affected build+test, e2e 13/13, live report() returns skipped: local with zero audit rows, live worker registration stamps 0.17.0, website build. Ingest worker deployment (wrangler deploy + smoke) is a manual follow-up and is not part of this commit.
🦋 Changeset detectedLatest commit: 56bef34 The changes in this PR will be included in the next version bump. This PR includes changesets to release 5 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
View your CI Pipeline Execution ↗ for commit 56bef34
💡 Verify your cache is correct by running tasks in a sandbox. Read docs ↗ ☁️ Nx Cloud last updated this comment at |
CI build-and-test failed on @pgflow/telemetry-worker:typecheck with TS5069: the @nx/js plugin infers 'tsc --build --emitDeclarationOnly' for every root-tsconfig project, but this wrangler-deployed worker's tsconfig sets noEmit and never emits declarations. A package.json 'typecheck' script overrides the inferred target with tsc --noEmit — the same per-project override pattern pkgs/client uses (project.json noEmit target) instead of touching nx.json. Verified: focused typecheck exit=0; nx affected -t lint typecheck test green except the known unrelated demo Groq tests (sandbox GROQ_API_KEY, decommissioned llama-3.1-8b-instant).
![graphite-app[bot]](https://avatars.githubusercontent.com/in/158384?s=60&v=4){kind=small}
Comment on lines
+120
to
+126
| for (const c of contributions) { | ||
| env.PGFLOW_TELEMETRY.writeDataPoint({ | ||
| indexes: [c.metric as string], | ||
| blobs: [c.bucket as string], | ||
| doubles: [1], | ||
| }); | ||
| } |
Contributor
There was a problem hiding this comment.
The count field from contributions is validated (lines 113-117) but never used when writing data points. All contributions are written with doubles: [1] regardless of their actual count value. This means aggregated count data (like "count": "8-15") is completely lost.
Fix:
for (const c of contributions) {
env.PGFLOW_TELEMETRY.writeDataPoint({
indexes: [c.metric as string],
blobs: [c.bucket as string, c.count as string ?? ''],
doubles: [1],
});
}Or if count should affect the double value, the SQL queries building these payloads (like line 27 in 0133_function_build_telemetry_payload.sql) would need to be updated to not include the count field at all since it cannot be meaningfully transmitted in the current schema.
Suggested change
| for (const c of contributions) { | |
| env.PGFLOW_TELEMETRY.writeDataPoint({ | |
| indexes: [c.metric as string], | |
| blobs: [c.bucket as string], | |
| doubles: [1], | |
| }); | |
| } | |
| for (const c of contributions) { | |
| env.PGFLOW_TELEMETRY.writeDataPoint({ | |
| indexes: [c.metric as string], | |
| blobs: [c.bucket as string, c.count as string ?? ''], | |
| doubles: [1], | |
| }); | |
| } | |
Spotted by Graphite
Is this helpful? React 👍 or 👎 to let us know.
added 2 commits
September 20, 2026 09:34
…aths Two independent reviews (GLM plan-conformance pass, Sol deep pass) blocked the original telemetry commit with nine findings; this carries the corrected candidate through four review rounds plus the final capability predicate fix. Behavior changes against 4f7f279: - pg_net send: pass the JSONB payload directly to net.http_post(url, body jsonb, ...). The old body => v_payload::text call hit a signature that does not exist locally, so every non-local send failed into the exception handler; is_local() short-circuit kept local pgTAP from ever seeing it. - Payload building (0133): anchor the semver filter (^\d+\.\d+\.\d+$) with a length cap so '1.2.3 evil-string' cannot ride into the payload; group all five histograms by emitted bucket instead of raw values, which duplicated contributions per bucket and could fan out past the caps. - Sender bounds: deterministic 64-contribution / 2048-serialized-UTF-8-byte limits with active_db_day preserved first and unique (metric, bucket) pairs. - report() (0135): initial inactive/already-reported/is_local gates moved inside exception blocks with explicit WHEN query_canceled (WHEN OTHERS cannot catch it), so every failure returns an error status silently instead of escaping into the cron log; queue/audit/prune stay one transaction. - disable() (0136/0137): opt-out no longer trusts cron.unschedule's not-found text or the job_registry. job_registry records the schedule jobid; job_is_scheduled() is a zero-arg SECURITY DEFINER, empty search_path one-bit check that proves actual cross-owner cron.job existence. Its capability proof requires superuser, BYPASSRLS, disabled RLS, or pg_has_role(..., 'usage') inherited table-owner privileges — 'member' would accept NOINHERIT dormant membership and prove nothing (Sol probe: member=t, usage=f, visible_rows=0). disable() returns only when no telemetry job exists across owners and fails closed with an actionable error otherwise, including for incapable helper owners. - Receiver (apps/telemetry-worker): reject duplicate (metric, bucket) pairs, unknown fields, bad content types, and >2048-byte bodies before any Analytics Engine write; store count in the second blob; bounded byte streaming. - Docs: remote AE retention vs local sent_reports pruning vs payload truncation separated; fail-closed disable documented. Migration: the unreleased 20260918145424 telemetry migration is replaced by a regenerated one; exactly one top-level initial cron.schedule remains and no later migration re-schedules an opted-out install. Evidence: focused red/green logs per round under telemetry-evidence/correction-* (sandbox); final gates on the pre-merge tree: pgTAP 316 files / 1663 tests, edge-worker integration 65/65, telemetry-worker 24/24 + typecheck + lint, E2E 13/13 after one bounded retry (CPU-clock-limit timing flake), affected composite green except the known local demo Groq 404s.
Main gained 20260919152659_pgflow_fix_force_skip_multi_queue.sql (PR #686) with schema changes to 0076/0100, which left our telemetry migration generated against a stale schema baseline and an atlas.sum that no longer matches the merged history. Regenerated the unreleased telemetry migration per the migration-management skill on top of the merged tree: reset with the old migration present, removed 20260919185953, rebuilt atlas.sum via atlas-migrate-hash, re-ran atlas-migrate-diff, re-appended the documented one-time top-level cron.schedule + job_registry patch (Atlas cannot express it), refreshed the hash. New migration 20260920093533 sorts after main's force_skip migration and carries the correction-5 pg_has_role 'usage' capability proof. Also folds in the final Sol xhigh blocking fix vs the local correction candidate: job_is_scheduled() capability proof now requires inherited privileges (pg_has_role 'usage'), because 'member' accepts NOINHERIT dormant membership and cannot prove cron.job RLS bypass (probe: member=t, usage=f, visible_rows=0); pg_catalog.format qualified; NOINHERIT regression pinned in optout.test.sql (red on the 'member' body, green on 'usage'). Final gates on the merged tree: verify-migrations/gen-types/verify-gen-types PASS, six telemetry pgTAP files PASS on migration replay, full pgTAP 318 files / 1674 tests PASS, edge-worker integration 65/65 PASS (4m17s). Evidence: telemetry-evidence/correction-5 (sandbox).
Contributor
🔍 Preview Deployment: Website✅ Deployment successful! 🔗 Preview URL: https://pr-684.pgflow.pages.dev 📝 Details:
_Last updated: _ |
Contributor
Author
This stack of pull requests is managed by Graphite. Learn more about stacking. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Anonymous opt-out telemetry
Summary
Ships the researched daily anonymous telemetry (plan:
Plans/2026-09-18-anonymous-telemetry.md, spec:Runs/anonymous-telemetry-research/research.md).pgflow_telemetryschema:sent_reportsaudit/dedup table, bucket helpers,build_payload()covering all 26 metrics,preview()(never sends),report()with gates (inactive day → already reported → local), onepg_netattempt with 5 s timeout, no retries, andenable()/disable()where thecron.jobrow is the switch. The Atlas migration schedules the daily job exactly once; later migrations never re-schedule it.pgflow.workers.pgflow_versioncolumn; edge workers stamp their package version at registration (Deno/JSR-compatible JSON import, vendored for e2e).apps/telemetry-worker: strict allowlist validation (metric names, bucket strings, semver), one Analytics Engine data point per contribution, no cookies, no body logging, 2 KB / 64-contribution caps.pgflow_telemetry.sent_reports;preview()shows any day's exact bytes; local dev stacks never send./reference/telemetry/page, sidebar + reference card + redirect for the renamed news article (combined 0.17.1), changeset (@pgflow/core,@pgflow/edge-worker: patch).Deferred (manual follow-up, not in this PR):
wrangler deployof the ingest worker and the live endpoint smoke test. The SQL endpoint constant ishttps://pgflow-telemetry.workers.dev; if the deployed subdomain differs, it needs a follow-up change per the plan's Task 8 Step 5.Checks
telemetry/{workers_version,schema,payload,report}.test.sqlnx affected -t build test— pass, exceptdemo:test: 4 Groq-API tests fail only when aGROQ_API_KEYis present in the environment (modelllama-3.1-8b-instantdecommissioned); demo sources untouched by this PR and CI has no such keyverify-migrations,gen-types,verify-gen-types— passpreview()shows real contributions;report()returnsskipped: local, zero audit rows; worker registration stampspgflow_version = 0.17.0