Skip to content

fix: correct Seal, compatibility, analytics and runtime-pin docs - #232

Merged
abrichr merged 6 commits into
mainfrom
fix/ledger-wave2-2026-10-14
Oct 14, 2026
Merged

abrichr merged 6 commits into
mainfrom
fix/ledger-wave2-2026-10-14

Conversation

@abrichr

@abrichr abrichr commented Oct 14, 2026

Copy link
Copy Markdown
Member

Four docs pages told readers things that the released software does not do. This change corrects them and adds a regression test for each one, so the same drift fails CI next time.

  • The Seal pages told readers to run openadapt-flow replay bundle --seal. No Flow release has that option.
  • The compatibility table gave package ranges that the newest openadapt and openadapt-flow releases no longer accept.
  • Google Analytics on the docs ran under Do-Not-Track, left Google signals on, and sent the raw search text.
  • Several pages called a Cloud runtime pin "current" when it was two releases out of date.

Fixes

Problem Change Regression test Ref
seal.md, oem-brief.md and llms.txt told readers to run openadapt-flow replay bundle --seal (and openadapt flow replay bundle --seal). Flow 1.35.1 rejects the option with unrecognized arguments: --seal. The pages now say that Execute issues the Seal for approved pilot partners, that the public CLI doesn't issue one yet, and that a replay option for it is planned. openadapt flow seal SOURCE --out DESTINATION is described as what it does: it copies a bundle and encrypts its workflow and template evidence. It doesn't issue a Seal. tests/test_documented_flow_options.py checks the subcommand and long options of all 218 Flow commands that appear as code in the docs against the argparse parser of Flow 1.35.1 (tests/fixtures/flow_cli_1.35.1.json). B209
docs/reference/compatibility.md said openadapt-flow >=1.7,<2, openadapt-capture >=1.1.0, and two openadapt-types ranges (0.7 and 0.9). The newest releases require Flow >=1.29,<2, Capture >=1.2.0,<2 and types >=0.17,<0.18, so following the table caused a resolver conflict. The table states the current ranges and merges the two types rows into one. scripts/check_published_version_claims.py checks the table's structure on every PR. The daily run compares each range with live PyPI requires_dist and requires_python, and files an issue when they drift. validate_docs.py no longer requires the stale literal ranges. tests/test_compatibility_table.py compares the committed table with a PyPI snapshot from 2026-10-14 and covers 8 failure cases. B220
Material's Google Analytics script loaded gtag.js and set _ga cookies under Do-Not-Track. It left Google signals on and sent the search text as search_term. Both contradict ANALYTICS.md and the privacy policy. overrides/partials/integrations/analytics/google.html replaces Material 9.7.7's partial. Under Do-Not-Track it loads nothing. It checks each DNT property separately, so one property set to "0" can't hide another set to "1". Every config command turns Google signals and ad personalization signals off. Search sends only query_length. tests/test_docs_google_analytics.py builds the site and runs the page's analytics scripts in Node against a stub browser. It covers 4 DNT signals, mixed signals, no DNT, and the search event. B221
The deployment matrix, hosted guide, security review and qualification page said "the current managed-runtime manifest pins Flow 1.31.0", with that release's commit and hashes. Cloud has pinned Flow 1.34.0 since 2026-08-28. oem-brief.md called openadapt-types 0.9.0 the released contract. Each pin sentence now reads "As of 2026-10-14" and gives the 1.34.0 values. The release commit and wheel and sdist SHA-256 values appear only on the deployment matrix. They match PyPI and the v1.34.0 tag. The OEM guide links to the compatibility table for the types range. tests/test_runtime_pins.py requires each pin sentence to be dated, to match docs/published-version-claims.json, and not to call the pin current. It also requires the matrix hashes to match the registry and every named openadapt-types release to fall in the table's range. B226

Not fixed here

  • Consent banner (B221). Visitors who don't send Do-Not-Track still get Google Analytics cookies without being asked. openadapt.ai has no consent banner either. ANALYTICS.md now says this plainly. Adding Material's extra.consent is a product decision.
  • Blog analytics (B221). The blog still loads gtag.js under Do-Not-Track. That code is in the openadapt-blog repository.
  • Docs auto-sync (B223). The code fix is already on main (fix: patch the docs XSS advisory and correct docs and ops claims #230). The repository has no DOCS_SYNC_APP_CLIENT_ID variable or DOCS_SYNC_APP_PRIVATE_KEY secret, so the sync job stops with its intended error and docs: auto-sync generated documentation #223 stays blocked.
  • PostHog on the docs (B225). The code fix is already on main (fix: patch the docs XSS advisory and correct docs and ops claims #230). The NEXT_PUBLIC_POSTHOG_KEY Actions variable isn't set, so the published site loads no PostHog.
  • Live Cloud pin comparison (B226). Nothing compares the registry with Cloud's runner/runtime-version.json automatically, because that repository is private. The pin sentences are dated, so they stay accurate until the date and registry are bumped.

How it was tested

  • I made a temporary worktree at origin/main and copied in only the new test files and fixtures. For B220 I also copied in the new checker script, because the test imports it. All 4 new test modules failed for the documented reasons: the 4 --seal lines, the 6 stale table ranges, all 8 analytics cases, and the undated 1.31.0 pin sentences and types 0.9.0 mentions. With this branch's registry swapped in, the B226 hash test also failed on the 1.31.0 commit. All of them pass on this branch.
  • I rebuilt the Flow 1.35.1 parser from build_parser() at tag v1.35.1, and it matches the fixture (82 subcommands, identical options). The PyPI fixture matches live PyPI for openadapt 1.16.0 and openadapt-flow 1.35.1. The 1.34.0 commit and hashes match openadapt-cloud origin/main and PyPI.
  • pytest tests/: 541 passed, 1 skipped (the PostgreSQL drill, which needs OPENADAPT_TEST_POSTGRES_BIN).
  • scripts/validate_docs.py, scripts/check_published_version_claims.py --require-network (including the new PyPI comparison), mkdocs build --strict and uv lock --check all pass. ruff check passes on every changed Python file. The existing ruff findings are in files this change doesn't touch.

Before merging

These are repository settings, not code:

  1. B223: install a GitHub App on this repository with Contents: write and Pull requests: write permissions. Then set the Actions variable DOCS_SYNC_APP_CLIENT_ID and the secret DOCS_SYNC_APP_PRIVATE_KEY, and rerun or reopen docs: auto-sync generated documentation #223.
  2. B225: set the Actions variable NEXT_PUBLIC_POSTHOG_KEY to the openadapt.ai project key. NEXT_PUBLIC_POSTHOG_HOST is optional.
  3. B221: decide whether the docs and openadapt.ai should ask for consent before Google Analytics sets cookies.

🤖 Generated with Claude Code

abrichr and others added 6 commits October 14, 2026 09:58
The Seal and Execute pilot pages and llms.txt told readers to run
`openadapt-flow replay bundle --seal`, and seal.md said the same verb was
available as `openadapt flow replay bundle --seal`. No Flow release has
that flag: Flow 1.35.1 answers `unrecognized arguments: --seal`, so the
documented proof step failed. The pages also described `openadapt flow
seal` only as "encrypts a bundle", without saying it doesn't issue a Seal.

seal.md, oem-brief.md and llms.txt now say that Execute issues the Seal
for approved pilot partners, that the public CLI doesn't issue one yet,
and that a replay option for it is planned. The `seal` command is
described as what it is: it copies a bundle to a new directory and
encrypts its workflow and template evidence with OPENADAPT_BUNDLE_KEY.

tests/test_documented_flow_options.py reads every `openadapt-flow ...`
and `openadapt flow ...` command written as code in the docs (218 today)
and checks its subcommand and long options against the argparse parser
of Flow 1.35.1, recorded in tests/fixtures/flow_cli_1.35.1.json. It
failed on the four `--seal` lines before this change.

Ledger: B209

Signed-off-by: abrichr <richard.abrich@mldsai.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
docs/reference/compatibility.md said openadapt needs `openadapt-flow
>=1.7,<2`, native capture needs `openadapt-capture >=1.1.0`, and gave two
`openadapt-types` ranges (0.7 for Flow, 0.9 for Execute clients) with the
advice not to substitute one for the other. The newest releases require
Flow >=1.29,<2, Capture >=1.2.0,<2, and types 0.17 for Flow's base
install and every extra, so pinning the documented types 0.9 next to Flow
was a resolver conflict. Nothing compared the table with the metadata,
and validate_docs.py required the stale literal ranges, so correcting
the table would have failed the docs check.

The table now states the current ranges, merges the two types rows into
one, and says a daily check compares it with PyPI. The version-policy
bullet no longer names minors that go stale.

check_published_version_claims.py gains a compatibility check. Offline
(every PR) it requires the table to name every tracked package and the
Python range. Online (the daily published-version-claims run, which
files an issue on drift) it compares each range by meaning with the
requires_dist and requires_python of the newest openadapt and
openadapt-flow releases, and fails if Flow ever pins different types
ranges per extra. validate_docs.py keeps only structural phrases.

tests/test_compatibility_table.py compares the committed table with a
PyPI snapshot taken 2026-10-14 and covers the failure cases. It failed
on the six stale ranges before this change.

Ledger: B220

Signed-off-by: abrichr <richard.abrich@mldsai.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Material's Google Analytics integration ran for every docs visitor. It
loaded gtag.js and sent the `config` command, which sets the _ga
cookies, even when the browser sent Do-Not-Track, and it left Google
signals on. The OpenAdapt privacy policy says every browser analytics
path honors Do-Not-Track and that Google signals are disabled. PostHog
on the same page already checked Do-Not-Track. Material's script also
sent the raw search text to Google Analytics, while ANALYTICS.md says
only the query length is recorded.

overrides/partials/integrations/analytics/google.html replaces
Material's partial. Under Do-Not-Track it returns before it queues any
gtag command or inserts gtag.js. Every `config` command sets
allow_google_signals and allow_ad_personalization_signals to false, and
the search event sends only the query length. Material still decides
when the script runs. ANALYTICS.md now describes this and says there is
no consent banner yet, which is an open decision.

tests/test_docs_google_analytics.py builds the site and runs the page's
analytics scripts in Node against a stub browser, for four
Do-Not-Track signals and for none. All five cases failed before this
change. A sixth test ties the override to the locked Material version so
an upgrade prompts a review of the copied partial.

Ledger: B221

Signed-off-by: abrichr <richard.abrich@mldsai.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rent

The deployment matrix, hosted guide, security review, and qualification
page said "the current managed-runtime manifest pins Flow 1.31.0", with
that release's commit and hashes. Cloud's runner/runtime-version.json
has pinned Flow 1.34.0 since 2026-08-28, so the version, release commit,
wheel hash, and sdist hash were all wrong. The OEM guide also called
`openadapt-types` 0.9.0 the released contract while Flow requires 0.17.

Each pin statement now says "As of 2026-10-14" and gives the 1.34.0
values read from openadapt-cloud origin/main 45fa1c8. The hashes stay
only on the deployment matrix, and the hosted guide links there. The OEM
guide points to the compatibility table for the openadapt-types range
instead of naming a release. The registered pin in
published-version-claims.json moves to 1.34.0, records the commit and
hashes as structured fields, and is dated 2026-10-14. Also fixes "an
self-service".

tests/test_runtime_pins.py requires every "manifest pins Flow X" sentence
to match the registered version and date and not to say "current", the
deployment matrix to state the registered commit and hashes (and no
other page to repeat them), and every named openadapt-types release to
fall in the compatibility table's range. All four tests failed before
this change.

Ledger: B226

Signed-off-by: abrichr <richard.abrich@mldsai.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Google Analytics override read the first truthy of
navigator.doNotTrack, window.doNotTrack and navigator.msDoNotTrack, then
compared that one value with "1". A browser that exposed "0" or
"unspecified" on one property and "1" on another loaded gtag.js and set
the _ga cookies. Each signal is now checked on its own, as the PostHog
check in overrides/main.html already does.

The harness now also simulates one search, and a new test requires the
search event to carry only query_length. Both new tests fail on
origin/main; the mixed-signal test also failed on the previous version
of this branch.

Ledger: B221

Signed-off-by: abrichr <richard.abrich@mldsai.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One line of the hosted recorder boundary paragraph ran past the wrap
width after the pin sentence was dated. The registered claim context in
published-version-claims.json follows the new line break.

Ledger: B226

Signed-off-by: abrichr <richard.abrich@mldsai.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@abrichr
abrichr merged commit f38e44e into main Oct 14, 2026
10 checks passed
@abrichr
abrichr deleted the fix/ledger-wave2-2026-10-14 branch October 14, 2026 16:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant