Docs testing framework: static checks, anchor validation, live smoke checks, and CI wiring

[emmanueletienne-blip]

Summary

A layered docs testing framework for buildwithfern.com/learn that fails at the earliest stage that can catch a problem:

Stage	Workflow	What runs
PR, source	docs-checks.yml	unit tests, scripts/docs_check static checks, fern check
PR, external links	docs-checks.yml	lychee over http(s) links in changed files (shared .github/lychee.toml, extracted from check-links.yml)
PR, rendered preview	preview-docs.yml	scripts/docs_check/smoke.py against the preview for every page the PR touches
push to main	publish-docs.yml	smoke.py against production for every derived URL after fern generate
weekday schedule	docs-checks.yml	full static run + full production smoke run; failures open/update a tracking issue

Static checker (python3 -m scripts.docs_check, PyYAML only) rebuilds every page's published /learn/... URL from fern/docs.yml + product navigation (product slug, skip-slug, slug: overrides, frontmatter slug relative to the product root, CamelCase splitting GitLab → git-lab, APIs → ap-is) and validates the tree against that model. Errors: nav-path-missing, broken-internal-link, broken-anchor, missing-snippet, missing-asset, changelog-filename, changelog-missing-tags. Warnings: redirected-link, relative-page-link, orphan-page, unused-snippet, missing-title, missing-description, stub-page, changelog-h1. Plus a per-product coverage table in the job summary and a JSON artifact.

broken-anchor: #fragment on same-page and /learn/... links must match an id the target page renders. Ids are derived from source in document order with a shared duplicate counter (api, api-1): headings (with optional [#id]), <Step>/<Tab>/<Accordion> titles, <ParamField path>, <Anchor id>, recursing into included snippets. A first version that only modeled headings produced 164 false positives; the shipped version reports 36, every one confirmed missing in the live HTML. A residual gap: some generated components apply their own id transformation (one link, #user-agent on the PHP configuration page, resolves in source but not on the site), documented in the README.

smoke.py: fetch derived URLs from a deployed origin; fail on non-200, Page not found / Something went wrong in the body, or an image under <main> that does not load. --changed maps edited pages to their URL and edited snippets to every including page; an edited navigation YAML widens the run to the whole site. 404/5xx are retried with backoff so a propagating deploy does not fail the run.

$ python3 -m scripts.docs_check.smoke
checking 335 pages on https://buildwithfern.com
0 failures across 335 pages

The one 404 the first run found (/learn/api-definitions/ferndef/depending-on-other-apis) was a slug-derivation bug, not an unpublished page: the platform splits APIs as ap-is. Fixed in slugify.

Baseline: scripts/docs_check/baseline.txt suppresses 194 pre-existing findings (<check> <path> <message>, one occurrence per line) so the job is green today and only new problems fail a PR; the 36 anchor entries are real broken fragments for triage. Content fixes: two broken internal links on main.

Verification

• python3 -m unittest discover -s scripts/docs_check/tests -t . → 13 tests OK
• python3 -m scripts.docs_check --strict → 0 errors, 0 warnings (194 suppressed by baseline)
• smoke.py against production → 0 failures across 335 pages; against the PR preview with --changed → 0 failures
• lychee 0.18 with .github/lychee.toml over sample changed files → 0 errors
• fern check --warnings unchanged; zizmor: no new findings; Vale passes

Link to Devin session: https://app.devin.ai/sessions/4b823dd919a841988f58f31b6dc722a2
Open in Devin Desktop: https://app.devin.ai/desktop/session/4b823dd919a841988f58f31b6dc722a2?variant=devin
Requested by: @emmanueletienne-blip

---

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[github-actions]

🌿 Preview your docs: https://fern-preview-devin-1789659014-docs-check-framework.docs.buildwithfern.com/learn

Here are the markdown pages you've updated:

• learn/docs/getting-started/technical-evaluation