Skip to content

feat(design): add advisory conformance check and local Studio - #8313

Closed
BillLeoutsakosvl346 wants to merge 69 commits into
stagingfrom
codex/design-system-rollout
Closed

BillLeoutsakosvl346 wants to merge 69 commits into
stagingfrom
codex/design-system-rollout

Conversation

@BillLeoutsakosvl346

@BillLeoutsakosvl346 BillLeoutsakosvl346 commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Add a warning-only product design diff check to the existing Test and Build workflow. Findings become file/line warnings; failed inspection remains an error.
  • Derive EMCN and global-style facts directly from each source revision. The diff check and full scanner share the analyzer, and the scanner publishes compact export, variant, slot, and diagnostic facts in versioned scan.json for Studio. No generated metadata file or freshness audit is required.
  • Keep the standalone local Studio in tools/design-studio, outside Sim's deployed routes. An explicit refresh inventories every public EMCN visual export and every detected Extra; missing fixtures stay visible.
  • Exclude landing, docs, native/build code including apps/sim/scripts/, Monaco presentation, provider branding, and block identity palettes. Browser desktop screens remain in scope.

Product appearance is unchanged. This draft targets staging, supersedes #7827, and will not be merged automatically.

Cleanup in this revision

  • Removed contracts.generated.json, its writer and freshness check, and the diff check's artifact dependency. Source-derived metadata remains independently computed for the base and proposed revisions.
  • Removed the control graph's no-op scalar SourceIndex bridge while keeping central recipe resolution. Inspection failures now carry an explicit marker rather than being inferred from diagnostic wording.
  • Kept every unchecked diagnostic in JSON. Terminal and GitHub summaries show changed-file or newly introduced diagnostics, with baseline counts and at most 20 examples. Long finding and diagnostic messages are bounded in human output; JSON retains full evidence.
  • Corrected Studio's Extras description to include findings in EMCN source. Removed obsolete artifact tests and updated the Studio handoff and agent guidance.
  • The earlier cleanup in this draft also removed redundant exception analyzers, unused associations, and duplicated scanner tests while retaining source-only colour tracing, component-chrome coverage, direct use locations, and source-backed fixtures.

Workflow

bun run check:design --base origin/staging --working-tree
bun run check:design --base origin/staging --head HEAD
bun run design:scan --repo . --working-tree --output /tmp/sim-design-scan
bun run studio:refresh
bun run studio:dev

The diff checker includes staged, unstaged, and nonignored new product files. CI compares immutable commits. Full scans and Studio refreshes are explicit local commands and publish reports outside the checkout.

Verification

  • Full scan on unchanged product source: 2,308 findings before and after, matched by file/context/rule/property/value; zero added or removed treatments and zero inspection failures. Styling unchecked diagnostics fell from 2,921 to 2,891: all 30 removed diagnostics were in the newly excluded apps/sim/scripts/ scope.
  • Immutable and working-tree diffs each retain the same 28 findings and 2,197 unchecked diagnostics as the baseline, with zero inspection failures. The working-tree human report shows 557 relevant diagnostics and 1,640 baseline diagnostics, listing at most 20 examples.
  • Studio refresh retains identical component and Extra inventories: 759 EMCN export/variant entries and 2,308 Extra entries. Its existing 237 fixture gaps remain visible; refresh exits 1 for incomplete fixture coverage. All 48 browser fixture cases passed in light/dark modes at 16px/20px.
  • All 675 script tests, design and Studio type checks, lint, and 51 repository audits passed. The production app build passed with CI environment values, and its route tree has no Studio route. A separate 4-case Extra specimen check passed in light/dark modes at 16px/20px. This cleanup removes 793 net lines: 124 production/tooling, 84 tests, and 585 metadata/guidance (including the 575-line committed artifact).

Coverage limits

New public exports and finite variants are discovered automatically, but an export requiring props or providers may need a fixture adapter. Unsupported static flows remain unchecked; a failed inspection is an error. A completed scan does not imply every Extra has a source-backed preview. See scripts/design-conformance/README.md and scripts/design-studio/README.md.

Review follow-up

  • Local text output now tells engineers how to rerun the same check with --format json or --output /tmp/design-check.json when the 20-item unchecked preview is abbreviated.
  • Warning annotations put source and permitted action before a bounded authored value. The JSON report retains the complete input.
  • A cross-process smoke test runs the scanner CLI against a small Git fixture and passes its unmodified scan.json into Studio refresh. It checks the export, variant, direct use, and Extra handoff.

Studio freshness follow-up (97121492c2)

  • scan.json version 3 includes the working-tree source revision; immutable scans record null. The scanner, refresh command, and page loader now use one revision calculation.
  • Studio refuses a stale or immutable injected scan before updating latest.json. Existing publications remain available on rejection.
  • Every detected Extra still renders. Generated samples say “Indicative preview”; fixtures mounting product or EMCN source say “Live source preview.” ready means renderable, not necessarily source-backed.
  • On identical product source, before/after scans had the same 2,308 findings, 2,891 styling diagnostics, 2,254 control records, and 3,018 control diagnostics. Studio retained all 759 EMCN entries and 2,308 Extra entries, with 237 fixture gaps. The only coverage-count difference was the new excluded tooling file.
  • Three real scanner-to-Studio regression cases cover tracked edits, nonignored new product files, and immutable scans. All 675 script tests, design and Studio type checks, lint, and 51 audits passed. Four light/dark and 16px/20px gallery cases and four component fixture cases passed, with screenshots and reports outside the checkout.

Central EMCN ownership follow-up (8d6fcb44bf)

  • Exceptional permissions and protections now live in packages/emcn/src/design-ownership.json: 18 reasoned decisions across ten components. Ordinary exports, variants, defaults, and styling slots remain source-derived. The ten component implementations now match staging byte-for-byte; this change does not alter runtime rendering.
  • Each diff revision reads its own ownership file. Historical revisions without the file have no explicit decisions; deleting an established file or supplying malformed, stale, contradictory, or ineffective decisions fails inspection. A new EMCN export needs no registration unless it has a deliberate exception. Policy changes appear as central design findings.
  • scan.json publishes the decisions with effective export metadata, so the scanner and Studio share the same ownership inventory. Authoring guidance now points to the central file.
  • On refreshed staging, the full scan remains at 2,308 findings, 2,893 styling diagnostics, 568 EMCN exports, and zero inspection failures. Every finding matches the pre-migration baseline by file, context, rule, property, and authored value; all effective styling slots match. Studio still has 759 EMCN export/variant entries, 2,308 detected Extras, and the same 237 fixture gaps. The immutable and working-tree diff checks each completed with 23 central findings, 805 unchecked diagnostics, and no inspection failures.
  • Verification: 678 script tests, 51 audits, design and Studio type checks, and lint passed. Four fixture browser checks passed across light/dark and 16px/20px. Reports and browser captures remain outside the checkout.
  • The PR's clean CI Build App job passed on this head. Two in-place macOS builds of the same checkout hit a Turbopack duplicate-output-path error, including after moving the generated .next directory aside. An isolated local build of current staging passed. This local/CI discrepancy remains unclassified; it did not affect the passing CI production build.

Test layout follow-up (6c3db54527)

  • Colocated all 11 design-conformance test files with their analyzer under scripts/design-conformance/; Git records them as renames. The former generate-design-contracts.test.ts is now generated-contracts.test.ts.
  • Updated Vitest discovery, CLI-relative test paths, the design type-check scope, and the script-test coverage audit. The audit now walks script subdirectories and catches tests missed by the configured globs; scripts/openapi keeps its separate suite. Removed a stale lint/format glob for the deleted generator test path.
  • Verification: the same 38 script test files and 678 tests pass; 51 audits, design type check, lint, and format checks pass. The committed design diff remains completed with 23 findings, 805 unchecked diagnostics, and zero inspection failures.

Tooling layout follow-up (66e1cd3abf)

  • Grouped the shared analyzer under scripts/design-conformance/{shared,system,styling,controls,diff}/; tests live beside their owners. The full scanner and local Studio remain separate entry points.
  • Updated package imports, CI paths, test discovery, Studio imports, and recursive scanner/implementation identity hashing. The relocated policy registry remains recognized at its historical path when inspecting earlier commits.
  • On the same source commit, the full scan retains 2,308 findings, 2,256 control records, 568 EMCN exports, identical coverage and unchecked diagnostics, and zero inspection failures. Eleven artwork findings only point to the policy JSON's new path. The local diff check reports two advisory central-definition findings for the policy-file move and zero inspection failures. All 678 script tests, 51 audits, design and Studio type checks, and lint pass.

@vercel

vercel Bot commented Sep 26, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
docs Ready Ready Preview Sep 29, 2026 1:47am UTC

Request Review

@BillLeoutsakosvl346

Copy link
Copy Markdown
Contributor Author

@greptile

@BillLeoutsakosvl346

Copy link
Copy Markdown
Contributor Author

@cubic-dev-ai review this PR

@cubic-dev-ai

cubic-dev-ai Bot commented Sep 26, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review this PR

@BillLeoutsakosvl346 I have started the AI code review. It will take a few minutes to complete.

@greptile-apps

greptile-apps Bot commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 5/5

[High risk] Adds design conformance checking and modifies build configuration.

The PR appears safe to merge based on the changes reviewed.

Summary

Adds advisory design-conformance checks and a local Studio. Since the previous review, the PR expands bounded CVA analysis, treats central extraction failures as incomplete comparisons, and resolves namespace styling references. No new actionable issue was established.

Reviews (18) · Last reviewed commit: "docs(design): document finite CVA analys..."

Comment thread scripts/design-scan/inventory.ts Outdated
Comment thread scripts/design-scan/report.ts Outdated

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 113 files

Tip: instead of fixing issues one by one fix them all with cubic

Re-trigger cubic

Comment thread scripts/design-conformance/styling/appearance.ts
Comment thread tools/design-studio/app/layout.tsx Outdated
Comment thread scripts/design-conformance/evaluate.ts Outdated
Comment thread scripts/design-conformance/control-associations.ts Outdated
Comment thread .agents/skills/emcn-design-review/SKILL.md Outdated
Comment thread scripts/design-conformance/control-simplifications.ts Outdated
Comment thread scripts/design-conformance/control-simplifications.ts Outdated
Comment thread scripts/design-conformance/catalogue.ts Outdated
Comment thread package.json Outdated
Comment thread tools/design-studio/_components/generated-gallery.tsx Outdated
@BillLeoutsakosvl346

Copy link
Copy Markdown
Contributor Author

@greptile

@BillLeoutsakosvl346

Copy link
Copy Markdown
Contributor Author

@cubic-dev-ai review this PR

@cubic-dev-ai

cubic-dev-ai Bot commented Sep 26, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review this PR

@BillLeoutsakosvl346 I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 114 files

Tip: instead of fixing issues one by one fix them all with cubic

Re-trigger cubic

Comment thread scripts/design-conformance/model.ts Outdated
Comment thread scripts/design-conformance/styling/artwork.ts
Comment thread scripts/design-conformance/control-associations.ts Outdated
Comment thread scripts/design-conformance/control-review.ts Outdated
Comment thread scripts/design-conformance/cli.ts Outdated
Comment thread scripts/design-conformance/central-recipes.ts Outdated
Comment thread scripts/design-conformance/controls/colour-assignments.ts
Comment thread scripts/design-conformance/shared/normalize.ts
Comment thread scripts/design-conformance/control-typography.ts Outdated
Comment thread scripts/design-conformance/lint.ts Outdated
@BillLeoutsakosvl346

Copy link
Copy Markdown
Contributor Author

@greptile

@BillLeoutsakosvl346

Copy link
Copy Markdown
Contributor Author

@cubic-dev-ai review this PR

@cubic-dev-ai

cubic-dev-ai Bot commented Sep 26, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review this PR

@BillLeoutsakosvl346 I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 114 files

Tip: instead of fixing issues one by one fix them all with cubic

Re-trigger cubic

Comment thread scripts/design-conformance/control-review.ts Outdated
Comment thread scripts/design-conformance/controls/colour-assignments.ts
Comment thread scripts/design-conformance/conformance.ts Outdated
Comment thread scripts/design-conformance/control-simplifications.ts Outdated
Comment thread scripts/design-conformance/control-simplifications.ts Outdated
Comment thread scripts/design-conformance/control-typography.ts Outdated
Comment thread scripts/design-conformance/control-imperative.ts Outdated
Comment thread scripts/design-conformance/control-imperative.ts Outdated
Comment thread scripts/design-conformance/control-imperative.ts Outdated
Comment thread scripts/design-conformance/shared/model.ts
@BillLeoutsakosvl346

Copy link
Copy Markdown
Contributor Author

@greptile

@BillLeoutsakosvl346

Copy link
Copy Markdown
Contributor Author

@cubic-dev-ai review this PR

@cubic-dev-ai

cubic-dev-ai Bot commented Sep 26, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review this PR

@BillLeoutsakosvl346 I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 119 files

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.

Re-trigger cubic

@BillLeoutsakosvl346

Copy link
Copy Markdown
Contributor Author

Superseded by draft #8396, which carries the complete design tooling work and includes the latest staging merge. Continuing CI and reviewer follow-up there.

@waleedlatif1
waleedlatif1 deleted the codex/design-system-rollout branch September 29, 2026 04:46

This branch was successfully deployed

1 active deployment
Preview — 66e1cd3a Deployed Sep 29, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant