Skip to content

Version-bump-only regressions surface only on the Version Packages PR — rehearse pnpm run version on main on a schedule and test the bumped tree #20562

Description

@hotlong

Problem

CI on main only tests the tree at its current version. Some code depends on the version number itself, for example:

  • tests that pin the current minor;
  • protocol-major checks;
  • scaffold templates;
  • docs image tags.

That code is first exercised after pnpm run version, and that runs only on changeset-release/main. So these failures show up at release time, on a PR that nobody can commit to: the version-pr lane rebuilds it with git reset --hard. The release then waits a full cycle:

  1. file the defect;
  2. fix it on main;
  3. merge the fix;
  4. wait for the next scheduled or dispatched refresh of the version PR;
  5. re-run CI.

Precedents of the same class

(Advisory-discovery reds such as #20561 are a different class. The daily OSV scan already covers them, and they are out of scope here.)

Proposal

Add a scheduled job (daily, or on each push to main that adds a changeset) that:

  1. clones main and prepares it with scripts/pm/release-rehearsal-clone.mjs --prepare;
  2. runs pnpm run version there, with no push and no publish;
  3. builds, then runs the test suites against the bumped tree. The whole Test Core matrix is the ideal. If that is too expensive, run at least the version-sensitive packages: create-objectstack, the CLI scaffolds, Scaffold E2E, and the protocol-version sync checks;
  4. on red, opens or updates one finding card. It never touches changeset-release/main.

Acceptance

Open question for maintainers

The cost trade-off: run the full Test Core matrix on the bumped tree, or only a curated subset. A curated subset needs its own gate so that the list does not rot.


Generated by Claude Code

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions