Skip to content

perf(build): reuse kernel builds across worktrees; add a quick test-only kernel build - #533

Merged
bompus merged 3 commits into
fork/consolidatedfrom
perf/kernel-build-cache
Oct 19, 2026
Merged

bompus merged 3 commits into
fork/consolidatedfrom
perf/kernel-build-cache

Conversation

@bompus

@bompus bompus commented Oct 19, 2026 •

Copy link
Copy Markdown
Owner

A new worktree gets its native kernel in under a second instead of about two minutes when its kernel sources match a recent build, and a kernel edit can be tested after a 4 s rebuild instead of 90 s.

Step (16 vCPUs, other work running) Before After Benefit
npm run build:kernel in a new worktree, kernel sources unchanged ~139 s, 595 MB target/ 0.8 s (restored from the cache) ~170× faster, no target/
ensure-kernel before npm test, prebuild missing ~139 s 0.08 s ~1,700× faster
Rebuild after a one-line resolver edit 89 s (release) 4.2 s (build:kernel:quick) 21× faster
Cold build ~139 s (release) 83 s (quick) 1.7× faster

One run each; these are single measurements, not a benchmark claim. The gaps are far outside run-to-run variation.

What changes

  • scripts/kernel-cache.mjs (new): a per-user cache of release kernel builds, keyed by the kernel source stamp that ensure-kernel already uses.
    • Only builds of committed sources are cached, and only plain release builds. A build with RUSTFLAGS or CARGO_PROFILE_* overrides is not cached.
    • It keeps the four most recently used entries per platform (about 82 MB each).
    • Location: CODEGRAPH_KERNEL_CACHE, else $XDG_CACHE_HOME/codegraph/kernel (%LOCALAPPDATA% on Windows); =off disables it.
  • scripts/build-kernel.mjs:
    • Host release builds restore from the cache before compiling and save after building. --no-cache compiles anyway.
    • --quick builds a new quick Cargo profile (no LTO, 16 codegen units, incremental). Its stamp is marked quick, and it is never cached.
    • CARGO_TARGET_DIR is honored. Before, the output path was hard-coded to target/.
    • The stamp is written only when the kernel sources did not change while cargo ran. Before, a file edited mid-build could leave a prebuild whose stamp did not describe its code.
  • scripts/ensure-kernel.mjs restores from the cache before compiling. It accepts a quick build of the current sources, with a reminder that a quick build is not for measuring.
  • npm run build:kernel:quick.
  • The build-validation skill gets refreshed timings and the --minWorkers=1 --maxWorkers=8 form; --maxWorkers alone errors in vitest 2.
  • Cross-target (--target) and release-workflow builds are unchanged: restore, save and --quick are host-only.

Verification (head 8320a70)

  • __tests__/ensure-kernel-stamp.test.ts, 23 tests. New ones use a fake cargo to cover build-kernel.mjs:

    • save then restore without compiling, and --no-cache;
    • quick builds never cached, with a marked stamp;
    • an edit during the build leaves the prebuild unstamped and uncached;
    • custom flags bypass the cache;
    • CARGO_TARGET_DIR;
    • restore from ensure-kernel without a toolchain;
    • no restore for uncommitted sources or with the cache off;
    • a quick stamp of other sources is rebuilt;
    • pruning keeps the four most recently used entries.

    The mid-build and custom-flag tests fail against the first version of this change and pass now.

  • Real builds: a release build saved to the cache; a second fresh worktree at the same commit restored it in 0.8 s, and the golden-dump tests passed on the restored kernel.

  • Full suite: 754 files passed, 40 tests skipped (vitest run --minWorkers=1 --maxWorkers=8). ensure-kernel restored a stale prebuild from the cache during one run and rebuilt-and-saved during the other.

  • Windows: the cache unit test is not gated by platform, but the fake-cargo tests are POSIX-only (shell-script cargo). Not run on Windows.

README rows checked: none apply (developer build tooling; no user-visible feature, command or quoted number).


Summary by cubic

Caches plain release kernel builds per user so a new worktree with unchanged kernel sources restores the prebuild in under a second instead of recompiling for ~two minutes, and adds an unoptimized quick build (npm run build:kernel:quick) that rebuilds after a one-line edit in ~4 s instead of ~90 s.

Details:

  • The cache is keyed by the kernel source stamp, keeps the four most recent entries per platform, and lives at CODEGRAPH_KERNEL_CACHE (or the platform cache dir; =off disables it).
  • Only plain release host builds of committed sources are cached; quick, cross-target, and RUSTFLAGS/CARGO_PROFILE_* builds bypass it, and build:kernel --no-cache compiles without it.
  • Both ensure-kernel.mjs and host release builds restore from the cache before compiling.
  • Builds with CARGO_TARGET_DIR now write there instead of hard-coded target/.
  • The prebuild stamp is written only when kernel sources did not change during the build, so a file edited mid-build leaves the prebuild unstamped and it gets rebuilt instead of carrying a wrong stamp.
  • ensure-kernel accepts a quick build of the current sources and warns not to measure performance with it.

Written for commit 8320a70. Summary will update on new commits.

View guided diff Turn on auto-fix

@bompus

bompus commented Oct 19, 2026

Copy link
Copy Markdown
Owner Author

@coderabbitai full review

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bompus has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.

@coderabbitai

coderabbitai Bot commented Oct 19, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 22 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration
  • Configuration used: Repository YAML (base), Organization UI (inherited)
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 188e909e-b76b-42ba-922e-e99dab8a1f01
📥 Commits

Reviewing files that changed from the base of the PR and between ccc852b and 8320a70.

📒 Files selected for processing: 8
  • .claude/skills/codegraph-build-validation/SKILL.md
  • .github/workflows/kernel-stamp.yml
  • __tests__/ensure-kernel-stamp.test.ts
  • codegraph-kernel/Cargo.toml
  • package.json
  • scripts/build-kernel.mjs
  • scripts/ensure-kernel.mjs
  • scripts/kernel-cache.mjs
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai

coderabbitai Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
⚠️ Action not completed

Review rate limited.


Your included review limit is currently reached under our Fair Usage Limits Policy. This review may still proceed through usage-based billing if eligible. Your next included review will be available in 22 minutes.

@bompus

bompus commented Oct 19, 2026

Copy link
Copy Markdown
Owner Author

Review bots skipped on 8320a70: CodeRabbit review rate limited (review limit reached), cubic out of quota, Greptile out of trial credits. A local review before opening this PR found one blocking defect (a kernel compiled while its sources were being edited could be cached under the clean commit's key) and several smaller gaps; all are fixed in the second commit, with tests that fail on the first version.

@bompus

bompus commented Oct 19, 2026

Copy link
Copy Markdown
Owner Author

The Windows lifecycle failures here are pre-existing timing flakes, not this change. Two runs on this PR timed out in two different runtime-control.test.ts startup tests (MCP readiness at 10 s). Control PR #534, which is the unchanged base with only a comment added to trigger the workflow, failed on Windows the same way in a third test (mcp-writer-lock.test.ts, a hook timed out at 10 s). Linux and macOS passed in all runs. This PR touches only the kernel build scripts and their tests.

@bompus
bompus merged commit 815c6cc into fork/consolidated Oct 19, 2026
12 of 14 checks passed
@bompus
bompus deleted the perf/kernel-build-cache branch October 19, 2026 07:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant