fix(nix): remove the nix store cache causing rebuild inconsistencies

[Venkumahanti Subhankar (V-Subhankar-infy)]

Pre-requisite: First merge the removal of debian 11 for the smoke tests to pass , mentioned in the PR #1736

Closes #1727. Closes #1505.

Problem

The Nix Feature caches /nix in a persistent Docker volume, and that cache makes rebuilds ignore option changes. Nix and any packages are installed into the image at build time, then the volume is mounted over them at container start:

{ "source": "nix-store-${devcontainerId}", "target": "/nix", "type": "volume" }

Docker fills a named volume from the image only while it is empty, once, at first create; after that it hides what the newer image built. ${devcontainerId} comes from the workspace path, not the config, so every rebuild meets the same frozen volume and options such as packages stop taking effect. Removing the cache is the fix.

Solution and changes

Remove the cache so the image is what runs.

• src/nix/devcontainer-feature.json: drop the mounts block, so nothing is layered over /nix; bump 1.3.1 → 1.4.0.
• src/nix/NOTES.md: tell affected users which versions were broken
• src/nix/README.md: regenerated with devcontainer features generate-docs
• test/nix/extra-config-packages.sh: new test asserting /nix is not a mountpoint, so the cache cannot return unnoticed.
• test/nix/scenarios.json: register that scenario with the extraNixConfig plus packages combination from extraNixConfig breaks package installation #1727.

Minor, not major: the volume was never documented, and users pin ghcr.io/devcontainers/features/nix:1, so a 2.0.0 would not reach anyone affected.

Impact

Measured with nix: {}, each pass from a full prune.

Benefits

• Every option applies on every rebuild; the environment depends on devcontainer.json alone, and a rebuild resets to zero again.
• First up is 26–33s faster: 135–142s against 168s, since Docker no longer copies all of /nix into a fresh volume.
• 2.9 GB less disk per devcontainer: /nix is stored once in the image instead of twice.
• No orphaned state: no nix-store-* volume is left behind, which matters because docker system prune -a --volumes -f does not remove it anyway.
• Nothing is given up at build time: no volume was ever attached during docker build, and Docker's layer cache still skips the install when options are unchanged.
• Anyone who wants the cache can still declare the same volume in their own devcontainer.json, making it a deliberate choice rather than a silent default.

Losses

• Packages installed by hand at runtime no longer survive container recreation: 534.5 MiB re-fetched in the test above. This is a real regression for dotfiles and home-manager workflows, which install everything at runtime and gain the most from the cache.
• Recreating a container is slower without a populated volume: 46.8s against 4.3s.

Why it went unnoticed

The first run always works; the volume is created there, so image and volume agree. When it breaks, the usual response is a clean rebuild, which drops the volume and hides the cause. CI never sees it either: devcontainer features test creates a fresh volume per scenario, so build A → create → build B against one volume is never run.

How is the current cache implementation incomplete ?

Six lines of static metadata, no code behind it. Once seeded the cache is one-way:

• The build cannot write to it. docker build has no volume attached, and cannot have one.
• Nix cannot reconcile it. The store is content-addressed and read-only; a rebuild creates new paths beside the old, and Nix has no merge.
• The volume owns the pointer. Which generation is current is recorded inside /nix, so the image's is never elected.

Nothing errors: the mount succeeds and puts older data on top, which is why the failure is silent. The only escapes are docker volume rm, or reinstalling by hand at runtime after every rebuild.

Why a complete cache implementation is expensive

Nix is large, so wanting a cache is reasonable; making one correct is not:

• It must run at container start, adding a runtime network dependency: a container that built fine may fail to converge offline or behind a proxy.
• It must detect drift, needing a build-time marker of the resolved request compared against one inside the volume on every start.
• Everything after that is still open: removal, version, garbage-collection, failure-policy, testing.

The breaking point in past that caused this issue

#1127 one partially wrong assumption done that:

Garbage collection is no issue since nix simply refetches everything that's missing.

Self-healing fixes absence, not staleness. It only fires when something installs against the volume, at runtime, never during a build. And nothing is missing: the volume holds a complete, self-consistent Nix that is merely older, so there are no holes to detect and nothing to repair. This was despite the concerns already raised in #284, where both the "reset to zero with a rebuild" invariant and the distinction between cache and runtime directories were stated.

[Copilot]

🟢 Approval recommended

The implementation and regression coverage are consistent, with CI sequencing on #1736 already documented.

Review details

• Files reviewed: 5/5 changed files
• Comments generated: 0 new
• Review effort level: Balanced