improvement(ci): measure where the layer-cache disk actually goes

[waleedlatif1]

The problem

The layer-cache cap is enforced against buildctl du, and on the busiest disk that number no longer describes the disk.

From a single real job (app image, amd64):

before prune -> Total: 65.87GB
after prune  -> Total: 65.87GB  (keep-storage 102400 MB)

The prune is behaving correctly — 65.87 GB is already under the 100 GB cap, so there is nothing to trim. Meanwhile that sticky disk reports 224.9 GB.

The control is the same Dockerfile on the other architecture:

disk	buildctl du	actual disk	gap
app.Dockerfile/linux-amd64	65.87 GB	~241 GB	3.7×
app.Dockerfile/linux-arm64	88.04 GB	~93 GB	~1×

Same image, same 100 GB cap, both disks created 2026-08-29. The only difference is build frequency — amd64 builds on every push to main and staging, arm64 only on main. So the divergence tracks write churn, not image content.

Why this is a diagnostic and not a fix

Two causes fit the symptom, and they need opposite fixes:

1. Thin-provisioned volume not releasing freed blocks. Sticky disks are Ceph block volumes cloned and committed per job, so a filesystem that deletes files without discard would grow monotonically. Fix would be fstrim.
2. Real files du's total does not cover. The RUN --mount=type=cache dirs (apt ×2, bun, npm, next, turbo) are a different record class from the layer cache --keep-storage trims. Fix would be bounding or dropping specific mounts.

One log line argues against (1) — Filesystem free space after mount: 527497363456 bytes (491.27 GiB) implies the filesystem really is holding those bytes — but that is a single indirect reading, not proof, and shipping fstrim or deleting a cache mount on it would be guessing.

So this adds the three readings that separate the cases:

• df — the filesystem's own view, versus buildkit's.
• buildctl du --verbose summarised by record type — whether cache mounts are accounted for at all.
• a bounded du -x -d1 walk — where the bytes actually live, if they are not.

One or two builds of output settles it, and the follow-up fix is then evidenced rather than assumed.

Safety

Composite steps run under bash -e -o pipefail, where a failing command inside an assignment or pipeline aborts the step and fails the build — the bug that broke a real app image build in #7252. Every command here is guarded, and the directory walk is bounded by timeout 120.

Verified rather than assumed: the block was run under bash --noprofile --norc -e -o pipefail with every command failing (no buildkitd, no /var/lib/buildkit locally). It reaches the end and exits 0. Shellcheck clean.

This step already could not fail a deploy by design — that property is preserved.

Type of Change

• Improvement

Testing

Executed the block under the exact composite-step shell flags with all commands failing (exit 0, reached end marker). Shellcheck clean. No behaviour change to the prune itself — this is additive output after the existing before/after readings.

Checklist

• Code follows project style guidelines
• Self-reviewed my changes
• Tests added/updated and passing
• No new warnings introduced
• I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
docs	Ready	Preview	Sep 5, 2026 5:24pm UTC

[greptile-apps]

Greptile Summary

This PR adds bounded, best-effort BuildKit disk-accounting diagnostics after layer-cache pruning.

• Reports the filesystem’s view of /var/lib/buildkit.
• Attempts to aggregate verbose BuildKit usage by record type.
• Lists the largest top-level BuildKit directories with a two-minute timeout.
• Preserves deployment continuity by guarding every diagnostic command.

Confidence Score: 4/5

The PR appears safe to merge operationally, but its diagnostic output and explanatory comment should be corrected so the investigation produces reliable conclusions.

The commands remain bounded and cannot fail the deployment step, but the record-type aggregation reports near-zero totals for human-readable BuildKit sizes, and the surrounding explanation overstates what existing evidence proves.

Files Needing Attention: .github/actions/docker-build/action.yml

Important Files Changed

Filename	Overview
.github/actions/docker-build/action.yml	Adds non-failing disk diagnostics, but the record-size aggregation mishandles BuildKit’s unit-suffixed output and an explanatory comment prematurely rules out one investigated cause.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Layer-cache prune completes] --> B[df /var/lib/buildkit]
  B --> C[buildctl du --verbose]
  C --> D[Aggregate sizes by record type]
  D --> E[Bounded du directory walk]
  E --> F[Continue Docker build setup]

Loading

_{Reviews (1): Last reviewed commit: "improvement(ci): measure where the layer..." | Re-trigger Greptile}

[greptile-apps]

Size Units Break Aggregation

buildctl du --verbose emits human-readable Size: values such as 65.87GB. Awk converts $2 to only the leading number before dividing it by 1e9, so non-empty record types appear as approximately 0.00 GB. This makes the new diagnostic unable to show which record types consume the cache disk.

[greptile-apps]

Comment Rules Out Cause

This explanation says thin provisioning has been ruled out, although the free-space reading is only indirect evidence. That could steer the follow-up investigation toward deleting cache mounts instead of also evaluating discard behavior. Please preserve both possible causes until the new diagnostics distinguish them.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!