Delete unused CodeQL bundles from the toolcache before downloading - #4124
Delete unused CodeQL bundles from the toolcache before downloading#4124henrymercer wants to merge 10 commits into
Conversation
`isHostedRunner` infers hostedness from the runner name and the toolcache path, so it also matches self-hosted runners that are configured to resemble hosted ones. Rename it to `looksLikeHostedRunner` so callers can see they are getting a heuristic, and add `isGitHubHostedRunner`, which reads the `RUNNER_ENVIRONMENT` value the Actions service reports. The existing callers keep the heuristic, so there is no behaviour change. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
`deleteToolcacheBundles` removes `$RUNNER_TOOL_CACHE/CodeQL` and reports which versions were there. It refuses to follow a symlinked CodeQL directory so that it can only ever delete paths that are really inside the toolcache, and reports failures rather than throwing. Not called yet. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
When we download a bundle the toolcache often already holds a different one that the job will not use, and on GitHub-hosted runners it shares a filesystem with the workspace, so it takes space away from the analysis. Empty the toolcache before downloading, which also frees space for the archive during extraction, and which is safe because getting as far as a download means the tools were not resolved from the toolcache. Skip this once a step has obtained the tools, since a later step may run a path it was given, and gate it on the runner being GitHub-hosted and on a feature flag that is off by default. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The check installs the tools from a URL, so most versions in its matrix are downloaded rather than found in the toolcache, which is when the cleanup runs, and it then builds and analyses seven languages, so a bundle we damaged on the way in would show up. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Warning
- Copilot's review of this pull request may be incomplete because some of the changed files are excluded by your Copilot content exclusion settings. See Excluding content from Copilot for details.
Copilot review overview
🟡 Changes recommended
Two moderate issues affect cleanup coverage and failure telemetry.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review tier: Balanced
Findings: 2
New issues introduced by this change (2)
| Severity | Finding |
|---|---|
 |
src/setup-codeql.ts — This condition excludes supported explicit tools URLs that do not contain a /codeql-bundle-.../… |
|
src/tools-download.ts — The catch also handles permission errors, I/O failures, and other lstat errors, but reports all… |
What changed in this PR
Adds feature-flagged cleanup of unused CodeQL toolcache bundles before downloads on GitHub-hosted runners.
Changes:
- Adds guarded toolcache cleanup with telemetry and tests.
- Tracks whether a job already obtained CodeQL tools.
- Clarifies hosted-runner detection APIs.
| File | Review |
|---|---|
src/util.ts |
Renames and documents the hosted-runner heuristic. |
src/tools-download.ts |
Implements cleanup; non-ENOENT inspection errors must be reported as failures. |
src/setup-codeql.ts |
Integrates cleanup; the current condition incorrectly excludes some explicit download URLs. |
src/setup-codeql.test.ts |
Covers cleanup behavior and safeguards. |
src/feature-flags.ts |
Defines the cleanup feature flag. |
src/environment.ts |
Adds job-scoped tool acquisition state. |
src/config-utils.ts |
Updates renamed heuristic usage. |
src/caching-utils.ts |
Updates renamed heuristic usage. |
src/actions-util.ts |
Adds service-reported hosted-runner detection. |
pr-checks/checks/multi-language-autodetect.yml |
Enables cleanup in integration checks. |
Files excluded by content exclusion policy (2)
- .github/workflows/__multi-language-autodetect.yml
- lib/entry-points.js
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| // If we are not going to add the tools to the toolcache, we are extracting them somewhere else | ||
| // and emptying the toolcache would not buy us the space we need. | ||
| if ( | ||
| destinationVersion === undefined || |
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Run the cleanup even when the download will not be cached in the toolcache, since the toolcache shares a filesystem with the directory we extract to, so freeing it helps either way, and report an error other than the toolcache being absent as a failure rather than as an empty toolcache. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Some runner images keep the toolcache on a different volume to the workspace, so deleting the tools there frees up disk space that the analysis cannot use, and costs a later step that wanted them in the toolcache a download. Windows runners are laid out this way, with the toolcache on `C:` and the workspace on `D:`. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
github-actions
Bot
added
size/XL
mbg
left a comment
mbg
left a comment
There was a problem hiding this comment.
This mostly looks good and makes a lot of sense, thank you! I don't think I noticed anything overly critical, but found a few minor-ish points, some of which we should probably address.
| .map((entry) => entry.name) | ||
| .sort(); | ||
|
|
||
| await fs.promises.rm(toolDirectory, { force: true, recursive: true }); |
There was a problem hiding this comment.
To be a bit extra safe, do you think it would make sense to perform the symlink check for each of the specific version directories as well before deleting them individually to make sure that the specific version directories aren't symlinks?
There was a problem hiding this comment.
This would be odd on a runner that claims to be GitHub-hosted, but can do. In most cases this shouldn't slow things down much as we expect exactly one CodeQL entry in the toolcache.
Delete each version directory individually so that a symlinked one is skipped rather than removed, take an `ActionState` so the environment is read through `ReadOnlyEnv` rather than the deprecated `getOptionalEnvVar`, let `deleteToolcacheBundles` report its own failure to locate the toolcache instead of having the caller catch it, quote paths in log messages, and rename `HAS_OBTAINED_CODEQL_TOOLS` to `HAS_SET_UP_CODEQL`, which is also set when we find the tools in the toolcache rather than downloading them. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
@copilot resolve the merge conflicts in this pull request |
…-bundle-cleanup Co-authored-by: henrymercer <14129055+henrymercer@users.noreply.github.com>
Merged main in — no actual conflicts, git auto-merged cleanly (313a0b9). Verified the build regenerates lib/ with no diff, and reran the relevant unit tests (tools-download, setup-codeql, init-action-post-helper, caching-utils) — all 80 pass. |
mbg
left a comment
mbg
left a comment
There was a problem hiding this comment.
Thanks for addressing the feedback! A few minor comments here for you to look at. I think maybe two of the log messages got mixed up in the change; otherwise just some style/minor safety points.
| toolDirectory = getToolcacheToolDirectory(); | ||
| } catch (e) { | ||
| logger.info( | ||
| `Unable to reclaim disk space from the toolcache: ${getErrorMessage(e)}`, |
There was a problem hiding this comment.
This could probably be something more specific like Unable to determine toolcache directory: ${getErrorMessage(e)}
| const versionDirectory = path.join(toolDirectory, entry.name); | ||
|
|
||
| // `isDirectory` is false for a symlink, so we never delete a version directory that is | ||
| // really somewhere else. | ||
| if (!entry.isDirectory()) { | ||
| logger.debug( | ||
| `Not deleting '${versionDirectory}' from the toolcache since it is not a directory.`, | ||
| ); | ||
| continue; | ||
| } |
There was a problem hiding this comment.
Minor: It's probably unlikely that path.join(toolDirectory, entry.name) would cause an issue before the entry.isDirectory() check (e.g. non-common entry types on Windows resulting in illegal paths), but I note that the order is reversed compared to the previous version and this kind of thing can sometimes lead to unexpected errors on Windows. It might be marginally safer to restore the previous order, at the expense of not being able to use versionDirectory in the log message.
| } catch (e) { | ||
| logger.info( | ||
| `Failed to delete the CodeQL tools at ${toolDirectory} from the toolcache: ${getErrorMessage(e)}`, | ||
| `Failed to read the CodeQL tools at '${toolDirectory}' from the toolcache: ${getErrorMessage(e)}`, |
There was a problem hiding this comment.
Undo? The previous message still seems correct.
| `Unable to reclaim disk space from the toolcache: ${util.getErrorMessage(e)}`, | ||
| ); | ||
| } | ||
| const result = await deleteToolcacheBundles(logger); |
There was a problem hiding this comment.
Now that we have removed the try/catch here because deleteToolcacheBundles handles all exceptions, it might be worth documenting this e.g. in the definition of deleteToolcacheBundles to make sure we don't accidentally change it down the line in a way that could result in an upstream failure here.
| await withTmpDir(async (tmpDir) => { | ||
| setupActionsVars(tmpDir, tmpDir); | ||
| process.env[ActionsEnvVars.RUNNER_ENVIRONMENT] = "github-hosted"; | ||
| delete process.env[ActionsEnvVars.RUNNER_TOOL_CACHE]; |
There was a problem hiding this comment.
If we update getToolcacheToolDirectory to use an Env parameter, then this test wouldn't need to be serial.
When a job downloads a CodeQL bundle, the toolcache is often already holding a different one that the job will not use, typically the bundle baked into the runner image. On GitHub-hosted runners the toolcache sits on the same filesystem we measure for the disk space check, so that unused bundle takes around 2 GB away from the analysis.
This adds a feature flag,
cleanup_toolcache_bundles, that empties$RUNNER_TOOL_CACHE/CodeQLimmediately before we download a bundle. It is off by default.Emptying it before the download rather than after means the space also covers the archive during extraction. We only get as far as downloading when the tools were not resolved from the toolcache, and a lookup there requires both the version directory and its completion marker, so nothing left behind is usable by the job.
That stops holding once a step has obtained the tools, because a later step may run a path it was given, as our own
codeql.ymldoes. The job records that it has the tools and skips the cleanup from then on.The cleanup only runs when
RUNNER_ENVIRONMENTreportsgithub-hosted, so we never delete from a toolcache that outlives the job, and only when the toolcache is on the same filesystem as the workspace, since otherwise the space we free is not space the analysis can use. Windows runner images are laid out that way, keeping the toolcache onC:while the workspace is onD:. Deletion is confined to the CodeQL directory, refuses to follow a symlink, and logs failures rather than throwing.This also renames
isHostedRunnertolooksLikeHostedRunner, with no behaviour change. It infers hostedness from the runner name and filesystem layout, so it can disagree with the newisGitHubHostedRunnercheck. Its callers are deciding whether caching is worthwhile, where the inferred answer is arguably the one they want, so switching them would be a behaviour change and is left for a separate PR.Risk assessment
Which use cases does this change impact?
Workflow types:
dynamicworkflows (Default Setup, Code Quality, ...).Products:
analysis-kinds: code-scanning.analysis-kinds: code-quality.upload-sarifaction.The cleanup sits on the shared bundle download path, so it can run for anything that obtains the CodeQL tools.
Environments:
github.comand/or GitHub Enterprise Cloud with Data Residency.How did/will you validate this change?
pr-checks)..test.tsfiles).Unit tests cover the flag being off, a non-hosted runner, an unreported runner environment, a toolcache on a different filesystem to the workspace, a hosted runner with bundles present, a download that will not be cached in the toolcache, a step having already obtained the tools, deletion failing, a toolcache we cannot inspect, and a symlinked CodeQL directory. The multi-language PR check runs with the flag enabled, so the cleanup happens ahead of a real build and analysis of seven languages.
This has not been run on a real hosted runner outside the PR checks, and the disk saving is inferred from bundle size rather than measured, so the rollout should confirm it.
If something goes wrong after this change is released, what are the mitigation and rollback strategies?
The rename is not behind the flag, but it has no behaviour change.
How will you know if something goes wrong after this change is released?
A
codeql-action/toolcache-bundle-cleanupdiagnostic reports which versions were deleted and whether the attempt failed. Note that diagnostics are only flushed by theinitaction, so a cleanup under the standalonesetup-codeqlaction reports nothing. That is pre-existing behaviour for diagnostics raised fromsetup-codeql.ts, but it does leave that entry point unmeasurable.Are there any special considerations for merging or releasing this change?
Merge / deployment checklist