Skip to content

fix: add trailing slash to NIGHTLY_RELEASES_DIR path - #1035

Merged
bmuenzenmeyer merged 1 commit into
mainfrom
nightly-kv
Sep 25, 2026
Merged

bmuenzenmeyer merged 1 commit into
mainfrom
nightly-kv

Conversation

@bmuenzenmeyer

@bmuenzenmeyer bmuenzenmeyer commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Possible fix for #1034. I have something else to look at here.... see LONG comment below


A frontier coding agent assisted in this analysis.

@bmuenzenmeyer
bmuenzenmeyer requested a review from a team as a code owner September 25, 2026 12:06
@MattIPv4

Copy link
Copy Markdown
Member

Happy for this to land immediately as an errata fix 👍

@bmuenzenmeyer

bmuenzenmeyer commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor Author

export from a frontier model session. i am including for review for those that know the code better than me


Investigation: v27 nightlies missing from nodejs.org/download/nightly/

Re: [nodejs/release-cloudflare-worker#1034](#1034)

TL;DR

Since #1002 (Aug 17), the worker serves directory listings for nodejs/nightly/ (and several other prefixes) from KV only, and a KV miss is a hard 404. The only thing that refreshes those KV entries is a manual full rebuild. #1019 tried to add a nightly refresh, but it uses 'nodejs/nightly' with no trailing slash. As a result, the scheduled job writes to a KV key the worker never reads.

The harden-runner egress warnings appear unrelated.

Symptoms (from #1034)

  • The https://nodejs.org/download/nightly/ listing stops at v27.0.0-nightly2026090729667e046b (Sep 7).
  • index.json is current: it lists v27.0.0-nightly20260925a3bb551ea7.
  • https://nodejs.org/download/nightly/v27.0.0-nightly20260925a3bb551ea7/ returns 404 through the worker.
  • The same path on direct.nodejs.org returns 200.

Root cause

1. KV is authoritative for these prefixes, with no fallback

[#1002](2876418) enabled USE_KV in prod with this KV_DIRECTORIES list:

metrics/, nodejs/chakracore-nightly/, nodejs/chakracore-rc/, nodejs/chakracore-release/,
nodejs/docs/, nodejs/nightly/, nodejs/rc/, nodejs/test/, nodejs/v8-canary/

In src/providers/r2Provider.ts, any path matching one of these prefixes returns the KV result directly:

if (path.startsWith(prefix)) {
  return await kvProvider.readDirectory(path);   // KV miss -> undefined -> 404
}

There is no fallback to R2/S3 listing. Any directory that isn't in KV returns 404.

Per-directory KV keys are written only by scripts/build-directory-cache.mjs, the manual Build Directory Cache workflow, which has run twice. Run #2 ([34127761210](https://github.com/nodejs/release-cloudflare-worker/actions/runs/34127761210)) sits between Aug 26 and Sep 8, which is consistent with the Sep 7 snapshot. (Its exact timestamp is not verified; it needs a signed-in view.)

2. The nightly refresh from #1019 writes the wrong key

#1019 added nightly to ALWAYS_UPDATED_PATHS in scripts/constants.mjs:

export const RELEASE_DIR = 'nodejs/release/';
export const DOCS_DIR = 'nodejs/docs/';
export const NIGHTLY_RELEASES_DIR = 'nodejs/nightly';   // <- missing trailing slash

update-directory-cache.mjs lists each path with ListObjectsV2 using Delimiter: '/':

  • Prefix: 'nodejs/nightly' returns a single CommonPrefix, nodejs/nightly/, and no files.
  • The result is stored under KV key nodejs/nightly, with one subdirectory named /.
  • The worker reads KV key nodejs/nightly/, which is never updated.

The scheduled update-links.yml runs succeed every night and merge a "chore: update redirect links" commit, so nothing appears broken.

3. New version subdirectories are never listed

Even with the slash fixed, the scheduled run (no version argument) lists ALWAYS_UPDATED_PATHS non-recursively. The recursive listing only runs for nodejs/release/${VERSION}/ when a version is passed, and nightlies never pass one. New nightly directories would then appear in the parent listing but still 404 when opened. That is the second curl in #1034.

The same staleness should apply to every other KV_DIRECTORIES prefix that nothing refreshes, notably nodejs/rc/ and nodejs/v8-canary/.

Why this isn't more widely noticed

Only directory listings go through KV. File downloads use R2_BUCKET.get(path) directly, so tarballs at exact URLs still work. index.json and index.tab are files, so they are also current.

Tooling such as nvm, fnm, n, setup-node and most CI scripts reads the index files and then fetches exact URLs. The only people affected are humans browsing HTML listings of nightly, rc or v8-canary builds. nodejs/release/, where most browsing happens, is excluded from KV_DIRECTORIES.

Egress warnings (harden-runner)

These are most likely unrelated:

It's still worth checking which endpoints are being flagged, but as a separate item. The logs require sign-in and were not reviewed here.

How to confirm

  1. Browse https://nodejs.org/download/v8-canary/ and https://nodejs.org/download/rc/. If those listings are also frozen around Sep 7, and a newer version directory 404s through the worker but returns 200 on direct.nodejs.org, the mechanism is confirmed.
  2. Inspect the KV namespace a8378d5444a2465ca7202ebaa5d8deb1. Expect a stray nodejs/nightly key updated nightly, and a nodejs/nightly/ key last written around Sep 7.

If those checks don't match, the mechanism above is wrong, and the KV contents should be examined directly.

Proposed fixes

  1. Fix the constant: NIGHTLY_RELEASES_DIR = 'nodejs/nightly/'. Add a guard that throws if any ALWAYS_UPDATED_PATHS entry lacks a trailing /. Delete the stray nodejs/nightly KV key.
  2. Populate new subdirectories: after listing a parent directory, recursively list any subdirectory with no existing KV key. Alternatives:
    • Recursively list the newest N entries (nightly names sort by date).
    • Have the nodejs/build promote step dispatch the update with the nightly or rc path.
  3. Cover the other prefixes: add nodejs/rc/ and nodejs/v8-canary/ (and any other active KV_DIRECTORIES prefixes) to the refresh.
  4. Consider a KV-miss fallback: on a KV miss, fall back to live R2 listing instead of returning 404. This would turn future sync gaps into slower listings rather than outages.
  5. Stopgap until the above lands: re-run Build Directory Cache. Alternatively, temporarily remove nodejs/nightly/, nodejs/rc/ and nodejs/v8-canary/ from KV_DIRECTORIES in wrangler.jsonc so those prefixes use live R2 listing.

@bmuenzenmeyer
bmuenzenmeyer merged commit ff13202 into main Sep 25, 2026
8 checks passed
@bmuenzenmeyer
bmuenzenmeyer deleted the nightly-kv branch September 25, 2026 12:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants