Skip to content

fix(storage): use the standard s3 client for cloudflare r2 - #96

Merged
TheMeinerLP merged 2 commits into
mainfrom
fix/r2-standard-client
Oct 14, 2026
Merged

TheMeinerLP merged 2 commits into
mainfrom
fix/r2-standard-client

Conversation

@TheMeinerLP

Copy link
Copy Markdown
Owner

Root cause

nio-spi-s3 2.5.0 builds the CRT-based S3AsyncClient (aws-crt 0.46.1). For every body-less GET, LIST and DELETE that client sends and signs Content-Length: 0, so content-length ends up in SignedHeaders. Cloudflare R2 builds its canonical request with an empty content-length value, so the signatures differ and every GET/LIST/DELETE fails with 403 SignatureDoesNotMatch. Only PUT works.

Canonical request, redacted (header block only):

client (CRT):   content-length:0
                host:<account>.r2.cloudflarestorage.com
                ...
                signedheaders: content-length;host;x-amz-content-sha256;x-amz-date;...

R2 (expected):  content-length:
                host:<account>.r2.cloudflarestorage.com
                ...

nio-spi's write path first GETs the existing object (S3WritableByteChannel -> S3TransferUtil.downloadToLocalFile), so every BlueMap write failed on R2. The behaviour was introduced upstream in awslabs/aws-c-s3 (related: aws-c-s3 issue #514). Path-style, region (auto/us-east-1), crossRegionAccess, checksum mode and the TransferManager open option do not change the outcome.

Fix

ResolvedEndpoint gets a pure useStandardClient flag (true for the r2 profile, false for generic/AWS). S3FileSystemFactory.build applies it: it sets s3.spi.client.custom-headers.enabled=true for R2 and clears the property otherwise, before PROVIDER.getFileSystem(uri). S3ClientProvider reads the property in its constructor during getFileSystem, so setting it afterwards has no effect. Clearing it for other profiles stops an earlier R2 storage from switching a later generic/AWS storage to the standard client (system properties are global).

Generic and AWS behaviour is unchanged (CRT client).

Tests: ProviderProfilesTest covers the decision (R2 via account-id, R2 via endpoint, generic). No network, filesystem or system-property access.

Docs (separate commit): provider-profiles explanation, SignatureDoesNotMatch on GetObject with R2 in the errors reference, and the checksum explanation no longer claims the CRT client is used for R2.

Side effects (R2 only)

  • Netty-based standard client instead of CRT.
  • No CRT automatic multipart transfers.
  • Different/extra user-agent headers.

Verification against real Cloudflare R2

Shaded jar of this branch on Paper 26.3 + BlueMap 5.29, maps with storage: "r2".

Startup line (account id redacted by BlueMap itself):

[BlueMap] S3 storage: profile=r2, endpoint=https://***.r2.cloudflarestorage.com, bucket=bluemap-s3-test, region=auto, path-style=true, checksum-validation=when_required
  • bluemap update world rendered all three maps; bluemap tasks reported "no pending tasks, all done".
  • No 403 / SignatureDoesNotMatch / S3 errors in the log.
  • rclone size r2:bluemap-s3-test: 121 objects, 1.883 MiB (1974091 bytes), of which 72 are tile objects under */tiles/. Layout: world/, world_the_nether/, world_the_end/ each with tiles/, live/, rstate/, settings.json, textures.json.gz.
  • Restarted the server: all three maps loaded from the existing data without errors (read-existing-data path), bluemap tasks clean.

https://claude.ai/code/session_01CAJNYTHfbTcqGiWQnaecY7

The CRT client signs Content-Length: 0 on body-less GET, LIST and DELETE,
which Cloudflare R2 rejects with 403 SignatureDoesNotMatch.

Claude-Session: https://claude.ai/code/session_01CAJNYTHfbTcqGiWQnaecY7
@TheMeinerLP
TheMeinerLP merged commit 1158e00 into main Oct 14, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant