Repository navigation
fix(storage): use the standard s3 client for cloudflare r2 - #96
Merged
Merged
Conversation
The CRT client signs Content-Length: 0 on body-less GET, LIST and DELETE, which Cloudflare R2 rejects with 403 SignatureDoesNotMatch. Claude-Session: https://claude.ai/code/session_01CAJNYTHfbTcqGiWQnaecY7
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Root cause
nio-spi-s3 2.5.0 builds the CRT-based
S3AsyncClient(aws-crt 0.46.1). For every body-less GET, LIST and DELETE that client sends and signsContent-Length: 0, socontent-lengthends up inSignedHeaders. Cloudflare R2 builds its canonical request with an emptycontent-lengthvalue, so the signatures differ and every GET/LIST/DELETE fails with403 SignatureDoesNotMatch. Only PUT works.Canonical request, redacted (header block only):
nio-spi's write path first GETs the existing object (
S3WritableByteChannel->S3TransferUtil.downloadToLocalFile), so every BlueMap write failed on R2. The behaviour was introduced upstream in awslabs/aws-c-s3 (related: aws-c-s3 issue #514). Path-style, region (auto/us-east-1), crossRegionAccess, checksum mode and the TransferManager open option do not change the outcome.Fix
ResolvedEndpointgets a pureuseStandardClientflag (true for ther2profile, false for generic/AWS).S3FileSystemFactory.buildapplies it: it setss3.spi.client.custom-headers.enabled=truefor R2 and clears the property otherwise, beforePROVIDER.getFileSystem(uri).S3ClientProviderreads the property in its constructor duringgetFileSystem, so setting it afterwards has no effect. Clearing it for other profiles stops an earlier R2 storage from switching a later generic/AWS storage to the standard client (system properties are global).Generic and AWS behaviour is unchanged (CRT client).
Tests:
ProviderProfilesTestcovers the decision (R2 via account-id, R2 via endpoint, generic). No network, filesystem or system-property access.Docs (separate commit): provider-profiles explanation,
SignatureDoesNotMatchon GetObject with R2 in the errors reference, and the checksum explanation no longer claims the CRT client is used for R2.Side effects (R2 only)
Verification against real Cloudflare R2
Shaded jar of this branch on Paper 26.3 + BlueMap 5.29, maps with
storage: "r2".Startup line (account id redacted by BlueMap itself):
bluemap update worldrendered all three maps;bluemap tasksreported "no pending tasks, all done".rclone size r2:bluemap-s3-test: 121 objects, 1.883 MiB (1974091 bytes), of which 72 are tile objects under*/tiles/. Layout:world/,world_the_nether/,world_the_end/each withtiles/,live/,rstate/,settings.json,textures.json.gz.bluemap tasksclean.https://claude.ai/code/session_01CAJNYTHfbTcqGiWQnaecY7