Skip to content

[filesystem][s3] Mirror s3.session-token to fs.s3a.session.token - #10228

Merged
JingsongLi merged 2 commits into
apache:masterfrom
thswlsqls:fix/s3-mirror-session-token
Oct 3, 2026
Merged

JingsongLi merged 2 commits into
apache:masterfrom
thswlsqls:fix/s3-mirror-session-token

Conversation

@thswlsqls

Copy link
Copy Markdown
Contributor

Purpose

fix #10227

  • S3FileIO mirrors hyphenated access-key/secret-key to the dotted S3A keys, but not the session token.
  • s3.session-token became fs.s3a.session-token, which S3A never reads, so temporary STS credentials passed via Flink/Spark/Hive catalog options were signed without the token (403).
  • Adds {"fs.s3a.session-token", "fs.s3a.session.token"} to MIRRORED_CONFIG_KEYS; precedence matches the existing mirrors. Dotted s3.session.token is unaffected.
  • Aligns with PyPaimon, which accepts s3.session-token ([python] Support S3 options in filesystem catalog #7712).
  • Documents s3.session-token in the S3 section of filesystems.mdx.

Tests

  • Added offline S3FileIOConfigTest: testHyphenSessionTokenIsMirrored (conf key plus resolved AwsSessionCredentials), testDotSessionTokenIsKept.
  • Without the fix, testHyphenSessionTokenIsMirrored fails (fs.s3a.session.token is null).
  • mvn -pl paimon-filesystems/paimon-s3-impl clean install green on JDK 11 (S3FileIOConfigTest 2/2, spotless/checkstyle); Docker-based MinIO tests not run locally.

S3FileIO mirrors the hyphenated access-key and secret-key options to the
dotted keys read by S3A, but not the session token. s3.session-token was
rewritten to fs.s3a.session-token, which S3A never reads, so temporary
credentials were signed without their token. Mirror it like its siblings
and document the option in the S3 filesystem docs.

Generated-by: Claude Code
@JingsongLi

Copy link
Copy Markdown
Contributor

Requirement fit: SUPPORTED. Implementation: CLEAN at e354f8a5c6.

The temporary-credential fix has caller-visible value, and I found no introduced code defect. Both new configuration tests pass locally with normal Maven checks. I also exercised real S3A/SDK HTTP reads against a local endpoint that rejects requests missing the session token: all three hyphenated prefixes (s3., s3a., fs.s3a.), plus the existing dotted option, return the expected content and include x-amz-security-token in the signed request. Local Docker was unavailable, so this probe does not claim live AWS/MinIO STS verification.

One production-validation gap remains: Flink 1 / Common was cancelled during “Build dependencies and run tests”, leaving the aggregate CI result failed. The last log reports completed passing tests, then cancellation; it does not establish completion of that suite. The Core/integrations JDK 8 and 11 checks are green. Please obtain a completed green Flink 1 check before merging.

@thswlsqls

Copy link
Copy Markdown
Contributor Author

@JingsongLi Thanks for the review. Flink 1 / Common now completes green on 84bc789a7.

The earlier run hit the suite's 100-minute timeout after SinkSavepointITCase stopped producing output, unrelated to the S3 option change. I merged the latest master to retrigger CI.

@JingsongLi
JingsongLi merged commit 826976f into apache:master Oct 3, 2026
34 of 36 checks passed
@thswlsqls
thswlsqls deleted the fix/s3-mirror-session-token branch October 4, 2026 03:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Support s3.session-token for temporary credentials in the S3 filesystem plugin

2 participants