[filesystem][s3] Mirror s3.session-token to fs.s3a.session.token - #10228
Conversation
S3FileIO mirrors the hyphenated access-key and secret-key options to the dotted keys read by S3A, but not the session token. s3.session-token was rewritten to fs.s3a.session-token, which S3A never reads, so temporary credentials were signed without their token. Mirror it like its siblings and document the option in the S3 filesystem docs. Generated-by: Claude Code
|
Requirement fit: SUPPORTED. Implementation: CLEAN at The temporary-credential fix has caller-visible value, and I found no introduced code defect. Both new configuration tests pass locally with normal Maven checks. I also exercised real S3A/SDK HTTP reads against a local endpoint that rejects requests missing the session token: all three hyphenated prefixes ( One production-validation gap remains: Flink 1 / Common was cancelled during “Build dependencies and run tests”, leaving the aggregate CI result failed. The last log reports completed passing tests, then cancellation; it does not establish completion of that suite. The Core/integrations JDK 8 and 11 checks are green. Please obtain a completed green Flink 1 check before merging. |
|
@JingsongLi Thanks for the review. Flink 1 / Common now completes green on The earlier run hit the suite's 100-minute timeout after |
Purpose
fix #10227
S3FileIOmirrors hyphenatedaccess-key/secret-keyto the dotted S3A keys, but not the session token.s3.session-tokenbecamefs.s3a.session-token, which S3A never reads, so temporary STS credentials passed via Flink/Spark/Hive catalog options were signed without the token (403).{"fs.s3a.session-token", "fs.s3a.session.token"}toMIRRORED_CONFIG_KEYS; precedence matches the existing mirrors. Dotteds3.session.tokenis unaffected.s3.session-token([python] Support S3 options in filesystem catalog #7712).s3.session-tokenin the S3 section offilesystems.mdx.Tests
S3FileIOConfigTest:testHyphenSessionTokenIsMirrored(conf key plus resolvedAwsSessionCredentials),testDotSessionTokenIsKept.testHyphenSessionTokenIsMirroredfails (fs.s3a.session.tokenis null).mvn -pl paimon-filesystems/paimon-s3-impl clean installgreen on JDK 11 (S3FileIOConfigTest2/2, spotless/checkstyle); Docker-based MinIO tests not run locally.