Repository navigation
Improve integration of hadoop s3a and comet #1829
Description
What is the problem the feature request solves?
#1817 introduces integration that allows usage of hadoop-aws's s3a configuration with the object_store used by the native scan implementations.
Some more work can be done to improve this -
-
The feature works with
native_datafusionbut not withnative_iceberg_compat. To be truly useful we need the same integration innative_iceberg_compat. -
To allow usage of custom authentication, we need a JNI based adapter that can get credentials from an AWS credentials provider (Both AWS SDK V1 and V2 credentials providers must be supported).
-
Document usage.
Describe the potential solution
No response
Additional context
No response
Activity
andygrove commentedon May 6, 2026
@parthchandra is this issue still valid?
parthchandra commentedon May 7, 2026
That depends on whether we want to keep the
hdfsimplementation or whether we want to useOpenDALexclusively. I'm not sure if anyone has spent time ensuring there are no gaps in coverage between the two implementations.dwsmith1983 commentedon Oct 4, 2026
All three parts look done on main:
native_iceberg_compatwas removed (#4019, #4363), S3A credential providers work through the JVM bridge and the built-in adapters from #4309 and #6023, ands3-credential-providers.mddocuments it. I think this can be closed.