You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
A native scan routed to libhdfs builds its client with create_hdfs_operator (native/core/src/parquet/parquet_support.rs), which sets only the name node on opendal's HDFS builder. libhdfs then creates a fresh Hadoop Configuration from the executor classpath, so it sees core-site.xml and hdfs-site.xml but none of the scan's Hadoop options: not the reader options, and not spark.hadoop.* settings. A filesystem whose fs.<scheme>.impl or other settings come only from those sources fails natively (No FileSystem for scheme ...) even though Spark can read it.
#6838 keeps the scan rule from claiming a libhdfs scheme that only a reader option lists, so those scans fall back to Spark. Settings for a libhdfs scheme still have to live in the executor's core-site.xml.
Describe the potential solution
Pass the scan's fs.* Hadoop options to libhdfs when building the client, for example through hdfsBuilderConfSetStr. opendal's HDFS builder has no way to set arbitrary configuration today, so this needs either an opendal change or building the libhdfs client directly. The forwarded keys should be limited to filesystem settings, and secrets among them handled the way the object_store path handles them.
Additional context
Once options reach libhdfs, the scan rule could again claim schemes listed only in reader options.
Activity