Skip to content

Pass a scan's Hadoop options to libhdfs #6967

Open

Description

@dwsmith1983

What is the problem the feature request solves?

Found in #6838 (review comment).

A native scan routed to libhdfs builds its client with create_hdfs_operator (native/core/src/parquet/parquet_support.rs), which sets only the name node on opendal's HDFS builder. libhdfs then creates a fresh Hadoop Configuration from the executor classpath, so it sees core-site.xml and hdfs-site.xml but none of the scan's Hadoop options: not the reader options, and not spark.hadoop.* settings. A filesystem whose fs.<scheme>.impl or other settings come only from those sources fails natively (No FileSystem for scheme ...) even though Spark can read it.

#6838 keeps the scan rule from claiming a libhdfs scheme that only a reader option lists, so those scans fall back to Spark. Settings for a libhdfs scheme still have to live in the executor's core-site.xml.

Describe the potential solution

Pass the scan's fs.* Hadoop options to libhdfs when building the client, for example through hdfsBuilderConfSetStr. opendal's HDFS builder has no way to set arbitrary configuration today, so this needs either an opendal change or building the libhdfs client directly. The forwarded keys should be limited to filesystem settings, and secrets among them handled the way the object_store path handles them.

Additional context

Once options reach libhdfs, the scan rule could again claim schemes listed only in reader options.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:scanParquet scan / data readingenhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions