HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning by difin · Pull Request #6789 · apache/hive

difin · 2026-09-15T20:09:28Z

What changes were proposed in this pull request?

Hive wires up Iceberg REST server-side scan planning: when scan-planning-mode=server, split planning can go through the REST catalog (POST /plan) instead of only the serialized table snapshot on Tez/MR workers. Adds a small RestCatalogScanPlanning helper for config and copying catalog settings into job conf, and updates the Iceberg storage handler / input path to reload a live REST table on executors where appropriate. Tests cover configuration, job propagation, and embedded REST planning behavior.

Why are the changes needed?

Iceberg REST catalogs can plan scans on the catalog service instead of on every client. That keeps planning close to authoritative metadata and catalog credentials, can reduce work on HiveServer2 and task containers, and lets the REST service apply catalog-side optimizations as Iceberg evolves.

Hive today mostly plans reads from a serialized table snapshot shipped in the job configuration, which fits local/Hadoop catalogs but does not use the REST server’s planning API. For deployments that already run a REST catalog with scan planning, operators expect scan-planning-mode=server to work for Hive queries on Tez/MR - not only for single-node Iceberg clients. This change closes that gap so Hive participates in the same server-side planning model as the rest of the Iceberg REST ecosystem.

Does this PR introduce any user-facing change?

Yes. For REST Iceberg catalogs, operators can set iceberg.catalog.<catalog>.scan-planning-mode=server (via hive-site.xml or session SET) alongside existing REST catalog properties. When enabled, read jobs that use the Iceberg input format can use server-side scan planning instead of local planning from the serialized snapshot. Default behavior is unchanged when the property is unset or not server. Hive and non-REST catalogs are unaffected.

How was this patch tested?

TestRestCatalogScanPlanning — Tests config keys, server-mode detection, job propagation gating and copying.
TestRestCatalogScanPlanningServerIT — Uses embedded REST catalog; Verifies that RESTTable / RESTTableScan and PlanTableScanRequest on planTasks() are used when file planning is on.
TestHiveIcebergServerSideScanPlanning — Checks that Hive only reloads a table from the REST catalog for split planning when server mode is on and it is safe to do so - not when planning stays local or when the job must use uncommitted metadata from the same transaction.
TestHiveIcebergServerSideScanPlanningServerIT — Tests that executor job conf with/without propagated catalog props and table reload over REST.

Aggarwal-Raghav · 2026-09-16T09:19:51Z

There were already discussions happening under HIVE-29599 for server side planning?

difin · 2026-09-16T13:50:53Z

There were already discussions happening under HIVE-29599 for server side planning?

This is different - this PR is about adding client-side support.

tanishq-chugh

Hi @difin
Checked out the PR, great improvement 🚀

tanishq-chugh · 2026-09-17T07:35:23Z

+   */
+  public static Table resolveTableForScanPlanning(Configuration conf, String tableIdentifier) {
+    if (shouldReloadForServerSideScanPlanning(conf)) {
+      Table table = Catalogs.loadTable(conf);


I believe this will create a new RestCatalog Object/HttpClient on every call, curious if this can cause any connection leak?

Yes - Catalogs.loadTable creates a new REST catalog (and HTTP client) on each call, and we don’t close it on this path today. That’s the same pattern as other Catalogs.loadTable usages in the handler. Here it runs once per getSplits during split planning, not per input split. A reload only happens when shouldReloadForServerSideScanPlanning is true.

deniskuzZ · 2026-09-29T15:01:59Z

        "If this is set to true the URI for auth will have the default location masked with DEFAULT_TABLE_LOCATION"),
    HIVE_ICEBERG_ALLOW_DATAFILES_IN_TABLE_LOCATION_ONLY("hive.iceberg.allow.datafiles.in.table.location.only", false,
        "If this is set to true, then all the data files being read should be withing the table location"),
+    HIVE_ICEBERG_REST_SERVER_SIDE_SCAN_PLANNING_ENABLED(


hive.iceberg.rest.scan-planning-mode {client, server} ?

Renamed to hive.iceberg.rest.scan-planning-mode with client/server to mirror Iceberg REST catalog property naming; default client preserves prior opt-in behavior.

deniskuzZ · 2026-09-29T15:05:27Z

    setCommonJobConf(jobConf);
    configureOutputTableJobConf(tableDesc, jobConf);
+    if (tableDesc != null && tableDesc.getProperties() != null) {
+      String catalogName = tableDesc.getProperties().getProperty(InputFormatConfig.CATALOG_NAME);


why not move helper into tableDesc? i.e. tableDesc.getCatalogName()?

If we move this to If we move this ti tableDesc we will add Iceberg-specific code to hive-exec module.
How about tableDesc.getProperty(InputFormatConfig.CATALOG_NAME) instead?

deniskuzZ · 2026-09-29T15:07:45Z

    <!-- Upgrade roaringbit version whenever upgrading iceberg version -->
    <iceberg.version>1.11.0</iceberg.version>
+    <!-- Jetty 12 for Iceberg REST catalog embedded-server tests (Hive uses jetty.version 9.x elsewhere). -->
+    <iceberg.rest.test.jetty.version>12.1.8</iceberg.rest.test.jetty.version>


is it misalligned?

Two Jetty stacks are required (9 for Hive, 12 for Iceberg REST embedded-server tests). Putting it in the root pom.xml was wrong since it's used only in one test module. Fixed.

deniskuzZ · 2026-09-29T15:13:58Z

+import static org.mockito.Mockito.verify;
+
+/** Embedded REST server tests for the Hive executor reload path for server-side scan planning. */
+class TestHiveIcebergServerSideScanPlanningServerIT extends TestBaseWithRESTServer {


don't we use HiveRESTCatalogServerExtension restCatalogExtension = getHiveRESTCatalogServerExtension() do decorate with additional fixtures?

We use HiveRESTCatalogServerExtension for tests that hit the HMS-backed REST catalog servlet. These scan-planning Server ITs intentionally use Iceberg’s TestBaseWithRESTServer, an in-process Iceberg REST harness, not the HMS servlet, so we can spy RESTCatalogAdapter, drive HiveTableUtil.resolveTableForScanPlanning / job-conf propagation, and keep the unshaded Iceberg + Jetty 12 classpath this module needs. Different server, different layer under test.

deniskuzZ · 2026-09-29T15:15:45Z

+import static org.mockito.Mockito.verify;
+
+/** Embedded REST server tests for the Hive executor reload path for server-side scan planning. */
+class TestHiveIcebergServerSideScanPlanningServerIT extends TestBaseWithRESTServer {


is this a test framefork class or a test class?

Test class. It’s a concrete JUnit tests. It uses the framework base TestBaseWithRESTServer (from iceberg-core tests); it isn’t a reusable fixture or extension itself.

deniskuzZ · 2026-09-29T15:17:46Z

+ *
+ * @see <a href="https://iceberg.apache.org/docs/latest/catalog-properties/">REST catalog properties</a>
+ */
+public final class RestCatalogScanPlanning {


should it be in util package or have util in the name?

Moved this class to iceberg-handler (because it's not rest client specific) and added Util to the class name.

deniskuzZ · 2026-09-29T15:20:48Z

+  }
+
+  public static String catalogPropertyKey(String catalogName) {
+    return IcebergCatalogProperties.catalogPropertyConfigKey(


i see you use mix of static imports, catalogPropertyKey in setScanPlanningMode

deniskuzZ · 2026-09-29T15:21:25Z

+    setScanPlanningMode(conf, catalogName, RESTCatalogProperties.ScanPlanningMode.fromString(mode));
+  }
+
+  public static RESTCatalogProperties.ScanPlanningMode getScanPlanningMode(


naming seems to repeat class name

deniskuzZ · 2026-09-29T15:22:30Z

+  /**
+   * Resolves the catalog name from per-table {@code iceberg.catalog} or the session default catalog.
+   */
+  public static String resolveCatalogName(Configuration conf, String catalogNameFromTable) {


this seem to be a repeated functionality

deniskuzZ · 2026-09-29T15:24:23Z

+   * Called from {@code configureJobConf} so split generation sees catalog URI/type/scan-planning-mode
+   * even when job properties were not copied yet.
+   */
+  public static void propagateCatalogPropertiesToJob(


please check visibility modifiers in public methods if the actually need thos

deniskuzZ · 2026-09-29T15:25:43Z

+ * {@link HiveTableUtil#resolveTableForScanPlanning}. Embedded REST server coverage is in
+ * {@code TestHiveIcebergServerSideScanPlanningServerIT} in {@code itests/hive-iceberg-rest-server}.
+ */
+class TestHiveIcebergServerSideScanPlanning {


does it test ServerSideScanPlanning? usesDeserializedTable

deniskuzZ · 2026-09-29T15:27:52Z

+    String tableIdentifier = conf.get(InputFormatConfig.TABLE_IDENTIFIER);
+    Table table = HiveTableUtil.resolveTableForScanPlanning(conf, tableIdentifier);
+    String executorTableId = tableIdentifier != null ? tableIdentifier : table.name();
+    if (tableIdentifier == null) {


you check tableIdentifier and set executorTableId, that is inconsistent.

deniskuzZ · 2026-09-29T15:28:51Z

-        });
+    String tableIdentifier = conf.get(InputFormatConfig.TABLE_IDENTIFIER);
+    Table table = HiveTableUtil.resolveTableForScanPlanning(conf, tableIdentifier);
+    String executorTableId = tableIdentifier != null ? tableIdentifier : table.name();


maybe simply tableId ?

deniskuzZ · 2026-09-29T15:31:18Z

+   * uses the deserialized snapshot so uncommitted metadata is visible.
+   */
+  public static Table resolveTableForScanPlanning(Configuration conf, String tableIdentifier) {
+    if (shouldReloadForServerSideScanPlanning(conf)) {


what is this reload, isn't it expensive? what if we have multiple requests in same query/session?

sonarqubecloud · 2026-09-30T04:17:30Z

Quality Gate passed

Issues
103 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
1.4% Duplication on New Code

See analysis details on SonarQube Cloud

asf-ci-hive added the tests pending label Sep 15, 2026

difin changed the title ~~File planning~~ HIVE-30056: Iceberg: Add REST catalog server-side scan planning support Sep 15, 2026

asf-ci-hive added tests failed and removed tests pending labels Sep 15, 2026

difin force-pushed the file_planning branch from 7e54757 to 769a954 Compare September 15, 2026 20:24

asf-ci-hive added tests pending tests failed and removed tests failed tests pending labels Sep 16, 2026

difin force-pushed the file_planning branch from 769a954 to 3ef3d65 Compare September 16, 2026 13:44

asf-ci-hive added tests pending and removed tests failed labels Sep 16, 2026

difin changed the title ~~HIVE-30056: Iceberg: Add REST catalog server-side scan planning support~~ HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning Sep 16, 2026

asf-ci-hive added tests failed and removed tests pending labels Sep 16, 2026

tanishq-chugh reviewed Sep 17, 2026

View reviewed changes

asf-ci-hive added tests pending tests failed and removed tests failed tests pending labels Sep 18, 2026

difin force-pushed the file_planning branch from 8ef09d9 to 0db254a Compare September 22, 2026 22:56

asf-ci-hive added tests pending tests failed and removed tests failed tests pending labels Sep 22, 2026

difin force-pushed the file_planning branch from 0db254a to e4ef113 Compare September 23, 2026 22:31

asf-ci-hive added tests pending and removed tests failed labels Sep 23, 2026

asf-ci-hive added tests pending tests failed and removed tests unstable tests pending labels Sep 25, 2026

HIVE-30056: Iceberg: Add REST catalog server-side scan planning support

19151e4

difin force-pushed the file_planning branch from 81f40f7 to 19151e4 Compare September 25, 2026 16:44

asf-ci-hive added tests pending tests failed and removed tests failed tests pending labels Sep 25, 2026

deniskuzZ reviewed Sep 29, 2026

View reviewed changes

Review comments from Sep 29 (partial).

45f0bc7

asf-ci-hive added tests pending and removed tests failed labels Sep 30, 2026

asf-ci-hive added tests failed and removed tests pending labels Sep 30, 2026

Conversation

difin commented Sep 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

Aggarwal-Raghav commented Sep 16, 2026

Uh oh!

difin commented Sep 16, 2026

Uh oh!

tanishq-chugh left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

deniskuzZ Sep 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

deniskuzZ Sep 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sonarqubecloud Bot commented Sep 30, 2026

Quality Gate passed

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

difin commented Sep 15, 2026 •

edited

Loading

deniskuzZ Sep 29, 2026 •

edited

Loading

deniskuzZ Sep 29, 2026 •

edited

Loading