[EPIC] Production-quality native Iceberg writes #5649

What is the problem the feature request solves?

Native Iceberg writes landed in three parts under #4322: the split writer/committer plan (#4658), the eligibility allowlist with fall-back reasons (#5298), and the iceberg-rust data-file writer with JVM-side manifest metrics (#5361). #4322 closed when #5361 merged, and no issue tracks what remains between "merged behind two off-by-default flags" and "production quality". The open follow-ups exist but are not linked to each other, and several gaps had no issue at all.

This epic is that plan. The goal is unchanged from #4322: an ETL job that scans, transforms, and writes Iceberg tables runs natively end to end, and falls back to iceberg-java wherever the native path cannot reproduce its outcome.

Current state (2026-10-13). Comet's Iceberg write path is on by default in 1.2.0, on every Spark version (#6664). spark.comet.write.iceberg.enabled (renamed from spark.comet.iceberg.write.enabled by #5306) defaults to true and plans both the split operator and the native writer, and spark.comet.write.iceberg.splitOperator.enabled is a testing setting. #5644 lists the evidence to gather before 1.2.0 ships. Each Iceberg Spark test job's summary reports which writer ran each write (#6155). In the merge-queue run that merged #6664, 7,175 of the 13,677 writes in the Iceberg 1.11 jobs ran natively (52.5%), up from 47.5% on 2026-10-06, mostly because format-version 3 tables are now admitted (#6668). Most of the rest are Avro and ORC tables (2,458), merge-on-read WriteDelta (1,477), input that is not in Arrow format (1,058), format-version 4 (581), and format-version 3 copy-on-write and compaction writes, which carry row lineage (215). Phase 1 is done, and Phase 2 has one open issue, with a PR.

Describe the potential solution

Phased, in priority order. Each item is a linked issue; check it here when it closes.

Phase 1: correctness blockers

• Native Iceberg writes use an iceberg-rust revision missing partition-path escaping #5636 Bump the iceberg-rust pin past the partition-path escaping fix (a # in an S3 partition value commits a location iceberg-java resolves to a different key)
• Native Iceberg writes drop Hadoop GCS configuration for HadoopFileIO + gs:// #5637 Native writes drop Hadoop GCS configuration for HadoopFileIO + gs://; fail closed until the translation is complete
• Stale native Iceberg write wording in iceberg-writes.md #5641 Fix the stale "not yet wired in" wording and the divergence list in iceberg-writes.md
• revertToSpark erases CometIcebergWriteExec / CometNativeWriteExec because originalPlan is the node's own child #5719 revertToSpark erases the write node, because CometIcebergWriteExec and CometNativeWriteExec report their own child as originalPlan (opt-in: needs spark.comet.exec.transitionRevert.enabled; PR fix: restore Spark write execs when reverting transition-heavy stages #5957)
• Native Iceberg write merges -0.0 and 0.0 rows into one float/double identity partition #6138 -0.0 and 0.0 rows merged into one float/double identity partition (silent wrong partitioning); such tables now fall back (PR fix: fall back for Iceberg writes with float or double identity partitions #6531)
• Native Iceberg write silently drops S3 settings it cannot honour (credentials provider, SSE, ACL, tags, remote signing) instead of falling back #6139 S3 settings the native writer cannot honour (credentials provider, SSE, ACL, tags, remote signing) are dropped instead of falling back (PR fix: gate unsupported Iceberg S3 write settings #6441)
• Native Iceberg write gate reads hdfs:/path as file, so the write fails natively instead of falling back #6140 The gate reads hdfs:/path as file, so the write fails natively instead of falling back (PR fix: read the Iceberg write gate's storage scheme the same way the native factory does #6502)
• Native Iceberg write fails on a V1 spec that mixes a live field with a void field whose source column was dropped #6141 A V1 spec mixing a live field with a void field whose source column was dropped fails every task; such writes now fall back, so the error is iceberg-java's own (PR fix: decline native Iceberg writes through a void partition field whose source column was dropped #6677)
• Native Iceberg write panics or writes a NULL partition value for dates and timestamps beyond year 262143 #6145 Dates and timestamps beyond year 262143 panic or get a NULL partition value
• Native Iceberg write over-counts NaNs for nested float fields when the input batch is already sliced #6146 NaN over-count for nested float fields when the input batch is already sliced (PR fix: compact already-sliced batches before the native Iceberg writer counts NaNs #6238)
• docs: fix stale and missing native write details in iceberg-writes.md and iceberg.md #6147 Stale and missing native write details in iceberg-writes.md and iceberg.md
• Native Iceberg writer counts NaNs under NULL structs and NULL list or map entries #6562 NaNs under a NULL struct, or under a NULL list or map entry that still holds elements, are counted in nan_value_counts, where iceberg-java counts none (PR fix: drop values under NULL parents before the native Iceberg writer counts NaNs #6658)
• Preserve catalog transactions for split Iceberg writes on Spark 4.2 #6586 On Spark 4.2, IcebergCommitExec does not implement TransactionalExec, so a catalog transaction Spark attaches to the write is never committed (PR fix(iceberg): preserve Spark 4.2 write transactions #6587)

Phase 2: failure handling parity with iceberg-java

• Native Iceberg write: clean up task-attempt data files on task failure #5618 Clean up task-attempt data files on task failure
• Remove orphaned files in comet iceberg split writers #5277 Remove orphaned files when the commit fails
• Failure-injection tests for native Iceberg writes #5646 Failure-injection tests that assert both the visible outcome and the storage state (PR test: cover speculation and executor loss in multi-task native Iceberg writes #6601 covers speculation and executor loss)
• Native Iceberg Write: close cleanup ownership gap during JVM handoff #6581 Keep native cleanup armed until the JVM has taken ownership of the written files, so a failure during the handoff cannot leave an uncommitted file with no cleanup owner (PR fix: close Iceberg cleanup ownership gap during JVM handoff #6582)
• Spark errors after the first batch of a JVM input stream reach the user as CometNativeException #6234 A Spark error thrown after the first input batch reaches the user wrapped in CometNativeException instead of as the exception Spark threw (PR fix: rethrow the exception a JVM input throws instead of a CometNativeException #6243)

Phase 3: coverage so common workloads stay native

• Support Iceberg's system functions (bucket, truncate, years/months/days/hours) as native expressions #5635 Iceberg system functions (bucket, truncate, years/months/days/hours) as native expressions, so the default hash-distribution write is native end to end (PR feat: Support Iceberg system functions (bucket, truncate, years/months/days/hours) natively #5638)
• feat: add native support for MergeRowsExec (row-level MERGE INTO) #5318 Native MergeRowsExec (Spark 3.5 and 4.0, behind spark.comet.exec.mergeRows.enabled, default off)
• Support native MergeRowsExec on Spark 4.1+ with MergeSummary #6606 Native MergeRowsExec on Spark 4.1+, where the V2 write reads its metrics to build MergeSummary (PR feat: support native MergeSummary on Spark 4.1+ #6640)
• Support Spark 4.2 insert-only MERGE execution #6612 Spark 4.2 insert-only MERGE, which Spark 4.2 can plan as InsertOnlyMergeExec instead of through MergeRowsExec (PR feat: support Spark 4.2 insert-only MERGE #6696)
• Accelerate row-level MERGE / UPDATE / DELETE plans (MergeRowsExec, ReplaceDataExec, WriteDeltaExec) #5122 Row-level MERGE / UPDATE / DELETE plans
• Support Iceberg SparkCopyOnWriteScan in IcebergReflection.ICEBERG_SCAN_CLASSES #5319 Recognise Iceberg's SparkCopyOnWriteScan as a native scan, so copy-on-write DELETE and UPDATE can feed the native writer without a Comet shuffle in between (draft PR feat: recognize SparkCopyOnWriteScan as a native Iceberg scan  #5320)
• Support Native Iceberg MOR write #6240 Native merge-on-read DELETE, UPDATE and MERGE: write the position deltas natively instead of through Iceberg's JVM WriteDelta path. The first step is done: Plan Iceberg WriteDelta in the split write path and keep it on the JVM DeltaWriter #6692 (PR feat(iceberg): plan WriteDelta in the split write path and keep it on the JVM DeltaWriter #6693) plans WriteDelta in the split write path and keeps it on the JVM DeltaWriter
• Decide keep-or-lift for each remaining native Iceberg write eligibility restriction #5643 A keep-or-lift decision for every remaining eligibility restriction, so the fallback surface is intentional (bloom filters: PR feat: support bloom filters in native Iceberg writes #5724; float/double identity partitions: PR feat: bump iceberg-rust and write float and double identity partitions natively #6851, which bumps the iceberg-rust pin past fix(spec): treat -0.0 and 0.0 as distinct float literals iceberg-rust#3327)
• Native Iceberg writes: support format-version 3 tables #6668 Native writes to format-version 3 tables, declining the copy-on-write and compaction writes that carry row lineage (PR feat: support native Iceberg writes to format-version 3 tables #6672)
• Native Iceberg writes: write row lineage for copy-on-write DML and compaction on format-version 3 tables #6669 Write row lineage natively, so v3 copy-on-write DML and rewrite_data_files can run natively too
• Reconcile the two config namespaces for native writes #5306 Reconcile the two config namespaces for native writes
• Native Iceberg write gate misses a custom location provider supplied by TableOperations #6144 Detect a custom location provider supplied by TableOperations, not only the table property

Phase 4: enable by default

• Iceberg write split-operator plan: CI failures when enabled by default #5259 Fix the four CI failure buckets when the split plan is on by default
• Enable the split-operator plan and native Iceberg writes by default #5644 Graduation criteria and the default flip. feat: enable native Iceberg writes by default #6664 turned both defaults on in 1.2.0, on every Spark version. What is left is the evidence to gather before 1.2.0 ships: a nightly soak, a classified fallback report, metrics parity per data type, and Decide keep-or-lift for each remaining native Iceberg write eligibility restriction #5643
• IcebergWriteStrategy ignores spark.comet.enabled, so the split-operator write plan survives the kill switch #6142 IcebergWriteStrategy ignores spark.comet.enabled (blocks flipping the split-operator default)
• IcebergCommitExec rethrows the raw cause when the abort fails, where Spark wraps it in writingJobFailedError #6143 IcebergCommitExec rethrows the raw cause when the abort fails, where Spark wraps it in writingJobFailedError (PR fix: wrap Iceberg split-write failures the way Spark does when abort fails #6153)
• Make native Iceberg write coverage visible in the Iceberg Spark test CI jobs #6148 Make native write coverage visible in the Iceberg Spark test CI jobs, so a green run means the native path ran (PR test: report which writer ran each Iceberg write in the Iceberg CI jobs #6155)

Phase 5: performance

• Add a native Iceberg write benchmark #5647 A native Iceberg write benchmark in the repository (unpartitioned, default hash distribution, fanout, copy-on-write delete)
• Account the native Iceberg writer's buffers in Comet's memory pool #5648 Account the writer's buffers in Comet's memory pool, especially the fanout writer's per-partition buffers (PR feat: charge native write buffers to the task's memory pool #6247)
• Charge native write buffers to the shared off-heap memory pool #6115 Charge native write buffers to the shared off-heap memory pool, for the Parquet writer as well as the Iceberg writer (PR feat: charge native write buffers to the task's memory pool #6247)
• Native fanout Iceberg writes: sort the remaining rows by partition when the memory pool runs short, instead of writing small files #6818 When the memory pool runs short, have a fanout write sort its remaining rows by partition instead of closing partitions early and writing small files
• Native Iceberg writes: count the DataFiles a task keeps in its memory reservation #6885 Count the DataFiles a task keeps in its memory reservation
• Native Sort-Merge Writer for Iceberg ClusteredWriter Path #3595 Fused sort plus write for the clustered path
• Native Iceberg writer keeps a dictionary page for high-cardinality columns where iceberg-java writes none #6114 Drop the dictionary for high-cardinality columns the way parquet-mr does, so selective reads of native-written files don't fetch a ~2 MB dictionary page per column (PR perf: choose dictionary encoding like iceberg-java in the native Iceberg writer #6241)
• Hand natively written Iceberg files' Parquet footers to the JVM instead of reading them back #6772 Hand each written file's Parquet footer to the JVM instead of reading it back from storage, which costs two small reads per file on object stores
• Native Iceberg writes do not use shuffle direct read for their input #6913 Read shuffle blocks directly when a shuffle feeds the native writer (PR perf: read shuffle blocks directly in native Iceberg writes #6926)
• Make the native Iceberg writer's DictionaryChooser pre-pass cheaper #6914 A cheaper DictionaryChooser pre-pass (PR perf: make the native Iceberg writer's dictionary pre-pass 1.8x to 3.9x faster #6937)
• Native Iceberg writes re-parse Hadoop's XML configuration in every task to re-read footers #6912 Stop re-parsing Hadoop's XML configuration in every task when reading footers back (PR perf: read native-written Parquet footers without parsing Hadoop's configuration #6923)
• Native Iceberg write eligibility builds two Hadoop configurations on every planning pass #6915 Stop building two Hadoop configurations for the eligibility check on every planning pass (PR perf: avoid duplicate Hadoop configs in Iceberg write detection #6925)
• Use RoaringTreemap for MergeRows cardinality tracking #6608 Track MergeRowsExec cardinality with a Roaring bitmap as Spark does; the HashSet<i64> it uses cannot spill and can exhaust memory where Spark fits
• Optimize MergeRows predicate evaluation and row selection #6611 Speed up MergeRowsExec predicate evaluation: evaluate later clauses only on rows no earlier clause claimed, and gather only the columns they reference

#6911 tracks the rest of what profiling the end-to-end benchmark (#6910) found, including the parquet-rs encoding work upstream.

Ongoing

• Adopt a pin-bump policy for the iceberg-rust dependency #5645 A pin-bump policy for iceberg-rust, so upstream fixes to file bytes, manifests, or partition layout reach Comet on a schedule
• [EPIC] Iceberg Feature Matrix: Spark-Iceberg vs iceberg-rust vs datafusion-comet #3756 Update the Iceberg feature matrix, which still lists the write path as not implemented

Additional context

Done and merged: #4658 (split-operator plan), #5298 (eligibility detection), #5361 (native writer). Related umbrellas: #5121 (DataSource V2 writes in general), #1625 (native Parquet writes for the V1 path).

Native write bugs filed and fixed since this epic was opened, outside the phases above: #5689, #5690, #5691, #5693, #5694, #5698, #5776, #5836, #5837, #6771. #6659 (float value and null counts under a NULL struct on Iceberg 1.9+) was closed as a divergence the user guide already lists; #6681 corrected the Iceberg version the guide gives for it. Open outside the phases: #6803, where five CometIcebergWriteDetectionSuite tests assume a split write plan and fail once #6780 makes a write over a local table scan fully native.

The accepted divergences between iceberg-rust and iceberg-java output are documented in iceberg-writes.md; changes to that list belong in the issues above rather than here.