Repository navigation
feat: Write Iceberg Parquet data files natively via iceberg-rust (part 3 of 3) #5308
Description
What is the problem the feature request solves?
This is the third and final part of #4322 (accelerating Iceberg V2 write operations), which was split into three deliverables:
- Split-operator plan — feat: Optionally split the Iceberg V2 write operator into distinct writer and committer operations #4658 (merged). Iceberg's single V2 write command is rewritten into a writer operator (inside AQE) and a committer operator, with file writing still running through iceberg-java.
- Native-write eligibility detection — feat: detect Iceberg V2 writes and emit fall-back reasons #5298. An allowlist over the write's effective configuration decides whether a native write would produce the same outcome as iceberg-java, and reports a fall-back reason via extended EXPLAIN when it would not. The supported configuration and the accepted divergences are documented in
iceberg-writes.md.
This issue tracks part 3: actually performing the per-task Parquet data-file write via iceberg-rust when detection accepts the write.
Describe the potential solution
High-level approach:
- Conversion:
CometIcebergNativeWrite.convert(currently a stub) produces a native write operator. The JVM resolves all write settings and passes them to the native side via protobuf; the vettedwrite.parquet.*configuration (compression codec/level, row-group/page/dictionary sizes, page row limit) is translated into parquet-rsWriterProperties. Footer statistics are written untruncated to match parquet-mr. - Writer modes: unpartitioned, clustered, and fanout writers, mirroring iceberg-java's writer selection.
- Transport: the native side serialises the resulting
DataFiles as an in-memory Iceberg manifest via iceberg-rust'sManifestWriter; the JVM decodes the bytes with Iceberg's ownManifestFiles.read. Both ends use their library's spec codec, so no hand-rolled Avro. - Commit: unchanged from part 1 — the recovered
DataFile[]goes intoSparkWrite$TaskCommitandBatchWrite.commit, preserving Iceberg's overwrite semantics, copy-on-write validation, WAP, and snapshot summaries. - Manifest metrics: assembled on the JVM at commit time using Iceberg's own
MetricsConfig/ParquetUtilcode against the written file's footer (plus NaN counts from the native writer, which the footer cannot provide). Java makes every metadata decision — metrics modes, the inferred-column cap, bound truncation, list/map bounds suppression — so thecounts/nonemetrics restrictions from part 2 can be lifted. - Testing: parity tests comparing native output against iceberg-java per data type (including a targeted float/double NaN metrics test), plus end-to-end tests showing compaction / sort / z-order rewrites running fully native.
The guiding rule from part 2 carries over: if the native write cannot reproduce iceberg-java's outcome for the effective configuration, the write falls back — the documented accepted divergences (row-group/file-rolling cadence, compressed page bytes, footer key-value metadata, partition-path escaping) are the only exceptions.
Additional context
Detection (#5298) is deliberately conservative so this part can land incrementally: individual restrictions can be relaxed as the native writer proves each behaviour byte- or semantics-identical.
Activity