Decide keep-or-lift for each remaining native Iceberg write eligibility restriction #5643

What is the problem the feature request solves?

The native Iceberg write eligibility gate (CometIcebergNativeWrite, documented in iceberg-writes.md) is an allowlist, and each restriction it enforces was added conservatively in #5298 so the writer could land incrementally. Most of the restrictions have no issue saying whether they are permanent or what would lift them, so there is no way to tell how much of the fallback surface is intended.

Current restrictions on main (2026-10-13), each of which makes a write fall back to iceberg-java. The eligibility table in the user guide's iceberg-writes.md gives the supported values for each, and its "Plans for each fallback" section (PR #6961) records the decisions below. A checked item has its decision recorded.

Table format, operation and schema:

• a write format other than Parquet (Avro or ORC). Keep, permanent: the native writer writes Parquet only, and iceberg-rust has no Avro or ORC data file writer.
• format version 3: lifted by Native Iceberg writes: support format-version 3 tables #6668 (PR feat: support native Iceberg writes to format-version 3 tables #6672), apart from the row lineage writes below
• format version 4 or newer. Keep, pending upstream: the version 4 spec is not final.
• on format version 3 with Iceberg 1.10 or newer, copy-on-write DELETE, UPDATE and MERGE and rewrite_data_files, which write the row lineage columns _row_id and _last_updated_sequence_number. Lift: Native Iceberg writes: write row lineage for copy-on-write DML and compaction on format-version 3 tables #6669.
• merge-on-read DELETE, UPDATE and MERGE, which plan WriteDelta and keep iceberg-java's DeltaWriter. Lift on format version 2: Support Native Iceberg MOR write #6240 (needs feat: add file-scoped position delete index loader iceberg-rust#3291 and feat: add sorting position-only delete writer iceberg-rust#3329). Keep, pending upstream on format version 3, which writes deletion vectors: Deletion-vector writer + RowDelta action iceberg-rust#2580.
• uuid columns (Spark plans them as strings, and no Arrow cast reaches fixed(16)). Lift: Native Iceberg writes: support uuid and unknown columns #6959.
• the v3 types variant, unknown, timestamp_ns, geometry and geography. unknown: Lift, Native Iceberg writes: support uuid and unknown columns #6959 (iceberg-java 1.10+ leaves unknown fields out of the data file). variant: Keep, pending Comet's native VariantType support ([Variant] Support whole-value VariantType in native Parquet writes #5433, part of [EPIC] Native Variant Support #5438). timestamp_ns, geometry, geography: Keep, pending upstream: Iceberg's Spark integration has no Spark type for them (TypeToSparkType throws), so Spark cannot plan such a write today; the rule is a guard for when it can.
• an identity partition on a float or double column, including a nested one, because iceberg-rust treats -0.0 and 0.0 as one partition value (Native Iceberg write merges -0.0 and 0.0 rows into one float/double identity partition #6138, PR fix: fall back for Iceberg writes with float or double identity partitions #6531). Lift: PR feat: bump iceberg-rust and write float and double identity partitions natively #6851 bumps the iceberg-rust pin past fix(spec): treat -0.0 and 0.0 as distinct float literals iceberg-rust#3327, which merged 2026-10-09.
• a format version 1 spec that mixes a live field with a void field whose source column was dropped. Keep, permanent: iceberg-java cannot write through it either, so falling back surfaces iceberg-java's own error (Native Iceberg write fails on a V1 spec that mixes a live field with a void field whose source column was dropped #6141).

Parquet properties:

• write.parquet.bloom-filter-max-bytes and per-column bloom filter enablement. Lift: PR feat: support bloom filters in native Iceberg writes #5724.
• write.parquet.page-version=v2. Lift: Native Iceberg writes: translate the Parquet settings that have a parquet-rs equivalent #6954 (WriterVersion::PARQUET_2_0; parquet-rs and parquet-mr choose the same v2 fallback encodings).
• write.parquet.shred-variants=true. Lift: Native Iceberg writes: stop falling back on settings iceberg-java's writer ignores #6955. Iceberg 1.11 shreds only top-level variant columns, which the type rule already declines, so the setting cannot change an eligible write.
• non-default write.parquet.row-group-check-min-record-count / max-record-count. Lift: Native Iceberg writes: translate the Parquet settings that have a parquet-rs equivalent #6954. The only native dependency is the first page size check in DictionaryChooser, which hard-codes the default of 100.
• a write.parquet.compression-level outside the native writer's per-codec range (zstd 1 to 22, gzip 0 to 9, brotli 0 to 11), such as zstd 0 or a negative zstd level, which iceberg-java accepts. zstd 0: Lift, Native Iceberg writes: translate the Parquet settings that have a parquet-rs equivalent #6954 (zstd defines 0 as its default level, 3). Negative zstd: Keep, pending upstream, Parquet: accept zstd compression level 0 and negative levels in ZstdLevel arrow-rs#11545. gzip and brotli outside their ranges: Keep, permanent. While checking this, Native Iceberg writes commit brotli files that no JVM reader can read without Hadoop's BrotliCodec #6960 was found: iceberg-java fails gzip with any numeric level and brotli without a codec jar, which the native writer currently accepts.
• a row-group-size-bytes, page-size-bytes, page-row-limit or dict-size-bytes that is not a positive Java int. Keep, permanent: iceberg-java fails such a write, so falling back surfaces its error.
• any unvetted write.parquet.* key, and any parquet.* table property (including parquet.enable.dictionary). Unvetted write.parquet.* keys: Keep, permanent, so a key a newer Iceberg version adds is never silently ignored; each known key has its own decision (bloom filters above; stats-enabled.column.*: Lift, Native Iceberg writes: translate the Parquet settings that have a parquet-rs equivalent #6954). parquet.* table properties: Lift. parquet.enable.dictionary and parquet.columnindex.truncate.length translate directly (Native Iceberg writes: translate the Parquet settings that have a parquet-rs equivalent #6954), and iceberg-java's writer reads no other parquet.* key except codec settings (Native Iceberg writes: stop falling back on settings iceberg-java's writer ignores #6955).
• any parquet.* key in the session Hadoop configuration (other than parquet.hadoop.vectored.io.enabled). Lift: Native Iceberg writes: stop falling back on settings iceberg-java's writer ignores #6955. From the Hadoop configuration, iceberg-java's writer reads only parquet.columnindex.truncate.length and codec settings, and with S3FileIO it does not see session settings at all.

Location, storage and encryption:

• write.object-storage.enabled=true, write.location-provider.impl, or any table.locationProvider() other than Iceberg's DefaultLocationProvider, such as one a custom TableOperations supplies (Native Iceberg write gate misses a custom location provider supplied by TableOperations #6144). Object storage: Lift, Native Iceberg writes: support the object-storage layout (write.object-storage.enabled=true) #6956 (iceberg-rust's ObjectStorageLocationGenerator already implements the Iceberg 1.7+ layout; 1.5.2 hashes differently). A custom provider: Keep, permanent: it is user code in the JVM that the native writer cannot call.
• io-impl set, or a table.io() outside the recognized class hierarchy, including the EncryptingFileIO family. io-impl as a table property: Lift, Native Iceberg writes: stop falling back on settings iceberg-java's writer ignores #6955 (iceberg-java reads io-impl from catalog properties only, and requireRecognizedTableFileIO already checks the instantiated FileIO). A custom FileIO: Keep, permanent (user code). EncryptingFileIO: see encryption.
• any encryption.* property, or a table.encryption() other than PlaintextEncryptionManager. Iceberg's standard table encryption: Keep, pending upstream, Support client-side encryption iceberg-rust#2034. iceberg-rust at the pinned revision can already write Iceberg-encrypted Parquet with per-file key metadata, but its encryption work is still landing; once it is complete, wiring it into the native writer is bounded. A custom encryption manager: Keep, permanent (user code).
• data location schemes other than file, memory, s3, s3a, gs (oss falls back because oss.* properties are not forwarded), and an s3, s3a or gs location with no bucket in its authority. Azure (abfs, abfss, wasb, wasbs), oss and S3-compatible aliases: Lift, Native Iceberg writes: support Azure and oss data locations, S3-compatible scheme aliases, and gs through HadoopFileIO #6958 (Azure after Native Iceberg scan: support abfs/abfss (Azure Data Lake Storage Gen2) locations #6058). hdfs: Keep, pending upstream, feat(storage): Add HDFS support via opendal services-hdfs-native iceberg-rust#3111; Support Iceberg tables on HDFS in the native scan #5894 tracks the Comet side, write gate included. A location with no bucket in its authority: Keep, permanent (iceberg-java rejects it, or resolves it through Hadoop). Any other scheme: Keep, permanent unless an iceberg-rust storage backend serves it.
• a gs location opened by a FileIO other than GCSFileIO, because a HadoopFileIO takes its settings from fs.gs.*, which is not forwarded (Native Iceberg writes drop Hadoop GCS configuration for HadoopFileIO + gs:// #5637). Lift: Native Iceberg writes: support Azure and oss data locations, S3-compatible scheme aliases, and gs through HadoopFileIO #6958 (translate the fs.gs.* settings that have an iceberg-rust equivalent).
• for an s3 or s3a location, Hadoop S3A settings other than the access key, secret key, session token, endpoint, endpoint region and path-style access (a few read and committer settings that cannot affect a data-file write are ignored), and Iceberg FileIO S3 settings the native storage path cannot honour, such as an Iceberg credentials provider, ACLs, tags, remote signing or dsse-kms encryption (Native Iceberg write silently drops S3 settings it cannot honour (credentials provider, SSE, ACL, tags, remote signing) instead of falling back #6139). Lift, Native Iceberg writes: admit S3 settings that cannot change the written object, and translate S3 credential providers and encryption settings #6957: settings that cannot change the written object (fs.s3a.impl, pools, timeouts, retries, multipart and buffering), well-known credential providers, S3A encryption, and the ACL, storage class, requester-pays and dsse-kms settings OpenDAL supports. Keep, permanent: object tags, access points, transfer acceleration, dual-stack and cross-region endpoints, remote signing and custom signers, which the native S3 client has no equivalent for.

Describe the potential solution

For each restriction, record one of three decisions in this issue and in iceberg-writes.md:

• Lift: the native path already reproduces iceberg-java's outcome, or can with a bounded change; file a follow-up and link it.
• Keep, pending upstream: needs an iceberg-rust or parquet-rs feature; link the upstream issue.
• Keep, permanent: intentionally out of scope (for example encryption); say why.

The ordering should follow how often the setting appears in real tables. Object-storage layout (write.object-storage.enabled) and bloom filters are the two most likely to be hit by production tables and should be decided first.

Additional context

Part of the native Iceberg writes epic, #5649. Related: #5298 (introduced the allowlist), #5361 (native writer).