docs: list known Parquet and Iceberg issues and add a data source audit log

docs/source/contributor-guide/data-source-audits/iceberg.md

@@ -0,0 +1,108 @@
+<!---
+  Licensed to the Apache Software Foundation (ASF) under one
+  or more contributor license agreements.  See the NOTICE file
+  distributed with this work for additional information
+  regarding copyright ownership.  The ASF licenses this file
+  to you under the Apache License, Version 2.0 (the
+  "License"); you may not use this file except in compliance
+  with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing,
+  software distributed under the License is distributed on an
+  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+  KIND, either express or implied.  See the License for the
+  specific language governing permissions and limitations
+  under the License.
+-->
+
+# Iceberg Audits
+
+> Audit notes for the native Iceberg scan and the native Iceberg writer. Absence of an entry means
+> the area has not been audited yet. Open issues are listed for users in
+> [Known Issues in Parquet and Iceberg Reads and Writes].
+
+## Native Iceberg scan
+
+### Nested types (audited 2026-10-15)
+
+- **Scope:** structs, arrays and maps. NULLs at every level, field ids, name mappings, schema
+  evolution, deletes and partitions.
+- **Method:** queries compared with Spark with Comet disabled, on Spark 4.1.3 with Iceberg 1.11 at
+  `main` 66c63bd307. A code review of iceberg-rust at the current pin ran alongside, and each
+  finding was reproduced before it was filed.
+- **Filed:** [#7033] (a data file that omits an optional nested field fails the scan), [#7038]
+  (residual predicates on struct fields are not pushed into iceberg-rust) and [#7039] (a file is
+  read in full when the query needs none of its columns). The audit also confirmed the open issues
+  [#6790], [#6794], [#6822], [#6993], [#6782], [#5540] and [#5257]. All are tracked in [#7040].
+- **Matched Spark:** the nested data of the Parquet scan audit written by iceberg-java,
+  merge-on-read position deletes across three-row pages (format v2, and v3 deletion vectors),
+  equality deletes including one with a NULL key, required nested fields under NULL parents, nested
+  UUID, fixed, timestamptz and timestamp_ntz leaves, nested type promotion inside structs, lists
+  and maps, the nested schema evolution fallback for lists and maps, `_partition` with NULL values
+  and spec evolution, and nested partition sources.
+- **Did not reproduce:** reordered struct siblings of the same type swapping values when the column
+  holds a map, in more than 20 probes, CTAS included.
+- **Not Comet bugs:** on a table identity-partitioned by a struct field, iceberg-java before 1.12
+  reads a NULL struct as a struct of NULLs when the partition source is the struct's first field.
+  [apache/iceberg#17320] fixed it in 1.12.0. Its title names nested initial defaults, but the
+  change also keeps a file column in the read when every projected field of a struct is a constant,
+  and an identity partition value is such a constant. The native scan returns NULL, as 1.12 does.
+- **Not covered:** Variant ([#5438]), encrypted tables, and Iceberg versions other than 1.11.
+
+## Native Iceberg writer
+
+### Nested types (audited 2026-10-16)
+
+- **Scope:** NULL structs, arrays and maps, NULLs inside them, field ids at every level, manifest
+  metrics, nested partition sources, sort orders, copy-on-write DML, and `NOT NULL` columns.
+- **Method:** each write made natively and with Comet disabled, on Iceberg 1.5.2 (Spark 3.4.3),
+  1.8.1 (Spark 3.5.9) and 1.11.0 (Spark 4.1.3) at `main` c230bf4e98. The comparison covered the
+  values read back, the manifest metrics of each data file (value, null and NaN counts, lower and
+  upper bounds), the Parquet footer, and the files pruned for nested predicates. A code review of
+  the writer, arrow-rs 59.3 and iceberg-rust ran alongside, and each finding was reproduced before
+  it was filed.
+- **Filed:** [#7049] (writes into `NOT NULL` columns fall back on Spark 3.5 and later when no
+  exchange sits in front of the writer, because Comet has no native `AssertNotNull`) and [#7050]
+  (the user guide's claim about float and double null counts). Both are tracked in [#7051].
+- **Matched iceberg-java:** values, pruning, and field ids for struct fields, list elements and map
+  keys and values, on all three versions. Manifest metrics under the default and `full` metrics
+  modes on all three versions, and on 1.11.0 also under `counts`, `truncate(4)`, per-column
+  overrides on nested fields, `max-inferred-column-defaults`, and tiny pages and row groups. Nested
+  partition sources (identity on `s.a`, `bucket`, `truncate` and `days(ts.t)`), a sort order on a
+  nested field, and copy-on-write `DELETE`, `UPDATE` and `MERGE`. Required nested fields under NULL
+  parents written straight from a native scan, `fixed` fields under NULL structs, and copy-on-write
+  `UPDATE` with `AssertNotNull`, whose check still runs in Spark in front of the columnar shuffle.
+- **Not Comet bugs:** on Iceberg 1.9 and later, the value and null counts of a float or double field
+  under a NULL struct differ from iceberg-java's ([#6659], closed), and iceberg-java's are the wrong
+  ones ([apache/iceberg#17561], with a fix in [apache/iceberg#17560]). With Comet disabled on
+  1.11.0, `WHERE s.x IS NULL` returned 0 rows instead of 5, while 1.5.2 and 1.8.1 returned 5.
+- **Did not reproduce:** a task failure from validating a required struct under an optional struct,
+  with a required child, when a native scan feeds the writer. A task failure from `fixed(16)` values
+  of the wrong length hidden under NULL structs.
+- **Not measured:** the copy of every column that the writer makes when a float or double sits
+  under a list or map, to count NaNs correctly.
+- **Not covered:** merge-on-read writes (`WriteDelta`), format v3 types, encryption, object stores,
+  and write throughput, which needs a release build.
+
+[Known Issues in Parquet and Iceberg Reads and Writes]: ../../user-guide/latest/compatibility/parquet-iceberg-known-issues.md
+[apache/iceberg#17320]: https://github.com/apache/iceberg/pull/17320
+[apache/iceberg#17560]: https://github.com/apache/iceberg/pull/17560
+[apache/iceberg#17561]: https://github.com/apache/iceberg/issues/17561
+[#5257]: https://github.com/apache/datafusion-comet/issues/5257
+[#5438]: https://github.com/apache/datafusion-comet/issues/5438
+[#5540]: https://github.com/apache/datafusion-comet/issues/5540
+[#6659]: https://github.com/apache/datafusion-comet/issues/6659
+[#6782]: https://github.com/apache/datafusion-comet/issues/6782
+[#6790]: https://github.com/apache/datafusion-comet/issues/6790
+[#6794]: https://github.com/apache/datafusion-comet/issues/6794
+[#6822]: https://github.com/apache/datafusion-comet/issues/6822
+[#6993]: https://github.com/apache/datafusion-comet/issues/6993
+[#7033]: https://github.com/apache/datafusion-comet/issues/7033
+[#7038]: https://github.com/apache/datafusion-comet/issues/7038
+[#7039]: https://github.com/apache/datafusion-comet/issues/7039
+[#7040]: https://github.com/apache/datafusion-comet/issues/7040
+[#7049]: https://github.com/apache/datafusion-comet/issues/7049
+[#7050]: https://github.com/apache/datafusion-comet/issues/7050
+[#7051]: https://github.com/apache/datafusion-comet/issues/7051

docs/source/contributor-guide/data-source-audits/index.md

@@ -0,0 +1,42 @@
+<!---
+  Licensed to the Apache Software Foundation (ASF) under one
+  or more contributor license agreements.  See the NOTICE file
+  distributed with this work for additional information
+  regarding copyright ownership.  The ASF licenses this file
+  to you under the Apache License, Version 2.0 (the
+  "License"); you may not use this file except in compliance
+  with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing,
+  software distributed under the License is distributed on an
+  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+  KIND, either express or implied.  See the License for the
+  specific language governing permissions and limitations
+  under the License.
+-->
+
+# Data Source Audits
+
+This section records audits of Comet's native data source paths, the scans and writers for Parquet
+and Iceberg, in the same way that the [Expression Audits](../expression-audits/index.md) record
+audits of expressions. Each page holds dated entries per component. An entry records:
+
+- what was audited, at which commit, against which Spark and library versions, and how
+- the issues the audit filed, and the open issues it confirmed
+- what was checked and matched, so that it is not re-audited without a reason
+- the candidates that did not reproduce, so that they are not chased again
+- the differences that are bugs in Spark or iceberg-java rather than in Comet
+- what the audit did not cover
+
+A later audit of the same area adds a new dated entry instead of editing an old one, so the history
+stays visible. The open issues that users can hit are listed in the user guide, in
+[Known Issues in Parquet and Iceberg Reads and Writes](../../user-guide/latest/compatibility/parquet-iceberg-known-issues.md).
+
+```{toctree}
+:maxdepth: 1
+
+Parquet <parquet>
+Iceberg <iceberg>
+```

docs/source/contributor-guide/data-source-audits/parquet.md

@@ -0,0 +1,108 @@
+<!---
+  Licensed to the Apache Software Foundation (ASF) under one
+  or more contributor license agreements.  See the NOTICE file
+  distributed with this work for additional information
+  regarding copyright ownership.  The ASF licenses this file
+  to you under the Apache License, Version 2.0 (the
+  "License"); you may not use this file except in compliance
+  with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing,
+  software distributed under the License is distributed on an
+  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+  KIND, either express or implied.  See the License for the
+  specific language governing permissions and limitations
+  under the License.
+-->
+
+# Parquet Audits
+
+> Audit notes for the native Parquet scan and the native Parquet writer. Absence of an entry means
+> the area has not been audited yet. Open issues are listed for users in
+> [Known Issues in Parquet and Iceberg Reads and Writes].
+
+## Native Parquet scan
+
+### Nested types (audited 2026-10-15)
+
+- **Scope:** structs, arrays and maps. NULL structs, arrays and maps, NULLs inside them, Parquet
+  field ids, case-insensitive names and read schemas that differ from the file's.
+- **Method:** about 3,000 queries compared with Spark with Comet disabled, on Spark 3.4.3, 3.5.9
+  and 4.1.3 at `main` 66c63bd307. A code review of DataFusion 55.2 and arrow-rs 59.3 ran alongside,
+  and each finding was reproduced before it was filed.
+- **Filed:** [#7029] (INT96 timestamps inside legacy-format lists), [#7030] (a nested column with
+  none of its requested fields in the file), [#7031] (Spark 3.4: a struct matched by field id under
+  a new name, under nested pruning), [#7032] (nested BINARY values that are not valid UTF-8, read as
+  STRING), [#7034] (`TIMESTAMP(NANOS)` read as `timestamp`), [#7035] (a nested encoding that
+  arrow-rs rejects fails the whole file), [#7036] and [#7037] (nested pruning gaps), and
+  [apache/datafusion#26393] upstream for [#7029]. All are tracked in [#7040].
+- **Matched Spark:** NULL structs, arrays and maps, empty arrays and maps, and NULLs inside them at
+  every level, in deep (eight levels) and wide (300 fields) structs. Spark-written files in five
+  layouts (default, tiny row groups and pages, the legacy format, v2 pages without dictionaries and
+  v2 tiny pages), read with batch sizes of 1 to 3, with row-filter pushdown on and off and with
+  filter pushdown off. Avro, Thrift, Hive and protobuf list and map encodings, maps with struct and
+  array keys, nested type promotion, and nested timestamp, decimal, boolean and binary leaves
+  (micros, millis, and INT96 in standard layouts). ENUM, JSON, BSON, unsigned and binary-decimal
+  annotations, files written by arrow-rs, partitioned tables, nested schemas that differ across the
+  files of one scan, case-insensitive nested names, nested field ids (renamed, missing and partly
+  present), and both modes of SPARK-53535 on Spark 4.1.
+- **Not Comet bugs:** with `spark.sql.parquet.fieldId.read.enabled=true` and a read schema that
+  swaps two nested field names, Spark's vectorized reader pairs the file's columns with the read
+  fields by the file's names, so it fails or returns no rows, while Comet resolves the fields by
+  id. On Spark 3.5 and later, nested schema pruning drops a root field id, so Spark reads a renamed
+  root as NULL, and Comet matches it there. A map with NULL keys from a non-Spark writer fails in
+  both engines.
+- **Not covered:** Variant ([#5438]), the V2 Parquet source, encrypted files, and the JVM-side
+  conversion paths.
+
+## Native Parquet writer
+
+### Nested types (audited 2026-10-16)
+
+- **Scope:** NULL structs, arrays and maps, NULLs inside them, Parquet field ids, column statistics,
+  and the file's schema and footer.
+- **Method:** each write made natively and by Spark with Comet disabled, on Spark 3.4.3, 3.5.9 and
+  4.1.3 at `main` c230bf4e98. The comparison covered the values read back by Spark and by Comet,
+  the Parquet schema with its repetition and field ids, the footer key-value metadata and the
+  statistics of every leaf. A code review of the writer and arrow-rs 59.3 ran alongside, and each
+  finding was reproduced before it was filed.
+- **Filed:** [#7047] (the datetime rebase modes are ignored, so Spark rejects or shifts the
+  writer's dates before 1582), [#7048] (`outputTimestampType`, `writeLegacyFormat` and Spark's
+  schema footer key), and a measurement on [#5304]: for nested rows the row-count cap gives one row
+  group per file, 239 MB for 100,000 rows of 100-element arrays, where Spark writes row groups of
+  about 128 MB. All are tracked in [#7051].
+- **Matched Spark:** values, repetition and field ids for NULL structs, arrays and maps, empty
+  arrays and maps, and NULLs inside them at every level, in deep and wide schemas. That includes
+  input sliced by `LIMIT` and `OFFSET`, filtered input, and nested values computed by expressions
+  before the write. Field ids at every level, with `spark.sql.parquet.fieldId.write.enabled` on and
+  off. Null counts, value counts, minimums and maximums for every leaf. Nested timestamp, decimal,
+  boolean and binary leaves.
+- **Did not reproduce:** a write of void (`NullType`) columns on Spark 3.4 and 3.5, which Spark
+  rejects at analysis before Comet plans the write. A write of ANSI interval columns, which did not
+  reach the native writer on any of the three versions. A task failure from the in-progress row
+  group outgrowing a 256 MB off-heap pool: the 239 MB row group fit.
+- **Not covered:** Variant ([#5433]), partitioned and bucketed writes, which the native writer does
+  not support ([#3193], [#3194]), object-store and HDFS outputs, and write throughput, which needs a
+  release build.
+
+[Known Issues in Parquet and Iceberg Reads and Writes]: ../../user-guide/latest/compatibility/parquet-iceberg-known-issues.md
+[apache/datafusion#26393]: https://github.com/apache/datafusion/issues/26393
+[#3193]: https://github.com/apache/datafusion-comet/issues/3193
+[#3194]: https://github.com/apache/datafusion-comet/issues/3194
+[#5304]: https://github.com/apache/datafusion-comet/issues/5304
+[#5433]: https://github.com/apache/datafusion-comet/issues/5433
+[#5438]: https://github.com/apache/datafusion-comet/issues/5438
+[#7029]: https://github.com/apache/datafusion-comet/issues/7029
+[#7030]: https://github.com/apache/datafusion-comet/issues/7030
+[#7031]: https://github.com/apache/datafusion-comet/issues/7031
+[#7032]: https://github.com/apache/datafusion-comet/issues/7032
+[#7034]: https://github.com/apache/datafusion-comet/issues/7034
+[#7035]: https://github.com/apache/datafusion-comet/issues/7035
+[#7036]: https://github.com/apache/datafusion-comet/issues/7036
+[#7037]: https://github.com/apache/datafusion-comet/issues/7037
+[#7040]: https://github.com/apache/datafusion-comet/issues/7040
+[#7047]: https://github.com/apache/datafusion-comet/issues/7047
+[#7048]: https://github.com/apache/datafusion-comet/issues/7048
+[#7051]: https://github.com/apache/datafusion-comet/issues/7051

docs/source/contributor-guide/index.md

@@ -95,6 +95,7 @@ Tracing <tracing>
 :hidden:
 
 Expression Audits <expression-audits/index>
+Data Source Audits <data-source-audits/index>
 Supported Spark Configurations <spark_configs_support>
 Configuration Conventions <config_conventions>
 ```

docs/source/user-guide/latest/compatibility/index.md

@@ -24,6 +24,8 @@ Comet aims to provide consistent results with the version of Apache Spark that i
 This guide documents areas where Comet's behavior is known to differ from Spark. Topics are grouped by subsystem:
 
 - **Parquet**: limitations when reading Parquet files.
+- **Parquet and Iceberg known issues**: open issues in which Parquet and Iceberg reads and writes
+  give different results from Spark, or fail where Spark succeeds, without falling back.
 - **Floating-point comparison**: NaN and signed-zero handling in comparisons.
 - **Regular expressions**: differences between the Rust regexp crate and Java's regex engine.
 - **Operators**: operator-level compatibility notes, including window functions and round-robin partitioning.
@@ -119,7 +121,9 @@ divergence:
 
 The following native paths silently return values that differ from Spark for edge-case inputs.
 Most also have entries in the per-category expression pages linked above; they are collected here
-so users hunting an unexpected value have a single place to check:
+so users hunting an unexpected value have a single place to check. Parquet and Iceberg reads and
+writes have their own list, in
+[Known Issues in Parquet and Iceberg Reads and Writes](parquet-iceberg-known-issues.md#results-that-differ-from-spark):
 
 - Native `RANGE` window frames with an explicit `PRECEDING` / `FOLLOWING` offset diverge from
   Spark when the boundary arithmetic overflows for `DATE` or `DECIMAL` `ORDER BY` columns