From 6fb68d108e4e614f48f9df488b9dbf6f0b02b2a2 Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Fri, 16 Oct 2026 09:02:19 -0600 Subject: [PATCH 1/4] docs: list known Parquet and Iceberg correctness issues and add a data source audit log Add a compatibility page that lists the open issues in which a Parquet or Iceberg read or write gives a different result from Spark without falling back, with links to the issues, and point to it from the compatibility overview, the Parquet scan page, the Iceberg guide and the Iceberg writes page. The datetime rebasing item moves from scans.md to the new page. Correct the Iceberg writes page's claim that the float/double null-count difference cannot change IS NULL pruning: iceberg-java's counts are the wrong ones (apache/iceberg#17561). Start a Data Source Audits section in the contributor guide, next to the expression audits, recording the nested-type audits of the native Parquet and Iceberg scans and writers. --- .../data-source-audits/iceberg.md | 106 +++++++++++ .../data-source-audits/index.md | 42 +++++ .../data-source-audits/parquet.md | 108 +++++++++++ docs/source/contributor-guide/index.md | 1 + .../user-guide/latest/compatibility/index.md | 6 +- .../parquet-iceberg-known-issues.md | 177 ++++++++++++++++++ .../user-guide/latest/compatibility/scans.md | 15 +- .../user-guide/latest/iceberg-writes.md | 13 +- docs/source/user-guide/latest/iceberg.md | 3 + docs/source/user-guide/latest/index.rst | 1 + 10 files changed, 457 insertions(+), 15 deletions(-) create mode 100644 docs/source/contributor-guide/data-source-audits/iceberg.md create mode 100644 docs/source/contributor-guide/data-source-audits/index.md create mode 100644 docs/source/contributor-guide/data-source-audits/parquet.md create mode 100644 docs/source/user-guide/latest/compatibility/parquet-iceberg-known-issues.md diff --git a/docs/source/contributor-guide/data-source-audits/iceberg.md b/docs/source/contributor-guide/data-source-audits/iceberg.md new file mode 100644 index 00000000000..912dd6d001a --- /dev/null +++ b/docs/source/contributor-guide/data-source-audits/iceberg.md @@ -0,0 +1,106 @@ + + +# Iceberg Audits + +> Audit notes for the native Iceberg scan and the native Iceberg writer. Absence of an entry means +> the area has not been audited yet. Open correctness issues are listed for users in +> [Known Correctness Issues in Parquet and Iceberg Reads and Writes]. + +## Native Iceberg scan + +### Nested types (audited 2026-10-15) + +- **Scope:** structs, arrays and maps. NULLs at every level, field ids, name mappings, schema + evolution, deletes and partitions. +- **Method:** queries compared with Spark with Comet disabled, on Spark 4.1.3 with Iceberg 1.11 at + `main` 66c63bd307. A code review of iceberg-rust at the current pin ran alongside, and each + finding was reproduced before it was filed. +- **Filed:** [#7033] (a data file that omits an optional nested field fails the scan), [#7038] + (residual predicates on struct fields are not pushed into iceberg-rust) and [#7039] (a file is + read in full when the query needs none of its columns). The audit also confirmed the open issues + [#6790], [#6794], [#6822], [#6993], [#6782], [#5540] and [#5257]. All are tracked in [#7040]. +- **Matched Spark:** the nested data of the Parquet scan audit written by iceberg-java, + merge-on-read position deletes across three-row pages (format v2, and v3 deletion vectors), + equality deletes including one with a NULL key, required nested fields under NULL parents, nested + UUID, fixed, timestamptz and timestamp_ntz leaves, nested type promotion inside structs, lists + and maps, the nested schema evolution fallback for lists and maps, `_partition` with NULL values + and spec evolution, and nested partition sources. +- **Did not reproduce:** reordered struct siblings of the same type swapping values when the column + holds a map, in more than 20 probes, CTAS included. +- **Not Comet bugs:** on a table identity-partitioned by a struct field, iceberg-java before 1.12 + reads a NULL struct as a struct of NULLs when the partition source is the struct's first field. + [apache/iceberg#17320] fixed it in 1.12.0. The native scan returns NULL, as 1.12 does. +- **Not covered:** Variant ([#5438]), encrypted tables, and Iceberg versions other than 1.11. + +## Native Iceberg writer + +### Nested types (audited 2026-10-16) + +- **Scope:** NULL structs, arrays and maps, NULLs inside them, field ids at every level, manifest + metrics, nested partition sources, sort orders, copy-on-write DML, and `NOT NULL` columns. +- **Method:** each write made natively and with Comet disabled, on Iceberg 1.5.2 (Spark 3.4.3), + 1.8.1 (Spark 3.5.9) and 1.11.0 (Spark 4.1.3) at `main` c230bf4e98. The comparison covered the + values read back, the manifest metrics of each data file (value, null and NaN counts, lower and + upper bounds), the Parquet footer, and the files pruned for nested predicates. A code review of + the writer, arrow-rs 59.3 and iceberg-rust ran alongside, and each finding was reproduced before + it was filed. +- **Filed:** [#7049] (writes into `NOT NULL` columns fall back on Spark 3.5 and later when no + exchange sits in front of the writer, because Comet has no native `AssertNotNull`) and [#7050] + (the user guide's claim about float and double null counts). Both are tracked in [#7051]. +- **Matched iceberg-java:** values, pruning, and field ids for struct fields, list elements and map + keys and values, on all three versions. Manifest metrics under the default and `full` metrics + modes on all three versions, and on 1.11.0 also under `counts`, `truncate(4)`, per-column + overrides on nested fields, `max-inferred-column-defaults`, and tiny pages and row groups. Nested + partition sources (identity on `s.a`, `bucket`, `truncate` and `days(ts.t)`), a sort order on a + nested field, and copy-on-write `DELETE`, `UPDATE` and `MERGE`. Required nested fields under NULL + parents written straight from a native scan, `fixed` fields under NULL structs, and copy-on-write + `UPDATE` with `AssertNotNull`, whose check still runs in Spark in front of the columnar shuffle. +- **Not Comet bugs:** on Iceberg 1.9 and later, the value and null counts of a float or double field + under a NULL struct differ from iceberg-java's ([#6659], closed), and iceberg-java's are the wrong + ones ([apache/iceberg#17561], with a fix in [apache/iceberg#17560]). With Comet disabled on + 1.11.0, `WHERE s.x IS NULL` returned 0 rows instead of 5, while 1.5.2 and 1.8.1 returned 5. +- **Did not reproduce:** a task failure from validating a required struct under an optional struct, + with a required child, when a native scan feeds the writer. A task failure from `fixed(16)` values + of the wrong length hidden under NULL structs. +- **Not measured:** the copy of every column that the writer makes when a float or double sits + under a list or map, to count NaNs correctly. +- **Not covered:** merge-on-read writes (`WriteDelta`), format v3 types, encryption, object stores, + and write throughput, which needs a release build. + +[Known Correctness Issues in Parquet and Iceberg Reads and Writes]: ../../user-guide/latest/compatibility/parquet-iceberg-known-issues.md +[apache/iceberg#17320]: https://github.com/apache/iceberg/pull/17320 +[apache/iceberg#17560]: https://github.com/apache/iceberg/pull/17560 +[apache/iceberg#17561]: https://github.com/apache/iceberg/issues/17561 +[#5257]: https://github.com/apache/datafusion-comet/issues/5257 +[#5438]: https://github.com/apache/datafusion-comet/issues/5438 +[#5540]: https://github.com/apache/datafusion-comet/issues/5540 +[#6659]: https://github.com/apache/datafusion-comet/issues/6659 +[#6782]: https://github.com/apache/datafusion-comet/issues/6782 +[#6790]: https://github.com/apache/datafusion-comet/issues/6790 +[#6794]: https://github.com/apache/datafusion-comet/issues/6794 +[#6822]: https://github.com/apache/datafusion-comet/issues/6822 +[#6993]: https://github.com/apache/datafusion-comet/issues/6993 +[#7033]: https://github.com/apache/datafusion-comet/issues/7033 +[#7038]: https://github.com/apache/datafusion-comet/issues/7038 +[#7039]: https://github.com/apache/datafusion-comet/issues/7039 +[#7040]: https://github.com/apache/datafusion-comet/issues/7040 +[#7049]: https://github.com/apache/datafusion-comet/issues/7049 +[#7050]: https://github.com/apache/datafusion-comet/issues/7050 +[#7051]: https://github.com/apache/datafusion-comet/issues/7051 diff --git a/docs/source/contributor-guide/data-source-audits/index.md b/docs/source/contributor-guide/data-source-audits/index.md new file mode 100644 index 00000000000..08eb6f0f763 --- /dev/null +++ b/docs/source/contributor-guide/data-source-audits/index.md @@ -0,0 +1,42 @@ + + +# Data Source Audits + +This section records audits of Comet's native data source paths, the scans and writers for Parquet +and Iceberg, in the same way that the [Expression Audits](../expression-audits/index.md) record +audits of expressions. Each page holds dated entries per component. An entry records: + +- what was audited, at which commit, against which Spark and library versions, and how +- the issues the audit filed, and the open issues it confirmed +- what was checked and matched, so that it is not re-audited without a reason +- the candidates that did not reproduce, so that they are not chased again +- the differences that are bugs in Spark or iceberg-java rather than in Comet +- what the audit did not cover + +A later audit of the same area adds a new dated entry instead of editing an old one, so the history +stays visible. The open correctness issues that users can hit are listed in the user guide, in +[Known Correctness Issues in Parquet and Iceberg Reads and Writes](../../user-guide/latest/compatibility/parquet-iceberg-known-issues.md). + +```{toctree} +:maxdepth: 1 + +Parquet +Iceberg +``` diff --git a/docs/source/contributor-guide/data-source-audits/parquet.md b/docs/source/contributor-guide/data-source-audits/parquet.md new file mode 100644 index 00000000000..0fe371b957b --- /dev/null +++ b/docs/source/contributor-guide/data-source-audits/parquet.md @@ -0,0 +1,108 @@ + + +# Parquet Audits + +> Audit notes for the native Parquet scan and the native Parquet writer. Absence of an entry means +> the area has not been audited yet. Open correctness issues are listed for users in +> [Known Correctness Issues in Parquet and Iceberg Reads and Writes]. + +## Native Parquet scan + +### Nested types (audited 2026-10-15) + +- **Scope:** structs, arrays and maps. NULL structs, arrays and maps, NULLs inside them, Parquet + field ids, case-insensitive names and read schemas that differ from the file's. +- **Method:** about 3,000 queries compared with Spark with Comet disabled, on Spark 3.4.3, 3.5.9 + and 4.1.3 at `main` 66c63bd307. A code review of DataFusion 55.2 and arrow-rs 59.3 ran alongside, + and each finding was reproduced before it was filed. +- **Filed:** [#7029] (INT96 timestamps inside legacy-format lists), [#7030] (a nested column with + none of its requested fields in the file), [#7031] (Spark 3.4: a struct matched by field id under + a new name, under nested pruning), [#7032] (nested BINARY values that are not valid UTF-8, read as + STRING), [#7034] (`TIMESTAMP(NANOS)` read as `timestamp`), [#7035] (a nested encoding that + arrow-rs rejects fails the whole file), [#7036] and [#7037] (nested pruning gaps), and + [apache/datafusion#26393] upstream for [#7029]. All are tracked in [#7040]. +- **Matched Spark:** NULL structs, arrays and maps, empty arrays and maps, and NULLs inside them at + every level, in deep (eight levels) and wide (300 fields) structs. Spark-written files in five + layouts (default, tiny row groups and pages, the legacy format, v2 pages without dictionaries and + v2 tiny pages), read with batch sizes of 1 to 3, with row-filter pushdown on and off and with + filter pushdown off. Avro, Thrift, Hive and protobuf list and map encodings, maps with struct and + array keys, nested type promotion, and nested timestamp, decimal, boolean and binary leaves + (micros, millis, and INT96 in standard layouts). ENUM, JSON, BSON, unsigned and binary-decimal + annotations, files written by arrow-rs, partitioned tables, nested schemas that differ across the + files of one scan, case-insensitive nested names, nested field ids (renamed, missing and partly + present), and both modes of SPARK-53535 on Spark 4.1. +- **Not Comet bugs:** with `spark.sql.parquet.fieldId.read.enabled=true` and a read schema that + swaps two nested field names, Spark's vectorized reader pairs the file's columns with the read + fields by the file's names, so it fails or returns no rows, while Comet resolves the fields by + id. On Spark 3.5 and later, nested schema pruning drops a root field id, so Spark reads a renamed + root as NULL, and Comet matches it there. A map with NULL keys from a non-Spark writer fails in + both engines. +- **Not covered:** Variant ([#5438]), the V2 Parquet source, encrypted files, and the JVM-side + conversion paths. + +## Native Parquet writer + +### Nested types (audited 2026-10-16) + +- **Scope:** NULL structs, arrays and maps, NULLs inside them, Parquet field ids, column statistics, + and the file's schema and footer. +- **Method:** each write made natively and by Spark with Comet disabled, on Spark 3.4.3, 3.5.9 and + 4.1.3 at `main` c230bf4e98. The comparison covered the values read back by Spark and by Comet, + the Parquet schema with its repetition and field ids, the footer key-value metadata and the + statistics of every leaf. A code review of the writer and arrow-rs 59.3 ran alongside, and each + finding was reproduced before it was filed. +- **Filed:** [#7047] (the datetime rebase modes are ignored, so Spark rejects or shifts the + writer's dates before 1582), [#7048] (`outputTimestampType`, `writeLegacyFormat` and Spark's + schema footer key), and a measurement on [#5304]: for nested rows the row-count cap gives one row + group per file, 239 MB for 100,000 rows of 100-element arrays, where Spark writes row groups of + about 128 MB. All are tracked in [#7051]. +- **Matched Spark:** values, repetition and field ids for NULL structs, arrays and maps, empty + arrays and maps, and NULLs inside them at every level, in deep and wide schemas. That includes + input sliced by `LIMIT` and `OFFSET`, filtered input, and nested values computed by expressions + before the write. Field ids at every level, with `spark.sql.parquet.fieldId.write.enabled` on and + off. Null counts, value counts, minimums and maximums for every leaf. Nested timestamp, decimal, + boolean and binary leaves. +- **Did not reproduce:** a write of void (`NullType`) columns on Spark 3.4 and 3.5, which Spark + rejects at analysis before Comet plans the write. A write of ANSI interval columns, which did not + reach the native writer on any of the three versions. A task failure from the in-progress row + group outgrowing a 256 MB off-heap pool: the 239 MB row group fit. +- **Not covered:** Variant ([#5433]), partitioned and bucketed writes, which the native writer does + not support ([#3193], [#3194]), object-store and HDFS outputs, and write throughput, which needs a + release build. + +[Known Correctness Issues in Parquet and Iceberg Reads and Writes]: ../../user-guide/latest/compatibility/parquet-iceberg-known-issues.md +[apache/datafusion#26393]: https://github.com/apache/datafusion/issues/26393 +[#3193]: https://github.com/apache/datafusion-comet/issues/3193 +[#3194]: https://github.com/apache/datafusion-comet/issues/3194 +[#5304]: https://github.com/apache/datafusion-comet/issues/5304 +[#5433]: https://github.com/apache/datafusion-comet/issues/5433 +[#5438]: https://github.com/apache/datafusion-comet/issues/5438 +[#7029]: https://github.com/apache/datafusion-comet/issues/7029 +[#7030]: https://github.com/apache/datafusion-comet/issues/7030 +[#7031]: https://github.com/apache/datafusion-comet/issues/7031 +[#7032]: https://github.com/apache/datafusion-comet/issues/7032 +[#7034]: https://github.com/apache/datafusion-comet/issues/7034 +[#7035]: https://github.com/apache/datafusion-comet/issues/7035 +[#7036]: https://github.com/apache/datafusion-comet/issues/7036 +[#7037]: https://github.com/apache/datafusion-comet/issues/7037 +[#7040]: https://github.com/apache/datafusion-comet/issues/7040 +[#7047]: https://github.com/apache/datafusion-comet/issues/7047 +[#7048]: https://github.com/apache/datafusion-comet/issues/7048 +[#7051]: https://github.com/apache/datafusion-comet/issues/7051 diff --git a/docs/source/contributor-guide/index.md b/docs/source/contributor-guide/index.md index 1abd8891d20..a773c1b2ed7 100644 --- a/docs/source/contributor-guide/index.md +++ b/docs/source/contributor-guide/index.md @@ -95,6 +95,7 @@ Tracing :hidden: Expression Audits +Data Source Audits Supported Spark Configurations Configuration Conventions ``` diff --git a/docs/source/user-guide/latest/compatibility/index.md b/docs/source/user-guide/latest/compatibility/index.md index a74daf64f75..635c94cf3f1 100644 --- a/docs/source/user-guide/latest/compatibility/index.md +++ b/docs/source/user-guide/latest/compatibility/index.md @@ -24,6 +24,8 @@ Comet aims to provide consistent results with the version of Apache Spark that i This guide documents areas where Comet's behavior is known to differ from Spark. Topics are grouped by subsystem: - **Parquet**: limitations when reading Parquet files. +- **Parquet and Iceberg known issues**: open issues in which Parquet and Iceberg reads and writes + give different results from Spark without falling back. - **Floating-point comparison**: NaN and signed-zero handling in comparisons. - **Regular expressions**: differences between the Rust regexp crate and Java's regex engine. - **Operators**: operator-level compatibility notes, including window functions and round-robin partitioning. @@ -119,7 +121,9 @@ divergence: The following native paths silently return values that differ from Spark for edge-case inputs. Most also have entries in the per-category expression pages linked above; they are collected here -so users hunting an unexpected value have a single place to check: +so users hunting an unexpected value have a single place to check. Parquet and Iceberg reads and +writes have their own list, in +[Known Correctness Issues in Parquet and Iceberg Reads and Writes](parquet-iceberg-known-issues.md): - Native `RANGE` window frames with an explicit `PRECEDING` / `FOLLOWING` offset diverge from Spark when the boundary arithmetic overflows for `DATE` or `DECIMAL` `ORDER BY` columns diff --git a/docs/source/user-guide/latest/compatibility/parquet-iceberg-known-issues.md b/docs/source/user-guide/latest/compatibility/parquet-iceberg-known-issues.md new file mode 100644 index 00000000000..186576398cc --- /dev/null +++ b/docs/source/user-guide/latest/compatibility/parquet-iceberg-known-issues.md @@ -0,0 +1,177 @@ + + +# Known Correctness Issues in Parquet and Iceberg Reads and Writes + +This page lists the open issues in which a Comet Parquet or Iceberg read or write gives a different +result from Spark without falling back to Spark: wrong values, a NULL where Spark returns a value or +the reverse, rows returned for a read that Spark rejects, and written files that Spark or other +engines read differently from the files Spark writes. Each entry links to the issue that tracks it. + +Queries that fail where Spark succeeds, and the cases where Comet falls back to Spark, are +documented in [Parquet Compatibility](scans.md), in the +[current limitations](../iceberg.md#current-limitations) of the Iceberg guide and in +[Iceberg Writes](../iceberg-writes.md). + +Which entries apply depends on which native paths are enabled: + +| Native path | Setting | Default | +| -------------- | ---------------------------------------- | ---------------------- | +| Parquet scan | `spark.comet.scan.enabled` | `true` | +| Iceberg scan | `spark.comet.scan.icebergNative.enabled` | `true` | +| Iceberg writer | `spark.comet.write.iceberg.enabled` | `false` | +| Parquet writer | `spark.comet.write.parquet.enabled` | `false` (testing only) | + +## Parquet reads + +- **Dates and timestamps written with the legacy calendar.** Comet does not rebase dates and + timestamps written with the hybrid Julian/Gregorian calendar, which Spark versions before 3.0 + wrote, and later versions write with `spark.sql.parquet.datetimeRebaseModeInWrite=LEGACY`. Comet + reads them as proleptic Gregorian values, so values before October 15, 1582 are wrong in both + projections and predicates. Comet also ignores `spark.sql.parquet.datetimeRebaseModeInRead` and + the file's `org.apache.spark.legacyDateTime` metadata, which would tell it to rebase + ([#5010](https://github.com/apache/datafusion-comet/issues/5010)). +- **INT96 timestamps inside legacy-format lists.** INT96 timestamps outside the years 1677 to 2262 + read back wrong inside lists written in a legacy layout, as Hive and + `spark.sql.parquet.writeLegacyFormat=true` write them + ([#7029](https://github.com/apache/datafusion-comet/issues/7029)). +- **A struct or array with none of its requested fields in the file.** Spark returns NULL for the + column, and Comet returns a non-null struct or array that holds NULLs. That is Spark's only + behavior on Spark 3.4, 3.5 and 4.0, and its behavior on Spark 4.1 and later with + `spark.sql.legacy.parquet.returnNullStructIfAllFieldsMissing=true` + ([#7030](https://github.com/apache/datafusion-comet/issues/7030)). +- **Nested BINARY values that are not valid UTF-8, read as STRING.** When a BINARY column without + the UTF8 annotation is read as STRING, because the read schema says `string` or because of + `spark.sql.parquet.binaryAsString=true`, Comet returns NULL for each nested value that is not + valid UTF-8 where Spark returns the bytes. Top-level values read this way skip UTF-8 validation + ([#7032](https://github.com/apache/datafusion-comet/issues/7032), part of + [#4764](https://github.com/apache/datafusion-comet/issues/4764)). +- **Spark 3.4: a struct matched by field id under a different name.** With + `spark.sql.parquet.fieldId.read.enabled=true`, a struct column that the read schema matches to + the file by field id but under another name reads as NULL when Spark prunes its nested fields + ([#7031](https://github.com/apache/datafusion-comet/issues/7031)). +- **Field-id reads in which one file column matches a field by id and a sibling by name.** With + `spark.sql.parquet.fieldId.read.enabled=true`, Comet returns NULL for the sibling at the top + level, and the values of the field matched by id inside a struct. Spark's own readers disagree + with each other here ([#6994](https://github.com/apache/datafusion-comet/issues/6994)). +- **`TIMESTAMP_MILLIS` values that overflow when converted to microseconds.** In nested fields, and + in every column of a scan that has data filters, Comet returns NULL where Spark fails with + `ArithmeticException: long overflow` + ([#5553](https://github.com/apache/datafusion-comet/issues/5553), + [#5739](https://github.com/apache/datafusion-comet/issues/5739)). +- **Reads that Spark rejects.** Comet returns rows where Spark fails the read for + `TIMESTAMP(NANOS)` columns read as `timestamp` or `timestamp_ntz` + ([#7034](https://github.com/apache/datafusion-comet/issues/7034)), `INT` columns read as + `INTERVAL DAY TO SECOND` ([#5188](https://github.com/apache/datafusion-comet/issues/5188)), and + Variant-annotated fields read as a plain struct + ([#5741](https://github.com/apache/datafusion-comet/issues/5741)). +- **Dynamic partition pruning under AQE.** Two exchanges whose scans differ only in their dynamic + partition pruning filter can compare equal, so AQE reuses the first for the second and the rows of + the second branch are lost. It shows when no coalesced shuffle read sits between the stages, as + with large shuffles or with partition coalescing off, and it applies to the native Iceberg scan + as well ([#6264](https://github.com/apache/datafusion-comet/issues/6264)). + +## Iceberg reads + +- **Migrated tables that use a name mapping.** For data files without Iceberg field ids that are + read through `schema.name-mapping.default`, which the `migrate`, `snapshot` and `add_files` + procedures set, the native scan returns NULL for nested fields (struct fields, list elements and + map values) whenever the query also reads a top-level primitive column, so filters on them select + the wrong rows ([#6790](https://github.com/apache/datafusion-comet/issues/6790)). The fix comes + with an iceberg-rust upgrade ([#6934](https://github.com/apache/datafusion-comet/issues/6934)). +- **Data files with neither field ids nor a name mapping.** Spark reads every nested field of such + a file as NULL, and the native scan returns the values stored in the file + ([#6794](https://github.com/apache/datafusion-comet/issues/6794)). +- **`VERSION AS OF` reads of a column renamed after the snapshot.** The renamed column reads as + NULL, and two columns that swapped names return each other's values + ([#6822](https://github.com/apache/datafusion-comet/issues/6822)). +- **Reflection failures while the scan is planned.** If Comet fails to read the table's schema or + metadata through reflection, it continues with an empty or partial result instead of falling + back, which can resolve a column to a different field id + ([#5257](https://github.com/apache/datafusion-comet/issues/5257)). + +## Parquet writes + +The native Parquet writer is experimental and disabled by default. With +`spark.comet.write.parquet.enabled=true`: + +- **The datetime rebase modes are ignored.** On Spark 3.4 and 3.5, whose default rebase mode is + `EXCEPTION`, the native writer writes dates and timestamps before October 15, 1582 that Spark + refuses to write, and Spark then fails to read them back under its default read mode. With + `spark.sql.parquet.datetimeRebaseModeInRead=LEGACY`, Spark reads them shifted by several days, on + every version. The files lack the `org.apache.spark.version` footer key that tells Spark they need + no rebasing ([#7047](https://github.com/apache/datafusion-comet/issues/7047), + [#3427](https://github.com/apache/datafusion-comet/issues/3427)). +- **`spark.sql.parquet.outputTimestampType` and `spark.sql.parquet.writeLegacyFormat` are + ignored.** Timestamps are always written as `INT64` microseconds, never as Spark's default + `INT96`, so with `TIMESTAMP_MILLIS` they keep the sub-millisecond digits that Spark drops. Spark's + schema footer key is not written either, so a path-based read loses column metadata such as + comments, and with `spark.sql.parquet.inferTimestampNTZ.enabled=false` it reads `TIMESTAMP_NTZ` + columns as `TIMESTAMP` ([#7048](https://github.com/apache/datafusion-comet/issues/7048), + [#3425](https://github.com/apache/datafusion-comet/issues/3425)). +- **Spark 3.4 and 3.5: `INSERT INTO` leaves the target stale.** The steps Spark runs after the + write (file index refresh, cache refresh, partition refresh and statistics update) are skipped, + so later queries can read the table's previous contents until `REFRESH TABLE` + ([#6315](https://github.com/apache/datafusion-comet/issues/6315)). +- **An input with no partitions writes no file.** Spark writes one file that holds the schema + ([#5303](https://github.com/apache/datafusion-comet/issues/5303)). +- **Spark SQL tests that still fail with the native writer**, not yet classified: CTAS schema + ([#3418](https://github.com/apache/datafusion-comet/issues/3418)), CTAS with `CHAR_AS_VARCHAR` + ([#3419](https://github.com/apache/datafusion-comet/issues/3419)), CTAS with a union + ([#3429](https://github.com/apache/datafusion-comet/issues/3429)), `INSERT INTO` with default + columns ([#3421](https://github.com/apache/datafusion-comet/issues/3421)) and after + `ALTER TABLE ADD COLUMNS` ([#3422](https://github.com/apache/datafusion-comet/issues/3422)), + `INSERT INTO` from non-projection queries + ([#3423](https://github.com/apache/datafusion-comet/issues/3423)), the `INSERT OVERWRITE` output + schema ([#3424](https://github.com/apache/datafusion-comet/issues/3424)), multiple inserts + ([#3430](https://github.com/apache/datafusion-comet/issues/3430)), + `INSERT OVERWRITE LOCAL DIRECTORY` + ([#3420](https://github.com/apache/datafusion-comet/issues/3420)), empty files + ([#3417](https://github.com/apache/datafusion-comet/issues/3417)) and `onTaskCommit` + ([#3428](https://github.com/apache/datafusion-comet/issues/3428)). + +## Iceberg writes + +The native Iceberg writer is disabled by default. With `spark.comet.write.iceberg.enabled=true`: + +- **Brotli tables.** A native write commits brotli-compressed files that no JVM Parquet reader can + read unless Hadoop's `BrotliCodec` is on its classpath, where iceberg-java fails the write + ([#6960](https://github.com/apache/datafusion-comet/issues/6960)). +- **`write.parquet.compression-codec=none`.** The native writer writes the table uncompressed, + where iceberg-java rejects the value. It also trims or lowercases a few other settings that + iceberg-java reads as written ([#7008](https://github.com/apache/datafusion-comet/issues/7008)). + +The physical differences that every native data file has, and why they are safe, are listed in +[Accepted divergences](../iceberg-writes.md#accepted-divergences). + +## Differences that are not Comet bugs + +In these cases Comet's result differs from Spark's, but the bug is in Spark or iceberg-java: + +- **iceberg-java 1.9 and later: `IS NULL` on a float or double field under a NULL struct.** + iceberg-java leaves the rows where the struct is NULL out of the field's null count, so a filter + such as `WHERE s.x IS NULL` can skip data files that hold matching rows, with or without Comet + ([apache/iceberg#17561](https://github.com/apache/iceberg/issues/17561)). Files written by Comet's + native Iceberg writer are not affected. +- **iceberg-java before 1.12: a NULL struct whose first field is an identity partition source.** + iceberg-java reads the struct as a struct of NULLs, and Comet returns NULL, as iceberg-java 1.12 + does ([apache/iceberg#17320](https://github.com/apache/iceberg/pull/17320)). +- **Field-id reads with a read schema that swaps two nested field names.** Spark's vectorized + reader pairs the file's columns with the read fields by the file's names, so it fails with a type + mismatch or returns no rows. Comet resolves the fields by id. diff --git a/docs/source/user-guide/latest/compatibility/scans.md b/docs/source/user-guide/latest/compatibility/scans.md index 7b7d2c198e8..7f827a275c7 100644 --- a/docs/source/user-guide/latest/compatibility/scans.md +++ b/docs/source/user-guide/latest/compatibility/scans.md @@ -55,16 +55,11 @@ The following features are not supported and cause Comet to fall back to Spark: `spark.sql.caseSensitive=false`. Spark's analyzer normally rejects such a schema before the scan is planned. -The following limitation may produce incorrect results without falling back to Spark: - -- No support for datetime rebasing. When reading Parquet files containing dates or timestamps - written with `spark.sql.parquet.datetimeRebaseModeInWrite=LEGACY` (which is Spark's default for - data written before Spark 3.0, using the hybrid Julian/Gregorian calendar), Comet reads them as - if they were written using the Proleptic Gregorian calendar. This produces silently-wrong - values for dates before October 15, 1582 in both projections and predicates. Comet also - ignores `spark.sql.parquet.datetimeRebaseModeInRead` and the file-level - `org.apache.spark.legacyDateTime` metadata that would tell it to rebase. Tracked by - [#5010](https://github.com/apache/datafusion-comet/issues/5010). +Some Parquet reads return results that differ from Spark without falling back to Spark, such as +reads of dates and timestamps written with the legacy hybrid Julian/Gregorian calendar +([#5010](https://github.com/apache/datafusion-comet/issues/5010)). +[Known Correctness Issues](parquet-iceberg-known-issues.md#parquet-reads) lists them, with the +issues that track them. The following limitations raise an error at scan time rather than falling back to Spark: diff --git a/docs/source/user-guide/latest/iceberg-writes.md b/docs/source/user-guide/latest/iceberg-writes.md index bba4ea8ec14..2b7fcfaebda 100644 --- a/docs/source/user-guide/latest/iceberg-writes.md +++ b/docs/source/user-guide/latest/iceberg-writes.md @@ -494,9 +494,14 @@ iceberg-java's _writer-tracked_ state would have recorded, and both are analyzed change pruning decisions. - On Iceberg 1.9+, manifest `value_counts` / `null_value_counts` for float/double columns nested under a nullable struct count rows whose parent struct is null (they come from the - parquet footer), while iceberg-java's writer-tracked counts do not. Both counts inflate by - the same amount, so the derived null ratios and `IS NULL` / `IS NOT NULL` pruning decisions - are unaffected. + parquet footer), while iceberg-java's writer-tracked counts do not. The native counts are the + correct ones. iceberg-java's leave out the rows where the struct is null, so when every + non-null struct in a file holds a value, iceberg-java records a null count of 0 and `IS NULL` + prunes the file even though its null-struct rows match + ([apache/iceberg#17561](https://github.com/apache/iceberg/issues/17561)). The native counts + never prune that file. `IS NOT NULL` pruning decisions are the same for both. All content not listed above — the logical data, statistics values, and manifest metadata — -must match iceberg-java exactly, or the write falls back. +must match iceberg-java exactly, or the write falls back. The open bugs in which a native write +still differs are listed in +[Known Correctness Issues](compatibility/parquet-iceberg-known-issues.md#iceberg-writes). diff --git a/docs/source/user-guide/latest/iceberg.md b/docs/source/user-guide/latest/iceberg.md index 24713a32b7d..a78d4e7e4ed 100644 --- a/docs/source/user-guide/latest/iceberg.md +++ b/docs/source/user-guide/latest/iceberg.md @@ -198,6 +198,9 @@ The following scenarios will fall back to the JVM Iceberg reader: change. The check uses the table's schema history, so the fallback stays after those files are rewritten +The cases in which the native reader returns different results from Spark without falling back +are listed in [Known Correctness Issues](compatibility/parquet-iceberg-known-issues.md#iceberg-reads). + Writes are not covered by this list. By default Spark plans Iceberg writes with its own V2 write operator, and iceberg-java writes the data files. With `spark.comet.write.iceberg.enabled=true`, Comet plans an Iceberg write with its split-operator plan and writes eligible data files natively; diff --git a/docs/source/user-guide/latest/index.rst b/docs/source/user-guide/latest/index.rst index 84b0064647d..b4cfe077ca8 100644 --- a/docs/source/user-guide/latest/index.rst +++ b/docs/source/user-guide/latest/index.rst @@ -60,6 +60,7 @@ the navigation menu to read more. Overview compatibility/scans + compatibility/parquet-iceberg-known-issues compatibility/floating-point compatibility/regex compatibility/operators From 968412acc16947b9a254ef01916d4c1c8e08254a Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Fri, 16 Oct 2026 09:18:53 -0600 Subject: [PATCH 2/4] docs: add Parquet and Iceberg queries that fail where Spark succeeds Add a section to the known issues page for the open issues in which a Parquet or Iceberg read or write fails with an error where Spark, or iceberg-java, succeeds: Parquet and Iceberg reads, object store and credential settings that the native clients resolve differently from Hadoop, and memory failures in the native writers. The page becomes "Known Issues in Parquet and Iceberg Reads and Writes". The existing sections move under "Results that differ from Spark" and keep their heading anchors, and the links from the other pages use the new title. --- .../data-source-audits/iceberg.md | 6 +- .../data-source-audits/index.md | 4 +- .../data-source-audits/parquet.md | 6 +- .../user-guide/latest/compatibility/index.md | 4 +- .../parquet-iceberg-known-issues.md | 111 ++++++++++++++++-- .../user-guide/latest/compatibility/scans.md | 6 +- .../user-guide/latest/iceberg-writes.md | 2 +- docs/source/user-guide/latest/iceberg.md | 5 +- 8 files changed, 117 insertions(+), 27 deletions(-) diff --git a/docs/source/contributor-guide/data-source-audits/iceberg.md b/docs/source/contributor-guide/data-source-audits/iceberg.md index 912dd6d001a..e0a8192ec80 100644 --- a/docs/source/contributor-guide/data-source-audits/iceberg.md +++ b/docs/source/contributor-guide/data-source-audits/iceberg.md @@ -20,8 +20,8 @@ # Iceberg Audits > Audit notes for the native Iceberg scan and the native Iceberg writer. Absence of an entry means -> the area has not been audited yet. Open correctness issues are listed for users in -> [Known Correctness Issues in Parquet and Iceberg Reads and Writes]. +> the area has not been audited yet. Open issues are listed for users in +> [Known Issues in Parquet and Iceberg Reads and Writes]. ## Native Iceberg scan @@ -84,7 +84,7 @@ - **Not covered:** merge-on-read writes (`WriteDelta`), format v3 types, encryption, object stores, and write throughput, which needs a release build. -[Known Correctness Issues in Parquet and Iceberg Reads and Writes]: ../../user-guide/latest/compatibility/parquet-iceberg-known-issues.md +[Known Issues in Parquet and Iceberg Reads and Writes]: ../../user-guide/latest/compatibility/parquet-iceberg-known-issues.md [apache/iceberg#17320]: https://github.com/apache/iceberg/pull/17320 [apache/iceberg#17560]: https://github.com/apache/iceberg/pull/17560 [apache/iceberg#17561]: https://github.com/apache/iceberg/issues/17561 diff --git a/docs/source/contributor-guide/data-source-audits/index.md b/docs/source/contributor-guide/data-source-audits/index.md index 08eb6f0f763..bc0df6d06a9 100644 --- a/docs/source/contributor-guide/data-source-audits/index.md +++ b/docs/source/contributor-guide/data-source-audits/index.md @@ -31,8 +31,8 @@ audits of expressions. Each page holds dated entries per component. An entry rec - what the audit did not cover A later audit of the same area adds a new dated entry instead of editing an old one, so the history -stays visible. The open correctness issues that users can hit are listed in the user guide, in -[Known Correctness Issues in Parquet and Iceberg Reads and Writes](../../user-guide/latest/compatibility/parquet-iceberg-known-issues.md). +stays visible. The open issues that users can hit are listed in the user guide, in +[Known Issues in Parquet and Iceberg Reads and Writes](../../user-guide/latest/compatibility/parquet-iceberg-known-issues.md). ```{toctree} :maxdepth: 1 diff --git a/docs/source/contributor-guide/data-source-audits/parquet.md b/docs/source/contributor-guide/data-source-audits/parquet.md index 0fe371b957b..ad7fad0efd4 100644 --- a/docs/source/contributor-guide/data-source-audits/parquet.md +++ b/docs/source/contributor-guide/data-source-audits/parquet.md @@ -20,8 +20,8 @@ # Parquet Audits > Audit notes for the native Parquet scan and the native Parquet writer. Absence of an entry means -> the area has not been audited yet. Open correctness issues are listed for users in -> [Known Correctness Issues in Parquet and Iceberg Reads and Writes]. +> the area has not been audited yet. Open issues are listed for users in +> [Known Issues in Parquet and Iceberg Reads and Writes]. ## Native Parquet scan @@ -87,7 +87,7 @@ not support ([#3193], [#3194]), object-store and HDFS outputs, and write throughput, which needs a release build. -[Known Correctness Issues in Parquet and Iceberg Reads and Writes]: ../../user-guide/latest/compatibility/parquet-iceberg-known-issues.md +[Known Issues in Parquet and Iceberg Reads and Writes]: ../../user-guide/latest/compatibility/parquet-iceberg-known-issues.md [apache/datafusion#26393]: https://github.com/apache/datafusion/issues/26393 [#3193]: https://github.com/apache/datafusion-comet/issues/3193 [#3194]: https://github.com/apache/datafusion-comet/issues/3194 diff --git a/docs/source/user-guide/latest/compatibility/index.md b/docs/source/user-guide/latest/compatibility/index.md index 635c94cf3f1..bc364e14a48 100644 --- a/docs/source/user-guide/latest/compatibility/index.md +++ b/docs/source/user-guide/latest/compatibility/index.md @@ -25,7 +25,7 @@ This guide documents areas where Comet's behavior is known to differ from Spark. - **Parquet**: limitations when reading Parquet files. - **Parquet and Iceberg known issues**: open issues in which Parquet and Iceberg reads and writes - give different results from Spark without falling back. + give different results from Spark, or fail where Spark succeeds, without falling back. - **Floating-point comparison**: NaN and signed-zero handling in comparisons. - **Regular expressions**: differences between the Rust regexp crate and Java's regex engine. - **Operators**: operator-level compatibility notes, including window functions and round-robin partitioning. @@ -123,7 +123,7 @@ The following native paths silently return values that differ from Spark for edg Most also have entries in the per-category expression pages linked above; they are collected here so users hunting an unexpected value have a single place to check. Parquet and Iceberg reads and writes have their own list, in -[Known Correctness Issues in Parquet and Iceberg Reads and Writes](parquet-iceberg-known-issues.md): +[Known Issues in Parquet and Iceberg Reads and Writes](parquet-iceberg-known-issues.md#results-that-differ-from-spark): - Native `RANGE` window frames with an explicit `PRECEDING` / `FOLLOWING` offset diverge from Spark when the boundary arithmetic overflows for `DATE` or `DECIMAL` `ORDER BY` columns diff --git a/docs/source/user-guide/latest/compatibility/parquet-iceberg-known-issues.md b/docs/source/user-guide/latest/compatibility/parquet-iceberg-known-issues.md index 186576398cc..b9af435d0e7 100644 --- a/docs/source/user-guide/latest/compatibility/parquet-iceberg-known-issues.md +++ b/docs/source/user-guide/latest/compatibility/parquet-iceberg-known-issues.md @@ -17,16 +17,18 @@ under the License. --> -# Known Correctness Issues in Parquet and Iceberg Reads and Writes +# Known Issues in Parquet and Iceberg Reads and Writes -This page lists the open issues in which a Comet Parquet or Iceberg read or write gives a different -result from Spark without falling back to Spark: wrong values, a NULL where Spark returns a value or -the reverse, rows returned for a read that Spark rejects, and written files that Spark or other -engines read differently from the files Spark writes. Each entry links to the issue that tracks it. +This page lists the open issues in Comet's Parquet and Iceberg reads and writes that a fallback to +Spark does not cover, with a link to the issue that tracks each one: -Queries that fail where Spark succeeds, and the cases where Comet falls back to Spark, are -documented in [Parquet Compatibility](scans.md), in the -[current limitations](../iceberg.md#current-limitations) of the Iceberg guide and in +- [Results that differ from Spark](#results-that-differ-from-spark): wrong values, a NULL where + Spark returns a value or the reverse, rows returned for a read that Spark rejects, and written + files that Spark or other engines read differently from the files Spark writes. +- [Queries that fail where Spark succeeds](#queries-that-fail-where-spark-succeeds). + +The cases where Comet falls back to Spark are documented in [Parquet Compatibility](scans.md), in +the [current limitations](../iceberg.md#current-limitations) of the Iceberg guide and in [Iceberg Writes](../iceberg-writes.md). Which entries apply depends on which native paths are enabled: @@ -38,7 +40,9 @@ Which entries apply depends on which native paths are enabled: | Iceberg writer | `spark.comet.write.iceberg.enabled` | `false` | | Parquet writer | `spark.comet.write.parquet.enabled` | `false` (testing only) | -## Parquet reads +## Results that differ from Spark + +### Parquet reads - **Dates and timestamps written with the legacy calendar.** Comet does not rebase dates and timestamps written with the hybrid Julian/Gregorian calendar, which Spark versions before 3.0 @@ -87,7 +91,7 @@ Which entries apply depends on which native paths are enabled: with large shuffles or with partition coalescing off, and it applies to the native Iceberg scan as well ([#6264](https://github.com/apache/datafusion-comet/issues/6264)). -## Iceberg reads +### Iceberg reads - **Migrated tables that use a name mapping.** For data files without Iceberg field ids that are read through `schema.name-mapping.default`, which the `migrate`, `snapshot` and `add_files` @@ -106,7 +110,7 @@ Which entries apply depends on which native paths are enabled: back, which can resolve a column to a different field id ([#5257](https://github.com/apache/datafusion-comet/issues/5257)). -## Parquet writes +### Parquet writes The native Parquet writer is experimental and disabled by default. With `spark.comet.write.parquet.enabled=true`: @@ -146,7 +150,7 @@ The native Parquet writer is experimental and disabled by default. With ([#3417](https://github.com/apache/datafusion-comet/issues/3417)) and `onTaskCommit` ([#3428](https://github.com/apache/datafusion-comet/issues/3428)). -## Iceberg writes +### Iceberg writes The native Iceberg writer is disabled by default. With `spark.comet.write.iceberg.enabled=true`: @@ -160,6 +164,89 @@ The native Iceberg writer is disabled by default. With `spark.comet.write.iceber The physical differences that every native data file has, and why they are safe, are listed in [Accepted divergences](../iceberg-writes.md#accepted-divergences). +## Queries that fail where Spark succeeds + +In these cases a Comet read or write fails with an error where Spark, or iceberg-java, succeeds. +Comet does not fall back to Spark for them. + +### Parquet reads that fail + +- **STRING columns that hold invalid UTF-8.** The read fails with + `Parquet error: encountered non UTF-8 data`, where Spark returns the bytes + ([#4121](https://github.com/apache/datafusion-comet/issues/4121), part of + [#4764](https://github.com/apache/datafusion-comet/issues/4764)). +- **Duplicate field names.** Selecting a field by name when several physical siblings match it, + including inside structs, arrays and maps, fails with a duplicate-field error + ([#5884](https://github.com/apache/datafusion-comet/issues/5884), + [#5964](https://github.com/apache/datafusion-comet/issues/5964)). + [Parquet Compatibility](scans.md) describes the reads this affects. +- **A duplicate field id in a column the query does not read.** With + `spark.sql.parquet.fieldId.read.enabled=true`, a file in which two root columns carry the same + field id fails the scan whenever the table schema has a field with that id, even when the query + does not read that field ([#7006](https://github.com/apache/datafusion-comet/issues/7006)). +- **A nested encoding that arrow-rs rejects.** A file fails to read when any column in its schema + uses such an encoding, even a column the query does not read + ([#7035](https://github.com/apache/datafusion-comet/issues/7035)). +- **A `DATE` column read as `INTERVAL YEAR TO MONTH`.** Spark reads it, and Comet fails the read + ([#5188](https://github.com/apache/datafusion-comet/issues/5188)). + +### Iceberg reads that fail + +- **A data file that omits an optional nested field.** The native scan fails on a data file that + lacks an optional struct field, which a write with `write.spark.accept-any-schema=true` can + produce without any schema change + ([#7033](https://github.com/apache/datafusion-comet/issues/7033)). +- **`VERSION AS OF` reads of a column whose type was promoted after the snapshot** + ([#6993](https://github.com/apache/datafusion-comet/issues/6993)). +- **An equality delete keyed on a nested field.** Every query the native scan runs on the table + fails with `field not found`, even one that reads the key + ([#6782](https://github.com/apache/datafusion-comet/issues/6782)). +- **Column names that only newer Unicode case data folds together**, such as a pair of Vithkuqi + letters, which Unicode 14 added. Queries on such a table fail with a duplicate-field error + ([#5540](https://github.com/apache/datafusion-comet/issues/5540)). + +### Object store and credential settings + +The native Parquet and Iceberg scans resolve storage settings with their own clients rather than +with Hadoop's. Where the two disagree, a native read can use a different identity or route than +Spark, and fail where Spark succeeds: + +- **S3 settings that Hadoop's S3A client resolves differently**, such as JCEKS-backed secrets and + the precedence of per-bucket settings. The native Parquet scan forwards them without checking + for a difference ([#5662](https://github.com/apache/datafusion-comet/issues/5662)). +- **AWS profile credential providers.** The native scans ignore `fs.s3a.auth.profile.name` and + `fs.s3a.auth.profile.file` ([#4245](https://github.com/apache/datafusion-comet/issues/4245)), + and resolve some profiles differently from Hadoop 3.4 and later, for example profiles that use + `credential_process` or chains of assumed roles + ([#6575](https://github.com/apache/datafusion-comet/issues/6575)). +- **Azure credentials.** Container-scoped OAuth settings, Fabric hosts and a few other + configurations resolve differently from Hadoop's ABFS driver + ([#6605](https://github.com/apache/datafusion-comet/issues/6605)). +- **`fs.comet.libhdfs.schemes` set only in the Hadoop configuration.** Comet's planner and its + native scan read the setting from different places, so they can route a scheme differently + ([#6837](https://github.com/apache/datafusion-comet/issues/6837)). +- **Iceberg tables whose metadata location has no host.** The native Iceberg scan skips the + credential provider configured with `s3.comet.credential.provider.class` + ([#6536](https://github.com/apache/datafusion-comet/issues/6536)). + +### Parquet writes that fail + +With `spark.comet.write.parquet.enabled=true`: + +- **Row groups that outgrow the memory pool.** The native writer caps row groups by row count + only and can't spill, so with wide or nested rows the row group it is building can outgrow the + off-heap pool and fail the task, where Spark's writer starts a new row group at + `parquet.block.size` ([#5304](https://github.com/apache/datafusion-comet/issues/5304)). + +### Iceberg writes that fail + +With `spark.comet.write.iceberg.enabled=true`: + +- **A write that keeps one file open and outgrows the off-heap pool.** An unpartitioned or + clustered write fails its task when its open file outgrows the pool, where iceberg-java writes + it on the JVM heap. [Failure handling](../iceberg-writes.md#failure-handling) describes the + error and the settings that let such a write fit. + ## Differences that are not Comet bugs In these cases Comet's result differs from Spark's, but the bug is in Spark or iceberg-java: diff --git a/docs/source/user-guide/latest/compatibility/scans.md b/docs/source/user-guide/latest/compatibility/scans.md index 7f827a275c7..ec6142713f2 100644 --- a/docs/source/user-guide/latest/compatibility/scans.md +++ b/docs/source/user-guide/latest/compatibility/scans.md @@ -58,10 +58,12 @@ The following features are not supported and cause Comet to fall back to Spark: Some Parquet reads return results that differ from Spark without falling back to Spark, such as reads of dates and timestamps written with the legacy hybrid Julian/Gregorian calendar ([#5010](https://github.com/apache/datafusion-comet/issues/5010)). -[Known Correctness Issues](parquet-iceberg-known-issues.md#parquet-reads) lists them, with the +[Known Issues](parquet-iceberg-known-issues.md#parquet-reads) lists them, with the issues that track them. -The following limitations raise an error at scan time rather than falling back to Spark: +The following limitations raise an error at scan time rather than falling back to Spark. More +reads that fail where Spark succeeds are listed in +[Known Issues](parquet-iceberg-known-issues.md#parquet-reads-that-fail): - Selecting a field by name when multiple physical siblings match, including inside structs, arrays, and maps. Comet raises a duplicate-field error instead of resolving the collision. diff --git a/docs/source/user-guide/latest/iceberg-writes.md b/docs/source/user-guide/latest/iceberg-writes.md index 2b7fcfaebda..bf87738811b 100644 --- a/docs/source/user-guide/latest/iceberg-writes.md +++ b/docs/source/user-guide/latest/iceberg-writes.md @@ -504,4 +504,4 @@ iceberg-java's _writer-tracked_ state would have recorded, and both are analyzed All content not listed above — the logical data, statistics values, and manifest metadata — must match iceberg-java exactly, or the write falls back. The open bugs in which a native write still differs are listed in -[Known Correctness Issues](compatibility/parquet-iceberg-known-issues.md#iceberg-writes). +[Known Issues](compatibility/parquet-iceberg-known-issues.md#iceberg-writes). diff --git a/docs/source/user-guide/latest/iceberg.md b/docs/source/user-guide/latest/iceberg.md index a78d4e7e4ed..0c26e16e68e 100644 --- a/docs/source/user-guide/latest/iceberg.md +++ b/docs/source/user-guide/latest/iceberg.md @@ -198,8 +198,9 @@ The following scenarios will fall back to the JVM Iceberg reader: change. The check uses the table's schema history, so the fallback stays after those files are rewritten -The cases in which the native reader returns different results from Spark without falling back -are listed in [Known Correctness Issues](compatibility/parquet-iceberg-known-issues.md#iceberg-reads). +The cases in which the native reader returns different results from Spark, or fails where Spark +succeeds, without falling back are listed in +[Known Issues](compatibility/parquet-iceberg-known-issues.md). Writes are not covered by this list. By default Spark plans Iceberg writes with its own V2 write operator, and iceberg-java writes the data files. With `spark.comet.write.iceberg.enabled=true`, From 9293815e09a41b4acdc7ca71d89c261da8cf7854 Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Fri, 16 Oct 2026 09:28:16 -0600 Subject: [PATCH 3/4] docs: drop the Spark 4.1 NullType read limitation parquet-rs accepts the UNKNOWN logical type on any physical type since apache/arrow-rs#9855, which the parquet 59 crate includes, so reading a Spark 4.1 Parquet file with a NullType column no longer fails (#4199). --- .../user-guide/latest/compatibility/spark-versions.md | 11 ----------- 1 file changed, 11 deletions(-) diff --git a/docs/source/user-guide/latest/compatibility/spark-versions.md b/docs/source/user-guide/latest/compatibility/spark-versions.md index 84baf6194fa..f178b679f94 100644 --- a/docs/source/user-guide/latest/compatibility/spark-versions.md +++ b/docs/source/user-guide/latest/compatibility/spark-versions.md @@ -69,17 +69,6 @@ Spark 4.0.4 is supported with Java 17/21 and Scala 2.13. Spark 4.1.3 is supported with Java 17/21 and Scala 2.13. -### Known Limitations - -- **`NullType` columns in Parquet files** - ([#4199](https://github.com/apache/datafusion-comet/issues/4199)): Spark encodes a `NullType` - column as a Parquet `BOOLEAN` physical type annotated with `LogicalType::Unknown`. The Rust - `parquet` crate that Comet depends on accepts `Unknown` only when paired with `INT32` and rejects - any other physical type with `Parquet error: Cannot annotate Unknown from BOOLEAN for field ''`. - Any attempt to read a Parquet file that contains a `NullType` column fails at decode time before - Comet's scan runs. Workaround: project the column away, cast it to a concrete type before - persisting, or read the file with Comet disabled for that query. - ## Spark 4.2 Spark 4.2.0 is supported with Java 17 and Scala 2.13. From 844e8a7cebefbcabf6f6906825750335c5f9cf99 Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Fri, 16 Oct 2026 19:45:32 -0600 Subject: [PATCH 4/4] docs: address review feedback on the known issues page Drop #6264 and #6837, which are fixed on main. Scope the datetime rebase entry to the native Parquet scan and say the Delta scan rebases. Add the ignored Parquet writer properties from #5304 and note that the row group failure did not reproduce. Explain how apache/iceberg#17320 covers the identity partition source case. Reword several entries, break the list of Spark SQL tests into one bullet each, and note that spark.comet.scan.enabled is a testing setting. --- .../data-source-audits/iceberg.md | 4 +- .../parquet-iceberg-known-issues.md | 124 +++++++++--------- .../user-guide/latest/iceberg-writes.md | 8 +- 3 files changed, 70 insertions(+), 66 deletions(-) diff --git a/docs/source/contributor-guide/data-source-audits/iceberg.md b/docs/source/contributor-guide/data-source-audits/iceberg.md index e0a8192ec80..41f89bd1e0c 100644 --- a/docs/source/contributor-guide/data-source-audits/iceberg.md +++ b/docs/source/contributor-guide/data-source-audits/iceberg.md @@ -46,7 +46,9 @@ holds a map, in more than 20 probes, CTAS included. - **Not Comet bugs:** on a table identity-partitioned by a struct field, iceberg-java before 1.12 reads a NULL struct as a struct of NULLs when the partition source is the struct's first field. - [apache/iceberg#17320] fixed it in 1.12.0. The native scan returns NULL, as 1.12 does. + [apache/iceberg#17320] fixed it in 1.12.0. Its title names nested initial defaults, but the + change also keeps a file column in the read when every projected field of a struct is a constant, + and an identity partition value is such a constant. The native scan returns NULL, as 1.12 does. - **Not covered:** Variant ([#5438]), encrypted tables, and Iceberg versions other than 1.11. ## Native Iceberg writer diff --git a/docs/source/user-guide/latest/compatibility/parquet-iceberg-known-issues.md b/docs/source/user-guide/latest/compatibility/parquet-iceberg-known-issues.md index b9af435d0e7..77a0a2501d9 100644 --- a/docs/source/user-guide/latest/compatibility/parquet-iceberg-known-issues.md +++ b/docs/source/user-guide/latest/compatibility/parquet-iceberg-known-issues.md @@ -19,8 +19,8 @@ # Known Issues in Parquet and Iceberg Reads and Writes -This page lists the open issues in Comet's Parquet and Iceberg reads and writes that a fallback to -Spark does not cover, with a link to the issue that tracks each one: +This page lists open issues in Comet's Parquet and Iceberg reads and writes where Comet does not +fall back to Spark. Each entry links to its tracking issue. The entries are in two sections: - [Results that differ from Spark](#results-that-differ-from-spark): wrong values, a NULL where Spark returns a value or the reverse, rows returned for a read that Spark rejects, and written @@ -40,40 +40,44 @@ Which entries apply depends on which native paths are enabled: | Iceberg writer | `spark.comet.write.iceberg.enabled` | `false` | | Parquet writer | `spark.comet.write.parquet.enabled` | `false` (testing only) | +`spark.comet.scan.enabled` is a testing setting that Comet's own test suites use. It is not meant +for production, so do not turn it off to avoid an entry on this page. + ## Results that differ from Spark ### Parquet reads -- **Dates and timestamps written with the legacy calendar.** Comet does not rebase dates and - timestamps written with the hybrid Julian/Gregorian calendar, which Spark versions before 3.0 - wrote, and later versions write with `spark.sql.parquet.datetimeRebaseModeInWrite=LEGACY`. Comet - reads them as proleptic Gregorian values, so values before October 15, 1582 are wrong in both - projections and predicates. Comet also ignores `spark.sql.parquet.datetimeRebaseModeInRead` and - the file's `org.apache.spark.legacyDateTime` metadata, which would tell it to rebase - ([#5010](https://github.com/apache/datafusion-comet/issues/5010)). +- **Dates and timestamps written with the legacy calendar.** Spark before 3.0, and later versions + with `spark.sql.parquet.datetimeRebaseModeInWrite=LEGACY`, write these values in the hybrid + Julian/Gregorian calendar. The native Parquet scan reads them as proleptic Gregorian, so values + before October 15, 1582 are wrong in projections and predicates. It ignores + `spark.sql.parquet.datetimeRebaseModeInRead` and the file's `org.apache.spark.legacyDateTime` + metadata ([#5010](https://github.com/apache/datafusion-comet/issues/5010)). The + [Delta scan](../delta.md) rebases. - **INT96 timestamps inside legacy-format lists.** INT96 timestamps outside the years 1677 to 2262 read back wrong inside lists written in a legacy layout, as Hive and `spark.sql.parquet.writeLegacyFormat=true` write them ([#7029](https://github.com/apache/datafusion-comet/issues/7029)). - **A struct or array with none of its requested fields in the file.** Spark returns NULL for the - column, and Comet returns a non-null struct or array that holds NULLs. That is Spark's only - behavior on Spark 3.4, 3.5 and 4.0, and its behavior on Spark 4.1 and later with + column. Comet returns a non-null struct or array that holds NULLs. This applies to Spark 3.4, 3.5 + and 4.0, and to Spark 4.1 and later with `spark.sql.legacy.parquet.returnNullStructIfAllFieldsMissing=true` ([#7030](https://github.com/apache/datafusion-comet/issues/7030)). -- **Nested BINARY values that are not valid UTF-8, read as STRING.** When a BINARY column without - the UTF8 annotation is read as STRING, because the read schema says `string` or because of - `spark.sql.parquet.binaryAsString=true`, Comet returns NULL for each nested value that is not - valid UTF-8 where Spark returns the bytes. Top-level values read this way skip UTF-8 validation +- **Nested BINARY values that are not valid UTF-8, read as STRING.** This applies to a BINARY + column without the UTF8 annotation that is read as `string`, through the read schema or + `spark.sql.parquet.binaryAsString=true`. Spark returns the bytes. Comet returns NULL for each + such nested value. Top-level values match Spark in a release build, because the invalid bytes + reach an Arrow string array unchecked. A debug build panics on them ([#7032](https://github.com/apache/datafusion-comet/issues/7032), part of [#4764](https://github.com/apache/datafusion-comet/issues/4764)). - **Spark 3.4: a struct matched by field id under a different name.** With `spark.sql.parquet.fieldId.read.enabled=true`, a struct column that the read schema matches to the file by field id but under another name reads as NULL when Spark prunes its nested fields ([#7031](https://github.com/apache/datafusion-comet/issues/7031)). -- **Field-id reads in which one file column matches a field by id and a sibling by name.** With +- **Field-id reads where one file column matches a field by id and a sibling by name.** With `spark.sql.parquet.fieldId.read.enabled=true`, Comet returns NULL for the sibling at the top - level, and the values of the field matched by id inside a struct. Spark's own readers disagree - with each other here ([#6994](https://github.com/apache/datafusion-comet/issues/6994)). + level. Inside a struct, it returns the values of the field matched by id. Spark's own readers + disagree here ([#6994](https://github.com/apache/datafusion-comet/issues/6994)). - **`TIMESTAMP_MILLIS` values that overflow when converted to microseconds.** In nested fields, and in every column of a scan that has data filters, Comet returns NULL where Spark fails with `ArithmeticException: long overflow` @@ -85,20 +89,15 @@ Which entries apply depends on which native paths are enabled: `INTERVAL DAY TO SECOND` ([#5188](https://github.com/apache/datafusion-comet/issues/5188)), and Variant-annotated fields read as a plain struct ([#5741](https://github.com/apache/datafusion-comet/issues/5741)). -- **Dynamic partition pruning under AQE.** Two exchanges whose scans differ only in their dynamic - partition pruning filter can compare equal, so AQE reuses the first for the second and the rows of - the second branch are lost. It shows when no coalesced shuffle read sits between the stages, as - with large shuffles or with partition coalescing off, and it applies to the native Iceberg scan - as well ([#6264](https://github.com/apache/datafusion-comet/issues/6264)). ### Iceberg reads -- **Migrated tables that use a name mapping.** For data files without Iceberg field ids that are - read through `schema.name-mapping.default`, which the `migrate`, `snapshot` and `add_files` - procedures set, the native scan returns NULL for nested fields (struct fields, list elements and - map values) whenever the query also reads a top-level primitive column, so filters on them select - the wrong rows ([#6790](https://github.com/apache/datafusion-comet/issues/6790)). The fix comes - with an iceberg-rust upgrade ([#6934](https://github.com/apache/datafusion-comet/issues/6934)). +- **Migrated tables that use a name mapping.** The `migrate`, `snapshot` and `add_files` + procedures set `schema.name-mapping.default` for files without Iceberg field ids. When a query + also reads a top-level primitive column, the native scan returns NULL for nested fields (struct + fields, list elements and map values). Filters on those fields then select the wrong rows + ([#6790](https://github.com/apache/datafusion-comet/issues/6790)). The fix needs an iceberg-rust + upgrade ([#6934](https://github.com/apache/datafusion-comet/issues/6934)). - **Data files with neither field ids nor a name mapping.** Spark reads every nested field of such a file as NULL, and the native scan returns the values stored in the file ([#6794](https://github.com/apache/datafusion-comet/issues/6794)). @@ -129,37 +128,39 @@ The native Parquet writer is experimental and disabled by default. With comments, and with `spark.sql.parquet.inferTimestampNTZ.enabled=false` it reads `TIMESTAMP_NTZ` columns as `TIMESTAMP` ([#7048](https://github.com/apache/datafusion-comet/issues/7048), [#3425](https://github.com/apache/datafusion-comet/issues/3425)). -- **Spark 3.4 and 3.5: `INSERT INTO` leaves the target stale.** The steps Spark runs after the - write (file index refresh, cache refresh, partition refresh and statistics update) are skipped, - so later queries can read the table's previous contents until `REFRESH TABLE` +- **Spark 3.4 and 3.5: `INSERT INTO` leaves the target stale.** Comet skips the steps Spark runs + after a write: file index refresh, cache refresh, partition refresh and statistics update. Later + queries can return the table's previous contents until `REFRESH TABLE` ([#6315](https://github.com/apache/datafusion-comet/issues/6315)). +- **Spark's Parquet writer properties are ignored.** `parquet.block.size`, page size, dictionary, + writer version and statistics settings never reach the native writer + ([#5304](https://github.com/apache/datafusion-comet/issues/5304)). - **An input with no partitions writes no file.** Spark writes one file that holds the schema ([#5303](https://github.com/apache/datafusion-comet/issues/5303)). -- **Spark SQL tests that still fail with the native writer**, not yet classified: CTAS schema - ([#3418](https://github.com/apache/datafusion-comet/issues/3418)), CTAS with `CHAR_AS_VARCHAR` - ([#3419](https://github.com/apache/datafusion-comet/issues/3419)), CTAS with a union - ([#3429](https://github.com/apache/datafusion-comet/issues/3429)), `INSERT INTO` with default - columns ([#3421](https://github.com/apache/datafusion-comet/issues/3421)) and after - `ALTER TABLE ADD COLUMNS` ([#3422](https://github.com/apache/datafusion-comet/issues/3422)), - `INSERT INTO` from non-projection queries - ([#3423](https://github.com/apache/datafusion-comet/issues/3423)), the `INSERT OVERWRITE` output - schema ([#3424](https://github.com/apache/datafusion-comet/issues/3424)), multiple inserts - ([#3430](https://github.com/apache/datafusion-comet/issues/3430)), - `INSERT OVERWRITE LOCAL DIRECTORY` - ([#3420](https://github.com/apache/datafusion-comet/issues/3420)), empty files - ([#3417](https://github.com/apache/datafusion-comet/issues/3417)) and `onTaskCommit` - ([#3428](https://github.com/apache/datafusion-comet/issues/3428)). +- **Spark SQL tests that still fail with the native writer**, not yet classified: + - CTAS schema ([#3418](https://github.com/apache/datafusion-comet/issues/3418)) + - CTAS with `CHAR_AS_VARCHAR` ([#3419](https://github.com/apache/datafusion-comet/issues/3419)) + - CTAS with a union ([#3429](https://github.com/apache/datafusion-comet/issues/3429)) + - `INSERT INTO` with default columns ([#3421](https://github.com/apache/datafusion-comet/issues/3421)) + - `INSERT INTO` after `ALTER TABLE ADD COLUMNS` ([#3422](https://github.com/apache/datafusion-comet/issues/3422)) + - `INSERT INTO` from non-projection queries ([#3423](https://github.com/apache/datafusion-comet/issues/3423)) + - the `INSERT OVERWRITE` output schema ([#3424](https://github.com/apache/datafusion-comet/issues/3424)) + - multiple inserts ([#3430](https://github.com/apache/datafusion-comet/issues/3430)) + - `INSERT OVERWRITE LOCAL DIRECTORY` ([#3420](https://github.com/apache/datafusion-comet/issues/3420)) + - empty files ([#3417](https://github.com/apache/datafusion-comet/issues/3417)) + - `onTaskCommit` ([#3428](https://github.com/apache/datafusion-comet/issues/3428)) ### Iceberg writes The native Iceberg writer is disabled by default. With `spark.comet.write.iceberg.enabled=true`: -- **Brotli tables.** A native write commits brotli-compressed files that no JVM Parquet reader can - read unless Hadoop's `BrotliCodec` is on its classpath, where iceberg-java fails the write - ([#6960](https://github.com/apache/datafusion-comet/issues/6960)). -- **`write.parquet.compression-codec=none`.** The native writer writes the table uncompressed, - where iceberg-java rejects the value. It also trims or lowercases a few other settings that - iceberg-java reads as written ([#7008](https://github.com/apache/datafusion-comet/issues/7008)). +- **Brotli tables.** The native writer commits brotli-compressed files. JVM Parquet readers can + read them only if Hadoop's `BrotliCodec` is on the classpath. iceberg-java fails the write + instead ([#6960](https://github.com/apache/datafusion-comet/issues/6960)). +- **`write.parquet.compression-codec=none`.** The native writer writes the table uncompressed, and + iceberg-java rejects the value. The native writer also trims or lowercases a few other settings + that iceberg-java reads as written + ([#7008](https://github.com/apache/datafusion-comet/issues/7008)). The physical differences that every native data file has, and why they are safe, are listed in [Accepted divergences](../iceberg-writes.md#accepted-divergences). @@ -180,7 +181,7 @@ Comet does not fall back to Spark for them. ([#5884](https://github.com/apache/datafusion-comet/issues/5884), [#5964](https://github.com/apache/datafusion-comet/issues/5964)). [Parquet Compatibility](scans.md) describes the reads this affects. -- **A duplicate field id in a column the query does not read.** With +- **A duplicate field id in a column the query does not read.** With a valid read schema and `spark.sql.parquet.fieldId.read.enabled=true`, a file in which two root columns carry the same field id fails the scan whenever the table schema has a field with that id, even when the query does not read that field ([#7006](https://github.com/apache/datafusion-comet/issues/7006)). @@ -222,9 +223,6 @@ Spark, and fail where Spark succeeds: - **Azure credentials.** Container-scoped OAuth settings, Fabric hosts and a few other configurations resolve differently from Hadoop's ABFS driver ([#6605](https://github.com/apache/datafusion-comet/issues/6605)). -- **`fs.comet.libhdfs.schemes` set only in the Hadoop configuration.** Comet's planner and its - native scan read the setting from different places, so they can route a scheme differently - ([#6837](https://github.com/apache/datafusion-comet/issues/6837)). - **Iceberg tables whose metadata location has no host.** The native Iceberg scan skips the credential provider configured with `s3.comet.credential.provider.class` ([#6536](https://github.com/apache/datafusion-comet/issues/6536)). @@ -233,10 +231,11 @@ Spark, and fail where Spark succeeds: With `spark.comet.write.parquet.enabled=true`: -- **Row groups that outgrow the memory pool.** The native writer caps row groups by row count - only and can't spill, so with wide or nested rows the row group it is building can outgrow the - off-heap pool and fail the task, where Spark's writer starts a new row group at - `parquet.block.size` ([#5304](https://github.com/apache/datafusion-comet/issues/5304)). +- **Row groups that outgrow the memory pool.** The native writer cuts row groups by row count + only, where Spark uses `parquet.block.size`, and it cannot spill. With wide or nested rows, the + open row group can outgrow the off-heap pool and fail the task. The audit did not reproduce a + failure, because a 239 MB row group fit in a 256 MB pool + ([#5304](https://github.com/apache/datafusion-comet/issues/5304)). ### Iceberg writes that fail @@ -258,7 +257,10 @@ In these cases Comet's result differs from Spark's, but the bug is in Spark or i native Iceberg writer are not affected. - **iceberg-java before 1.12: a NULL struct whose first field is an identity partition source.** iceberg-java reads the struct as a struct of NULLs, and Comet returns NULL, as iceberg-java 1.12 - does ([apache/iceberg#17320](https://github.com/apache/iceberg/pull/17320)). + does. [apache/iceberg#17320](https://github.com/apache/iceberg/pull/17320) is titled for nested + initial defaults, but it fixes this case too: it keeps one file column in the read, so the + struct's definition level survives when every projected field is a constant, and an identity + partition value is such a constant. - **Field-id reads with a read schema that swaps two nested field names.** Spark's vectorized reader pairs the file's columns with the read fields by the file's names, so it fails with a type mismatch or returns no rows. Comet resolves the fields by id. diff --git a/docs/source/user-guide/latest/iceberg-writes.md b/docs/source/user-guide/latest/iceberg-writes.md index bf87738811b..840916411e4 100644 --- a/docs/source/user-guide/latest/iceberg-writes.md +++ b/docs/source/user-guide/latest/iceberg-writes.md @@ -495,11 +495,11 @@ iceberg-java's _writer-tracked_ state would have recorded, and both are analyzed - On Iceberg 1.9+, manifest `value_counts` / `null_value_counts` for float/double columns nested under a nullable struct count rows whose parent struct is null (they come from the parquet footer), while iceberg-java's writer-tracked counts do not. The native counts are the - correct ones. iceberg-java's leave out the rows where the struct is null, so when every - non-null struct in a file holds a value, iceberg-java records a null count of 0 and `IS NULL` - prunes the file even though its null-struct rows match + correct ones. iceberg-java's counts leave out rows whose struct is null. When every non-null + struct in a file holds a value, iceberg-java records a null count of 0, and `IS NULL` prunes a + file that contains matching rows ([apache/iceberg#17561](https://github.com/apache/iceberg/issues/17561)). The native counts - never prune that file. `IS NOT NULL` pruning decisions are the same for both. + never prune such a file. `IS NOT NULL` pruning is the same for both. All content not listed above — the logical data, statistics values, and manifest metadata — must match iceberg-java exactly, or the write falls back. The open bugs in which a native write