diff --git a/docs/source/contributor-guide/data-source-audits/iceberg.md b/docs/source/contributor-guide/data-source-audits/iceberg.md new file mode 100644 index 00000000000..41f89bd1e0c --- /dev/null +++ b/docs/source/contributor-guide/data-source-audits/iceberg.md @@ -0,0 +1,108 @@ + + +# Iceberg Audits + +> Audit notes for the native Iceberg scan and the native Iceberg writer. Absence of an entry means +> the area has not been audited yet. Open issues are listed for users in +> [Known Issues in Parquet and Iceberg Reads and Writes]. + +## Native Iceberg scan + +### Nested types (audited 2026-10-15) + +- **Scope:** structs, arrays and maps. NULLs at every level, field ids, name mappings, schema + evolution, deletes and partitions. +- **Method:** queries compared with Spark with Comet disabled, on Spark 4.1.3 with Iceberg 1.11 at + `main` 66c63bd307. A code review of iceberg-rust at the current pin ran alongside, and each + finding was reproduced before it was filed. +- **Filed:** [#7033] (a data file that omits an optional nested field fails the scan), [#7038] + (residual predicates on struct fields are not pushed into iceberg-rust) and [#7039] (a file is + read in full when the query needs none of its columns). The audit also confirmed the open issues + [#6790], [#6794], [#6822], [#6993], [#6782], [#5540] and [#5257]. All are tracked in [#7040]. +- **Matched Spark:** the nested data of the Parquet scan audit written by iceberg-java, + merge-on-read position deletes across three-row pages (format v2, and v3 deletion vectors), + equality deletes including one with a NULL key, required nested fields under NULL parents, nested + UUID, fixed, timestamptz and timestamp_ntz leaves, nested type promotion inside structs, lists + and maps, the nested schema evolution fallback for lists and maps, `_partition` with NULL values + and spec evolution, and nested partition sources. +- **Did not reproduce:** reordered struct siblings of the same type swapping values when the column + holds a map, in more than 20 probes, CTAS included. +- **Not Comet bugs:** on a table identity-partitioned by a struct field, iceberg-java before 1.12 + reads a NULL struct as a struct of NULLs when the partition source is the struct's first field. + [apache/iceberg#17320] fixed it in 1.12.0. Its title names nested initial defaults, but the + change also keeps a file column in the read when every projected field of a struct is a constant, + and an identity partition value is such a constant. The native scan returns NULL, as 1.12 does. +- **Not covered:** Variant ([#5438]), encrypted tables, and Iceberg versions other than 1.11. + +## Native Iceberg writer + +### Nested types (audited 2026-10-16) + +- **Scope:** NULL structs, arrays and maps, NULLs inside them, field ids at every level, manifest + metrics, nested partition sources, sort orders, copy-on-write DML, and `NOT NULL` columns. +- **Method:** each write made natively and with Comet disabled, on Iceberg 1.5.2 (Spark 3.4.3), + 1.8.1 (Spark 3.5.9) and 1.11.0 (Spark 4.1.3) at `main` c230bf4e98. The comparison covered the + values read back, the manifest metrics of each data file (value, null and NaN counts, lower and + upper bounds), the Parquet footer, and the files pruned for nested predicates. A code review of + the writer, arrow-rs 59.3 and iceberg-rust ran alongside, and each finding was reproduced before + it was filed. +- **Filed:** [#7049] (writes into `NOT NULL` columns fall back on Spark 3.5 and later when no + exchange sits in front of the writer, because Comet has no native `AssertNotNull`) and [#7050] + (the user guide's claim about float and double null counts). Both are tracked in [#7051]. +- **Matched iceberg-java:** values, pruning, and field ids for struct fields, list elements and map + keys and values, on all three versions. Manifest metrics under the default and `full` metrics + modes on all three versions, and on 1.11.0 also under `counts`, `truncate(4)`, per-column + overrides on nested fields, `max-inferred-column-defaults`, and tiny pages and row groups. Nested + partition sources (identity on `s.a`, `bucket`, `truncate` and `days(ts.t)`), a sort order on a + nested field, and copy-on-write `DELETE`, `UPDATE` and `MERGE`. Required nested fields under NULL + parents written straight from a native scan, `fixed` fields under NULL structs, and copy-on-write + `UPDATE` with `AssertNotNull`, whose check still runs in Spark in front of the columnar shuffle. +- **Not Comet bugs:** on Iceberg 1.9 and later, the value and null counts of a float or double field + under a NULL struct differ from iceberg-java's ([#6659], closed), and iceberg-java's are the wrong + ones ([apache/iceberg#17561], with a fix in [apache/iceberg#17560]). With Comet disabled on + 1.11.0, `WHERE s.x IS NULL` returned 0 rows instead of 5, while 1.5.2 and 1.8.1 returned 5. +- **Did not reproduce:** a task failure from validating a required struct under an optional struct, + with a required child, when a native scan feeds the writer. A task failure from `fixed(16)` values + of the wrong length hidden under NULL structs. +- **Not measured:** the copy of every column that the writer makes when a float or double sits + under a list or map, to count NaNs correctly. +- **Not covered:** merge-on-read writes (`WriteDelta`), format v3 types, encryption, object stores, + and write throughput, which needs a release build. + +[Known Issues in Parquet and Iceberg Reads and Writes]: ../../user-guide/latest/compatibility/parquet-iceberg-known-issues.md +[apache/iceberg#17320]: https://github.com/apache/iceberg/pull/17320 +[apache/iceberg#17560]: https://github.com/apache/iceberg/pull/17560 +[apache/iceberg#17561]: https://github.com/apache/iceberg/issues/17561 +[#5257]: https://github.com/apache/datafusion-comet/issues/5257 +[#5438]: https://github.com/apache/datafusion-comet/issues/5438 +[#5540]: https://github.com/apache/datafusion-comet/issues/5540 +[#6659]: https://github.com/apache/datafusion-comet/issues/6659 +[#6782]: https://github.com/apache/datafusion-comet/issues/6782 +[#6790]: https://github.com/apache/datafusion-comet/issues/6790 +[#6794]: https://github.com/apache/datafusion-comet/issues/6794 +[#6822]: https://github.com/apache/datafusion-comet/issues/6822 +[#6993]: https://github.com/apache/datafusion-comet/issues/6993 +[#7033]: https://github.com/apache/datafusion-comet/issues/7033 +[#7038]: https://github.com/apache/datafusion-comet/issues/7038 +[#7039]: https://github.com/apache/datafusion-comet/issues/7039 +[#7040]: https://github.com/apache/datafusion-comet/issues/7040 +[#7049]: https://github.com/apache/datafusion-comet/issues/7049 +[#7050]: https://github.com/apache/datafusion-comet/issues/7050 +[#7051]: https://github.com/apache/datafusion-comet/issues/7051 diff --git a/docs/source/contributor-guide/data-source-audits/index.md b/docs/source/contributor-guide/data-source-audits/index.md new file mode 100644 index 00000000000..bc0df6d06a9 --- /dev/null +++ b/docs/source/contributor-guide/data-source-audits/index.md @@ -0,0 +1,42 @@ + + +# Data Source Audits + +This section records audits of Comet's native data source paths, the scans and writers for Parquet +and Iceberg, in the same way that the [Expression Audits](../expression-audits/index.md) record +audits of expressions. Each page holds dated entries per component. An entry records: + +- what was audited, at which commit, against which Spark and library versions, and how +- the issues the audit filed, and the open issues it confirmed +- what was checked and matched, so that it is not re-audited without a reason +- the candidates that did not reproduce, so that they are not chased again +- the differences that are bugs in Spark or iceberg-java rather than in Comet +- what the audit did not cover + +A later audit of the same area adds a new dated entry instead of editing an old one, so the history +stays visible. The open issues that users can hit are listed in the user guide, in +[Known Issues in Parquet and Iceberg Reads and Writes](../../user-guide/latest/compatibility/parquet-iceberg-known-issues.md). + +```{toctree} +:maxdepth: 1 + +Parquet +Iceberg +``` diff --git a/docs/source/contributor-guide/data-source-audits/parquet.md b/docs/source/contributor-guide/data-source-audits/parquet.md new file mode 100644 index 00000000000..ad7fad0efd4 --- /dev/null +++ b/docs/source/contributor-guide/data-source-audits/parquet.md @@ -0,0 +1,108 @@ + + +# Parquet Audits + +> Audit notes for the native Parquet scan and the native Parquet writer. Absence of an entry means +> the area has not been audited yet. Open issues are listed for users in +> [Known Issues in Parquet and Iceberg Reads and Writes]. + +## Native Parquet scan + +### Nested types (audited 2026-10-15) + +- **Scope:** structs, arrays and maps. NULL structs, arrays and maps, NULLs inside them, Parquet + field ids, case-insensitive names and read schemas that differ from the file's. +- **Method:** about 3,000 queries compared with Spark with Comet disabled, on Spark 3.4.3, 3.5.9 + and 4.1.3 at `main` 66c63bd307. A code review of DataFusion 55.2 and arrow-rs 59.3 ran alongside, + and each finding was reproduced before it was filed. +- **Filed:** [#7029] (INT96 timestamps inside legacy-format lists), [#7030] (a nested column with + none of its requested fields in the file), [#7031] (Spark 3.4: a struct matched by field id under + a new name, under nested pruning), [#7032] (nested BINARY values that are not valid UTF-8, read as + STRING), [#7034] (`TIMESTAMP(NANOS)` read as `timestamp`), [#7035] (a nested encoding that + arrow-rs rejects fails the whole file), [#7036] and [#7037] (nested pruning gaps), and + [apache/datafusion#26393] upstream for [#7029]. All are tracked in [#7040]. +- **Matched Spark:** NULL structs, arrays and maps, empty arrays and maps, and NULLs inside them at + every level, in deep (eight levels) and wide (300 fields) structs. Spark-written files in five + layouts (default, tiny row groups and pages, the legacy format, v2 pages without dictionaries and + v2 tiny pages), read with batch sizes of 1 to 3, with row-filter pushdown on and off and with + filter pushdown off. Avro, Thrift, Hive and protobuf list and map encodings, maps with struct and + array keys, nested type promotion, and nested timestamp, decimal, boolean and binary leaves + (micros, millis, and INT96 in standard layouts). ENUM, JSON, BSON, unsigned and binary-decimal + annotations, files written by arrow-rs, partitioned tables, nested schemas that differ across the + files of one scan, case-insensitive nested names, nested field ids (renamed, missing and partly + present), and both modes of SPARK-53535 on Spark 4.1. +- **Not Comet bugs:** with `spark.sql.parquet.fieldId.read.enabled=true` and a read schema that + swaps two nested field names, Spark's vectorized reader pairs the file's columns with the read + fields by the file's names, so it fails or returns no rows, while Comet resolves the fields by + id. On Spark 3.5 and later, nested schema pruning drops a root field id, so Spark reads a renamed + root as NULL, and Comet matches it there. A map with NULL keys from a non-Spark writer fails in + both engines. +- **Not covered:** Variant ([#5438]), the V2 Parquet source, encrypted files, and the JVM-side + conversion paths. + +## Native Parquet writer + +### Nested types (audited 2026-10-16) + +- **Scope:** NULL structs, arrays and maps, NULLs inside them, Parquet field ids, column statistics, + and the file's schema and footer. +- **Method:** each write made natively and by Spark with Comet disabled, on Spark 3.4.3, 3.5.9 and + 4.1.3 at `main` c230bf4e98. The comparison covered the values read back by Spark and by Comet, + the Parquet schema with its repetition and field ids, the footer key-value metadata and the + statistics of every leaf. A code review of the writer and arrow-rs 59.3 ran alongside, and each + finding was reproduced before it was filed. +- **Filed:** [#7047] (the datetime rebase modes are ignored, so Spark rejects or shifts the + writer's dates before 1582), [#7048] (`outputTimestampType`, `writeLegacyFormat` and Spark's + schema footer key), and a measurement on [#5304]: for nested rows the row-count cap gives one row + group per file, 239 MB for 100,000 rows of 100-element arrays, where Spark writes row groups of + about 128 MB. All are tracked in [#7051]. +- **Matched Spark:** values, repetition and field ids for NULL structs, arrays and maps, empty + arrays and maps, and NULLs inside them at every level, in deep and wide schemas. That includes + input sliced by `LIMIT` and `OFFSET`, filtered input, and nested values computed by expressions + before the write. Field ids at every level, with `spark.sql.parquet.fieldId.write.enabled` on and + off. Null counts, value counts, minimums and maximums for every leaf. Nested timestamp, decimal, + boolean and binary leaves. +- **Did not reproduce:** a write of void (`NullType`) columns on Spark 3.4 and 3.5, which Spark + rejects at analysis before Comet plans the write. A write of ANSI interval columns, which did not + reach the native writer on any of the three versions. A task failure from the in-progress row + group outgrowing a 256 MB off-heap pool: the 239 MB row group fit. +- **Not covered:** Variant ([#5433]), partitioned and bucketed writes, which the native writer does + not support ([#3193], [#3194]), object-store and HDFS outputs, and write throughput, which needs a + release build. + +[Known Issues in Parquet and Iceberg Reads and Writes]: ../../user-guide/latest/compatibility/parquet-iceberg-known-issues.md +[apache/datafusion#26393]: https://github.com/apache/datafusion/issues/26393 +[#3193]: https://github.com/apache/datafusion-comet/issues/3193 +[#3194]: https://github.com/apache/datafusion-comet/issues/3194 +[#5304]: https://github.com/apache/datafusion-comet/issues/5304 +[#5433]: https://github.com/apache/datafusion-comet/issues/5433 +[#5438]: https://github.com/apache/datafusion-comet/issues/5438 +[#7029]: https://github.com/apache/datafusion-comet/issues/7029 +[#7030]: https://github.com/apache/datafusion-comet/issues/7030 +[#7031]: https://github.com/apache/datafusion-comet/issues/7031 +[#7032]: https://github.com/apache/datafusion-comet/issues/7032 +[#7034]: https://github.com/apache/datafusion-comet/issues/7034 +[#7035]: https://github.com/apache/datafusion-comet/issues/7035 +[#7036]: https://github.com/apache/datafusion-comet/issues/7036 +[#7037]: https://github.com/apache/datafusion-comet/issues/7037 +[#7040]: https://github.com/apache/datafusion-comet/issues/7040 +[#7047]: https://github.com/apache/datafusion-comet/issues/7047 +[#7048]: https://github.com/apache/datafusion-comet/issues/7048 +[#7051]: https://github.com/apache/datafusion-comet/issues/7051 diff --git a/docs/source/contributor-guide/index.md b/docs/source/contributor-guide/index.md index 1abd8891d20..a773c1b2ed7 100644 --- a/docs/source/contributor-guide/index.md +++ b/docs/source/contributor-guide/index.md @@ -95,6 +95,7 @@ Tracing :hidden: Expression Audits +Data Source Audits Supported Spark Configurations Configuration Conventions ``` diff --git a/docs/source/user-guide/latest/compatibility/index.md b/docs/source/user-guide/latest/compatibility/index.md index a74daf64f75..bc364e14a48 100644 --- a/docs/source/user-guide/latest/compatibility/index.md +++ b/docs/source/user-guide/latest/compatibility/index.md @@ -24,6 +24,8 @@ Comet aims to provide consistent results with the version of Apache Spark that i This guide documents areas where Comet's behavior is known to differ from Spark. Topics are grouped by subsystem: - **Parquet**: limitations when reading Parquet files. +- **Parquet and Iceberg known issues**: open issues in which Parquet and Iceberg reads and writes + give different results from Spark, or fail where Spark succeeds, without falling back. - **Floating-point comparison**: NaN and signed-zero handling in comparisons. - **Regular expressions**: differences between the Rust regexp crate and Java's regex engine. - **Operators**: operator-level compatibility notes, including window functions and round-robin partitioning. @@ -119,7 +121,9 @@ divergence: The following native paths silently return values that differ from Spark for edge-case inputs. Most also have entries in the per-category expression pages linked above; they are collected here -so users hunting an unexpected value have a single place to check: +so users hunting an unexpected value have a single place to check. Parquet and Iceberg reads and +writes have their own list, in +[Known Issues in Parquet and Iceberg Reads and Writes](parquet-iceberg-known-issues.md#results-that-differ-from-spark): - Native `RANGE` window frames with an explicit `PRECEDING` / `FOLLOWING` offset diverge from Spark when the boundary arithmetic overflows for `DATE` or `DECIMAL` `ORDER BY` columns diff --git a/docs/source/user-guide/latest/compatibility/parquet-iceberg-known-issues.md b/docs/source/user-guide/latest/compatibility/parquet-iceberg-known-issues.md new file mode 100644 index 00000000000..77a0a2501d9 --- /dev/null +++ b/docs/source/user-guide/latest/compatibility/parquet-iceberg-known-issues.md @@ -0,0 +1,266 @@ + + +# Known Issues in Parquet and Iceberg Reads and Writes + +This page lists open issues in Comet's Parquet and Iceberg reads and writes where Comet does not +fall back to Spark. Each entry links to its tracking issue. The entries are in two sections: + +- [Results that differ from Spark](#results-that-differ-from-spark): wrong values, a NULL where + Spark returns a value or the reverse, rows returned for a read that Spark rejects, and written + files that Spark or other engines read differently from the files Spark writes. +- [Queries that fail where Spark succeeds](#queries-that-fail-where-spark-succeeds). + +The cases where Comet falls back to Spark are documented in [Parquet Compatibility](scans.md), in +the [current limitations](../iceberg.md#current-limitations) of the Iceberg guide and in +[Iceberg Writes](../iceberg-writes.md). + +Which entries apply depends on which native paths are enabled: + +| Native path | Setting | Default | +| -------------- | ---------------------------------------- | ---------------------- | +| Parquet scan | `spark.comet.scan.enabled` | `true` | +| Iceberg scan | `spark.comet.scan.icebergNative.enabled` | `true` | +| Iceberg writer | `spark.comet.write.iceberg.enabled` | `false` | +| Parquet writer | `spark.comet.write.parquet.enabled` | `false` (testing only) | + +`spark.comet.scan.enabled` is a testing setting that Comet's own test suites use. It is not meant +for production, so do not turn it off to avoid an entry on this page. + +## Results that differ from Spark + +### Parquet reads + +- **Dates and timestamps written with the legacy calendar.** Spark before 3.0, and later versions + with `spark.sql.parquet.datetimeRebaseModeInWrite=LEGACY`, write these values in the hybrid + Julian/Gregorian calendar. The native Parquet scan reads them as proleptic Gregorian, so values + before October 15, 1582 are wrong in projections and predicates. It ignores + `spark.sql.parquet.datetimeRebaseModeInRead` and the file's `org.apache.spark.legacyDateTime` + metadata ([#5010](https://github.com/apache/datafusion-comet/issues/5010)). The + [Delta scan](../delta.md) rebases. +- **INT96 timestamps inside legacy-format lists.** INT96 timestamps outside the years 1677 to 2262 + read back wrong inside lists written in a legacy layout, as Hive and + `spark.sql.parquet.writeLegacyFormat=true` write them + ([#7029](https://github.com/apache/datafusion-comet/issues/7029)). +- **A struct or array with none of its requested fields in the file.** Spark returns NULL for the + column. Comet returns a non-null struct or array that holds NULLs. This applies to Spark 3.4, 3.5 + and 4.0, and to Spark 4.1 and later with + `spark.sql.legacy.parquet.returnNullStructIfAllFieldsMissing=true` + ([#7030](https://github.com/apache/datafusion-comet/issues/7030)). +- **Nested BINARY values that are not valid UTF-8, read as STRING.** This applies to a BINARY + column without the UTF8 annotation that is read as `string`, through the read schema or + `spark.sql.parquet.binaryAsString=true`. Spark returns the bytes. Comet returns NULL for each + such nested value. Top-level values match Spark in a release build, because the invalid bytes + reach an Arrow string array unchecked. A debug build panics on them + ([#7032](https://github.com/apache/datafusion-comet/issues/7032), part of + [#4764](https://github.com/apache/datafusion-comet/issues/4764)). +- **Spark 3.4: a struct matched by field id under a different name.** With + `spark.sql.parquet.fieldId.read.enabled=true`, a struct column that the read schema matches to + the file by field id but under another name reads as NULL when Spark prunes its nested fields + ([#7031](https://github.com/apache/datafusion-comet/issues/7031)). +- **Field-id reads where one file column matches a field by id and a sibling by name.** With + `spark.sql.parquet.fieldId.read.enabled=true`, Comet returns NULL for the sibling at the top + level. Inside a struct, it returns the values of the field matched by id. Spark's own readers + disagree here ([#6994](https://github.com/apache/datafusion-comet/issues/6994)). +- **`TIMESTAMP_MILLIS` values that overflow when converted to microseconds.** In nested fields, and + in every column of a scan that has data filters, Comet returns NULL where Spark fails with + `ArithmeticException: long overflow` + ([#5553](https://github.com/apache/datafusion-comet/issues/5553), + [#5739](https://github.com/apache/datafusion-comet/issues/5739)). +- **Reads that Spark rejects.** Comet returns rows where Spark fails the read for + `TIMESTAMP(NANOS)` columns read as `timestamp` or `timestamp_ntz` + ([#7034](https://github.com/apache/datafusion-comet/issues/7034)), `INT` columns read as + `INTERVAL DAY TO SECOND` ([#5188](https://github.com/apache/datafusion-comet/issues/5188)), and + Variant-annotated fields read as a plain struct + ([#5741](https://github.com/apache/datafusion-comet/issues/5741)). + +### Iceberg reads + +- **Migrated tables that use a name mapping.** The `migrate`, `snapshot` and `add_files` + procedures set `schema.name-mapping.default` for files without Iceberg field ids. When a query + also reads a top-level primitive column, the native scan returns NULL for nested fields (struct + fields, list elements and map values). Filters on those fields then select the wrong rows + ([#6790](https://github.com/apache/datafusion-comet/issues/6790)). The fix needs an iceberg-rust + upgrade ([#6934](https://github.com/apache/datafusion-comet/issues/6934)). +- **Data files with neither field ids nor a name mapping.** Spark reads every nested field of such + a file as NULL, and the native scan returns the values stored in the file + ([#6794](https://github.com/apache/datafusion-comet/issues/6794)). +- **`VERSION AS OF` reads of a column renamed after the snapshot.** The renamed column reads as + NULL, and two columns that swapped names return each other's values + ([#6822](https://github.com/apache/datafusion-comet/issues/6822)). +- **Reflection failures while the scan is planned.** If Comet fails to read the table's schema or + metadata through reflection, it continues with an empty or partial result instead of falling + back, which can resolve a column to a different field id + ([#5257](https://github.com/apache/datafusion-comet/issues/5257)). + +### Parquet writes + +The native Parquet writer is experimental and disabled by default. With +`spark.comet.write.parquet.enabled=true`: + +- **The datetime rebase modes are ignored.** On Spark 3.4 and 3.5, whose default rebase mode is + `EXCEPTION`, the native writer writes dates and timestamps before October 15, 1582 that Spark + refuses to write, and Spark then fails to read them back under its default read mode. With + `spark.sql.parquet.datetimeRebaseModeInRead=LEGACY`, Spark reads them shifted by several days, on + every version. The files lack the `org.apache.spark.version` footer key that tells Spark they need + no rebasing ([#7047](https://github.com/apache/datafusion-comet/issues/7047), + [#3427](https://github.com/apache/datafusion-comet/issues/3427)). +- **`spark.sql.parquet.outputTimestampType` and `spark.sql.parquet.writeLegacyFormat` are + ignored.** Timestamps are always written as `INT64` microseconds, never as Spark's default + `INT96`, so with `TIMESTAMP_MILLIS` they keep the sub-millisecond digits that Spark drops. Spark's + schema footer key is not written either, so a path-based read loses column metadata such as + comments, and with `spark.sql.parquet.inferTimestampNTZ.enabled=false` it reads `TIMESTAMP_NTZ` + columns as `TIMESTAMP` ([#7048](https://github.com/apache/datafusion-comet/issues/7048), + [#3425](https://github.com/apache/datafusion-comet/issues/3425)). +- **Spark 3.4 and 3.5: `INSERT INTO` leaves the target stale.** Comet skips the steps Spark runs + after a write: file index refresh, cache refresh, partition refresh and statistics update. Later + queries can return the table's previous contents until `REFRESH TABLE` + ([#6315](https://github.com/apache/datafusion-comet/issues/6315)). +- **Spark's Parquet writer properties are ignored.** `parquet.block.size`, page size, dictionary, + writer version and statistics settings never reach the native writer + ([#5304](https://github.com/apache/datafusion-comet/issues/5304)). +- **An input with no partitions writes no file.** Spark writes one file that holds the schema + ([#5303](https://github.com/apache/datafusion-comet/issues/5303)). +- **Spark SQL tests that still fail with the native writer**, not yet classified: + - CTAS schema ([#3418](https://github.com/apache/datafusion-comet/issues/3418)) + - CTAS with `CHAR_AS_VARCHAR` ([#3419](https://github.com/apache/datafusion-comet/issues/3419)) + - CTAS with a union ([#3429](https://github.com/apache/datafusion-comet/issues/3429)) + - `INSERT INTO` with default columns ([#3421](https://github.com/apache/datafusion-comet/issues/3421)) + - `INSERT INTO` after `ALTER TABLE ADD COLUMNS` ([#3422](https://github.com/apache/datafusion-comet/issues/3422)) + - `INSERT INTO` from non-projection queries ([#3423](https://github.com/apache/datafusion-comet/issues/3423)) + - the `INSERT OVERWRITE` output schema ([#3424](https://github.com/apache/datafusion-comet/issues/3424)) + - multiple inserts ([#3430](https://github.com/apache/datafusion-comet/issues/3430)) + - `INSERT OVERWRITE LOCAL DIRECTORY` ([#3420](https://github.com/apache/datafusion-comet/issues/3420)) + - empty files ([#3417](https://github.com/apache/datafusion-comet/issues/3417)) + - `onTaskCommit` ([#3428](https://github.com/apache/datafusion-comet/issues/3428)) + +### Iceberg writes + +The native Iceberg writer is disabled by default. With `spark.comet.write.iceberg.enabled=true`: + +- **Brotli tables.** The native writer commits brotli-compressed files. JVM Parquet readers can + read them only if Hadoop's `BrotliCodec` is on the classpath. iceberg-java fails the write + instead ([#6960](https://github.com/apache/datafusion-comet/issues/6960)). +- **`write.parquet.compression-codec=none`.** The native writer writes the table uncompressed, and + iceberg-java rejects the value. The native writer also trims or lowercases a few other settings + that iceberg-java reads as written + ([#7008](https://github.com/apache/datafusion-comet/issues/7008)). + +The physical differences that every native data file has, and why they are safe, are listed in +[Accepted divergences](../iceberg-writes.md#accepted-divergences). + +## Queries that fail where Spark succeeds + +In these cases a Comet read or write fails with an error where Spark, or iceberg-java, succeeds. +Comet does not fall back to Spark for them. + +### Parquet reads that fail + +- **STRING columns that hold invalid UTF-8.** The read fails with + `Parquet error: encountered non UTF-8 data`, where Spark returns the bytes + ([#4121](https://github.com/apache/datafusion-comet/issues/4121), part of + [#4764](https://github.com/apache/datafusion-comet/issues/4764)). +- **Duplicate field names.** Selecting a field by name when several physical siblings match it, + including inside structs, arrays and maps, fails with a duplicate-field error + ([#5884](https://github.com/apache/datafusion-comet/issues/5884), + [#5964](https://github.com/apache/datafusion-comet/issues/5964)). + [Parquet Compatibility](scans.md) describes the reads this affects. +- **A duplicate field id in a column the query does not read.** With a valid read schema and + `spark.sql.parquet.fieldId.read.enabled=true`, a file in which two root columns carry the same + field id fails the scan whenever the table schema has a field with that id, even when the query + does not read that field ([#7006](https://github.com/apache/datafusion-comet/issues/7006)). +- **A nested encoding that arrow-rs rejects.** A file fails to read when any column in its schema + uses such an encoding, even a column the query does not read + ([#7035](https://github.com/apache/datafusion-comet/issues/7035)). +- **A `DATE` column read as `INTERVAL YEAR TO MONTH`.** Spark reads it, and Comet fails the read + ([#5188](https://github.com/apache/datafusion-comet/issues/5188)). + +### Iceberg reads that fail + +- **A data file that omits an optional nested field.** The native scan fails on a data file that + lacks an optional struct field, which a write with `write.spark.accept-any-schema=true` can + produce without any schema change + ([#7033](https://github.com/apache/datafusion-comet/issues/7033)). +- **`VERSION AS OF` reads of a column whose type was promoted after the snapshot** + ([#6993](https://github.com/apache/datafusion-comet/issues/6993)). +- **An equality delete keyed on a nested field.** Every query the native scan runs on the table + fails with `field not found`, even one that reads the key + ([#6782](https://github.com/apache/datafusion-comet/issues/6782)). +- **Column names that only newer Unicode case data folds together**, such as a pair of Vithkuqi + letters, which Unicode 14 added. Queries on such a table fail with a duplicate-field error + ([#5540](https://github.com/apache/datafusion-comet/issues/5540)). + +### Object store and credential settings + +The native Parquet and Iceberg scans resolve storage settings with their own clients rather than +with Hadoop's. Where the two disagree, a native read can use a different identity or route than +Spark, and fail where Spark succeeds: + +- **S3 settings that Hadoop's S3A client resolves differently**, such as JCEKS-backed secrets and + the precedence of per-bucket settings. The native Parquet scan forwards them without checking + for a difference ([#5662](https://github.com/apache/datafusion-comet/issues/5662)). +- **AWS profile credential providers.** The native scans ignore `fs.s3a.auth.profile.name` and + `fs.s3a.auth.profile.file` ([#4245](https://github.com/apache/datafusion-comet/issues/4245)), + and resolve some profiles differently from Hadoop 3.4 and later, for example profiles that use + `credential_process` or chains of assumed roles + ([#6575](https://github.com/apache/datafusion-comet/issues/6575)). +- **Azure credentials.** Container-scoped OAuth settings, Fabric hosts and a few other + configurations resolve differently from Hadoop's ABFS driver + ([#6605](https://github.com/apache/datafusion-comet/issues/6605)). +- **Iceberg tables whose metadata location has no host.** The native Iceberg scan skips the + credential provider configured with `s3.comet.credential.provider.class` + ([#6536](https://github.com/apache/datafusion-comet/issues/6536)). + +### Parquet writes that fail + +With `spark.comet.write.parquet.enabled=true`: + +- **Row groups that outgrow the memory pool.** The native writer cuts row groups by row count + only, where Spark uses `parquet.block.size`, and it cannot spill. With wide or nested rows, the + open row group can outgrow the off-heap pool and fail the task. The audit did not reproduce a + failure, because a 239 MB row group fit in a 256 MB pool + ([#5304](https://github.com/apache/datafusion-comet/issues/5304)). + +### Iceberg writes that fail + +With `spark.comet.write.iceberg.enabled=true`: + +- **A write that keeps one file open and outgrows the off-heap pool.** An unpartitioned or + clustered write fails its task when its open file outgrows the pool, where iceberg-java writes + it on the JVM heap. [Failure handling](../iceberg-writes.md#failure-handling) describes the + error and the settings that let such a write fit. + +## Differences that are not Comet bugs + +In these cases Comet's result differs from Spark's, but the bug is in Spark or iceberg-java: + +- **iceberg-java 1.9 and later: `IS NULL` on a float or double field under a NULL struct.** + iceberg-java leaves the rows where the struct is NULL out of the field's null count, so a filter + such as `WHERE s.x IS NULL` can skip data files that hold matching rows, with or without Comet + ([apache/iceberg#17561](https://github.com/apache/iceberg/issues/17561)). Files written by Comet's + native Iceberg writer are not affected. +- **iceberg-java before 1.12: a NULL struct whose first field is an identity partition source.** + iceberg-java reads the struct as a struct of NULLs, and Comet returns NULL, as iceberg-java 1.12 + does. [apache/iceberg#17320](https://github.com/apache/iceberg/pull/17320) is titled for nested + initial defaults, but it fixes this case too: it keeps one file column in the read, so the + struct's definition level survives when every projected field is a constant, and an identity + partition value is such a constant. +- **Field-id reads with a read schema that swaps two nested field names.** Spark's vectorized + reader pairs the file's columns with the read fields by the file's names, so it fails with a type + mismatch or returns no rows. Comet resolves the fields by id. diff --git a/docs/source/user-guide/latest/compatibility/scans.md b/docs/source/user-guide/latest/compatibility/scans.md index 7b7d2c198e8..ec6142713f2 100644 --- a/docs/source/user-guide/latest/compatibility/scans.md +++ b/docs/source/user-guide/latest/compatibility/scans.md @@ -55,18 +55,15 @@ The following features are not supported and cause Comet to fall back to Spark: `spark.sql.caseSensitive=false`. Spark's analyzer normally rejects such a schema before the scan is planned. -The following limitation may produce incorrect results without falling back to Spark: - -- No support for datetime rebasing. When reading Parquet files containing dates or timestamps - written with `spark.sql.parquet.datetimeRebaseModeInWrite=LEGACY` (which is Spark's default for - data written before Spark 3.0, using the hybrid Julian/Gregorian calendar), Comet reads them as - if they were written using the Proleptic Gregorian calendar. This produces silently-wrong - values for dates before October 15, 1582 in both projections and predicates. Comet also - ignores `spark.sql.parquet.datetimeRebaseModeInRead` and the file-level - `org.apache.spark.legacyDateTime` metadata that would tell it to rebase. Tracked by - [#5010](https://github.com/apache/datafusion-comet/issues/5010). - -The following limitations raise an error at scan time rather than falling back to Spark: +Some Parquet reads return results that differ from Spark without falling back to Spark, such as +reads of dates and timestamps written with the legacy hybrid Julian/Gregorian calendar +([#5010](https://github.com/apache/datafusion-comet/issues/5010)). +[Known Issues](parquet-iceberg-known-issues.md#parquet-reads) lists them, with the +issues that track them. + +The following limitations raise an error at scan time rather than falling back to Spark. More +reads that fail where Spark succeeds are listed in +[Known Issues](parquet-iceberg-known-issues.md#parquet-reads-that-fail): - Selecting a field by name when multiple physical siblings match, including inside structs, arrays, and maps. Comet raises a duplicate-field error instead of resolving the collision. diff --git a/docs/source/user-guide/latest/compatibility/spark-versions.md b/docs/source/user-guide/latest/compatibility/spark-versions.md index 84baf6194fa..f178b679f94 100644 --- a/docs/source/user-guide/latest/compatibility/spark-versions.md +++ b/docs/source/user-guide/latest/compatibility/spark-versions.md @@ -69,17 +69,6 @@ Spark 4.0.4 is supported with Java 17/21 and Scala 2.13. Spark 4.1.3 is supported with Java 17/21 and Scala 2.13. -### Known Limitations - -- **`NullType` columns in Parquet files** - ([#4199](https://github.com/apache/datafusion-comet/issues/4199)): Spark encodes a `NullType` - column as a Parquet `BOOLEAN` physical type annotated with `LogicalType::Unknown`. The Rust - `parquet` crate that Comet depends on accepts `Unknown` only when paired with `INT32` and rejects - any other physical type with `Parquet error: Cannot annotate Unknown from BOOLEAN for field ''`. - Any attempt to read a Parquet file that contains a `NullType` column fails at decode time before - Comet's scan runs. Workaround: project the column away, cast it to a concrete type before - persisting, or read the file with Comet disabled for that query. - ## Spark 4.2 Spark 4.2.0 is supported with Java 17 and Scala 2.13. diff --git a/docs/source/user-guide/latest/iceberg-writes.md b/docs/source/user-guide/latest/iceberg-writes.md index bba4ea8ec14..840916411e4 100644 --- a/docs/source/user-guide/latest/iceberg-writes.md +++ b/docs/source/user-guide/latest/iceberg-writes.md @@ -494,9 +494,14 @@ iceberg-java's _writer-tracked_ state would have recorded, and both are analyzed change pruning decisions. - On Iceberg 1.9+, manifest `value_counts` / `null_value_counts` for float/double columns nested under a nullable struct count rows whose parent struct is null (they come from the - parquet footer), while iceberg-java's writer-tracked counts do not. Both counts inflate by - the same amount, so the derived null ratios and `IS NULL` / `IS NOT NULL` pruning decisions - are unaffected. + parquet footer), while iceberg-java's writer-tracked counts do not. The native counts are the + correct ones. iceberg-java's counts leave out rows whose struct is null. When every non-null + struct in a file holds a value, iceberg-java records a null count of 0, and `IS NULL` prunes a + file that contains matching rows + ([apache/iceberg#17561](https://github.com/apache/iceberg/issues/17561)). The native counts + never prune such a file. `IS NOT NULL` pruning is the same for both. All content not listed above — the logical data, statistics values, and manifest metadata — -must match iceberg-java exactly, or the write falls back. +must match iceberg-java exactly, or the write falls back. The open bugs in which a native write +still differs are listed in +[Known Issues](compatibility/parquet-iceberg-known-issues.md#iceberg-writes). diff --git a/docs/source/user-guide/latest/iceberg.md b/docs/source/user-guide/latest/iceberg.md index 24713a32b7d..0c26e16e68e 100644 --- a/docs/source/user-guide/latest/iceberg.md +++ b/docs/source/user-guide/latest/iceberg.md @@ -198,6 +198,10 @@ The following scenarios will fall back to the JVM Iceberg reader: change. The check uses the table's schema history, so the fallback stays after those files are rewritten +The cases in which the native reader returns different results from Spark, or fails where Spark +succeeds, without falling back are listed in +[Known Issues](compatibility/parquet-iceberg-known-issues.md). + Writes are not covered by this list. By default Spark plans Iceberg writes with its own V2 write operator, and iceberg-java writes the data files. With `spark.comet.write.iceberg.enabled=true`, Comet plans an Iceberg write with its split-operator plan and writes eligible data files natively; diff --git a/docs/source/user-guide/latest/index.rst b/docs/source/user-guide/latest/index.rst index 84b0064647d..b4cfe077ca8 100644 --- a/docs/source/user-guide/latest/index.rst +++ b/docs/source/user-guide/latest/index.rst @@ -60,6 +60,7 @@ the navigation menu to read more. Overview compatibility/scans + compatibility/parquet-iceberg-known-issues compatibility/floating-point compatibility/regex compatibility/operators