docs: list known Parquet and Iceberg issues and add a data source audit log

[andygrove]

Which issue does this PR close?

Closes #7050.

Rationale for this change

The nested-type audits of the native Parquet and Iceberg scans (#7040) and writers (#7051) filed or confirmed a set of open issues in which Comet gives a different result from Spark, or fails where Spark succeeds, without falling back. The user guide named few of them: datetime rebasing (#5010) among the wrong results, and invalid UTF-8 and duplicate field names among the failures. A user who sees an unexpected value or error has no page that lists the rest. The audits' results, including what was checked and matched and which candidates did not reproduce, are recorded only in those two epics, which is not where the next audit of the same area would look. The expression audits have a log in the contributor guide, and the data sources have none.

The Iceberg writes page also says the float/double null-count difference can't change IS NULL pruning. It can: iceberg-java's counts are the wrong ones (apache/iceberg#17561), and the native counts are correct (#7050).

What changes are included in this PR?

• A new compatibility page, "Known Issues in Parquet and Iceberg Reads and Writes", with a link to the issue that tracks each entry. A table at the top says which native paths are on by default.
  • "Results that differ from Spark" lists, for Parquet reads, Iceberg reads, Parquet writes and Iceberg writes, the open issues in which Comet returns wrong values, a NULL where Spark returns a value or the reverse, rows for a read that Spark rejects, or files that readers see differently from Spark's.
  • "Queries that fail where Spark succeeds" lists the Parquet and Iceberg reads that fail, the object store and credential settings that the native clients resolve differently from Hadoop, and the memory failures of the native writers.
  • A last section lists the differences that are bugs in Spark or iceberg-java.
  • The cases that fall back to Spark stay on the existing pages.
• Pointers to the new page from the compatibility overview (its topic list and "Known result-value divergences"), from scans.md, whose datetime rebasing item moves to the new page, from the Iceberg guide's current limitations, and from the Iceberg writes page's accepted divergences.
• The corrected float/double null-count bullet in iceberg-writes.md.
• The Spark 4.1 known limitation for NullType columns in Parquet files is removed from spark-versions.md. parquet-rs accepts the UNKNOWN logical type on any physical type since [PARQUET] Allow UNKNOWN logical type annotation on any physical type arrow-rs#9855, which the parquet 59 crate includes, so those reads no longer fail (Spark 4.1 NullType parquet: parquet-rs rejects BOOLEAN + Unknown logical type #4199).
• A new "Data Source Audits" section in the contributor guide, next to Expression Audits, with a Parquet page and an Iceberg page. Each holds dated entries for the 2026-10-15 scan audit and the 2026-10-16 writer audit: scope, method and versions, the issues filed, what matched, what did not reproduce, the differences that are not Comet bugs, and what was not covered.

Two issues are left off the page because they no longer reproduce on main and are closed: reading a Spark 4.1 file with a NullType column (#4199) and native Parquet writes in overwrite mode into an existing directory (#2970, checked on Spark 3.5 and 4.1).

How are these changes tested?

Documentation only. prettier --check (3.9.10) passes on the changed files. A local Sphinx build gives the same 80 warnings as main, so every new link and heading anchor resolves.

[kazuyukitanimura]

Thank you @andygrove
My comments below may not be a blocker...

[kazuyukitanimura]

Thanks for collecting these. Several of them look like Spark SQL test names, and the bullet says they are not yet classified. Since this section is about results that differ from Spark, could we avoid listing them as divergences until someone has checked what a user would actually see? Could they move to the contributor guide's Parquet audit as an open question, or could the bullet say what differs for each one?

[kazuyukitanimura]

The linked apache/iceberg#17320 is titled 'Fix nested initial default applied when ancestor struct is null'. Is it the same fix as the partition-source behavior described here? If so, could the entry say how, for example that the identity partition value is read as a constant through the same path? If it is a different change, could you link the one that fixed this?

[comphead]

I checked the docs against the code at 9293815e09. The config defaults, heading anchors and issue titles match. A few things need a change.

Consistency

1. #6264 (dynamic partition pruning under AQE) and #6837 (fs.comet.libhdfs.schemes) were closed today by #6270 and #6838, and both are on main. The page still lists them as open, so please drop both entries.
2. The "Row groups that outgrow the memory pool" entry sits under failures, but data-source-audits/parquet.md says the failure did not reproduce (a 239 MB row group fit in a 256 MB pool). Please say so on the page. #5304 is also about the writer ignoring parquet.block.size, page size, dictionary, writer version and statistics. That part is missing from "Parquet writes" under "Results that differ".
3. The datetime rebase entry says Comet does not rebase. native/core/src/parquet/datetime_rebase.rs and delta.md show the Delta scan does. Please scope the entry to the native Parquet scan.
4. dev/diffs/4.1.3.diff and 4.2.0.diff still tag the SPARK-54220 test with IgnoreComet(".../issues/4199"). This PR says #4199 no longer reproduces. I did not run the test. Either the tag is stale or the test still fails for another reason. A stale tag should be regenerated through the documented workflow, not edited by hand.
5. spark.comet.scan.enabled is in the testing category and documented as not for production. The table shows it as the on-switch for the Parquet scan, so a note would help.
6. The #7006 entry reads like the scans.md fallback for a read schema that repeats a field id. #7006 is a file that repeats a field id, so please say the read schema is valid.

Suggested rephrases

Intro:

This page lists open issues in Comet's Parquet and Iceberg reads and writes where Comet does not fall back to Spark. Each entry links to its tracking issue.

Legacy calendar:

Dates and timestamps written with the legacy calendar. Spark before 3.0, and later versions with spark.sql.parquet.datetimeRebaseModeInWrite=LEGACY, write these values in the hybrid Julian/Gregorian calendar. The native Parquet scan reads them as proleptic Gregorian, so values before October 15, 1582 are wrong in projections and predicates. It ignores spark.sql.parquet.datetimeRebaseModeInRead and the file's org.apache.spark.legacyDateTime metadata (#5010). The Delta scan rebases.

#7030:

A struct or array with none of its requested fields in the file. Spark returns NULL for the column. Comet returns a non-null struct or array that holds NULLs. This applies to Spark 3.4, 3.5 and 4.0, and to Spark 4.1 and later with spark.sql.legacy.parquet.returnNullStructIfAllFieldsMissing=true.

#7032:

Nested BINARY values that are not valid UTF-8, read as STRING. This applies to a BINARY column without the UTF8 annotation that is read as string, through the read schema or spark.sql.parquet.binaryAsString=true. Spark returns the bytes. Comet returns NULL for each such nested value. (Please confirm that top-level values match Spark here. The current text says only that they skip validation.)

#6994:

Field-id reads where one file column matches a field by id and a sibling by name. With spark.sql.parquet.fieldId.read.enabled=true, Comet returns NULL for the sibling at the top level. Inside a struct, it returns the values of the field matched by id. Spark's own readers disagree here.

#6790:

Migrated tables that use a name mapping. The migrate, snapshot and add_files procedures set schema.name-mapping.default for files without Iceberg field ids. When a query also reads a top-level primitive column, the native scan returns NULL for nested fields (struct fields, list elements and map values). Filters on those fields then select the wrong rows. The fix needs an iceberg-rust upgrade (#6934).

#6960 and #7008:

Brotli tables. The native writer commits brotli-compressed files. JVM Parquet readers can read them only if Hadoop's BrotliCodec is on the classpath. iceberg-java fails the write instead.

write.parquet.compression-codec=none. The native writer writes the table uncompressed, and iceberg-java rejects the value. The native writer also trims or lowercases a few other settings that iceberg-java reads as written.

#6315:

Spark 3.4 and 3.5: INSERT INTO leaves the target stale. Comet skips the steps Spark runs after a write: file index refresh, cache refresh, partition refresh and statistics update. Later queries can return the table's previous contents until REFRESH TABLE.

#5304:

Row groups that outgrow the memory pool. The native writer cuts row groups by row count only, where Spark uses parquet.block.size, and it cannot spill. With wide or nested rows, the open row group can outgrow the off-heap pool and fail the task. The audit did not reproduce a failure, because a 239 MB row group fit in a 256 MB pool.

and under "Parquet writes":

Spark's Parquet writer properties are ignored. parquet.block.size, page size, dictionary, writer version and statistics settings never reach the native writer (#5304).

The 11-issue sentence under "Spark SQL tests that still fail with the native writer" is hard to scan. A nested list with one bullet per issue would read better.

iceberg-writes.md null-count bullet. "iceberg-java's leave out" reads like a typo:

The native counts are the correct ones. iceberg-java's counts leave out rows whose struct is null. When every non-null struct in a file holds a value, iceberg-java records a null count of 0, and IS NULL prunes a file that contains matching rows (apache/iceberg#17561). The native counts never prune such a file. IS NOT NULL pruning is the same for both.

[comphead]

Inline versions of the points in my earlier review, anchored to the lines they concern.

[comphead]

Suggested intro, one fact per sentence:

This page lists open issues in Comet's Parquet and Iceberg reads and writes where Comet does not fall back to Spark. Each entry links to its tracking issue.

[comphead]

spark.comet.scan.enabled is in the testing category in CometConf.scala and is documented as not intended for production. Showing it as the Parquet scan switch could invite users to toggle it. A short note under the table would help.

[comphead]

Two things here. First, native/core/src/parquet/datetime_rebase.rs says per-file rebase is enabled only for the Delta scan (rebase_from_file_metadata), and delta.md documents it. "Comet does not rebase" is true only for the plain native Parquet scan, so please scope the entry. Second, a shorter wording:

Dates and timestamps written with the legacy calendar. Spark before 3.0, and later versions with spark.sql.parquet.datetimeRebaseModeInWrite=LEGACY, write these values in the hybrid Julian/Gregorian calendar. The native Parquet scan reads them as proleptic Gregorian, so values before October 15, 1582 are wrong in projections and predicates. It ignores spark.sql.parquet.datetimeRebaseModeInRead and the file's org.apache.spark.legacyDateTime metadata (#5010). The Delta scan rebases.

[comphead]

"That is Spark's only behavior ... and its behavior on Spark 4.1" is hard to parse. Suggested:

A struct or array with none of its requested fields in the file. Spark returns NULL for the column. Comet returns a non-null struct or array that holds NULLs. This applies to Spark 3.4, 3.5 and 4.0, and to Spark 4.1 and later with spark.sql.legacy.parquet.returnNullStructIfAllFieldsMissing=true.

[comphead]

Suggested:

Nested BINARY values that are not valid UTF-8, read as STRING. This applies to a BINARY column without the UTF8 annotation that is read as string, through the read schema or spark.sql.parquet.binaryAsString=true. Spark returns the bytes. Comet returns NULL for each such nested value.

The current last sentence says top-level values "skip UTF-8 validation". Please say whether they match Spark, since a reader will ask.

[comphead]

This reads like the scans.md fallback for a read schema that repeats a field id. This entry is about a file that repeats a field id, so please say the read schema is valid, for example "With a valid read schema and spark.sql.parquet.fieldId.read.enabled=true, a file in which two root columns carry the same field id fails the scan ...".

[comphead]

#6837 was closed today by #6838, which is on main (3a4a159). Please drop this entry.

[comphead]

Two points. First, data-source-audits/parquet.md records that this failure did not reproduce (a 239 MB row group fit in a 256 MB pool), yet this page lists it under failures. Suggested wording:

Row groups that outgrow the memory pool. The native writer cuts row groups by row count only, where Spark uses parquet.block.size, and it cannot spill. With wide or nested rows, the open row group can outgrow the off-heap pool and fail the task. The audit did not reproduce a failure, because a 239 MB row group fit in a 256 MB pool (#5304).

Second, #5304 is mainly about the writer ignoring Spark's Parquet writer properties. That part is missing from "Parquet writes" under "Results that differ from Spark". Suggested entry there:

Spark's Parquet writer properties are ignored. parquet.block.size, page size, dictionary, writer version and statistics settings never reach the native writer (#5304).

[comphead]

"iceberg-java's leave out" reads like a typo. Suggested wording for the whole bullet tail:

The native counts are the correct ones. iceberg-java's counts leave out rows whose struct is null. When every non-null struct in a file holds a value, iceberg-java records a null count of 0, and IS NULL prunes a file that contains matching rows (apache/iceberg#17561). The native counts never prune such a file. IS NOT NULL pruning is the same for both.

[comphead]

Removing the NullType limitation matches #4199 being closed. dev/diffs/4.1.3.diff (around line 3750) and dev/diffs/4.2.0.diff (around line 3831) still tag the SPARK-54220 test with IgnoreComet(".../issues/4199"). I did not run that test. Either the tag is stale or the test still fails for another reason. If it is stale, it should be regenerated through the documented Spark SQL tests workflow, not edited by hand.

[andygrove]

Thanks both. I pushed 844e8a7, which takes all the suggested rewordings.

Dropped the #6264 and #6837 entries, since both are closed by #6270 and #6838 on main.

The datetime rebase entry now covers only the native Parquet scan and says the Delta scan rebases. #5304 now has an entry for the ignored Parquet writer properties under Parquet writes, and the row-group entry says the audit did not reproduce a failure. The #7006 entry says the read schema is valid. The Spark SQL test issues are one bullet each. I also added a note under the table that spark.comet.scan.enabled is a testing setting.

On #7032, top-level values match Spark in a release build, because the invalid bytes reach an Arrow string array unchecked. A debug build panics. The entry now says so.

On iceberg#17320, it is the same fix even though the title names nested initial defaults. The change keeps one file column in the read when every projected field of a struct is a constant, so the struct's definition level survives, and an identity partition value is such a constant. It is in 1.12.0. I added that to the entry and to the Iceberg audit page. I read the change but did not run it against a partitioned table.

The IgnoreComet tags for SPARK-54220 in 4.1.3.diff and 4.2.0.diff are not part of this PR. #7054 re-enables them through the regenerate-the-diff workflow.

[andygrove]

Some of the review feedback was that this docs PR was out of date because other PRs had merged to main while this was going through review. This will keep happening if we keep this PR open, so I'll go ahead and merge now that the first round of feedback has been addressed.