You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Spark writes `NullType` columns to parquet as `BOOLEAN` physical type with an `Unknown` logical type annotation (comment in `ParquetSchemaConverter.scala`: "Selected primitive type here doesn't have significance"). parquet-rs only accepts `LogicalType::Unknown` paired with `PhysicalType::INT32` and rejects any other physical type with `Cannot annotate Unknown from BOOLEAN for field '…'` (parquet-57.2.0/src/schema/types.rs:401, :423).
Result: any attempt to read a Spark-written parquet file that contains a `NullType` field fails in Comet with:
```
org.apache.comet.CometNativeException: Parquet error: Cannot annotate Unknown from BOOLEAN for field '_1'
```
The SPARK-54220 test in `ParquetIOSuite` (`SPARK-54220: vectorized reader: missing all struct fields, struct with NullType only`) is the concrete reproducer. It was unignored as part of PR #4190 / issue #4136 but crashes on the parquet read path before the new fix in `parquet_convert_struct_to_struct` is reached.
Reproducer
See `issue #4136: struct with only NullType fields in file (SPARK-54220)` in `CometNativeReaderSuite`. The failure manifests for both `native_datafusion` and `native_iceberg_compat`.
Suspected fix
Either:
Upstream parquet-rs to accept `(Unknown, BOOLEAN)` (and arguably any physical type, since Spark's comment makes clear the physical type is a don't-care), or
Work around in Comet: in the schema adapter or parquet reader factory, rewrite the physical type to INT32 before passing it to parquet-rs' validator — or special-case the Unknown-annotated field at read time.
This no longer reproduces on main. apache/arrow-rs#9855 lets parquet-rs accept the UNKNOWN logical type on any physical type, and the parquet 59 crate Comet uses includes it.
I checked at c230bf4 on Spark 4.1.3. A read of a Spark-written file with a NullType column runs in the native scan when the query prunes that column, and falls back to Spark when the query reads it, and both match Spark. issue #4136: struct with only NullType fields in file (SPARK-54220) in CometNativeReaderSuite also passes with its assume(false) removed.
#7052 drops the Spark 4.1 known-limitation entry for this from spark-versions.md. Two skips still point here and can go in a follow-up: that assume(false), and the IgnoreComet on the SPARK-54220 test in dev/diffs/4.1.3.diff and dev/diffs/4.2.0.diff. Closing.
Activity
fix: skip SPARK-54220 NullType test; upstream parquet-rs gap
docs: start Spark 4.1 known-limitations section, seeded with #4199 (#…
andygrove commentedon May 4, 2026
This is possibly fixed already upstream in apache/arrow-rs#9855
andygrove commentedon May 6, 2026
Note that this compatibility issue is documented:
docs: drop the Spark 4.1 NullType read limitation
andygrove commentedon Oct 16, 2026
This no longer reproduces on
main. apache/arrow-rs#9855 lets parquet-rs accept theUNKNOWNlogical type on any physical type, and the parquet 59 crate Comet uses includes it.I checked at c230bf4 on Spark 4.1.3. A read of a Spark-written file with a
NullTypecolumn runs in the native scan when the query prunes that column, and falls back to Spark when the query reads it, and both match Spark.issue #4136: struct with only NullType fields in file (SPARK-54220)inCometNativeReaderSuitealso passes with itsassume(false)removed.#7052 drops the Spark 4.1 known-limitation entry for this from
spark-versions.md. Two skips still point here and can go in a follow-up: thatassume(false), and theIgnoreCometon the SPARK-54220 test indev/diffs/4.1.3.diffanddev/diffs/4.2.0.diff. Closing.docs: list known Parquet and Iceberg issues and add a data source aud…