Native Parquet writer ignores Spark's Parquet writer properties (block size, page size, dictionary, writer version, statistics) #5304
Description
What is the problem the feature request solves?
ParquetWriterExec builds its WriterProperties with compression and nothing else (native/core/src/execution/operators/parquet_writer.rs):
let props = WriterProperties::builder()
.set_compression(compression)
.build();Every other Parquet write knob Spark exposes is therefore silently ignored, and files come out with arrow-rs defaults rather than the ones the user configured:
parquet.block.size(row group size) — tracked as an unchecked item on [EPIC] Improve Comet Native writer #2967parquet.page.size/parquet.page.row.count.limitparquet.enable.dictionaryand dictionary page sizespark.sql.parquet.writer.version(PARQUET_1_0/PARQUET_2_0)- statistics level, and bloom filter settings
spark.sql.files.maxRecordsPerFile(needs file rolling, so partly separate)
Row group size is the one with real consequences: the default row group size decides read parallelism and memory for every downstream consumer, and a user who set parquet.block.size gets no error, just a differently shaped file.
This also matters beyond the V1 write path — #4658 splits Iceberg writes into writer + committer operators with the stated goal of writing data files natively next, and Iceberg data files are read by engines other than Spark.
Describe the potential solution
Thread the resolved options through the ParquetWriter protobuf message and set them on WriterProperties. Spark's ParquetOptions / ParquetWriteSupport are the reference for precedence and defaults.
Related: #2814 (compression settings), #3425 (INT96), #3427 (Spark version in footer metadata).
Activity