Repository navigation
[FEATURE] Native scan support for VariantType columns (Iceberg + Spark 4.0) #4295
Description
What is the problem the feature request solves?
Today, Comet falls back to JVM Spark for any query touching a VariantType column. This eliminates Comet's acceleration for the entire query, including unrelated operators that could otherwise run natively. This issue tracks adding native execution for Variant column scans, both for plain Parquet and Iceberg tables, by consuming new Variant primitives from arrow-rs and iceberg-rust.
Describe the potential solution
spark/src/main/scala/org/apache/comet/rules/CometScanRule.scala (line 729):
case s: StructType if isVariantStruct(s) =>
fallbackReasons +=
s"Unsupported $name of type VariantType (shredded; not supported by $scanImpl scan)"
false
Additional context
Follow-ups
• Comet support for parse_json / to_variant (write side)
• Comet support for schema_of_variant (metadata operation)
• Predicate pushdown on variant_get(...) = literal into Parquet's column index
• Variant in shuffle / spill paths (separate issue)
Activity
Shekharrajak commentedon May 12, 2026
Ref apache/arrow-rs#9963
apache/arrow-rs#8480
apache/iceberg-rust#2423
apache/iceberg-rust#2411
kazuyukitanimura commentedon May 18, 2026
Changed the priority to high because this is an important feature
peterxcli commentedon Aug 19, 2026
I would like to implement first step of this - basic native VARIANT projection for ordinary Parquet (protobuf/type identity, Arrow-to-Spark bridge, unshredding/reordering, and Spark VariantType vectors).
peterxcli commentedon Aug 19, 2026
take