[Feature] spark 4.2 #8901

Activity

1. melin

   added

   enhancementNew feature or requestNew feature or request

   on Jul 29, 2026
2. LuciferYang commented on Aug 17, 2026

   

   LuciferYang

   on Aug 17, 2026

   Contributor

   More actions

   Opened #9265 for this. It raises the spark4 profile baseline from 4.1.2 to 4.2.0, adds a paimon-spark-4.2 module, and keeps 4.0 and 4.1 working under the new baseline — the same shape as #7648 did for 4.1.

   The bump is what makes this more than a version string: paimon-spark-common and paimon-spark4-common are compiled once against the newest supported Spark and shipped to every 4.x runtime, and 4.2 turned CatalogManager into an interface, added case-class fields, renamed RewriteRowLevelCommand's DELTA_OPERATIONS_WITH_* constants, gave V2WriteCommand a new supertype, and split DESCRIBE ... PARTITION into its own plan node (SPARK-39660). So the PR also carries the neutralization needed to keep the older modules linking.

   A follow-up PR will add the cross-version binary linkage scanner used to find those incompatibilities. It is held back because its test fixtures reference classes that only exist once #9265 lands.