feat: support native wide-decimal hashing

[rich7420]

Which issue does this PR close?

Closes #5994.

Rationale for this change

Native wide-decimal hashing used different bytes from Spark, so mixed native/JVM shuffle could lose matching join rows. This makes precision 19–38 hashing Spark-compatible and enables native SQL hash/xxhash64.

What changes are included in this PR?

• Encode unscaled values as minimal signed big-endian bytes and optimize decimal-list hashing; precision at most 18 keeps its existing path. Retain Comet's wide-decimal xxhash64 kernel because the pinned upstream kernel differs.
• Remove the wide-decimal guards from typed Dataset and shuffle-input conversion: eligible converted inputs now use native shuffle. Preserve metadata-sensitive conversion restrictions.
• Cover scalar/nested NULLs, signed-byte references, partition parity, mixed-shuffle joins and AQE, with benchmarks and support documentation.

How are these changes tested?

Head 8b1d26a32 merges main 78b0eb0df, including #6740's Spark 4.2 timestamp fix. Current CI is running without failures; lint, Rust tests and native build pass; Spark 4.1 suites and TPC-H/TPC-DS checks are running. Spark SQL and Iceberg checks are skipped; selected broader coverage is still needed before the merge queue. Local formatting and fixture-parser checks pass.

Historical same-feature CI covered mixed shuffle on Spark 4.1/4.2 and Spark 4.1 SQL. Its Spark 4.2 expression failure is addressed by the main update, with the new-profile verdict pending. Historical benchmarks at 27faa50ed measured 5.0–14.2x Murmur3 and 2.1–15.8x xxhash64 improvements against the pre-optimization native decimal-array path; whole-query native array hashing averaged 83–89 ms versus 159–176 ms with Spark hash fallback. Decimal algorithms are unchanged; these are historical local workload measurements. Benchmark source.

[sunchao]

Reviewed all 17 changed files against base 78b0eb0df342e631a619c4951fd9b531f0375a3e at head 8b1d26a3207b0c8fe7fc80a9782ed750bc0907f8. The PR is not a draft. Snapshot and live discussion checks contained no existing reviews, comments, or threads.

Used review-comet-pr, review-comet-expression-pr, and review-comet-shuffle-pr, plus their relevant contributor documentation.

No introduced P1/P2 issues found within this review.

Summary

• Prior state and problem: Native wide-decimal hashing used fixed-width little-endian bytes, differing from Spark. Mixed native/JVM shuffles could place matching keys in different partitions and lose join matches. SQL hash and xxhash64 declined wide-decimal inputs.
• Design approach: A shared helper hashes the unscaled value’s minimal signed big-endian representation. Typed decimal-list loops avoid per-element slicing. The PR then removes the decimal-specific expression and shuffle restrictions.
• Correctness: Checked sign-byte trimming, zero, negative values, precision boundaries, null handling, slices, and seed chaining. A disposable harness using the head’s encoding helper and Murmur3 implementation passed 75,565 comparisons against Java BigInteger and Spark’s Java hash functions. Exact-head CI passed the added nested-value, partition-parity, mixed-shuffle, and AQE tests.
• Compatibility analysis: Spark’s interpreted and generated decimal hashing agree across supported versions 3.4.3, 3.5.9, 4.0.4, 4.1.3, and 4.2.0. Precision at most 18 retains unscaled-long hashing. Comparison with branch-1.1 at e9efd9f764ee0a59b7898ff028d6985d4a7a28e1 confirms the intended encoding correction and wider native eligibility.
• Key design decisions: Keeping wide decimals on Comet’s xxhash64 kernel is justified: DataFusion 55.2.0 still hashes their fixed-width little-endian bytes. Recursive eligibility checks also prevent nested wide decimals from reaching that upstream path. String-conversion restrictions and nested-partitioning configuration gates remain intact.
• Implementation sketch: hash_decimal! trims redundant sign bytes from a stack buffer. Scalar and list hashing reuse it. HashUtils accepts wide decimals, and CometShuffleExchangeExec removes the corresponding conversion guards. Tests, benchmarks, and support documentation accompany these changes.
• Performance: The list specialization removes observable allocation and dispatch work without introducing a new buffer or reservation. Benchmarks cover scalar controls, list variants, slices, and several null densities. The reported speedups are historical author measurements, not measurements reproduced in this review. No evidence-backed P1/P2 performance regression was identified.
• Design: Fixing the shared hash encoding addresses both SQL expressions and shuffle partitioning consistently. Removing the obsolete fallback logic simplifies routing without changing unrelated shuffle policies.
• Abstraction & complexity: The small helper fits the existing macro structure and shares the encoding between both hash algorithms. The typed-list specialization reuses existing null and offset handling. No P1/P2 abstraction or complexity issue was identified.
• Behavioral changes worth calling out: Native wide-decimal hash values and partition assignments intentionally change to match Spark. SQL hashing of precision 19–38 decimals, including nested decimals, becomes native. Eligible typed-Dataset and converted-row shuffles can now use native shuffle with these keys.
• Suggested improvements: Obtain the broader Spark SQL verdict before queueing, using run-spark-4.1-tests or the corresponding local suite. No additional code change met the P1/P2 reporting bar.

Exact-head CI: 28 successful checks, 15 skipped, and none unfinished. Run 38115070155 passed Rust tests, native build, Spark 4.1 expression/shuffle/execution/scan suites, and TPC-H/TPC-DS checks. Logs confirm the new decimal tests actually ran. Spark SQL, Iceberg, macOS, and benchmark checks were skipped. Other supported Spark profiles had compilation/lint coverage, not runtime-suite coverage.

Validation limits: The full local Rust test attempt failed during dependency resolution because the configured registry lacks DataFusion 55.2.0. JVM suites and benchmarks were not rerun locally. End-to-end validation therefore relies on exact-head CI, supplemented by source comparison and the independent local hash checks. Project code was unchanged, and no GitHub state was modified.