fix(writer,reader): stop zone-map pruning from silently disabling itself

[dfa1]

Summary

Three related bugs, one shared write-path root cause plus an independent read-path cost bug:

• RowFilter zone-map pruning silently no-ops for Utf8 columns under the default globalDict=true #378 — VortexWriter.flushZoneMaps required every chunk in a column to carry min/max stats before emitting the MIN/MAX zone-map fields at all. One stats-less chunk (e.g. an all-null chunk, common for optional string columns) silently dropped MIN/MAX for the entire column, not just that chunk. Confirmed against the real Rust reference (vortex-jni) that zone-map stats are always per-zone nullable there (vortex-layout/src/layouts/zoned/schema.rs wraps every stat .as_nullable()) — this was a genuine spec deviation, not just an internal inconsistency. Fixed to mark only the offending zone invalid, matching Rust.
• Zone-map min/max stats lost for any column encoded via the cascade (allowedCascading > 0) #379 — FrameOfReferenceEncodingEncoder and DictEncodingEncoder's primitive path hardcoded null min/max stats. Combined with the RowFilter zone-map pruning silently no-ops for Utf8 columns under the default globalDict=true #378 gate, any chunk the cascading ("better compression" / WriteOptions.cascading) path routed through FOR or Dict silently lost zone-map pruning for the whole column. Both now compute real stats via the existing shared PrimitiveEncodingEncoder.minMaxStats.
• RowFilter zone-map pruning fetches each candidate chunk's full segment over HTTP just to check its stats #380 — ScanIterator.canPruneChunk read each chunk's pruning stats via readFlatStats, which fetches that chunk's entire data segment over HTTP just to read its embedded stats footer — the same bytes a real (non-pruned) read would fetch. So checking whether a chunk could be pruned cost as much as reading it. It now decodes the column's compact vortex.stats zone-map table instead (once per column per scan, cached).

Fixed in response to review

A /code-review high pass on the initial commit surfaced two real correctness gaps, both now fixed (second commit):

1. Zone-table lookup was mis-aligned for foreign files. The RowFilter zone-map pruning fetches each candidate chunk's full segment over HTTP just to check its stats #380 fast path indexed the decoded zone table by chunk ordinal unconditionally — correct for this writer's own vortex.stats (one zone per chunk, by construction), but Rust's vortex.zoned format uses a genuinely independent, uniform zone length with no 1:1 relationship to chunk boundaries. ScanIterator now dispatches by layout id: ordinal lookup for our own format, and a zone-length containment check (bail unless a chunk's whole row range fits inside one zone) for vortex.zoned.
2. MaskedEncodingEncoder computed wrong (not just missing) stats for nullable columns. While verifying the RowFilter zone-map pruning silently no-ops for Utf8 columns under the default globalDict=true #378 fix by comparing vortex-java's zone-map output against the real Rust reader on identical data, found that a nullable column's MIN/MAX was computed over the dense, placeholder-filled values array (a long[]/int[]/... has no way to represent "no value"), so a column with any nulls could report a wrong non-null stat (e.g. 0) instead of excluding those rows — corrupting pruning and any consumer trusting the zone-map's actual values, not just losing pruning power. Now computed directly from (values, validity), excluding invalid rows.

Also applied the review's minor findings: PrimitiveEncodingEncoder.minOf/maxOf helpers remove duplicated stats-unpacking across the FOR/Dict encoders, and a test's // When result variable renamed to match convention.

Test plan

• ./mvnw test -pl reader,writer — full unit suites green
• FrameOfReferenceEncodingEncoderTest/DictEncodingEncoderTest — encode()/encodeCascade() now surface correct min/max (unordered input, so a broken read would surface a wrong value)
• WriterZoneMapTest.chunkWithoutStats_marksOnlyThatZoneInvalid — rewritten to assert the fixed per-zone-nullable behavior
• WriterZoneMapHttpPruningTest — mocks HttpClient; filters so every chunk is pruned; asserts exactly 1 additional HTTP request (the shared zone-table fetch) instead of 4 (one per chunk, pre-fix)
• ScanIteratorChunkGridTest — pins both the chunk-ordinal invariant (our own format) and the windowStart/sliceOffsetFor derivation (foreign-format containment check)
• MaskedEncodingEncoderTest — a nullable column with a leading invalid slot and a placeholder-polluted value range reports the true min/max, excluding invalid rows; an all-invalid column reports no stats at all (not the 0 placeholder)
• JavaWritesRustReadsIntegrationTest — javaWriter_jniReader_zoneMapped_allNullChunkStillRoundTrips (new), javaWriter_rustReader_masked_nullableI64, javaWriter_rustReader_masked_nullableUtf8 all pass against the real Rust reader (./mvnw verify -pl integration -am -Dit.test=...)

Closes #378, closes #379, closes #380.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Mh2Mh6vemzhJ6knfo2eoME