fix(writer,reader): stop zone-map pruning from silently disabling itself

CHANGELOG.md

@@ -5,6 +5,15 @@ All notable changes to **vortex-java** are documented here.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [Unreleased]
+
+### Fixed
+
+- Zone-map `MIN`/`MAX` is now per-zone nullable instead of all-or-nothing: one chunk without stats (e.g. an all-null chunk) used to blank `MIN`/`MAX` for every chunk in the column, silently disabling zone-map pruning for the whole column rather than just that chunk — matching Rust, which always wraps zone-map stats nullable (`vortex-layout/src/layouts/zoned/schema.rs`). ([#378](https://github.com/dfa1/vortex-java/issues/378))
+- `FrameOfReferenceEncodingEncoder` and `DictEncodingEncoder`'s primitive path now surface `MIN`/`MAX` stats instead of hardcoding `null`; combined with cascading compression (`WriteOptions.cascading`/`allowedCascading`), a numeric column whose chunks won the FOR or Dict competition previously lost zone-map pruning for those chunks with no indication. ([#379](https://github.com/dfa1/vortex-java/issues/379))
+- `ScanIterator`'s zone-map pruning check now decodes the column's compact zone-map table (once per scan, cached) instead of fetching each chunk's own full data segment just to read its embedded stats — over HTTP, checking whether a chunk could be pruned used to cost exactly as much as fetching it outright, so pruning saved nothing for the checked column itself (only for other, unfiltered columns in the same row window). ([#380](https://github.com/dfa1/vortex-java/issues/380))
+- `MaskedEncodingEncoder` (nullable columns) now computes `MIN`/`MAX` from only the row-valid elements instead of trusting whichever inner encoder ran over the dense, placeholder-filled values array — a `long[]`/`int[]`/... has no way to represent "no value", so invalid slots carry a placeholder (commonly `0`) that was silently folding into the reported stats; a column with any nulls could report a wrong non-null `MIN`/`MAX` (e.g. `0`) instead of excluding those rows, corrupting both pruning and any consumer trusting the zone-map's actual values. Found while verifying the #378 fix against the real Rust reader on the same data. ([#381](https://github.com/dfa1/vortex-java/pull/381))
+
 ## [0.14.1] — 2026-09-06
 
 ### Fixed

core/src/main/java/io/github/dfa1/vortex/core/compute/PrimitiveArrays.java

@@ -15,8 +15,9 @@
 ///
 /// [#toLongs(Object, PType, EncodingId)] and [#fromLongs(long[], PType, SegmentAllocator)] are
 /// inverses: the first widens any 8–64 bit integer array to a `long[]`, the second writes a
-/// `long[]` back to a little-endian off-heap segment of the target width. Floating-point ptypes
-/// are not handled here — they reinterpret to raw bits or take type-specific encode paths instead.
+/// `long[]` back to a little-endian off-heap segment of the target width. Both are integer-only —
+/// floating-point ptypes reinterpret to raw bits or take type-specific encode paths instead.
+/// [#compact(PType, Object, boolean[])] covers every primitive ptype, integer and floating alike.
 public final class PrimitiveArrays {
 
     private PrimitiveArrays() {
@@ -109,4 +110,97 @@ public static MemorySegment fromLongs(long[] longs, PType ptype, SegmentAllocato
         }
         return seg;
     }
+
+    /// Copies only the elements at `true` positions in `mask` from `data`, preserving `ptype`'s
+    /// storage array shape (`byte[]` for I8/U8, `short[]` for I16/U16/F16, `int[]` for I32/U32,
+    /// `long[]` for I64/U64, `float[]` for F32, `double[]` for F64). Covers every primitive ptype,
+    /// unlike [#toLongs(Object, PType, EncodingId)]/[#fromLongs(long[], PType, SegmentAllocator)],
+    /// which are integer-only.
+    ///
+    /// Used to strip a nullable column's dense, placeholder-filled values down to its real
+    /// (row-valid) values before an operation that must not see the placeholder — a `byte[]`/
+    /// `int[]`/... has no way to represent "no value", so a nullable column's invalid slots carry
+    /// some placeholder chosen by the caller, commonly `0`; running e.g. a min/max scan over the
+    /// raw array would fold that placeholder in as if it were real data.
+    ///
+    /// @param ptype the logical primitive type of `data`
+    /// @param data  the value array; its runtime type must match `ptype`
+    /// @param mask  per-element validity, aligned with `data` (`true` = keep)
+    /// @return a new array of the same runtime type as `data`, holding only the `true`-masked elements
+    public static Object compact(PType ptype, Object data, boolean[] mask) {
+        int n = 0;
+        for (boolean v : mask) {
+            if (v) {
+                n++;
+            }
+        }
+        return switch (ptype) {
+            case I8, U8 -> {
+                byte[] src = (byte[]) data;
+                byte[] out = new byte[n];
+                int j = 0;
+                for (int i = 0; i < src.length; i++) {
+                    if (mask[i]) {
+                        out[j++] = src[i];
+                    }
+                }
+                yield out;
+            }
+            case I16, U16, F16 -> {
+                short[] src = (short[]) data;
+                short[] out = new short[n];
+                int j = 0;
+                for (int i = 0; i < src.length; i++) {
+                    if (mask[i]) {
+                        out[j++] = src[i];
+                    }
+                }
+                yield out;
+            }
+            case I32, U32 -> {
+                int[] src = (int[]) data;
+                int[] out = new int[n];
+                int j = 0;
+                for (int i = 0; i < src.length; i++) {
+                    if (mask[i]) {
+                        out[j++] = src[i];
+                    }
+                }
+                yield out;
+            }
+            case I64, U64 -> {
+                long[] src = (long[]) data;
+                long[] out = new long[n];
+                int j = 0;
+                for (int i = 0; i < src.length; i++) {
+                    if (mask[i]) {
+                        out[j++] = src[i];
+                    }
+                }
+                yield out;
+            }
+            case F32 -> {
+                float[] src = (float[]) data;
+                float[] out = new float[n];
+                int j = 0;
+                for (int i = 0; i < src.length; i++) {
+                    if (mask[i]) {
+                        out[j++] = src[i];
+                    }
+                }
+                yield out;
+            }
+            case F64 -> {
+                double[] src = (double[]) data;
+                double[] out = new double[n];
+                int j = 0;
+                for (int i = 0; i < src.length; i++) {
+                    if (mask[i]) {
+                        out[j++] = src[i];
+                    }
+                }
+                yield out;
+            }
+        };
+    }
 }

core/src/test/java/io/github/dfa1/vortex/core/compute/PrimitiveArraysTest.java

@@ -170,4 +170,131 @@ private static long readElement(MemorySegment seg, PType ptype, int i) {
             default -> throw new IllegalArgumentException("not an integer ptype: " + ptype);
         };
     }
+
+    @Test
+    void compact_i8_keepsOnlyValidElements() {
+        // Given a leading and a trailing invalid slot around real values
+        byte[] data = {9, 10, 20, 9};
+        boolean[] mask = {false, true, true, false};
+
+        // When
+        byte[] result = (byte[]) PrimitiveArrays.compact(PType.I8, data, mask);
+
+        // Then
+        assertThat(result).containsExactly((byte) 10, (byte) 20);
+    }
+
+    @Test
+    void compact_i16_keepsOnlyValidElements() {
+        // Given
+        short[] data = {9, 10, 20, 9};
+        boolean[] mask = {false, true, true, false};
+
+        // When
+        short[] result = (short[]) PrimitiveArrays.compact(PType.I16, data, mask);
+
+        // Then
+        assertThat(result).containsExactly((short) 10, (short) 20);
+    }
+
+    @Test
+    void compact_f16_keepsOnlyValidElements() {
+        // Given — F16 shares I16/U16's short[] storage shape
+        short[] data = {9, 10, 20, 9};
+        boolean[] mask = {false, true, true, false};
+
+        // When
+        short[] result = (short[]) PrimitiveArrays.compact(PType.F16, data, mask);
+
+        // Then
+        assertThat(result).containsExactly((short) 10, (short) 20);
+    }
+
+    @Test
+    void compact_i32_keepsOnlyValidElements() {
+        // Given
+        int[] data = {9, 10, 20, 9};
+        boolean[] mask = {false, true, true, false};
+
+        // When
+        int[] result = (int[]) PrimitiveArrays.compact(PType.I32, data, mask);
+
+        // Then
+        assertThat(result).containsExactly(10, 20);
+    }
+
+    @Test
+    void compact_i64_keepsOnlyValidElements() {
+        // Given
+        long[] data = {9L, 10L, 20L, 9L};
+        boolean[] mask = {false, true, true, false};
+
+        // When
+        long[] result = (long[]) PrimitiveArrays.compact(PType.I64, data, mask);
+
+        // Then
+        assertThat(result).containsExactly(10L, 20L);
+    }
+
+    @Test
+    void compact_f32_keepsOnlyValidElements() {
+        // Given
+        float[] data = {9f, 10f, 20f, 9f};
+        boolean[] mask = {false, true, true, false};
+
+        // When
+        float[] result = (float[]) PrimitiveArrays.compact(PType.F32, data, mask);
+
+        // Then
+        assertThat(result).containsExactly(10f, 20f);
+    }
+
+    @Test
+    void compact_f64_keepsOnlyValidElements() {
+        // Given
+        double[] data = {9.0, 10.0, 20.0, 9.0};
+        boolean[] mask = {false, true, true, false};
+
+        // When
+        double[] result = (double[]) PrimitiveArrays.compact(PType.F64, data, mask);
+
+        // Then
+        assertThat(result).containsExactly(10.0, 20.0);
+    }
+
+    @Test
+    void compact_allValid_returnsEveryElement() {
+        // Given no placeholder slots at all
+        long[] data = {1L, 2L, 3L};
+        boolean[] mask = {true, true, true};
+
+        // When
+        long[] result = (long[]) PrimitiveArrays.compact(PType.I64, data, mask);
+
+        // Then
+        assertThat(result).containsExactly(1L, 2L, 3L);
+    }
+
+    @Test
+    void compact_allInvalid_returnsEmptyArray() {
+        // Given — no real value exists anywhere, e.g. an all-null zone/chunk
+        long[] data = {0L, 0L, 0L};
+        boolean[] mask = {false, false, false};
+
+        // When
+        long[] result = (long[]) PrimitiveArrays.compact(PType.I64, data, mask);
+
+        // Then
+        assertThat(result).isEmpty();
+    }
+
+    @Test
+    void compact_empty_returnsEmptyArray() {
+        // Given no elements at all
+        // When
+        long[] result = (long[]) PrimitiveArrays.compact(PType.I64, new long[0], new boolean[0]);
+
+        // Then
+        assertThat(result).isEmpty();
+    }
 }

integration/src/test/java/io/github/dfa1/vortex/integration/JavaWritesRustReadsIntegrationTest.java

@@ -536,6 +536,51 @@ void javaWriter_jniReader_zoneMapped_multipleZones(@TempDir Path tmp) throws IOE
         assertThat(decodedVals).containsExactly(vals);
     }
 
+    @Test
+    void javaWriter_jniReader_zoneMapped_allNullChunkStillRoundTrips(@TempDir Path tmp) throws IOException {
+        // Given — #378 regression: a nullable I64 column across 3 zone-mapped chunks where the
+        // middle chunk is entirely null (no chunk-level min/max to record). Before the fix, one
+        // stats-less chunk anywhere in the column dropped MIN/MAX from the zone-map for every
+        // chunk, not just the offending one; Rust's own zoned schema always wraps MIN/MAX nullable
+        // per zone (vortex-layout/src/layouts/zoned/schema.rs), so an all-or-nothing Java writer was
+        // silently out of step with the format it claims to speak, even though every file it wrote
+        // still parsed. This pins that the fixed writer round-trips through the real Rust reader:
+        // it isn't enough that vortex-java's own reader tolerates the shape it now writes.
+        Path file = tmp.resolve("java_zoned_null_chunk.vtx");
+        DType.Struct schema = new DType.Struct(
+                List.of(ColumnName.of("v")), List.of(new DType.Primitive(PType.I64, true)), false);
+        WriteOptions zoneMapped = new WriteOptions(4, true, 0.90, 0, false, false, MemorySize.ofMiB(256), Map.of());
+        Long[] data = {
+                0L, 1L, 2L, 3L,
+                null, null, null, null,
+                100L, 101L, 102L, 103L};
+        try (var ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE);
+             var sut = VortexWriter.create(ch, schema, zoneMapped)) {
+            for (int start = 0; start < data.length; start += 4) {
+                sut.writeChunk(Map.of(ColumnName.of("v"), Arrays.copyOfRange(data, start, start + 4)));
+            }
+        }
+
+        // Then — Rust parses the zone-map layout and returns every row, nulls included
+        String uri = file.toAbsolutePath().toUri().toString();
+        DataSource ds = DataSource.open(SESSION, uri);
+        Scan scan = ds.scan(ScanOptions.of());
+        var values = new ArrayList<Long>();
+        while (scan.hasNext()) {
+            Partition partition = scan.next();
+            try (ArrowReader reader = partition.scanArrow(ALLOCATOR)) {
+                while (reader.loadNextBatch()) {
+                    VectorSchemaRoot root = reader.getVectorSchemaRoot();
+                    BigIntVector vec = (BigIntVector) root.getVector("v");
+                    for (int i = 0; i < root.getRowCount(); i++) {
+                        values.add(vec.isNull(i) ? null : vec.get(i));
+                    }
+                }
+            }
+        }
+        assertThat(values).containsExactly(data);
+    }
+
     @Test
     void javaWriter_jniReader_i32Column(@TempDir Path tmp) throws IOException {
         // Given