RowFilter zone-map pruning silently no-ops for Utf8 columns under the default globalDict=true

[dfa1]

Summary

WriteOptions.defaults()/cascading(depth) enable globalDict=true by default. For a Utf8 column that qualifies as a global-dict candidate, this silently defeats RowFilter-driven zone-map pruning on that column — every chunk is scanned regardless of whether it could possibly match, even though the column's zone-map statistics are correctly computed and available.

Root cause

ScanIterator#canPruneChunk (via the RowFilter.Column case) reads a chunk's stats through chunk.layoutFor(col) → readFlatStats(flat), which decodes the per-flat-node embedded stats attached directly to that chunk's own encoded ArrayNode (the statsMin/statsMax an EncodingEncoder sets on its EncodeResult, e.g. in DictEncodingEncoder/VarBinEncodingEncoder).

When globalDict=true, VortexWriter.flushDictColumns() → writeGlobalDictUtf8Column() takes over writing the column entirely, bypassing the normal per-chunk cascade encode path. The per-chunk flat node ends up holding just integer dictionary codes, and — as far as I can tell — this path never populates statsMin/statsMax with the domain string values on that node. canPrune() requires non-null min/max to prove no row can match, so it always returns false (never prune) once those are absent.

This is separate from the zone-map stats table (vortex.stats/vortex.zoned, read by ScanIterator#columnZoneStats), which stays correctly populated with proper per-zone string min/max regardless of globalDict. So columnZoneStats("symbol") returns correct, useful stats, while RowFilter.eq("symbol", ...) pruning silently does nothing — two different stats sources that have drifted out of sync for this configuration.

Reproduction

Confirmed empirically on a 2,000,000-row synthetic dataset (timestamp/symbol/price, 30 symbols, sorted by symbol, chunkSize=65_536, WriteOptions.cascading(3)):

globalDict	Chunks yielded for RowFilter.eq("symbol", "SYM015")	Rows returned
true (default)	31 of 31 (all)	2,000,000 (all)
false	2 of 31	131,072

iter.columnZoneStats("symbol") returns correct, precise per-zone min/max in both cases — confirming the zone-map table itself is fine; only the pruning check's stats source is affected.

Impact

Silent, not a crash — the scan still returns correct results, just without the pruning speedup. Since globalDict=true is the default and WriteOptions.defaults()/cascading(depth) both enable it, this means filtered scans on low/medium-cardinality Utf8 columns get none of the zone-map pruning benefit by default — a real performance cliff that's easy to hit unknowingly (a user writing with defaults and filtering on a dict-friendly string column, e.g. a symbol/category/status field, gets a full scan every time).

Suggested fix directions (not investigated in depth)

• Have the global-dict write path also populate real per-chunk embedded stats (domain values, not codes) on the flat node it emits, matching what the normal per-chunk DictEncodingEncoder path does.
• Or: have ScanIterator#canPruneChunk consult the zone-map stats table (the same one columnZoneStats reads) instead of / in addition to per-flat-node embedded stats, when present.

How this was found

Found while building a demo (vortex-server/vortex-demo, see #377) showing VortexHttpReader's partial-fetch story over HTTP — a single-symbol filtered scan was unexpectedly fetching ~55% of the file instead of the few percent expected from zone-map pruning. The demo works around it via WriteOptions#withGlobalDict(false).

🤖 Filed via Claude Code

https://claude.ai/code/session_01P4ijFsGW1MHEcGiu26vNzi