RowFilter zone-map pruning fetches each candidate chunk's full segment over HTTP just to check its stats

[dfa1]

Summary

ScanIterator#canPruneChunk decides whether a chunk can be skipped by reading that chunk's own embedded stats via readFlatStats, which calls file.rawSegment(spec) on the chunk's entire data segment, then peeks at a small stats footer embedded at the tail (segLen - 4 - fbLen to segLen, per the comment above that code). For VortexReader (memory-mapped), fetching "the whole segment" to read its last few bytes is free — zero-copy, no I/O proportional to size. For VortexHttpReader, the same call is a real HTTP Range request for the segment's full byte range: checking whether a chunk is prunable costs as much bandwidth as decoding it outright.

This means RowFilter-driven zone-map pruning saves real HTTP bandwidth only when the filter column happens to be small per chunk (e.g. dict-encoded with few distinct values) — not in general, and not for the common case of a plain numeric column. Given #379 (cascade-selected encoders drop min/max, forcing cascading(0) for pruning to work at all), the column being pruned on is now also forced to be uncompressed, which is exactly the case where this issue bites hardest — plain 8-byte values, one full-segment fetch per chunk in the file, matched or not.

Reproduction

2,000,000-row file (timestamp:i64:series(...), symbol, price, volume; cascading(0) so stats stay populated per #379), single-symbol-sized time-range filter on timestamp covering ~2 of 31 chunks (RowFilter.gte("timestamp", ...).and(RowFilter.lte("timestamp", ...))), read via VortexHttpReader:

• Expected: a handful of small requests for the ~2 matching chunks' price data, plus cheap stats checks for the other ~29.
• Observed (from vortex-server's request log): 31 separate large GET requests, one per chunk in the file, each ~524 KB — i.e. every chunk's full segment was fetched, several of them fetched more than once (duplicate identical ranges logged back-to-back). Total bytes served: 49% of the 53 MB file, for a query that should have touched a couple of percent.

Where this lives

reader/src/main/java/io/github/dfa1/vortex/reader/ScanIterator.java:

• canPruneChunk (private, ~line 831) → readFlatStats(Layout flat) (private) → file.rawSegment(spec) where spec is the chunk's own full data-segment SegmentSpec.
• This is shared code between VortexReader and VortexHttpReader (both implement VortexHandle) — the method has no awareness that rawSegment is free on one backend and a real network fetch on the other.

There is a genuinely cheap alternative already in the same class: ScanIterator#columnZoneStats reads the file's dedicated zone-map table (vortex.stats/vortex.zoned, one small segment covering every chunk's min/max/sum/null-count at once — see decodeZoneTable) — but canPruneChunk never consults it, always going through the expensive per-chunk readFlatStats path instead.

Impact

Not a crash, not incorrect results — purely a silent performance/cost regression specific to remote/HTTP-backed reads. Since VortexHttpReader is one of vortex-java's headline capabilities (partial reads over plain HTTP Range requests, no query service required), this significantly undercuts the pruning story for exactly the workload it's meant to serve: a filtered scan over a remote file ends up transferring close to the same bytes as scanning the whole thing, whenever the filtered/projected column isn't already small per chunk.

Suggested fix direction (not implemented here)

Have canPruneChunk prefer the zone-map table path (decodeZoneTable, the same one columnZoneStats uses) when one exists, falling back to per-chunk embedded stats only when no zone-map table is present (e.g. a file with enableZoneMaps=false, or one written by another implementation without a stats table). That would make a pruning check one small shared read per column instead of one full-segment read per chunk.

How this was found

Found while building a demo (vortex-server/vortex-demo/vortex-fakedata-generator, see #377) showing VortexHttpReader's partial-fetch story over HTTP. After working around #378 (globalDict) and #379 (cascade stats), a numeric time-range filter still fetched roughly half the file; the server's own request log made the full-segment-per-chunk pattern directly visible.

🤖 Filed via Claude Code

https://claude.ai/code/session_01P4ijFsGW1MHEcGiu26vNzi