Add vortex-server + vortex-demo: live HTTP range demo - #377
Merged
Conversation
This was referenced Sep 11, 2026
Closed
Closed
dfa1
force-pushed
the
feature/vortex-server-demo
branch
from
September 12, 2026 17:20
4461d13 to
23a53b1
Compare
…ange demo Two new standalone, non-published modules: - vortex-server: a minimal, dependency-free object-storage HTTP server (GET with byte-range support, HEAD, PUT, a plain-text object listing, a /_stats endpoint) standing in for real object storage in a demo. Exists because the JDK's own jwebserver (SimpleFileServer) does not implement Range requests at all -- VortexHttpReader's first segment fetch against it throws VortexException since the returned byte count never matches the requested range. Path-traversal guarded and covered by regression tests using literal (non-normalized) URIs, verified against both GET and PUT with raw sockets during development. - vortex-demo: generates a synthetic tick dataset (timestamp/symbol/ price/volume, sorted by symbol), uploads it to a vortex-server (embedded by default, or an already-running one given on the command line for a two-process live demo), then runs a filtered + projected VortexHttpReader scan against it and reports how many bytes were actually fetched over HTTP against the object's full size. Along the way, found a real gap in the reader (not fixed here): the default WriteOptions.globalDict=true silently defeats RowFilter zone-map pruning for Utf8 columns, because ScanIterator#canPruneChunk reads per-chunk embedded stats that the global-dict write path never populates -- the separate zone-map stats table stays correct and is readable via ScanIterator#columnZoneStats, but pruning doesn't consult it. The demo works around it via WriteOptions#withGlobalDict(false). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P4ijFsGW1MHEcGiu26vNzi
…ient The server is conceptually part of the HTTP range demo, not a separate top-level concern -- demo/ is now a small aggregator (vortex-demo-parent, packaging=pom) with two children: demo/server (vortex-server, unchanged) and demo/client (vortex-demo, unchanged). Artifact coordinates, package names, and behavior are unchanged; only the directory layout moved. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P4ijFsGW1MHEcGiu26vNzi
…er vortex.demo.* Splits the client's ad-hoc synthetic-data generation into its own standalone CLI tool, vortex-fakedata-generator: given a compact column-description grammar (name:type:generator(args), e.g. "price:f64:range(50,150)"), inspired by PostgreSQL's generate_series, it writes a Vortex file of synthetic data. Generators: series(start,step) (the generate_series-style arithmetic progression), range(min,max), normal(mean,stddev), enum(prefix,count) (random categorical labels), constant(value), and bool(). A --sort-by flag clusters rows by a given column afterward, independent of how that column's values were generated -- useful for zone-map-pruning-friendly demo data. demo/client (vortex-demo) no longer generates its own data: it now takes an existing .vortex file via --file, uploads it to a vortex-server (embedded by default, or --server for an already-running one), and scans it with configurable --filter-column/--filter-value/--project. All three demo modules' packages now nest under io.github.dfa1.vortex.demo: vortex-server -> io.github.dfa1.vortex.demo.server, vortex-fakedata-generator -> io.github.dfa1.vortex.demo.fakedata, vortex-demo -> io.github.dfa1.vortex.demo.client. Reinforces that this is all demo/presentation tooling, not part of the library's public API surface. Full three-tool pipeline verified end to end: generate -> serve -> scan, 2M rows sorted by symbol, single-symbol filter fetches 2.09% of the file over HTTP. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P4ijFsGW1MHEcGiu26vNzi
Step-by-step for presenting: build once, two-terminal live run (server log visibly showing small Range fetches vs. one big download), a one-terminal convenience mode, and how to customize the filter/schema for a different story. Cross-links issue #378 (the globalDict pruning gap this demo surfaced) since it explains why the generator forces globalDict=false. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P4ijFsGW1MHEcGiu26vNzi
A throttled (max 10 redraws/sec), in-place terminal progress bar printed to stderr during the chunk-writing loop -- the I/O/encoding- bound phase, unlike in-memory column materialization which is near- instant even at millions of rows. Shows rows written, elapsed time, and an ETA extrapolated from the current completion fraction. Verified against a 50M-row generation (~11s) to see a real non-zero ETA. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P4ijFsGW1MHEcGiu26vNzi
A virtual-thread background poller hits the server's /_stats endpoint every 100ms while the scan runs, printing an in-place-updating "bytes downloaded so far" line to stderr. Verified against a 20M-row/245MB file: the counter genuinely climbs through real intermediate values (0.8% -> 11.7% -> 26.8% -> 43.9% -> final 59.2%) rather than just jumping straight to the final number -- the point being visible on stage that the scan stops well short of the file's full size, not just told after the fact. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P4ijFsGW1MHEcGiu26vNzi
…me-range filter Now that #378/#379/#380 are fixed upstream, the workarounds and the --sort-by knob are no longer needed: - FakeDataGenerator no longer materializes every column fully in memory before writing (Object[] columnArrays + a post-hoc sort permutation). Generation is now truly one-chunk-at-a-time streaming: ColumnMaterializer.materialize takes a rowOffset so a `series(...)` column stays a correct arithmetic progression across chunk boundaries. Verified against 200,000,000 rows with a 512MB heap cap: completes in ~80s, no OutOfMemoryError (the old materialize-everything path OOM'd on this same input). The progress bar now genuinely tracks the whole run instead of jumping straight to 100% after a silent materialization phase. - --sort-by is gone: real data never arrives pre-sorted by whatever column a later query filters on, so faking that clustering misrepresented the workload. The withGlobalDict(false) workaround for #378 is gone too, now that the fix landed. - vortex-demo's default filter switches from symbol equality (meaningless without --sort-by -- an enum() column is scattered uniformly across every chunk by design) to a numeric range on timestamp: a series(...) column is naturally ordered by row position with no sorting needed, matching real append-order ingestion. New --filter-min/--filter-max flags replace --filter-value. End-to-end verified on the default 2,000,000-row example: 5.55% of the file fetched over HTTP, real compression (cascading(3), the generator's own default), zero workarounds. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P4ijFsGW1MHEcGiu26vNzi
…flow) lastDrawNanos started at Long.MIN_VALUE. The very first #update call computed `nowNanos - lastDrawNanos` -- a moderate positive nanoTime() value minus the most negative possible long -- which silently overflows a signed long and wraps to a negative Duration. A negative duration always compares as "less than the 100ms redraw interval", so the first (and every subsequent, since lastDrawNanos never advances away from MIN_VALUE either) non-final update got throttle-skipped for the entire run. Only the final call ever drew, since it bypasses the throttle check entirely -- exactly matching what looked like "the bar does nothing until the very end". Fixed by backdating the initial value by one redraw interval instead of using a sentinel, avoiding the overflow entirely (both operands stay close to System.nanoTime()'s own range). Verified empirically, not just reasoned about: polled the output file's size every second during a 200M-row run. Before the fix, it sat at 0 bytes for the entire ~65s run and jumped to ~60KB in one write at the very end. After, it grows continuously and roughly linearly throughout (0 -> 859 -> 1750 -> ... -> 60973 bytes). Also confirmed, by temporarily removing it and rerunning the same test, that an explicit System.err.flush() per redraw was never actually necessary -- PrintStream#write(String) already pushes every write through its internal encoding buffers unconditionally, and the underlying FileOutputStream has no buffering of its own to flush. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P4ijFsGW1MHEcGiu26vNzi
… progress redraws ColumnMaterializer.materializeUtf8's enum(prefix,count) generator was calling String.formatted() once per row to build a zero-padded label, even though only `count` distinct labels ever exist. Measured: ~35% of total generation time for a 200M-row run with a 30-value enum column was attributable to this one column (67s -> 44s with it removed). Precomputing the small label pool once and indexing into it per row instead: 67.3s -> 57.5s for the same run (~15% faster overall; the remaining gap vs. the no-enum baseline is real write-side cost of encoding a 4th column, not a generation-side inefficiency). Also: ProgressBar's redraws didn't erase a longer previous line (\r only returns the cursor to column 0, it doesn't clear anything), so a shrinking field -- the ETA's digit count dropping as it counts down, for instance -- left trailing characters on screen (observed live: "eta=51s" rendered as "eta=51ss"). Now pads every redraw to the longest line drawn so far. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P4ijFsGW1MHEcGiu26vNzi
Every run re-uploaded the file via PUT, even when iterating on --filter-column/--filter-min/--filter-max/--project against a file already sitting on the server from a previous run -- wasteful and slow for exactly the "try different queries" workflow this client is meant to support. --no-upload skips the PUT and queries the existing object directly; requires --server since an embedded server always starts empty. Verified: two runs against the same file, second with --no-upload and a different filter, server log shows exactly one PUT total. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P4ijFsGW1MHEcGiu26vNzi
…file argument Replaces --file/--server/--no-upload flags with two clear modes: --upload FILE SERVER_URL copies a file and exits; a query takes a remote object URL directly (no upload, no local file at all -- file size comes from VortexHandle#fileSize on the opened reader) or a local file path (embeds a server, uploads, then queries), dropping the separate --file/--server/--no-upload flag combination entirely.
A 404/connection failure isn't a syntax problem -- printing the usage block after it (as if the command line were wrong) was misleading. Only IllegalArgumentException (missing/unknown argument) shows usage now; other failures print just the error message.
…ve paths "vortex-server 8080 ." (a relative dataDir) served a correct directory listing but 404'd every single-object GET/HEAD: #resolve normalized the resolved candidate before its startsWith containment check, but never normalized dataDir itself, so a bare "." never structurally startsWith-matched a candidate that normalize() had already stripped the "." from. Normalizing dataDir to an absolute path once at start() fixes it for every relative form, not just ".".
…sizes ScanOptions#withFilter only prunes whole chunks via zone-map stats -- it never filters individual rows within a surviving chunk. Summing chunk.rowCount() therefore reported every row in every non-pruned chunk as "matched", regardless of whether it actually satisfied the filter. A price:range(50,150) filter of [500,501] -- outside the column's entire range -- was reporting all 200,000,000 rows as matched, live on the user's own machine. Compute#filteredAggregate(chunk, filter, null) runs the same filter row-by-row (fused single pass, COUNT(*)-style with no aggregate column) to get the true matched-row count. Verified against the user's real 2.4GB/200M-row file: a [100,102] filter over a uniform [50,150] price column now reports 3,996,842 matched rows, ~2% of 200M as expected, instead of 200,000,000.
price never prunes regardless of filter overlap, unlike the "no natural clustering" story that explains symbol/enum columns -- worth distinguishing since it looks the same in the demo's own output (same ~54% either way) but has a different, currently-open root cause.
…range One compact COLUMN:MIN:MAX argument instead of three separate flags to remember for a live demo; --project stays separate since it's optional. Fixes #383.
Rebasing onto main pulled in the #382 fix. Verified live: a price filter entirely outside the data's range now prunes to <1% of the file instead of the ~54% it fetched when ALP-RD reported no min/max at all. A filter that overlaps price's range still touches most chunks, but that's the column's lack of natural clustering, not the stats bug.
dfa1
force-pushed
the
feature/vortex-server-demo
branch
from
September 12, 2026 19:02
1082424 to
a141f7a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two new standalone, non-published demo modules for showing
VortexHttpReader's partial-fetch story over plain HTTP -- built for a presentation on Vortex's HTTP range-read use case.vortex-server(vortex-server.jar): a minimal, dependency-free object-storage HTTP server --GETwith byte-range support,HEAD,PUT, a plain-text object listing, and a/_statsendpoint. Exists because the JDK's ownjwebserver(SimpleFileServer) doesn't implementRangerequests at all;VortexHttpReader's first segment fetch against it throwsVortexExceptionbecause the returned byte count never matches the requested range.vortex-demo(vortex-demo.jar): generates a synthetic tick dataset (timestamp/symbol/price/volume, 2M rows sorted by symbol), uploads it to avortex-server(embedded by default, or an already-running one given on the command line for a two-process live demo), then runs a filtered + projectedVortexHttpReaderscan and reports bytes fetched over HTTP vs. the object's full size.Live demo output (single-symbol filter, one-column projection, 22.4 MB file): 3.95% of the object fetched (883 KB), 131,072 matched rows.
Notable finding (not fixed here)
While building the demo I found a real gap in the reader: the default
WriteOptions.globalDict=truesilently defeatsRowFilterzone-map pruning forUtf8columns --ScanIterator#canPruneChunkreads per-chunk embedded stats that the global-dict write path never populates, even though the separate zone-map stats table stays correct and is readable viaScanIterator#columnZoneStats. The demo works around it viaWriteOptions#withGlobalDict(false). Flagging for a possible follow-up issue/fix.Test plan
./mvnw verifygreen across the whole reactorservermodule: 10 tests, including two path-traversal regression tests (rejectsPathTraversalOnGet/rejectsPathTraversalOnPut) verified against literal, non-normalized URIs (raw sockets used during manual verification to rule out client-side../normalization masking a real gap)java -jar server/target/vortex-server.jar <port> <dir>in one terminal,java -jar demo/target/vortex-demo.jar http://127.0.0.1:<port>/in another🤖 Generated with Claude Code
https://claude.ai/code/session_01P4ijFsGW1MHEcGiu26vNzi