DDIR server: tail <export> as corgi, a columnar subscription

[frankmcsherry]

tail <export> as corgi streams a columnar export as binary frames instead of text lines. It is the subscription a remote client would use. Plain tail and tail … as rows are unchanged.

Frames. Each frame is a header line <id> frame <kind> <nbytes>\n followed by exactly nbytes of payload. On TCP and stdin they follow each other on the stream; on WebSocket, each frame is one binary message holding both. Integers are little-endian.

Kind	When	Payload
schema	once, before any data	text: export=<name> encoding=corgi-container/1 time=u64 diff=i64 data=worker:u64-le,container
data	per batch, per worker	u64 worker, then one batch in the container format workers already exchange between processes (corgi/bytes.rs)
progress	when the subscription's frontier advances	u64 upper: every update before upper has been sent, from every worker

How it works.

• Encoding. Each worker encodes its share of each imported batch straight from the export's Corgi chunks, with no rows built (Published::import_encoded, backend::corgi::export_containers), and sends it to worker 0. Workers hold disjoint keys, so a tick's data frames together are its changes.
• Replay. The frames before the ok replay the export's compacted contents, as text tail does.
• Row traces. A row trace has no columnar form, because an untyped variant has no shape, so as corgi on one is refused with err. Text tail reads any trace. Since DDIR server: columnar exports and export taps #913, corgi exports are columnar by default.
• Slow clients. A client more than 256 MiB behind gets err client too slow and end, and that subscription stops sending.

Measured in one run at 4 workers, tailing the tour's scored export (tour with inspect removed, 50k nodes / 100k edges, 500 edges churned per tick). Client time includes receiving and decoding every row.

	Text tail	as corgi
Replay, 99,997 rows	9.83 MB, 66.2 ms	5.60 MB (−43%), 28.9 ms
Ten churn ticks, 10,000 rows	0.99 MB, 26.1 ms	0.57 MB, 23.2 ms

Tests. New server/tests/columnar_tail.rs, at 1 and 4 workers:

• a subscription started mid-stream decodes frames that match text tail update for update, across retractions, lists and sums;
• the decoded frames sum to peek after each tick;
• as corgi on a row trace is refused.

cargo test --release -p interactive -p ddir-server: 141 passed, 6 ignored (the 5 ignored before this change, plus the measurement above). The WebSocket binary path has no test.

Open choices:

• whether schema should carry declared DDIR types (Int vs F64, variant names), which are erased today;
• progress only once per tick;
• dropping a stopped slow subscription's dataflow; today it stays until stop or the end of the session, since dropping it on worker 0 alone isn't safe;
• an Arrow adapter;
• per-worker streams instead of gathering to worker 0.

🤖 Generated with Claude Code