DDIR server: columnar exports and export taps

[frankmcsherry]

Today, with the corgi backend, every export leaves its program as Corgi columns, is converted to rows, and is re-arranged as a row trace. With Server::set_columnar_exports(true), exports stay columnar instead: they are arranged as traces of Corgi chunks, and rows are made only when something reads them. Every reader goes through one path, Server::published → Published::import_rows:

• snapshot (peek);
• another program's import;
• bind;
• the server binary's tail.

It is on by default for the corgi backend (second commit). set_columnar_exports(false), or DDIR_COLUMNAR_EXPORTS=0 for the ddir-server binary, keeps row traces. With it on by default, every existing corgi-backend test now runs through columnar exports; all pass.

Server::set_export_taps(true) (together with columnar exports) keeps each export's new batches on each worker, for an embedding program to drain as a change stream without a snapshot dataflow:

• take_changes;
• for_each_change_batch, which drains in bounded row batches.

Worldgen's viewer and its shared-album server consume their DDIR programs this way.

Measured on the same binary with columnar exports off and then on: corgi backend, three pairs, order reversed in the middle pair, M4. The tour and SCC programs had their inspects removed. Medians, off → on, with the number of pairs (of 3) where on was worse:

Workload	Workers	Initial tick	Ten churn ticks	Peak footprint
tour 5k/10k	1	16.98 → 9.50 ms (−44%, 0/3)	6.71 → 6.01 (−10%, 0/3)	40.0 → 20.0 MiB (−50%, 0/3)
tour 5k/10k	4	5.49 → 3.36 (−39%, 0/3)	4.06 → 3.68 (−9%, 0/3)	51.4 → 34.8 (−32%, 0/3)
tour 50k/100k	1	205.5 → 125.9 (−39%, 0/3)	330.3 → 317.0 (−4%, 0/3)	279.7 → 153.1 (−45%, 0/3)
tour 50k/100k	4	65.6 → 40.7 (−38%, 0/3)	138.0 → 130.4 (−6%, 0/3)	331.9 → 222.2 (−33%, 0/3)
SCC	1 / 4	+0.8% / 0%	0% / −0.2%	−0.5% / −5% (noisy)
AoC runner (wall)	1 / 4	0% / +0.5%

• Tour: every case separates cleanly. The saving is the conversion to rows and the second arrangement, which the tour pays on every export.
• SCC: it exports one small collection, so there is little to save.
• AoC: mostly process start-up and inspect output.
• Worldgen's DDIR tour (earlier, on the spike): peak fell from 1.3 to 0.45 GB with the system allocator.
• Outputs: every timed run's peek output matched one oracle, agreed across vec, corgi rows and corgi columnar at 1 and 4 workers. All 33 AoC answers were correct in every run.

Tests.

• New server_columnar_exports.rs: columnar and row exports read identically through snapshot, import and bind, including a counter advanced only by its binding.
• server_change_batches.rs: bounded draining matches a full drain and consumes each change once.
• cargo test --release -p interactive -p ddir-server: 136 passed, 5 ignored (all ignored before this change too).

🤖 Generated with Claude Code