Skip to content

DDIR server: park idle workers while waiting for progress - #907

Merged
frankmcsherry merged 1 commit into
master-nextfrom
ddir-parking
Sep 28, 2026
Merged

frankmcsherry merged 1 commit into
master-nextfrom
ddir-parking

Conversation

@frankmcsherry

Copy link
Copy Markdown
Member

Server::tick and Server::snapshot called Worker::step in a loop even when no operator could run. They now use Timely's existing step_or_park(None). Its short idle spin covers brief gaps, and remote data or progress wakes a parked worker. There is no new API and no new policy knob. Two files, +46/−2, including a regression test where one peer is delayed.

Measured on three workloads: SCC, the worldgen tour, and the stock AoC runner (33 answers). Three paired runs each, order reversed in the middle run, M4 release build, nothing else running. Graph peaks are the server process footprint. AoC times cover the whole runner.

Workload / workers Initial ms Ten churn ticks ms Process CPU s Peak footprint MiB
scc 1 70.63 → 71.51 (+1.2%) 773.71 → 773.88 (+0.0%) 0.84 → 0.84 79.8 → 79.8
scc 4 32.43 → 32.79 (+1.1%) 375.39 → 380.63 (+1.4%) 1.63 → 1.48 (−9.2%) 77.6 → 83.5 (+7.7%)
tour 1 1540 → 1550 (+0.6%) 331.83 → 332.95 (+0.3%) 1.86 → 1.87 37.2 → 37.3
tour 4 1750 → 1730 (−1.1%) 361.94 → 362.58 (+0.2%) 3.41 → 2.23 (−34.6%) 45.3 → 46.2 (+1.9%)
AoC workers Runner elapsed s Process CPU s
1 2.12 → 2.12 2.14 → 2.14
4 2.24 → 2.11 (−5.8%) 8.54 → 2.94 (−65.6%)

At 4 workers, the CPU ranges separate for all three workloads: SCC 1.62–1.63 → 1.48 s, tour 3.40–3.45 → 2.22–2.26 s, AoC 8.54–8.55 → 2.94 s. Elapsed time moves by at most about 1.4% at the median, and one worker is flat. AoC elapsed improves in every pair.

Slower cases. SCC at 4 workers has a 7.7% higher peak footprint in all 3 pairs, and the ranges do not overlap (76.6–78.4 vs 83.3–87.3 MiB). The case for this change is CPU, not memory. SCC churn is also +1.4% at the median.

Tests. cargo test --release -p interactive: 105 passed, 5 ignored (all ignored before this change too). All 33 AoC answers match. The supported sessions and the graph references match on both backends.

Measurements and the port are by Astra (OpenAI Codex). I re-ran the tests on master-next 9cfc909.

🤖 Generated with Claude Code

Server::tick and Server::snapshot repeatedly called Worker::step even when
no operator could run. Use Timely's existing step_or_park(None) policy:
its brief idle spin absorbs short gaps; remote data/progress wakes parked
workers. No new policy knob, API change, or worldgen-specific logic.

Co-Authored-By: Astra (OpenAI Codex)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@frankmcsherry
frankmcsherry merged commit 57402e8 into master-next Sep 28, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant