Repository navigation
Conversation
tools/headless_batch/run_batch.py runs many headless engine processes in parallel, one mission, workspace, seed and UDP port per process, and collects each run's -sim-result file into results.jsonl and summary.json. A run counts as ok only if the engine exited 0 and wrote a readable result saying the mission ran (time_limit, player_died or mission_exit); crashes, missing results, start failures and timeouts are recorded as failures, and a timed-out process tree is killed. Plans are validated up front: unknown keys, out-of-range values, a missing pilot or mod folder, a base workspace and output directory on different volumes (the .vp packs are hardlinked), and, on Windows, an output directory so deep that the engine could not open a staged file are all errors that say what to change. Workspaces hardlink .vp packs and copy everything else so a run does not write through to the base. Supports --dry-run and --resume. README.md documents the headless options, what runs and what is skipped, the plan format, outputs, reproducibility and known limits. test_run_batch.py covers validation, command construction, staging, the volume and path checks, and a stub engine (success, crash, missing result, start failure, player death, timeout) and needs no game files.
With -noninteractive the engine prints Error() messages, and now Lua errors, to
stderr before it exits. A failed run's problems now include that message
("engine said: ..."), so a crash at load (for example a Lua syntax error in a
script table) says why instead of only "engine wrote no -sim-result file".
When its parent has a console, the Windows engine attaches to it and reopens stdout and stderr there unless FSO_KEEP_STDOUT is 1, so a run's stdout.txt and stderr.txt stayed empty and the engine's error text (and its warnings) went to the runner's console instead. The runner now sets FSO_KEEP_STDOUT=1 for the engine, which keeps the redirection, so a failed run can quote the engine's error.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Split out of #7874. Depends on #7897 (the headless options it passes) and #7893 (for its reproducibility notes); it only adds files under
tools/headless_batch/.run_batch.pyruns many-headlessengine processes in parallel, one portable workspace, seed and UDP port each, and collects their-sim-resultfiles intoresults.jsonlandsummary.json. It uses only the Python standard library..vppacks and copy everything else, so a run cannot write through to the base.time_limit,player_died,mission_exit). Crashes, timeouts, missing results and start failures are failures, and a failed run quotes the engine's own error text from stderr.--dry-runprints the commands;--resumereruns only unfinished or failed runs.FSO_KEEP_STDOUT=1; without it the engine reattaches its output to the runner's console and the per-run log files stay empty.Testing:
python -m unittest tools/headless_batch/test_run_batch.py(18 tests, no game files needed; a stub engine covers success, crash, missing result, start failure, player death, Lua error and timeout). Real batches of retail missions with 1–4 workers.