Shard the Frontier benchmark across two SLURM jobs so it fits the 2 h QOS cap - #1890
Open
sbryngelson wants to merge 2 commits into
Open
sbryngelson wants to merge 2 commits into
sbryngelson wants to merge 2 commits into
Conversation
… QOS cap The Frontier bench has been failing on every PR since the 7th benchmark case was added: one job spends ~30 min building and ~17 min per case, so the list needs ~2.5 h against a 01:59:00 walltime -- the "normal" QOS caps a job at 2 h, so the limit cannot go up -- and SLURM kills it on case 6/7 each time. The red X is non-blocking (continue-on-error) but it means no PR has had a Frontier speedup table in weeks. Split each tree's run across two concurrent shards, using the same i/N round-robin the case-optimization lanes already use for the same reason: - `mfc.sh bench --shard i/N` runs cases i, i+N, ... of bench.yaml. - bench.sh passes $job_shard (from submit-slurm-job.sh) through. - run_parallel_benchmarks.sh submits 1/2 and 2/2 per tree on frontier and frontier_amd, monitors each, and folds them back with the new `mfc.sh bench_merge` so bench_diff still sees one YAML per tree. The merge strips --shard/-o from the recorded invocation so the two trees' metadata compare equal, and refuses a duplicated case or a shard run under a different lock. Every other cluster gets an empty shard list and is unchanged. The worst shard (4 cases) is ~98 min, 21 min inside the cap. Diagnosed and written with Claude Code.
Contributor
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Pull request overview
This PR addresses Frontier benchmark CI jobs being killed by SLURM’s 2h QOS cap by sharding the benchmark case list across two concurrent jobs, then merging the shard outputs back into a single YAML so the existing bench_diff flow remains unchanged.
Changes:
- Add
--shard i/Nsupport tomfc.sh benchto run a round-robin subset of benchmark cases. - Introduce
mfc.sh bench_mergeto merge shard YAML outputs back into a single results file with consistent metadata. - Update CI scripts to submit/monitor per-shard SLURM jobs on Frontier and merge results before diffing.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| toolchain/mfc/cli/docs_gen.py | Adds bench_merge to the generated CLI reference under CI commands. |
| toolchain/mfc/cli/commands.py | Adds --shard to bench and introduces the new bench_merge command. |
| toolchain/mfc/bench.py | Implements sharded case selection and the bench_merge YAML fold step. |
| toolchain/main.py | Wires the new bench_merge command into the toolchain entrypoint. |
| .github/workflows/common/bench.sh | Passes $job_shard through to mfc.sh bench when present. |
| .github/scripts/run_parallel_benchmarks.sh | Submits per-shard jobs on Frontier and merges shard YAMLs post-run. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+94
to
+101
| for dir in pr master; do | ||
| for shard in ${bench_shards:-""}; do | ||
| slug=$(tree_slugs "$shard") | ||
| echo "Submitting ${dir} benchmark${shard:+ (shard $shard)}..." | ||
| (cd "$dir" && SUBMIT_ONLY=1 bash "${SCRIPT_DIR}/submit-slurm-job.sh" "$PR_BENCH_SCRIPT" "$device" "$interface" "$cluster" "$shard") | ||
| echo "${dir} job submitted: $(cat "${dir}/${slug}.slurm_job_id")" | ||
| done | ||
| done |
Comment on lines
+145
to
+152
| for shard in ${bench_shards:-""}; do | ||
| echo "" | ||
| echo "=== Monitoring ${dir} job $(cat "${dir}/$(tree_slugs "$shard").slurm_job_id")${shard:+ (shard $shard)} ===" | ||
| monitor_bench_with_resubmit "$dir" "$shard" | ||
| if [ "$BENCH_MON_RC" -ne 0 ] && [ "$BENCH_TREE_RC" -eq 0 ]; then | ||
| BENCH_TREE_RC="$BENCH_MON_RC" | ||
| fi | ||
| done |
Comment on lines
+264
to
+274
| out, skip = [], False | ||
| for word in argv: | ||
| if skip: | ||
| skip = False | ||
| continue | ||
| if word in ("--shard", "-o", "--output"): | ||
| skip = True | ||
| continue | ||
| if word.startswith("--shard=") or word.startswith("--output="): | ||
| continue | ||
| out.append(word) |
Comment on lines
+280
to
+284
| meta = {"invocation": _strip_shard_words(shard["metadata"]["invocation"]), "lock": shard["metadata"]["lock"]} | ||
| if merged is None: | ||
| merged = {"metadata": meta, "cases": {}} | ||
| elif merged["metadata"] != meta: | ||
| raise MFCException(f"Shard {path} was not run the same way as {inputs[0]}: {meta} vs {merged['metadata']}.") |
The sharding here treats a symptom. The Frontier bench passed in ~60 min through 2026-09-15 and has failed at the 2 h limit on every run since 09-16, with no change to bench.yaml (7 cases since July), the walltime, the toolchain run path, the modules, or the nodes. Per-target solver times are identical between a passing and a failing run (ibm simulation: 129.1 s vs 128.9 s), and every case still finishes in ~3 min inside mfc.sh run. The whole difference is the gap between one case's End-time and the next case's start: 4 s on 09-15, 14 min 15 s on 09-17, the same to the second on every case -- a fixed timeout in process exit, silent in every log. Darshan, which Frontier preloads into every MPI job and which flushes its log in MPI_Finalize, asserted in that phase the same day (MFlowCode#1866, syscheck). So: - bench.py prints, per case, launch-to-return wall time and how much of it fell after the run's own End-time. That interval appears nowhere else. - On frontier/frontier_amd, shard 1 runs with DARSHAN_DISABLE=1 and shard 2 with Darshan on, in the same allocation, so this PR's own bench lane is the A/B. If shard 1's post-exit time drops to seconds, the stall is Darshan's and the fix is one environment variable, not sharding. Diagnosed with Claude Code.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The Frontier bench lane has failed on every PR since 2026-09-16 with
It is
continue-on-error, so nothing has been blocked — but no PR has had a Frontier speedup table since.What actually changed (this is not a workload-growth problem)
bench.yamlhas had 7 cases since July and the 01:59:00 walltime since June. The lane passed in 55–66 min through 09-15 and has failed at 126+ min on every run since 09-16. Comparing a passing and a failing run's per-targetexectimes from the archived YAMLs:Identical. Every case still finishes in ~3 min inside
mfc.sh run. The entire difference is the gap between one case'sEnd-timeand the next case's start:A fixed stall in process exit, silent in every log. No change in the window to the toolchain run path, the loaded modules, or the nodes (
frontier10198/9in both runs). Darshan — which Frontier preloads into every MPI job and flushes atMPI_Finalize— asserted in exactly that phase the same day (#1866,heatmap_mpi_redux: Assertion 'heatmap_runtime' failedinsyscheck). That is the prime suspect but is unproven.What this PR does
Diagnose.
bench.pynow prints, per case, launch-to-return wall time and how much of it fell after the run's ownEnd-time. Onfrontier/frontier_amd, shard 1 runs withDARSHAN_DISABLE=1and shard 2 with Darshan on, in the same allocation. This PR's own Frontier bench lane is the A/B. If shard 1's post-exit time drops to seconds, the fix is one environment variable and the sharding below is just headroom.Shard each tree's bench across two concurrent SLURM jobs, using the
i/Nround-robin the case-optimization lanes already use:mfc.sh bench --shard i/N,bench.shpasses$job_shardthrough,run_parallel_benchmarks.shsubmits1/2+2/2per tree on Frontier and folds them back with a newmfc.sh bench_merge(strips--shard/-ofrom the recorded invocation so both trees' metadata compare equal; refuses a duplicated case or a lock mismatch). Every other cluster gets an empty shard list and is unchanged. Whether sharding stays depends on what the A/B says — I'll update this PR with the result.Type of change
Scope
How Has This Been Tested?
bench_merge+bench_diffon synthetic shard files with the realoutput_summary/exec/grindschema: normal speedup table, all cases present, no metadata warning.bench_mergerejects a duplicated case and a lock mismatch.bench.yaml(every case in exactly one shard).ibm../mfc.sh precheck: passes.Checklist
./mfc.sh formatbefore committing my codeDiagnosed and written with Claude Code.