perf(trainer-rank): recover CUDA cache before internal splits

[bradhilton]

Agent: Schulman

Try the existing budgeted CUDA-cache release before accepting a split forward for the smallest input group. A successful release triggers a fresh search and admission check; denial retains the split only if its refreshed memory check passes. This older candidate retains the original unsplit plan during recovery; #1053 is the newer implementation under review.

The reviewed head c15b1f20 passed hosted CPU checks. Local two-H200 validation passed on predecessor f93b6d85; hosted GPU validation failed to obtain capacity before tests. Representative steady-state overhead remains unqualified. The original detailed validation history is preserved in the owner’s review evidence.

No public API or art.megatron changes. Packing and numerical layout can change. Closed as superseded by #1053, which uses a fresh search after recovery and has separate source/native evidence. #1053 still requires its own final CI, reviews and Brad’s behavioral decision; none of its qualification is attributed to this historical PR.

[bradhilton]

Root integration record for exact head f93b6d85338b62c52c6e0e2e66b95ceddd8b7d24:

Three independent agent source reviews are CLEAR: recovery-before-split reviewer (0b19c6a4), collector/source reviewer (71b41b4b), and independent successor reviewer (ef36111f). All explicitly preserve the merge hold for changed packing, additional executed recovery/search collectives, and use of the shared first-release allowance/budget. Public APIs and art.megatron are unchanged.

Review reproduced an oversized-override lifetime defect in the original candidate. The final two-line correction clears the temporary recovery target on that successful return, preserving the selected plan, inputs, indices and refreshed check. The regression fails before the correction and passes afterward; the independent focused four-case run passed. The original failing review is preserved.

Native evidence is deliberately narrower than exact-final-head GPU qualification: the predecessor's ordinary path completed six flat forwards/backwards with two actual releases, and a separate four-case NCCL fixture passed. The final cleanup-only branch has CPU evidence; hosted CI for this head has just started. The later budget decision's denominator was 93.4% warmup/compilation-associated work, so no steady-state or total-overhead guarantee follows. All owned native processes and GPUs are physically closed.

Private exact review/result receipts are retained under /home/brad/.local/share/schulman/, indexed by krennic-packed-boundary-20260925/recovery-pr972-root-integration/REVIEWS.json. This remains a draft; no merge, deployment, frozen-run adoption or general OOM-resolution claim.

[bradhilton]

Published cc3591f68c62d2fac6bd1ac1a19de7a0b5861aab as a fast-forward type-check correction. Three independent exact-head source reviews clear the delta: 093db0e2, c08f4419, 01ee2e45. The complete production AST equals f93b6d85 after removing the single existing typing.cast identity call; there is no new branch, collective or changed error ordering. Test fixture casts and explicit narrowing preserve prior assertions.

The retained 13 type errors reproduce before and clear after; affected-file type/lint/format checks and 11 focused CPU cases pass. One reviewer additionally checked 13 small outcome/error cases without Torch. These do not replace fresh hosted CI. The currently running local GPU suite remains frozen at f93b6d85, with source equivalence documented separately. The feature remains draft and held for Brad because fresh recovery can change layout/packing. No API or art.megatron changes and no merge/deployment.

[bradhilton]

Independent local two-H200 qualification is complete on frozen f93b6d85338b62c52c6e0e2e66b95ceddd8b7d24: all five exact CI command groups exited 0; 95 pytest cases passed and 2 skipped, plus passing CP2/TP2 trainer checks. The two skip reasons are inferred from exact source/device count, since -rs was not enabled.

All 97 source pins matched. Root and independent review joined the 254 tracked identities, 10 groups and 9 sessions as retired, with both assigned H200s idle. Runtime 870.291 seconds; peak sampled process-family RSS 8.565 GiB. Audit receipt 512ec8db9cf2c7f0cd042bdafc7cd9bf7032303966ea7a142f8e7869fea33ac7, root closure 1348dba44bf494c885b7f20fed328b8a9e7d968baef2584d8c9deb246751253c.

The current cc3591f successor adds one reviewed identity cast and test narrowing; this supports relevance of the f93 result without relabeling the executed head. Both hosted GPU attempts failed acquisition before tests; the required check remains failed. No full-size OOM, throughput, numerical-equivalence or merge clearance is implied.