Repository navigation
fix(trainer-rank): credit staged adapter gradients and shared captures in graph admission - #1081
Merged
Merged
Conversation
…s in graph admission Graph admission added the 3x gradient staging reserve beside each unit's workspace, but the recompute walk's pending adapter gradients (#1002) are the same _GradientBatch clones the reserve's staged share holds, so they were charged twice. Credit them against the unit's workspace; the full staging reserve (current, staged, replacement) stays for multi-forward transactions. Split children and groups of one slot and staleness reuse the first capture's live LoRA weight snapshot; charge it once per admission. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
bradhilton
had a problem deploying
to
trainer-rank-gpu-validation
October 3, 2026 03:50 — with
GitHub Actions
Error
…tinct targets The backward walk repeats a slot's pending inventory for each gradient group priced together (routed and unrouted requests of one slot and policy), so AG can reach 2P - B while one transaction stages each distinct target once. Cap the credit at the pending bytes of the unit's distinct gradient targets: the walk's pending gradients are a subset of that staged aggregate. A no-grad correction forward allocates no adapter gradients and can coexist with the current gradient, staged head gradients and a current-weight capture, so its workspace keeps no credit. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
bradhilton
deployed
to
trainer-rank-gpu-validation
October 3, 2026 04:15 — with
GitHub Actions
Active
bradhilton
marked this pull request as ready for review
October 3, 2026 04:42
bradhilton
added a commit
that referenced
this pull request
Oct 3, 2026
Brings in #1068 (eager head workspaces in admission and replay), #1067 (dense CP1 no-grad floor), #1081 (staged adapter-gradient credit), #1077, #1080 and CI fixes. Conflicts and reconciliation: - _group_head_workspace_bytes: TP1 keeps #1068 verbatim (capacity reserves the eager 7-buffer statistics, 3 buffers for requested logits). TP > 1 goes through _tp_head_workspace_bytes, which uses #1068's model per chunk: the same 7-buffer eager charge where no kernel is attempted, the kernel path plus the bounded statistics' increment where one is (after a failure the bounded statistics, not the 7-buffer path, run), and #1068's logits copies gathered to the full vocabulary (1 + 2 TP buffers). - Replay: one _frozen_head_bytes over #1068's version-3 facts (head_statistics, head_target_backward). TP1 is #1068's formula; TP > 1 prices the kernel path, and capture declines TP > 1 waves with an eager chunk or with requested logits beside statistics, which the facts do not record. The optional rank facts (mixer_top_gaps, lora_modules_per_layer) sit beside #1067's defaulted dense_fc1_adapted. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Agent: Dalinar
#925's graph-placement admission over-prices the first gradient wave of a slot by charging the same memory twice:
This change credits the staged adapter gradients against the unit's workspace, so the full 3× staging reserve stays and rollback, the multi-forward prepare and the in-flight margin remain covered. It also charges each shared snapshot once. Nothing else in admission changes: CP no-release credit, custom heads, checkpoint sharing and learned split floors are untouched.
On a Qwen-like CPU fixture (40 layers, 920 MiB slot):
Testing: new admission tests fail on main and pass here. 2,257 trainer-rank CPU tests pass. GPU CI runs on this PR.
Limits:
🤖 Generated with Claude Code