fix(trainer-rank): raise routed-row estimates from checkpoint observations

[bradhilton]

Agent: Dalinar

A fixed expert-routing allowance can repeatedly underprice a skewed checkpoint. Observe each checkpoint’s worst routed share and raise later row estimates to that share plus 0.10 when it exceeds the allowance. Never lower a charge. This applies to HybridEP with matching EP/CP groups and TP/ETP1.

Only single-group micro-batch waves produce observations; split, multi-group and dp_rank_forward paths use prior observations. Ranks exchange shares with checkpoint epochs and record nothing on unsupported, failed, cancelled or mismatched observations. Loads, snapshots and discards check matching identities. Disk loads reset observations, snapshots inherit them, failed loads restore the previous epoch, and optimizer steps retain the high-water mark. Updated rows feed admission, splitting, costs and telemetry.

Recorded validation: 24 new CPU cases and the trainer-rank suite passed, including a two-process Gloo exchange and real checkpoint paths. A local two-H200 Qwen3.6-35B-A3B CP2/EP2 run made 12 real-data calls per arm. Zero-learning-rate shares stayed at 1.15–1.16; training reached 1.364 and raised later charges. No measured call was underestimated. These observations do not bound all routing or model configurations.

Includes #1002 via #963/#978; review those unmerged dependencies first. A warm discount is excluded. Planner replay still does not cover expert-parallel plans; EP1 records no share.