Investigate large hidden-state differences when independent requests are co-packed (Qwen3.6, no-grad) #901

Activity

1. bradhilton commented on Sep 25, 2026

   

   bradhilton

   on Sep 25, 2026

   CollaboratorAuthor

   More actions

   September 25 follow-up: a bounded one-H200 diagnostic has isolated a repeatable physical-normalization sensitivity associated with packing-dependent differences and ruled out one proposed recurrence explanation for the tested prepared stream. The original numerical qualification remains failed; no tolerance or model code changed.

   The source is ART aab5d54, Qwen3.6-35B-A3B, exact saved step-4 checkpoint, no gradients or optimizer updates. The two unchanged histories have 33,706 and 31,196 prediction positions (64,902 together). Thirteen complete model calls cover B alone and A+B, with warmups, repeats, observation controls, and one anchored operator experiment. Complete same-layout target-logprob vectors repeat byte-for-byte; observation does not change those vectors. Captured adapter/optimizer tensors, optimizer-step counters, slot configuration and training modes are unchanged before and after.

   Findings:

   • Full prepared inputs were not equal. Complete Q, K, V, decay-gate and beta hashes for B differ between the singleton-model and packed-model calls, even though every previously sampled row matched. These differences exist before FLA's input guard; the guard preserves the logical bytes. Initial recurrent state is exactly zero in both cases. We have not yet measured the differing coordinates or magnitudes in those complete streams.
   • Identical prepared inputs give identical recurrence outputs in the tested control. Replaying the original packed-prepared B tensors alone versus within the original internally ordered B,A pair gives the same complete 255,557,632-byte B output, with zero changed elements, in warm and repeated controls. This does not support a multi-sequence recurrence-context explanation for this specific prepared stream. It is not a general proof about all FLA configurations.
   • Normalization differs for identical physical input rows. At layer-0 pre-MLP RMSNorm, B positions 6 and 24, and layer-3 RMSNorm, B position 0, physical input rows, weight values and consumed settings are equal between singleton and packed calls, while output rows and reciprocal RMS statistics differ.
   • A value-preserving gamma storage/address change removes that selected-row difference. Passing byte-identical gamma values at a one-BF16-element storage offset to the actual native normalization call makes those output rows and reciprocal RMS statistics match across singleton and packed layouts. Restoring original gamma restores the original difference. Packed A+B's entire normalization output/statistics remain unchanged by the offset; singleton B's do change. Layer-0 position 0 has equal output already, but its reciprocal-RMS difference also disappears. There are 36 shadow native calls; their outputs never replace the original model result, and no model Parameter is replaced.

   This establishes physical native-normalization sensitivity to a value-preserving gamma storage/address change at the selected rows. The actual kernel dispatch was not traced, so a particular tuned kernel is not yet proved responsible. These shadow calls did not correct the whole model, test gradients or establish performance overhead. Full upstream-stream differences remain a separate lead, and this experiment does not resolve #902 or establish the cause of the older September-15 fixture.

   Resources are closed: native and observer exit 0, 54 registered/sampled process identities absent, owned groups/sessions empty, selected GPU idle. Closure is limited to those recorded identities/current owned groups; no universal historical-child claim.

   Private evidence (no prompt bodies attached):

   • /var/tmp/schulman-krennic-packed-boundary-20260925/attempt-03/science/result/result.json, SHA256 cfec4f06f02782f7d1e384bf7d7b318d8c5d41148ac96e710dc8b424dc695d2f.
   • operator-control.json, SHA256 b702b09d59bbe71929cad6e63cfe2d24f9304c5da6acbdfdb99d04d077c48d0a.
   • Root closure /home/brad/.local/share/schulman/krennic-packed-boundary-20260925/ATTEMPT-03-CLOSED.json, SHA256 f999ca83f88de9c3d47eaea4d8c79e16772fcf37ba78d57c2000205678f93758.

   Independent retained-result review: /home/brad/.local/share/schulman/krennic-packed-boundary-20260925/review-v4/RETAINED-RESULT-REVIEW-RECEIPT.json, SHA256 8f9bf9fdad899df6a3ef914e803adf2753cf090d5b2d7c18c47d71f8ef24a8b6, PASS_RETAINED_RESULT_AUDIT. The next bounded source preparation adds complete per-history block hashes at upstream first-layer boundaries to locate where those prepared input streams first diverge.
2. bradhilton commented on Sep 25, 2026

   

   bradhilton

   on Sep 25, 2026

   CollaboratorAuthor

   More actions

   Schulman: The new two-history, one-H200 diagnostic localizes the first complete-tensor difference to the normalization output returned by the first GDN fused input projection. This is a localization result, not a fix or an accuracy qualification.

   On frozen ART aab5d54163eda3431b716c2f46b2fbfb9c0f79f1, the complete common B tensor is byte-identical through embedding, layer input, GDN hidden input and projection input. The returned normalized B tensor differs in 112 of 488 consecutive 64-token blocks; the first differing block is [128,192). The base projection, combined projection and projected QKV/gate have the same 112-block difference set. Convolution adds six blocks, and recurrence output differs in 486 blocks. Actual invocation sizes are 31,196 rows alone and 64,902 rows packed; B's physical offset also changes.

   All 12 ordinary no-grad model calls completed. Complete outputs, plans, captured tensors, state and compiler checks repeat within each layout; measured plain/captured output vectors are exact. All three output-vector families also reproduce the prior diagnostic exactly. Independent review verified all retained file hashes and block inventories. No backward, optimizer update, save or maintained-source change occurred.

   This fused projection boundary is different from the standalone pre-MLP RMSNorm examined by the previous gamma-storage shadows. Their observed gamma sensitivity cannot be transferred directly to this newly localized operation. TE source suggests row-count/alignment-dependent normalization dispatch as a candidate, but actual fused-call kernel provenance, raw reference inputs and numerical error against a higher-precision reference remain missing. The next private witness targets that specific gap. No tolerance changes or original failure reclassification.

   Evidence: /home/brad/.local/share/schulman/krennic-packed-boundary-20260925/attempt-04-interpretation/REPORT.md SHA256 ad87c7c72f1168a40a977c1aa86dbdc9e6bb25027a8959b3be519663e071632c; independent audit /home/brad/.local/share/schulman/successor-reviews-20260925/attempt04-audit/REVIEW.md SHA256 26f67872b06b8628fe77cfe575f3e495c3ee429cbc5c040206a2b70f9d7f720f. Root closure ATTEMPT-04-CLOSED.json SHA256 be745bdc3caa444ed5c94c8622ae127c29a0e3e57c955eabacd597de133cf7f3 verifies the 50 recorded/sampled identities absent, owned groups/sessions empty and GPU idle; universal historical descendant absence is not claimed. Private raw tensors remain local.
3. bradhilton commented on Oct 2, 2026

   

   bradhilton

   on Oct 2, 2026

   Last edited by bradhilton

   CollaboratorAuthor

   More actions

   A new one-H200 backward diagnostic reproduces a grouping-dependent difference on baseline ART 413c237173db665714643460bbac15dd17e67d60. Both sides use that source; this is not evidence against #1061.

   The same process ran warm split/joined, then measured split/joined/joined/split. Split uses public request slices [1,15]; joined uses [16]. All calls use the same checkpoint 25, 16 retained trajectories, 663 selected loss tokens, logical input/target order, masks, old log-probabilities and advantages. Recorded model, adapter, optimizer and buffer state match; Python/NumPy/Torch/CUDA/Megatron RNG is restored. No optimizer update occurs.

   Comparison	Selected log-probability relative L2 difference	Full gradient relative L2 difference
   Split repeat	0	1.0695%
   Joined repeat	0	1.0739%
   Split → joined, first measured pair	7.7924%	17.9643%
   Split → joined, second measured pair	7.7924%	17.9957%

   617/663 selected log-probabilities differ across grouping (maximum absolute difference 2.6197); all 16 trajectories are affected. Same-grouping log-probabilities also repeat exactly from warmup. Gradient comparison covers all 58,869,760 present elements, with 660 consistently absent slots checked separately. Split loss repeats at 0.0100762201473; joined repeats at 0.0104971593246. Visible FLA autotuner caches are stable throughout measured calls, but are not complete kernel-execution provenance.

   This extends the issue to a warmed forward/backward fixture. It does not identify the cause or transfer the earlier fused-normalization diagnosis to this fixture. No acceptance tolerance was introduced. There is no matched per-case wall timer, so no speedup claim.

   All six inner captures completed and their delivered bytes were independently accepted. The outer diagnostic still failed cleanup (exit 1/INCOMPLETE); resources subsequently closed. Valid captures do not make the overall run successful or qualified.

   Private retained evidence, with no prompt payloads attached:

   • Execution e0ca5c4387414bb2a64da582efd2ba60, original input SHA256 23b27707ac353b1097bc8483cdaabb4f85388857302f7aedcc948097371ea871.
   • /home/brad/.local/share/hayek/consumer-acceptance-20261002/warmed-geometry-analysis/FINAL.json, SHA256 68527d8758fe387fc45399eb81900999e719bd899bf464e134e7de14dfc7184f.
   • The offline reader validates the original inventory, all tensor hashes/extents, logical/state/RNG bindings and actual geometry; ten arithmetic/negative controls pass. Independent interpretation review passed: warmed-geometry-analysis-review/REVIEW.json, SHA256 76f693bbd83adbf772a46502a2c0e92c80903383a53a4899b12d8d3fc40c4469.

   The retained CPU audit found no token, position, selected-head or GDN parent-routing mismatch. All 16 trajectories differ, including trajectory 0. Joined packing shares a 5,046-token prefix; GDN processes 4,992 prefix tokens once and duplicates the remaining 54 into child sequences. The next diagnostic will compare the actual first-layer fused projection inputs and returned normalization, using logical-row mapping, bounded values and plain/captured controls. No new GPU run or geometry adoption is authorized by this update.
4. bradhilton commented on Oct 2, 2026

   

   bradhilton

   on Oct 2, 2026

   CollaboratorAuthor

   More actions

   Follow-up on the same 16-trajectory fixture: the seven recorded trajectory-0 boundaries are bit-identical between split and joined grouping, through layer 0’s combined projection. This includes embedding, layer input, GDN input, projection input, returned normalization, base projection and LoRA-combined projection. The full normalization tensors match in all 11,304,960 BF16 elements; logical row mappings were checked against the original tokens.

   The later difference persists: 617/663 final logprobs differ (relative L2 7.79%), including 32/38 for trajectory 0. Gradient relative L2 is 17.93–17.98% across geometries. Captured and plain runs have identical forward outputs within each geometry; their ~1.08% gradient difference cannot distinguish hook effects from repeat variation. Inputs, checkpoint state and RNG match; no optimizer updates occurred.

   This rules out those recorded early boundaries for this reproduction. It does not identify the first differing internal operation, or establish a TE/FLA cause. We are preparing a broader layer-output and GDN-intermediate capture to locate the later divergence.

   The six-callback diagnostic completed successfully. Analysis passed 17 reader checks and an independent review. Evidence manifest: 6116c24d23baeee8ef94db8ceeab5b66b2ce167f279271b5c516ddcf9a2b7f21; review: d0f3e00058e48f5c29ecab813dd7d34283541d3709d6ebe9396683dce1487a60.
5. bradhilton commented on Oct 2, 2026

   

   bradhilton

   on Oct 2, 2026

   CollaboratorAuthor

   More actions

   The broader capture locates the first observed difference at decoder layer 3’s output (the fourth layer). Embedding, the recorded layer-0 internals and decoder outputs 0–2 are bit-identical for split [1,15] versus joined [16].

   At layer 3, 4,868,295 of 11,304,960 BF16 elements differ across 5,519 of 5,520 trajectory-0 rows. Relative L2 is 0.416624%; maximum absolute difference is 0.015625. By final normalization, relative L2 reaches 28.648828%. These describe the difference; they are not acceptance tolerances.

   We checked all 42 complete boundary pairs against their logical token maps. Recorded input, checkpoint state and RNG match. Same-geometry warm, captured and plain calls have identical selected logprobs; split versus joined still differs at 617/663 selected logprobs across the full 16-history batch. No optimizer update occurred.

   This narrows the search to an operation inside layer 3. It does not yet identify that operation or establish a TE/FLA cause. We are preparing internal-boundary capture there. The diagnostic completed successfully and all resources are closed; a separate reader independently recounted the raw arrays.

   Evidence: analysis 7ee43f8dafea2d806387367f47cbc6447aa927a45252c6649c6d8cca1cbac62e; independent review 69a60ec8fc1e564af883e8f3183a52e6cc4c70b851b8585a6be57661c8e01fb9. No gradient or performance claim from these activation captures.