Repository navigation
Add float16 head_dim 256 TK=1 SDPA vector kernel for M1/M2 - #4719
Open
jmmarkiewicz wants to merge 1 commit into
Open
jmmarkiewicz wants to merge 1 commit into
jmmarkiewicz wants to merge 1 commit into
Conversation
ml-explore#4643 added the TK=1 fallback for bfloat16 and float32, but float16 with head_dim 256 still exceeds the 896-thread limit on M1/M2 when there is no causal mask. Also test the no-mask and bool-mask cases. Fixes ml-explore#4718 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On M1/M2, fp16 SDPA with head_dim 256 fails in the 2-pass vector kernel with
Maximum threads per threadgroup is 896 but requested 960when the 2-pass route is taken (around 1024+ keys on M1 Ultra), 32 × gqa × q_len falls between 928 and 1024, and the mask is a bool mask or none (causal compiles to its own_cpipeline and runs fine in my tests). It was reported in oMLX #4609 by @JerilynFranz, running MTP verify on an fp16 copy of Qwen3.8-27B; fp16 copies are common on M1 because bf16 is emulated there.I reproduced it with synthetic tensors, and Qwen3.6-35B-A3B should hit it too (16/2 heads × MTP width 4 = 1024 threads), though I haven't run that model. #4643 added
_tk1fallbacks for bf16/256 and fp32/192 and 256 but not fp16/256, and its test only ran those shapes withmask="causal", so the gap never surfaced. This PR adds the missinginstantiate_sdpa_vector_tk1(float16_t, 256)and the matching case inhas_tk1_variant, and extendstest_sdpa_vector_2pass_threadgroup_limitto run every shape with causal, no mask, and a random bool mask against the fp32 reference._tk1is only selected when the default kernel would exceed the limit, which used to throw, so paths that already worked are unchanged.Closes #4718.
Testing (Mac Studio M1 Ultra,
applegpu_g13d, macOS 27.2)Stock 0.32.4 repro: 4/36 failures, all fp16/256 with bool or no mask; fp16/192 never fails. At 256 keys the same shapes run fine.
test_sdpa_vector_2pass_threadgroup_limitonmain: before, 4 errors (exactly the fp16/256 no-mask and bool subtests); after, OK on all 24 subtests (8 shapes × 3 masks).Full
test_fast_sdpa.py(32 tests):main8 errors, this branch 4 errors. The remaining 4 are the existingtest_sdpa_vector_head_dim_512failures on M1: the 1-pass d=512 kernel is capped at 640 threads (fp32) and 832 (fp16), but MLX requests 1024. They're identical with or without this PR.☑️ I understand it is strictly prohibited to use AI to write PR description
AI usage disclosure: The root cause was found, and the patch and test written and validated, with Claude Code; I reviewed all changes.