Skip to content

Add float16 head_dim 256 TK=1 SDPA vector kernel for M1/M2 - #4719

Open
jmmarkiewicz wants to merge 1 commit into
ml-explore:mainfrom
jmmarkiewicz:fp16-sdpa-tk1
Open

jmmarkiewicz wants to merge 1 commit into
ml-explore:mainfrom
jmmarkiewicz:fp16-sdpa-tk1

Conversation

@jmmarkiewicz

Copy link
Copy Markdown

On M1/M2, fp16 SDPA with head_dim 256 fails in the 2-pass vector kernel with Maximum threads per threadgroup is 896 but requested 960 when the 2-pass route is taken (around 1024+ keys on M1 Ultra), 32 × gqa × q_len falls between 928 and 1024, and the mask is a bool mask or none (causal compiles to its own _c pipeline and runs fine in my tests). It was reported in oMLX #4609 by @JerilynFranz, running MTP verify on an fp16 copy of Qwen3.8-27B; fp16 copies are common on M1 because bf16 is emulated there.

I reproduced it with synthetic tensors, and Qwen3.6-35B-A3B should hit it too (16/2 heads × MTP width 4 = 1024 threads), though I haven't run that model. #4643 added _tk1 fallbacks for bf16/256 and fp32/192 and 256 but not fp16/256, and its test only ran those shapes with mask="causal", so the gap never surfaced. This PR adds the missing instantiate_sdpa_vector_tk1(float16_t, 256) and the matching case in has_tk1_variant, and extends test_sdpa_vector_2pass_threadgroup_limit to run every shape with causal, no mask, and a random bool mask against the fp32 reference. _tk1 is only selected when the default kernel would exceed the limit, which used to throw, so paths that already worked are unchanged.

Closes #4718.

Testing (Mac Studio M1 Ultra, applegpu_g13d, macOS 27.2)

  • Stock 0.32.4 repro: 4/36 failures, all fp16/256 with bool or no mask; fp16/192 never fails. At 256 keys the same shapes run fine.

  • test_sdpa_vector_2pass_threadgroup_limit on main: before, 4 errors (exactly the fp16/256 no-mask and bool subtests); after, OK on all 24 subtests (8 shapes × 3 masks).

  • Full test_fast_sdpa.py (32 tests): main 8 errors, this branch 4 errors. The remaining 4 are the existing test_sdpa_vector_head_dim_512 failures on M1: the 1-pass d=512 kernel is capped at 640 threads (fp32) and 832 (fp16), but MLX requests 1024. They're identical with or without this PR.

  • ☑️ I understand it is strictly prohibited to use AI to write PR description

  • AI usage disclosure: The root cause was found, and the patch and test written and validated, with Claude Code; I reviewed all changes.

ml-explore#4643 added the TK=1 fallback for bfloat16 and float32, but float16 with
head_dim 256 still exceeds the 896-thread limit on M1/M2 when there is no
causal mask. Also test the no-mask and bool-mask cases.

Fixes ml-explore#4718

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] float16 head_dim 256 SDPA still exceeds the M1/M2 threadgroup limit: #4643's _tk1 fallback covers bfloat16 and float32 only

1 participant