[Vulkan] Make expand_copy resizable and fully delegate dynamic transformer blocks by mergennachin · Pull Request #23254 · pytorch/executorch

mergennachin · 2026-09-29T15:03:06Z

Dynamic eager-attention and SDPA transformer blocks were split around expand_copy despite its runtime resize callback. Mark expand_copy as resizable so these blocks lower to one Vulkan delegate across changing sequence lengths. Integration tests compare against ATen while sequence lengths shrink and grow, including fully masked SDPA rows, and verify exact dynamic expansion results.

Fixes #23156.

This is part 15/15 of the Vulkan transformer and operator-conformance series. Parts 1–14 (#23240 through #23253) have landed. This PR is rebased directly onto origin/main at 541a2fab62 and contains only the expansion registration and integration tests.

The transformer test selects supported storages before entering subtests. SDPA execution has a separate test with a top-level SwiftShader skip: its masked-row checks allocate internal bool buffers even with texture preference, and SwiftShader lacks the required 8-bit storage support. A direct runtime probe confirmed initialization fails with Vulkan bool buffer tensors require 8-bit storage buffer support. Eager attention and dynamic expansion remain covered on SwiftShader; MoltenVK executes both attention paths.

Test Plan: Rebuilt ExecuTorch with Vulkan enabled from the rebased source on macOS arm64 and ran the four focused tests through the public Buck Python test adapter with each driver ICD explicitly selected.

MoltenVK: 4 passed, no skips (26.747s). Eager attention, SDPA including fully masked rows, and expansion executed with both texture and buffer preferences.
SwiftShader: 3 passed, 1 expected top-level SDPA execution skip (18.642s). Both transformer modes passed the partitioning test; eager attention executed with texture preference, and expansion executed with both storage types.
No failures, errors, or nested skips. Lintrunner on both changed files, git diff --check, and pip check passed.

Recreates #23162 through ghstack. Prior review discussion remains on that PR.

Authored with OpenAI Codex; split planned with Claude Code.

cc @SS-JIA @manuelcandales @digantdesai @cbilgin

…ghstack [ghstack-poisoned]

mergennachin · 2026-09-29T15:03:07Z

Stack from ghstack (oldest at bottom):

-> [Vulkan] Make expand_copy resizable and fully delegate dynamic transformer blocks #23254

pytorch-bot · 2026-09-29T15:03:11Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/23254

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

⏳ No Failures, 5 Pending

As of commit 5f93cf9 with merge base 541a2fa ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[ghstack-poisoned]

@SS-JIA

…ormer blocks The dynamic transformer reproductions now lower to one Vulkan delegate across changing sequence lengths. This final PR makes `expand_copy` resizable and adds the eager-attention and SDPA integration tests, including fully masked rows. Fixes #23156. This is part 15 of the original ghstack series. Parts 1–7 (#23240 through #23246) have landed in main; the remaining eight PRs run from #23247 through #23254. Review each PR against its selected base branch. This PR contains the final expansion and transformer changes; part 10 owns the scalar-tensor issue #23158. | Part | PR | Change | | --- | --- | --- | | 1 | #23240 (landed) | Hardware Vulkan CI | | 2 | #23241 (landed) | Scalar cache type and signed-zero keys | | 3 | #23242 (landed) | Vulkan-local signed-zero serialization | | 4 | #23243 (landed) | GELU modes and view kwargs | | 5 | #23244 (landed) | Reduction clamp, NaN, and FP16 rounding | | 6 | #23245 (landed) | Reduction and arg-reduction dimension/storage guards | | 7 | #23246 (landed) | Bool staging and logical_not | | 8 | #23247 | Scalar representability and symbolic guards | | 9 | #23248 | 64-bit dtype and fusion policy | | 10 | #23249 | scalar_tensor with exact integer values | | 11 | #23250 | Typed, resizable full | | 12 | #23251 | Power special values and logical FP16 dtype | | 13 | #23252 | mul.Scalar | | 14 | #23253 | any.dim textures with either keepdim setting | | 15 | #23254 | Dynamic expand and transformer integration | The stack retains the FACTO and ATen conformance fixes and their regression tests: typed integer fills, NaN and signed-zero behavior, reduction range and accumulation fixes, safe scalar fallbacks, and the FP16 power contract. On native FP16 devices, the scalar-tensor, full texture, single-dimension texture reduction, and binary scalar shaders round their FP16 outputs to nearest-even. The binary scalar shaders also preserve the requested FP16 dtype when storage is emulated with FP32. Other operators retain their existing FP32 intermediate behavior on those devices. Coverage includes scalar exponents 2.0001 and 2049 and a two-node FP16 mul.Scalar chain in both texture and buffer storage. PR4 introduces `test_vulkan_dynamic.py` and its CI/Buck references. In #23253, `any.dim` now supports both `keepdim` settings on textures. With `keepdim=False`, a reduction into a temporary texture is followed by a GPU view/repack into the squeezed output shape. Both nodes resize dynamically, and empty reduced axes produce false. This adds one temporary texture and one GPU dispatch without introducing a bool-buffer requirement for texture models. Unsupported scalar inputs and 4D batch/channel axes use CPU fallback. Current rebase: Rebased onto main at `5f72739cf26e` after #23246 landed. All eight remaining code patches are unchanged, and the rebase had no conflicts. All 1153 shader variants compile with glslc, the 15 graph-builder/serialization tests pass, and lintrunner and `git diff --check` pass. Native GPU execution was validated on an earlier base; hardware and SwiftShader CI will rerun on the updated ghstack heads. Prior texture-path validation on main at `903cef063774`: the Release Vulkan/portable runtime built and eleven focused native tests passed on Apple M1 Pro / MoltenVK. Coverage included dynamic and chained any reductions, actual texture storage, scalar outputs, singleton and empty dimensions, growth after a zero-length dimension, transformer integration, supported and unsupported 4D reductions, special values, and FP16 rounding. An additional 27 FACTO-generated bool any.dim cases and six boundary cases matched ATen and portable kernels exactly through the shared `backends/test/` harness: 23 ran on Vulkan textures and 10 exercised expected CPU fallback. The subsequent review of #23245 with #23253 verified 784 support/storage combinations and eight mixed dynamic models; all nine review tests passed, with execution results matching ATen and portable kernels. These GPU execution results precede the latest upstream shader-indexing change. Earlier validation of the operator changes passed 116 FACTO cases against both ATen and portable kernels, plus 144 boundary cases against ATen, with texture and buffer preferences: all 520 Vulkan-configuration executions matched ATen, including CPU fallback where unsupported. The additional 239-case sweep covered all 91 available FACTO specifications for registered ATen overloads and 57 targeted examples. Exact-input replays of every candidate Vulkan failure on a separately built main at `a31838280f9309f04af1b375147375b702d29339` found no new regressions or unresolved comparisons. The prior full native run passed 51 tests with one expected SwiftShader-only skip. Subsequent review coverage added exact FP16 rounding boundaries, int32 buffer amax/amin range checks against both ATen and portable kernels, and 28 argmax/argmin execution and fallback cases. These are prior validation results, not a repeat of the full suite for this update. Validation uses Release portable CPU kernels and Vulkan; optimized kernels and XNNPACK are disabled. Buffer preference uses `texture_limits=(1,1,1)`, with actual storage and fallback recorded separately. Single-texel tensors can still select textures. FACTO revision: `3b8c778c99766a8b4d0d04563ae0b16cbb276829`, seed 0. Recreates #23162 through ghstack. Prior review discussion remains on that PR. Authored with OpenAI Codex; split planned with Claude Code. cc @SS-JIA @manuelcandales @digantdesai @cbilgin ghstack-source-id: a7c2766 ghstack-comment-id: 5892899249 Pull-Request: #23254

@SS-JIA

With downcast_64_bit disabled, the graph builder refuses int64 and float64 tensors, but the partitioner still claimed those nodes, so lowering failed late or the delegate aborted at load. The partitioner now rejects nodes that need 64-bit dtypes in that mode, and keeps a fusable pattern entirely on CPU when it touches a 64-bit non-constant tensor rather than splitting it. Constant parameters are exempt because quantization parameters are folded away during fusion. Part 9/15 of the Vulkan transformer and operator-conformance stack. Depends on #23247; review against the selected base branch. Integration PR: #23254. Validation: 4 passed, 2 warnings in 32.37s. Lintrunner and git diff --check pass. Native tests use MoltenVK with portable CPU kernels; hardware and SwiftShader CI are pending. Recreates #23209 through ghstack. Prior review discussion remains on that PR. Authored with OpenAI Codex; split planned with Claude Code. cc @SS-JIA @manuelcandales @digantdesai @cbilgin

[ghstack-poisoned]

@SS-JIA

…ormer blocks The dynamic transformer reproductions now lower to one Vulkan delegate across changing sequence lengths. This final PR makes `expand_copy` resizable and adds the eager-attention and SDPA integration tests, including fully masked rows. Fixes #23156. This is part 15 of the original ghstack series. Parts 1–7 (#23240 through #23246) have landed in main; the remaining eight PRs run from #23247 through #23254. Review each PR against its selected base branch. This PR contains the final expansion and transformer changes; part 10 owns the scalar-tensor issue #23158. | Part | PR | Change | | --- | --- | --- | | 1 | #23240 (landed) | Hardware Vulkan CI | | 2 | #23241 (landed) | Scalar cache type and signed-zero keys | | 3 | #23242 (landed) | Vulkan-local signed-zero serialization | | 4 | #23243 (landed) | GELU modes and view kwargs | | 5 | #23244 (landed) | Reduction clamp, NaN, and FP16 rounding | | 6 | #23245 (landed) | Reduction and arg-reduction dimension/storage guards | | 7 | #23246 (landed) | Bool staging and logical_not | | 8 | #23247 | Scalar representability and symbolic guards | | 9 | #23248 | 64-bit dtype and fusion policy | | 10 | #23249 | scalar_tensor with exact integer values | | 11 | #23250 | Typed, resizable full | | 12 | #23251 | Power special values and logical FP16 dtype | | 13 | #23252 | mul.Scalar | | 14 | #23253 | any.dim textures with either keepdim setting | | 15 | #23254 | Dynamic expand and transformer integration | The stack retains the FACTO and ATen conformance fixes and their regression tests: typed integer fills, NaN and signed-zero behavior, reduction range and accumulation fixes, safe scalar fallbacks, and the FP16 power contract. On native FP16 devices, the scalar-tensor, full texture, single-dimension texture reduction, and binary scalar shaders round their FP16 outputs to nearest-even. The binary scalar shaders also preserve the requested FP16 dtype when storage is emulated with FP32. Other operators retain their existing FP32 intermediate behavior on those devices. Coverage includes scalar exponents 2.0001 and 2049 and a two-node FP16 mul.Scalar chain in both texture and buffer storage. PR4 introduces `test_vulkan_dynamic.py` and its CI/Buck references. In #23253, `any.dim` now supports both `keepdim` settings on textures. With `keepdim=False`, a reduction into a temporary texture is followed by a GPU view/repack into the squeezed output shape. Both nodes resize dynamically, and empty reduced axes produce false. This adds one temporary texture and one GPU dispatch without introducing a bool-buffer requirement for texture models. Unsupported scalar inputs and 4D batch/channel axes use CPU fallback. Current rebase: Rebased onto main at `5f72739cf26e` after #23246 landed. All eight remaining code patches are unchanged, and the rebase had no conflicts. All 1153 shader variants compile with glslc, the 15 graph-builder/serialization tests pass, and lintrunner and `git diff --check` pass. Native GPU execution was validated on an earlier base; hardware and SwiftShader CI will rerun on the updated ghstack heads. Prior texture-path validation on main at `903cef063774`: the Release Vulkan/portable runtime built and eleven focused native tests passed on Apple M1 Pro / MoltenVK. Coverage included dynamic and chained any reductions, actual texture storage, scalar outputs, singleton and empty dimensions, growth after a zero-length dimension, transformer integration, supported and unsupported 4D reductions, special values, and FP16 rounding. An additional 27 FACTO-generated bool any.dim cases and six boundary cases matched ATen and portable kernels exactly through the shared `backends/test/` harness: 23 ran on Vulkan textures and 10 exercised expected CPU fallback. The subsequent review of #23245 with #23253 verified 784 support/storage combinations and eight mixed dynamic models; all nine review tests passed, with execution results matching ATen and portable kernels. These GPU execution results precede the latest upstream shader-indexing change. Earlier validation of the operator changes passed 116 FACTO cases against both ATen and portable kernels, plus 144 boundary cases against ATen, with texture and buffer preferences: all 520 Vulkan-configuration executions matched ATen, including CPU fallback where unsupported. The additional 239-case sweep covered all 91 available FACTO specifications for registered ATen overloads and 57 targeted examples. Exact-input replays of every candidate Vulkan failure on a separately built main at `a31838280f9309f04af1b375147375b702d29339` found no new regressions or unresolved comparisons. The prior full native run passed 51 tests with one expected SwiftShader-only skip. Subsequent review coverage added exact FP16 rounding boundaries, int32 buffer amax/amin range checks against both ATen and portable kernels, and 28 argmax/argmin execution and fallback cases. These are prior validation results, not a repeat of the full suite for this update. Validation uses Release portable CPU kernels and Vulkan; optimized kernels and XNNPACK are disabled. Buffer preference uses `texture_limits=(1,1,1)`, with actual storage and fallback recorded separately. Single-texel tensors can still select textures. FACTO revision: `3b8c778c99766a8b4d0d04563ae0b16cbb276829`, seed 0. Recreates #23162 through ghstack. Prior review discussion remains on that PR. Authored with OpenAI Codex; split planned with Claude Code. cc @SS-JIA @manuelcandales @digantdesai @cbilgin ghstack-source-id: 6b879b2 ghstack-comment-id: 5892899249 Pull-Request: #23254

[ghstack-poisoned]

…idate on MoltenVK and SwiftShader, isolate the unsupported SwiftShader SDPA case as a top-level skip. [ghstack-poisoned]

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

mergennachin added 15 commits September 29, 2026 11:02

[INITIAL] Recreate the Vulkan transformer and conformance stack with …

aae8159

…ghstack [ghstack-poisoned]

[INITIAL] Recreate the Vulkan transformer and conformance stack with …

bcab9d0

…ghstack [ghstack-poisoned]

[INITIAL] Recreate the Vulkan transformer and conformance stack with …

d6db604

…ghstack [ghstack-poisoned]

[INITIAL] Recreate the Vulkan transformer and conformance stack with …

446a5d6

…ghstack [ghstack-poisoned]

[INITIAL] Recreate the Vulkan transformer and conformance stack with …

fb905cf

…ghstack [ghstack-poisoned]

[INITIAL] Recreate the Vulkan transformer and conformance stack with …

2131a32

…ghstack [ghstack-poisoned]

[INITIAL] Recreate the Vulkan transformer and conformance stack with …

f310b91

…ghstack [ghstack-poisoned]

[INITIAL] Recreate the Vulkan transformer and conformance stack with …

f588b1e

…ghstack [ghstack-poisoned]

[INITIAL] Recreate the Vulkan transformer and conformance stack with …

0dfae81

…ghstack [ghstack-poisoned]

[INITIAL] Recreate the Vulkan transformer and conformance stack with …

5bc8471

…ghstack [ghstack-poisoned]

[INITIAL] Recreate the Vulkan transformer and conformance stack with …

836dc54

…ghstack [ghstack-poisoned]

[INITIAL] Recreate the Vulkan transformer and conformance stack with …

b708c1a

…ghstack [ghstack-poisoned]

[INITIAL] Recreate the Vulkan transformer and conformance stack with …

bebd775

…ghstack [ghstack-poisoned]

[INITIAL] Recreate the Vulkan transformer and conformance stack with …

fcf87f1

…ghstack [ghstack-poisoned]

[INITIAL] Recreate the Vulkan transformer and conformance stack with …

8d46517

…ghstack [ghstack-poisoned]

mergennachin added the module: vulkan Issues related to the Vulkan delegate and code under backends/vulkan/ label Sep 29, 2026

mergennachin added 7 commits October 7, 2026 11:41

[UPDATE] Rebase remaining Vulkan stack onto main ad434a3

80283ae

[ghstack-poisoned]

[UPDATE] Rebase remaining Vulkan stack onto main ad434a3

d248f3a

[ghstack-poisoned]

[UPDATE] Rebase remaining Vulkan stack onto main ad434a3

e8b545b

[ghstack-poisoned]

[UPDATE] Rebase remaining Vulkan stack onto main ad434a3

7132779

[ghstack-poisoned]

[UPDATE] Rebase remaining Vulkan stack onto main ad434a3

8bee7c7

[ghstack-poisoned]

[UPDATE] Rebase remaining Vulkan stack onto main ad434a3

ee495eb

[ghstack-poisoned]

[UPDATE] Rebase remaining Vulkan stack onto main ad434a3

fe403c3

[ghstack-poisoned]

mergennachin deployed to cadence October 7, 2026 15:42 — with GitHub Actions Active

mergennachin deployed to cadence October 7, 2026 15:59 — with GitHub Actions Active

mergennachin deployed to cadence October 7, 2026 16:04 — with GitHub Actions Active

mergennachin added 6 commits October 7, 2026 13:35

[UPDATE] Rebase remaining Vulkan stack after #23248 landed

a7e71de

[ghstack-poisoned]

[UPDATE] Rebase remaining Vulkan stack after #23248 landed

247f43f

[ghstack-poisoned]

[UPDATE] Rebase remaining Vulkan stack after #23248 landed

8e9e2a6

[ghstack-poisoned]

[UPDATE] Rebase remaining Vulkan stack after #23248 landed

9b3dfcb

[ghstack-poisoned]

[UPDATE] Rebase remaining Vulkan stack after #23248 landed

8429e10

[ghstack-poisoned]

[UPDATE] Rebase remaining Vulkan stack after #23248 landed

487bf1d

[ghstack-poisoned]

mergennachin deployed to cadence October 7, 2026 17:36 — with GitHub Actions Active

mergennachin added 6 commits October 7, 2026 15:56

[UPDATE] Rebase remaining Vulkan stack after #23249 landed

1f18190

[ghstack-poisoned]

[UPDATE] Rebase remaining Vulkan stack after #23249 landed

6311011

[ghstack-poisoned]

[UPDATE] Rebase remaining Vulkan stack after #23249 landed

48b288c

[ghstack-poisoned]

[UPDATE] Rebase remaining Vulkan stack after #23249 landed

d54b03c

[ghstack-poisoned]

[UPDATE] Rebase remaining Vulkan stack after #23249 landed

d140d0b

[ghstack-poisoned]

[UPDATE] Rebase dynamic transformer integration onto origin/main; val…

5f93cf9

…idate on MoltenVK and SwiftShader, isolate the unsupported SwiftShader SDPA case as a top-level skip. [ghstack-poisoned]

Copilot AI reviewed Oct 8, 2026

Navigation Menu

Repository navigation

[Vulkan] Make expand_copy resizable and fully delegate dynamic transformer blocks - #23254

[Vulkan] Make expand_copy resizable and fully delegate dynamic transformer blocks#23254
mergennachin merged 130 commits into
mainfrom
gh/mergennachin/38/head

mergennachin commented Sep 29, 2026 •

edited

Loading

Uh oh!

mergennachin commented Sep 29, 2026 •

edited

Loading

Uh oh!

pytorch-bot Bot commented Sep 29, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

This branch was successfully deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

mergennachin commented Sep 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mergennachin commented Sep 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot Bot commented Sep 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/23254

⏳ No Failures, 5 Pending

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

This branch was successfully deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mergennachin commented Sep 29, 2026 •

edited

Loading

mergennachin commented Sep 29, 2026 •

edited

Loading

pytorch-bot Bot commented Sep 29, 2026 •

edited

Loading