WIP: ENH: Replace the shell wheel build system with Python (supersedes #302)

[hjmjohnson]

Replaces the shell and PowerShell wheel-build system with a pixi-driven
Python one, and moves ITK 6 to stable-ABI (abi3) wheels on
manylinux_2_28, macOS arm64 14.0, and Windows x86_64. Supersedes #302.

Status: WIP — refinements pending; not yet ready for in-depth review.
Successfully built on Windows, macOS arm64, Linux x86_64, and Linux
aarch64. The Doxygen/environment defect found by @dzenanz is fixed: build
tools now resolve inside the target --platform-env, and launching from
the wrong environment fails immediately with the environment name to use.

One item needs a decision before merge: the rebuilt macOS cache is
verified but not yet published to ITKPythonBuilds. The two Linux caches
and the Windows cache are published.

Platform verification — clean builds from this branch

Every row is a from-scratch build of this branch, not an incremental one.

Platform	Result	Time	Wheels
Linux x86_64	pass	—	7 × cp311-abi3-manylinux_2_28_x86_64
Linux aarch64	pass on retry	39 min	7 × cp311-abi3-manylinux_2_28_aarch64
macOS arm64	pass	93 min	7 × cp311-abi3-macosx_14_0_arm64
Windows x86_64	pass	—	7 × cp311-abi3-win_amd64

macOS binaries were checked with otool -l: real .so files report
minos 14.0, matching their macosx_14_0_arm64 tag. That check matters
because a component wheel tagged below its true minos installs on older
macOS and then fails in dyld at import — installable and broken, rather
than loudly uninstallable.

Remote-module end-to-end (the consumer path)

Proven on Windows x86_64 and Linux x86_64, each run as a fresh
consumer with zero ITK wheels present beforehand, so a leftover
artifact could not produce a false pass.

Linux x86_64, ITKBioCell, 8 minutes, rc=0:

• itk_biocell-1.0.0-cp311-abi3-manylinux_2_28_x86_64.whl, 0.58 MB
• Requires-Dist: itk<7,>=6.0.0b1 — the dependency rewrite fired
• 2 bundled libraries — genuinely repaired by auditwheel
• pyproject.toml restored byte-identical after the build

An earlier Windows pass was found to be riding on leftover wheels; the
zero-wheel precondition was added in response and both platforms were
re-run under it.

Unit tests and hooks

• pytest: 97 passed. pytest is a developer-only dependency; the
  normal build path does not import it.
• pre-commit run --all-files: exit 0, every hook Passed
  (black, ruff, shellcheck, shfmt, taplo, yaml/toml, whitespace).

Known caveat: intermittent aarch64 link truncation

One clean aarch64 build failed while linking:

FAILED: lib/libITKIOStimulate-6.0.a
ranlib: error reading itkStimulateImageIO.cxx.o: file truncated

The identical clean build then passed, with no truncation and no short
object files. Evidence points at the container file-sharing layer rather
than this branch: 158 GB free at the time, the truncated archive was
exactly 49,152 bytes (12 × 4096, a dropped write tail rather than a
partial compile), and aarch64 is the only platform writing through a
Docker bind mount onto a macOS filesystem.

Docker Desktop updated itself between the two runs (29.7.2 → 29.8.0), so
the version cannot be credited with the fix, and one pass does not
disprove an intermittent fault. Recommended handling: retry a clean
build first; do not treat a truncation as a defect in this branch.

Commit series

Ten topical commits, no fixups and no merge commits. Bug fixes found
during platform testing were folded into the commits that introduced the
code, so the series reads as though the defects never existed.

Cavan Riley is credited via Co-Authored-By on the seven commits
carrying his original work.

[dzenanz]

I have taken a look. There is too much code to look carefully. I tried running locally:

M:\a\IPP>pixi run python3 "C:\Misc\ITKPythonPackage\scripts\build_wheels.py" --platform-env windows-py311 --itk-source-dir C:/Dev/ITK-git
Error:   × could not find pixi.toml or pyproject.toml with tool.pixi at directory M:\a\IPP


M:\a\IPP>cd C:\Misc\ITKPythonPackage

M:\a\IPP>c:

C:\Misc\ITKPythonPackage>pixi run python3 "C:\Misc\ITKPythonPackage\scripts\build_wheels.py" --platform-env windows-py311 --itk-source-dir C:/Dev/ITK-git
 WARN Encountered 2 warnings while parsing the manifest:
  ⚠ the `[system-requirements]` table is deprecated in favor of virtual packages on `platforms`
    ╭─[C:\Misc\ITKPythonPackage\pixi.toml:82:1]
 81 │
 82 │ ╭─▶ [feature.manylinux228-build.system-requirements]
 83 │ │   linux = "5.4"
 84 │ ├─▶ libc = { family = "glibc", version = "2.28" }
    · ╰──── declare these on the `platforms` entries instead
 85 │
    ╰────
  help: e.g. platforms = [{ platform = "linux-64", cuda = "12" }]

  ⚠ the `[system-requirements]` table is deprecated in favor of virtual packages on `platforms`
    ╭─[C:\Misc\ITKPythonPackage\pixi.toml:95:23]
 94 │ platforms = ["osx-arm64"]
 95 │ system-requirements = { macos = "14.0" }
    ·                       ─────────┬────────
    ·                                ╰── declare these on the `platforms` entries instead
 96 │
    ╰────
  help: e.g. platforms = [{ platform = "linux-64", cuda = "12" }]

Error launching 'python3': The system cannot find the path specified. (os error 3)

C:\Misc\ITKPythonPackage>pixi run python "C:\Misc\ITKPythonPackage\scripts\build_wheels.py" --platform-env windows-py311 --itk-source-dir C:/Dev/ITK-git
 WARN Encountered 2 warnings while parsing the manifest:
  ⚠ the `[system-requirements]` table is deprecated in favor of virtual packages on `platforms`
    ╭─[C:\Misc\ITKPythonPackage\pixi.toml:82:1]
 81 │
 82 │ ╭─▶ [feature.manylinux228-build.system-requirements]
 83 │ │   linux = "5.4"
 84 │ ├─▶ libc = { family = "glibc", version = "2.28" }
    · ╰──── declare these on the `platforms` entries instead
 85 │
    ╰────
  help: e.g. platforms = [{ platform = "linux-64", cuda = "12" }]

  ⚠ the `[system-requirements]` table is deprecated in favor of virtual packages on `platforms`
    ╭─[C:\Misc\ITKPythonPackage\pixi.toml:95:23]
 94 │ platforms = ["osx-arm64"]
 95 │ system-requirements = { macos = "14.0" }
    ·                       ─────────┬────────
    ·                                ╰── declare these on the `platforms` entries instead
 96 │
    ╰────
  help: e.g. platforms = [{ platform = "linux-64", cuda = "12" }]

Running >>>>>: git fetch --tags origin  ; # in cwd=C:\Dev\ITK-git with check=False

Traceback (most recent call last):
  File "C:\Misc\ITKPythonPackage\scripts\build_wheels.py", line 628, in <module>
    build_wheels_main()
    ~~~~~~~~~~~~~~~~~^^
  File "C:\Misc\ITKPythonPackage\scripts\build_wheels.py", line 450, in build_wheels_main
    raise RuntimeError(
    ...<2 lines>...
    )
RuntimeError: Failed to fetch ITK tags in C:\Dev\ITK-git: fatal: 'origin' does not appear to be a git repository
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.


C:\Misc\ITKPythonPackage>

[dzenanz]

This seems to be doing something: C:\Misc\ITKPythonPackage>pixi run python scripts/build_wheels.py --platform-env windows-py311 --itk-git-tag v6.0b01

[dzenanz]

That failed too: run.log

[hjmjohnson]

Windows verification complete — handing back to Linux/macOS

Everything Windows-specific on this branch is now verified by real builds, and one cross-platform defect found along the way is fixed. Head is 1af2b15 (10 commits, no BUG: commits — every fix is folded into the commit that introduced the code). Coordination detail lives in the shared gist; this is the summary a reviewer needs.

What was verified on Windows 11 / VS 2022 (i7-9800X, 32 GB)

step	result
clean cache build from ef0c2d4 (pixi reinstalled, everything deleted first)	7 × cp311-abi3-win_amd64, cache 226,973,373 B, published to v6.0rc01.dev20260915, byte-identical
DLL bundling	9 DLLs in itk_core incl. itk_core.libs/tbb12.dll; wheels import in a clean venv with PATH stripped to System32 and ITK_EAGER_IMPORT=1
remote module (ITKBioCell) via the documented script, as a fresh consumer	step 06 reached; Requires-Dist: itk<7,>=6.0.0b1; repaired wheel (with msvcp140.dll) in the module's own dist/; pyproject.toml restored byte-identical
unit tests	97 passed on Windows (was 11 failed / 81 passed)

Defects fixed on this branch by the Windows work

All Windows-only unless noted; none were reachable from Linux/macOS testing.

1. TargetPlatform.detect() read PROCESSOR_ARCHITECTURE from the config mapping, not the environment — Unknown machine '' on any Windows build (POSIX hidden by os.uname).
2. Nested pixi run from inside an activated env overflowed the cmd.exe 8191-char limit (The input line is too long) — in the interpreter probe and in echo_check_call.
3. build_tarball() deleted the seven freshly built wheels from dist/ and reported rc=0.
4. pixi's vs2022 activation exports CMAKE_GENERATOR="Visual Studio 17 2022"; scikit-build-core honoured it, and the multi-config generator broke ITK wrapping's file(GENERATE) (castxml.inc "written multiple times"). Module wheels now pass -GNinja.
5. Cross-platform: the dependency rewrite emitted itk >= 6.0, < 7, which PEP 440 makes unsatisfiable by any 6.0.0bN (6.0.0b2 < 6.0); --pre cannot rescue it. Now >= 6.0.0b1, < 7.
6. delvewheel --add-path pointed at a non-existent oneTBB directory.
7. pixi global install aria2 always fails on win-64 and printed a red error that looked fatal.
8. A failed git checkout <tag> forced git checkout main, moving a cache-restored ITK source off the commit its binaries were built from.
9. 7-Zip -r treated build as a recursive pattern and swept a remote module's CMake tree into the cache; consumers then hit generator ... does not match.
10. Windows fixup_wheel ignored remote_module_wheel and wrote the repaired module wheel to the main dist/, leaving the unrepaired one as the module's only artifact.
11. Cross-platform (gist T7): on the --skip-itk-wheel-build path, 04_post_build_fixup and 05_final_import_test still ran against ITK wheels that were never built, and step 05 aborted the run before the module wheel. Both are now registered as _skipped no-ops when 03 is skipped; the step table was extracted into build_step_table() and is covered by tests. The earlier Windows "pass" of step 05 was an artifact of leftover wheels on the build machine — verified by reproducing the failure with dist/ empty, then re-running after the fix.

Plus three test fixes so the suite runs on Windows (os.uname monkeypatching, a cross-build test that inferred the host OS, a symlink that needs SeCreateSymbolicLinkPrivilege), and install-pixi.ps1 added to .gitignore.

For Linux/macOS

• T2 (Linux remote-module test) is unblocked by fix 11 — git fetch, rebuild from 1af2b15 or later. Expected Requires-Dist is itk<7,>=6.0.0b1, not the older >= 6.0, < 7.
• The published caches were built from ef0c2d4 and carry the pre-fix-11 scripts. The documented module scripts re-clone the branch when ITKPYTHONPACKAGE_TAG is set, so consumers get the fix; a consumer without that variable does not. Rebuilding caches is not urgent but is why cache and branch can disagree.
• Not on this branch: an experiment bounding ninja jobs by physical memory (-j = min(cpu, ram/4) with a link pool). Motivated by two OOM kills at -j16 on 32 GB; peak memory dropped to 9.9 GB, but the timing run was confounded and is parked on a separate local branch. Worth revisiting for CI runner sizing.

[hjmjohnson]

Thanks for running it locally — your run.log found a real defect in this
branch, and your two earlier comments are now answered by the Windows
verification. Rebased onto current main and pushed; head is afdfaf8.

Your build failure is our bug, not your invocation. The log ends at:

CMake Error: Could NOT find Doxygen (missing: DOXYGEN_EXECUTABLE)
  -DITK_WRAP_DOC:BOOL='ON'  -DDOXYGEN_EXECUTABLE:FILEPATH='C:\Dev\doxygen'

ITK_WRAP_DOC=ON is set unconditionally and DOXYGEN_EXECUTABLE comes from
which_required("doxygen"), which aborts when it cannot resolve. But
doxygen is not declared in pixi.toml, so it is an undeclared external
prerequisite. My Windows runs passed only because that machine happens to have
a working Doxygen. That defeats the point of using pixi for a hermetic
toolchain, and it is being fixed before this leaves draft.

310-x64 is removed (your inline comment). It was a leftover, and it could
never have resolved: the only environments in pixi.toml are dev, publish,
linux-py311, manylinux228-py311, macosx-py311, and windows-py311.
ITK 5.4 is out of scope for this build system.

This PR stays WIP until the Doxygen prerequisite is resolved. "Too much
code to look carefully" is fair — please don't spend review time on the whole
diff yet; I'll ping you when it is ready for a real pass.

Platform verification status as of afdfaf8

Platform	Clean build	Remote-module end-to-end
Linux x86_64	pass	pass (ITKBioCell, 8 min)
Linux aarch64	pass on retry	—
macOS arm64	pass (93 min)	—
Windows x86_64	pass	pass

97 unit tests pass; pre-commit run --all-files exits 0.

One known caveat: a single aarch64 clean build failed while linking with a
truncated object file, and the identical rebuild passed. Evidence points at the
Docker bind-mount layer rather than this branch (the truncated archive was
exactly 49,152 bytes, a dropped write tail, and aarch64 is the only platform
writing through a bind mount onto a macOS filesystem). Docker updated itself
between the two runs, so the version cannot be credited with the fix.

[hjmjohnson]

@dzenanz thank you for the log file — it identified a real bug, and that bug is
now fixed in e53db8d.

This should have been a WIP: PR from the start, and that is my fault. It
was opened primarily to get the code onto my Windows machine for testing and to
track the outstanding testing tasks, not to request review. It is now retitled
WIP:. There are still refinements to make, but I have successfully run it on
Windows, macOS arm64, Linux x86_64, and Linux aarch64 builds.

What your log found, and the fix

Your run failed at:

CMake Error: Could NOT find Doxygen (missing: DOXYGEN_EXECUTABLE)
  -DDOXYGEN_EXECUTABLE:FILEPATH='C:\Dev\doxygen'

doxygen is declared in pixi.toml and is present in every build
environment. The bug was that your command used pixi run without -e,
which lands in the default environment. The script then resolved its build
tools from whatever was on the system PATH rather than from the target
environment, and handed CMake a C:\Dev\doxygen that is not a Doxygen
executable. Reproduced here — in default, cmake, ninja, doxygen, and
git all resolve to system paths.

Two changes:

1. cmake, ninja, doxygen, and git now resolve inside the target
   --platform-env, the same way the Python interpreter already did. A stray
   same-named executable on PATH can no longer reach CMake.
2. Running outside the requested environment now fails immediately, before
   any work, naming the environment you want:

==============================================================
  WRONG ENVIRONMENT - build aborted before doing any work
==============================================================
  running in : the 'default' pixi environment
  required   : 'windows-py311'

  Re-run with the environment selected:
      pixi run -e windows-py311 python scripts/build_wheels.py ...

  'pixi run' without '-e' uses the 'default' environment, which
  does NOT carry the pinned cmake, ninja, doxygen and git. The
  build would silently pick up whatever is on the system PATH.
==============================================================

Six regression tests cover both (103 total, up from 97). Your exact command
shape now exits 1 with that message instead of failing minutes later inside
CMake.

[hjmjohnson]

Windows x86_64 status at f19b202

Everything below was run natively on Windows 11, VS 2022, 16 threads / 31.8 GiB, against ITK f1e39a58.

Verified by real builds

• Clean ITK build + cache (make_windows_zip.ps1 py311): 7 wheels, all cp311-abi3-win_amd64; ITKPythonBuilds-windows.zip 226,973,373 B published to v6.0rc01.dev20260915. DLL bundling confirmed by inspecting the wheel (MSVC runtimes + itk_core.libs/tbb12.dll), not by step status.
• Remote module (ITKBioCell) as a fresh consumer: build/, dist/ and the module clone deleted first, so nothing could be satisfied by leftovers. Steps 04/05 record _skipped, step 06 runs, repaired wheel lands in the module's own dist/ with DELVEWHEEL marker, Requires-Dist: itk<7,>=6.0.0b1, and pyproject.toml restored byte-identical.
• Test suite: 119 passed on Windows; pre-commit run --all-files clean on an LF checkout.

Windows defects fixed on this branch (folded into their originating commits)

• Repaired module wheel written to the main dist/ instead of the module's; failed delvewheel repair ignored.
• 7-Zip invoked with -r, sweeping a remote module's CMake tree into the cache and causing "generator does not match" on consumers.
• oneTBB runtime path, arch detection, PATH overflow past cmd.exe's 8191-char limit, and >=6.0 excluding the RC series under PEP 440 ordering (now >=6.0.0b1,<7).
• 05_final_import_test running on the --skip-itk-wheel-build path where no wheels exist (the caches deliberately exclude them). Now 04/05 are _skipped alongside 03; build_step_table() was extracted so the table is unit-tested.
• test_tool_resolution_env.py compared a stringified Path against a POSIX literal and failed on Windows.

Build parallelism (the two PERF commits)

Cold builds at the historical -j<cpu_count> were killed for memory pressure in 2 of 3 runs here: 16 compilers on 32 GB is 2 GB each, and a single ITK wrapping TU peaks at 3.9 GB in cl.exe.

jobs	step 02	peak build RSS sum	min free RAM	outcome
-j16	—	—	—	killed, 2 of 3 cold runs
-j12	4842 s	15.7 GB	5.75 GB	pass
-j8	5291 s	12.2 GB	8.96 GB	pass

What changed:

• scripts/build_jobs.py: jobs = min(cpu_count, floor(ram_gb / 2.5)), dependency-free RAM detection on all three OSes, falls back to cpu_count if RAM is unknown. Overrides: ITK_BUILD_JOBS, ITK_BUILD_LOAD_LIMIT.
• ninja -l is now the thread count rather than the job count. ninja honours -l on Windows (verified on 1.13.2), so -l<jobs> on a box with more threads than jobs throttled the build to one compiler.
• ITK wheels on Windows now build with -GNinja, as the module wheels already did.
• No link pool: link.exe peaked at 1.45 GB and the link phase is ~1.3 min of an ~80 min step.

The 2.5 GB/job constant is calibrated on MSVC only. Linux and macOS builds on this head should report the # Build parallelism: line the builder prints and their step-02 time; a slowdown over ~10 % there would mean gcc/clang deserve a different budget. Inside dockcross, sysconf reports host RAM, not a cgroup limit, so a memory-limited container may need ITK_BUILD_JOBS set explicitly.

Notes for other platforms

• The branch was rewritten today; reset local branches to origin before rebasing.
• Published caches predate the step-table and parallelism changes and carry the old IPP/scripts/. The documented consumer path re-clones the branch via ITKPYTHONPACKAGE_TAG, so it gets the fixes; a consumer without that variable does not.
• Full detail and the task board are in the coordination gist: https://gist.github.com/87d847e38a2c50c48730817042b475d3