Repository navigation
Use a unified mechanism for build, test, packaging, and release #2204
Description
Presently, local development and GitHub Actions use different build mechanisms. These mechanisms also provide different capabilities and levels of test coverage.
GitHub Actions is the only standardized entry point for several classes of validation, particularly release-canary installation tests, release-artifact smoke matrices, and multi-platform release build and packaging workflows. Other tests can technically run locally, but their CI configurations may lack equivalent, documented mise entry points. The reverse gap also exists: several tests included in mise run pre-commit do not appear to run in GitHub Actions, including the explicit Rust workspace compile check, SBOM tooling tests, focused install.sh tests, static packaging-asset tests, and documentation synchronization tests.
Release builds have a similar reproducibility gap. GitHub Actions provides the toolchains and target-specific environments used to build release binaries, and developers cannot reliably produce every target artifact through a common local entry point. Because packaging and installation tests depend on those binaries, this limitation propagates downstream. For example, a developer may be unable to build an RPM, install it in a Fedora VM, and reproduce the corresponding release test locally. The shared build mechanism should make cross-compilation, packaging, and artifact testing locally executable wherever practical.
CI and developer workstations should invoke the same shared task runner(s) so that neither environment silently provides unique coverage. Intentional differences should be limited to tests requiring specialized hardware or operating systems. For example, testing macOS artifacts or creating VMs with Hypervisor.framework requires a real Mac, while GPU end-to-end tests require an NVIDIA GPU. Even when execution environments differ, entry points and assertions should remain consistent, with GitHub Actions primarily orchestrating the same tasks developers can run locally.
This issue is intended to supersede the following PRs, which add desired coverage to the current system in different ways and must be considered when developing the shared task runner:
- PR #2093, which expands GitHub Actions coverage across additional targets.
- PR #1465, which adds a method for running some tests locally.
- PR #1981, which expands the release canary GitHub Action to cover upgrades.
Activity
zyga commentedon Jul 14, 2026
I'm open in extending spread so that it works both on MacOS and Windows, if there's interest in making this the adopted tool. Locally I started doing some early prep work for this and moving some shell-heavy pieces to native go so that it doesn't have to differ across systems.
krishicks commentedon Jul 14, 2026
I chose lima in my POC #2205 for managing local machines and xtask for orchestrating tests in those machines instead of using spread because lima gives us a lot of stuff for free, including macOS and Windows support, but also managing downloading of images and the ability to snapshot machines for re-use to allow fast feedback loops. Driving it with xtask also gives us the ability to keep the orchestration in-repo and in Rust rather than shelling out to spread. There's also a desire to just spin up a machine to use for an arbitrary purpose outside of a specific test, which the combination of xtask and lima enables. The actual machine provider, whether local (like lima) or external (a theoretical GCE provider) is fungible, however.
Does that make sense?
zyga commentedon Jul 14, 2026
That is actually nice. Note that spread could still be used as a test description / runner, just like image-garden , lima could be slotted into the task of "give me a test machine". Since xtask already exists I see the desire to get things into a consistent format. I don't have much experience with it but if you think it can arrange the test code into test suites (spread mostly handles getting a system into a state where certain kind of test is easy to do), then this is fine.
zyga commentedon Jul 14, 2026
@krishicks I had a look at #2205 but at 2K lines I wonder what's the strategy towards reviewing it. Can you suggest something?
krishicks commentedon Jul 14, 2026
I split it into commits; each commit is meant to showcase a different thing. There are two major themes: one is introducing Dagger, and one is introducing xtask. Keep in mind also that it's just a POC and will not be merged. #2254 is a better PR to look at, which is scoped down just to adding xtask for running tests, optionally in VMs.
krishicks commentedon Jul 14, 2026
I think we'll be moving away from suites as they exist today (e.g. e2e:podman/e2e:docker) and more towards features, which is what the mise tasks end up doing anyway through parameters. For example, we should have a base set of features that all drivers are expected to support, and then specialized tests for features that only apply to specific environments/drivers, like GPU tests, or tests that take advantage of SELinux. And then we should be able to arbitrarily say "run tests that exercise these features [in this VM]".
olivercalder commentedon Jul 15, 2026
Thinking about building and packaging for linux, I think there are a few key questions to answer early on:
openshell-sandboxbinary is built withgnufor the snap andmuslviacargo-zigbuildfor the docker imagez3in the binaries -- my sense is yes, but there may be issues with the deb packaging rules if there is ever intention to have the deb in official reposmusltarget for deb/rpm/snap package binaries even though they all assume aglibcruntime environmentmuslbinaries (does it use an ubuntu base image?) then perhaps we can simplify the current status quo dramaticallyglibcLinux environment for building binaries (natively, in CI runners, in a non-alpine docker container, in whatever dagger provides), and running them as wellglibcsystems (the snap's base provides this regardless of the actual host on which it runs), so the question is mostly can we assume this for the Docker image in which OpenShell runs (is it alpine or ubuntu, basically)glibcLinux environment for building and running binaries, then it should be simple to provide that environment on any host, either natively, via docker, dagger, CI, wsl, or something elsemusland non-native targets, and by extensionzigorcargo-zigbuilddaggerwhich spins up VMs/containers to run tasks/builds/packaging/tests, will we expect to invokedaggerfrom within CI as well, or will we expect CI to run the scripts directly which would otherwise be run from within adaggerVM?daggeridentically from CI or locallydaggerby the providers on all host platforms, such as kernels, LSMs, package repos, networking, etc., or will we need to accept some variation between runners on different hosts?zigandcargo-zigbuild?x86_64-unknown-linux-gnuoraarch64-unknown-linux-gnubut notaarch64-apple-darwin, or vice versa?daggerspin up VMs with all supported architectures, or just those matching the host architecture? And does this vary based on provider?For a bit of context, here's the current state of binaries and packaging for Linux, from what I could surmise with the help of an agent:
openshellopenshell-gateway--target) or gnu.2.28 (cargo-zigbuild if cross-compiling), bundled-z3openshell-sandboxopenshell-driver-vmSDAChess commentedon Jul 16, 2026
Thanks, this is a useful inventory. I think the key distinction is between the release contract we want and the current build paths, which have accumulated some inconsistencies.
From yesterday’s discussion, the current direction for Linux is:
openshell: musl for x86_64 and aarch64, published in tarballs, debs, RPMs, and Snaps.openshell-gateway: glibc 2.28 for x86_64 and aarch64, published in tarballs, debs, RPMs, Snaps, and gateway OCI images.openshell-sandbox: musl for x86_64 and aarch64, published in the supervisor OCI images used by Docker, Podman, and Kubernetes.The target is chosen per component, rather than per packaging format. Within a component and architecture, the aim is to build the binary once and use the same output in every applicable package. That deliberately means putting the musl CLI binary into the deb, RPM, and Snap packages. Those formats do not require their payloads to be dynamically linked against glibc.
The gateway is deliberately different. It uses glibc with a 2.28 compatibility floor, along with dynamically linked Z3 and the C++ runtime. Its OCI image therefore needs to provide a compatible glibc environment; Docker does not inherently require us to make the gateway musl. The sandbox remains musl because it is the portable supervisor used across the different backends.
For Z3, the intended dependency matrix has no Z3 dependency in the CLI. Removing it from
openshell, is an action item (#2318). The gateway continues to use Z3. One terminology wrinkle is that Cargo’sbundled-z3feature and “Z3 is statically linked into the executable” are not necessarily the same statement, so we still need to be precise about how its shared libraries are included in each artifact. Possible future inclusion in official Debian repositories may impose additional rules, but that is separate from producing our own deb artifacts for beta.The build implementation itself is not settled.
cargo-zigbuildis how the current Cargo pipeline produces the musl and explicit glibc 2.28 targets, but we are also actively investigating Bazel for the beta. Bazel could model the Rust and native toolchains, sysroots, libc compatibility floors, architectures, and packaging inputs in one hermetic build graph. That may let us remove much of the bespoke Cargo/Zig machinery and make the “build once, package many times” model more reliable.Bazel would manage the cross-compilation constraints rather than make them disappear. Installing a Rust target alone does not provide the target linker, libc/sysroot, C/C++ dependencies, or Apple SDKs. Rust Tier 1 status also does not imply that every Tier 1 host can build every other Tier 1 target. We would still need appropriate Linux toolchains, macOS runners and SDKs, and native or emulated support for each CPU architecture.
The same applies to Dagger. If we use it, the expectation would be for local development and CI to invoke the same pipeline. It can pin the build userspace and toolchain, but it cannot make every provider identical: kernels, LSMs, networking, CPU architectures, and emulation support still depend on the engine environment. Build and packaging steps can be made portable, while tests that depend on kernel or container-runtime behavior should run on explicitly selected runners.
The current-state table is therefore a good description of what the scripts do today, but not every difference in it is intentional. For example, native local deb builds should eventually consume the same release binaries as CI, and the GNU sandbox used by the current Snap path does not match the target matrix under discussion.
The VM bundle is still an open area. We are looking at publishing it separately, possibly as an OCI artifact, with glibc 2.28 compatibility, after checking exactly what is inside its guest bundle. That could also let us stop treating the standalone
openshell-driver-*targets as release artifacts. The Docker driver is linked directly into the gateway, while the Kubernetes and Podman standalone Cargo targets are exercised by workspace CI but are not published; all three drivers’ functionality is included in every gateway build.These are very active internal discussions for the beta. The matrix captures the intended published outputs, but the build system and some of the packaging details may still change as we work through Bazel, Dagger, the VM bundle, and dependency packaging.