Repository navigation
Discussion: Strategy to remove shell-exec from compilation path #202
Description
Activity
TinyGo, by design and convention, has depended on external tools to handle parts of compilation (LLVM, asyncify, wasm-ld, among others). Based on statements made by the Go team and by historical precedent, I (and others, I believe) assume that this strategy would not apply to mainline ("big") Go. The
gocommand (and associated toolchain) contains the necessary capability to compile (and cross-compile) binaries for a variety of platforms, including wasm.- Is the issue that shell exec exists at all in the current Component Model tooling for Go?
- Do we (the authors and maintainers of this tooling, TinyGo, and Go) agree that this is an issue worth addressing?
- Wasm in Go has so far taken an approach of "prototype in TinyGo, learn and implement later in big Go."
- With respect to tooling and compilation, what if TinyGo skipped the
wasm-tools componentsteps, and generated awasm32-wasip2-moduleform suitable for consumption bywasm-toolsor conforming runtimes?
we chatted about it during BA's meeting and no decisive actions other than researching what wasm-tools is actually doing under the hood and follow up next meeting.
We identified a few options to pass down wit:
- Continue with
witdirectory so wasm-tools (or something else) can perform embedding - Nuke the
witdirectory and rely on information from generated code in downstream tooling. Options:go embedper package: Attaching each interface wit to its corresponding go packagego embedper project: Attach the whole wit to the entrypoint package under compilation- A combination of
go:wasm{import,export}and markers to carry type information, so the wit can be discarded after codegen
With respect to tooling and compilation, what if TinyGo skipped the wasm-tools component steps, and generated a WebAssembly/component-model#378 form suitable for consumption by wasm-tools or conforming runtimes?
This is a 🔑 point. We touched briefly on the point wrt making TinyGo / BigGo emit wasm32 module and leave
wasm-toolssteps to other tools. It would allow us to defer implementations based on upstream stability ( and yes, I'm looking at you WIT spec! 😂 )The interesting bit here is
wasm-tools-gois already a requirement, so something like this is not far from reality:# generate bindings go run wasm-tools/cmd/wit-bindgen-go # compile module ( core wasm ) tinygo build ... -o module.wasm # componetize it go run wasm-tools/cmd/assemble-component -in module.wasm -out component.wasmwasm-tools-goas pre & post compilation steps, havingwit-bindgen-goandassemble-componentdefining their own api / data passing mechanism.The first incarnation of
assemble-componentcould be a full shell exec intowasm-tools, and feels this is a pattern acceptable in Big Go.again, no decisions made but wanted to relay the discussion.
- Continue with
I have WIT generation working with tree shaking in a branch, so we can emit minimal WIT files containing a single world to a world's Go package directory.
wasm-tools-go is only a prerequisite for code generation, so it's not really required to compile and build a Go program. This is similar to other tools used by big Go projects like protoc or avo.
Reacted by Ethan M LewisWork is progressing here:
- @Mossaka has put up a draft PR to run
wasm-toolsas a WASI 0.1 module inside ofwit-bindgen-go: internal/wasmtools: run wasm-tools.wasm in embedded Wazero runtime #252 - @lukewagner is working on new module mode linker name mangling spec: Add BuildTargets.md WebAssembly/component-model#378
- @alexcrichton merged early support for the new linker name mangling into wasm-tools: Implement support for the standard mangling scheme in the component model wasm-tools#1828
- @Mossaka has put up a draft PR to run
TinyGo's current behavior is to handle module -> component conversion for
-target wasip2.Where is shell exec today?
wit-bindgen-go
https://github.com/bytecodealliance/wasm-tools-go/blob/main/wit/load.go#L54
wit: parsing / world resolver
Uses wasm-tools to convert wit -> resolved world json.
The resulting json is converted into structs for codegen.
TinyGo
https://github.com/tinygo-org/tinygo/blob/release/builder/build.go#L864
wit: parsing / world resolver
wasm metadata: attach wit to wasm (
wasm-tools component embed)wasm component encoder: convert wasm module to component (
wasm-tools component new)Uses
wasm-toolsto adapt a wasm module to component.Discussion
Should TinyGo / Go generate core wasm modules and leave metadata/component encoding to 3rd party tools; or should they be part of a wasi target ( or goos/goarch combination )?
Assuming we implement wit parsing / wasm patching in Go: