AIPCC-31724: Add airgapped prefetch bundles

docs/concepts/architecture-overview.rst

@@ -97,6 +97,21 @@ The ``build`` command uses the same source acquisition and build
 subsystems but skips resolution and recursion -- it compiles a
 single package given a name, version, and source URL.
 
+Prefetch Bundles
+----------------
+
+.. versionadded:: 0.92.0
+
+``bootstrap --prefetch-dir`` separates network-enabled discovery from an
+airgapped production build. It runs the sdist-only bootstrap pipeline to
+dependency closure, then exports prepared source archives, bootstrap wheels, prebuilt
+wheels, ``graph.json``, and ``build-order.json`` into a versioned bundle.
+
+``build-sequence --prefetch-dir`` verifies the bundle manifest, skips source
+resolution, downloading, and source preparation, and builds from the prepared
+source archives. Build-environment dependencies are installed from local wheelhouse
+directories with the package index disabled.
+
 Extension Points
 ----------------
 
@@ -126,7 +141,7 @@ breakdown.
 Key Data Structures
 -------------------
 
-Four data structures flow between subsystems:
+Five data structures flow between subsystems:
 
 .. list-table::
    :header-rows: 1
@@ -148,3 +163,6 @@ Four data structures flow between subsystems:
    * - ``BuildEnvironment``
      - Isolated virtual environment for building one package.  Created
        per source build, cleaned up after completion.
+   * - ``PrefetchBundle``
+     - Verified manifest and local artifacts used to move dependency discovery
+       and source preparation ahead of an airgapped build.

docs/concepts/bootstrap-vs-build.rst

@@ -107,3 +107,7 @@ Typical Workflow
    ``build-order.json`` from bootstrap to rebuild deterministically
 
 3. **Fixes:** Use ``build`` to rebuild individual packages after applying patches
+
+4. **Airgapped builds:** Use ``bootstrap --prefetch-dir`` while connected, copy
+   the resulting bundle into the isolated environment, then run
+   ``build-sequence --prefetch-dir``.

docs/reference/files.md

@@ -114,6 +114,34 @@ The following example shows `graph.json` file for the top-level dependency `whee
 }
 ```
 
+## Prefetch bundle
+
+`bootstrap --prefetch-dir` exports the files needed by a later airgapped build:
+
+```text
+prefetch-output/
+├── manifest.json
+├── graph.json
+├── build-order.json
+├── constraints.txt
+├── merged-constraints.txt
+├── sdists/
+├── wheels/
+└── prebuilt/
+```
+
+`sdists/` contains source archives after `prepare_source` hooks, project
+overrides, patches, and Rust vendoring have run. When a hook prepares files
+beside the source root, the archive preserves those sibling directories too.
+`wheels/` contains wheels needed to create build environments, and `prebuilt/`
+contains packages intentionally used as binary distributions.
+
+`manifest.json` records the format version, build variant, Python environment,
+effective settings fingerprint, package-to-artifact mapping, exact build
+requirements, file sizes, and SHA-256 digests. All paths in the manifest are
+relative to the bundle directory. `build-sequence --prefetch-dir` validates the
+manifest before starting any builds.
+
 ## Output Directories
 
 During the wheel building process, fromager generates multiple output directories namely `sdists-repo`, `wheels-repo` and `work-dir`. These directories contain important information related to the wheel build.

docs/using.md

@@ -127,6 +127,33 @@ Production builds use separate commands for the steps described as
 part of bootstrapping, and accept arguments to control the servers
 that are used for downloading source or built wheels.
 
+### Airgapped builds
+
+Use `bootstrap --prefetch-dir` to prepare a dependency graph and all source
+artifacts while network access is available. Prefetch mode uses the sdist-only
+bootstrap path, runs source preparation hooks, and exports prepared source
+archives (including patched sdists and any prepared sibling directories),
+bootstrap wheels, prebuilt wheels, the dependency graph, and the build order.
+
+```bash
+fromager -O bootstrap-output bootstrap \
+  --prefetch-dir prefetch-output \
+  -r requirements.txt
+```
+
+Copy `prefetch-output` into the isolated environment, then build directly from
+the bundle. The build validates all recorded file sizes and SHA-256 digests,
+does not resolve or download sources, and installs build dependencies from the
+local wheel directories using the exact versions selected during prefetch.
+
+```bash
+fromager -O build-output build-sequence --prefetch-dir prefetch-output
+```
+
+The build variant and package settings must match the prefetch run. The first
+implementation supports `build-sequence`; parallel bundle builds are not yet
+supported.
+
 Two commands support building wheels from source.
 
 ### The build command

src/fromager/bootstrapper/_bootstrapper.py

@@ -65,6 +65,7 @@ def __init__(
         test_mode: bool = False,
         multiple_versions: bool = False,
         num_bg_threads: int = DEFAULT_BG_THREADS,
+        capture_build_requirements: bool = False,
     ) -> None:
         if test_mode and sdist_only:
             raise ValueError(
@@ -78,6 +79,7 @@ def __init__(
         self.sdist_only = sdist_only
         self.test_mode = test_mode
         self.multiple_versions = multiple_versions
+        self.capture_build_requirements = capture_build_requirements
         self.why: list[tuple[RequirementType, Requirement, Version]] = []
         self._num_bg_threads = max(1, num_bg_threads)
         self._bg_pool: concurrent.futures.ThreadPoolExecutor | None = (
@@ -699,6 +701,9 @@ def add_to_build_order(
         source_type: SourceType,
         prebuilt: bool = False,
         constraint: Requirement | None = None,
+        build_requirements: typing.Mapping[str, Version] | None = None,
+        prepared_source_root: str | None = None,
+        prepared_source_workspace: str | None = None,
     ) -> None:
         """Append a package to the build-order list if not already present.
 
@@ -715,6 +720,11 @@ def add_to_build_order(
             prebuilt: True if the wheel was taken from a prebuilt cache rather
                 than compiled locally.
             constraint: The version constraint active for this package, if any.
+            build_requirements: Exact distributions installed in the build environment.
+            prepared_source_root: Source root relative to ``ctx.work_dir`` when
+                capturing a prefetch build.
+            prepared_source_workspace: Source workspace relative to ``ctx.work_dir``
+                when capturing a prefetch build.
         """
         # We only care if this version of this package has been built,
         # and don't want to trigger building it twice. The "extras"
@@ -735,6 +745,15 @@ def add_to_build_order(
             "source_url": source_url,
             "source_url_type": str(source_type),
         }
+        if build_requirements is not None:
+            info["build_requirements"] = [
+                f"{canonicalize_name(name)}=={version}"
+                for name, version in sorted(build_requirements.items())
+            ]
+        if prepared_source_root is not None:
+            info["prepared_source_root"] = prepared_source_root
+        if prepared_source_workspace is not None:
+            info["prepared_source_workspace"] = prepared_source_workspace
         self._build_stack.append(info)
 
     def _check_write_error(self, fut: concurrent.futures.Future[int]) -> None:

src/fromager/bootstrapper/_build.py

@@ -58,6 +58,9 @@ def run(self, bt: Bootstrapper) -> list[Phase]:
         if remaining_deps.isdisjoint(wi.build_system_deps):
             wi.build_env.install(remaining_deps)
 
+        if bt.capture_build_requirements:
+            wi.resolved_build_requirements = dict(wi.build_env.get_distributions())
+
         wheel_filename, sdist_filename = self.do_build(bt.ctx, bt.explain)
 
         source_type = sources.get_source_type(bt.ctx, wi.req)

src/fromager/bootstrapper/_prepare_source.py

@@ -28,21 +28,23 @@ def _bg_prepare_source(
     req: Requirement,
     resolved_version: Version,
     source_url: str,
+    use_wheel_cache: bool = True,
 ) -> PreparedSourceData:
     """Background-safe source download+unpack: no Bootstrapper state accessed."""
     # Thread-safe: _seen_requirements in the main thread prevents the same
     # package from being submitted to the thread pool more than once.
     # Paths from download_source() and prepare_source() include {name}-{version},
     # making them unique across concurrent threads processing different packages.
     logger.info("preparing source")
-    cached_wheel, unpacked = _cache.find_cached_wheel(
-        ctx, cache_wheel_server_url, req, resolved_version
-    )
-    if unpacked is not None:
-        return PreparedSourceData(
-            sdist_root_dir=unpacked / unpacked.stem,
-            cached_wheel_filename=cached_wheel,
+    if use_wheel_cache:
+        cached_wheel, unpacked = _cache.find_cached_wheel(
+            ctx, cache_wheel_server_url, req, resolved_version
         )
+        if unpacked is not None:
+            return PreparedSourceData(
+                sdist_root_dir=unpacked / unpacked.stem,
+                cached_wheel_filename=cached_wheel,
+            )
     source_filename = sources.download_source(
         ctx=ctx, req=req, version=resolved_version, download_url=source_url
     )
@@ -96,7 +98,12 @@ def do_prepare_prebuilt() -> PreparedSourceData:
         def do_prepare_source() -> PreparedSourceData:
             with req_ctxvar_context(req, resolved_version):
                 return _bg_prepare_source(
-                    ctx, cache_wheel_server_url, req, resolved_version, source_url
+                    ctx,
+                    cache_wheel_server_url,
+                    req,
+                    resolved_version,
+                    source_url,
+                    use_wheel_cache=not bt.capture_build_requirements,
                 )
 
         return do_prepare_source

src/fromager/bootstrapper/_process_install_deps.py

@@ -149,13 +149,32 @@ def run(self, bt: Bootstrapper) -> list[Phase]:
 
         pbi = bt.ctx.package_build_info(wi.req)
         constraint = bt.ctx.constraints.get_constraint(wi.req.name)
+        build_order_kwargs: dict[str, typing.Any] = {
+            "req": wi.req,
+            "version": wi.resolved_version,
+            "source_url": wi.source_url,
+            "source_type": wi.build_result.source_type,
+            "prebuilt": pbi.pre_built,
+            "constraint": constraint,
+            "build_requirements": (
+                wi.resolved_build_requirements
+                if bt.capture_build_requirements
+                else None
+            ),
+        }
+        if bt.capture_build_requirements and wi.build_result.sdist_root_dir is not None:
+            build_order_kwargs.update(
+                {
+                    "prepared_source_root": str(
+                        wi.build_result.sdist_root_dir.relative_to(bt.ctx.work_dir)
+                    ),
+                    "prepared_source_workspace": str(
+                        wi.build_result.unpack_dir.relative_to(bt.ctx.work_dir)
+                    ),
+                }
+            )
         bt.add_to_build_order(
-            req=wi.req,
-            version=wi.resolved_version,
-            source_url=wi.source_url,
-            source_type=wi.build_result.source_type,
-            prebuilt=pbi.pre_built,
-            constraint=constraint,
+            **build_order_kwargs,
         )
 
         dep_items: list[Phase] = bt.create_unresolved_work_items(

src/fromager/bootstrapper/_work_item.py

@@ -50,6 +50,9 @@ class WorkItem:
     build_system_deps: set[Requirement] = dataclasses.field(default_factory=set)
     build_backend_deps: set[Requirement] = dataclasses.field(default_factory=set)
     build_sdist_deps: set[Requirement] = dataclasses.field(default_factory=set)
+    resolved_build_requirements: dict[str, Version] = dataclasses.field(
+        default_factory=dict
+    )
 
     def is_build_requirement_context(self) -> bool:
         """Return True if this item is being processed as part of a build requirement.

src/fromager/build_environment.py

@@ -48,6 +48,9 @@ def __init__(
         self.all_reqs = all_reqs
         resolutions = []
         for r in all_reqs:
+            if ctx.offline:
+                resolutions.append(f"{r} -> missing from prefetch bundle")
+                continue
             try:
                 _, version = resolver.resolve(
                     ctx=ctx,
@@ -154,6 +157,17 @@ def get_venv_environ(
         venv_environ["UV_NO_MANAGED_PYTHON"] = "true"
         venv_environ["UV_PYTHON_DOWNLOADS"] = "never"
         venv_environ["UV_PYTHON"] = str(self.python)
+        if self._ctx.offline:
+            venv_environ["UV_OFFLINE"] = "true"
+            venv_environ["UV_NO_CACHE"] = "true"
+            venv_environ["PIP_NO_INDEX"] = "1"
+            venv_environ["PIP_FIND_LINKS"] = " ".join(
+                os.fspath(path) for path in self._ctx.wheelhouse_dirs
+            )
+            if self._ctx.offline_constraints_file is not None:
+                venv_environ["PIP_CONSTRAINT"] = os.fspath(
+                    self._ctx.offline_constraints_file
+                )
 
         return venv_environ
 
@@ -233,7 +247,9 @@ def install(self, reqs: typing.Iterable[Requirement]) -> None:
         self.run(
             cmd,
             cwd=str(self.path.parent),
-            network_isolation=False,
+            network_isolation=(
+                self._ctx.network_isolation if self._ctx.offline else False
+            ),
         )
         logger.info(
             "installed dependencies %s into build environment in %s",
@@ -340,6 +356,40 @@ def prepare_build_environment(
     return build_env
 
 
+@metrics.timeit(description="prepare prefetched build environment")
+def prepare_build_environment_from_prefetch(
+    *,
+    ctx: context.WorkContext,
+    req: Requirement,
+    sdist_root_dir: pathlib.Path,
+    build_requirements: typing.Iterable[Requirement],
+) -> BuildEnvironment:
+    """Create a build environment from graph-resolved requirements.
+
+    Args:
+        ctx: Offline work context.
+        req: Requirement being built.
+        sdist_root_dir: Root directory of the unpacked prepared source archive.
+        build_requirements: Exact requirements captured during prefetch.
+
+    Returns:
+        Prepared build environment.
+    """
+    build_env = BuildEnvironment(
+        ctx=ctx,
+        req=req,
+        sdist_root_dir=sdist_root_dir,
+    )
+    _safe_install(
+        ctx=ctx,
+        req=req,
+        build_env=build_env,
+        deps=set(build_requirements),
+        dep_req_type=RequirementType.BUILD_SYSTEM,
+    )
+    return build_env
+
+
 def _safe_install(
     ctx: context.WorkContext,
     req: Requirement,