diff --git a/.agents/skills/alpine.md b/.agents/skills/alpine.md index ab7ace9..e8841db 100644 --- a/.agents/skills/alpine.md +++ b/.agents/skills/alpine.md @@ -145,6 +145,85 @@ Apptainer image. For NF1, prefer grouping compatible ZedProfiler feature familie by image-set when possible so each task loads image data once and writes validated outputs. +## uv For ZedProfiler + +CURC documents a `uv` module for Python environments. The documented flow is: + +```bash +module load uv +uv venv "$UV_ENVS/mycustomenv" --python 3.12 +source "$UV_ENVS/mycustomenv/bin/activate" +uv pip install +``` + +The documentation says `module load uv` creates and sets `UV_ENVS` to +`/projects/$USER/software/uv/envs`. During live validation on `2026-08-04`, +neither the regular login node nor `Persistence1` exposed an `uv` module via +`module avail uv` or `module spider uv`. + +The tested fallback is a project-owned `uv` install that keeps the same CURC +environment layout: + +```bash +export UV_HOME="/projects/$USER/software/uv" +export UV_INSTALL_DIR="$UV_HOME/bin" +export UV_ENVS="$UV_HOME/envs" +export UV_CACHE_DIR="/scratch/alpine/$USER/uv-cache" +export PATH="$UV_INSTALL_DIR:$PATH" +mkdir -p "$UV_INSTALL_DIR" "$UV_ENVS" "$UV_CACHE_DIR" + +if [[ ! -x "$UV_INSTALL_DIR/uv" ]]; then + curl -LsSf https://astral.sh/uv/install.sh | sh +fi + +uv venv "$UV_ENVS/zedprofiler-simple" --python 3.12 +source "$UV_ENVS/zedprofiler-simple/bin/activate" +uv pip install zedprofiler +python - <<'PY' +import importlib.metadata as metadata +import zedprofiler +print(metadata.version("zedprofiler")) +print(zedprofiler.__file__) +PY +``` + +Observed validation: + +- `uv 0.12.1` installed under `/projects/$USER/software/uv/bin` +- `uv venv --python 3.12` used `/usr/bin/python3.12` +- ZedProfiler `0.1.1` installed and imported +- heavy dependencies resolved as wheels, including `mahotas`, `pyarrow`, + `numpy`, `pandas`, and `scikit-image` +- with `UV_CACHE_DIR` on scratch and the env under `/projects`, `uv` warned + that hardlinking was not available and fell back to copying; set + `UV_LINK_MODE=copy` to make that explicit + +Validated simple experiment using the `uv` environment: + +```bash +cd /scratch/alpine/$USER/formascute-codex-test +module load nextflow/25.10.2 +source "/projects/$USER/software/uv/envs/zedprofiler-simple/bin/activate" +make run EXPERIMENT=zp_synthetic_features ITEMS=4 ACCOUNT=amc-general PROFILE=alpine RUN_ID=zp-sim-uv-01 +``` + +Observed result: + +- Nextflow ran from `/curc/sw/install/bio/nextflow/25.10.2_env/bin/nextflow` +- `python3` resolved to the `uv` environment +- preflight recorded Python `3.12.13` +- `completion_status.txt` reported `nextflow_exit_status: 0` +- `validation.json` reported `"valid": true` +- 4 feature tasks plus 1 validation task were submitted to Slurm +- 16 simulated feature rows were emitted + +For current repository behavior, prefer this direct `Persistence1` run pattern +when testing `uv`: load the Nextflow module first, then activate the `uv` +environment before invoking `make run`. The generated `make submit` coordinator +currently treats `FORMASCUTE_ENV_DIR` as a replacement environment and skips +module loading when it exists, so it is not yet the right interface for a +combined Nextflow-module plus `uv` Python environment. + ## Important Failure Mode Submitting directly from the regular Alpine login node failed because the batch diff --git a/docs/alpine-findings.md b/docs/alpine-findings.md index 3284090..335e22b 100644 --- a/docs/alpine-findings.md +++ b/docs/alpine-findings.md @@ -14,6 +14,9 @@ image data were used. not expose that module during testing. - `/usr/bin/apptainer` and `/usr/bin/singularity` are available on the tested compute environment. +- A project-owned `uv` install under `/projects/$USER/software/uv` can build a + Python 3.12 ZedProfiler environment when the documented CURC `uv` module is + not visible. The validated smoke command was: @@ -48,6 +51,7 @@ All runs validated successfully. | sequential | `nf1_featurization_batched` | 16 | 4 | 00:00:41 | 5 | 1 s | | sequential | `nf1_queue_size_4` | 16 | 1 | 00:01:02 | 17 | 1 s | | burst | `zp_synthetic_features` | 4 | 1 | about 00:00:42 | 5 | 0-1 s | +| direct `uv` env | `zp_synthetic_features` | 4 | 1 | about 00:00:16 task span | 5 | 0-1 s | ## ZedProfiler-Shaped Probe @@ -87,6 +91,61 @@ present in the tested base environment. A production NF1/ZedProfiler run should therefore use a project-owned environment or container instead of the default `Persistence1` Python. +## uv Environment Probe + +CURC documents `module load uv`, with `$UV_ENVS` set to +`/projects/$USER/software/uv/envs`, as the preferred `uv` environment location. +During validation on `2026-08-04`, `module avail uv` and `module spider uv` did +not find that module on either the regular login node or `Persistence1`. + +A user-space fallback worked: + +```bash +export UV_HOME="/projects/$USER/software/uv" +export UV_INSTALL_DIR="$UV_HOME/bin" +export UV_ENVS="$UV_HOME/envs" +export UV_CACHE_DIR="/scratch/alpine/$USER/uv-cache" +export PATH="$UV_INSTALL_DIR:$PATH" +export UV_LINK_MODE=copy + +uv venv "$UV_ENVS/zedprofiler-simple" --python 3.12 +source "$UV_ENVS/zedprofiler-simple/bin/activate" +uv pip install zedprofiler +``` + +Observed result: + +- `uv 0.12.1` installed under `/projects/$USER/software/uv/bin`. +- The environment used `/usr/bin/python3.12`, reporting Python `3.12.13`. +- ZedProfiler `0.1.1` installed and imported. +- Wheels resolved for previously missing dependencies including `mahotas`, + `pyarrow`, `numpy`, `pandas`, and `scikit-image`. +- Keeping `UV_CACHE_DIR` on scratch and the environment under `/projects` causes + cross-filesystem hardlink fallback; `UV_LINK_MODE=copy` should be set + intentionally for this layout. + +The simple Nextflow experiment also completed with the `uv` environment active: + +```bash +cd /scratch/alpine/$USER/formascute-codex-test +module load nextflow/25.10.2 +source "/projects/$USER/software/uv/envs/zedprofiler-simple/bin/activate" +make run EXPERIMENT=zp_synthetic_features ITEMS=4 ACCOUNT=amc-general PROFILE=alpine RUN_ID=zp-sim-uv-01 +``` + +Result: + +- Nextflow exit status was `0`. +- validation passed for 4 expected items. +- the run emitted 16 simulated feature rows. +- `trace.tsv` contained 4 feature tasks and 1 validation task. +- `slurm.tsv` was collected for account `amc-general`. + +This upgrades the ZedProfiler runtime recommendation from "project-owned +environment or container" to "`uv` environment first, Apptainer if native wheels +or system libraries fail." It still does not validate real NF1 image I/O or a +real ZedProfiler feature call inside a workflow task. + ## Interpretation - Nextflow can successfully submit and track Slurm jobs on Alpine when submitted @@ -102,6 +161,12 @@ therefore use a project-owned environment or container instead of the default control knob for NF1-style work. - The ZedProfiler-shaped probe behaved like the earlier tiny fan-out tests: scheduler wait was negligible, and task overhead dominated actual work. +- `uv` is a practical first runtime path for ZedProfiler on Alpine, even though + the documented module was not visible during testing. +- The current `make submit` coordinator is not yet the best `uv` interface + because it treats `FORMASCUTE_ENV_DIR` as a replacement for module loading. + For now, direct `Persistence1` runs should load Nextflow first and then + activate the `uv` environment. ## Recommended NF1 Orchestration Direction @@ -114,7 +179,10 @@ Recommended first implementation: - Use one Nextflow process for CPU featurization work items. - Keep the existing Python feature scripts as the task payload. - Use Slurm executor with `acpu`, `cpu-normal`, and account configuration. -- Build an explicit ZedProfiler runtime environment before real data tests. +- Use a project-owned `uv` environment as the first ZedProfiler runtime before + real data tests. +- Update the coordinator bootstrap so it can load the Nextflow module and + activate a separate Python runtime environment in the same submitted job. - Start with independent jobs for real feature tasks when each task loads a single image-set and runs enough feature work to amortize Slurm overhead. - Add optional batching when individual feature calls are very short or repeatedly @@ -135,6 +203,10 @@ Avoid for the first production-facing iteration: - Mixing CPU and GPU work in one executor profile. - Running real NF1 image data before a tiny real-work probe is designed. - Relying on the base `Persistence1` Python environment for ZedProfiler. +- Assuming the documented CURC `uv` module is available without checking + `module spider uv`. +- Pointing `FORMASCUTE_ENV_DIR` at the `uv` environment before the submit + wrapper can separately load Nextflow. ## Questions For CURC @@ -150,5 +222,7 @@ Ask CURC these before scaling: independent submission easier for accounting and retries? - Should CPU featurization and SAMMed3D GPU work be split into separate Nextflow profiles and runs? -- Should ZedProfiler be installed as a project conda/uv environment or as an - Apptainer image for production NF1 runs? +- Is CURC's documented `uv` module expected to be available on Alpine now, and + if so, which module tree or host should expose it? +- For production NF1 runs, is a project-owned `uv` environment acceptable, or + should CURC prefer an Apptainer image once image I/O dependencies are known? diff --git a/docs/nf1-pipeline-target.md b/docs/nf1-pipeline-target.md index 27dcaa9..b70c80d 100644 --- a/docs/nf1-pipeline-target.md +++ b/docs/nf1-pipeline-target.md @@ -56,6 +56,15 @@ make submit EXPERIMENT=nf1_submit_rate_4 ITEMS=16 ACCOUNT= SUBMIT_HO make submit EXPERIMENT=zp_synthetic_features ITEMS=8 ACCOUNT= SUBMIT_HOST=Persistence1 ``` +For `uv` runtime checks, use a direct `Persistence1` run until the submit +wrapper can load the Nextflow module and activate a separate Python environment: + +```bash +module load nextflow/25.10.2 +source "/projects/$USER/software/uv/envs/zedprofiler-simple/bin/activate" +make run EXPERIMENT=zp_synthetic_features ITEMS=4 ACCOUNT= PROFILE=alpine +``` + Interpretation: - `nf1_minimal_overhead` estimates Nextflow + Slurm coordinator overhead before @@ -94,9 +103,17 @@ tasks. On Alpine, the base `Persistence1` Python environment is not ready for real ZedProfiler work because it provides Python 3.9 and lacks required packages such as `mahotas`, `pyarrow`, and `zedprofiler`. +A project-owned `uv` environment is now the preferred first runtime path: +Python 3.12.13, ZedProfiler `0.1.1`, `mahotas`, `pyarrow`, `numpy`, `pandas`, +and `scikit-image` installed and imported successfully from +`/projects/$USER/software/uv/envs/zedprofiler-simple`. The documented CURC `uv` +module was not visible during validation, so the tested fallback installs `uv` +under `/projects/$USER/software/uv/bin` and uses scratch for `UV_CACHE_DIR`. + For the NF1 refactor, prefer: -- a project-owned Python 3.11+ environment or Apptainer image for ZedProfiler +- a project-owned `uv` Python 3.11+ environment first, with Apptainer as the + fallback if native wheels or system libraries fail - a Nextflow process that consumes a manifest of image-set work items - task payloads that can compute multiple compatible feature families after one image load @@ -106,9 +123,9 @@ For the NF1 refactor, prefer: ## What This Does Not Answer Yet These experiments do not measure image-processing runtime, GPU memory, CellPose, -SAMMed3D, DuckDB, parquet performance, or real ZedProfiler dependency -installation. They only characterize scheduler, submission, retry, validation, -and artifact behavior with a small controlled workload. +SAMMed3D, DuckDB, parquet performance, or real ZedProfiler feature execution. +They only characterize scheduler, submission, retry, validation, artifact +behavior, and the basic viability of installing/importing ZedProfiler with `uv`. Do not use NF1 image data in this phase. Data-based testing belongs in a later phase, after synthetic runs have shown stable submission, accounting, diff --git a/docs/north-star.md b/docs/north-star.md index fe739d4..1b668c2 100644 --- a/docs/north-star.md +++ b/docs/north-star.md @@ -26,8 +26,10 @@ This should be a thin workflow layer around real work, not a broad rewrite. separate profiles or separate runs. - Do not rely on the base `Persistence1` Python for ZedProfiler. It is Python 3.9 and lacks required packages. -- Use a project-owned Python 3.11+ environment or an Apptainer image for - ZedProfiler. +- Use a project-owned `uv` Python 3.11+ environment as the first ZedProfiler + runtime path; use Apptainer if native wheels or system libraries fail. +- The documented CURC `uv` module was not visible during validation, but a + user-space `uv` install under `/projects/$USER/software/uv/bin` worked. ## Target Shape @@ -140,7 +142,7 @@ feature extraction with a tiny controlled manifest. Recommended slice: 1. Add a manifest builder for a few known-safe image-sets. -2. Add a ZedProfiler runtime environment definition. +2. Add a ZedProfiler `uv` runtime environment definition. 3. Add one Nextflow process for CPU feature extraction. 4. Add per-task validation and summary output. 5. Run one image-set with real ZedProfiler on Alpine. @@ -155,7 +157,7 @@ or production data movement in the first slice. Scale only after these gates pass: - formascute synthetic Alpine checks pass from a fresh clone -- ZedProfiler environment builds reproducibly on Alpine +- ZedProfiler `uv` environment builds reproducibly on Alpine - one real image-set completes and validates - 4-8 real image-sets complete and validate - `trace.tsv` and `slurm.tsv` show acceptable queue wait and memory use @@ -178,7 +180,7 @@ Avoid these as first moves: The best direction today is a thin Nextflow wrapper around NF1 Stage 3 that consumes a manifest of image-set work items and runs ZedProfiler in a controlled -Python 3.11+ environment. Start with independent CPU tasks on `acpu` / +`uv` Python 3.11+ environment. Start with independent CPU tasks on `acpu` / `cpu-normal` through `Persistence1`, validate every output, collect trace and Slurm accounting, and tune batching only after real image-set runtimes show whether repeated image loading or scheduler overhead dominates.