Cytotable skills

.agents/skills/alpine.md → .agents/skills/alpine/SKILL.md

@@ -1,3 +1,8 @@
+---
+name: alpine
+description: Use before changing, testing, or scaling formascute on CURC Alpine, especially for Nextflow, Slurm submission, Persistence1, Python runtimes, Apptainer/Singularity, uv environments, queue limits, or Alpine filesystem behavior.
+---
+
 # Alpine Skill
 
 Use this note before changing, testing, or scaling formascute on CURC Alpine,
@@ -7,7 +12,11 @@ environment behind the project's imaging/feature-extraction workload.
 ## Current Position
 
 - Submit and orchestrate through `Persistence1`; treat this as selected project
-  policy, not an open question.
+  policy for Nextflow and other long-lived workflow managers, not an open
+  question for that workflow shape.
+- For non-Nextflow, single-node Python tools, a plain Slurm batch job submitted
+  from the Alpine login node is acceptable. Do not introduce `Persistence1`
+  just to run one compute-node Python process.
 - Keep compute work on Slurm. `Persistence1` is for the long-lived workflow
   manager, not image processing or other heavy work.
 - Use `queueSize = 200` as the production default. It matches a real, hard
@@ -169,18 +178,19 @@ Nextflow-orchestrated container tasks.
 
 ## Slurm Defaults
 
-*Scope: `account`/`partition`/`QoS`/`submit host` are general Slurm settings —
-they apply to any job on Alpine, including a plain `sbatch` script with no
-Nextflow involved. `executor`, `queueSize`, and the submit throttle are
-Nextflow executor settings and only mean something when using Nextflow.*
+*Scope: `account`/`partition`/`QoS` are general Slurm settings — they apply to
+any job on Alpine, including a plain `sbatch` script with no Nextflow involved.
+`submit host`, `executor`, `queueSize`, and the submit throttle are Nextflow
+or workflow-manager settings and only mean something for that orchestration
+shape.*
 
 Use these defaults for CPU work unless the user or CURC gives a newer allocation
 policy:
 
 - account: project allocation supplied by the user
 - partition: `acpu`
 - QoS: `cpu-normal`
-- submit host: `Persistence1`
+- submit host: `Persistence1` *(Nextflow / workflow-manager orchestration only)*
 - executor: Slurm *(Nextflow only)*
 - production `queueSize`: `200` *(Nextflow only)*
 - production submit throttle: do not enable by default; see Queue And
@@ -891,6 +901,11 @@ blockers — there are no known blockers on the Apptainer side beyond the
 
 ### General (Alpine / Apptainer, Not Nextflow-Specific)
 
+- Do not let `$HOME` fill up. Alpine SSH/session setup can fail before a remote
+  command runs when home is `100%` full; observed failure text included
+  `sed: couldn't open temporary file /home/.../.ssh/sed...: No space left on
+  device`. Keep archives, virtual environments, caches, benchmark outputs, and
+  Parsl/Nextflow work directories on scratch or project storage, not home.
 - Do not rely on the base `Persistence1` Python environment for real
   production work — it lacks the project's required Python version and
   scientific dependencies.

.agents/skills/cytotable/SKILL.md

@@ -0,0 +1,254 @@
+---
+name: cytotable
+description: Use before running, tuning, or documenting CytoTable conversions on CURC Alpine, especially for Parsl executor choice, Slurm CPU and memory shape, DuckDB/Arrow settings, S3-backed large tests, scratch-backed HOME, or uv runtime setup.
+---
+
+# Cytotable Skill
+
+Use this note before running, tuning, or documenting CytoTable conversions on
+CURC Alpine, especially when choosing a Parsl executor, Slurm CPU shape, Python
+runtime, or scratch/project storage layout.
+
+Also read `.agents/skills/alpine/SKILL.md` for Alpine-wide Slurm, storage, and
+SSH facts. This skill is Cytotable-specific; keep general Alpine policy in the
+Alpine skill instead.
+
+## Current Position
+
+- Run Cytotable as a regular Slurm batch job on compute nodes. Do not route it
+  through `Persistence1` unless another orchestrator actually needs that host.
+- Put source data, destination parquet, Parsl run directories, and Cytotable
+  benchmark logs under `/scratch/alpine/$USER`.
+- Put reusable Python environments under `/projects/$USER/software/uv/envs` and
+  keep the `uv` cache on scratch with `UV_LINK_MODE=copy`.
+- Run the Cytotable Python process with a scratch-backed `HOME` so DuckDB
+  extension installs do not touch Alpine's small home quota. Cytotable loads
+  DuckDB extensions (`sqlite_scanner`, `httpfs`) via DuckDB's default
+  extension cache under `$HOME/.duckdb`; a full home directory causes
+  immediate conversion failure before executor differences matter.
+- Set `CYTOTABLE_MAX_THREADS` before Python starts. Cytotable reads it at import
+  time and passes it into DuckDB thread configuration.
+- Start with one Cytotable conversion per Slurm job, using one node, explicit
+  `--time`, explicit `--mem`, and `--cpus-per-task` matched to
+  `CYTOTABLE_MAX_THREADS`.
+- Use `ThreadPoolExecutor(max_threads=<cpus>)` as the first candidate for
+  local Alpine conversion jobs. In one validated `4` CPU comparison on the
+  bundled CSV fixture, threadpool was slightly faster than capped HTEX.
+- Keep capped `HighThroughputExecutor(max_workers_per_node=<cpus>)` as the
+  comparison candidate when memory is the concern. In the same single
+  comparison, capped HTEX used less Slurm-reported peak RSS but ran slightly
+  slower. Do not use default uncapped HTEX for production guidance until it has
+  passed the same scratch-home benchmark.
+- A practical first production-shaped request is `--cpus-per-task=4`,
+  `--mem=8G`, `--time=00:20:00` for the bundled test-size CSV conversion, then
+  scale from measured walltime and RSS. Re-measure for larger real datasets.
+
+## DuckDB And Arrow Settings
+
+- Set `CYTOTABLE_MAX_THREADS="$SLURM_CPUS_PER_TASK"` in the batch job before
+  Python starts. Cytotable reads this at import time and uses it for DuckDB
+  `PRAGMA threads`; setting it later in Python is too late.
+- Set `HOME` to a scratch job-home after activating the reusable environment
+  and before starting Python, for example:
+
+  ```bash
+  export CYTOTABLE_JOB_HOME="/scratch/alpine/$USER/cytotable-job-home/$SLURM_JOB_ID"
+  mkdir -p "$CYTOTABLE_JOB_HOME"
+  export HOME="$CYTOTABLE_JOB_HOME"
+  ```
+
+  This redirects DuckDB's default extension cache (`$HOME/.duckdb`) to scratch.
+  A validated run created about `54 MB` per scratch job-home.
+- Leave Cytotable's default Arrow memory mapping enabled for initial runs:
+  `CYTOTABLE_ARROW_USE_MEMORY_MAPPING=1`. This is the default and should be a
+  reasonable fit for scratch-backed parquet reads.
+- If a run shows unexpectedly high `MaxVMSize`, file-handle/unlink behavior, or
+  memory pressure during concat/join, run a controlled comparison with
+  `CYTOTABLE_ARROW_USE_MEMORY_MAPPING=0`.
+- Use `chunk_size` as the first memory tuning knob. Smaller chunks lower peak
+  memory and larger chunks reduce intermediate-file overhead. Start with
+  `chunk_size=1000` for validation, then tune upward only after Slurm `MaxRSS`
+  is comfortably below the memory request.
+- Do not try to tune DuckDB `memory_limit` first. Cytotable exposes thread
+  control through `CYTOTABLE_MAX_THREADS`, but not a clean per-run DuckDB memory
+  limit hook. Control memory with Slurm `--mem`, `chunk_size`, and one
+  conversion per Python process.
+- Keep `sort_output=True` when reproducibility or output comparison matters.
+  Test `sort_output=False` only when downstream consumers do not require stable
+  sorted output; it can avoid some `ORDER BY` work.
+
+## Runtime Setup
+
+Use a project-owned `uv` environment:
+
+```bash
+export UV_HOME="/projects/$USER/software/uv"
+export UV_INSTALL_DIR="$UV_HOME/bin"
+export UV_ENVS="$UV_HOME/envs"
+export UV_CACHE_DIR="/scratch/alpine/$USER/uv-cache"
+export UV_LINK_MODE=copy
+export PATH="$UV_INSTALL_DIR:$PATH"
+mkdir -p "$UV_INSTALL_DIR" "$UV_ENVS" "$UV_CACHE_DIR"
+
+if [[ ! -x "$UV_INSTALL_DIR/uv" ]]; then
+  curl -LsSf https://astral.sh/uv/install.sh | sh
+fi
+
+uv venv "$UV_ENVS/cytotable-alpine" --python 3.12
+source "$UV_ENVS/cytotable-alpine/bin/activate"
+uv pip install cytotable
+```
+
+If installing from a GitHub source archive rather than PyPI or a real git
+checkout, Cytotable's `setuptools-scm` build needs an explicit pretend version:
+
+```bash
+export SETUPTOOLS_SCM_PRETEND_VERSION_FOR_CYTOTABLE=0.0.0+alpinebench
+uv pip install /scratch/alpine/$USER/cytotable-alpine-bench-20260821/CytoTable-main
+```
+
+## Parsl Config Pattern
+
+Thread pool baseline:
+
+```python
+from parsl.config import Config
+from parsl.executors import ThreadPoolExecutor
+
+parsl_config = Config(
+    executors=[ThreadPoolExecutor(label="cytotable_tpe", max_threads=cpus)],
+    run_dir=f"{scratch_root}/parsl-runs/{run_name}",
+)
+```
+
+HTEX comparison candidates:
+
+```python
+from parsl.config import Config
+from parsl.executors import HighThroughputExecutor
+
+default_htex = Config(
+    executors=[HighThroughputExecutor(label="cytotable_htex")],
+    run_dir=f"{scratch_root}/parsl-runs/{run_name}-htex-default",
+)
+
+capped_htex = Config(
+    executors=[
+        HighThroughputExecutor(
+            label="cytotable_htex",
+            max_workers_per_node=cpus,
+        )
+    ],
+    run_dir=f"{scratch_root}/parsl-runs/{run_name}-htex-capped",
+)
+```
+
+Call `convert(...)` with the config:
+
+```python
+from cytotable import convert
+
+convert(
+    source_path=source_path,
+    dest_path=dest_path,
+    dest_datatype="parquet",
+    source_datatype="csv",
+    preset="cellprofiler_csv",
+    chunk_size=1000,
+    parsl_config=parsl_config,
+)
+```
+
+## Alpine Benchmark State
+
+On `2026-08-21`, a benchmark harness was created under:
+
+```text
+/scratch/alpine/$USER/cytotable-alpine-bench-20260821
+```
+
+The source fixture was Cytotable's current `main` archive, using:
+
+```text
+tests/data/cellprofiler/cpg0043-segmentation
+```
+
+Fixture size observed on Alpine:
+
+- `48` CSV files
+- `256020900` CSV bytes
+
+Environment installed successfully:
+
+- Python `3.12.13` environment requested via `uv`
+- Cytotable source archive installed as `0.0.0+alpinebench`
+- Parsl `2026.8.10`
+- DuckDB `1.5.5`
+- PyArrow `25.0.1`
+
+The first matrix submitted and failed because the harness called
+`convert.convert(...)` after `from cytotable import convert`. This was a harness
+bug, not a Cytotable or Slurm finding. Failed job IDs:
+
+```text
+31536860-31536868
+```
+
+The corrected matrix submitted as:
+
+```text
+31536906 rerun-threadpool-1c
+31536907 rerun-threadpool-4c
+31536908 rerun-threadpool-8c
+31536909 rerun-htex-default-1c
+31536910 rerun-htex-default-4c
+31536911 rerun-htex-default-8c
+31536912 rerun-htex-capped-1c
+31536913 rerun-htex-capped-4c
+31536914 rerun-htex-capped-8c
+```
+
+Those jobs left the queue within a few minutes, but their result files could
+not be collected until Alpine home space was freed. After collection, all nine
+corrected jobs were found to have failed before conversion because DuckDB tried
+to install extensions under `$HOME/.duckdb` while home was full:
+
+```text
+IO Error: Cannot open file "/home/.../.duckdb/extensions/v1.5.5/linux_amd64/..."
+No space left on device
+```
+
+This is an Alpine/Cytotable runtime finding, not an executor-performance
+finding: all executor variants failed in about `5-6s`, before the conversion
+work began.
+
+Validated scratch-home follow-up runs:
+
+```text
+31541899 homefix-threadpool-4c: completed, 43.83s Cytotable elapsed,
+         397 rows x 6049 columns, 22.85 MB parquet, Slurm elapsed 1:24,
+         Slurm batch MaxRSS 5966192K
+31541900 homefix-htex-capped-4c: completed, 49.35s Cytotable elapsed,
+         397 rows x 6049 columns, 22.85 MB parquet, Slurm elapsed 1:30,
+         Slurm batch MaxRSS 4110988K
+```
+
+Interpretation: the scratch-home pattern fixes the DuckDB extension-cache
+failure. Threadpool is the current first-choice baseline for speed at `4` CPUs;
+capped HTEX remains worth testing for memory-sensitive larger runs. Treat the
+memory comparison as one-run evidence, not a universal rule.
+
+## Failure Modes
+
+- Alpine home at `100%` full can prevent even simple SSH commands from running;
+  commands were observed failing with `sed: couldn't open temporary file
+  /home/.../.ssh/sed...: No space left on device`.
+- Avoid writing Cytotable data, archives, virtual environments, caches, or
+  Parsl run directories under home. Use scratch/project paths as above.
+- Do not let DuckDB use `$HOME/.duckdb` on Alpine. Either run Cytotable with
+  scratch-backed `HOME` as above, or use a code-level DuckDB
+  `extension_directory` configuration if Cytotable exposes one in the future.
+- Do not use an existing Cytotable `dest_path`; Cytotable intentionally raises
+  instead of overwriting.
+- Set `parsl_config` on the first Cytotable call in a Python process. Cytotable
+  reuses an already-loaded Parsl DFK and will not switch configs mid-process.

.agents/skills/isilon.md → .agents/skills/isilon/SKILL.md

@@ -1,11 +1,16 @@
+---
+name: isilon
+description: Use before mounting, testing, transferring, or documenting CU Anschutz DBMI Isilon storage access, especially SMB mounts, local transfer behavior, data-handling rules, or Isilon-to-PetaLibrary movement.
+---
+
 # Isilon Skill
 
 Use this note before mounting, testing, or transferring data to/from CU
 Anschutz's DBMI Isilon storage from a local machine. See
-`.agents/skills/petalibrary.md` for the PetaLibrary side of an
-Isilon-to-PetaLibrary transfer, and `.agents/skills/alpine.md` for Alpine/HPC
-knowledge — this file covers Isilon specifically. For measured performance
-numbers, see `docs/storage-mount-benchmark.md` and the reproducible script at
+`.agents/skills/petalibrary/SKILL.md` for the PetaLibrary side of an
+Isilon-to-PetaLibrary transfer, and `.agents/skills/alpine/SKILL.md` for
+Alpine/HPC knowledge — this file covers Isilon specifically. For measured
+performance numbers, see `docs/storage-mount-benchmark.md` and the reproducible script at
 `examples/benchmark_storage_mount.sh` — this skill intentionally does not
 carry exact benchmark figures itself, since they're time/location/session-
 specific and go stale fast; re-run the script rather than trusting a number

.agents/skills/petalibrary.md → .agents/skills/petalibrary/SKILL.md

@@ -1,10 +1,15 @@
+---
+name: petalibrary
+description: Use before mounting, transferring to or from, or reasoning about CU Boulder/Anschutz PetaLibrary storage from local machines or Alpine, including sshfs, Globus, rclone, quotas, access paths, and data-handling rules.
+---
+
 # PetaLibrary Skill
 
 Use this note before mounting, transferring to/from, or reasoning about quotas
 and access methods for CU Boulder/Anschutz's PetaLibrary storage service, as
-used from a local machine or from Alpine. See `.agents/skills/alpine.md` for
-general Alpine/SSH/Slurm knowledge — this file covers PetaLibrary specifically
-and cross-references that file rather than repeating it. For measured
+used from a local machine or from Alpine. See
+`.agents/skills/alpine/SKILL.md` for general Alpine/SSH/Slurm knowledge — this
+file covers PetaLibrary specifically and cross-references that file rather than repeating it. For measured
 performance numbers, see `docs/storage-mount-benchmark.md` and the
 reproducible script at `examples/benchmark_storage_mount.sh` — this skill
 intentionally does not carry exact benchmark figures itself, since they're
@@ -13,7 +18,7 @@ than trusting a number recorded here. It does carry a rough, order-of-
 magnitude expectation for off-campus-VPN vs. on-campus speed below, since
 that's useful for planning a transfer even without a fresh benchmark run.
 
-**Data handling rule, same as `.agents/skills/isilon.md`: never record real
+**Data handling rule, same as `.agents/skills/isilon/SKILL.md`: never record real
 allocation/directory names, file names, or directory listings from inside
 `/pl/active` (or any other PetaLibrary path) in this skill, in commit
 messages, or anywhere else committed to the repo.** Allocation subdirectory

AGENTS.md

@@ -1,5 +1,5 @@
 # Agent Notes
 
-Use `.agents/skills/alpine.md` before working on Alpine validation or Slurm
-submission behavior. It records the connection and module-loading details already
-validated for this project.
+Use `.agents/skills/alpine/SKILL.md` before working on Alpine validation or
+Slurm submission behavior. It records the connection and module-loading details
+already validated for this project.

bin/formascute

@@ -108,7 +108,7 @@ check_repo() {
     "CODE_OF_CONDUCT.md"
     "CONTRIBUTING.md"
     "AGENTS.md"
-    ".agents/skills/alpine.md"
+    ".agents/skills/alpine/SKILL.md"
     ".markdownlint-cli2.yaml"
     "nextflow.config"
     "conf/alpine.reference.config"

docs/alpine-findings.md

@@ -401,7 +401,7 @@ residual risk, but no longer the only one.
 ### CURC answered directly (2026-08-07): mostly good news, one new seasonal risk
 
 CURC's contact (Gregory Way) responded to the questions above. Full detail in
-`.agents/skills/alpine.md` under "CURC's Direct Answer"; summary:
+`.agents/skills/alpine/SKILL.md` under "CURC's Direct Answer"; summary:
 
 - **Institution fairshare confirmed and quantified**: AMC holds `6,459` of
   Alpine's priority shares (`~1,085,112` SU/week sustainable budget