diff --git a/.agents/skills/alpine.md b/.agents/skills/alpine/SKILL.md similarity index 97% rename from .agents/skills/alpine.md rename to .agents/skills/alpine/SKILL.md index 1de8f17..45d94f6 100644 --- a/.agents/skills/alpine.md +++ b/.agents/skills/alpine/SKILL.md @@ -1,3 +1,8 @@ +--- +name: alpine +description: Use before changing, testing, or scaling formascute on CURC Alpine, especially for Nextflow, Slurm submission, Persistence1, Python runtimes, Apptainer/Singularity, uv environments, queue limits, or Alpine filesystem behavior. +--- + # Alpine Skill Use this note before changing, testing, or scaling formascute on CURC Alpine, @@ -7,7 +12,11 @@ environment behind the project's imaging/feature-extraction workload. ## Current Position - Submit and orchestrate through `Persistence1`; treat this as selected project - policy, not an open question. + policy for Nextflow and other long-lived workflow managers, not an open + question for that workflow shape. +- For non-Nextflow, single-node Python tools, a plain Slurm batch job submitted + from the Alpine login node is acceptable. Do not introduce `Persistence1` + just to run one compute-node Python process. - Keep compute work on Slurm. `Persistence1` is for the long-lived workflow manager, not image processing or other heavy work. - Use `queueSize = 200` as the production default. It matches a real, hard @@ -169,10 +178,11 @@ Nextflow-orchestrated container tasks. ## Slurm Defaults -*Scope: `account`/`partition`/`QoS`/`submit host` are general Slurm settings — -they apply to any job on Alpine, including a plain `sbatch` script with no -Nextflow involved. `executor`, `queueSize`, and the submit throttle are -Nextflow executor settings and only mean something when using Nextflow.* +*Scope: `account`/`partition`/`QoS` are general Slurm settings — they apply to +any job on Alpine, including a plain `sbatch` script with no Nextflow involved. +`submit host`, `executor`, `queueSize`, and the submit throttle are Nextflow +or workflow-manager settings and only mean something for that orchestration +shape.* Use these defaults for CPU work unless the user or CURC gives a newer allocation policy: @@ -180,7 +190,7 @@ policy: - account: project allocation supplied by the user - partition: `acpu` - QoS: `cpu-normal` -- submit host: `Persistence1` +- submit host: `Persistence1` *(Nextflow / workflow-manager orchestration only)* - executor: Slurm *(Nextflow only)* - production `queueSize`: `200` *(Nextflow only)* - production submit throttle: do not enable by default; see Queue And @@ -891,6 +901,11 @@ blockers — there are no known blockers on the Apptainer side beyond the ### General (Alpine / Apptainer, Not Nextflow-Specific) +- Do not let `$HOME` fill up. Alpine SSH/session setup can fail before a remote + command runs when home is `100%` full; observed failure text included + `sed: couldn't open temporary file /home/.../.ssh/sed...: No space left on + device`. Keep archives, virtual environments, caches, benchmark outputs, and + Parsl/Nextflow work directories on scratch or project storage, not home. - Do not rely on the base `Persistence1` Python environment for real production work — it lacks the project's required Python version and scientific dependencies. diff --git a/.agents/skills/cytotable/SKILL.md b/.agents/skills/cytotable/SKILL.md new file mode 100644 index 0000000..c0ee4da --- /dev/null +++ b/.agents/skills/cytotable/SKILL.md @@ -0,0 +1,254 @@ +--- +name: cytotable +description: Use before running, tuning, or documenting CytoTable conversions on CURC Alpine, especially for Parsl executor choice, Slurm CPU and memory shape, DuckDB/Arrow settings, S3-backed large tests, scratch-backed HOME, or uv runtime setup. +--- + +# Cytotable Skill + +Use this note before running, tuning, or documenting CytoTable conversions on +CURC Alpine, especially when choosing a Parsl executor, Slurm CPU shape, Python +runtime, or scratch/project storage layout. + +Also read `.agents/skills/alpine/SKILL.md` for Alpine-wide Slurm, storage, and +SSH facts. This skill is Cytotable-specific; keep general Alpine policy in the +Alpine skill instead. + +## Current Position + +- Run Cytotable as a regular Slurm batch job on compute nodes. Do not route it + through `Persistence1` unless another orchestrator actually needs that host. +- Put source data, destination parquet, Parsl run directories, and Cytotable + benchmark logs under `/scratch/alpine/$USER`. +- Put reusable Python environments under `/projects/$USER/software/uv/envs` and + keep the `uv` cache on scratch with `UV_LINK_MODE=copy`. +- Run the Cytotable Python process with a scratch-backed `HOME` so DuckDB + extension installs do not touch Alpine's small home quota. Cytotable loads + DuckDB extensions (`sqlite_scanner`, `httpfs`) via DuckDB's default + extension cache under `$HOME/.duckdb`; a full home directory causes + immediate conversion failure before executor differences matter. +- Set `CYTOTABLE_MAX_THREADS` before Python starts. Cytotable reads it at import + time and passes it into DuckDB thread configuration. +- Start with one Cytotable conversion per Slurm job, using one node, explicit + `--time`, explicit `--mem`, and `--cpus-per-task` matched to + `CYTOTABLE_MAX_THREADS`. +- Use `ThreadPoolExecutor(max_threads=)` as the first candidate for + local Alpine conversion jobs. In one validated `4` CPU comparison on the + bundled CSV fixture, threadpool was slightly faster than capped HTEX. +- Keep capped `HighThroughputExecutor(max_workers_per_node=)` as the + comparison candidate when memory is the concern. In the same single + comparison, capped HTEX used less Slurm-reported peak RSS but ran slightly + slower. Do not use default uncapped HTEX for production guidance until it has + passed the same scratch-home benchmark. +- A practical first production-shaped request is `--cpus-per-task=4`, + `--mem=8G`, `--time=00:20:00` for the bundled test-size CSV conversion, then + scale from measured walltime and RSS. Re-measure for larger real datasets. + +## DuckDB And Arrow Settings + +- Set `CYTOTABLE_MAX_THREADS="$SLURM_CPUS_PER_TASK"` in the batch job before + Python starts. Cytotable reads this at import time and uses it for DuckDB + `PRAGMA threads`; setting it later in Python is too late. +- Set `HOME` to a scratch job-home after activating the reusable environment + and before starting Python, for example: + + ```bash + export CYTOTABLE_JOB_HOME="/scratch/alpine/$USER/cytotable-job-home/$SLURM_JOB_ID" + mkdir -p "$CYTOTABLE_JOB_HOME" + export HOME="$CYTOTABLE_JOB_HOME" + ``` + + This redirects DuckDB's default extension cache (`$HOME/.duckdb`) to scratch. + A validated run created about `54 MB` per scratch job-home. +- Leave Cytotable's default Arrow memory mapping enabled for initial runs: + `CYTOTABLE_ARROW_USE_MEMORY_MAPPING=1`. This is the default and should be a + reasonable fit for scratch-backed parquet reads. +- If a run shows unexpectedly high `MaxVMSize`, file-handle/unlink behavior, or + memory pressure during concat/join, run a controlled comparison with + `CYTOTABLE_ARROW_USE_MEMORY_MAPPING=0`. +- Use `chunk_size` as the first memory tuning knob. Smaller chunks lower peak + memory and larger chunks reduce intermediate-file overhead. Start with + `chunk_size=1000` for validation, then tune upward only after Slurm `MaxRSS` + is comfortably below the memory request. +- Do not try to tune DuckDB `memory_limit` first. Cytotable exposes thread + control through `CYTOTABLE_MAX_THREADS`, but not a clean per-run DuckDB memory + limit hook. Control memory with Slurm `--mem`, `chunk_size`, and one + conversion per Python process. +- Keep `sort_output=True` when reproducibility or output comparison matters. + Test `sort_output=False` only when downstream consumers do not require stable + sorted output; it can avoid some `ORDER BY` work. + +## Runtime Setup + +Use a project-owned `uv` environment: + +```bash +export UV_HOME="/projects/$USER/software/uv" +export UV_INSTALL_DIR="$UV_HOME/bin" +export UV_ENVS="$UV_HOME/envs" +export UV_CACHE_DIR="/scratch/alpine/$USER/uv-cache" +export UV_LINK_MODE=copy +export PATH="$UV_INSTALL_DIR:$PATH" +mkdir -p "$UV_INSTALL_DIR" "$UV_ENVS" "$UV_CACHE_DIR" + +if [[ ! -x "$UV_INSTALL_DIR/uv" ]]; then + curl -LsSf https://astral.sh/uv/install.sh | sh +fi + +uv venv "$UV_ENVS/cytotable-alpine" --python 3.12 +source "$UV_ENVS/cytotable-alpine/bin/activate" +uv pip install cytotable +``` + +If installing from a GitHub source archive rather than PyPI or a real git +checkout, Cytotable's `setuptools-scm` build needs an explicit pretend version: + +```bash +export SETUPTOOLS_SCM_PRETEND_VERSION_FOR_CYTOTABLE=0.0.0+alpinebench +uv pip install /scratch/alpine/$USER/cytotable-alpine-bench-20260821/CytoTable-main +``` + +## Parsl Config Pattern + +Thread pool baseline: + +```python +from parsl.config import Config +from parsl.executors import ThreadPoolExecutor + +parsl_config = Config( + executors=[ThreadPoolExecutor(label="cytotable_tpe", max_threads=cpus)], + run_dir=f"{scratch_root}/parsl-runs/{run_name}", +) +``` + +HTEX comparison candidates: + +```python +from parsl.config import Config +from parsl.executors import HighThroughputExecutor + +default_htex = Config( + executors=[HighThroughputExecutor(label="cytotable_htex")], + run_dir=f"{scratch_root}/parsl-runs/{run_name}-htex-default", +) + +capped_htex = Config( + executors=[ + HighThroughputExecutor( + label="cytotable_htex", + max_workers_per_node=cpus, + ) + ], + run_dir=f"{scratch_root}/parsl-runs/{run_name}-htex-capped", +) +``` + +Call `convert(...)` with the config: + +```python +from cytotable import convert + +convert( + source_path=source_path, + dest_path=dest_path, + dest_datatype="parquet", + source_datatype="csv", + preset="cellprofiler_csv", + chunk_size=1000, + parsl_config=parsl_config, +) +``` + +## Alpine Benchmark State + +On `2026-08-21`, a benchmark harness was created under: + +```text +/scratch/alpine/$USER/cytotable-alpine-bench-20260821 +``` + +The source fixture was Cytotable's current `main` archive, using: + +```text +tests/data/cellprofiler/cpg0043-segmentation +``` + +Fixture size observed on Alpine: + +- `48` CSV files +- `256020900` CSV bytes + +Environment installed successfully: + +- Python `3.12.13` environment requested via `uv` +- Cytotable source archive installed as `0.0.0+alpinebench` +- Parsl `2026.8.10` +- DuckDB `1.5.5` +- PyArrow `25.0.1` + +The first matrix submitted and failed because the harness called +`convert.convert(...)` after `from cytotable import convert`. This was a harness +bug, not a Cytotable or Slurm finding. Failed job IDs: + +```text +31536860-31536868 +``` + +The corrected matrix submitted as: + +```text +31536906 rerun-threadpool-1c +31536907 rerun-threadpool-4c +31536908 rerun-threadpool-8c +31536909 rerun-htex-default-1c +31536910 rerun-htex-default-4c +31536911 rerun-htex-default-8c +31536912 rerun-htex-capped-1c +31536913 rerun-htex-capped-4c +31536914 rerun-htex-capped-8c +``` + +Those jobs left the queue within a few minutes, but their result files could +not be collected until Alpine home space was freed. After collection, all nine +corrected jobs were found to have failed before conversion because DuckDB tried +to install extensions under `$HOME/.duckdb` while home was full: + +```text +IO Error: Cannot open file "/home/.../.duckdb/extensions/v1.5.5/linux_amd64/..." +No space left on device +``` + +This is an Alpine/Cytotable runtime finding, not an executor-performance +finding: all executor variants failed in about `5-6s`, before the conversion +work began. + +Validated scratch-home follow-up runs: + +```text +31541899 homefix-threadpool-4c: completed, 43.83s Cytotable elapsed, + 397 rows x 6049 columns, 22.85 MB parquet, Slurm elapsed 1:24, + Slurm batch MaxRSS 5966192K +31541900 homefix-htex-capped-4c: completed, 49.35s Cytotable elapsed, + 397 rows x 6049 columns, 22.85 MB parquet, Slurm elapsed 1:30, + Slurm batch MaxRSS 4110988K +``` + +Interpretation: the scratch-home pattern fixes the DuckDB extension-cache +failure. Threadpool is the current first-choice baseline for speed at `4` CPUs; +capped HTEX remains worth testing for memory-sensitive larger runs. Treat the +memory comparison as one-run evidence, not a universal rule. + +## Failure Modes + +- Alpine home at `100%` full can prevent even simple SSH commands from running; + commands were observed failing with `sed: couldn't open temporary file + /home/.../.ssh/sed...: No space left on device`. +- Avoid writing Cytotable data, archives, virtual environments, caches, or + Parsl run directories under home. Use scratch/project paths as above. +- Do not let DuckDB use `$HOME/.duckdb` on Alpine. Either run Cytotable with + scratch-backed `HOME` as above, or use a code-level DuckDB + `extension_directory` configuration if Cytotable exposes one in the future. +- Do not use an existing Cytotable `dest_path`; Cytotable intentionally raises + instead of overwriting. +- Set `parsl_config` on the first Cytotable call in a Python process. Cytotable + reuses an already-loaded Parsl DFK and will not switch configs mid-process. diff --git a/.agents/skills/isilon.md b/.agents/skills/isilon/SKILL.md similarity index 96% rename from .agents/skills/isilon.md rename to .agents/skills/isilon/SKILL.md index 137c519..cc0d0cd 100644 --- a/.agents/skills/isilon.md +++ b/.agents/skills/isilon/SKILL.md @@ -1,11 +1,16 @@ +--- +name: isilon +description: Use before mounting, testing, transferring, or documenting CU Anschutz DBMI Isilon storage access, especially SMB mounts, local transfer behavior, data-handling rules, or Isilon-to-PetaLibrary movement. +--- + # Isilon Skill Use this note before mounting, testing, or transferring data to/from CU Anschutz's DBMI Isilon storage from a local machine. See -`.agents/skills/petalibrary.md` for the PetaLibrary side of an -Isilon-to-PetaLibrary transfer, and `.agents/skills/alpine.md` for Alpine/HPC -knowledge — this file covers Isilon specifically. For measured performance -numbers, see `docs/storage-mount-benchmark.md` and the reproducible script at +`.agents/skills/petalibrary/SKILL.md` for the PetaLibrary side of an +Isilon-to-PetaLibrary transfer, and `.agents/skills/alpine/SKILL.md` for +Alpine/HPC knowledge — this file covers Isilon specifically. For measured +performance numbers, see `docs/storage-mount-benchmark.md` and the reproducible script at `examples/benchmark_storage_mount.sh` — this skill intentionally does not carry exact benchmark figures itself, since they're time/location/session- specific and go stale fast; re-run the script rather than trusting a number diff --git a/.agents/skills/petalibrary.md b/.agents/skills/petalibrary/SKILL.md similarity index 97% rename from .agents/skills/petalibrary.md rename to .agents/skills/petalibrary/SKILL.md index a4f83aa..4517376 100644 --- a/.agents/skills/petalibrary.md +++ b/.agents/skills/petalibrary/SKILL.md @@ -1,10 +1,15 @@ +--- +name: petalibrary +description: Use before mounting, transferring to or from, or reasoning about CU Boulder/Anschutz PetaLibrary storage from local machines or Alpine, including sshfs, Globus, rclone, quotas, access paths, and data-handling rules. +--- + # PetaLibrary Skill Use this note before mounting, transferring to/from, or reasoning about quotas and access methods for CU Boulder/Anschutz's PetaLibrary storage service, as -used from a local machine or from Alpine. See `.agents/skills/alpine.md` for -general Alpine/SSH/Slurm knowledge — this file covers PetaLibrary specifically -and cross-references that file rather than repeating it. For measured +used from a local machine or from Alpine. See +`.agents/skills/alpine/SKILL.md` for general Alpine/SSH/Slurm knowledge — this +file covers PetaLibrary specifically and cross-references that file rather than repeating it. For measured performance numbers, see `docs/storage-mount-benchmark.md` and the reproducible script at `examples/benchmark_storage_mount.sh` — this skill intentionally does not carry exact benchmark figures itself, since they're @@ -13,7 +18,7 @@ than trusting a number recorded here. It does carry a rough, order-of- magnitude expectation for off-campus-VPN vs. on-campus speed below, since that's useful for planning a transfer even without a fresh benchmark run. -**Data handling rule, same as `.agents/skills/isilon.md`: never record real +**Data handling rule, same as `.agents/skills/isilon/SKILL.md`: never record real allocation/directory names, file names, or directory listings from inside `/pl/active` (or any other PetaLibrary path) in this skill, in commit messages, or anywhere else committed to the repo.** Allocation subdirectory diff --git a/AGENTS.md b/AGENTS.md index c5b4921..a8e2e36 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,5 +1,5 @@ # Agent Notes -Use `.agents/skills/alpine.md` before working on Alpine validation or Slurm -submission behavior. It records the connection and module-loading details already -validated for this project. +Use `.agents/skills/alpine/SKILL.md` before working on Alpine validation or +Slurm submission behavior. It records the connection and module-loading details +already validated for this project. diff --git a/bin/formascute b/bin/formascute index 739d69c..9afe865 100755 --- a/bin/formascute +++ b/bin/formascute @@ -108,7 +108,7 @@ check_repo() { "CODE_OF_CONDUCT.md" "CONTRIBUTING.md" "AGENTS.md" - ".agents/skills/alpine.md" + ".agents/skills/alpine/SKILL.md" ".markdownlint-cli2.yaml" "nextflow.config" "conf/alpine.reference.config" diff --git a/docs/alpine-findings.md b/docs/alpine-findings.md index 67d1d7f..f96925a 100644 --- a/docs/alpine-findings.md +++ b/docs/alpine-findings.md @@ -401,7 +401,7 @@ residual risk, but no longer the only one. ### CURC answered directly (2026-08-07): mostly good news, one new seasonal risk CURC's contact (Gregory Way) responded to the questions above. Full detail in -`.agents/skills/alpine.md` under "CURC's Direct Answer"; summary: +`.agents/skills/alpine/SKILL.md` under "CURC's Direct Answer"; summary: - **Institution fairshare confirmed and quantified**: AMC holds `6,459` of Alpine's priority shares (`~1,085,112` SU/week sustainable budget diff --git a/docs/cytotable-alpine-findings.md b/docs/cytotable-alpine-findings.md new file mode 100644 index 0000000..43d0ded --- /dev/null +++ b/docs/cytotable-alpine-findings.md @@ -0,0 +1,173 @@ +# Cytotable Alpine Findings + +These notes record the `2026-08-21` CytoTable-on-Alpine experiment setup and +what was learned before Alpine account-home exhaustion blocked result +collection. + +## Goal + +Compare Cytotable Parsl configuration on Alpine compute nodes: + +- `ThreadPoolExecutor` +- Cytotable-style default `HighThroughputExecutor` +- capped `HighThroughputExecutor` +- Slurm CPU requests of `1`, `4`, and `8` CPUs + +The run intentionally avoided `Persistence1`; each Cytotable conversion was a +plain Slurm batch job submitted from the Alpine login node. + +## Dataset + +The benchmark used Cytotable's current `main` source archive and bundled test +fixture: + +```text +tests/data/cellprofiler/cpg0043-segmentation +``` + +Observed on Alpine: + +- `48` CSV files +- `256020900` CSV bytes + +This fixture is large enough to exercise CSV parsing, chunking, joining, and +Parquet writing, while still being appropriate for short `acpu` jobs. + +## Runtime + +Scratch root: + +```text +/scratch/alpine/$USER/cytotable-alpine-bench-20260821 +``` + +Environment: + +- `uv` environment under `/projects/$USER/software/uv/envs/cytotable-alpine` +- `UV_CACHE_DIR=/scratch/alpine/$USER/uv-cache` +- `UV_LINK_MODE=copy` +- Cytotable source archive installed with + `SETUPTOOLS_SCM_PRETEND_VERSION_FOR_CYTOTABLE=0.0.0+alpinebench` +- Parsl `2026.8.10` +- DuckDB `1.5.5` +- PyArrow `25.0.1` + +GitHub source archives do not include `.git` metadata, so Cytotable's +`setuptools-scm` build fails unless a pretend version is supplied or the package +is installed from PyPI or a real git checkout. + +## Experiment Matrix + +First submitted job IDs: + +| job IDs | status | finding | +| --- | --- | --- | +| `31536860-31536868` | failed | harness called `convert.convert(...)` after importing `convert` as a function | + +The failed jobs still confirmed submission mechanics and showed one Slurm +accounting quirk: nominal `--cpus-per-task=1` jobs appeared in `sacct` with +`AllocCPUS=3` on this `acpu` run, while `4` and `8` CPU requests reported as +requested. + +Corrected submitted job IDs: + +| job ID | run | +| ---: | --- | +| `31536906` | `rerun-threadpool-1c` | +| `31536907` | `rerun-threadpool-4c` | +| `31536908` | `rerun-threadpool-8c` | +| `31536909` | `rerun-htex-default-1c` | +| `31536910` | `rerun-htex-default-4c` | +| `31536911` | `rerun-htex-default-8c` | +| `31536912` | `rerun-htex-capped-1c` | +| `31536913` | `rerun-htex-capped-4c` | +| `31536914` | `rerun-htex-capped-8c` | + +The corrected jobs left the queue within a few minutes. Their JSON result files +and Slurm accounting could not be collected until the account home directory was +freed enough for SSH to work again. + +After collection, all nine corrected jobs had failed before conversion. The +common failure was DuckDB attempting to install/load extensions under +`$HOME/.duckdb` while Alpine home was full: + +```text +IO Error: Cannot open file "/home/.../.duckdb/extensions/v1.5.5/linux_amd64/..." +No space left on device +``` + +All executor variants failed in about `5-6s`, so those runs do not compare +ThreadPoolExecutor and HTEX performance. + +## Scratch-Home Fix + +Two focused reruns set `HOME` to a scratch-backed per-job directory after +activating the reusable `uv` environment and before starting Python. This +redirected DuckDB's extension cache away from Alpine home. + +Job IDs: + +| job ID | run | status | +| ---: | --- | --- | +| `31541899` | `homefix-threadpool-4c` | completed | +| `31541900` | `homefix-htex-capped-4c` | completed | + +Results: + +| run | Cytotable elapsed | Slurm elapsed | Slurm batch MaxRSS | rows | columns | parquet size | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | +| `homefix-threadpool-4c` | `43.83s` | `1:24` | `5966192K` | `397` | `6049` | `22.85 MB` | +| `homefix-htex-capped-4c` | `49.35s` | `1:30` | `4110988K` | `397` | `6049` | `22.85 MB` | + +The scratch job-home directories were about `54 MB` each, dominated by DuckDB +extension cache state. + +Interpretation: + +- Scratch-backed `HOME` is a required Alpine best practice for Cytotable unless + Cytotable exposes a DuckDB `extension_directory` hook in the future. +- Threadpool is the current first-choice `4` CPU baseline for speed on this + fixture. +- Capped HTEX is still worth testing for larger memory-sensitive runs because + it used less Slurm-reported peak RSS in this one comparison. +- Default uncapped HTEX should not be used for production guidance until it + passes a scratch-home benchmark. + +## Current Guidance + +Until the corrected matrix is collected, use this as provisional guidance: + +- Use one Cytotable conversion per Slurm job on `acpu`. +- Submit directly from the Alpine login node; do not use `Persistence1` for + this non-Nextflow shape. +- Run the Cytotable process with a scratch-backed `HOME` so DuckDB extension + installs use scratch rather than `$HOME/.duckdb`. +- Start with `ThreadPoolExecutor(max_threads=)`, `--cpus-per-task=4`, + `--mem=8G`, and `CYTOTABLE_MAX_THREADS=`. +- Keep all Cytotable inputs, outputs, Parsl `run_dir`, and logs on scratch. +- Keep reusable `uv` environments under project storage and the `uv` cache on + scratch. +- Use capped HTEX as a comparison candidate for memory-sensitive larger runs, + but do not treat default uncapped HTEX as validated. + +## Collection Commands + +After freeing Alpine home space: + +```bash +ROOT=/scratch/alpine/$USER/cytotable-alpine-bench-20260821 +cd "$ROOT" + +for f in results/rerun-*.json; do + printf 'FILE %s\n' "$f" + cat "$f" +done + +sacct -j 31536906,31536907,31536908,31536909,31536910,31536911,31536912,31536913,31536914 \ + --format=JobIDRaw,JobName%26,State,ExitCode,Elapsed,AllocCPUS,ReqCPUS,ReqMem,MaxRSS,MaxVMSize,NNodes,NodeList -P + +for f in logs/rerun-*.err; do + printf 'ERR %s\n' "$f" + grep -E 'Elapsed|Maximum resident|Percent of CPU|Exit status|Command exited' "$f" || true +done +``` diff --git a/docs/storage-mount-benchmark.md b/docs/storage-mount-benchmark.md index 1407bd7..61a1296 100644 --- a/docs/storage-mount-benchmark.md +++ b/docs/storage-mount-benchmark.md @@ -2,9 +2,9 @@ A small, reproducible experiment comparing basic operation timing on two network-mounted storage systems used by this project: DBMI Isilon (SMB) and -CURC PetaLibrary (sshfs). See `.agents/skills/isilon.md` and -`.agents/skills/petalibrary.md` for what these systems are and how to mount -them — this doc is just the benchmark methodology and results, kept separate +CURC PetaLibrary (sshfs). See `.agents/skills/isilon/SKILL.md` and +`.agents/skills/petalibrary/SKILL.md` for what these systems are and how to +mount them — this doc is just the benchmark methodology and results, kept separate so the skills don't carry numbers that go stale. **Data handling:** results below are aggregate timings only. No real share