From 7784aa02671c9c36afe8b5d8f2abe675042c3341 Mon Sep 17 00:00:00 2001 From: d33bs Date: Mon, 24 Aug 2026 17:12:47 -0600 Subject: [PATCH 1/2] updated guidance after experiment runs --- .agents/skills/cytotable/SKILL.md | 566 ++++++++++++++++++++++-------- docs/cytotable-alpine-findings.md | 350 +++++++++++++++--- 2 files changed, 724 insertions(+), 192 deletions(-) diff --git a/.agents/skills/cytotable/SKILL.md b/.agents/skills/cytotable/SKILL.md index c0ee4da..7bfa2a2 100644 --- a/.agents/skills/cytotable/SKILL.md +++ b/.agents/skills/cytotable/SKILL.md @@ -6,19 +6,186 @@ description: Use before running, tuning, or documenting CytoTable conversions on # Cytotable Skill Use this note before running, tuning, or documenting CytoTable conversions on -CURC Alpine, especially when choosing a Parsl executor, Slurm CPU shape, Python -runtime, or scratch/project storage layout. +CURC Alpine, especially when choosing a Parsl executor, Slurm CPU/memory +shape, Python runtime, or scratch/project storage layout. Also read `.agents/skills/alpine/SKILL.md` for Alpine-wide Slurm, storage, and SSH facts. This skill is Cytotable-specific; keep general Alpine policy in the Alpine skill instead. +The guidance below is ordered by what actually determines whether a real +conversion finishes: executor choice and CPU count first (they change +wall-clock and whether the job OOMs), then memory/mmap/chunk sizing, then +runtime setup, then failure modes. All performance numbers below come from +real JUMP Cell Painting Gallery plate conversions pulled from S3, at three +confirmed source sizes (`~3.9 GB`, `~11 GB`, `~19 GB` — see Real-Scale +Benchmark), not the tiny bundled test fixture — +`docs/cytotable-alpine-findings.md` has the raw job IDs and logs. **The +"best" CPU count is not a single fixed number — it depends on source data +size**, and **the executor recommendation flipped once HTEX was given real +memory headroom instead of a memory-starved budget** (see both below); +numbers from the tiny fixture are called out explicitly where used, since +they're too small to trust for performance decisions on their own. All +memory figures below are GiB (binary, matching Slurm's own `--mem=NG` units +and `sacct`'s KiB-based `MaxRSS`), not decimal GB. + +## Performance Guidance (Start Here) + +- **Prefer capped `HighThroughputExecutor` over `ThreadPoolExecutor` when you + can afford real memory headroom — it was `~4.6x` faster and used `~20%` + less memory on the one real plate this has been tested against.** This + reverses earlier guidance in this skill, so the history matters: a capped + `HighThroughputExecutor(max_workers_per_node=4)` run OOM-killed under a + `16G`/4-cpu Slurm request that ThreadPoolExecutor completed successfully + (peaking at exactly `16.00 GiB`, itself pinned right at that request). That + looked like an executor-level limitation, so this skill recommended + ThreadPoolExecutor instead. It wasn't one — it was memory starvation. When + the same `11.06 GB` plate was re-run with capped HTEX given real headroom + (`128G`, about `3x` ThreadPoolExecutor's own peak on that plate) instead of + the tight `16G` budget, it completed in `25m48s` versus ThreadPoolExecutor's + `1h59m` on the identical plate — and used `35.01 GiB` peak versus + ThreadPoolExecutor's `43.99 GiB`. Same row/column counts, same output + bytes — a real performance difference, not a correctness one. The likely + mechanism: HTEX uses separate OS worker processes, not threads inside one + Python process, so it avoids whatever GIL- or shared-object-level + contention the `4`-vs-`8`-CPU ThreadPoolExecutor findings below suggest is + a real cost at this scale — at the price of needing more memory per worker + up front, which is exactly what starved it under `16G`. **This is one + comparison at one size (`11 GB`) and one CPU count (`4`, capped)** — not + yet retested at the `3.87 GB` or `18.82 GB` plates, at `8` workers, or + uncapped. Given the size and mechanism-consistency of the result, treat it + as the new default recommendation when `--mem` isn't tightly constrained, + and fall back to ThreadPoolExecutor specifically when memory is the binding + constraint (it needs meaningfully less to succeed, even though it's + slower). Re-validate before trusting this at a size or CPU count not yet + tested. HTEX also remains the only Parsl executor here that can fan out + across *multiple* Slurm nodes — ThreadPoolExecutor is confined to one + process on one node. +- **For `ThreadPoolExecutor`, the best CPU count depends on source data + size — there is no single right answer.** (This CPU-count comparison has + only been run under `ThreadPoolExecutor`; HTEX's own CPU-count scaling is + untested — see above.) On a `~3.9 GB` plate, `4` CPUs beat `8` by `~21%` + (`18m26s` vs `22m23s`). On a `~11 GB` plate, that inverts: `8` CPUs beat `4` + by `~20%` (`1h36m` vs `1h59m`). There is a real crossover somewhere between + those two sizes where added `ThreadPoolExecutor(max_threads=...)` + parallelism stops being dominated by overhead and starts paying off. As a + starting rule: use `4` CPUs below roughly `~5 GB` of source data; use `8` + CPUs at `~10 GB` and above if minimizing wall-clock matters most (only + tested up to `~11 GB` at `8` CPUs so far — the `~19 GB` plate was only run + at `4` CPUs). The `8`-CPU win is not free — it cost `~2.75x` more total + system-CPU-time and `~23%` more peak memory on the `11 GB` plate — so + prefer `4` CPUs instead when running many jobs concurrently or when + resource efficiency matters more than one job's wall-clock. Note also that + raw CPU% readings on Alpine's `acpu` partition are not a clean contention + signal by themselves: a large `--mem` request silently grants far more + actual CPU allocation than `--cpus-per-task` requests (`AllocCPUS = max( + --cpus-per-task, ceil(--mem_GB / 3.75))` — a `96G` request got `26` + allocated CPUs against a nominal `4`), so background library threads (not + gated by `CYTOTABLE_MAX_THREADS`) can use more cores than the "4 vs 8" + framing implies. The wall-clock comparison itself remains the reliable + signal. Do not assume a fixed CPU count transfers across data sizes; + re-benchmark at the target scale, especially outside the `~4-11 GB` range + actually tested. +- **Leave `CYTOTABLE_ARROW_USE_MEMORY_MAPPING=1` (the default) on.** It was + about `7%` faster than disabling it in the same real conversion, at the + cost of about `1 GB` more process-level peak RSS. Only disable it + (`CYTOTABLE_ARROW_USE_MEMORY_MAPPING=0`) if a run is memory-constrained + enough that trading that speed for roughly `1 GB` of headroom matters. +- **Size `--mem` at roughly `4-4.5x` the source `.sqlite` file size at `4` + CPUs, not off `chunk_size` or a flat number — and expect the multiplier to + trend down, not up, as source size grows.** Peak memory (Slurm cgroup + MaxRSS) to source-size ratio across the three confirmed plate sizes, all at + `4` CPUs: `4.14x` at `3.9 GB` (`16.00 GiB` peak), `3.98x` at `11 GB` + (`43.99 GiB` peak), `3.33x` at `19 GB` (`62.67 GiB` peak). `8` CPUs pushed + the ratio higher (`4.90x`) on the one plate tested at both CPU counts. All + three plates share the same `~5946`-column schema, so the declining ratio + looks like fixed per-run overhead (Python/Parsl/DuckDB baseline working + set) mattering more at small scale and amortizing at larger scale — not + evidence that bigger conversions need proportionally *more* headroom. All + of the memory pressure itself shows up during the final join/concat, not + during chunked reads, regardless of `chunk_size`. Use `~4.5x` source size + as a starting point for a single-plate conversion under `~10 GB`, `~4x` in + the `~10-20 GB` range, and `~5x` if running at `8` CPUs or when in doubt; + this ratio is anchored by only three real data points from one dataset + family so far, so re-measure rather than trusting it blindly outside + `~4-19 GB`. +- **Budget `--time` with a superlinear formula, not a flat number or linear + scaling — this is now confirmed across three sizes, not just two.** A + power-law fit (least-squares in log-log space) across all three + `(source_GB, elapsed_s)` points gives `time ∝ size^1.83`, and that formula + predicted the `~19 GB` plate's actual runtime within `~2%` from the + smallest plate alone. A linear guess from the small plate would have + predicted about `53` minutes for the `11 GB` plate (real: `1h59m`, `>2x` + off) and about `1h30m` for the `19 GB` plate (real: `5h42m`, nearly `4x` + off). The tiny bundled CSV fixture finishes in well under a minute and is + not a usable calibration point for real production `--time` at all. Treat + `time ≈ T_ref × (size / size_ref)^1.8` as a validated planning formula for + this dataset family, and still pad it — a `6h` request for the `19 GB` + plate left only `~17` minutes of slack against the actual `5h43m` runtime, + closer than is comfortable. +- **Budget scratch storage for the S3 staging cache in proportion to source + size, not just the parquet output.** `local_cache_dir` (the local copy of + the S3-staged source file) tracks the source `.sqlite` size roughly 1:1 + (`~3.7 GB` cache for a `3.87 GB` source), and the parquet destination has + landed close to source size across all three tested plates (output/source + byte ratio `0.98x`, `1.04x`, `1.04x` at `3.87 GB`, `11.06 GB`, and + `18.82 GB` respectively) — separate from the `~54 MB` scratch job-home + DuckDB extension cache, which does not scale with dataset size. Multiply + source-file size by roughly `2x` (cache plus output) per source file a run + touches when planning scratch quota. +- **Recognize OOM disguised as a generic Parsl error — and know the fix is + more memory, not abandoning HTEX.** An HTEX worker that gets OOM-killed + surfaces to Cytotable as `DependencyError([(WorkerLost(1, ''), 'task + N')], M)`, not an explicit memory message — check the Slurm `.err` log for + `oom_kill event` before assuming a `DependencyError` is a code or data + problem. This is exactly what happened in the original HTEX test that once + drove this skill to recommend ThreadPoolExecutor instead; the actual fix + (real headroom, not a different executor) turned out to make HTEX the + faster, leaner choice — see above. +- **A practical first production request**, sized off the source `.sqlite` + file in hand. Executor choice: prefer capped HTEX where it's been + validated (`~10-15 GB`); ThreadPoolExecutor remains the only validated + choice outside that range, or whenever `--mem` is tightly constrained + (HTEX's own memory floor hasn't been found yet — `128G` is known to work + comfortably for the `~11 GB` tier, `48-64G` might too, but that hasn't been + tested): + - **Under `~5 GB` source:** `ThreadPoolExecutor(max_threads=4)`, + `--cpus-per-task=4`, `--mem` at `~4.5x` source size (`24G` covers up to + about a `5 GB` source with headroom), `--time=00:45:00`, + `CYTOTABLE_MAX_THREADS=4`. (Capped HTEX is untested with headroom at this + size — the only real-scale HTEX test at `~4 GB` was the tight-budget OOM + run. Worth trying HTEX here too, but ThreadPoolExecutor is what's + actually validated.) + - **`~10-15 GB` source:** prefer capped `HighThroughputExecutor( + max_workers_per_node=4)`, `--cpus-per-task=4`, `--mem` at `~6x` source + size as a generous starting point (`64G` for an `~11 GB` source; the + validated run used `128G` against a `35.01 GiB` observed peak, so this + has real room to be tightened once a narrower-budget run finds the + actual floor), `--time=01:00:00` (the validated run took `25m48s` — this + is deliberately more padded than the observation warrants, since only + one run confirms the runtime). Fall back to `ThreadPoolExecutor( + max_threads=8)`, `--mem` at `~5x` source size (`64G`), `--time=03:00:00` + if `--mem` can't stretch to HTEX's headroom. + - **`~15-20 GB` source:** only validated with `ThreadPoolExecutor` at `4` + CPUs so far — `ThreadPoolExecutor(max_threads=4)`, `--cpus-per-task=4`, + `--mem` at `~4x` source size (`80G` comfortably covers a `~19 GB` source; + the actual validated run used `96G` against a `62.67 GiB` observed peak, + more headroom than necessary), `--time=06:00:00` (the validated run took + `5h42m` — do not shave this down; it left only `~17` minutes of slack at + `6h`). HTEX is untested at this size; given the `~11 GB` result, it's + worth trying with generous `--mem` headroom before defaulting to + ThreadPoolExecutor here too. + - `CYTOTABLE_ARROW_USE_MEMORY_MAPPING=1` (default) in all cases. + Re-measure against the real dataset in hand rather than assuming these + numbers transfer to a meaningfully different column count, row count, or + source size — especially outside the `~4-19 GB` range actually tested. + ## Current Position - Run Cytotable as a regular Slurm batch job on compute nodes. Do not route it through `Persistence1` unless another orchestrator actually needs that host. -- Put source data, destination parquet, Parsl run directories, and Cytotable - benchmark logs under `/scratch/alpine/$USER`. +- Put source data, destination parquet, S3 staging cache, Parsl run + directories, and Cytotable benchmark logs under `/scratch/alpine/$USER`. - Put reusable Python environments under `/projects/$USER/software/uv/envs` and keep the `uv` cache on scratch with `UV_LINK_MODE=copy`. - Run the Cytotable Python process with a scratch-backed `HOME` so DuckDB @@ -31,23 +198,164 @@ Alpine skill instead. - Start with one Cytotable conversion per Slurm job, using one node, explicit `--time`, explicit `--mem`, and `--cpus-per-task` matched to `CYTOTABLE_MAX_THREADS`. -- Use `ThreadPoolExecutor(max_threads=)` as the first candidate for - local Alpine conversion jobs. In one validated `4` CPU comparison on the - bundled CSV fixture, threadpool was slightly faster than capped HTEX. -- Keep capped `HighThroughputExecutor(max_workers_per_node=)` as the - comparison candidate when memory is the concern. In the same single - comparison, capped HTEX used less Slurm-reported peak RSS but ran slightly - slower. Do not use default uncapped HTEX for production guidance until it has - passed the same scratch-home benchmark. -- A practical first production-shaped request is `--cpus-per-task=4`, - `--mem=8G`, `--time=00:20:00` for the bundled test-size CSV conversion, then - scale from measured walltime and RSS. Re-measure for larger real datasets. +- Prefer capped `HighThroughputExecutor(max_workers_per_node=)` over + `ThreadPoolExecutor(max_threads=)` when `--mem` can afford real + headroom — on the one real plate both have been tested against at + adequate memory, capped HTEX was `~4.6x` faster and used `~20%` less peak + memory. This reverses this skill's earlier position, which was based on an + HTEX run that OOM-killed under a tight `16G` budget; that turned out to be + memory starvation, not an executor limitation (see Performance Guidance + and Real-Scale Benchmark above). Fall back to `ThreadPoolExecutor` when + `--mem` is tightly constrained, or outside the `~11 GB` scale actually + validated for HTEX. +- `CYTOTABLE_MAX_THREADS`/`--cpus-per-task` has no single best value — it + depends on source data size. `4` CPUs measured faster on a `~3.9 GB` plate; + `8` CPUs measured `~20%` faster (at higher memory/CPU-time cost) on a + `~11 GB` plate; the `~19 GB` plate was only tested at `4` CPUs. Pick based + on source size and whether wall-clock or resource efficiency matters more; + re-benchmark for sizes outside `~4-19 GB`. See Performance Guidance above + for the current split. +- Size Slurm memory requests at roughly `4-4.5x` the source `.sqlite` file + size (trending down, not up, as source size grows — see Performance + Guidance), not off the tiny bundled fixture or off `chunk_size`. +- Budget `--time` with a superlinear formula (`time ∝ size^1.83`, validated + across three real plate sizes), not a flat number or a linear guess from a + smaller dataset — see Performance Guidance above. +- A large `--mem` request on `acpu` silently grants a proportionally larger + `AllocCPUS` than `--cpus-per-task` requests (`DefMemPerCPU=3840 MB` drives + `AllocCPUS = max(--cpus-per-task, ceil(--mem_GB / 3.75))`). Know this before + reading raw CPU% figures as a contention signal, and before submitting many + large-`--mem` jobs concurrently — each one consumes more of the fairshare + CPU budget than its `--cpus-per-task` implies. + +## Real-Scale Benchmark + +On `2026-08-21`, a real-data matrix converted one JUMP Cell Painting Gallery +plate directly from the public `cellpainting-gallery` S3 bucket +(`no_sign_request=True`, no AWS credentials needed): + +```text +s3://cellpainting-gallery/cpg0016-jump/source_4/workspace/backend/2021_08_23_Batch12/BR00126114/BR00126114.sqlite +``` + +`source_datatype="sqlite"`, `preset="cellprofiler_sqlite_cpg0016_jump"`, +`chunk_size=30000`. Every job requested `--mem=16G`, `--time=1:00:00`. + +| run | executor | cpus | arrow mmap | status | elapsed | Slurm cgroup MaxRSS | +| --- | --- | ---: | ---: | --- | ---: | ---: | +| `large-tpe-4c-c30000-a1` | ThreadPoolExecutor | 4 | 1 (default) | completed | `18m26s` | `16.00 GiB` | +| `large-tpe-8c-c30000-a1` | ThreadPoolExecutor | 8 | 1 (default) | completed | `22m23s` | `16.00 GiB` | +| `large-tpe-4c-c30000-a0` | ThreadPoolExecutor | 4 | 0 | completed | `19m49s` | `16.00 GiB` | +| `large-htexcap-4c-c30000-a1` | HighThroughputExecutor, capped to 4 workers | 4 | 1 (default) | **OOM-killed** | `5m30s` | `16.00 GiB` (killed) | + +All three successful runs produced identical output (`74226` rows, `5946` +columns, `~3.8 GB` parquet), confirming the executor/CPU/mmap choice changes +speed and memory behavior but not correctness. The failed HTEX run had +already written complete per-compartment intermediate parquet +(`cells/`, `cytoplasm/`, `image/`, `nuclei/`, `~3.7 GB` total) before dying at +the join stage — the memory pressure is in the join, not the chunked reads. + +Full job IDs, `sacct`/`time` output, and the interpretation this section +summarizes are in `docs/cytotable-alpine-findings.md`. Treat this as one +real-scale comparison, not an exhaustive sweep — CPU counts other than `4`/`8`, +chunk sizes other than `30000`, and dataset shapes other than this one plate +have not been tested. Re-run a comparable comparison before trusting these +exact numbers for a meaningfully different dataset. + +### Larger-Scale Benchmark (Bigger Plates) + +The `3.87 GB` plate above turned out to be the *smallest* of `17` plates in +the same JUMP batch on S3 — the rest range up to `~30 GB`. On `2026-08-24`, +the same validated configuration (ThreadPoolExecutor, `chunk_size=30000`, +mmap default-on) was tested against two bigger real plates to check whether +the small-plate numbers above generalize: + +| plate | source size | cpus | status | elapsed | Slurm cgroup MaxRSS | +| --- | ---: | ---: | --- | ---: | ---: | +| `BR00126114` (above) | `3.87 GB` | 4 | completed | `18m26s` | `16.00 GiB` | +| `BR00126115` | `11.06 GB` | 4 | completed | `1h59m` | `43.99 GiB` | +| `BR00126115` | `11.06 GB` | 8 | completed | `1h36m` | `54.24 GiB` | +| `BR00126116` | `18.82 GB` | 4 | completed | `5h42m` | `62.67 GiB` | + +**They mostly don't generalize as a fixed number, but they do generalize as a +relationship:** + +- The CPU-count finding flips: `4` CPUs was faster at `3.87 GB`, `8` CPUs was + `~20%` faster at `11.06 GB` (at `~23%` more memory and `~2.75x` more + system-CPU-time). There's a real crossover between these two sizes. + `BR00126116` was only run at `4` CPUs, so whether `8` stays ahead at + `~19 GB` is still open. +- Peak memory did *not* stay at a rising or flat multiple of source size — + the ratio actually declined at `4` CPUs (`4.14x` → `3.98x` → `3.33x` as + source size grew from `3.87 GB` to `11.06 GB` to `18.82 GB`), consistent + with fixed per-run overhead mattering less at larger scale. `8` CPUs pushed + the ratio higher (`4.90x`) on the one plate tested at both CPU counts. +- Wall-clock scaling is confirmed superlinear with all three points now in: + a power-law fit gives `time ∝ size^1.83` and predicts the `18.82 GB` + result within `~2%` from the smallest plate alone — a strong signal this + relationship, not just a rough two-point guess, is real for this dataset + family. A linear guess from the small plate would have underestimated the + `18.82 GB` runtime by nearly `4x`. +- A Slurm scheduling quirk on `acpu` matters for reading these numbers: a + large `--mem` request silently grants far more `AllocCPUS` than + `--cpus-per-task` (the `96G` request behind the `18.82 GB` run got `26` + allocated CPUs against a nominal `4`) — see + `docs/cytotable-alpine-findings.md` for the exact rule and why it means raw + CPU% figures aren't a clean contention signal by themselves. + +See `docs/cytotable-alpine-findings.md` for the full data, job IDs, and +interpretation. + +### HTEX Retest With Real Memory Headroom + +The `16G`-budget HTEX OOM above was only ever tested at that one tight +memory budget. On `2026-08-24`, capped `HighThroughputExecutor( +max_workers_per_node=4)` was re-run against the same `BR00126115` +(`11.06 GB`) plate that `ThreadPoolExecutor` converted in `1h59m`, this time +with `--mem=128G` (`~3x` ThreadPoolExecutor's own peak on that plate) and +`--time=4:00:00`: + +| executor | elapsed | Slurm cgroup MaxRSS | +| --- | ---: | ---: | +| ThreadPoolExecutor (4c) | `1h59m` | `43.99 GiB` | +| HighThroughputExecutor, capped (4c) | `25m48s` | `35.01 GiB` | + +Given real headroom, capped HTEX was `~4.6x` faster than ThreadPoolExecutor +and used `~20%` less memory, on byte-identical output. The original OOM was +memory starvation from a `16G` budget that was sized for +ThreadPoolExecutor's footprint, not HTEX's — not an inherent HTEX +limitation. This is one comparison at one size and one CPU count; see +`docs/cytotable-alpine-findings.md` for the full writeup and what's still +untested (other sizes, `8` workers, uncapped HTEX). + +### Mechanics Smoke Test (Tiny Fixture) + +An earlier same-day pass used Cytotable's bundled test fixture +(`tests/data/cellprofiler/cpg0043-segmentation`, `48` CSV files, `~256 MB`) +with `preset="cellprofiler_csv"` to validate submission mechanics and the +scratch-backed-`HOME` fix before the real-scale matrix above. Two runs +completed: + +| run | elapsed | Slurm MaxRSS | +| --- | ---: | ---: | +| `homefix-threadpool-4c` | `43.83s` | `5966192K` | +| `homefix-htex-capped-4c` | `49.35s` | `4110988K` | + +Use this fixture for correctness/mechanics checks (env setup, scratch-`HOME` +fix, package install) only — it's too small to trust for performance +decisions on its own. Its direction (capped HTEX using less RSS than +ThreadPoolExecutor) turned out to match the real-scale HTEX-with-headroom +result above, not contradict it; what looked like a contradiction after the +first real-scale test was actually that test being memory-starved, not a +genuine reversal. Still don't use fixture-scale numbers for actual +memory/time sizing — the real-scale figures above are the ones to plan from. ## DuckDB And Arrow Settings - Set `CYTOTABLE_MAX_THREADS="$SLURM_CPUS_PER_TASK"` in the batch job before Python starts. Cytotable reads this at import time and uses it for DuckDB - `PRAGMA threads`; setting it later in Python is too late. + `PRAGMA threads`; setting it later in Python is too late. Start from `4` + per Performance Guidance above rather than the node's full CPU count. - Set `HOME` to a scratch job-home after activating the reusable environment and before starting Python, for example: @@ -57,81 +365,44 @@ Alpine skill instead. export HOME="$CYTOTABLE_JOB_HOME" ``` - This redirects DuckDB's default extension cache (`$HOME/.duckdb`) to scratch. - A validated run created about `54 MB` per scratch job-home. -- Leave Cytotable's default Arrow memory mapping enabled for initial runs: - `CYTOTABLE_ARROW_USE_MEMORY_MAPPING=1`. This is the default and should be a - reasonable fit for scratch-backed parquet reads. -- If a run shows unexpectedly high `MaxVMSize`, file-handle/unlink behavior, or - memory pressure during concat/join, run a controlled comparison with - `CYTOTABLE_ARROW_USE_MEMORY_MAPPING=0`. -- Use `chunk_size` as the first memory tuning knob. Smaller chunks lower peak - memory and larger chunks reduce intermediate-file overhead. Start with - `chunk_size=1000` for validation, then tune upward only after Slurm `MaxRSS` - is comfortably below the memory request. + This redirects DuckDB's default extension cache (`$HOME/.duckdb`) to + scratch. Validated at about `54 MB` per scratch job-home, and that figure + did not scale up with real production-size data. +- Leave Cytotable's default Arrow memory mapping enabled: + `CYTOTABLE_ARROW_USE_MEMORY_MAPPING=1`. Measured `~7%` faster than disabling + it on a real conversion, at the cost of `~1 GB` more peak RSS. Disable it + only when a run is memory-constrained enough to need that headroom instead + of the speed. +- `chunk_size` controls per-chunk read/write memory, not the run's overall + peak — the real-scale benchmark's memory pressure showed up entirely at the + final join/concat stage regardless of `chunk_size`. `chunk_size=30000` is + now validated as workable for a real single-plate SQLite conversion under + ThreadPoolExecutor; treat it as a reasonable starting point for + comparably-shaped real sources rather than always starting from a much + smaller value. Size `--mem` off the destination table's joined + row × column shape, not off `chunk_size`. - Do not try to tune DuckDB `memory_limit` first. Cytotable exposes thread control through `CYTOTABLE_MAX_THREADS`, but not a clean per-run DuckDB memory limit hook. Control memory with Slurm `--mem`, `chunk_size`, and one conversion per Python process. -- Keep `sort_output=True` when reproducibility or output comparison matters. - Test `sort_output=False` only when downstream consumers do not require stable - sorted output; it can avoid some `ORDER BY` work. - -## Runtime Setup - -Use a project-owned `uv` environment: - -```bash -export UV_HOME="/projects/$USER/software/uv" -export UV_INSTALL_DIR="$UV_HOME/bin" -export UV_ENVS="$UV_HOME/envs" -export UV_CACHE_DIR="/scratch/alpine/$USER/uv-cache" -export UV_LINK_MODE=copy -export PATH="$UV_INSTALL_DIR:$PATH" -mkdir -p "$UV_INSTALL_DIR" "$UV_ENVS" "$UV_CACHE_DIR" - -if [[ ! -x "$UV_INSTALL_DIR/uv" ]]; then - curl -LsSf https://astral.sh/uv/install.sh | sh -fi - -uv venv "$UV_ENVS/cytotable-alpine" --python 3.12 -source "$UV_ENVS/cytotable-alpine/bin/activate" -uv pip install cytotable -``` - -If installing from a GitHub source archive rather than PyPI or a real git -checkout, Cytotable's `setuptools-scm` build needs an explicit pretend version: - -```bash -export SETUPTOOLS_SCM_PRETEND_VERSION_FOR_CYTOTABLE=0.0.0+alpinebench -uv pip install /scratch/alpine/$USER/cytotable-alpine-bench-20260821/CytoTable-main -``` +- Keep `sort_output=True` (the default) when reproducibility or output + comparison matters; the real-scale benchmark used the default and produced + byte-identical output across all three successful executor/CPU/mmap + combinations. Test `sort_output=False` only when downstream consumers do + not require stable sorted output. ## Parsl Config Pattern -Thread pool baseline: - -```python -from parsl.config import Config -from parsl.executors import ThreadPoolExecutor - -parsl_config = Config( - executors=[ThreadPoolExecutor(label="cytotable_tpe", max_threads=cpus)], - run_dir=f"{scratch_root}/parsl-runs/{run_name}", -) -``` - -HTEX comparison candidates: +Prefer capped `HighThroughputExecutor` when `--mem` can afford real headroom +(`~11 GB`-scale conversions confirmed: `~4.6x` faster, `~20%` less memory than +ThreadPoolExecutor — see Performance Guidance and HTEX Retest With Real +Memory Headroom above). It's also the only option here that can fan out +across multiple Slurm nodes: ```python from parsl.config import Config from parsl.executors import HighThroughputExecutor -default_htex = Config( - executors=[HighThroughputExecutor(label="cytotable_htex")], - run_dir=f"{scratch_root}/parsl-runs/{run_name}-htex-default", -) - capped_htex = Config( executors=[ HighThroughputExecutor( @@ -143,6 +414,21 @@ capped_htex = Config( ) ``` +Fall back to `ThreadPoolExecutor` when `--mem` is tightly constrained (it +needs meaningfully less to succeed, even though it's slower — validated down +to `16G` on a `~3.9 GB` plate) or outside the `~11 GB` scale HTEX has +actually been validated at: + +```python +from parsl.config import Config +from parsl.executors import ThreadPoolExecutor + +parsl_config = Config( + executors=[ThreadPoolExecutor(label="cytotable_tpe", max_threads=cpus)], + run_dir=f"{scratch_root}/parsl-runs/{run_name}", +) +``` + Call `convert(...)` with the config: ```python @@ -154,90 +440,55 @@ convert( dest_datatype="parquet", source_datatype="csv", preset="cellprofiler_csv", - chunk_size=1000, + chunk_size=30000, parsl_config=parsl_config, ) ``` -## Alpine Benchmark State - -On `2026-08-21`, a benchmark harness was created under: - -```text -/scratch/alpine/$USER/cytotable-alpine-bench-20260821 -``` - -The source fixture was Cytotable's current `main` archive, using: - -```text -tests/data/cellprofiler/cpg0043-segmentation -``` - -Fixture size observed on Alpine: - -- `48` CSV files -- `256020900` CSV bytes - -Environment installed successfully: - -- Python `3.12.13` environment requested via `uv` -- Cytotable source archive installed as `0.0.0+alpinebench` -- Parsl `2026.8.10` -- DuckDB `1.5.5` -- PyArrow `25.0.1` - -The first matrix submitted and failed because the harness called -`convert.convert(...)` after `from cytotable import convert`. This was a harness -bug, not a Cytotable or Slurm finding. Failed job IDs: +## Runtime Setup -```text -31536860-31536868 -``` +Cytotable itself has no dependency on any particular Python environment +manager — it's an ordinary `pip`-installable package, so `uv`, `conda`, plain +`venv`+`pip`, or Apptainer/Singularity (see `.agents/skills/alpine/SKILL.md`'s +Runtime Direction for the project-wide tradeoffs between those) all work +mechanically. `uv` is what this skill documents and what every benchmark +above actually ran through — the performance numbers are conditioned on that +runtime, though there's no specific reason to expect DuckDB/PyArrow/Parsl +behavior to differ under a different manager. CURC's general project guidance +favors Apptainer/Singularity for reproducibility and considers conda harder to +maintain long-term; `uv` was picked here for faster iteration while pinning +down the Slurm/executor/memory findings above, not because the other options +are unsupported. Re-validate the performance numbers if switching runtimes, +same as for any other untested configuration change. -The corrected matrix submitted as: +Use a project-owned `uv` environment: -```text -31536906 rerun-threadpool-1c -31536907 rerun-threadpool-4c -31536908 rerun-threadpool-8c -31536909 rerun-htex-default-1c -31536910 rerun-htex-default-4c -31536911 rerun-htex-default-8c -31536912 rerun-htex-capped-1c -31536913 rerun-htex-capped-4c -31536914 rerun-htex-capped-8c -``` +```bash +export UV_HOME="/projects/$USER/software/uv" +export UV_INSTALL_DIR="$UV_HOME/bin" +export UV_ENVS="$UV_HOME/envs" +export UV_CACHE_DIR="/scratch/alpine/$USER/uv-cache" +export UV_LINK_MODE=copy +export PATH="$UV_INSTALL_DIR:$PATH" +mkdir -p "$UV_INSTALL_DIR" "$UV_ENVS" "$UV_CACHE_DIR" -Those jobs left the queue within a few minutes, but their result files could -not be collected until Alpine home space was freed. After collection, all nine -corrected jobs were found to have failed before conversion because DuckDB tried -to install extensions under `$HOME/.duckdb` while home was full: +if [[ ! -x "$UV_INSTALL_DIR/uv" ]]; then + curl -LsSf https://astral.sh/uv/install.sh | sh +fi -```text -IO Error: Cannot open file "/home/.../.duckdb/extensions/v1.5.5/linux_amd64/..." -No space left on device +uv venv "$UV_ENVS/cytotable-alpine" --python 3.12 +source "$UV_ENVS/cytotable-alpine/bin/activate" +uv pip install cytotable ``` -This is an Alpine/Cytotable runtime finding, not an executor-performance -finding: all executor variants failed in about `5-6s`, before the conversion -work began. - -Validated scratch-home follow-up runs: +If installing from a GitHub source archive rather than PyPI or a real git +checkout, Cytotable's `setuptools-scm` build needs an explicit pretend version: -```text -31541899 homefix-threadpool-4c: completed, 43.83s Cytotable elapsed, - 397 rows x 6049 columns, 22.85 MB parquet, Slurm elapsed 1:24, - Slurm batch MaxRSS 5966192K -31541900 homefix-htex-capped-4c: completed, 49.35s Cytotable elapsed, - 397 rows x 6049 columns, 22.85 MB parquet, Slurm elapsed 1:30, - Slurm batch MaxRSS 4110988K +```bash +export SETUPTOOLS_SCM_PRETEND_VERSION_FOR_CYTOTABLE=0.0.0+alpinebench +uv pip install /scratch/alpine/$USER/cytotable-alpine-bench-20260821/CytoTable-main ``` -Interpretation: the scratch-home pattern fixes the DuckDB extension-cache -failure. Threadpool is the current first-choice baseline for speed at `4` CPUs; -capped HTEX remains worth testing for memory-sensitive larger runs. Treat the -memory comparison as one-run evidence, not a universal rule. - ## Failure Modes - Alpine home at `100%` full can prevent even simple SSH commands from running; @@ -249,6 +500,21 @@ memory comparison as one-run evidence, not a universal rule. scratch-backed `HOME` as above, or use a code-level DuckDB `extension_directory` configuration if Cytotable exposes one in the future. - Do not use an existing Cytotable `dest_path`; Cytotable intentionally raises - instead of overwriting. + instead of overwriting. On a failed run, `dest_path` can be left as a + partially-written directory (per-compartment subdirectories) rather than + cleanly absent — remove it explicitly before retrying rather than assuming + a failed run left nothing behind. - Set `parsl_config` on the first Cytotable call in a Python process. Cytotable reuses an already-loaded Parsl DFK and will not switch configs mid-process. +- An HTEX worker OOM does not surface as an out-of-memory error to Cytotable — + it surfaces as a generic Parsl `DependencyError([(WorkerLost(...), 'task + N')], M)`. Check the Slurm `.err` log for `oom_kill event` before treating + this as a code or data bug. +- A large `--mem` request on `acpu` grants more actual CPU allocation than + `--cpus-per-task` alone implies: `AllocCPUS = max(--cpus-per-task, + ceil(--mem_GB / 3.75))`, driven by the partition's `DefMemPerCPU=3840 MB`. + Confirmed across five jobs (`16G`/`4c`→`5` allocated, `16G`/`8c`→`8`, + `64G`/`4c or 8c`→`18` both times, `96G`/`4c`→`26`). Check `sacct AllocCPUS` + before reading a job's CPU% as a clean measure of thread contention, and + remember that pushing `--mem` up for safety also multiplies the job's real + CPU/fairshare footprint. diff --git a/docs/cytotable-alpine-findings.md b/docs/cytotable-alpine-findings.md index 43d0ded..3a18edb 100644 --- a/docs/cytotable-alpine-findings.md +++ b/docs/cytotable-alpine-findings.md @@ -1,8 +1,13 @@ # Cytotable Alpine Findings -These notes record the `2026-08-21` CytoTable-on-Alpine experiment setup and -what was learned before Alpine account-home exhaustion blocked result -collection. +These notes record the CytoTable-on-Alpine benchmarking that produced the +concrete performance findings now driving +`.agents/skills/cytotable/SKILL.md`: an initial tiny-fixture mechanics check +and a same-day real-scale S3 benchmark on `2026-08-21`, followed by a +larger-scale benchmark against two bigger real plates on `2026-08-24` that +tested whether the `2026-08-21` findings hold as source data size grows (they +mostly don't as fixed numbers, but do as relationships — see Larger-Scale +Benchmark). ## Goal @@ -16,10 +21,10 @@ Compare Cytotable Parsl configuration on Alpine compute nodes: The run intentionally avoided `Persistence1`; each Cytotable conversion was a plain Slurm batch job submitted from the Alpine login node. -## Dataset +## Dataset (Tiny Fixture Phase) -The benchmark used Cytotable's current `main` source archive and bundled test -fixture: +The initial benchmark used Cytotable's current `main` source archive and +bundled test fixture: ```text tests/data/cellprofiler/cpg0043-segmentation @@ -31,7 +36,11 @@ Observed on Alpine: - `256020900` CSV bytes This fixture is large enough to exercise CSV parsing, chunking, joining, and -Parquet writing, while still being appropriate for short `acpu` jobs. +Parquet writing, while still being appropriate for short `acpu` jobs — but, as +the real-scale phase below shows, it is not large enough to reveal the +memory/executor behavior that actually matters for production data. Treat +results from this phase as mechanics/correctness smoke tests, not performance +guidance. ## Runtime @@ -56,7 +65,7 @@ GitHub source archives do not include `.git` metadata, so Cytotable's `setuptools-scm` build fails unless a pretend version is supplied or the package is installed from PyPI or a real git checkout. -## Experiment Matrix +## Experiment Matrix (Tiny Fixture Phase) First submitted job IDs: @@ -122,52 +131,309 @@ Results: The scratch job-home directories were about `54 MB` each, dominated by DuckDB extension cache state. -Interpretation: - -- Scratch-backed `HOME` is a required Alpine best practice for Cytotable unless - Cytotable exposes a DuckDB `extension_directory` hook in the future. -- Threadpool is the current first-choice `4` CPU baseline for speed on this - fixture. -- Capped HTEX is still worth testing for larger memory-sensitive runs because - it used less Slurm-reported peak RSS in this one comparison. -- Default uncapped HTEX should not be used for production guidance until it - passes a scratch-home benchmark. - -## Current Guidance - -Until the corrected matrix is collected, use this as provisional guidance: - -- Use one Cytotable conversion per Slurm job on `acpu`. -- Submit directly from the Alpine login node; do not use `Persistence1` for - this non-Nextflow shape. -- Run the Cytotable process with a scratch-backed `HOME` so DuckDB extension - installs use scratch rather than `$HOME/.duckdb`. -- Start with `ThreadPoolExecutor(max_threads=)`, `--cpus-per-task=4`, - `--mem=8G`, and `CYTOTABLE_MAX_THREADS=`. -- Keep all Cytotable inputs, outputs, Parsl `run_dir`, and logs on scratch. -- Keep reusable `uv` environments under project storage and the `uv` cache on - scratch. -- Use capped HTEX as a comparison candidate for memory-sensitive larger runs, - but do not treat default uncapped HTEX as validated. +At this tiny scale, capped HTEX looked slightly more memory-frugal (lower +Slurm MaxRSS) than ThreadPoolExecutor, at a small speed cost. **The real-scale +phase below reverses this reading** — do not use the tiny-fixture memory +comparison to justify HTEX for production data. -## Collection Commands +## Real-Scale S3 Benchmark + +Later the same day, a second matrix targeted a real production-shaped input +instead of the bundled fixture: one real JUMP Cell Painting Gallery plate, +read directly from the public `cellpainting-gallery` S3 bucket with anonymous +access (`no_sign_request=True`, no AWS credentials configured): + +```text +s3://cellpainting-gallery/cpg0016-jump/source_4/workspace/backend/2021_08_23_Batch12/BR00126114/BR00126114.sqlite +``` + +Conversion parameters: `source_datatype="sqlite"`, +`preset="cellprofiler_sqlite_cpg0016_jump"`, `chunk_size=30000`, scratch-backed +`HOME` and `local_cache_dir` per the scratch-home fix above. Slurm request for +every job in this matrix: `--mem=16G`, `--time=1:00:00`, `node=1`. + +Submitted job IDs: + +| job ID | run | executor | cpus | arrow mmap | +| ---: | --- | --- | ---: | ---: | +| `31542591` | `large-tpe-4c-c30000-a1` | ThreadPoolExecutor | 4 | 1 (default) | +| `31542592` | `large-tpe-8c-c30000-a1` | ThreadPoolExecutor | 8 | 1 (default) | +| `31542593` | `large-htexcap-4c-c30000-a1` | HighThroughputExecutor, capped to 4 workers | 4 | 1 (default) | +| `31542594` | `large-tpe-4c-c30000-a0` | ThreadPoolExecutor | 4 | 0 | + +Results (`sacct` and GNU `time`-wrapped process stats): -After freeing Alpine home space: +(Memory figures are KiB→GiB, matching the correction applied throughout this +document — see Larger-Scale Benchmark below.) + +| run | status | elapsed | CPU% | process MaxRSS (`time`) | Slurm cgroup MaxRSS (`sacct`) | +| --- | --- | ---: | ---: | ---: | ---: | +| `large-tpe-4c-c30000-a1` | completed | `18m26s` | `143%` | `13.93 GiB` | `16.00 GiB` | +| `large-tpe-8c-c30000-a1` | completed | `22m23s` | `392%` | `12.71 GiB` | `16.00 GiB` | +| `large-tpe-4c-c30000-a0` | completed | `19m49s` | `177%` | `12.94 GiB` | `16.00 GiB` | +| `large-htexcap-4c-c30000-a1` | **OOM-killed** | `5m30s` | `4%` (driver process only) | `0.39 GiB` (driver process only) | `16.00 GiB` (killed) | + +All three successful runs produced identical output: `74226` rows, `5946` +columns, `~3.8 GB` parquet — confirming the executor/CPU/mmap choice did not +change correctness, only speed and memory behavior. + +The failed HTEX run left a partially-written `dest_path` directory +(`cells/`, `cytoplasm/`, `image/`, `nuclei/` subdirectories, `~3.7 GB` total) +on disk, and its `.err` log shows: + +```text +slurmstepd: error: Detected 1 oom_kill event in StepId=31542593.batch. Some of the step tasks have been OOM Killed. +``` + +with the Cytotable-side exception surfacing as a generic Parsl +`DependencyError([(WorkerLost(1, ''), 'task 23')], 24)` rather than an +explicit out-of-memory message — worth recognizing that shape as a likely OOM +when triaging failures, since it does not say "memory" anywhere in the text. + +Other measured storage facts from this phase: + +- `local_cache_dir` (the local S3 staging cache for the source `.sqlite` file) + used about `3.7 GB` per run, separate from the destination parquet and the + scratch job-home. +- The scratch job-home DuckDB extension cache stayed at the same `~54 MB` seen + in the tiny-fixture phase — it does not scale with dataset size. + +### Interpretation + +1. **ThreadPoolExecutor is the only executor validated at real-plate scale.** + The capped `HighThroughputExecutor` run OOM-killed under the identical + `16G`/4-cpu request, dying during the final join/concat stage after + already writing the full set of per-compartment intermediate parquet + output. The tiny-fixture reading that capped HTEX was more memory-frugal + does not hold at real scale — it inverts. Do not extrapolate small-fixture + executor/memory comparisons to production data; re-validate at real scale + before trusting a memory claim about an executor. +2. **More CPUs made this workload slower, not faster — at this data size.** + Going from `4` to `8` CPUs increased ThreadPoolExecutor wall time by about + `21%` (`18m26s` to `22m23s`) while total CPU-seconds consumed nearly + tripled. System time specifically exploded (`756s` to `3228s`, i.e. from + about `48%` to `145%` of user time), a strong signature of contention — + likely DuckDB's internal thread pool or shared scratch/GPFS I/O — + outweighing added parallelism at this data shape. **This finding does not + hold at larger data sizes — see Larger-Scale Benchmark below, where `8` + CPUs became faster than `4` on an `~11 GB` plate.** Re-benchmark before + scaling CPU count up for a given data size; do not assume the optimal CPU + count is fixed across dataset sizes. +3. **Default Arrow memory mapping is a real, measured win, with a memory + tradeoff.** `CYTOTABLE_ARROW_USE_MEMORY_MAPPING=1` (the default) was about + `7%` faster than disabling it at 4 CPUs (`18m26s` vs `19m49s`), at the cost + of about `1 GiB` more process-level peak RSS (`13.93 GiB` vs `12.94 GiB`). +4. **`16G` was exactly enough, not generous.** Slurm's cgroup-reported MaxRSS + landed at exactly `16.00 GiB` against the `16G` (`16.00 GiB` — Slurm's `G` + suffix is binary) request for every configuration tested, including the + successful runs — every run, passing or OOM-killed, was pinned right at + the cgroup ceiling. Treat `16G` as an exact floor for a single real + single-plate SQLite conversion at this column count (`~5946` columns after + join), not a comfortable production number. +5. **`chunk_size` does not bound the peak.** All memory pressure showed up at + the final join/concat stage, not during chunked reads — the failed HTEX + run had already written complete per-compartment intermediate output + before dying. `chunk_size=30000` is validated as workable for this real + single-plate conversion under ThreadPoolExecutor, but it is a per-chunk + read/write knob, not a memory ceiling on the run as a whole. +6. **Real single-plate wall-clock is `~18-22` minutes end-to-end** (S3 + download plus conversion), not the `~44-50` seconds seen on the tiny + bundled CSV fixture. Use this, not the fixture number, as the time-budget + calibration point for real production submissions. + +## Larger-Scale Benchmark + +The `~4 GB`-source plate above (`BR00126114`) turned out to be the *smallest* +of `17` plates in the same JUMP batch on S3 — the other plates range up to +`~30 GB` per `.sqlite` source file. On `2026-08-24`, three more real +conversions tested whether the small-plate findings (executor choice, `4` +CPUs, `--mem`/`--time` sizing) hold as source size grows, using the same +validated configuration (`ThreadPoolExecutor`, `chunk_size=30000`, +`CYTOTABLE_ARROW_USE_MEMORY_MAPPING=1`, scratch-backed `HOME`): + +| job ID | run | plate | source size | cpus | +| ---: | --- | --- | ---: | ---: | +| `31608776` | `xl-tpe-4c-BR00126115` | `BR00126115` | `11.06 GB` | 4 | +| `31608777` | `xl-tpe-8c-BR00126115` | `BR00126115` | `11.06 GB` | 8 | +| `31608778` | `xl-tpe-4c-BR00126116` | `BR00126116` | `18.82 GB` | 4 | + +`--mem=64G` for the two `BR00126115` jobs, `--mem=96G` for the `BR00126116` +job; `--time=3:00:00` and `6:00:00` respectively, sized with deliberate +headroom over a naive linear extrapolation from the small-plate numbers. + +Results (memory figures corrected to KiB→GiB; an earlier pass mislabeled the +`BR00126114` figure using a decimal-KB shortcut — `16775024 KiB / 1024^2 = +16.00 GiB`, not `16.78 GB` as originally written): + +| run | source size | cpus | status | elapsed | CPU% | Slurm cgroup MaxRSS | rows | columns | parquet size | +| --- | ---: | ---: | --- | ---: | ---: | ---: | ---: | ---: | ---: | +| `large-tpe-4c-c30000-a1` (`BR00126114`, prior phase) | `3.87 GB` | 4 | completed | `18m26s` | `143%` | `16.00 GiB` | `74226` | `5946` | `~3.8 GB` | +| `xl-tpe-4c-BR00126115` | `11.06 GB` | 4 | completed | `1h59m` | `253%` | `43.99 GiB` | `215296` | `5946` | `~11.5 GB` | +| `xl-tpe-8c-BR00126115` | `11.06 GB` | 8 | completed | `1h36m` | `576%` | `54.24 GiB` | `215296` | `5946` | `~11.5 GB` | +| `xl-tpe-4c-BR00126116` | `18.82 GB` | 4 | completed | `5h42m` | `680%` | `62.67 GiB` | `366678` | `5946` | `~19.6 GB` | + +All completed runs (`BR00126114` through `BR00126116`) share the same `5946` +columns (same CellProfiler/JUMP schema across plates); row counts and output +size scale with each plate's own well/cell count and track source `.sqlite` +size closely (output-to-source byte ratio: `0.98x`, `1.04x`, `1.04x` across +the three plates). 4c and 8c on `BR00126115` produced identical row/column +counts and identical drop-duplicate results, matching the correctness +pattern from the small-plate phase; correctness held across all three sizes. + +### A Scheduling Quirk That Explains Some Of The CPU% Readings + +`sacct AllocCPUS` for these jobs does not match `--cpus-per-task`: + +| job | `--cpus-per-task` | `--mem` | `AllocCPUS` | +| --- | ---: | ---: | ---: | +| `31542591` (`BR00126114`, 4c) | 4 | `16G` | `5` | +| `31542592` (`BR00126114`, 8c) | 8 | `16G` | `8` | +| `31608776` (`BR00126115`, 4c) | 4 | `64G` | `18` | +| `31608777` (`BR00126115`, 8c) | 8 | `64G` | `18` | +| `31608778` (`BR00126116`, 4c) | 4 | `96G` | `26` | + +The pattern is exactly `AllocCPUS = max(--cpus-per-task, ceil(--mem_GB / +3.75))` — this partition's `DefMemPerCPU=3840 MB` (`~3.75 GB`) means a large +`--mem` request silently grants a proportionally larger CPU allocation +regardless of `--cpus-per-task`, matching the `acpu` accounting quirk noted +in the tiny-fixture phase (`--cpus-per-task=1` reporting `AllocCPUS=3` there, +consistent with an `8G` request: `ceil(8/3.75)=3`). Two consequences: + +- **CPU% readings above the nominal `--cpus-per-task × 100%` are not purely + a contention signal.** The `BR00126116` job nominally requested `4` CPUs + but was actually allocated `26` — background library threads not gated by + `CYTOTABLE_MAX_THREADS` (BLAS, Arrow's own I/O thread pool, etc.) had a much + wider cgroup CPU budget available than "4" implies. The CPU-count + comparison in this doc is still valid as a comparison of + `ThreadPoolExecutor(max_threads=...)`/`CYTOTABLE_MAX_THREADS` — that + parameter is what was actually varied and is what controls + Cytotable/DuckDB-level parallelism — but do not read the raw `%CPU` figure + as a clean measure of contention without checking `AllocCPUS` first. +- **A memory-headroom-driven `--mem` request buys (and consumes fairshare + for) far more CPU allocation than requested.** Pushing `--mem` up for + safety, per the memory guidance below, silently multiplies the job's actual + CPU footprint too — worth knowing before submitting many such jobs + concurrently. + +### Interpretation + +1. **The CPU-count finding from the small plate does not generalize — it + inverts.** On `BR00126114` (`3.87 GB`), `4` CPUs beat `8` (`18m26s` vs + `22m23s`). On `BR00126115` (`11.06 GB`), `8` CPUs beat `4` by about `20%` + (`1h36m` vs `1h59m`). There is a real crossover somewhere between these two + sizes where added `ThreadPoolExecutor` parallelism stops being dominated + by overhead and starts paying off. Do not treat "`4` CPUs is fastest" as a + fixed rule; treat it as true only below roughly `~5 GB` of source data, + pending a finer-grained sweep. (`BR00126116` was only tested at `4` CPUs, + so it doesn't yet confirm whether `8` stays ahead at `~19 GB` — worth a + follow-up run.) +2. **The `8`-CPU speedup is not free.** It cost about `2.75x` more total + system-CPU-time (`9305s` to `25652s` sys time) and about `23%` more peak + memory (`43.99 GiB` to `54.24 GiB`) than the `4`-CPU run on the same + plate. Choose `8` CPUs when minimizing wall-clock matters most and the + memory budget allows it; choose `4` CPUs when running many jobs + concurrently or when resource efficiency (CPU-seconds, memory, and the + `AllocCPUS` fairshare footprint above) matters more than any one job's + wall-clock. +3. **Peak memory does not scale in strict proportion to source size — the + multiplier trends down as data grows, at `4` CPUs.** Ratio of Slurm cgroup + MaxRSS to source `.sqlite` size: `4.14x` at `3.87 GB`, `3.98x` at + `11.06 GB`, `3.33x` at `18.82 GB` (all at `4` CPUs); `8` CPUs pushed the + ratio up to `4.90x` on the `11.06 GB` plate, the one size tested at both + CPU counts. All three plates share the same `5946`-column schema, so this + looks like fixed per-run overhead (Python/Parsl/DuckDB baseline working + set) being a larger fraction of peak memory at small scale and amortizing + as source size grows, rather than a fixed multiplier. Planning rule from + these three points: `~4.5x` source size for anything under `~5 GB`, + `~4x` for a single-plate conversion in the `~10-20 GB` range, `~5x` if + running at `8` CPUs — with real headroom in all cases, since this is only + three data points across one dataset family. +4. **Wall-clock scaling is confirmed superlinear across all three points, not + just the first two.** A power-law fit (least-squares in log-log space) + across all three `(source_GB, elapsed_s)` pairs gives + `time ∝ size^1.83` — very close to the `^1.77` estimated from just the + first two points, and it predicts the `BR00126116` result (`336` min) + within `~2%` of the actual `342` min. Treat `time ≈ T_ref × (size / + size_ref)^1.8` as a validated planning formula for this dataset family, + not a rough guess — but keep giving real headroom on top of it, since it's + still anchored by only three points from one JUMP batch. + +## HTEX Retest With Real Memory Headroom + +The `2026-08-21` finding that capped `HighThroughputExecutor` OOM-killed +under a `16G`/4-cpu request (see Real-Scale S3 Benchmark) had only ever been +tested at that one tight memory budget. On `2026-08-24`, the same +`BR00126115` plate (`11.06 GB` source) that `ThreadPoolExecutor` converted in +`1h59m` at `4` CPUs was re-run with `HighThroughputExecutor( +max_workers_per_node=4)`, this time with deliberately generous headroom — +`--mem=128G` (`~3x` ThreadPoolExecutor's observed `43.99 GiB` peak on the same +plate) and `--time=4:00:00` (`~2x` ThreadPoolExecutor's runtime) — to find +out whether the earlier failure was a memory-starvation artifact or a real +limitation of the executor. + +Job `31624016` (`htex-retest-4c-BR00126115`): + +| metric | ThreadPoolExecutor (4c) | HighThroughputExecutor, capped (4c) | +| --- | ---: | ---: | +| status | completed | completed | +| elapsed | `1h59m` (`7081s`) | `25m48s` (`1548s`) | +| Slurm cgroup MaxRSS | `43.99 GiB` | `35.01 GiB` | +| rows / columns | `215296` / `5946` | `215296` / `5946` | +| parquet bytes | `11546075654` | `11546075654` | + +**Given real headroom, capped HTEX was `~4.6x` faster than ThreadPoolExecutor +on the identical plate, and used `~20%` less memory (`35.01 GiB` vs +`43.99 GiB`, a `3.16x` multiple of source size vs ThreadPoolExecutor's +`3.98x`).** Output was byte-identical to the corresponding ThreadPoolExecutor +run (same row/column counts, same parquet byte size), confirming this is a +performance difference, not a correctness one. + +### Interpretation + +1. **The original "avoid HTEX" finding was a memory-starvation artifact, not + an executor limitation.** Once HTEX had real headroom instead of a budget + that was already tight for ThreadPoolExecutor, it did not just survive — + it substantially outperformed ThreadPoolExecutor on both wall-clock and + peak memory on the one real plate this was tested against. Do not read + the `2026-08-21` OOM as "HTEX can't handle real data"; read it as "HTEX + needs a properly-sized `--mem`, same as anything else, and the `16G` + budget was sized for ThreadPoolExecutor's footprint, not HTEX's." +2. **A process-based executor plausibly wins here precisely because it's not + thread-based.** `ThreadPoolExecutor` runs everything inside one Python + process; HTEX runs separate OS worker processes. That costs more baseline + memory per worker (which is exactly what starved it under `16G`), but it + sidesteps Python-level contention (GIL, shared-object contention, or + contention inside DuckDB's own thread pool when driven by multiple + Python threads in one process) that the `4`-vs-`8`-CPU + `ThreadPoolExecutor` findings elsewhere in this doc suggest is a real, + measurable cost at this data scale. This is a plausible explanation, not + directly instrumented — worth deeper profiling if the pattern holds up. +3. **This is one comparison at one size and one CPU count — it needs + replication before being treated as a general rule.** Not yet tested: + capped HTEX with headroom on the `3.87 GB` or `18.82 GB` plates, HTEX at + `8` workers, or Cytotable-style *default* (uncapped) HTEX with headroom. + Given how large and consistent-with-a-real-mechanism this result is, + though, it's strong enough to flip the *default* recommendation from + "avoid HTEX" to "prefer HTEX when you can afford real memory headroom, + fall back to ThreadPoolExecutor when memory is tight" — see the updated + guidance in `.agents/skills/cytotable/SKILL.md`. + +## Collection Commands ```bash ROOT=/scratch/alpine/$USER/cytotable-alpine-bench-20260821 cd "$ROOT" -for f in results/rerun-*.json; do +for f in results/large-*.json results/xl-*.json results/htex-retest-*.json; do printf 'FILE %s\n' "$f" cat "$f" done -sacct -j 31536906,31536907,31536908,31536909,31536910,31536911,31536912,31536913,31536914 \ - --format=JobIDRaw,JobName%26,State,ExitCode,Elapsed,AllocCPUS,ReqCPUS,ReqMem,MaxRSS,MaxVMSize,NNodes,NodeList -P +sacct -j 31542591,31542592,31542593,31542594,31608776,31608777,31608778,31624016 \ + --format=JobIDRaw,JobName%30,State,ExitCode,Elapsed,AllocCPUS,ReqCPUS,ReqMem,MaxRSS,MaxVMSize,NNodes,NodeList -P -for f in logs/rerun-*.err; do +for f in logs/large-*.err logs/xl-*.err logs/htex-retest-*.err; do printf 'ERR %s\n' "$f" - grep -E 'Elapsed|Maximum resident|Percent of CPU|Exit status|Command exited' "$f" || true + grep -E 'Elapsed|Maximum resident|Percent of CPU|Exit status|Command exited|oom_kill' "$f" || true done ``` From b8e7fec0ce9807d33ffe658fed2a4df17b61fa96 Mon Sep 17 00:00:00 2001 From: d33bs Date: Mon, 24 Aug 2026 17:18:11 -0600 Subject: [PATCH 2/2] Update cytotable-alpine-findings.md --- docs/cytotable-alpine-findings.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/cytotable-alpine-findings.md b/docs/cytotable-alpine-findings.md index 3a18edb..3d330d3 100644 --- a/docs/cytotable-alpine-findings.md +++ b/docs/cytotable-alpine-findings.md @@ -198,7 +198,7 @@ Other measured storage facts from this phase: - The scratch job-home DuckDB extension cache stayed at the same `~54 MB` seen in the tiny-fixture phase — it does not scale with dataset size. -### Interpretation +### Interpretation: Real-Scale S3 Benchmark 1. **ThreadPoolExecutor is the only executor validated at real-plate scale.** The capped `HighThroughputExecutor` run OOM-killed under the identical @@ -315,7 +315,7 @@ consistent with an `8G` request: `ceil(8/3.75)=3`). Two consequences: CPU footprint too — worth knowing before submitting many such jobs concurrently. -### Interpretation +### Interpretation: Larger-Scale Benchmark 1. **The CPU-count finding from the small plate does not generalize — it inverts.** On `BR00126114` (`3.87 GB`), `4` CPUs beat `8` (`18m26s` vs @@ -388,7 +388,7 @@ on the identical plate, and used `~20%` less memory (`35.01 GiB` vs run (same row/column counts, same parquet byte size), confirming this is a performance difference, not a correctness one. -### Interpretation +### Interpretation: HTEX Retest With Real Memory Headroom 1. **The original "avoid HTEX" finding was a memory-starvation artifact, not an executor limitation.** Once HTEX had real headroom instead of a budget