Release 0.6.0

.github/workflows/benchmark.yml

@@ -0,0 +1,137 @@
+# Comparative benchmark: adata-cli against anndata and scanpy.
+#
+# Report-only, by design. Wall time on a shared runner is too noisy to gate a
+# release on, and a benchmark that can block a publish stops being run. What
+# *does* gate merges is tests/test_performance.py, which counts operations
+# rather than seconds and lives in the ordinary test job.
+#
+# Its own file rather than a job in publish.yml: at the `ci` tier this takes
+# the better part of an hour, and hanging that off the release graph would
+# either delay the PyPI publish or paint the release run red for a report.
+name: Benchmark
+
+on:
+  push:
+    tags: ["*"]
+  workflow_dispatch:
+    inputs:
+      tier:
+        description: Input size
+        type: choice
+        options: [smoke, ci, large]
+        default: ci
+      publish:
+        description: Commit the results to docs/ on main
+        type: boolean
+        default: false
+
+concurrency:
+  group: benchmark-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  benchmark:
+    name: adata-cli vs anndata/scanpy
+    runs-on: ubuntu-latest
+    timeout-minutes: 90
+    # Job level, not step level: a report must never mark a release red.
+    continue-on-error: true
+    permissions:
+      contents: write
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      - uses: astral-sh/setup-uv@v3
+        with:
+          enable-cache: true
+
+      # The checked-out source, not the published wheel. Benchmarking the
+      # wheel would mean waiting on publish-pypi and then on index
+      # propagation, for a measurement that would come out the same.
+      - name: Install adata-cli
+        run: uv sync --extra dev --frozen
+
+      - name: Show free space
+        run: df -h /
+
+      - name: Run
+        run: |
+          uv run python -m benchmarks.run \
+            --tier "${{ inputs.tier || 'ci' }}" \
+            --out results.json \
+            --work "$RUNNER_TEMP/bench"
+
+      - name: Render
+        run: |
+          uv run python -m benchmarks.report results.json --out summary.md
+          cat summary.md >> "$GITHUB_STEP_SUMMARY"
+
+      - uses: actions/upload-artifact@v4
+        if: always()
+        with:
+          name: benchmark-${{ github.ref_name }}
+          path: |
+            results.json
+            summary.md
+          retention-days: 90
+
+      # docs/ is the durable home: artifacts expire, and one absolute number
+      # with nothing to compare it against says very little. The page is
+      # already served by Pages from docs/, so this needs no extra machinery.
+      - name: Publish to docs
+        if: github.ref_type == 'tag' || inputs.publish
+        run: |
+          git config user.name  "github-actions[bot]"
+          git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
+          # A tag build is on a detached HEAD; the page belongs on main.
+          git fetch origin main
+          git checkout -B main origin/main
+          uv run python -m benchmarks.report results.json --docs docs --publish
+          git add docs/BENCHMARKS.md docs/benchmarks
+          if git diff --cached --quiet; then
+            echo "nothing to publish"; exit 0
+          fi
+          git commit -m "Benchmark results for ${{ github.ref_name }} [skip ci]"
+          # Another job may have landed on main in the meantime.
+          for attempt in 1 2 3; do
+            if git push origin main; then exit 0; fi
+            git pull --rebase origin main
+          done
+          echo "could not push benchmark results"; exit 1
+
+      # Best effort, and last: releases here are cut by hand, so this job may
+      # well run before one exists. The docs page is authoritative either way.
+      - name: Append to the release notes
+        if: github.ref_type == 'tag'
+        continue-on-error: true
+        env:
+          GH_TOKEN: ${{ github.token }}
+        run: |
+          tag="${{ github.ref_name }}"
+          gh release view "$tag" >/dev/null 2>&1 || {
+            echo "no release for $tag yet; skipping"; exit 0
+          }
+          gh release view "$tag" --json body -q .body > body.md
+          # Idempotent: replace any block this job wrote before.
+          python - <<'PY'
+          import pathlib, re
+          body = pathlib.Path("body.md").read_text()
+          summary = pathlib.Path("summary.md").read_text()
+          block = (
+              "<!-- benchmark:start -->\n\n"
+              "<details><summary>Benchmark vs anndata/scanpy</summary>\n\n"
+              f"{summary}\n</details>\n\n<!-- benchmark:end -->"
+          )
+          pattern = re.compile(
+              r"<!-- benchmark:start -->.*?<!-- benchmark:end -->", re.S
+          )
+          body = (
+              pattern.sub(block, body)
+              if pattern.search(body)
+              else body.rstrip() + "\n\n" + block + "\n"
+          )
+          pathlib.Path("body.md").write_text(body)
+          PY
+          gh release edit "$tag" --notes-file body.md

.gitignore

@@ -10,3 +10,9 @@ htmlcov/
 .pytest_cache/
 pytest-results*.xml
 compat-results.xml
+
+# Benchmark working directory and results -- the defaults of
+# `python -m benchmarks.run`, both written into the repo root.
+.bench/
+results.json
+summary.md

CHANGELOG.md

@@ -3,6 +3,117 @@
 Notable changes to `adata-cli`. Versions are `MAJOR.MINOR.PATCH`; tags carry no
 `v` prefix.
 
+## 0.6.0
+
+Adds `adata convert`, and fixes four quadratic paths that made `concat` and
+`split` appear to hang on real data. Three of the four were found by a new
+suite of complexity guards, which now run on every pull request; the first
+was reported from a pipeline that had to kill twelve tasks after 98 minutes.
+
+### Added
+
+- **`adata convert` changes a matrix's dtype, layout or density on disk**
+  ([#13](https://github.com/cellgeni/adata-cli/issues/13)). Counts stored as
+  float64 halve with `--dtype float32`; `--indices-dtype int32` halves the
+  index arrays of a matrix small enough to address that way; `--layout
+  csr|csc` transposes between the sparse encodings; `--layout dense|sparse`
+  changes the density. Works on `X`, any layer, `raw/X` or any 2-D array by
+  path, and on all of them at once with `--all`.
+
+  A cast that would not round-trip is refused before anything is written --
+  every value is cast and cast back, because whether float64 counts survive
+  float32 depends on the counts, not on the dtypes. So is a densification
+  that would inflate the store beyond four times its size. `--force`
+  overrides either, and both messages say what they measured.
+
+  Transposing streams by default, holding one bucket of nonzeros rather than
+  the matrix, so it works on files too large to load; `--in-memory` is
+  faster when the matrix fits. Buckets are balanced by nonzero count rather
+  than by coordinate range, because a single-cell matrix is skewed -- a few
+  genes carry most of the counts -- and equal-width bounds put most of one
+  in a single bucket.
+
+  `--indices-dtype` is checked against both what `indices` must address and
+  what `indptr` must reach, which differ: a narrow matrix with more than
+  2^31 nonzeros needs int64 offsets over int32 coordinates. Both are
+  preserved from the source when not specified.
+
+- **`concat` now names the command to run** when inputs disagree about a
+  matrix encoding. The check itself is not new, but nothing tested it and it
+  could not suggest a fix, because there was none.
+
+- **Complexity guards in the test suite** (`tests/test_performance.py`).
+  Cost regressions now fail at merge time. They count operations rather than
+  seconds -- h5py and zarr reads, Zarr store traffic, Python allocation and
+  executed lines -- and assert that successive increments grow no faster than
+  linearly, so nothing here can fail because a CI runner was busy. See
+  [docs/TESTING.md](docs/TESTING.md#performance). Every subcommand is covered:
+  `ls`, `view`, `create`, all five `export` and all five `import` variants,
+  `split` on both axes, and the `concat` options nothing else reached.
+  Two claims are now enforced rather than described -- `view` and `ls` read
+  **zero** data elements at any store size, and streaming stays far below the
+  input curve at a fixed `--chunk`.
+
+- **A comparative benchmark** (`benchmarks/`), run on every tag against
+  anndata and against scanpy where scanpy has a real equivalent. Reports peak
+  RSS, wall time and output size; publishes to
+  [docs/BENCHMARKS.md](docs/BENCHMARKS.md) and the release notes. Report-only
+  -- it never fails a build. Fifteen cases, covering every command with a real
+  baseline, including `h5ls -r` for `ls` and the rows where adata-cli is the
+  slower of the two.
+
+- **`--merge drop` and `--uns-merge drop` are accepted.** `drop` was already
+  the documented default behaviour but was rejected as a value, so a config
+  could not state it explicitly.
+
+### Fixed
+
+- **`concat --merge` never finished on a real store.** Aligning a var column
+  onto the target index re-read the whole column from disk once per target
+  variable, so the cost was quadratic: at 36,601 variables a merge that should
+  take a fraction of a second ran for hours at 100% CPU with the output file
+  never growing past its header. Reported against 0.5.1 (REQ-71798), where 12
+  of 13 pipeline tasks had to be killed after 98 minutes. The column is now
+  read once per input, and `--merge first` / `--merge only`, which decide on
+  presence alone, read no column values at all.
+
+- **`concat` was quadratic in the number of categories** in an obs column.
+  Category merging probed a list rather than a dict: 2,096,128 string
+  comparisons to union 1,024 categories, and around 5e9 for a 100k-category
+  column. Found by the new guards.
+
+- **`split --by` was quadratic**, O(n_rows x n_groups). `group_indices` grouped
+  rows with `np.nonzero(values == label)` inside a loop over distinct labels,
+  rescanning each chunk once per label: at 4,096 rows, 16,384 elements scanned
+  for 4 groups and 1,048,576 for 256. A million cells split by a thousand
+  samples is ~10^9 comparisons. One `np.unique` pass per chunk makes it flat
+  in the group count. Found by the new guards; order of first appearance,
+  which names the output files, is unchanged.
+
+- **`concat` built a Python object per row** for nullable and string obs
+  columns, then walked the list twice more. Filling a typed buffer by slice
+  removes three full passes over every such column.
+
+- **Copying variable-length strings ignored its own read budget.** The width
+  of a vlen element was assumed to be 64 bytes, because h5py reports the
+  itemsize of a pointer, so the step was the same 524,288 elements whatever
+  the data held: 2 GiB per read at 4 KiB elements against a stated 32 MiB
+  budget, and for any array shorter than that step, the whole array in one
+  go. Copying 200,000 strings of 4 KiB peaked at 827 MB. The width is now
+  sampled from the first 256 elements. Reported by an automated review on
+  PR #14 and confirmed by measurement; `uns` can hold arbitrary text, so this
+  was not a width the layer could assume.
+
+- **Peak RSS in the benchmark was floored by the runner's own memory on Linux.**
+  A forked child inherits its parent's resident pages and `execve` folds that
+  into the `maxrss` the kernel reports, so every contender would have measured
+  at least what `benchmarks/run.py` used to build the fixtures — around
+  200 MB — and the tables would have read "everything costs about the same".
+  Commands are now forked from a small shim: with a 330 MB parent, a no-op
+  child goes from 326 MB to 8 MB. Caught by `test_benchmark_harness.py`, which
+  exists for exactly this. The published figures were measured on macOS, which
+  resets the high-water mark at exec, and are unchanged.
+
 ## 0.5.1
 
 Makes the container image usable from Nextflow, and stops `copy_dataset`

README.md

@@ -4,7 +4,7 @@ A command-line tool for exploring huge AnnData stores (`.h5ad` and `.zarr`) with
 
 ## Features
 
-- Streaming access to very large `.h5ad` and `.zarr` stores
+- Streaming access to very large `.h5ad` and `.zarr` stores — see [the benchmarks](docs/BENCHMARKS.md) for what that costs in practice, including where loading the file outright is faster
 - Auto-detects `.h5ad` files vs `.zarr` directories
 - Chunked processing for dense and sparse matrices (CSR/CSC)
 - Reads every AnnData on-disk layout, from 0.7.x through the current spec, and always writes the current one
@@ -56,6 +56,7 @@ Run help at any level (e.g. `adata --help`, `adata export --help`).
 - `subset` – stream and write a filtered copy, selected by obs/var name lists (`--obs`/`--var`) or by expression (`--obs-query`/`--var-query`).
 - `split` – write one store per distinct value of an annotation column, with a CSV manifest.
 - `concat` – concatenate stores along the obs axis, with `--join inner|outer` and merge strategies for var and uns.
+- `convert` – change a matrix's dtype, layout (CSR/CSC/dense) or density, streaming; refuses a lossy cast or a large size increase unless forced.
 - `export` – extract data from a store; subcommands: `dataframe` (any dataframe group to CSV), `array` (dense to `.npy`), `sparse` (CSR/CSC to `.mtx`), `dict` (JSON), `image` (PNG). Results go to stdout when no `--output` is given.
 - `import` – write new data into a store at any path; subcommands: `dataframe` (CSV), `array` (`.npy`), `sparse` (`.mtx`), `dict` (JSON), `image` (PNG/JPEG/TIFF).
 
@@ -78,12 +79,26 @@ adata split  data.h5ad --by sample -o per_sample/
 adata concat per_sample/*.h5ad -o merged.h5ad --join outer --label sample
 ```
 
+### Shrinking a store, and making encodings agree
+
+```bash
+adata convert data.h5ad X --inplace --dtype float32
+adata convert data.h5ad --all -o small.h5ad --dtype float32 --indices-dtype int32
+adata convert data.h5ad X -o csc.h5ad --layout csc
+```
+
+Counts written as float64 halve with no loss — and `convert` proves that
+before it writes, by casting every value and casting it back. `concat`
+refuses inputs whose matrices disagree about CSR versus CSC; `--layout` is
+how you make them agree.
+
 ## Documentation
 
 - [Get started](docs/GET_STARTED.md) — a short tutorial
 - [Command reference](docs/COMMANDS.md) — every command and flag
 - [Element spec: HDF5](docs/ELEMENTS_h5ad.md) / [Zarr](docs/ELEMENTS_zarr.md) — the on-disk format, and what this tool does with it
-- [Testing](docs/TESTING.md) — how the suite is organised, and how compatibility is verified against six anndata releases
+- [Testing](docs/TESTING.md) — how the suite is organised, how compatibility is verified against six anndata releases, and the complexity guards that keep cost regressions out
+- [Benchmarks](docs/BENCHMARKS.md) — peak memory and wall time against anndata and scanpy, remeasured on every release
 - [Changelog](CHANGELOG.md)
 
 ## Docker

benchmarks/__init__.py

@@ -0,0 +1,10 @@
+"""Comparative benchmark for adata-cli, against anndata and scanpy.
+
+Report-only, and deliberately separate from `tests/`. The tests measure
+operation counts and gate merges; this measures wall time and peak RSS, runs
+on tags, and never fails a build -- timing on a shared runner is too noisy to
+gate on.
+
+    uv run python -m benchmarks.run --tier smoke
+    uv run python -m benchmarks.report results.json
+"""