Support bounded curation and full-source replay for long recordings

[rossant]

Problem and behavior

Opening a complete chronic sorting in the existing in-memory curation model does not scale to 1000-hour recordings. This PR prepares bounded contiguous observation windows, supports semantic curation of that evidence, and replays committed actions against the complete immutable source into a separate validated output.

phy prepare-large SOURCE_PARAMS OUTPUT validates the source, preserves exact int64 sample/global-row coordinates, generates full-duration counts and exposure-aware summaries, and writes a budgeted evidence package. phy template-gui OUTPUT/params.py activates a separate bounded controller only when its manifest is present. phy apply-curation SOURCE_PARAMS BOUNDED OUTPUT streams supported actions into a new complete sorting.

Implementation

• Shared deterministic survey/rescue windows, explicit acquisition gaps and waveform coverage; no silent spike thinning or unknown per-spike-array copying.
• Full counts/rates and Unit Stability alongside window-relative evidence amplitudes/features and explicitly sampled ISI/correlograms. Window boundaries prevent artificial inter-window intervals/pairs.
• Whole-unit merges, labels, and replayable source-scalar amplitude splits with explicit IDs, guarded mutation APIs, transactional rollback/undo/redo, atomic checkpoint generations, and stale-writer rejection. Predicate descendants display pending full metrics until replay/re-preparation.
• Source identity/revision and bounded asset hashes checked before replay; immutable source files retained; outputs verified before exclusive publication. Bounded loading avoids dense allocations indexed by sparse unit IDs.
• Ordinary datasets retain their default controller/layout/save workflow. Shared hooks have unchanged defaults; sparse-ID handling has regression coverage.

The commit sequence covers the contract, services, GUI, hardening, and documentation within this single PR. The guide is docs/long-recordings.md; measured evidence and acceptance limits are in design/large-recording-validation.md.

Validation

The subsequent window-labeling update passed 19 focused GUI/controller tests, lint, formatting, codespell, and make doc-check; CI is rerunning for that revision.

CI before the window-labeling update, on 94adf852 is fully green: all nine Linux/macOS/Windows × Python 3.10–3.12 test jobs, documentation, codespell, and package build. The Windows 3.11 log reports 760 passed, 1 existing skip. CI run.

• make test-full: 756 passed, 1 existing benchmark skip (496 core + 260 application tests).
• Final geometry/nonidentity waveform additions passed 33 focused source/preparation tests; after the Windows flushing fix, all 99 long-recording tests passed locally.
• make lint, make format-check, make doc-check, and uv build passed. Built wheel contains all long-recording runtime modules.
• Independent streamed replay oracle matched every assignment for synthetic 250k and 2m-spike sources over a 1000-hour clock. Peak process RSS was approximately 204–206 MiB with the measured settings.
• Three-run ordinary-recording controller/provider comparison returned identical dimensions/counts with no observed regression in the small synthetic fixture.
• Delegated architecture and scientific-statistics audit found no remaining severe concrete defect in the reviewed paths.

Screenshots

Actual Qt application captures from a reproducible synthetic eight-hour fixture. These illustrate the UI; they are not measurements of chronic recording quality. Regenerate with tools/capture_long_recording_views.py under Xvfb/software Mesa; the command and captions are in docs/long-recordings.md.

Unit Stability: complete-source rates and finite-value amplitude/depth means, with acquisition gaps in gray and retained windows in cyan.

Windowed evidence: source amplitudes at top left, explicitly labeled as seconds from window start with the recording-time range above the plot; sampled ISI at top right, sampled correlograms below it, and window-relative features along the bottom.

Acceptance limits

Draft pending real chronic-recording and curator acceptance. Synthetic sparse-clock fixtures do not establish billion-spike capacity, Qt responsiveness on every hardware/source layout, or biological identity over 1000 hours. Survey/rescue evidence is not a probability sample, and sampled diagnostics must not be interpreted as full-recording quality metrics. The workflow currently requires a single fixed-rate monotonic sample clock, explicit authoritative exposure, supported canonical arrays, and nonnegative int32-compatible unit IDs; time/global-row coordinates are int64. Producer-derived quality metrics are inherited, not recomputed.