[unsupervised AI] Prototype bounded pre-submission permits

[YusefSyed]

Warning

This PR was written autonomously by an AI agent and has not been reviewed
by a human yet. Maintainers should ignore it until the human author has reviewed,
understood, and approved everything that the AI agent wrote.

Related to #8876 (the client-side submission stage), building on #8877.

A client can spend longer than the scheduler's idle timeout preparing a graph before update_graph reaches the scheduler. This experimental draft implements a finite pre-submission permit and an explicit protected compute/persist path. The scheduler acknowledges the permit before preparation starts. The client prepares the graph locally, rechecks the connection and deadline before sending, and exposes its newly-created Futures only after admission.

The path is opt-in: protected_compute and protected_persist live in a private module, support one collection per operation, and require manual scheduler-extension registration. Ordinary calls retain their existing path. The caller supplies lease duration, network timeout, clock-rate bound and margin; no production defaults or automatic coverage are proposed.

The protocol uses server-issued connection epochs, monotonic submission sequences, globally bounded pending permits, bounded retained outcomes per client, and a compacted deadline heap. Pending permits prevent idle shutdown and transfer into the existing active-update guard without an await gap. Expired, aborted, stale and reused submissions are rejected before graph work. The client releases only its operation-owned Future objects on rejection, preserving other Futures that share a key. Once dispatch is attempted, the outcome may be indeterminate: the client never aborts or resends that graph.

Initial implementation validation on Python 3.12 / macOS arm64 (historical; current follow-up below):

• 90 focused registry, scheduler, sync-client, async-client and legacy regression cases passed. This includes preparation over 400ms against a 50ms idle timeout armed at grant time, cancellation/connection loss before and after dispatch, delayed replies, shared-key cleanup, legacy same-ID cleanup with the extension disabled, and 20,000-operation expiry-index churn.
• 11 additional existing compute/persist/annotation/Future tests passed in focused runs.
• All repository pre-commit hooks passed via pre-commit run --all-files: Ruff 0.15.16 lint/format, codespell, and Mypy 1.20.2. The initial run did not use the Pixi launcher; the September 23 follow-up below uses the repository Pixi environment.
• Earlier resource-aware runs reported a one-FD finding in the process/client fixture and the existing reconnect test. Both reproduced with ordinary calls on unchanged upstream; those runs are not described as fully green.

Local measurements exposed an expiry scan that grew with every connected client. The bounded heap reduced the fixed-clock, 1,000-client status microbenchmark from about 108–132 microseconds to about 1.0–1.1 microseconds. Record-count bounds were checked separately from timing and tracemalloc observations. These are bookkeeping measurements, not network-capacity claims.

Sequential loopback measurements also make the API tradeoff visible: for the tiny graph case, median ordinary/protected API return was 0.43/11.13ms, while completion was 14.05/14.46ms (30 pairs). The protected return includes admission; the ordinary return only queues work. Larger graphs and a 4MiB payload were measured too, without claiming a speedup or a deployment-wide regression guarantee.

This remains a design prototype. The API shape, deployment clock assumptions, defaults and whether an opt-in surface belongs upstream need review. It does not automatically protect unchanged calls, define safe graph retransmission, or replace existing accepted-graph error semantics.

• Tests added / passed (focused cases described above)
• Passes pixi run lint

September 23 validation follow-up

Added focused tests for acquisition ownership and failure boundaries: unsupported duration and reused operations, duplicate admission waiters, capability replacement while waiting for the acquisition lock, non-finite clocks before RPC, missing/duplicate graph capture, and continuing owned-Future cleanup after one release fails. The production protocol is unchanged.

• Canonical Pixi run of the complete permit test set: 100 passed.
• Scoped pre-commit Ruff check/format and mypy, plus git diff --check: passed.
• Local coverage of the three permit implementation modules in the complete permit run increased from 488/508 to 491/508 statements. These local measurements are distinct from hosted patch coverage.
• The preceding head ba7364f1 reached hosted patch coverage of 95.12%, below the configured 100% target. Fresh hosted coverage for this update is pending; no aggregate-pass claim is made.
• On that preceding head, all permit tests passed in the Windows Py312 job that failed on a separate shuffle timeout. The nightly job failed on SciPy sparse deprecation warnings. These observations do not replace current-head CI.

The remaining local misses include defensive cleanup exception paths. No threshold, exclusion or production behavior was changed to make coverage pass. This remains a draft requiring human review.

September 26 validation and CI triage

Current head: 2e49df7b04e70ca3843a9c9f98f2a9ca6cb51f47. The latest commits add tests; the production protocol is unchanged. Recorded validation of all five permit test files is 111 passed, with 615/615 locally measured changed executable statements covered. Hosted coverage is separate and is not claimed here.

The hosted matrix is not fully green. The nightly SciPy sparse deprecation failures also occur on base dc182bda in scheduled upstream runs and are tracked by #9353. One Windows Py311 job passed its tests and then failed while downloading a dependency for post-processing. Other timeout, worker-start and cleanup failures still need platform-specific attribution; being outside the new permit test files does not prove they are unrelated.

A controlled local comparison ran test_retire_workers, test_close_connections and test_stress_scatter_death twice on both the base and PR head, using the same macOS arm64 / Python 3.14 environment and resource-leak checks. Each run reported 3 passed and the same one-FD leak in test_close_connections. This is baseline evidence for that local leak, not a claim that the Linux minimum-dependency or Windows CI failures are resolved. No test threshold, leak suppression or production behavior was changed during this triage.

[github-actions]

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

    40 files  ±    0      40 suites  ±0   14h 38m 56s ⏱️ - 17m 39s
 4 271 tests +  111   4 088 ✅ +  109    178 💤 ±0  4 ❌ +1  1 🔥 +1 
83 179 runs  +2 218  78 934 ✅ +2 217  4 240 💤  - 1  4 ❌ +1  1 🔥 +1 

For more details on these failures and errors, see this check.

Results for commit 2e49df7. ± Comparison against base commit dc182bd.

♻️ This comment has been updated with latest results.