[anth-volk]

Fixes #711
Fixes #792

Same-repository draft PR check

• This PR is a draft.
• This PR is from a branch in PolicyEngine/policyengine-sim-api, not a personal fork.

Summary

• Split each Stage 12 US baseline and reform simulation into 20 deterministic segment calls, while keeping the implementation independent of API v1 runner modules.
• Add versioned dataset and baseline cache construction, storage, image loading, manifest identity, and deployment validation.
• Observe baseline and reform concurrently within one shared wait window. Cancel every unfinished simulation, and record it as failed, when one simulation fails, when the window closes, or when the coordinator itself fails after starting a simulation.
• Preserve structured policy-validation errors, including invalid effective-period details, through simulation and report records.
• Document Stage 12 cache deployment, segmented execution, aggregation, and failure handling.
• Rebased onto main after Release Stage 12 database sessions and keep original failures visible #789. Every Stage 12 database operation in this PR opens its connection through Release Stage 12 database sessions and keep original failures visible #789's connection opener, and every report and worker failure record goes through Release Stage 12 database sessions and keep original failures visible #789's persist_failure_record.
• Review fixes: an error reading or writing a failed simulation's own record no longer replaces its failure, each failure is reported to telemetry exactly once, and the Stage 12 precompute app mounts only the read-only pe-uk-private-hf-read-token Hugging Face secret. See "Fixes after review" below.
• Second review fixes: baseline cache files now have the names .py saves and loads, the v2 manifest stays readable by readers released before the cache, the immutable cache has no forced rebuild, and a failed determinism check cannot be skipped by running again. See "Fixes from the second review" below.
• Merged main after Enable ruff structure and import rules (INP, PLC, SLF, N, A, FAST, E) #799 and the .py 6.7.5 update: twelve conflicts are resolved, and main's ruff rule groups 2 to 7 are applied to this PR's code. See "Merged main after Enable ruff structure and import rules (INP, PLC, SLF, N, A, FAST, E) #799 and the .py 6.7.5 update" below.

Merged main after #799 and the .py 6.7.5 update (28759410)

Main is merged in as a merge commit, as before. Main brought .py 6.7.5 (#818, #820, #822, #826), ruff rule groups 2 to 7 (#773, #777, #779, #783, #785, #799), the Auth0 permission checks on the entrypoint routes (#758), job stage reporting (#781) and the policyengine-observability 3.0.2 lockfiles (#816).

Conflicts

Twelve files conflicted.

• Coordination. This PR's cancellation handling, shared wait window and failure reporting are kept. Main's _require_artifact_digest helper is kept; its merged call sites already use it. Main's _mark_child_failed is dropped, because this PR had already replaced it with _record_child_failure and the cancellation records. The wait-window constant still comes from stage12_timeouts.
• Aggregation. This PR's PrecomputedSimulation stays in the aggregation module, and main's module-level build_spm_comparison_provenance import is kept.
• Everywhere else. The conflicts were main's docstrings, prebuilt exception messages and TYPE_CHECKING imports against this PR's cache, segmentation and cancellation additions. Each kept both: this PR's behaviour, in main's style.

Ruff rules from main (28759410, 2a81eba1)

Main's new rule groups found 372 findings in this PR's code. All are fixed. The only suppressions this PR adds are nine catch-all handlers (BLE001), each with its reason: they record a redacted cause and raise a fixed error from None, or log a failed best-effort write or cancellation without replacing the original failure. Ruff accepts a catch-all only if it re-raises, chains or logs the original error with its message, which these handlers deliberately avoid. Main's coordinator suppresses its own catch-all for the same reason.

• Deferred imports are module-level imports. This covers the precompute app, the image-build download, the cache build, worker validation, qualification, segment fan-out and the v2 app. Where a test patches a module, the code imports the module and calls through it.
• Two import cycles are broken by moving code. Stage12CachedSimulation, qualifying_cache_identity and the cache outcomes move from stage12_cache.runtime to stage12_runtime.cached_simulation. should_segment_simulation moves into stage12_runtime.partition. The simulation module now imports both at module level, and the stage12_cache package no longer imports .py.
• An incomplete stored baseline clears .py's in-memory cache. .py's Simulation.ensure keeps a process-wide in-memory cache of up to 100 simulations. When a stored baseline fails the output plan, Stage12CachedSimulation.ensure recalculates and saves it. It then calls the cache's public clear(), instead of replacing one private entry, so the incomplete output cannot be served from memory; the next ensure loads the saved file. A new test covers this.
• Types. The precompute's Modal handles are annotated modal.Function, which replaces this PR's copy of the RemoteFunction protocol; Stage 12 code must not import the v1 precompute_models. The fan-out's Modal module parameter is a ModuleType. The other Any parameters are typed where a precise type exists.
• Raises. Two raises inside try blocks move into helpers. The image build's two manifest type checks raise TypeError.
• Names. Stage12Cancellation is renamed Stage12CancellationError (N818); it is internal to this PR.
• Logging. A segment cancellation that raises is logged with its error type instead of being ignored silently.

.py 6.7.5

The lockfiles are main's; this PR doesn't change them. The test that runs the real bundle and US model, and checks that the planned baseline file is the one .py saves and loads, passes on 6.7.5. Between 6.4.1 and 6.7.5, Simulation.ensure only gained reruns of saved outputs that .py marks as outdated. The in-memory cache copy has existed since 6.0.0.

Environment variables

This merge adds or removes no environment variable, workflow input, secret or Modal secret attachment. The conflict resolutions and ruff fixes touch only Python modules and tests. The deployed v2 application name follows the .py version, as on main.

Validation

On .py 6.7.5, merged with main at 35857b2e:

• Executor suite: 1,292 passed, 22 skipped, 3 deselected. This includes main's tests that every runtime-evaluated Pydantic model resolves its annotations.
• Simulation entry: 534 passed. Contract: 195 passed. Stage 12 persistence: 19 passed. Observability: 32 passed.
• ruff check . and ruff format --check projects/*/src libs/*/src with the pinned ruff 0.15.22 pass.
• python -m scripts.rules guard --base-ref origin/main and scripts/run_quality_guards.py pass. Shellcheck passes on the changed scripts.
• Not run: new testing, staging or production deployments.

Fixes from the second review (b68e5545, ea82c92f)

Main is merged in first (d007697a, after #765), with ruff's group 1 rules applied to this PR's files.

Baseline cache files now have the names .py uses

• Problem. .py saves and loads a simulation's output as Simulation.storage_id + ".h5". A US simulation always resolves an SPM selection, so storage_id is <id>-spm-<sha256 of the selection>. The cache named baselines <simulation_id>.h5. compute_baseline_impl therefore never found the file it had just calculated ("Stage 12 baseline calculation produced no cache file"), which would fail every Stage 12 deploy, and no worker could have loaded a baked baseline.
• Fix. The baseline identity now holds the resolved SPM selection the worker builds the simulation with: normalize_runtime_spm on the same parameters, which are now built by one shared simulation_runtime_params. Its filename mirrors storage_id. The selection is part of the identity digest, so an explicit selection equal to the defaults shares the file, and a different selection gets its own. Plan entries carry the file name, and the manifest, the determinism check and the image use it.
• Test. test_planned_baseline_file_is_the_one_the_worker_saves_and_loads plans with the real bundle, builds the worker's model with the real US model through build_stage12_simulation, and compares the name from .py's own output_dataset_filepath with the planned name. It fails with the old name.

The v2 manifest stays readable by readers released before the cache

• Problem. V2WorkerVersion and V2WorkerValidation gained cache, and the manifest models forbid unknown fields. Readers already deployed would reject the published manifest, and a cold reader would then have no valid manifest. The manifest is published before the reading revision is promoted, staging deploys do not promote, and a rollback or revert would read the same document.
• Fix. publish_v2_manifest writes the manifest under a new active-with-cache key, and the same manifest without the cache fields under active. The loader and the publisher read active-with-cache, falling back to active only before the first publish that writes both keys. Removing the active copy needs a later change, once no deployed or rollback revision reads it.
• Checked against main's models. Main's loader reads the active copy and rejects active-with-cache, and main's publisher (as after a revert) still publishes on top of it. A contract test pins the exact fields earlier readers accept.

No forced rebuild of the immutable cache

• Problem. The v1 precompute's force_recompute input, which main deploys pass through, also reached the Stage 12 cache step. Forced uploads hit create-only paths, and HDF5 baselines are not byte-reproducible. The store therefore raised "immutable Stage 12 cache path contains other data", failing staging and skipping production.
• Fix. The Stage 12 cache has no force option. The workflow passes only the Modal environment, the script and the app take no --force, and select_work selects only missing files. The v1 precompute still receives force_recompute. To replace a bad file, delete it and deploy again: the run recomputes it and repeats the determinism check.

A failed determinism check cannot be skipped by running again

• Problem. Baselines are uploaded before the check. After a failed check, the next run found every baseline present, skipped the check and published them.
• Fix. A pass is recorded at determinism/us/<digest>.json, keyed by the planned baseline identities. A run checks when it computed baselines or when no pass is recorded, and publish_manifest_impl refuses to publish without the record. The check now also requires the stored baseline to load as a cache hit, so it cannot compare two fresh calculations.

Not changed

Policy effective periods stay strict ISO 8601: a date such as 2027-4-6 is rejected with invalid_policy_period. This is intended.

Environment variables

No variable is added or removed.

• The cache step still reads MODAL_TOKEN_ID and MODAL_TOKEN_SECRET (GitHub Actions secrets) and STAGE12_CACHE_BUCKET (GitHub Actions variable, in the step's env). The script writes to the runner's GITHUB_OUTPUT. Only the positional force argument is gone.
• The determinism record is written by the precompute functions with the same stage12-evaluation-gcp-credentials Modal secret that uploads the baselines, in the same bucket under a new determinism/ prefix.
• The new manifest key is written by the existing publisher step with the same Modal token.
• Not verified in this update: the cache buckets' IAM grants for the precompute service account. That is unchanged from before, but the new prefix relies on the same grants.

Validation

Merged with main at 9cd0ca3c, on .py 6.4.1:

• executor: 1,238 passed, 22 skipped;
• entry: 449 passed;
• persistence: 19 passed;
• contract: 156 passed.

ruff 0.15.22's lint and format checks, shellcheck on the changed script, and the rules-registry and ci-scripts guards also pass.

Fixes after review

A failed simulation's own state error no longer replaces its failure (b6c691d1)

Before this fix, the coordinator read a failed child's record, to choose its failure detail, outside any guard. A reproduction in which one database read of the failed baseline's record fails recorded the report as report_coordination_failed instead of simulation_invocation_failed, recorded the failed baseline itself as simulation_cancelled ("Cancelled after the coordinator failed."), and gave the reform the same wrong reason. A failing write of that record was already caught, but it was logged without the report or simulation identifiers.

• In both the child-wait path and the dispatch-failure path, reading and writing the failed simulation's own record is now non-fatal, like the cancellation writes. The error is logged at ERROR with evaluation_id, simulation_execution_id, the role and the error type, in the message and as record attributes. The error's message is not logged.
• The report keeps the simulation's failure as its code and cause (simulation_invocation_failed or simulation_dispatch_failed, or the typed failure the worker recorded).
• The failed simulation is recorded as failed with its own cause whenever its record can still be written. If the first read fails, the conditional write is still attempted. It is never recorded as simulation_cancelled, and it is not cancelled through Modal.
• Its peers are cancelled with "Cancelled after the simulation failed.".

Each failure is reported once

#789's contract is that each failure is reported once. After the rebase, an unexpected error reached telemetry twice: once through #789's report_original_failure ("Stage 12 execution failed", with the error), and again through this PR's failure_detail_from_exception, which called log_and_redact_exception. That second report was a standard-library ERROR log with the stack trace, runtime.record_exception and a gateway_error_redacted event. It happened in the worker, child-wait, child-dispatch and report-coordination handlers. A segmented simulation's failure was also reported twice: the fan-out logged the segment error, and the worker then reported the wrapping Stage12ExecutionError.

• A new report_failure helper builds the redacted detail without logging it and emits one report_original_failure record. That record carries the same correlation_id as the persisted Simulation failed (correlation_id=...) summary, plus evaluation_id and, for a simulation, simulation_execution_id and the role. Without a runtime, the one standard-library log line includes the correlation ID.
• The worker, the segment fan-out, and the coordinator's child-wait, dispatch and report handlers all use it. The worker does not report a segment failure again, because the fan-out has already reported it.
• failure_detail_from_exception is replaced by redacted_failure_detail, which only describes a failure and never reports it.
• The expected attributes in Release Stage 12 database sessions and keep original failures visible #789's report-ordering tests now include the identifiers and the correlation ID. Each test also checks that the correlation ID matches the persisted summary.

The Stage 12 precompute app mounts only the read-only Hugging Face secret (eef03c3c)

src/modal/stage12_precompute_app.py runs on every Stage 12 deployment. It mounted the retiring policyengine-data-credentials and huggingface-token (required_keys=["HF_TOKEN"]) Modal secrets on its image build and all five functions.

• Its image build (policyengine bundle install ... --country us) and all five functions now mount pe-uk-private-hf-read-token with required_keys=["HUGGING_FACE_TOKEN"]. The secret is declared through the same HF_MODAL_SECRET_NAME and HF_RUNTIME_ENV_NAME constants as app.py and v2_app.py. The functions keep stage12-evaluation-gcp-credentials. The app never read HF_TOKEN itself, and nothing else changes.
• Read scope. Every Hugging Face read the app makes resolves through the US bundle manifest to hf://policyengine/populace-us/.... This covers the bundle install in the image build and any dataset materialization during ensure_datasets or a calculation. PolicyEngine 6.3.4's runtime code names no other Hugging Face repository for the US. policyengine/populace-us is a public, ungated dataset repository. The token's scope covers only the two private UK repositories, and this public repository needs no grant. The same token already downloads these artifacts in the v2 US worker image build. That layer was rebuilt with only this secret in today's 6.3.4 deployment (run 38545891000, staging) and installed populace_us_2024.h5 from policyengine/populace-us. No credential was widened.
• test_deployment_sources_exclude_legacy_hf_secret_names now includes this app. A new declaration test checks the secret on the image build and every function. Both fail on the previous app source.

Environment variables and wiring

Variable	Used by	Provider	Wiring
HUGGING_FACE_TOKEN	Image build (bundle install) and all five functions	GitHub organization secret PE_UK_PRIVATE_HF_READ_TOKEN, copied into Modal secret pe-uk-private-hf-read-token	The prepare job validates the GitHub value, then runs modal secret create pe-uk-private-hf-read-token HUGGING_FACE_TOKEN=... --env=<modal_environment> --force when deploy_existing_stack or deploy_stage12_v2 is true. deploy_stage12_v2 needs: prepare, and its implicit success() condition means the sync has succeeded, in the same Modal environment, before "Build the independent Stage 12 cache" runs. The sync is in the prepare job, not in the same job as the precompute step. The app attaches the secret to the image build and every function.
GOOGLE_APPLICATION_CREDENTIALS_JSON	All five functions (cache bucket access)	Secret Manager secret named by vars.STAGE12_GCP_CREDENTIALS_SECRET_NAME, copied into Modal secret stage12-evaluation-gcp-credentials	Unchanged: stage12-sync-modal-secrets.sh, in the same deploy_stage12_v2 job, runs before the precompute step.
POLICYENGINE_DATA_FOLDER	Functions	Image environment	Unchanged.
STAGE12_CACHE_BUCKET	Local entrypoint on the runner, which passes the bucket to each function	GitHub environment variable vars.STAGE12_CACHE_BUCKET	Unchanged: precompute step env:.
MODAL_TOKEN_ID, MODAL_TOKEN_SECRET	modal secret create and modal run	GitHub secrets	Unchanged: each step's env:.

• Both environments. Staging (release_environment: beta, Modal staging) and production (prod, Modal main) call the reusable workflow with deploy_stage12_v2: true and secrets: inherit, from both simulation-deploy.yml and stage12-deploy.yml.
• Modal secret names. A read-only modal secret list showed pe-uk-private-hf-read-token and stage12-evaluation-gcp-credentials in both staging and main, each used on 2026-10-14. Secret values and the GitHub secret were not inspected.
• Not verified directly. Whether PE_UK_PRIVATE_HF_READ_TOKEN is granted to this repository and is valid. Today's successful deployments, which validate and synchronize it, indicate that it is. The precompute app has not run with the new secret, because no deployment was run.

Rebased onto main after #789

The branch is rebased onto 5484278455e7fd48bb8d874f0e26245895d2730c, the merge of #789. The earlier merge commit from main is gone, and each commit of this PR now builds on #789.

Connection opener

• Stage12PersistenceStore.replace_simulation_if_status, which this PR adds, opens its connection through _transaction, like every other store operation. Only the connect step can be retried; its statements and commit are never retried.
• The coordinator and the simulation worker build their stores with runtime_store(runtime), so a refused connection is retried and each retry is logged with the operation name. The Cloud Run entrypoint keeps the non-retrying default.

Failure recording

• Each failure is reported to telemetry once, through Release Stage 12 database sessions and keep original failures visible #789's report_original_failure, before any database write. This covers worker failures, segment failures, child dispatch failures, child failures seen while observing, and report failures. See "Fixes after review" for how the report now carries the correlation ID.
• Report and worker failure records are written through persist_failure_record, which logs a failed write instead of raising it. A failed simulation's own record is updated by the coordinator through _record_child_failure, which logs a failed read or write with the report and simulation identifiers, like the cancellation records.
• This PR's structured failure details are kept. The records now hold the redacted summary with a correlation ID, or the typed public error, instead of the exception type name. Release Stage 12 database sessions and keep original failures visible #789's tests were adapted to these summaries and to the conditional replace_simulation_if_status writes.
• A child dispatch failure, a child failure and a closed wait window each raise Stage12ExecutionError after being reported where they occur. The report handler therefore reports only other errors, so each failure is reported once.
• A closed wait window is now reported too, with failure scope child_wait_timeout, before the cancellation records and the report record are written. A new test makes those writes fail and checks that the report still reached telemetry first.
• Release Stage 12 database sessions and keep original failures visible #789's _mark_child_failed is folded into this PR's _replace_running_simulation, which the dispatch, child-failure and cancellation paths share. It only changes a running record, which this PR requires.
• Cancellation records keep this PR's own handling. A failed cancellation write is logged with evaluation_id and simulation_execution_id, and an unfinished child is then cancelled through Modal. They do not use persist_failure_record, because the coordinator must know whether the record was written.

Known open issue (fixed after review)

The coordinator read a failed child's record outside the failure-recording guard, so an error on that read replaced the child's failure. This is fixed in b6c691d1; see "Fixes after review". #789's test for this case, test_coordinator_reports_a_child_failure_once_when_its_record_is_unwritable, is no longer marked xfail and passes.

Connection retries and the finishing time

#789 retries a refused connection for up to 60 s. The last attempt can start just before 60 s and then wait up to the 10 s connect timeout, so one store operation can take about 70 s when the pooler refuses throughout. After the wait window closes:

• Wait timeout: at most 5 operations (a read and a conditional write for each unfinished child, then the report record), so about 350 s. This path has no production-result wait, so 1,500 s remain.
• Child failure at the last poll: at most 6 operations, so about 420 s, which also fits.
• Direct report succeeds: 1 operation (2 if that write fails), so 70 to 140 s of the 600 s.
• Automatic (shadow) report succeeds: at most 5 operations outside the 900 s production-result wait, so about 350 s. That leaves about 250 s of the 600 s to read the artifacts, aggregate and write. A complete testing-environment US report took 288 s including every calculation, so the budget still holds, but in this worst case the margin falls from 600 s to about 250 s.

If the coordinator ever exceeded 5,400 s, Modal would stop it and the report would stay running without a failure record. The timeouts are unchanged in this rebase.

Replaces #793 and covers #792

This PR replaces #793 and adopts its practices. Issue #792 asks for one shared wait deadline that fits inside the coordinator's timeout, and for every unfinished worker to be cancelled and recorded as failed. Observing baseline and reform concurrently was always part of this PR.

Wait window and timeouts

This PR follows #793's rule: the shared wait equals the workers' Modal timeout, and the coordinator's timeout is that wait plus 900 s for the production-result wait plus 600 s to finish.

• One shared window from coordinator entry. The window opens when coordinate_report starts. It covers claiming, output planning (including loading the country model), retained-artifact reads, dispatch, and the concurrent wait for both simulations. Setup time therefore shortens the simulations' window, not the time the coordinator needs afterwards.
• Window = longest simulation worker timeout = 3,900 s. This is the US worker's timeout, and it covers the UK worker (3,000 s) and each US region-group call (3,600 s). Before this update, the window was 3,000 s, so a US simulation could never use the 3,900 s Modal allowed it.
• Coordinator timeout = 3,900 + 900 + 600 = 5,400 s (was 5,100 s):
  • 3,900 s: shared wait window;
  • 900 s: production-result wait (PRODUCTION_RESULT_WAIT_TIMEOUT_SECONDS);
  • 600 s: read and validate both simulation artifacts, aggregate, write the aggregate and comparison artifacts, and record the outcome or failure.
• Why 600 s is enough. A complete testing-environment US report took 288.2 s, including all 40 region-group calculations, merging, reads, aggregation and writes. A failed report took 26.8 s. So 600 s is ample, and I kept Share one Stage 12 child wait deadline and cancel unfinished children #793's value. See "Connection retries and the finishing time" above for the worst case with Release Stage 12 database sessions and keep original failures visible #789's retries.
• No overrun. The coordinator's last poll no longer sleeps past the deadline, so the window closes on time rather than up to 1 s late, and the 600 s is not eroded.
• Named constants. A new dependency-free module, stage12_timeouts.py, defines the worker timeouts, the window and the finishing time, and derives the coordinator timeout from them. The Modal app declaration, the coordinator and the comparison step all read it.
• Dependents. I searched for anything that relies on the old 5,100 s or 3,000 s values. The Modal app declaration, its declaration test, the coordinator comment and the docs did, and are updated. The entry service, the persistence library, deployment workflows and scripts, and release validation do not depend on them. I also checked the Public API's Stage 12 code in a local clone. It has no dependency on these timeouts, and its status rules let a simulation record move from running to failed and from failed back to running, so failed cancellation records and retries still fit.
• Environment variables. Only the timeout values of run_single_simulation_us, run_single_simulation_uk and coordinate_report change. Their images, worker_secrets (the GCP, Hugging Face and comparison-runtime Modal secrets) and environment are unchanged. The new constants module ships in the existing policyengine_simulation_executor package source that every Stage 12 image already includes. No CI, Docker or Terraform changes.

Cancellation

• Recorded as failed. Every simulation the coordinator cancels is recorded as failed with simulation_cancelled. Its error_summary gives the reason:

  • "Cancelled after the baseline simulation failed." (or reform);
  • "Cancelled after the 3900-second wait window closed.", where the report itself fails with simulation_wait_timeout;
  • "Cancelled after the coordinator failed."

  The earlier incomplete status and the cancelled_after_peer_failure and cancelled_after_coordinator_failure codes are gone. Nothing else needed them: workers treat any status other than running as a cancellation, and the entry service passes statuses and codes through unchanged.

• Only running records change. A simulation that has already succeeded or failed keeps its own outcome.

• Always stopped in Modal. An unsegmented simulation is always cancelled through Modal, even if its record could not be written. A segmented simulation reads its failed record, cancels its own region-group calls and exits. Modal does not cancel spawned calls recursively, so the coordinator cancels a segmented simulation through Modal only if it could not write the record.

• Startup failures too. Any coordinator failure after the first spawn stops every started, unfinished simulation. Examples are a failed dispatch placeholder write, a failed invocation attach (including the simulation whose attach failed), a failed restore and a retained-artifact error. A call restored from an earlier coordinator attempt is stopped too. Before this update, only a failed spawn did this.

• Errors never mask the original failure. Cancellation and state-update errors, including errors reading or writing the failed simulation's own record, are logged, and the report keeps the original failure.

• Traceable logs. When recording or requesting a cancellation fails, the log includes evaluation_id and the simulation's simulation_execution_id, in the message and as record attributes, alongside the role and error type. The error message itself is not logged. Share one Stage 12 child wait deadline and cancel unfinished children #793 used event-name messages with these fields only as attributes. This PR also puts them in the message, because the observability log handler exports only the message.

Carried over from #793

• The timing rule and its declaration test, which reads the real Modal timeouts through fixtures.fake_modal, and the setup-time and exhausted-window tests.
• simulation_cancelled failed records, the running-only guard, and Modal cancellation even when the state write fails.
• Tests: a cancelled and recorded reform, a finished baseline left alone, a restored reform call cancelled on retry, a sibling that already succeeded or failed left unchanged, every unfinished child cancelled on timeout, and cancellation and state errors that do not mask the failure.
• ImmediateCall.cancel now counts cancellations.
• A changelog line. changelog_entry.yaml keeps Release Stage 12 database sessions and keep original failures visible #789's line and this PR's line.
• Not carried over: Share one Stage 12 child wait deadline and cancel unfinished children #793's tests of sequential call.get timeouts, because this PR polls both simulations without blocking.

Validation of the review fixes

• Full executor suite on Python 3.13, with CI's uv sync --extra test environment: 1149 passed, 22 skipped, 3 deselected, 0 xfailed (was 1134 passed and 1 xfailed).
• Stage 12 executor tests (tests/test_stage12_*.py): 213 passed.
• Simulation entry suite: 381 passed. Stage 12 persistence suite: 19 passed.
• New tests, each run against the previous source and failing there:
  • the failed simulation's own record read or write fails once, in the child-wait path (2 cases) and in the dispatch-failure path (2 cases);
  • each unexpected failure is reported exactly once, with a runtime and without one, in the worker, child-wait, dispatch and report paths (8 cases), plus a segment failure (1 case);
  • the precompute app mounts only the read-only secret (1 test), and the guard now includes this app.
• test_coordinator_reports_a_child_failure_once_when_its_record_is_unwritable no longer has its xfail marker and passes. Its expected report attributes now include the identifiers and the correlation ID.
• Pinned Ruff (0.15.22) format --check and check on the 10 changed Python files: clean. Ruff format --check on every projects/*/src and libs/*/src directory: clean. git diff --check: clean.
• Not run: new testing, staging or production deployments.

Merged main after #791

#791 stores a redacted failure cause: the type, the stage, the simulation role and the message. For every unexpected error, this PR stored a generic message with a correlation ID instead. Main is merged in as a merge commit rather than a rebase, because 16 of this PR's 25 commits touch the three files that conflicted. This PR keeps its own failure model and uses #791's cause within it:

• Unexpected errors: redacted_failure_detail stores the redacted cause, followed by the correlation ID, for example OperationalError in stage12_child_state_create: ... (correlation_id=3f9c...). The cause is shortened, never the ID, to fit the 512-character summary. Typed input and SPM failures, failures a worker already recorded, and cancellations are unchanged.
• Stages and roles: the coordinator and the workers open their spans and operations through Record a redacted Stage 12 failure cause instead of only the exception class #791's tracked_stage, so a failure names its stage. The coordinator waits on both simulations at once, so it records each one's role while checking it.
• Telemetry: the original error is still reported once, with its correlation ID. As on main, the redacted cause is also attached to the failing operation, with the original stack.
• Error codes: the report keeps this PR's codes. For example, a failed child gives the report simulation_invocation_failed, where Record a redacted Stage 12 failure cause instead of only the exception class #791 always used report_coordination_failed.
• Tests:
  • Tests that expected the generic message, or no message at all, now expect the redacted cause.
  • Record a redacted Stage 12 failure cause instead of only the exception class #791's six cause tests are ported with this PR's error codes.
  • A new test checks that a long cause is shortened and keeps its correlation ID.
  • The exactly-once telemetry tests now check for one log of the original error and one recorded redacted cause.
• Docs: the README and the Stage 12 migration notes describe the new summary.
• Validation: executor suite: 1230 passed, 22 skipped, 3 deselected. Simulation entry suite: 449 passed. Stage 12 persistence suite: 19 passed. Quality guards, pinned Ruff format --check on every src directory, and git diff --check all pass.

Rebased onto main after #762, #763 and #795

• Move the deploy pipelines' multi-line steps into tested scripts #763 moved the worker validation step into .github/scripts/stage12-validate-v2-workers.sh, which conflicted with this PR's --cache-manifest-sha256 argument. The script now takes the digest as a second required argument and passes it to validate_v2_worker_release. The workflow passes steps.stage12_cache.outputs.manifest_digest, the same pattern Move the deploy pipelines' multi-line steps into tested scripts #763 uses to pass step outputs to its other scripts.
• The digest comes from the Build the independent Stage 12 cache step (stage12-cache-precompute.sh), which runs earlier in the same deploy_stage12_v2 job and writes manifest_digest to GITHUB_OUTPUT. RUNNER_TEMP and GITHUB_OUTPUT come from the runner. MODAL_TOKEN_ID and MODAL_TOKEN_SECRET come from GitHub Actions secrets, unchanged.
• test_deploy_workflow_scripts.py checks that the workflow passes the digest and that the script forwards it. A new test checks that the script fails, without calling uv, when the digest is missing.
• Simulation entry suite: 449 passed. Full executor suite: 1169 passed, 22 skipped, 3 deselected. Quality guards (rules-registry, ci-scripts), pinned Ruff and bash -n pass.

Rebased onto main after #757 and #787

• changelog_entry.yaml keeps Accept null SPM provenance in Stage 12 aggregate artifacts #787's line and this PR's line.
• test_coordinator_records_bounded_timeout_without_an_aggregate combines Accept null SPM provenance in Stage 12 aggregate artifacts #787's valid aggregate stub with this PR's wait_timeout_seconds=0.
• Accept null SPM provenance in Stage 12 aggregate artifacts #787 made the coordinator tests' artifact double write aggregates through the real Stage12ArtifactStore. This PR's four aggregator stubs that must not run now return _aggregate_payload(...), as Accept null SPM provenance in Stage 12 aggregate artifacts #787 did for main's, so an unexpected run shows up as a written aggregate instead of as a rejection hidden by the failure the test expects.
• Full executor suite on Python 3.13: 1152 passed, 22 skipped, 3 deselected. Quality guards (rules-registry, ci-scripts) pass.

Rebased onto the rules registry (#752)

• Rebased onto main after Add a rules registry that makes every enforced and unenforced rule visible #752. Code and the changelog line are unchanged from the previous head (eef03c3); the only differences are main's changes and the registry entry below.
• The rules-registry guard from Add a rules registry that makes every enforced and unenforced rule visible #752 flagged test_stage12_only_deployment_skips_existing_stack_integration_suite, which this PR adds, as a guard test no rule references. It is now one of the nodes of sim.stage12.deployment-scope. The rule's statement now also says that a Stage 12-only deploy never runs the existing stack's complete integration suite. docs/generated/rules.md and the AGENTS.md Rules block were re-rendered.
• python -m scripts.rules guard --base-ref origin/main, scripts/run_quality_guards.py and the rules tool's own tests pass. tests/test_deployment_assets.py: 113 passed.

Validation after rebasing onto #789

• Executor environment synced with CI's uv sync --extra test, plus --locked to confirm the lockfile is current: PolicyEngine 6.3.4 (policyengine-us 2.2.1, policyengine-uk 2.126.12, policyengine-core 3.33.0).
• Full executor suite on Python 3.13: 1134 passed, 22 skipped, 3 deselected, 1 xfailed (the known open issue above).
• Stage 12 executor tests (tests/test_stage12_*.py): 198 passed, 1 xfailed.
• Simulation entry suite: 381 passed. Stage 12 persistence suite: 19 passed.
• Every rebased commit was checked out in turn and its Stage 12 executor tests and persistence suite were run. They pass from Reconcile segmented Stage 12 execution with current typed provenance onwards. The four commits before it fail one worker-validation test, test_us_validation_normalizes_the_frontend_reform_interval, exactly as they did before this rebase: main's reform-interval normaliser predates them, and that commit updates the test.
• Regression check: removing the closed-window report makes the new timeout test fail.
• Pinned Ruff (0.15.22) format --check and check on the 49 changed Python files: clean. Ruff format --check on every projects/*/src and libs/*/src directory, as CI runs it: clean. git diff --check: clean.
• Not run: new testing, staging or production deployments.

Validation of the cancellation and timeout update (before rebasing onto #789)

• Full executor suite on Python 3.13 with uv run --frozen: 809 passed, 23 skipped, 3 deselected. Before that update it was 786 passed.

• Stage 12 executor tests (tests/test_stage12_*.py): 181 passed.

• tests/test_stage12_runtime.py gained 23 tests in that update, covering startup cancellation, cancellation records, Modal fallback, logs, and timing.

• Regression checks. I reverted each behaviour in turn, and each time the matching new tests failed:

  • removing the Modal fallback after a failed state write;
  • letting the last poll sleep past the deadline;
  • overwriting a sibling that already finished;
  • anchoring the deadline at observation instead of coordinator entry.

  The startup-cancellation, cancellation-log and coordinator-entry tests also failed against the head before that update.

• Simulation entry suite: 162 passed. Stage 12 persistence suite: 14 passed. Their cancellation fixtures now use the failed simulation_cancelled record.

• Scoped Pyright on the coordination, comparison and timeouts modules: 0 errors.

Prior validation before the earlier rebases

• Stage 12 simulation executor tests: 130 passed, with 1 warning.
• Stage 12 simulation entry tests: 81 passed.
• Stage 12 persistence tests: 14 passed.
• Ruff lint checks passed for the changed Python surface.
• Repository-wide Ruff formatting found one pre-existing formatting difference in projects/policyengine-simulation-executor/tests/test_simulation_output_geographic.py, which this branch does not modify.
• Deployed and validated all seven Stage 12 functions in the Modal testing environment, including US and UK worker validation and a cache-hit check for the 20-segment, three-year cache.
• Completed a testing-environment US report with 1 coordinator call, 2 simulation calls, and 40 successful segment calls in 288.2 seconds.
• Completed an invalid-reform report in 26.8 seconds: the reform retained invalid_policy_period, the baseline became incomplete with cancelled_after_peer_failure (the behaviour at that commit; a cancelled baseline is now failed with simulation_cancelled), and the report failed with the original validation summary.

Earlier testing-environment deployment note

A fresh UK dataset download was blocked because the configured Hugging Face credential could not access the private UK dataset. The testing deployment reused the previously validated UK and coordinator images from the same commit and rebuilt the US image layer with the validated Stage 12 cache.

Earlier current-main reconciliation

The executor keeps main's PolicyEngine 6.3.4 requirements and frozen lockfile; it does not restore older country packages.

Conflict resolution preserves main's output-planning instrumentation and UK metadata validation while retaining concurrent baseline/reform monitoring and segmented US calculation.

Segment merging accepts the current typed SPMProvenance contract. Every segment must supply the same complete receipt, including configuration, forecast hash, years and runtime versions; incomplete or inconsistent receipts fail. The merged logical simulation keeps one compact receipt, not a legacy spm_config object or a growing receipt list. Five regressions cover preservation, missing receipts and mismatched scenario, runtime versions and years.

The worker-validation unit test expects the complete reform date interval, matching main's normalizer rather than discarding its end date.

The earlier live results above describe an earlier commit on an older PolicyEngine version, not a fresh 6.3.4 qualification. No deployment has been run on .py 6.3.4 or any later version; the branch is now on 6.7.5.

🤖 Generated with Claude Code