Package UK area lists and certify UK regional releases

[juaristi22]

Part of #553 (the infrastructure half; the certification PR will close it)

The infrastructure half of #553. The UK default stays the enhanced FRS (1.58.1 on main) for now, so manifest.json and both TRO sidecars are unchanged. This PR adds what a later certification PR needs to switch the default to a Microcosm national release. That works with a Microcosm local-area release, or, if the national release is ready first, with the enhanced FRS serving constituency and local-authority runs.

Rebased on main at 4773da7f, after #574, #605, #617 and #649.

Commits

1. Package UK area lists and certify UK regional releases. Everything below except the next two items.
2. Name builds by cut tag in qualification and check a regional release's core. Adapts this PR to Qualify UK model and data candidates automatically before promotion #574; see "Fit with Qualify UK model and data candidates automatically before promotion #574's qualification" below.
3. Say which bundle entries lack regional_release_kind. A docs correction: the US entry routes to ACS-local again since Restore ACS-local US regions on the microcosm#1263 rebuild #649, but was certified before the field existed.

Packaged area lists

• Lists: src/policyengine/data/uk_geography/ ships constituencies_2024.csv and local_authorities_2023.csv. Their sources and hashes are in SOURCES.md.
  • Constituencies: 650 on the 2024 boundaries, including Northern Ireland, copied from policyengine-uk-data.
  • Local authorities: 361 on the April 2023 frame, with ONS names and sim-api's hex layout. This adds Ards and North Down (N09000011), and the four 2023 unitaries get their own hexes instead of a shared (0, 0).
• Regions: constituency and local-authority regions read these lists. On main the opt-in build_uk_region_registry(include_constituencies=True, include_local_authorities=True) returns no areas: the GCS helper it imports no longer exists in policyengine-core 3.33.0, and the loader swallows the ImportError.
• Outputs: compute_uk_constituency_impacts and compute_uk_local_authority_impacts label areas from the packaged lists. Add disclosure control and uncertainty for UK geography outputs #605 deprecated both in favour of compute_uk_geography_disclosure, but they still run. An explicit CSV path still wins. They no longer search local folders or download from GCS, because the local search included the old download cache and would prefer a stale 360-row file.
• Matrix lookups unchanged: the lookup CSVs paired with the legacy weight matrices (local_authorities_2021.csv) are untouched. A matrix indexes areas by that file's row order, and sim-api's fallback depends on it.

UK regional certification (certify-data --regional-manifest-uri)

#552's regional path was US-only. It now covers both UK cases:

• Ship together: a Microcosm non_default_local_area release routes region_datasets.constituency and .local_authority to its H5. This records regional_release_kind: local_area.

• National first: --regional-release-kind area_coded_national takes a certified policyengine-uk-data release, such as the enhanced FRS manifest. It carries:

  • that release's national dataset, with area codes;
  • its weights artifacts: the constituency and local-authority matrices and local_geography_weights.

  All of them keep their own repo and revision, so sim-api's same-release matrix binding still holds. The constituency and LA region types route to the enhanced FRS, and the run records regional_release_kind: area_coded_national.

• Either way:

  • Both area types are routed, which satisfies sim-api's uk_area_regions_routed.
  • The regional release must serve the pinned model, and the pinned core when --core-version is given, by its own build or claims.
  • Every carried artifact is HEAD-checked. Microcosm UK's vendored artifacts are HEAD-checked too, as US Populace's already were.

• Guard: re-certifying a bundle that routes regions, without --regional-manifest-uri, is refused instead of silently dropping the routing. --drop-regional-release removes it on purpose. This applies to the US as well. Since Restore ACS-local US regions on the microcosm#1263 rebuild #649 the US entry routes states and districts to ACS-local again, so it applies to US re-certification today, as the runbook already asks.

Cut tag as build identity

Microcosm keeps build.build_id equal to the line name on every cut and records the cut in build.cut_tag. Certification and the import-time compatibility check both use the cut tag when there is one, through release_build_id. It feeds version, data_build_id and the changelog fragment name, so two cuts of one line no longer look identical. Releases without a cut tag, such as the enhanced FRS, are unchanged.

Fit with #574's qualification (commit 2)

• Build ids: Qualify UK model and data candidates automatically before promotion #574's runtime check for a qualified exact pair compared the bundle's data_build_id with the release manifest's build.build_id. With commit 1 the certificate names a Microcosm cut by its cut tag. Without this commit, a qualified cut would have been refused at import, and another cut of the same line, which shares the line name, would have matched. The runtime check, the UK candidate pair's data build id and the Zenodo description now use release_build_id. A test covers both cases.
• Core: with --core-version, the regional release must also cover the pinned core by its own build or claims. A --qualified-pair-id names the primary release only, so it does not vouch for the regional one.
• Merge note: git's automatic merge had moved Qualify UK model and data candidates automatically before promotion #574's core and qualified-pair logic into the regional helper, where its variables don't exist. validate_release_manifest keeps Qualify UK model and data candidates automatically before promotion #574's logic as merged; it shares only the build-version and claim lookups with the regional check (_model_claims).

UK region routing

• Constituency and local-authority regions take dataset_path=resolve_region_dataset_path("uk", …) and keep their row filters. That is None today, so behaviour is unchanged.
• build_uk_region_registry() adds the area regions by default only when regional_release_kind == "local_area". Explicit True/False still wins.

Universal Credit coverage floor (#491)

scripts/check_uk_uc_floor.py checks two Great Britain measures, each within 10% of its benchmark:

• benefit units with Universal Credit, against DWP's households on UC (6.7m in 2025);
• UC spend, against the OBR's March 2026 EFO Table 4.9 (£79.3bn in 2025-26).

These are the figures policyengine-uk-data calibrates to. Sources are in data/qualification/uk_uc_floor.json. The receipt holds weighted national totals only. On the enhanced FRS 1.58.0 at 2025 it passes: benefit units at 0.925 of DWP's figure and spend at 0.979 of the OBR's.

Checks

• Tests:
  • New: test_uk_area_lists.py, test_uk_uc_floor.py, UK merge and end-to-end certification tests (both modes, the model and core compatibility refusals, the re-certification guard, cut-tag identity, the sim-api predicate), UK routing tests, and runtime cut-tag tests (including a qualified pair on a cut).
  • Updated: three US certification tests now use a temporary bundle instead of the repo's manifest, and the label-lookup tests now expect the packaged lists.
  • Local, after the rebase: the touched and neighbouring modules pass (844). CI runs the full suite.
• Bundle: bundle.py check --include-tros passes, and src/policyengine/data/bundle/ is unchanged.
• Live dry run (14 Oct, scratch copy, nothing committed): the published Microcosm national cut …5c6b3f68 was certified national-first against the enhanced FRS 1.58.0 manifest.
  • It is refused at pe-uk 2.100.0 (the cut's build version) on the enhanced FRS's own claim, as it should be.
  • Merging the live manifests routes both area types to enhanced_frs_2024_25.h5, with the matrices at policyengine-uk-data-private@1.58.0, and names the cut tag as the build.
  • All 12 dataset references and the carried artifacts answer HEAD requests.

Follow-up (certification PR, not this one)

• Certify a Microcosm national cut built on the pinned pe-uk, using one of the two modes above.
• Before it, decide how Qualify UK model and data candidates automatically before promotion #574's automated qualification keeps the routing. uk_qualification.py runs certify-data without --regional-manifest-uri, so once UK regions are routed the guard above refuses every automated run. One option is for the flow to pass the bundle's current regional release. That release would then have to claim each candidate's core.
• Add dataset_overlays.uk entries for the enhanced-FRS names no longer certified.
• Update the 14 tests pinned to the enhanced FRS, plus the docs and examples.
• Attach the Qualify UK model and data candidates automatically before promotion #574 qualification receipt and the UC floor receipt; the floor could also become one of Qualify UK model and data candidates automatically before promotion #574's gates.
• Coordinate with sim-api#728.

🤖 Generated with Claude Code

[anth-volk]

Could we avoid adding cut_tag to .py's schema?

From Microcosm's tools/assemble_uk_release_dir.py, build.cut_tag is the Hugging Face git tag each UK build is published under (<release id>-<YYYYMMDDTHHMMSSZ>-<8 hex chars>). The assembler writes the same string as every artifact's revision. Certification already records it in the bundle: data_package.release_manifest_revision, the @revision in source_manifest_uri, release_manifest_uri and certified_data_artifact.uri, and each dataset's revision. The certified H5's sha256 is recorded too.

So reading cut_tag makes .py track a third producer-side build identifier (build_id, attempt_id, cut_tag). The bundle already pins that value, and the assembler's --cut-tag flag can override it. The underlying problem is that Microcosm sets build.build_id to the fixed release name (UK_NATIONAL_RELEASE_ID) for every build.

Two alternatives:

1. Fix it in Microcosm: write a per-build build.build_id, as Populace US releases already do (there, build_id, the release id and the tag are the same string). .py then needs no change here. UK: certify a Microcosm national dataset as the UK default, with separate national and local-area datasets and a corrected local-authority list #553 listed this as the first option.
2. If Microcosm can't change: take the build id from the revision certification already pins (uri_parts["revision"]) rather than from a producer field.

I'd lean towards 1. Either way, cut_tag and release_build_id() could come out of DataBuildInfo, the runtime compatibility check, uk_qualification.py and zenodo.py.

[anth-volk]

Would it make sense to automate this check and make it more durable? Today someone runs scripts/check_uk_uc_floor.py by hand and attaches the JSON to the certification PR, so nothing stops a UK certification that skips it or fails it. That is the situation #491 describes.

Some options:

• Run it automatically: make it one of the required checks in Qualify UK model and data candidates automatically before promotion #574's automated UK qualification (uk_qualification.py and certify-uk-release.yaml), or in certify-data --country uk. A dataset outside the tolerance would then be refused, instead of relying on a reviewer to notice a missing or failing receipt. The PR description already lists this as a possible follow-up.
• Store the result with the certification: keep the receipt in the qualification ledger or the bundle entry, not only in a PR attachment.
• Pin what it measures: it runs on whichever policyengine-uk is installed; it records that version but doesn't compare it with the bundle pin. It also identifies the dataset by file name or URI, not by sha256. Running it on the pinned model version (or refusing a mismatch) and recording the dataset's sha256 would make the receipt identify exactly what was checked.
• One source for the benchmarks: uk_uc_floor.json hand-copies the DWP and OBR totals that policyengine-uk-data calibrates to, for 2025 and 2026 only. Could they come from the same source as the calibration targets, so they're updated in one place when DWP or the OBR publish new figures?

Since policyengine-uk-data calibrates to these exact totals, the enhanced FRS is expected to pass. The check mainly protects Microcosm builds, so it seems worth making it hard to skip.