diff --git a/changelog.d/648.changed.md b/changelog.d/648.changed.md new file mode 100644 index 000000000..6f2cbbc05 --- /dev/null +++ b/changelog.d/648.changed.md @@ -0,0 +1 @@ +US state and congressional-district regions use the ACS-local Populace dataset again: the rebuilt release `populace-us-2024-buildo-acs-local-766b7c7-20261017T030317Z` (PolicyEngine/microcosm#1263), in which every SPM unit has a classified adult. This ends the temporary national-file filter of #604. The rebuild is registered as a legacy SPM source with the ACS household-population scope rule, so its group-quarters units stay outside SPM measurement, as for the previous release. diff --git a/docs/bundles.md b/docs/bundles.md index 39e5da6e4..dc06c6cad 100644 --- a/docs/bundles.md +++ b/docs/bundles.md @@ -147,9 +147,7 @@ python scripts/bundle.py certify-data \ --model-version ``` -**Temporarily**, state and congressional-district regions filter the certified national dataset (`populace_us_2024`), as in policyengine 6.2.1. The certified ACS-local release has no source-backed SPM independence, so canonical SPM refuses thousands of its SPM units (PolicyEngine/microcosm#1263, PolicyEngine/policyengine-sim-api#747). They move back to the ACS-local dataset with a release whose SPM units all have a classified adult. - -When they use it, US state and congressional-district regions scope one certified ACS-local +US state and congressional-district regions scope one certified ACS-local Populace dataset with row filters. Pass its non-default local-area release manifest through `--regional-manifest-uri`; certification adds the pinned H5 artifact to `data_releases.us.datasets` and writes both regional path templates diff --git a/docs/engineering/runbooks/build-m-us-populace-certification.md b/docs/engineering/runbooks/build-m-us-populace-certification.md index 7b8f8fe4c..c99190436 100644 --- a/docs/engineering/runbooks/build-m-us-populace-certification.md +++ b/docs/engineering/runbooks/build-m-us-populace-certification.md @@ -63,19 +63,6 @@ This one command: - writes the changelog fragment `changelog.d/certify-us-$NATIONAL_RELEASE_ID.changed.md`. -**Interim exception (from policyengine 6.3.4; PolicyEngine/policyengine.py#604).** -State and congressional-district regions filter the national dataset until an -ACS-local release has no SPM unit without a classified adult -(PolicyEngine/microcosm#1263). Run the command above unchanged, with -`--regional-manifest-uri`: without it, certification drops the ACS-local -dataset and leaves no state or district template. Then, before Step 3, set -`region_datasets.state` and `region_datasets.congressional_district` in -`data_releases.us` to `{"path_template": "populace_us_2024.h5"}` by hand. In -Step 5, expect those two templates to select the national artifact, not the -local-area one. The data-certification skill -(`docs/engineering/skills/data-certification.md`) says what must hold before -the exception ends. - ## Step 3 — regenerate TRACE sidecars The certify step does not touch TRACE sidecars. Regenerate them so they record @@ -103,16 +90,15 @@ PE_UPDATE_SNAPSHOTS=1 pytest tests/test_household_calculator_snapshot.py ## Step 5 — certify and verify the local-area release -The certification command passes the immutable ACS-local -release manifest with `--regional-manifest-uri`. Confirm all of the following: +The certification command passes the immutable ACS-local release manifest with +`--regional-manifest-uri`. Confirm all of the following: - `default_dataset` remains the national Populace dataset; - the local-area artifact appears in `data_releases.us.datasets` with its repository type, immutable revision, and SHA-256; - `region_datasets.national` selects the national artifact; - `region_datasets.state` and `region_datasets.congressional_district` select - the shared local-area artifact (the national artifact while the interim - exception in Step 2 holds); + the shared local-area artifact; - no derived `states/*.h5` or `districts/*.h5` artifacts are runtime inputs. ```bash diff --git a/docs/engineering/runbooks/us-legacy-spm-sources.md b/docs/engineering/runbooks/us-legacy-spm-sources.md index 20f90d844..583b9c420 100644 --- a/docs/engineering/runbooks/us-legacy-spm-sources.md +++ b/docs/engineering/runbooks/us-legacy-spm-sources.md @@ -1,9 +1,13 @@ # Legacy US SPM sources (backward compatibility) > **Backward compatibility, not a data contract.** This layer exists so that -> certified US sources published *before* SPM declarations existed can still -> be measured. New certified sources must carry the declarations themselves; -> do not register them here. +> certified US sources that cannot carry an SPM declaration can still be +> measured: sources published *before* the declarations existed, and sources +> that cannot store a declaration because the pinned policyengine-us does not +> define its variable (today `spm_unit_spm_universe_status`, before +> PolicyEngine/policyengine-us#9462). Register only the declarations a source +> lacks. Every other certified source must carry the declarations itself; do +> not register it here. policyengine-us measures SPM poverty only when a source declares two things: @@ -29,6 +33,7 @@ Sources are keyed by certified SHA-256 in `legacy_spm/registry.py`. | Source | Scope rule | Independence list | | --- | --- | --- | | `populace_us_2024_acs_local` Build O (`769756c3…a7ec`) | `acs_household_population` | national release `source_spm_person_independence.csv` (`22b5968d…3ce5`) | +| `populace_us_2024_acs_local` rebuilt for microcosm#1263 (`3ce781c4…8d6c`, the bundle's certified ACS-local file) | `acs_household_population` | none: the source stores `is_spm_independent_minor_role` | ### Scope rule `acs_household_population` (#597) @@ -43,6 +48,10 @@ Age is never used. Unknown channels, missing or inconsistent evidence, and SPM units that mix group quarters with household people all raise. On the full Build O source, 183,206 SPM units are outside the measurement. +On the microcosm#1263 rebuild, 181,260 are: Build O's group-quarters +units less the 1,946 one-person group-quarters records of children under 15, +which the rebuild leaves out of the population until the model reads scope +natively (PolicyEngine/microcosm#1263). ### Independence list (#598) @@ -111,6 +120,12 @@ existing files must be made again. - **A rebuilt source declares scope and role itself.** Point the bundle at it. Keep the Build O registry entry only while anything still reads Build O, then delete it. +- **A rebuilt source can store the role but not the scope.** The pinned + policyengine-us does not define `spm_unit_spm_universe_status`, so the + source cannot store it (Microcosm's stored-inputs gate refuses a variable + the model does not define). Register it with its scope rule and no + independence list, as for the microcosm#1263 rebuild. Remove the entry once + a release of the source carries the scope natively. ## Related diff --git a/docs/engineering/skills/data-certification.md b/docs/engineering/skills/data-certification.md index 94211889c..5f1d9abe0 100644 --- a/docs/engineering/skills/data-certification.md +++ b/docs/engineering/skills/data-certification.md @@ -35,33 +35,8 @@ python scripts/bundle.py certify-data --country us --data-producer populace \ --model-version "" ``` -**Interim exception (from policyengine 6.3.4; PolicyEngine/policyengine.py#604).** -State and congressional-district regions currently filter the certified -national dataset, not the ACS-local one. The certified ACS-local release has -no source-backed SPM independence, so canonical SPM refuses its SPM units that -have no classified adult, and with them every state-region run -(PolicyEngine/microcosm#1263, PolicyEngine/policyengine-sim-api#747). The 6.3.4 -manifest was certified with `--regional-manifest-uri`, as below, and then -edited by hand. Until the exception ends, do the same: - -- still pass `--regional-manifest-uri`. Without it, certification rewrites - `data_releases.us` from the national manifest alone, which drops the - ACS-local dataset and leaves no state or district template at all; -- then set `region_datasets.state` and `region_datasets.congressional_district` - to `{"path_template": "populace_us_2024.h5"}` by hand, before regenerating - the TRACE sidecars. `tests/test_release_manifests.py` fails if they name - the ACS-local file, and also if the ACS-local dataset is no longer - certified. - -Restore the ACS-local regions only with an ACS-local release in which every -SPM unit has at least one classified adult under the pinned spm-calculator's -rule (zero unclassified units), and after a state-region run through the -public wrapper succeeds on it. Then remove this exception, the temporary test -and the matching notes in `docs/regions.md` and `docs/bundles.md`. - -Outside that exception, US state and congressional-district regions are row -filters over one shared, certified ACS-local Populace dataset. Its release -manifest must declare +US state and congressional-district regions are row filters over one shared, +certified ACS-local Populace dataset. Its release manifest must declare `dataset_role: non_default_local_area`, `is_default: false`, no default datasets, and exactly one pinned H5 microdata artifact. Certification writes: diff --git a/docs/regions.md b/docs/regions.md index 6770a9780..a6f3e2373 100644 --- a/docs/regions.md +++ b/docs/regions.md @@ -23,13 +23,9 @@ ca_snap = Aggregate( ca_snap.run() ``` -Each state is a region in the US registry. State regions load the Populace -dataset declared by `region.dataset_path`, then scope its rows by `state_fips`. -They do not require separate state H5 files. - -**Temporarily**, state and congressional-district regions filter the certified national dataset (`populace_us_2024`), as in policyengine 6.2.1. The certified ACS-local release has no source-backed SPM independence, so canonical SPM refuses thousands of its SPM units (PolicyEngine/microcosm#1263, PolicyEngine/policyengine-sim-api#747). They move back to the ACS-local dataset with a release whose SPM units all have a classified adult. - -For example: +Each state is a region in the US registry. State regions load the ACS-local +Populace dataset declared by `region.dataset_path`, then scope its rows by +`state_fips`. They do not require separate state H5 files: ```python states = pe.us.model.region_registry.get_by_type("state") @@ -56,8 +52,7 @@ for row in impacts.district_results: `district_geoid` is the SSDD integer (state FIPS × 100 + district number; at-large districts use `00`). Congressional district regions load the same -file as state regions (for now the national dataset; see above) and scope its rows by -`congressional_district_geoid`. +ACS-local file and scope its rows by `congressional_district_geoid`. ## UK countries diff --git a/docs/release-bundles.md b/docs/release-bundles.md index 6bc65c3ad..80b6f8500 100644 --- a/docs/release-bundles.md +++ b/docs/release-bundles.md @@ -207,8 +207,7 @@ python scripts/bundle.py certify-data --country us --data-producer populace \ That produces one US bundle manifest entry containing the Populace national default dataset and the certified ACS-local regional dataset. State and -congressional-district simulations use one of them with runtime row filters (for now the -national dataset; see [Regions](regions.md)), so +congressional-district simulations use the latter with runtime row filters, so derived `states/*.h5` or `districts/*.h5` files are not vendored into `data_releases.us.datasets`. diff --git a/src/policyengine/data/bundle/manifest.json b/src/policyengine/data/bundle/manifest.json index 4ae9379f8..b3b0d07a5 100644 --- a/src/policyengine/data/bundle/manifest.json +++ b/src/policyengine/data/bundle/manifest.json @@ -175,8 +175,8 @@ "path": "populace_us_2024_acs_local.h5", "repo_id": "policyengine/populace-us", "repo_type": "dataset", - "revision": "populace-us-2024-buildo-acs-local-767312d60-20260923T074941Z", - "sha256": "769756c31f3ca646d12c272511744dec04c0e68870c6946dd945fbba65b6a7ec" + "revision": "populace-us-2024-buildo-acs-local-766b7c7-20261017T030317Z", + "sha256": "3ce781c440a7f933b1d952ff4e9d9f65f4698c64b94d2fad165af0c662838d6c" }, "source_enrichment": { "path": "releases/populace-us-2024-spm-20260915/source_enrichment.json", @@ -220,17 +220,17 @@ "policyengine_version": "6.7.4", "region_datasets": { "congressional_district": { - "path_template": "populace_us_2024.h5" + "path_template": "populace_us_2024_acs_local.h5" }, "national": { "path_template": "populace_us_2024.h5" }, "state": { - "path_template": "populace_us_2024.h5" + "path_template": "populace_us_2024_acs_local.h5" } }, - "regional_release_manifest_uri": "https://huggingface.co/datasets/policyengine/populace-us/resolve/populace-us-2024-buildo-acs-local-767312d60-20260923T074941Z/releases/populace-us-2024-buildo-acs-local-767312d60-20260923T074941Z/release_manifest.json", - "regional_source_manifest_uri": "hf://dataset/policyengine/populace-us@populace-us-2024-buildo-acs-local-767312d60-20260923T074941Z/releases/populace-us-2024-buildo-acs-local-767312d60-20260923T074941Z/release_manifest.json", + "regional_release_manifest_uri": "https://huggingface.co/datasets/policyengine/populace-us/resolve/populace-us-2024-buildo-acs-local-766b7c7-20261017T030317Z/releases/populace-us-2024-buildo-acs-local-766b7c7-20261017T030317Z/release_manifest.json", + "regional_source_manifest_uri": "hf://dataset/policyengine/populace-us@populace-us-2024-buildo-acs-local-766b7c7-20261017T030317Z/releases/populace-us-2024-buildo-acs-local-766b7c7-20261017T030317Z/release_manifest.json", "release_manifest_uri": "https://huggingface.co/datasets/policyengine/populace-us/resolve/populace-us-2024-spm-20260915/releases/populace-us-2024-spm-20260915/release_manifest.json", "schema_version": 1, "source_manifest_uri": "hf://dataset/policyengine/populace-us@populace-us-2024-spm-20260915/releases/populace-us-2024-spm-20260915/release_manifest.json", diff --git a/src/policyengine/data/bundle/uk.trace.tro.jsonld b/src/policyengine/data/bundle/uk.trace.tro.jsonld index 319f4ddb1..6c1eb41e1 100644 --- a/src/policyengine/data/bundle/uk.trace.tro.jsonld +++ b/src/policyengine/data/bundle/uk.trace.tro.jsonld @@ -75,7 +75,7 @@ "@type": "trov:ResearchArtifact", "schema:name": "policyengine.py bundle manifest for uk", "trov:mimeType": "application/json", - "trov:sha256": "ba941346a5d4c77426756e38cb1dc6c518985bbc3d05fb9ebfb0c0aeef3d7151" + "trov:sha256": "edff66681e6944cb4dbfd92031b9d4f764a4bf025d4872d4d47afbb32c725b47" }, { "@id": "composition/1/artifact/data_release_manifest", @@ -102,7 +102,7 @@ "trov:hasFingerprint": { "@id": "composition/1/fingerprint", "@type": "trov:CompositionFingerprint", - "trov:sha256": "e52daf7bac16573b5f91057ea25906990ed6689576b143deb4fa67652644df5c" + "trov:sha256": "3ff039a36b4ef8fbd3f97c86945762fd271440393eca770f6af36a4576510df8" } }, "trov:hasPerformance": { diff --git a/src/policyengine/data/bundle/us.trace.tro.jsonld b/src/policyengine/data/bundle/us.trace.tro.jsonld index ff681b554..240fc3fea 100644 --- a/src/policyengine/data/bundle/us.trace.tro.jsonld +++ b/src/policyengine/data/bundle/us.trace.tro.jsonld @@ -74,7 +74,7 @@ "@type": "trov:ResearchArtifact", "schema:name": "policyengine.py bundle manifest for us", "trov:mimeType": "application/json", - "trov:sha256": "ba941346a5d4c77426756e38cb1dc6c518985bbc3d05fb9ebfb0c0aeef3d7151" + "trov:sha256": "edff66681e6944cb4dbfd92031b9d4f764a4bf025d4872d4d47afbb32c725b47" }, { "@id": "composition/1/artifact/data_release_manifest", @@ -101,7 +101,7 @@ "trov:hasFingerprint": { "@id": "composition/1/fingerprint", "@type": "trov:CompositionFingerprint", - "trov:sha256": "e329a374b3ca516be0e7e0273a02b68bad9b4c8e670f88e8c80cd74fceb974b1" + "trov:sha256": "4f134adadf3f0c63b619621cca8a89020ae071733a4cf96e7c43320f8ea90d0e" } }, "trov:hasPerformance": { diff --git a/src/policyengine/tax_benefit_models/us/legacy_spm/__init__.py b/src/policyengine/tax_benefit_models/us/legacy_spm/__init__.py index 26905ff3e..13ca85c05 100644 --- a/src/policyengine/tax_benefit_models/us/legacy_spm/__init__.py +++ b/src/policyengine/tax_benefit_models/us/legacy_spm/__init__.py @@ -11,8 +11,10 @@ # # # This package DERIVES the declarations from evidence those sources do # # carry, for the sources registered in ``registry.LEGACY_SPM_SOURCES`` and # -# for nothing else. New certified sources must carry the declarations # -# themselves: do NOT register them here. # +# for nothing else. A new certified source is registered only for a # +# declaration the pinned policyengine-us cannot read (today the scope, # +# before PolicyEngine/policyengine-us#9462); see ``registry``. Otherwise new # +# certified sources must carry the declarations themselves. # # # # ``model_adapter`` is the part that must eventually go: it overrides SPM # # variables only while the installed policyengine-us cannot read the scope # diff --git a/src/policyengine/tax_benefit_models/us/legacy_spm/registry.py b/src/policyengine/tax_benefit_models/us/legacy_spm/registry.py index 2d2b35928..4806066ee 100644 --- a/src/policyengine/tax_benefit_models/us/legacy_spm/registry.py +++ b/src/policyengine/tax_benefit_models/us/legacy_spm/registry.py @@ -1,8 +1,20 @@ -"""Certified US sources that predate SPM declarations. - -BACKWARD COMPATIBILITY (#597, #598): register a source here only when it was -certified before SPM measurement-scope and independence-role declarations -existed. New certified sources must carry the declarations themselves. +"""Certified US sources that cannot carry their SPM declarations. + +BACKWARD COMPATIBILITY (#597, #598). Register a certified source here only +when it cannot carry a declaration itself: + +- it was certified before SPM measurement-scope and independence-role + declarations existed (Build O); or +- the pinned policyengine-us does not define that declaration's variable, so + the source cannot store it. Today that is ``spm_unit_spm_universe_status``, + which policyengine-us defines only from PolicyEngine/policyengine-us#9462; + Microcosm's stored-inputs gate refuses a variable the model does not define. + +Register only the declarations the source lacks: a source that stores +``is_spm_independent_minor_role`` takes no independence list. Delete an entry +once nothing reads that source, or once a release of it carries the +declaration natively. Every other certified source must carry the +declarations itself. """ from __future__ import annotations @@ -63,7 +75,26 @@ class LegacySPMSource(BaseModel): ), ) -LEGACY_SPM_SOURCES: tuple[LegacySPMSource, ...] = (POPULACE_US_2024_ACS_LOCAL_BUILD_O,) +#: ``populace_us_2024_acs_local`` rebuilt for PolicyEngine/microcosm#1263. It +#: stores the SPM independence role for every person, so it takes no list. It +#: cannot store the measurement scope: policyengine-us 2.2.1 does not define +#: ``spm_unit_spm_universe_status``. Its ACS group-quarters units are outside +#: the measurement under Census's ACS household-population rule, as Build O's +#: are. Remove the entry once a release carries the scope natively +#: (PolicyEngine/policyengine-us#9462). +POPULACE_US_2024_ACS_LOCAL_MICROCOSM_1263 = LegacySPMSource( + name=( + "populace_us_2024_acs_local@" + "populace-us-2024-buildo-acs-local-766b7c7-20261017T030317Z" + ), + source_sha256="3ce781c440a7f933b1d952ff4e9d9f65f4698c64b94d2fad165af0c662838d6c", + spm_universe_rule="acs_household_population", +) + +LEGACY_SPM_SOURCES: tuple[LegacySPMSource, ...] = ( + POPULACE_US_2024_ACS_LOCAL_BUILD_O, + POPULACE_US_2024_ACS_LOCAL_MICROCOSM_1263, +) def legacy_spm_source(source_sha256: Optional[str]) -> Optional[LegacySPMSource]: diff --git a/tests/test_release_manifests.py b/tests/test_release_manifests.py index 6c64909c5..132e1599c 100644 --- a/tests/test_release_manifests.py +++ b/tests/test_release_manifests.py @@ -65,11 +65,9 @@ # immutable release. It remains non-default nationally and is selected for # states and congressional districts. US_LOCAL_AREA_DATASET = "populace_us_2024_acs_local" -US_LOCAL_AREA_RELEASE_ID = ( - "populace-us-2024-buildo-acs-local-767312d60-20260923T074941Z" -) +US_LOCAL_AREA_RELEASE_ID = "populace-us-2024-buildo-acs-local-766b7c7-20261017T030317Z" US_LOCAL_AREA_SHA256 = ( - "769756c31f3ca646d12c272511744dec04c0e68870c6946dd945fbba65b6a7ec" + "3ce781c440a7f933b1d952ff4e9d9f65f4698c64b94d2fad165af0c662838d6c" ) US_LOCAL_AREA_DATASET_URI = ( "hf://policyengine/populace-us/populace_us_2024_acs_local.h5" @@ -330,15 +328,12 @@ def test__given_local_area_dataset__then_national_default_is_unchanged(self): assert manifest.region_datasets["national"].path_template == ( "populace_us_2024.h5" ) - # Temporary (microcosm#1263): state and district regions filter the - # national file until the ACS-local release supplies source-backed SPM - # independence; see the test below. assert manifest.region_datasets["state"].path_template == ( - "populace_us_2024.h5" + "populace_us_2024_acs_local.h5" ) assert ( manifest.region_datasets["congressional_district"].path_template - == "populace_us_2024.h5" + == "populace_us_2024_acs_local.h5" ) def test__given_local_area_name__then_resolves_to_its_immutable_tag(self): @@ -365,21 +360,10 @@ def test__given_local_area_dataset__then_registered_with_sha_but_not_default(sel assert US_LOCAL_AREA_DATASET in resolve_default_datasets("us") assert US_LOCAL_AREA_DATASET != manifest.default_dataset - def test__given_us_manifest__then_local_regions_filter_the_national_file(self): - """Temporary revert, until microcosm#1263 lands. - - The ACS-local release (``populace-us-2024-buildo-acs-local-767312d60-…``) - has no ``is_spm_independent_minor_role``, so 5,274 SPM units have no - classified adult and canonical SPM refuses every state-region run - (SPM_COMPOSITION_REQUIRED; policyengine-sim-api#747). State and district - regions filter the national file again, as in policyengine 6.2.1, which - carries the role. Point them back at the ACS-local dataset only with a - release whose SPM units all have a classified adult. - """ - + def test__given_us_manifest__then_local_regions_resolve_to_acs_local(self): assert ( resolve_region_dataset_path("us", "state", state_code="CA") - == US_CERTIFIED_DATASET_URI + == US_LOCAL_AREA_DATASET_URI ) assert ( resolve_region_dataset_path( @@ -387,7 +371,7 @@ def test__given_us_manifest__then_local_regions_filter_the_national_file(self): "congressional_district", district_code="CA-01", ) - == US_CERTIFIED_DATASET_URI + == US_LOCAL_AREA_DATASET_URI ) assert not any( key.startswith(("states/", "districts/")) diff --git a/tests/test_us_legacy_spm.py b/tests/test_us_legacy_spm.py index 26329c022..7654cee5d 100644 --- a/tests/test_us_legacy_spm.py +++ b/tests/test_us_legacy_spm.py @@ -40,6 +40,7 @@ ) from policyengine.tax_benefit_models.us.legacy_spm.registry import ( POPULACE_US_2024_ACS_LOCAL_BUILD_O, + POPULACE_US_2024_ACS_LOCAL_MICROCOSM_1263, ) from policyengine.tax_benefit_models.us.legacy_spm.universe import ( classify_acs_household_population, @@ -47,6 +48,8 @@ ACS = "acs_2024_1yr" BUILD_O = POPULACE_US_2024_ACS_LOCAL_BUILD_O +#: The bundle's certified ACS-local file: scope rule only, roles stored. +REBUILD = POPULACE_US_2024_ACS_LOCAL_MICROCOSM_1263 def _tables(): @@ -88,16 +91,27 @@ def test_registry_keys_are_unique_and_match_the_bundle(): hashes = [source.source_sha256 for source in LEGACY_SPM_SOURCES] assert len(hashes) == len(set(hashes)) manifest = get_release_manifest("us") + # The bundle certifies the microcosm#1263 rebuild. Build O stays + # registered so runs over it still reproduce. assert manifest.datasets["populace_us_2024_acs_local"].sha256 == ( - BUILD_O.source_sha256 + REBUILD.source_sha256 ) pin = BUILD_O.independence_list assert pin is not None assert manifest.datasets[pin.bundle_dataset].sha256 == pin.sha256 +def test_the_rebuild_needs_only_the_scope(): + """It stores the independence role, so it takes no list (registry rule).""" + assert REBUILD.spm_universe_rule == "acs_household_population" + assert REBUILD.independence_list is None + assert REBUILD in LEGACY_SPM_SOURCES + assert BUILD_O in LEGACY_SPM_SOURCES + + def test_only_registered_sources_are_legacy(): assert legacy_spm_source(BUILD_O.source_sha256) == BUILD_O + assert legacy_spm_source(REBUILD.source_sha256) == REBUILD assert legacy_spm_source(None) is None assert legacy_spm_source("0" * 64) is None default = get_release_manifest("us").datasets["populace_us_2024"] @@ -322,6 +336,51 @@ def test_unexpected_list_columns_are_refused(tmp_path): independence_roles(person, path) +# --- The microcosm#1263 rebuild: scope only --------------------------------- + + +def test_the_rebuild_takes_scope_from_record_types_and_keeps_its_own_roles(tmp_path): + from policyengine.tax_benefit_models.us.legacy_spm import derive_declarations + + person, household = _tables() + stored_roles = [True, False, False, False, True, False, True] + person[INDEPENDENCE_ROLE] = stored_roles + declarations = derive_declarations(REBUILD, person, household, data_dir=tmp_path) + # ACS group quarters (TYPEHUGQ 2/3 with RELSHIPP 37/38) are outside; the + # ACS housing unit and the ASEC and PUF units are measured. + assert declarations.universe.to_dict() == { + 10: "INCLUDED", + 20: "OUTSIDE", + 30: "OUTSIDE", + 40: "INCLUDED", + 50: "INCLUDED", + } + # No list: the roles the source stores are the model's input, untouched. + assert declarations.independence_role is None + assert person[INDEPENDENCE_ROLE].tolist() == stored_roles + assert declarations.record.independence_list_people is None + assert declarations.record.outside_spm_units == 2 + assert declarations.record.included_spm_units == 3 + assert declarations.record.profile == expected_profile(REBUILD) + assert not any(tmp_path.iterdir()) # nothing was downloaded + + +def test_a_rebuild_year_file_needs_the_scope_but_no_derived_role(): + from policyengine.tax_benefit_models.us.legacy_spm import check_year_file_frames + + record = LegacySPMRecord( + profile=expected_profile(REBUILD), + outside_spm_units=1, + included_spm_units=1, + ) + person, spm_unit = _declared_frames() + person.drop(columns=[INDEPENDENCE_ROLE], inplace=True) + check_year_file_frames(person, spm_unit, record) + spm_unit.drop(columns=["spm_unit_spm_universe_status"], inplace=True) + with pytest.raises(ValueError, match="spm_unit_spm_universe_status"): + check_year_file_frames(person, spm_unit, record) + + # --- Records: total and exact --------------------------------------------- @@ -458,11 +517,21 @@ def test_a_file_that_is_not_hdf5_is_out_of_date(tmp_path): def test_only_legacy_year_files_must_record_declarations(tmp_path): path = _empty_h5(tmp_path / "year.h5") acs_local = us_datasets._expected_legacy_spm_profile("populace_us_2024_acs_local") - assert acs_local == expected_profile(BUILD_O) + assert acs_local == expected_profile(REBUILD) assert us_datasets._expected_legacy_spm_profile("populace_us_2024") is None assert us_datasets._expected_legacy_spm_profile("hf://other/repo/x.h5") is None assert year_file_problem(path, acs_local) is not None + # A Build O record is stale for the rebuild the bundle now certifies. write_record(path, _record()) + assert year_file_problem(path, acs_local) is not None + write_record( + path, + _record( + name=REBUILD.name, + source_sha256=REBUILD.source_sha256, + independence_list=None, + ), + ) assert year_file_problem(path, acs_local) is None assert year_file_problem(path, None) is not None @@ -472,7 +541,7 @@ def test_a_managed_dataset_is_recognised_by_its_uri_too(): uri = resolve_dataset_reference("us", "populace_us_2024_acs_local") assert uri.startswith("hf://") - assert us_datasets._expected_legacy_spm_profile(uri) == expected_profile(BUILD_O) + assert us_datasets._expected_legacy_spm_profile(uri) == expected_profile(REBUILD) default_uri = resolve_dataset_reference("us", "populace_us_2024") assert us_datasets._expected_legacy_spm_profile(default_uri) is None diff --git a/tests/test_us_legacy_spm_integration.py b/tests/test_us_legacy_spm_integration.py index 45bdc8940..22c3b85ca 100644 --- a/tests/test_us_legacy_spm_integration.py +++ b/tests/test_us_legacy_spm_integration.py @@ -70,6 +70,7 @@ ) from policyengine.tax_benefit_models.us.legacy_spm.registry import ( # noqa: E402 POPULACE_US_2024_ACS_LOCAL_BUILD_O, + POPULACE_US_2024_ACS_LOCAL_MICROCOSM_1263, ) from policyengine.tax_benefit_models.us.model import PolicyEngineUSLatest # noqa: E402 @@ -571,9 +572,26 @@ def materialize(country, dataset=None, **kwargs): for module in (us_datasets, us_model, declarations): monkeypatch.setattr(module, "materialize_dataset", materialize) + # The bundle now certifies the microcosm#1263 rebuild. These tests serve + # the synthetic file as Build O, which stays registered, so the bundle's + # ACS-local name and URI resolve to Build O's hash here. + managed = us_datasets._managed_source_sha256 + bundle_sha256 = _bundle_acs_local_sha256() + + def build_o_identity(dataset): + sha256 = managed(dataset) + return BUILD_O.source_sha256 if sha256 == bundle_sha256 else sha256 + + monkeypatch.setattr(us_datasets, "_managed_source_sha256", build_o_identity) return SimpleSources(source=source, requests=requests) +def _bundle_acs_local_sha256() -> str: + from policyengine.provenance.manifest import get_release_manifest + + return get_release_manifest("us").datasets["populace_us_2024_acs_local"].sha256 + + class SimpleSources: def __init__(self, source, requests): self.source = source @@ -770,3 +788,100 @@ def test_other_sources_get_no_declarations(tmp_path): assert RECORD_KEY not in dataset.metadata assert UNIVERSE_VARIABLE not in pd.DataFrame(dataset.data.spm_unit).columns assert INDEPENDENCE_ROLE not in pd.DataFrame(dataset.data.person).columns + + +# --- The microcosm#1263 rebuild: scope rule, roles stored ------------------- + +REBUILD = POPULACE_US_2024_ACS_LOCAL_MICROCOSM_1263 +#: The roles the rebuilt source stores: ACS head, ASEC head and the ASEC teen +#: who heads their own SPM unit. +REBUILD_ROLES = [True, False, False, False, True, True] + + +@pytest.fixture +def rebuild_files(tmp_path, monkeypatch): + """Serve a synthetic source as the bundle's certified ACS-local file.""" + frames = _source_frames() + frames["person"][INDEPENDENCE_ROLE] = REBUILD_ROLES + path = tmp_path / "populace_us_2024_acs_local.h5" + with pd.HDFStore(path, mode="w") as store: + for entity, frame in frames.items(): + store.put(entity, frame, format="table", data_columns=True) + store.put("_time_period", pd.Series([2024]), format="table") + requests = [] + + def materialize(country, dataset=None, **kwargs): + requests.append(dataset) + return _bundle_source(path, REBUILD.source_sha256) + + for module in (us_datasets, us_model, declarations): + monkeypatch.setattr(module, "materialize_dataset", materialize) + return SimpleSources(source=str(path), requests=requests) + + +def test_the_bundle_certifies_the_rebuild(): + assert _bundle_acs_local_sha256() == REBUILD.source_sha256 + + +def test_the_rebuilds_year_files_carry_its_scope_and_keep_its_roles( + tmp_path, rebuild_files +): + from policyengine.provenance.manifest import resolve_dataset_reference + + data_folder = str(tmp_path / "data") + created = create_datasets( + datasets=["populace_us_2024_acs_local"], years=[YEAR], data_folder=data_folder + ) + # No independence list is fetched: the source stores the roles. + assert rebuild_files.requests == ["populace_us_2024_acs_local"] + (dataset,) = created.values() + spm_unit = pd.DataFrame(dataset.data.spm_unit).set_index("spm_unit_id") + person = pd.DataFrame(dataset.data.person).set_index("person_id") + assert spm_unit[UNIVERSE_VARIABLE].tolist() == [ + "INCLUDED", + "OUTSIDE", + "OUTSIDE", + "INCLUDED", + "INCLUDED", + ] + assert person[INDEPENDENCE_ROLE].tolist() == REBUILD_ROLES + assert LegacySPMRecord.model_validate( + dataset.metadata[RECORD_KEY] + ) == LegacySPMRecord( + profile=expected_profile(REBUILD), + outside_spm_units=2, + included_spm_units=3, + ) + + # The region registry names the file by URI. + uri = resolve_dataset_reference("us", "populace_us_2024_acs_local") + (loaded,) = load_datasets( + datasets=[uri], years=[YEAR], data_folder=data_folder + ).values() + simulation = Simulation( + dataset=loaded, + tax_benefit_model_version=pe.us.model, + extra_variables=SPM_EXTRAS, + ) + simulation.run() + spm_unit = _frame(simulation, "spm_unit") + assert spm_unit.loc[OUTSIDE_UNITS, "spm_unit_spm_threshold"].isna().all() + assert spm_unit.loc[OUTSIDE_UNITS, "spm_unit_is_in_spm_poverty"].isna().all() + assert spm_unit.loc[MEASURED_UNITS, "spm_unit_spm_threshold"].notna().all() + assert spm_unit.loc[MEASURED_UNITS, "spm_unit_net_income"].notna().all() + assert simulation.output_dataset.metadata[RECORD_KEY]["model_mode"] == "override" + + +def test_a_managed_simulation_of_the_rebuild_masks_group_quarters(rebuild_files): + simulation = pe.us.managed_microsimulation(dataset="populace_us_2024_acs_local") + thresholds = simulation.calculate("spm_unit_spm_threshold", YEAR).values + assert np.isnan(thresholds[[1, 2]]).all() + assert np.isfinite(thresholds[[0, 3, 4]]).all() + in_poverty = simulation.calculate("spm_unit_is_in_spm_poverty", YEAR).values + assert np.isnan(in_poverty[[1, 2]]).all() + recorded = LegacySPMRecord.model_validate( + simulation.policyengine_bundle[RECORD_KEY] + ) + assert recorded.model_mode == "override" + assert recorded.outside_spm_units == 2 + assert recorded.independence_list_people is None diff --git a/tests/test_us_regions.py b/tests/test_us_regions.py index 56dae9f06..54f20cd60 100644 --- a/tests/test_us_regions.py +++ b/tests/test_us_regions.py @@ -11,12 +11,9 @@ ) from policyengine.provenance.manifest import CountryReleaseManifest -# Temporary (microcosm#1263, policyengine-sim-api#747): state and district -# regions filter the certified national file until the ACS-local release -# supplies source-backed SPM independence, so every SPM unit has a classified -# adult. Restore the ACS-local URI here when the bundle points back at it. -US_REGION_DATASET_URI = ( - "hf://policyengine/populace-us/populace_us_2024.h5@populace-us-2024-spm-20260915" +US_LOCAL_AREA_DATASET_URI = ( + "hf://policyengine/populace-us/populace_us_2024_acs_local.h5" + "@populace-us-2024-buildo-acs-local-766b7c7-20261017T030317Z" ) @@ -146,7 +143,7 @@ def test__given_california_region__then_has_correct_format(self): assert ca.label == "California" assert ca.region_type == "state" assert ca.parent_code == "us" - assert ca.dataset_path == US_REGION_DATASET_URI + assert ca.dataset_path == US_LOCAL_AREA_DATASET_URI assert ca.requires_filter assert ca.scoping_strategy is not None assert ca.scoping_strategy.variable_name == "state_fips" @@ -179,7 +176,7 @@ def test__given_ca_first_district__then_has_correct_format(self): assert "1st" in ca01.label.lower() or "1 " in ca01.label assert ca01.region_type == "congressional_district" assert ca01.parent_code == "state/ca" - assert ca01.dataset_path == US_REGION_DATASET_URI + assert ca01.dataset_path == US_LOCAL_AREA_DATASET_URI assert ca01.requires_filter assert ca01.scoping_strategy is not None assert ca01.scoping_strategy.variable_name == "congressional_district_geoid" @@ -286,7 +283,7 @@ def test__given_states_and_districts__then_all_use_local_area_dataset(self): assert len(local_regions) == 51 + 436 assert {region.dataset_path for region in local_regions} == { - US_REGION_DATASET_URI + US_LOCAL_AREA_DATASET_URI } assert all(region.requires_filter for region in local_regions) @@ -296,8 +293,8 @@ def test__given_dc_state_and_district__then_both_use_local_area_dataset(self): assert dc is not None assert dc_al is not None - assert dc.dataset_path == US_REGION_DATASET_URI - assert dc_al.dataset_path == US_REGION_DATASET_URI + assert dc.dataset_path == US_LOCAL_AREA_DATASET_URI + assert dc_al.dataset_path == US_LOCAL_AREA_DATASET_URI assert dc.scoping_strategy is not None assert dc_al.scoping_strategy is not None