Skip to content
1 change: 1 addition & 0 deletions changelog.d/648.changed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
US state and congressional-district regions use the ACS-local Populace dataset again: the rebuilt release `populace-us-2024-buildo-acs-local-766b7c7-20261017T030317Z` (PolicyEngine/microcosm#1263), in which every SPM unit has a classified adult. This ends the temporary national-file filter of #604. The rebuild is registered as a legacy SPM source with the ACS household-population scope rule, so its group-quarters units stay outside SPM measurement, as for the previous release.
4 changes: 1 addition & 3 deletions docs/bundles.md
Original file line number Diff line number Diff line change
Expand Up @@ -147,9 +147,7 @@ python scripts/bundle.py certify-data \
--model-version <policyengine-us-version>
```

**Temporarily**, state and congressional-district regions filter the certified national dataset (`populace_us_2024`), as in policyengine 6.2.1. The certified ACS-local release has no source-backed SPM independence, so canonical SPM refuses thousands of its SPM units (PolicyEngine/microcosm#1263, PolicyEngine/policyengine-sim-api#747). They move back to the ACS-local dataset with a release whose SPM units all have a classified adult.

When they use it, US state and congressional-district regions scope one certified ACS-local
US state and congressional-district regions scope one certified ACS-local
Populace dataset with row filters. Pass its non-default local-area release
manifest through `--regional-manifest-uri`; certification adds the pinned H5
artifact to `data_releases.us.datasets` and writes both regional path templates
Expand Down
20 changes: 3 additions & 17 deletions docs/engineering/runbooks/build-m-us-populace-certification.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,19 +63,6 @@ This one command:
- writes the changelog fragment
`changelog.d/certify-us-$NATIONAL_RELEASE_ID.changed.md`.

**Interim exception (from policyengine 6.3.4; PolicyEngine/policyengine.py#604).**
State and congressional-district regions filter the national dataset until an
ACS-local release has no SPM unit without a classified adult
(PolicyEngine/microcosm#1263). Run the command above unchanged, with
`--regional-manifest-uri`: without it, certification drops the ACS-local
dataset and leaves no state or district template. Then, before Step 3, set
`region_datasets.state` and `region_datasets.congressional_district` in
`data_releases.us` to `{"path_template": "populace_us_2024.h5"}` by hand. In
Step 5, expect those two templates to select the national artifact, not the
local-area one. The data-certification skill
(`docs/engineering/skills/data-certification.md`) says what must hold before
the exception ends.

## Step 3 — regenerate TRACE sidecars

The certify step does not touch TRACE sidecars. Regenerate them so they record
Expand Down Expand Up @@ -103,16 +90,15 @@ PE_UPDATE_SNAPSHOTS=1 pytest tests/test_household_calculator_snapshot.py

## Step 5 — certify and verify the local-area release

The certification command passes the immutable ACS-local
release manifest with `--regional-manifest-uri`. Confirm all of the following:
The certification command passes the immutable ACS-local release manifest with
`--regional-manifest-uri`. Confirm all of the following:

- `default_dataset` remains the national Populace dataset;
- the local-area artifact appears in `data_releases.us.datasets` with its
repository type, immutable revision, and SHA-256;
- `region_datasets.national` selects the national artifact;
- `region_datasets.state` and `region_datasets.congressional_district` select
the shared local-area artifact (the national artifact while the interim
exception in Step 2 holds);
the shared local-area artifact;
- no derived `states/*.h5` or `districts/*.h5` artifacts are runtime inputs.

```bash
Expand Down
21 changes: 18 additions & 3 deletions docs/engineering/runbooks/us-legacy-spm-sources.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,13 @@
# Legacy US SPM sources (backward compatibility)

> **Backward compatibility, not a data contract.** This layer exists so that
> certified US sources published *before* SPM declarations existed can still
> be measured. New certified sources must carry the declarations themselves;
> do not register them here.
> certified US sources that cannot carry an SPM declaration can still be
> measured: sources published *before* the declarations existed, and sources
> that cannot store a declaration because the pinned policyengine-us does not
> define its variable (today `spm_unit_spm_universe_status`, before
> PolicyEngine/policyengine-us#9462). Register only the declarations a source
> lacks. Every other certified source must carry the declarations itself; do
> not register it here.

policyengine-us measures SPM poverty only when a source declares two things:

Expand All @@ -29,6 +33,7 @@ Sources are keyed by certified SHA-256 in `legacy_spm/registry.py`.
| Source | Scope rule | Independence list |
| --- | --- | --- |
| `populace_us_2024_acs_local` Build O (`769756c3…a7ec`) | `acs_household_population` | national release `source_spm_person_independence.csv` (`22b5968d…3ce5`) |
| `populace_us_2024_acs_local` rebuilt for microcosm#1263 (`3ce781c4…8d6c`, the bundle's certified ACS-local file) | `acs_household_population` | none: the source stores `is_spm_independent_minor_role` |

### Scope rule `acs_household_population` (#597)

Expand All @@ -43,6 +48,10 @@ Age is never used. Unknown channels, missing or inconsistent evidence, and SPM
units that mix group quarters with household people all raise.

On the full Build O source, 183,206 SPM units are outside the measurement.
On the microcosm#1263 rebuild, 181,260 are: Build O's group-quarters
units less the 1,946 one-person group-quarters records of children under 15,
which the rebuild leaves out of the population until the model reads scope
natively (PolicyEngine/microcosm#1263).

### Independence list (#598)

Expand Down Expand Up @@ -111,6 +120,12 @@ existing files must be made again.
- **A rebuilt source declares scope and role itself.** Point the bundle at it.
Keep the Build O registry entry only while anything still reads Build O,
then delete it.
- **A rebuilt source can store the role but not the scope.** The pinned
policyengine-us does not define `spm_unit_spm_universe_status`, so the
source cannot store it (Microcosm's stored-inputs gate refuses a variable
the model does not define). Register it with its scope rule and no
independence list, as for the microcosm#1263 rebuild. Remove the entry once
a release of the source carries the scope natively.

## Related

Expand Down
29 changes: 2 additions & 27 deletions docs/engineering/skills/data-certification.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,33 +35,8 @@ python scripts/bundle.py certify-data --country us --data-producer populace \
--model-version "<policyengine-us-version>"
```

**Interim exception (from policyengine 6.3.4; PolicyEngine/policyengine.py#604).**
State and congressional-district regions currently filter the certified
national dataset, not the ACS-local one. The certified ACS-local release has
no source-backed SPM independence, so canonical SPM refuses its SPM units that
have no classified adult, and with them every state-region run
(PolicyEngine/microcosm#1263, PolicyEngine/policyengine-sim-api#747). The 6.3.4
manifest was certified with `--regional-manifest-uri`, as below, and then
edited by hand. Until the exception ends, do the same:

- still pass `--regional-manifest-uri`. Without it, certification rewrites
`data_releases.us` from the national manifest alone, which drops the
ACS-local dataset and leaves no state or district template at all;
- then set `region_datasets.state` and `region_datasets.congressional_district`
to `{"path_template": "populace_us_2024.h5"}` by hand, before regenerating
the TRACE sidecars. `tests/test_release_manifests.py` fails if they name
the ACS-local file, and also if the ACS-local dataset is no longer
certified.

Restore the ACS-local regions only with an ACS-local release in which every
SPM unit has at least one classified adult under the pinned spm-calculator's
rule (zero unclassified units), and after a state-region run through the
public wrapper succeeds on it. Then remove this exception, the temporary test
and the matching notes in `docs/regions.md` and `docs/bundles.md`.

Outside that exception, US state and congressional-district regions are row
filters over one shared, certified ACS-local Populace dataset. Its release
manifest must declare
US state and congressional-district regions are row filters over one shared,
certified ACS-local Populace dataset. Its release manifest must declare
`dataset_role: non_default_local_area`, `is_default: false`, no default
datasets, and exactly one pinned H5 microdata artifact. Certification writes:

Expand Down
13 changes: 4 additions & 9 deletions docs/regions.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,13 +23,9 @@ ca_snap = Aggregate(
ca_snap.run()
```

Each state is a region in the US registry. State regions load the Populace
dataset declared by `region.dataset_path`, then scope its rows by `state_fips`.
They do not require separate state H5 files.

**Temporarily**, state and congressional-district regions filter the certified national dataset (`populace_us_2024`), as in policyengine 6.2.1. The certified ACS-local release has no source-backed SPM independence, so canonical SPM refuses thousands of its SPM units (PolicyEngine/microcosm#1263, PolicyEngine/policyengine-sim-api#747). They move back to the ACS-local dataset with a release whose SPM units all have a classified adult.

For example:
Each state is a region in the US registry. State regions load the ACS-local
Populace dataset declared by `region.dataset_path`, then scope its rows by
`state_fips`. They do not require separate state H5 files:

```python
states = pe.us.model.region_registry.get_by_type("state")
Expand All @@ -56,8 +52,7 @@ for row in impacts.district_results:

`district_geoid` is the SSDD integer (state FIPS × 100 + district number;
at-large districts use `00`). Congressional district regions load the same
file as state regions (for now the national dataset; see above) and scope its rows by
`congressional_district_geoid`.
ACS-local file and scope its rows by `congressional_district_geoid`.

## UK countries

Expand Down
3 changes: 1 addition & 2 deletions docs/release-bundles.md
Original file line number Diff line number Diff line change
Expand Up @@ -207,8 +207,7 @@ python scripts/bundle.py certify-data --country us --data-producer populace \

That produces one US bundle manifest entry containing the Populace national
default dataset and the certified ACS-local regional dataset. State and
congressional-district simulations use one of them with runtime row filters (for now the
national dataset; see [Regions](regions.md)), so
congressional-district simulations use the latter with runtime row filters, so
derived `states/*.h5` or `districts/*.h5` files are not vendored into
`data_releases.us.datasets`.

Expand Down
12 changes: 6 additions & 6 deletions src/policyengine/data/bundle/manifest.json
Original file line number Diff line number Diff line change
Expand Up @@ -175,8 +175,8 @@
"path": "populace_us_2024_acs_local.h5",
"repo_id": "policyengine/populace-us",
"repo_type": "dataset",
"revision": "populace-us-2024-buildo-acs-local-767312d60-20260923T074941Z",
"sha256": "769756c31f3ca646d12c272511744dec04c0e68870c6946dd945fbba65b6a7ec"
"revision": "populace-us-2024-buildo-acs-local-766b7c7-20261017T030317Z",
"sha256": "3ce781c440a7f933b1d952ff4e9d9f65f4698c64b94d2fad165af0c662838d6c"
},
"source_enrichment": {
"path": "releases/populace-us-2024-spm-20260915/source_enrichment.json",
Expand Down Expand Up @@ -220,17 +220,17 @@
"policyengine_version": "6.7.4",
"region_datasets": {
"congressional_district": {
"path_template": "populace_us_2024.h5"
"path_template": "populace_us_2024_acs_local.h5"
},
"national": {
"path_template": "populace_us_2024.h5"
},
"state": {
"path_template": "populace_us_2024.h5"
"path_template": "populace_us_2024_acs_local.h5"
}
},
"regional_release_manifest_uri": "https://huggingface.co/datasets/policyengine/populace-us/resolve/populace-us-2024-buildo-acs-local-767312d60-20260923T074941Z/releases/populace-us-2024-buildo-acs-local-767312d60-20260923T074941Z/release_manifest.json",
"regional_source_manifest_uri": "hf://dataset/policyengine/populace-us@populace-us-2024-buildo-acs-local-767312d60-20260923T074941Z/releases/populace-us-2024-buildo-acs-local-767312d60-20260923T074941Z/release_manifest.json",
"regional_release_manifest_uri": "https://huggingface.co/datasets/policyengine/populace-us/resolve/populace-us-2024-buildo-acs-local-766b7c7-20261017T030317Z/releases/populace-us-2024-buildo-acs-local-766b7c7-20261017T030317Z/release_manifest.json",
"regional_source_manifest_uri": "hf://dataset/policyengine/populace-us@populace-us-2024-buildo-acs-local-766b7c7-20261017T030317Z/releases/populace-us-2024-buildo-acs-local-766b7c7-20261017T030317Z/release_manifest.json",
"release_manifest_uri": "https://huggingface.co/datasets/policyengine/populace-us/resolve/populace-us-2024-spm-20260915/releases/populace-us-2024-spm-20260915/release_manifest.json",
"schema_version": 1,
"source_manifest_uri": "hf://dataset/policyengine/populace-us@populace-us-2024-spm-20260915/releases/populace-us-2024-spm-20260915/release_manifest.json",
Expand Down
4 changes: 2 additions & 2 deletions src/policyengine/data/bundle/uk.trace.tro.jsonld
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@
"@type": "trov:ResearchArtifact",
"schema:name": "policyengine.py bundle manifest for uk",
"trov:mimeType": "application/json",
"trov:sha256": "ba941346a5d4c77426756e38cb1dc6c518985bbc3d05fb9ebfb0c0aeef3d7151"
"trov:sha256": "edff66681e6944cb4dbfd92031b9d4f764a4bf025d4872d4d47afbb32c725b47"
},
{
"@id": "composition/1/artifact/data_release_manifest",
Expand All @@ -102,7 +102,7 @@
"trov:hasFingerprint": {
"@id": "composition/1/fingerprint",
"@type": "trov:CompositionFingerprint",
"trov:sha256": "e52daf7bac16573b5f91057ea25906990ed6689576b143deb4fa67652644df5c"
"trov:sha256": "3ff039a36b4ef8fbd3f97c86945762fd271440393eca770f6af36a4576510df8"
}
},
"trov:hasPerformance": {
Expand Down
4 changes: 2 additions & 2 deletions src/policyengine/data/bundle/us.trace.tro.jsonld
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@
"@type": "trov:ResearchArtifact",
"schema:name": "policyengine.py bundle manifest for us",
"trov:mimeType": "application/json",
"trov:sha256": "ba941346a5d4c77426756e38cb1dc6c518985bbc3d05fb9ebfb0c0aeef3d7151"
"trov:sha256": "edff66681e6944cb4dbfd92031b9d4f764a4bf025d4872d4d47afbb32c725b47"
},
{
"@id": "composition/1/artifact/data_release_manifest",
Expand All @@ -101,7 +101,7 @@
"trov:hasFingerprint": {
"@id": "composition/1/fingerprint",
"@type": "trov:CompositionFingerprint",
"trov:sha256": "e329a374b3ca516be0e7e0273a02b68bad9b4c8e670f88e8c80cd74fceb974b1"
"trov:sha256": "4f134adadf3f0c63b619621cca8a89020ae071733a4cf96e7c43320f8ea90d0e"
}
},
"trov:hasPerformance": {
Expand Down
6 changes: 4 additions & 2 deletions src/policyengine/tax_benefit_models/us/legacy_spm/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,10 @@
# #
# This package DERIVES the declarations from evidence those sources do #
# carry, for the sources registered in ``registry.LEGACY_SPM_SOURCES`` and #
# for nothing else. New certified sources must carry the declarations #
# themselves: do NOT register them here. #
# for nothing else. A new certified source is registered only for a #
# declaration the pinned policyengine-us cannot read (today the scope, #
# before PolicyEngine/policyengine-us#9462); see ``registry``. Otherwise new #
# certified sources must carry the declarations themselves. #
# #
# ``model_adapter`` is the part that must eventually go: it overrides SPM #
# variables only while the installed policyengine-us cannot read the scope #
Expand Down
43 changes: 37 additions & 6 deletions src/policyengine/tax_benefit_models/us/legacy_spm/registry.py
Original file line number Diff line number Diff line change
@@ -1,8 +1,20 @@
"""Certified US sources that predate SPM declarations.

BACKWARD COMPATIBILITY (#597, #598): register a source here only when it was
certified before SPM measurement-scope and independence-role declarations
existed. New certified sources must carry the declarations themselves.
"""Certified US sources that cannot carry their SPM declarations.

BACKWARD COMPATIBILITY (#597, #598). Register a certified source here only
when it cannot carry a declaration itself:

- it was certified before SPM measurement-scope and independence-role
declarations existed (Build O); or
- the pinned policyengine-us does not define that declaration's variable, so
the source cannot store it. Today that is ``spm_unit_spm_universe_status``,
which policyengine-us defines only from PolicyEngine/policyengine-us#9462;
Microcosm's stored-inputs gate refuses a variable the model does not define.

Register only the declarations the source lacks: a source that stores
``is_spm_independent_minor_role`` takes no independence list. Delete an entry
once nothing reads that source, or once a release of it carries the
declaration natively. Every other certified source must carry the
declarations itself.
"""

from __future__ import annotations
Expand Down Expand Up @@ -63,7 +75,26 @@ class LegacySPMSource(BaseModel):
),
)

LEGACY_SPM_SOURCES: tuple[LegacySPMSource, ...] = (POPULACE_US_2024_ACS_LOCAL_BUILD_O,)
#: ``populace_us_2024_acs_local`` rebuilt for PolicyEngine/microcosm#1263. It
#: stores the SPM independence role for every person, so it takes no list. It
#: cannot store the measurement scope: policyengine-us 2.2.1 does not define
#: ``spm_unit_spm_universe_status``. Its ACS group-quarters units are outside
#: the measurement under Census's ACS household-population rule, as Build O's
#: are. Remove the entry once a release carries the scope natively
#: (PolicyEngine/policyengine-us#9462).
POPULACE_US_2024_ACS_LOCAL_MICROCOSM_1263 = LegacySPMSource(
name=(
"populace_us_2024_acs_local@"
"populace-us-2024-buildo-acs-local-766b7c7-20261017T030317Z"
),
source_sha256="3ce781c440a7f933b1d952ff4e9d9f65f4698c64b94d2fad165af0c662838d6c",
spm_universe_rule="acs_household_population",
)

LEGACY_SPM_SOURCES: tuple[LegacySPMSource, ...] = (
POPULACE_US_2024_ACS_LOCAL_BUILD_O,
POPULACE_US_2024_ACS_LOCAL_MICROCOSM_1263,
)


def legacy_spm_source(source_sha256: Optional[str]) -> Optional[LegacySPMSource]:
Expand Down
Loading
Loading