Impute UC deductions in constituency and local-authority runs

[MaxGhenis]

Fixes #1902

Reproducer: policyengine_uk/tests/test_data_built_imputations.py::test_region_filtered_from_data_matches_the_national_run; policyengine_uk/tests/test_data_built_imputations.py::test_small_data_built_simulation_gets_hashed_draws

Builds on #1899 (merged), which adds Simulation.built_from_dataset. Current main is merged in.

Summary

Three variables chose between microdata imputation and household defaults by testing whether total weight was below 1e6:

Variable	Old test	Default below 1e6
uc_deduction_random_draw	benefit-unit weight	draw of 1.0, so no UC deductions
uc_deduction_type_random_draw	benefit-unit weight	1.0
attends_private_school	household weight projected to each person (person.household(...)), so in effect people	no private school attendance

policyengine.py builds constituency and local-authority simulations by filtering rows from the national data:

• src/policyengine/countries/uk/regions.py uses RowFilterStrategy on constituency_code_oa / la_code_oa.
• filter_dataset_by_household_ids in src/policyengine/utils/entity_utils.py keeps the rows without rescaling weights.
• run() in src/policyengine/tax_benefit_models/uk/model.py wraps them in UKSingleYearDataset and calls Microsimulation.

So every constituency and local-authority run fell below the threshold and modelled no UC deductions at all.

What changes

• UC deduction draws now gate on built_from_data(simulation) (new, policyengine_uk/utils/data_source.py). It is true for any simulation built from data, however little weight it carries. It reads Simulation.built_from_dataset and falls back to policyengine-core's is_over_dataset, so a core Simulation built over the UK system from data counts too. Set State Pension age from date of birth, including the rise to 67 #1899's months_since_last_birthday now uses the same helper. UC draws hash benunit_id, so a filtered region gets exactly the national run's draws.
• The builders keep the flag current. build_from_situation clears it, and build_from_ids, which every data path goes through, sets it. Previously only __init__ set it, so a simulation rebuilt in place kept a stale value. The builders also drop core's simulation-level caches (_fast_cache, _user_input_keys), so a calculated clone rebuilt in place never reads arrays sized for its old population. clone() and get_branch() copy the instance __dict__, so the flag reaches baseline and branch simulations.
• Private school attendance: household situations never impute it, whatever their weight (main imputed it for a situation with over a million of weight). The dead hasattr(person.simulation, "dataset") check is replaced by built_from_data.
• For data, private school attendance keeps main's size test. Data under a million people of weight still imputes none, so constituency and local-authority runs are unchanged. Imputing there would rank each area's incomes against themselves, which sums to about twice the national run's pupils (measured below). Whether to impute by local rank or wait for a national rank in the data is decision d1435.
• filter_dataset carries each person's attends_private_school into the household it extracts, as Set State Pension age from date of birth, including the rise to 67 #1899 does for months_since_last_birthday. For the extracted year, the extract keeps the attendance it had in the full simulation. A multi-year dataset holding only that year recomputes later years.

Invariants (stated and tested)

policyengine_uk/tests/test_data_built_imputations.py:

1. Construction decides UC draws, not weight. In a data-built simulation, scaling every weight by 2^k for any k in [-14, 30] leaves the draws, uc_has_deduction, uc_deduction_combination and uc_deductions unchanged (Hypothesis). That covers totals from about a hundred to about 10^15. Powers of two scale exactly in floating point, so the property holds exactly.
2. Row-filter invariance for UC deductions (differential test). Take a national-scale data-built simulation (about 1.6m of household weight). Filter one region (about 0.4m, or 0.8m people) the way RowFilterStrategy does. Every benefit unit gets the same draws, deduction flag, combination and amount as in the full simulation.
3. Hashed draws in small data. A data-built simulation with under a million units of weight gets splitmix64_uniform(benunit_id) draws and some deductions.
4. Private school needs a national population. The national-scale simulation imputes some attendance; the filtered region imputes none.
5. Extracts keep their imputations. For the extracted year, a filter_dataset extract reproduces the full simulation's UC deductions and private school attendance. The test asserts that both sets are non-empty.
6. Data without weight. A data-built simulation with every household at zero weight attends no private school and still gets hashed draws.
7. Situations get defaults whatever their weight. A household situation with 1e9 of weight gets draws of 1.0, no deductions and no private school attendance.
8. The flag follows the builder. A calculated clone of a data simulation, rebuilt in place from a situation, gets the household defaults. A calculated clone of a situation, rebuilt from data, gets the imputations, sized for the new population.
9. Core simulations over data. A policyengine-core Simulation(tax_benefit_system=system, dataset=<DataFrame>) gets hashed draws and private school attendance.

Against a git archive of main (db2c36b), 7 of the 9 fail. Two pass on main by design: test 4 pins main's behaviour, and test 9's weight is large enough for main's draws. The first new YAML case (a 1e9-weight situation) also fails on main.

Each fix was also checked by removing it and confirming its test fails:

• The extract test fails without the filter_dataset carry.
• The rebuild test fails with the flag set only in __init__, and again without the cache reset (it reproduces the broadcast error).
• The core test fails without the is_over_dataset fallback.

Impact: constituency and local-authority runs, 2026

Real runs, following policyengine.py's path: filter the national tables by constituency_code_oa / la_code_oa / region, build UKSingleYearDataset + Microsimulation, and calculate 2026.

• Data: enhanced FRS 2024-25, policyengine-uk-data-private 1.58.0, the current policyengine.py bundle.
• Before: main at db2c36b. After: this branch at bc3cd66, which differs from the final head only in private school attendance for data, now back to main's behaviour.
• National run: the unfiltered simulation, grouped by the same geography. It is identical before and after on every metric: UC deductions £1,933m across 3.03m benefit units.
• Coverage: 632 GB constituencies (Northern Ireland records carry no constituency or LA code), 363 GB local authorities and Northern Ireland, each under both versions.

UC deductions, summed over 632 constituency runs

	Before	After	National run
Constituencies with any UC deductions	0	600	600
UC benefit units with deductions (k)	0	2,966	2,966
UC deductions (£m)	0	1,891	1,891
UC paid (£m)	83,250	81,359	81,662

UC deductions, summed over 363 local-authority runs

	Before	After	National run
LAs with any UC deductions	0	348	348
UC benefit units with deductions (k)	0	2,965	2,966
UC deductions (£m)	0	1,891	1,891

• Household net income falls by £1,890.2m summed over constituencies, approximately equal to the £1,890.8m of deductions.
• UC deductions match the national run to within float32 rounding (relative gap ≤ 1e-6) in 624 of 632 constituencies and 357 of 363 LAs.
• The rest differ because UC entitlement itself differs between the filtered and national runs, not because of the draws. UC paid differs by more than 0.1% in 219 constituencies and 138 LAs. One source is Set State Pension age from date of birth, including the rise to 67 #1899's months_since_last_birthday, which spreads birthdays within the simulated population and so can move State Pension age status near the boundary. I did not trace every case.
• Northern Ireland as a filtered region is unchanged: in 2026 it carries 1,000,496 of benefit-unit weight, just above the old threshold.

Private school attendance (decision d1435). The same runs, with attendance imputed in every data-built simulation as first proposed:

Pupils (k)	Main	Imputed by local rank	National run
632 constituencies	0	1,485	751
363 local authorities	24	1,366	751

• Per constituency, the correlation with the national run is 0.54. The ratio to the national run has a median of 1.42 and a 90th percentile of 55.
• On the earlier 1.56.16 data, the same comparison gave 811k against 865k, so the overstatement depends on the data. I have not traced why.
• This PR keeps main's behaviour (the "Main" column) for data.
• Once Charge VAT on private school fees at current law #2319 (VAT on fees at current law) lands, this imputation feeds every constituency's baseline taxes.

Caveats and follow-ups

• Separate issue found while measuring, not changed here. Several incidence variables spread a national total over the simulated population by each household's share of a weighted sum, e.g. shareholding and corporate_land_value, which feed corporate_tax_incidence and business_rates. In a filtered constituency the whole national total lands on the constituency: for one constituency of 87 records, household net income is −£35.6bn in the constituency run against +£2.3bn for the same households nationally (1.56.16 data). This affects every constituency run in policyengine.py (follow-up task). The net-income change above is a difference, in which it cancels.
• Core's stale flag. policyengine-core's SimulationBuilder.build_from_dict(..., simulation=existing) leaves is_over_dataset set when it overwrites a data simulation with a situation, so the fallback treats that core-only path as data. No caller does this, and policyengine_uk.Simulation's own rebuild is correct and tested. Replace a simulation's population when the builder writes a situation into it policyengine-core#611 fixes it in core. A follow-up policyengine-uk PR after this one will raise the core floor, drop the caveat in data_source.py, and test that path.
• Ordinary household situations (well under a million of weight) are unchanged: they got the defaults before and still do.

Tests run

• test_data_built_imputations.py, test_uc_deductions.py, test_state_pension_age.py, test_uc_rebalancing.py, test_dataframe_dataset.py (-n 4): 146 passed.
• tests/policy/baseline/contrib/labour/ YAML: 5 passed.
• ruff format / ruff check clean.
• Reviews, all in ~/reviews/policyengine-uk-1905-constituency-imputations/:
  • Opus 5.5 on 40aaab0: approved with nits, all addressed.
  • GPT-6.1 Sol on 2fad4c0: requested changes (3 findings), fixed in 77141d9.
  • Sol on 77141d9: requested changes (clone crash, body overstatement), fixed in e951934.
  • Sol on e951934: requested changes on the core stale-flag path; answered in review4_response.md: the fix belongs in core (now policyengine-core#611), and dropping the fallback would bring back the earlier finding.
  • Sol on de97e0c: requested changes (reproducer line, extract-year wording, "equal" to "approximately equal"). Body fixed; docstring fixed in 475c705. It found the core limitation adequately scoped for this PR, and the narrowed gate passed its threshold, situation, zero-weight and extract checks.
  • Sol on the current head: pending.

axiom: n/a: microsimulation imputation

🤖 Generated with Claude Code

[MaxGhenis]

Hand-off to the UK hub at head 475c70525.

• Review: GPT-6.1 Sol APPROVE at 475c705, no findings (~/reviews/policyengine-uk-1905-constituency-imputations/review6_out.md). Earlier rounds (Opus, then Sol ×4) are in the same folder, each answered or fixed.
• Passed locally: 146 related tests (-n 4), labour YAML, ruff, and the fix-evidence checker on this body.
• Pending: CI on 475c705 (Actions is slow today). The PR is marked ready.
• Depends on: nothing open; Set State Pension age from date of birth, including the rise to 67 #1899 is merged and main (db2c36b) is merged in.
• Not a hold: decision d1435 asks whether constituency and local-authority runs should impute private school attendance by local income rank. This PR keeps main's behaviour there, so it can land on gates whatever the ruling.
• Follow-ups owed by others:
  • Replace a simulation's population when the builder writes a situation into it policyengine-core#611 fixes core's stale is_over_dataset. Its session will open the policyengine-uk follow-up that drops the data_source.py caveat.
  • Tasks are filed for national-total incidence in filtered regions (corporate_tax_incidence / business_rates), for ranking private school attendance by national income, and for subsample() on UK multi-year datasets.
• Evidence: ~/reviews/policyengine-uk-1905-constituency-imputations/ (the 1.58.0 run is in run-2026-10-12/).

Merge method: merge commit.

[MaxGhenis]

Merged by the UK hub (audit). Merged with a merge commit at the tested head 475c70525 (--match-head-commit, through the main merge queue). Gates:

• CI: gh pr checks passed at the head.
• MERGEABLE, not a draft. No outstanding change request: earlier rounds (Opus, then Sol ×4) were each answered or fixed.
• Independent review: GPT-6.1 Sol round 6 APPROVE at 475c70525, no findings (~/reviews/policyengine-uk-1905-constituency-imputations/review6_out.md).
• Real-run impact: constituency and local-authority runs for 2026, following policyengine.py's row-filter path on uk-data 1.58.0. Figures are in the PR body; evidence is in run-2026-10-12/. Before this, every constituency and LA run fell below the size threshold and modelled no UC deductions. Private school attendance keeps main's size test, so those figures are unchanged.
• Not a hold: d1435 (whether to rank private school attendance by local income) is independent; this PR keeps main's behaviour there.
• Follow-ups owed by others: Replace a simulation's population when the builder writes a situation into it policyengine-core#611 (stale is_over_dataset), then a policyengine-uk PR dropping the data_source.py caveat. Tasks are filed for regional incidence, national-rank private schooling and multi-year subsample().