Skip to content

Reduce and rank over a population with no persons - #590

Merged
MaxGhenis merged 4 commits into
masterfrom
fix-empty-group-reductions
Oct 18, 2026
Merged

MaxGhenis merged 4 commits into
masterfrom
fix-empty-group-reductions

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

What this fixes

With no persons in a simulation, GroupPopulation.all, max and min raise:

ValueError: zero-size array to reduction operation maximum which has no identity

reduce fills each group with its neutral element, but it first took numpy.max of the persons' positions to size its loop, and that array is empty. members_position (numpy.max(members_entity_id)) and Population.get_rank (numpy.max(positions)) have the same problem. All three now size their loop as 0 when there are no persons, so it runs zero times. They use a length check rather than numpy.max(..., initial=-1), which would raise for an unsigned index array. get_rank reshapes its value matrix to (count, 0) so it stays two-dimensional. sum and nb_persons already returned 0 for this case.

The independent review found that an empty membership given as [], or built by build_default_simulation(count=0) (np.array(range(0))), is float64. NumPy indexes only with integers, so ranks and projections over no persons still raised. The members_entity_id setter now stores an empty membership as an integer array, and build_default_simulation uses np.arange(count, dtype=np.int64), which keeps IDs integer for zero counts and for Python and NumPy integer counts.

The audit verifier's witness was one household and no persons. With this PR, all returns [True], any [False], max [-inf], min [inf], sum [0], nb_persons [0], and get_rank returns an empty array.

Invariants

For any number of groups (1 to 6) and any assignment of 0 to 30 persons to them (so groups may be empty and there may be no persons at all), given as an integer array or a plain list:

  • sum, nb_persons, any, all, max and min equal a plain-Python oracle, with 0, False, True, -inf and inf for a group with no members;
  • get_rank gives each person the position of their value among their group's values, ties going to the person listed first.
  • A default simulation built with a person count given as int, np.int32, np.int64 or np.uint64 has int64 IDs equal to master's, and project maps group values onto persons correctly.

Tests

  • New test_group_population_empty.py: the witness for every reduction, get_rank with no persons, a plain [] membership, and a default simulation of no one (rank, nth person, project, projector max, all). Also checks default simulation IDs and projections for counts 0, 1, and 4 supplied as int, np.int32, np.int64, and np.uint64.
  • New test_group_population_reductions_property.py: one Hypothesis property, 300 examples, drawing the count integer type and checking default simulation IDs and projection as well as reductions and ranks.

Before the fix, on master's policyengine_core/populations:

test_group_population_empty.py:               2 failed   (ValueError: zero-size array to reduction operation maximum which has no identity)
test_group_population_reductions_property.py: 1 failed   (minimal: one group, no persons)

Round 2 (after the independent review), the new tests on the first head eca6a621:

test_group_population_empty.py:                 2 failed, 2 passed (plain [] and count=0 cases)
test_group_population_reductions_property.py:   1 failed   (minimal: one group, no persons, given as a list)

After the unsigned-count review, the extended tests on the unchanged head 8d662ec7e8729ae640ec270ca06ee2f5c81352fd:

test_group_population_empty.py:                3 failed, 13 passed (np.uint64 counts 0, 1, 4 have float64 IDs)
test_group_population_reductions_property.py:   1 failed (minimal: count=np.uint64(1), no members)

After this review fix (Python 3.13.9, locked NumPy 2.4.2):

test_group_population_empty.py:                16 passed
test_group_population_reductions_property.py:   1 passed (300 examples)
test_simulation_builder.py:                    35 passed, 7 warnings
test_entities.py:                              23 passed

Earlier validation on prior PR heads also passed test_dump_restore.py (1), test_bincount_minlength.py (2), and test_projectors.py (6); those files were not rerun for this review fix.

Ruff format and ruff check pass. I ran these files one at a time and left the full suite to CI, because the shared host is saturated.

Impact

For non-empty memberships, the length guards retain the existing numpy.max(x) + 1 calculation, and the rank reshape preserves the existing matrix dimensions. Default simulations now use np.arange(count, dtype=np.int64). On 64-bit platforms, positive counts supplied as int, np.int32, np.int64, or np.uint64 retain the old np.array(range(count)) ID values and int64 dtype. Zero counts now produce empty integer IDs instead of empty float64 IDs. Regression tests cover counts 0, 1, and 4 for all four types; the property test also draws the count type. These focused tests do not establish universal country-output equivalence; full suites and microsimulations remain with the hub and CI.

Changes after review: Specify np.int64 for default population IDs to preserve unsigned-count projections, and cover all four count types in regression and property tests.

axiom: n/a: engine fix in population reductions, no policy change

Other audit findings, handled elsewhere

The 2026-10-06 invariants audit of master 757147c7 also confirmed defects that this PR leaves alone, because other work owns them:

Branch staleness after set_input and raw-file restore are documented behaviour, not defects.

🤖 Generated with Claude Code

MaxGhenis and others added 4 commits October 6, 2026 12:59
GroupPopulation.reduce (behind all, max and min), members_position and
Population.get_rank each took numpy.max of the persons' positions or group
indices to size a loop. With no persons that array is empty, so they raised
"zero-size array to reduction operation maximum which has no identity"
instead of returning the neutral element reduce already fills each group
with. They now start the maximum at -1, so the loop runs zero times; get_rank
keeps its value matrix two-dimensional with no columns.

Adds the audit witness (one household, no persons) and a Hypothesis
property: for 1 to 6 groups and 0 to 30 persons, sum, nb_persons, any, all,
max, min and get_rank equal a plain-Python oracle, empty groups included.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
numpy.max(x, initial=-1) raises OverflowError for an unsigned index
array, which master handled for any non-empty population. A length check
gives 0 for no persons and leaves every other case exactly as on master.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Addresses the independent review of eca6a62 (one should-fix finding).

An empty list or numpy.array([]) is float64, and so were the IDs and
memberships build_default_simulation(count=0) made (np.array(range(0))).
NumPy indexes only with integers or booleans, so get_rank, project,
value_nth_person and projector reductions over no persons still raised
IndexError or TypeError. The members_entity_id setter now stores an empty
membership as an integer array, and build_default_simulation uses
np.arange(count), which is integer even for 0.

Adds regressions for a plain [] membership and a default simulation of no
one, and lets the Hypothesis property pass memberships as plain lists.
All fail on eca6a62 (the property shrinks to one group, no persons,
given as a list).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis
MaxGhenis merged commit dd7dc1b into master Oct 18, 2026
19 checks passed
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

US + core hub merge audit, core#590 at 941b96ba334068c0a1f1fdc4abd06d27a9b7083a (squash): reduce and rank over a population with no persons.

  • Review:
    • A GPT-6.1 Sol shell review (20261018-115342-hub-core590-review-sh) found the empty-population behaviour correct. It requested one change (P2): np.arange(count) gave float IDs for np.uint64 counts, so project() raised.
    • A hub build fixed it with explicit np.int64 IDs and a regression test across int, np.int32, np.int64 and np.uint64 that fails before the fix.
    • A Sol delta review (20261018-190057-hub-core590-review-delta) APPROVED this exact head. IDs match master exactly for every positive integer count across 14 integer types. The only differences from master are at zero counts: master gave float IDs and IndexError there, and this PR gives int64 IDs and working reductions, as intended. This is a bug fix with one defensible reading.
  • PE-US gate: PE-US main 83e03a92f6 against this PR merged into master aa5fd8820a (tested tree f97b9f17d2): 25 core test files, 494 passed, rc 0.
  • Gates checked live at merge:
    • every check passes (19 of 19 at this head);
    • MERGEABLE;
    • not a draft;
    • no CHANGES_REQUESTED review;
    • head pinned.
  • Impact: the hub's full-sample A/B on PE-US's default dataset (2026; core 757147c747 against 8d662ec7e8): household_net_income, income_tax, state_income_tax and snap identical, with 0 records changed. The fix commit since then changes only the ID dtype for unsigned counts in build_default_simulation, which the dataset path doesn't use.
  • Follow-ups (proposed separately): restoring dumped simulations with no persons (simulation_dumper.py:134), and rejecting non-integer counts at build (operator.index).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant