Repository navigation
Reduce and rank over a population with no persons - #590
Merged
Merged
Conversation
GroupPopulation.reduce (behind all, max and min), members_position and Population.get_rank each took numpy.max of the persons' positions or group indices to size a loop. With no persons that array is empty, so they raised "zero-size array to reduction operation maximum which has no identity" instead of returning the neutral element reduce already fills each group with. They now start the maximum at -1, so the loop runs zero times; get_rank keeps its value matrix two-dimensional with no columns. Adds the audit witness (one household, no persons) and a Hypothesis property: for 1 to 6 groups and 0 to 30 persons, sum, nb_persons, any, all, max, min and get_rank equal a plain-Python oracle, empty groups included. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
numpy.max(x, initial=-1) raises OverflowError for an unsigned index array, which master handled for any non-empty population. A length check gives 0 for no persons and leaves every other case exactly as on master. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Addresses the independent review of eca6a62 (one should-fix finding). An empty list or numpy.array([]) is float64, and so were the IDs and memberships build_default_simulation(count=0) made (np.array(range(0))). NumPy indexes only with integers or booleans, so get_rank, project, value_nth_person and projector reductions over no persons still raised IndexError or TypeError. The members_entity_id setter now stores an empty membership as an integer array, and build_default_simulation uses np.arange(count), which is integer even for 0. Adds regressions for a plain [] membership and a default simulation of no one, and lets the Hypothesis property pass memberships as plain lists. All fail on eca6a62 (the property shrinks to one group, no persons, given as a list). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Author
|
US + core hub merge audit, core#590 at
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this fixes
With no persons in a simulation,
GroupPopulation.all,maxandminraise:reducefills each group with its neutral element, but it first tooknumpy.maxof the persons' positions to size its loop, and that array is empty.members_position(numpy.max(members_entity_id)) andPopulation.get_rank(numpy.max(positions)) have the same problem. All three now size their loop as 0 when there are no persons, so it runs zero times. They use a length check rather thannumpy.max(..., initial=-1), which would raise for an unsigned index array.get_rankreshapes its value matrix to(count, 0)so it stays two-dimensional.sumandnb_personsalready returned 0 for this case.The independent review found that an empty membership given as
[], or built bybuild_default_simulation(count=0)(np.array(range(0))), isfloat64. NumPy indexes only with integers, so ranks and projections over no persons still raised. Themembers_entity_idsetter now stores an empty membership as an integer array, andbuild_default_simulationusesnp.arange(count, dtype=np.int64), which keeps IDs integer for zero counts and for Python and NumPy integer counts.The audit verifier's witness was one household and no persons. With this PR,
allreturns[True],any[False],max[-inf],min[inf],sum[0],nb_persons[0], andget_rankreturns an empty array.Invariants
For any number of groups (1 to 6) and any assignment of 0 to 30 persons to them (so groups may be empty and there may be no persons at all), given as an integer array or a plain list:
sum,nb_persons,any,all,maxandminequal a plain-Python oracle, with 0, False, True, -inf and inf for a group with no members;get_rankgives each person the position of their value among their group's values, ties going to the person listed first.int,np.int32,np.int64ornp.uint64hasint64IDs equal to master's, andprojectmaps group values onto persons correctly.Tests
test_group_population_empty.py: the witness for every reduction,get_rankwith no persons, a plain[]membership, and a default simulation of no one (rank, nth person, project, projector max, all). Also checks default simulation IDs and projections for counts 0, 1, and 4 supplied asint,np.int32,np.int64, andnp.uint64.test_group_population_reductions_property.py: one Hypothesis property, 300 examples, drawing the count integer type and checking default simulation IDs and projection as well as reductions and ranks.Before the fix, on master's
policyengine_core/populations:Round 2 (after the independent review), the new tests on the first head
eca6a621:After the unsigned-count review, the extended tests on the unchanged head
8d662ec7e8729ae640ec270ca06ee2f5c81352fd:After this review fix (Python 3.13.9, locked NumPy 2.4.2):
Earlier validation on prior PR heads also passed
test_dump_restore.py(1),test_bincount_minlength.py(2), andtest_projectors.py(6); those files were not rerun for this review fix.Ruff format and ruff check pass. I ran these files one at a time and left the full suite to CI, because the shared host is saturated.
Impact
For non-empty memberships, the length guards retain the existing
numpy.max(x) + 1calculation, and the rank reshape preserves the existing matrix dimensions. Default simulations now usenp.arange(count, dtype=np.int64). On 64-bit platforms, positive counts supplied asint,np.int32,np.int64, ornp.uint64retain the oldnp.array(range(count))ID values andint64dtype. Zero counts now produce empty integer IDs instead of emptyfloat64IDs. Regression tests cover counts 0, 1, and 4 for all four types; the property test also draws the count type. These focused tests do not establish universal country-output equivalence; full suites and microsimulations remain with the hub and CI.Changes after review: Specify
np.int64for default population IDs to preserve unsigned-count projections, and cover all four count types in regression and property tests.axiom: n/a: engine fix in population reductions, no policy change
Other audit findings, handled elsewhere
The 2026-10-06 invariants audit of master
757147c7also confirmed defects that this PR leaves alone, because other work owns them:Branch staleness after
set_inputand raw-file restore are documented behaviour, not defects.🤖 Generated with Claude Code