ENH: reproducible Monte Carlo via per-simulation-index seeding - #1054
ENH: reproducible Monte Carlo via per-simulation-index seeding#1054thc1006 wants to merge 10 commits into
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## develop #1054 +/- ##
===========================================
+ Coverage 89.96% 90.70% +0.73%
===========================================
Files 131 131
Lines 17527 17554 +27
===========================================
+ Hits 15769 15922 +153
+ Misses 1758 1632 -126 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
0d37ed6 to
761c092
Compare
phmbressan
left a comment
There was a problem hiding this comment.
The implementation is very clear and throughout, nice work.
The explanation on the concepts behind per index seeding (both in the issue and PR description) were rather helpful. I agree having reproducible results was an issue with the parallel per worker seeding.
Regarding the decisions on parameter naming, I agree with most of the decisions taken here. Moreover, the rng attribute is well docstringed, so it shouldn't be a matter of confusion to the user.
@MateusStano could you give your two cents on the changes here before we proceed with a merge?
Addresses review feedback on RocketPy-Team#1054. Parallel workers claimed the next index with an unlocked keep_simulating() + increment(), so near the end of a run two workers could both pass the count < n check and then claim sim_idx == n; the per-index child_seeds lookup turned that into an IndexError (before, it only wrote one extra record). Move the claim into a _claim_next_index helper that holds the shared mutex across the check and the increment, so each index is handed out once and the counter never overshoots. A deterministic unit test (a barrier plus a widened check-to-increment window) over-claims and fails if the lock is dropped. __root_seed_sequence returned the caller's SeedSequence, and spawn() advances its child counter, so passing the same object to simulate() twice produced different children. Copy it from its full state instead, which leaves the caller untouched and keeps repeated calls reproducible. Also drop Generator/BitGenerator from the accepted types: a stateful generator is not a seed, and reducing it to its underlying SeedSequence ignores how far it has been consumed. random_seed now takes an int, a sequence of ints, or a SeedSequence. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
Addresses review feedback on RocketPy-Team#1054. Parallel workers claimed the next index with an unlocked keep_simulating() + increment(), so near the end of a run two workers could both pass the count < n check and then claim sim_idx == n; the per-index child_seeds lookup turned that into an IndexError (before, it only wrote one extra record). Move the claim into a _claim_next_index helper that holds the shared mutex across the check and the increment, so each index is handed out once and the counter never overshoots. A deterministic unit test (a barrier plus a widened check-to-increment window) over-claims and fails if the lock is dropped. __root_seed_sequence returned the caller's SeedSequence, and spawn() advances its child counter, so passing the same object to simulate() twice produced different children. Copy it from its full state instead, which leaves the caller untouched and keeps repeated calls reproducible. Also drop Generator/BitGenerator from the accepted types: a stateful generator is not a seed, and reducing it to its underlying SeedSequence ignores how far it has been consumed. random_seed now takes an int, a sequence of ints, or a SeedSequence. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
9c020b6 to
3e22729
Compare
Addresses review feedback on RocketPy-Team#1054. Parallel workers claimed the next index with an unlocked keep_simulating() + increment(), so near the end of a run two workers could both pass the count < n check and then claim sim_idx == n; the per-index child_seeds lookup turned that into an IndexError (before, it only wrote one extra record). Move the claim into a _claim_next_index helper that holds the shared mutex across the check and the increment, so each index is handed out once and the counter never overshoots. A deterministic unit test (a barrier plus a widened check-to-increment window) over-claims and fails if the lock is dropped. __root_seed_sequence returned the caller's SeedSequence, and spawn() advances its child counter, so passing the same object to simulate() twice produced different children. Copy it from its full state instead, which leaves the caller untouched and keeps repeated calls reproducible. Also drop Generator/BitGenerator from the accepted types: a stateful generator is not a seed, and reducing it to its underlying SeedSequence ignores how far it has been consumed. random_seed now takes an int, a sequence of ints, or a SeedSequence. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
3e22729 to
6bf8bb6
Compare
|
@MateusStano friendly ping when you have a moment. Both points from your last pass are addressed: the parallel index claim now holds the shared mutex across the check-and-increment (with a deterministic test that goes red if the lock is removed), and a supplied |
6bf8bb6 to
c529d0a
Compare
Addresses review feedback on RocketPy-Team#1054. Parallel workers claimed the next index with an unlocked keep_simulating() + increment(), so near the end of a run two workers could both pass the count < n check and then claim sim_idx == n; the per-index child_seeds lookup turned that into an IndexError (before, it only wrote one extra record). Move the claim into a _claim_next_index helper that holds the shared mutex across the check and the increment, so each index is handed out once and the counter never overshoots. A deterministic unit test (a barrier plus a widened check-to-increment window) over-claims and fails if the lock is dropped. __root_seed_sequence returned the caller's SeedSequence, and spawn() advances its child counter, so passing the same object to simulate() twice produced different children. Copy it from its full state instead, which leaves the caller untouched and keeps repeated calls reproducible. Also drop Generator/BitGenerator from the accepted types: a stateful generator is not a seed, and reducing it to its underlying SeedSequence ignores how far it has been consumed. random_seed now takes an int, a sequence of ints, or a SeedSequence. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
|
Heads up that this changed enough since the last look to be worth a fresh pass rather than merging on the earlier approval. @MateusStano @phmbressan when you have a moment. What is new since the review:
Both earlier concerns are still handled: the parallel claim holds the mutex across the check and the increment, and a supplied |
c529d0a to
e5ba940
Compare
|
@MateusStano @phmbressan a follow-up pass turned up a few more things worth fixing, so I have pushed them and would appreciate another look when you have time. Since your reviews:
I also marked the earlier threads resolved. The race and the SeedSequence copy are both fixed in the current code, and the dangling-files question checked out: the run writes only under A few larger items from the same review are better as their own issues, so I opened #1075 (append continuation), #1076 (a full parallel test under spawn and forkserver) and #1077 (a seed for |
5a9c119 to
da2ba5c
Compare
|
I opened #1071 for the same issue before spotting this one, and have closed it in favor of this PR, yours is the more complete solution (per-component child seeds, list-valued attribute sampling, int seeds for custom |
for sure go ahead thx!!! |
696c516 to
8366640
Compare
|
Tidying up after myself here, and stating where this actually stands. TL;DR: the branch was rewritten as promised. Head is What is left in this PR is the thing #1053 asked for and nothing else:
Measured over real flights, four simulations: Where the rest of the old branch went, all separately reviewable:
Correcting something I wrote here earlier. I said the split-independence guarantee held under Measured with the models seeded by hand and no per-index seeding in the way: the parent, a forked child and a spawned child build the same rocket, mass to the last digit. There is a test on the spawn start method now rather than an assumption, and taking the per-index seeding away still turns three of these red, so dropping those two fields has not made the comparison vacuous. Still true and worth saying: appending continues the same stream only if you pass the same seed again, since nothing here records it. That check is #1075. This stays draft until #1181 lands, which is the only thing under it. I have also deleted nine of my own older comments on this PR. They were progress notes naming commits that no longer exist in the branch, so following them led nowhere. Everything that answered one of you is untouched. Reporting every push here was the wrong habit, and I have stopped. |
A parallel run spawns a SeedSequence per worker and passes it to environment, rocket and flight. _sampler_seed then fed it to SeedSequence(entropy=...), which takes an int or a sequence of ints, so the first worker raised TypeError before drawing anything. The call was reached only from the custom sampler reset until RocketPy-Team#1117 added the list-choice generator, which every model goes through. A real two-worker run passes at d21abde^ in 2.32s and does not finish on develop: the worker's own error path raises UnboundLocalError on inputs_json, so the parent never learns it died and the run hangs. The children of one root share their entropy and differ by spawn_key, so the value is folded through generate_state rather than read off entropy, which would put every worker on one sampler stream. Nothing is consumed, and an int or None seed keeps the stream it had. The fold lives in rocketpy.tools, since the component streams and the per-index seeding both need the same one and three copies would drift on width and word order. _sampler_seed does its own final fold through it as well rather than repeating the four lines. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
The two run paths named the same simulation differently. Three of them wrote 1, 2, 3 through the serial path and 0, 1, 2 through the parallel one, so a row could not be compared with its counterpart and an index meant nothing on its own. Serial counts from zero now, which is what the parallel path already did and what append already assumed: num_of_loaded_sims counts rows, so a two-row checkpoint resumes at 2, an index the serial path never used. Existing serial results are numbered one higher than the same run would be numbered now. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
A run seeded its workers, one seed each, from fresh entropy every time, and the serial path never reseeded at all. So the same study gave different results run to run, different results in the two modes, and different results again when the worker count changed. Addresses RocketPy-Team#1053. simulate() takes random_seed now, keyword-only, and every simulation takes the child of that root belonging to its index. The child is derived directly rather than by spawning the ones before it: spawn appends n_children_spawned + i to the parent key, so rebuilding that one child reproduces it bit for bit, and a worker reaches any index from four picklable values instead of a list a million long. There is a test comparing it with spawn(n)[i] for every seed type. Measured over real flights, four simulations: serial(42) == serial(42) True serial(42) == parallel(2 workers, 42) True serial(42) == parallel(4 workers, 42) True serial(42) == serial(7) False The per-worker seed is gone rather than kept alongside, since a worker now decides nothing about sampling and how many there are cannot reach it. Appending continues the same stream when the same seed is given, since an index maps to a seed and nothing else. Nothing here checks that the caller did give the same one; persisting the root so it can be checked is RocketPy-Team#1075. Fixed-seed results change: every study is sampled from a different place. Nothing that was reproducible before stops being so, because nothing was. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
simulate() gained random_seed with no entry in its own Parameters block, and the stochastic page hands users to the MonteCarlo class without saying that a run is fixed there rather than on the models. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
RocketPyEncoder records hash(obj) beside every serialized Function, and
that is the object's identity in the process that wrote the row. Two
runs sharing memory agree on it and two that do not, differ, so the
comparison was answering a question about the start method rather than
about the seed.
Windows uses spawn, and both its legs failed on exactly those fields.
Reproduced on Linux with set_start_method("spawn").
Every hash is dropped at any depth before comparing now. This does not
make the split-independence test pass under spawn: with the identity
gone it still differs on power_off_drag and power_on_drag, which is a
separate and so far unexplained difference.
Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
The Windows legs of this branch hung and then reported a mismatch on power_off_drag and power_on_drag, and I read that as the guarantee failing under spawn. It was the test. Two fields a serialized Function carries belong to the writer rather than to the draw. hash is the object's identity in that process. A callable source is its pickle, and the same callable pickles to different bytes in a spawned child: measured on one drag curve, the parent and a forked child agree and a spawned child does not. Both of those drag curves are declared None on the fixture, so nothing varies them and neither field ever carried a draw. Measured with the models seeded by hand, no per-index seeding in the way, one seed: the parent, a forked child and a spawned child all build the same rocket, mass to the last digit. So the guarantee does hold on the start method Windows uses, and there is now a test that says so rather than an assumption. Removing the per-index seeding still turns three of these red, so dropping those two fields has not made the comparison vacuous. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
8366640 to
f9a10fa
Compare
_set_stochastic re-validates every declared input, and validation reads the nominal off the wrapped object. create_object writes the sampled value back onto that same object on purpose, so re-reading it on a reseed took one simulation's output as the next one's nominal: a wind factor compounded 10 -> 8.576 -> 7.355 -> 6.308 under a single fixed seed, and a plain scalar spec drifted the same way. Read the nominal once and keep it. Containers are copied on the way in, so writing through the wrapped object cannot reach it either. A component position arrives through an injected getter, reads an attribute nothing writes back to, and shares one name across every component, so those are read live rather than cached. Extracted from RocketPy-Team#1054. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
_set_stochastic handed the same seed to the rocket body and to every surface, motor, rail button and parachute, so two components built from one spec drew identical values: a main and a drogue with the same cd_s and lag spec drew the same cd_s and the same lag, every time, and a study of both was a study of one counted twice. Air brakes were worse. They are built and sampled in create_object and were not in the reseed at all, so their values came from wherever the generator had been left rather than from the seed: 0.683, then 0.586, then 0.488 for one seed asked three times. Each component now takes its own child of a SeedSequence root, spawned in a fixed order so one seed still reproduces the whole rocket. The collections are named in one place and checked against create_object's own source, since the collection no fixture populates is the one that gets missed. Extracted from RocketPy-Team#1054. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
The serial numbering only reaches whoever ran it through that message, and the producer loop is otherwise only ever watched from another process. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
The workflow that would have written it has not run since RocketPy-Team#1112, which is what RocketPy-Team#1173 is about. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
keep_simulating() and increment() were two calls on a manager proxy, so two workers could both see the last slot free and then claim an index each, one of them past the end. claim_next_index() does both under a lock in the manager's own process, and the two methods it replaces are gone so the pair cannot be written again. The index also moves after the data collectors, so one named "index" cannot take the number the row's seed was derived from. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
|
Pushed 3aada2c, which restores the atomic claim from the earlier review.
I want to be straight about the test, because I could not build one that observes the race. A thread-level version does not reproduce it even at The other half of the commit: the index is written after the data collectors, so one named |
|
@MateusStano thanks for your patience on this one, and for the race in particular. You were right, and it is fixed in 3aada2c. Your reading was exactly it. One thing I want to be straight about: I could not build a test that observes the race. A thread-level version does not reproduce it even at Your second point, the The branch is out of draft now and all nine checks are green. It is still stacked on #1181, so the diff shown here carries that commit until it lands. |
A stochastic input reads its nominal from the object it wraps, and a run moves that as it goes, so two runs of the same index can draw the same inputs and still fly differently. Measured over four simulations with a wind factor: the recorded inputs match serial to two workers for every index, and three of the four apogees do not. What this branch fixes is the inputs an index draws, which is what the note says now. The index test also stubs the root state, which a later branch writes into the same row. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
|
Pushed 97f3b45: a narrowing of the note, and one test stub. I checked the guarantee this branch documents against an input whose nominal moves. The wording said simulation 7 draws the same inputs whichever way the run was split, which is true, and I had been reading it as meaning the simulation is the same, which it is not. Four simulations with a wind factor, serial against two workers:
The recorded inputs match for every index. Three of the four apogees do not. A stochastic input reads its nominal from the object it wraps and a run moves that as it goes, so an index seeded identically can still fly differently depending on which worker took it. That is what #1169 fixes, and it is a prerequisite for reading this note the way it invites. Rather than deepen the stack I narrowed the note to what this branch delivers on its own. The combined regression belongs on whichever of the two lands second. |
|
Closing this in favour of #1187 — not because anything here was rejected, but because that PR already contains this work in full and carrying both would mean reviewing the same code twice. Concretely, #1187 is a strict superset of this branch:
On the review state here: the Thanks @thc1006 for the work and the iterations, and @phmbressan for the review — none of it is lost, it just continues on #1187. Reopen if you think something here does not actually survive in that branch. |
* BUG: give each StochasticRocket component its own random stream _set_stochastic handed the same seed to the rocket body and to every surface, motor, rail button and parachute, so two components built from one spec drew identical values: a main and a drogue with the same cd_s and lag spec drew the same cd_s and the same lag, every time, and a study of both was a study of one counted twice. Air brakes were worse. They are built and sampled in create_object and were not in the reseed at all, so their values came from wherever the generator had been left rather than from the seed: 0.683, then 0.586, then 0.488 for one seed asked three times. Each component now takes its own child of a SeedSequence root, spawned in a fixed order so one seed still reproduces the whole rocket. The collections are named in one place and checked against create_object's own source, since the collection no fixture populates is the one that gets missed. Extracted from #1054. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com> * DOC: shorten the comments on the component seeding Measured against the register the repository uses: inline comments in flight.py average 5.6 words and none of its docstrings run longer than the code they describe. The three added here were four to seven lines of prose where a line would do, and the seed helper carried seven lines of docstring over two lines of code. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com> * BUG: leave the rocket body's stream where it was, and isolate each collection Moving the body to child zero broke every fixed-seed baseline for mass, radius and the body inputs, and nothing about the nested-component fix needed that. The body keeps the seed as given now: stochastic_calisto under seed 42 reads mass=14.906007947 on develop and the same here. Components were also addressed by one global traversal index, so adding a fin moved every motor, rail button, parachute and air brake. Each collection has a root of its own now, spawned from the same seed, so an unrelated component in one of them leaves the others where they were. The source scan compares the two sets both ways. A collection left in the reseed after create_object stops using it still spawns a child and moves every stream after it, which the subset check let through. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com> * TST: count the reseeds, and cover two air brakes on one spec The source scan reads create_object for a literal loop over self.collection, so a helper, a local alias or a getattr would hide a collection from it. Counting what each entry actually receives is the check that survives a refactor, and it is the only one that fails when an entry is reseeded twice. The air brakes are a plain list and take a different route through the reseed than the positioned collections, so two of them on one spec are worth their own case. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com> * TST: stop the isolation test from storing one wrapper twice stochastic_calisto already holds the stochastic_nose_cone fixture, so adding it again put the test into the state #1172 describes: one wrapper in two entries, its position overwritten, and two reseeds landing on the same object. The assertion looked at a different collection and passed anyway. It adds the deterministic nose now, so add_nose builds a wrapper of its own. Nothing pinned the body keeping the seed as given either. Reproducibility and seed uniqueness both hold with the body on a spawned child, so neither would have noticed it going back there. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com> * DOC: say how a rocket's components are seeded The change moves every fixed-seed component baseline and nothing in the user documentation said how components are seeded at all, before or after. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com> * DOC: say what separate streams do and do not promise Two independent streams are not made to consume the same draws; they can still land on equal values, and a specification with no spread always will. The text promised unequal results, which is a stronger claim than spawning gives. It also said each kind of component is spawned separately. The unit is the collection: a nose cone, the fins and the tail share one root. And a stream belongs to one wrapper, so storing one twice or sharing it between rockets is outside what this establishes. That is #1172. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com> * DOC: say that the reset builds the tree, not the add A rocket resets itself while being constructed, when it holds no components yet, so a parachute added afterwards keeps the generator it was built with until the next reset. The text read as though attaching a component gave it a stream, which is only true once something resets the rocket, and a Monte Carlo is what does that. Two wrappers sharing a CustomSampler seed_group are also one stream on purpose. Separate component streams are not meant to take that apart, so the note says so rather than leaving it to be discovered. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com> * DOC: correct what Monte Carlo and a shared sampler group actually do A serial MonteCarlo run never resets the rocket, and a parallel one resets each worker once rather than once per simulation, so the text saying a run resets the rocket for you was wrong for both. Per-simulation reset is the Monte Carlo seeding work, not this change. CustomSampler.seed_group already documents that a group belongs to one model and that the last to seed it wins. Saying two components sharing one stay one stream on purpose read as a guarantee this does not make. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com> * DOC: the parallel path does not get as far as building the tree Saying it resets each worker once reads as though it works and only the grain differs. It hands the model a SeedSequence where an integer is wanted, so it stops before the tree exists, which the PR already records as the Monte Carlo seeding work rather than this change. The two air brake test also says what it is not: both are added with one controller because the rocket keeps a single one, which is #1172. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com> * TST: pin every word of the seed, and say what append-only means Dropping the fourth word left all 33 tools tests passing: the checks were that the high bits are not zero, that the low word matches, that two children differ and that reading twice agrees, none of which a 96 bit truncation breaks. It compares against the integer rebuilt from all four words now, and that mutation fails. A collection's stream is addressed by where its name falls in the two tuples read end to end, so appending to the first moves every name in the second. The comment said append rather than reorder, which reads as though appending to either one is safe. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com> * BUG: take the seed type a parallel run hands the rocket A parallel run spawns a SeedSequence per worker and passes it down, and SeedSequence does not take another one as entropy, so rooting the collections from it raised TypeError. It was unreachable until now: the base _set_stochastic refuses the same type one frame earlier, so a worker never got this far. Once that is fixed the call here is the next one to fail, which is why it is fixed in the same series rather than left for whoever hits it. Copied from the full state rather than spawned from directly. spawn() advances the counter of an object the caller still holds, and a second use of the same seed would then build the components a different tree. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com> * MNT: drop the _seed_sequence_to_int this branch no longer needs to add #1181 landed the same helper on develop, with the same body, so merging develop in left tools.py defining it twice a hundred lines apart. Git had no conflict to report: the two copies were added at different points in the file, so the second simply shadowed the first, and pylint would have failed the branch with E0102 rather than anything explaining why. Develop's copy is kept, its docstring being the fuller of the two. What this branch still adds on its own is _seed_sequence_from, which is left where it was, and the test in test_tools.py now covers the surviving definition. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com> Co-authored-by: Gui-FernandesBR <guilherme.fernandes@maggu.ai> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* BUG: sample around the nominal a stochastic model was built with _set_stochastic re-validates every declared input, and validation reads the nominal off the wrapped object. create_object writes the sampled value back onto that same object on purpose, so re-reading it on a reseed took one simulation's output as the next one's nominal: a wind factor compounded 10 -> 8.576 -> 7.355 -> 6.308 under a single fixed seed, and a plain scalar spec drifted the same way. Read the nominal once and keep it. Containers are copied on the way in, so writing through the wrapped object cannot reach it either. A component position arrives through an injected getter, reads an attribute nothing writes back to, and shares one name across every component, so those are read live rather than cached. Extracted from #1054. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com> * BUG: keep the nominal out of reach of what is generated from it Copying on the way in was not enough. _nominal handed back the kept object itself, and on the empty-spec path that one object reached the model attribute, last_rnd_dict and the FreeFormFins create_object returns, so a write through any of them moved what the next reseed sampled around. _snapshot_of stopped at a tuple as well, which left an array inside an airfoil pair shared with the object it came from. Copy on the way out too, and recurse through the built-in containers. The documented contract now names the four cases that stay outside it: an input added after construction, a component position, an ensemble wind factor, and anything that is not an array or a built-in container. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com> * TST: pin when a late input is captured, and the spread-tuple path An add_* input is configured after __init__, so its nominal is read then. The new test writes the rocket's eccentricity before add_cp_eccentricity and again after it, and only the first one may reach the draw. The (std, distribution) form now runs its own seed histories rather than repeating one seed, which is what a cache keyed by the seed instead of by the model actually fails. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com> * TST: pin the Function nominal as a boundary, not a footnote The documented exception said a Function is held as it was given. Nothing enforced it, so closing the hole later would have gone unnoticed and the documentation would have quietly become wrong. Measured: set_source on the rocket's drag curve moves the drawn value from 0.377 to 0.890, and deepcopy of that curve costs 6 microseconds. Cost is not the reason to leave it. _snapshot_of cannot raise today, and deepcopying whatever a user passed, on a path that runs on every reseed, would make it able to. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com> * BUG: read the nominal again when a late input is configured again Keeping the nominal gave the second add_cp_eccentricity nothing to replace, so it went on sampling around the value the rocket held at the first call: 0.5 where 0.8 was asked for. Reproducible, and around the wrong centre, which is harder to notice than a value that moves. Late configuration drops the kept nominal before validation reads one, and puts it back if validation raises, so a refused call leaves the previous configuration standing. Only the reconfiguration path. Passing None still leaves the earlier declaration in place, which is develop's behaviour and not this branch's. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com> * DOC: say what the snapshot does not do Measured on the current implementation: a cycle recurses until Python stops it, two references to one list come back as two lists, and the elements of an object-dtype array stay shared. None of those reach a supported nominal, but the docstring read like a general deep copy and should not. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com> * BUG: take a late input away when it is configured to None None is a configuration too. The nominal was refreshed but the earlier distribution stayed declared, so the next reseed validated it again and drew an uncertainty the caller had asked to remove. Filed as #1171 while the removal lived elsewhere; it belongs in the replacement helper this branch added, so it is here rather than in a second PR that owns the other half of one state transition. None still means an axis that was never given, and removing what was never declared stays a no-op. Both meanings have a test. The snapshot test asserted a dict entry was not None, which held whether or not anything had been copied, and no test reached the set branch at all. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com> * BUG: replace a pair of late inputs together or not at all add_cp_eccentricity takes x and y in one call, so a y that will not validate left x already replaced and declared. Validation happens for the whole group before anything is committed now. The test gives only y first, so x is undeclared going in and a partial commit shows up as an eccentricity the caller never successfully asked for. Asserting the nominal alone did not catch it, since x's nominal was restored either way. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com> * BUG: give every draw its own copy of a mutable value Copying on the way out of the kept nominal was not the last boundary. The list branch handed back the candidate itself, which for an empty spec is the model's own working value, and FreeFormFins keeps shape_points by reference. Writing through the first generated fins reached the second ones: first = stochastic.create_object() first.shape_points[1] = (9.9, 9.9) second = stochastic.create_object() # (9.9, 9.9) as well No reseed in between, which is how create_object is documented to be used and how a serial Monte Carlo runs it. last_rnd_dict was the same dictionary the values were built from, so it moved with them too. It records what was drawn now, which matters because a Monte Carlo writes it out after the flight rather than before. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com> * BUG: record a draw after the subclass that adjusts it, not before Recording in the base generator put the record before StochasticFreeFormFins had pulled the fin root back onto the body line. Under seed 7 the two root points drifted to 0.000299 and 0.001340, the correction returned them to zero, and the record kept the outline the fins were never built from. The rocket copies each component's record into its own, so the Monte Carlo input log carried it too. That is the failure class #1090 was about, arriving from the other side. _record_draw is the one place a model publishes what it drew, and a subclass that changes a value calls it again. A source scan holds the next subclass to the same rule, since the one that gets it wrong is the one nobody wrote a fixture for. _declare_stochastic_input and the _MISSING sentinel had no callers left after the grouped reconfiguration landed, and the first still carried the None handling that #1171 was about, so they are gone rather than left as a second lifecycle for someone to reach for. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com> * DOC: _choose returns a copy, and say so where it is documented The docstring still promised values itself when there are no candidates, which stopped being true when the draw started handing back a copy. Nothing reaches that branch through a validated input, since an empty list validates to the object's own value, so it is a guard against integers(0) rather than a path with a caller. It has a test now, which is also the one line of this change Codecov had no coverage for. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com> * BUG: record the parachute noise seed the parachute was built with StochasticParachute.create_object derives the pressure noise seed after the draw, and #1134 relied on last_rnd_dict being the same dictionary to carry it into the record. Snapshotting the draw broke that link: the parachute is still built with the seed, but the record loses it, so the Monte Carlo inputs stop describing the parachute that flew. develop recorded 37773913418288439290323614982376424810 before recorded <absent> The source scan missed it because it only read dict_generator overrides. It reads create_object too now, and tracks the names a method binds from a draw rather than guessing at a variable name, so a local a method fills in for its own use is not mistaken for a record. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com> * TST: walk the subclass tree, not the package exports StochasticMotorModel is a StochasticModel subclass that rocketpy.stochastic does not export, so the scan could not see it. It overrides neither method today, which is why nothing was wrong, and which is also why the gap would have gone unnoticed until something did. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com> * BUG: stop an omitted axis from taking away what was declared Removing on None looked like one line inside the new replacement helper, and it is not. add_cp_eccentricity(x=..., y=...) defaults both to None, so an omitted axis and an explicit None read identically, and the removal took away an axis the caller never mentioned: add_cp_eccentricity(x=0.001, y=0.002) add_cp_eccentricity(x=0.005) # y quietly gone develop keeps y here, and so does this again. Removing an earlier declaration needs an argument omission cannot supply, which is a signature change and its own decision, so it stays in #1171 rather than arriving inside a change about nominal ownership. The test that asked for removal is replaced by one that holds the omitted axis in place, since that is the behaviour anything already written depends on. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com> * DOC: say what a second add_cp_eccentricity call does Both arguments read as optional and nothing said what happens when the method is called again, which is the whole of the question behind #1171. Each public docstring now states it: a later call replaces what was configured, an omitted axis keeps what it had, and taking one away is not supported. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com> * BUG: leave an axis that was left out entirely alone Keeping its declaration was not enough. The omitted axis still went through the whole replacement: its kept nominal was dropped, None was validated again into a lone nominal, and that was written back over its distribution. The private side then said the axis was random while the attribute dict_generator reads said it was not, so it stopped varying: add_cp_eccentricity(x=0.001, y=0.002) add_cp_eccentricity(x=0.005) eight draws of y -> one distinct value A serial Monte Carlo never resets, so a whole study would have run with that axis switched off and nothing raised. Dropping the nominal also moved the centre. With the rocket's own y changed between the two calls, the next reset centred the old distribution on 9.0 rather than the 0.0 it was configured around. An axis given as None that already has a configuration is now left out of the transaction: not revalidated, its nominal not re-read, its attribute not rewritten. The test covers both eccentricity methods and looks before the reset as well as after, which is where the previous one missed it. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com> * DOC: say what a snapshot does not reach inside an object array ndarray.copy() copies an object array without copying its entries, so a later write through one of them is still visible. The scope was numeric arrays already; this says so. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com> --------- Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
Two entries named two pull requests, and RocketPy-Team#1054 is closed rather than merged: its work is in this branch. A reader following that link lands on a closed page. One entry now, on the pull request the release actually contains, with both issues it answers. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
simulate(append=True) derived a fresh root and carried on writing into the file it was given, so a study resumed after a restart held two lineages with nothing afterwards to say which simulation came from which. Every input row now records the root that drew it, and an append reads it back rather than needing to be given it again. A seed that disagrees with the rows is refused, as is a log whose rows disagree with each other, and one whose rows carry no root at all, which is how a study written before this looks. Output rows carry a digest of that root, so a log belonging to another study is refused even when its indices line up with this one's. Per-simulation-index seeding comes with it: a simulation takes its seed from its own index, so it draws the same inputs however the run was split up. That was RocketPy-Team#1054, closed in favour of this. Addresses RocketPy-Team#1053 and RocketPy-Team#1075. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
A Monte Carlo run cannot be seeded on develop: simulate() takes no random_seed. Seeding it per simulation index is not enough on its own, because an append then derives a fresh root and writes it into the same file, so a study resumed after a restart holds two lineages with nothing afterwards to say which simulation came from which. Both halves are here. A simulation takes its seed from its own index, so a serial run and a run split over workers draw the same inputs for the same index. Every input row records the root that drew it, and an append reads it back rather than needing to be given it again. A seed that disagrees with the rows is refused, as is a log whose rows disagree with each other, and one whose rows carry no root at all, which is how a log written before this looks. Output rows carry a digest of that root, so a log belonging to another study is refused even when its indices line up with this one's. The seeding half was RocketPy-Team#1054, closed in favour of this. Addresses RocketPy-Team#1053 and RocketPy-Team#1075. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
A Monte Carlo run cannot be seeded on develop: simulate() takes no random_seed. Seeding it per simulation index is not enough on its own, because an append then derives a fresh root and writes it into the same file, so a study resumed after a restart holds two lineages with nothing afterwards to say which simulation came from which. Both halves are here. A simulation takes its seed from its own index, so a serial run and a run split over workers draw the same inputs for the same index. Every input row records the root that drew it, and an append reads it back rather than needing to be given it again. A seed that disagrees with the rows is refused, as is a log whose rows disagree with each other, and one whose rows carry no root at all, which is how a log written before this looks. Output rows carry a digest of that root, so a log belonging to another study is refused even when its indices line up with this one's. RocketPy-Team#1182's worker tests drive the producer with a stand-in monitor, so they move to the claim along with it. A reseed failure now names the index it was seeding for rather than worker startup, because the seeding happens after the claim rather than once above the loop. The seeding half was RocketPy-Team#1054, closed in favour of this. Addresses RocketPy-Team#1053 and RocketPy-Team#1075. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
Rewritten from current
develop. The old head is not replayed: it is kept atthc1006/RocketPy@backup/pr-1054-cf99e872, and what came out of it went up separately as #1169, #1170, #1181 and #1182.What is left here is the thing #1053 asked for and nothing else.
The other three files are the two test files and a paragraph in
docs/user/stochastic.rst.Stacked on #1181. This branch carries that commit, so the diff shown here includes it. Happy to rebase once it lands.
Pull request type
Checklist
ruff check/ruff format --check,pylint) has passed locallyCHANGELOG.md: one line, written by hand. The workflow that would have written it has not run since CI: run the changelog job for pull requests from forks #1112, which is BUG: the changelog workflow stopped running, and CHANGELOG.md is 37 merged pull requests behind #1173.Current behavior
simulate()has no seed at all. The two run paths seed differently, and one of them does not seed:So a study gives different results run to run, different results in the two modes, and different results again when the worker count changes. That is #1053.
Underneath it, the two paths do not even agree on what a simulation is called. Three simulations, measured:
An index therefore meant nothing on its own, and no index-keyed guarantee was possible before that was settled. It is the first commit here, separately revertible.
New behavior
simulate(..., random_seed=None), keyword-only. Every simulation takes the child of that root belonging to its index.The child is derived directly rather than by spawning the ones before it.
spawnappendsn_children_spawned + ito the parent key, so rebuilding that one child reproduces it bit for bit and a worker reaches any index from four picklable values instead of a list a million long. There is a test comparing it againstspawn(n)[i], for every index, for each ofint,None, a sequence, and aSeedSequence.Measured over real flights, four simulations:
The last line is the control. Without it the first three pass on a run that ignores the seed.
The per-worker seed is gone rather than kept alongside. A worker decides nothing about sampling now, so
__sim_producerno longer takes a seed and__run_in_parallelno longer spawns one per worker.pylintfinding the argument unused is what made that obvious.An index is claimed in one operation.
keep_simulating()andincrement()were two calls on a manager proxy, so two workers could both see the last slot free and then claim one index each, the second past the end.claim_next_index()does both under a lock in the manager's own process, and the two methods it replaces are gone, so the pair cannot be written again by accident. The index is also written after the data collectors now, so one namedindexcannot take the number the row's seed was derived from.Addresses #1053. Not
Closes, because the default branch ismasterand a closing keyword only fires into it.It also addresses the spawn half of #1076.
test_an_index_keeps_its_inputs_when_the_workers_are_spawnedruns a realsimulate()on two and four workers underspawnand compares the sampled inputs against serial. The forkserver half is still only the derivation helpers, so that issue stays open.Breaking change
Two, both worth stating plainly.
Fixed-seed results change, because there were none: nothing in
simulate()was reproducible before, so nothing that was reproducible stops being so. Every study is now sampled from a different place than it would have been.Serial results are numbered one lower. A three-simulation serial run wrote indices 1, 2, 3 and now writes 0, 1, 2. Anything reading those indices back sees the shift. Append already assumed zero-based, since
num_of_loaded_simscounts rows and a two-row checkpoint resumes at index 2, which the serial path never used.What this does not do
Appending continues the same stream when the same seed is given again, since an index maps to a seed and to nothing else. Nothing here records the seed, so nothing here can tell you whether a later append was given the same one. Persisting the root so that can be checked is #1075, and the docstring says so rather than implying more.
Not in here, and each already has its own place: the nominal lifetime (#1169), component streams (#1170), worker failure reporting (#1182), checkpoint and manifest work (#1075). The old branch had all of them at once, which is what made it unreviewable.
Additional information
Verification, on a clean tree:
Seven
tests/unitfailures on my machine are optional dependencies it does not have: four intest_sensitivity.pyforstatsmodels, three intest_environment_analysis.pyforwindroseandipywidgets. All seven areImportErrorbefore any assertion.Each mechanism is pinned by a mutation, and each leaves a control standing:
SeedSequenceused as givenThe last two rows are new. The patch had two lines uncovered, both reachable only when something goes wrong or when the worker loop runs in another process, so each now has a test that reaches it here: the failure message names the index the parallel path would, and the loop a worker runs, driven in this process, produces the run serial produces.
One mutation survived, and finding that was the point. I had written the entropy copy twice, once in
_root_seed_sequenceand once in_root_state_of. Removing either alone changed nothing, because the other still did the job; only removing both turned a test red. So one of them could not be distinguished from its own absence. Kept the one at the boundary where the value stops being the caller's, deleted the other.A second mutation survives, and I would rather say so than imply a test I do not have. Putting the two-call claim back leaves everything green. A thread-level test does not reproduce the race even at
setswitchinterval(1e-9)with eight claimers over five thousand slots, and the existing parallel test stayed green over five runs with the old loop restored. The window is one IPC round trip and two workers have to arrive inside it. The two tests I added pin the claim contract and both die to a boundary mutation, but the fix rests on construction rather than on a failing test.The copy is not decoration.
SeedSequencekeeps a sequence entropy by reference:Merged with #1169, #1170, #1182 and #1171's follow-up into a throwaway tree on
developand run there. That found two things no branch shows on its own:pylintR0915 on__sim_producer, which is under the limit on each branch and over it once both grew, and ten of #1182's tests calling the producer with the seed argument this branch removes, which git merged without a conflict marker. Both are noted for whichever lands second.