add job-assignment internals notes and heap sampler - #90
Open
Alistair-Afton wants to merge 3 commits into
Open
Alistair-Afton wants to merge 3 commits into
Alistair-Afton wants to merge 3 commits into
Conversation
doc/job-assignment.rst documents DF's job-assignment auction (job_applicationst bids into a max-heap; do_now adds ~20000000; continuation ~1000000; posting recycling), observed value bands, the task-interruption taxonomy, and reproduction methodology for v0.53.16. job_heap_sampler.py samples world.jobs.job_application_heap via ReadProcessMemory, catching the sub-millisecond in-tick passes that in-process Lua cannot observe.
Per ab9rf (DFHack/scripts#1639): handle_job_applications pushes a crashlog minidump entry of type 0xf on entry and removes it on exit, giving a deterministic in-process signal for when an auction pass is running. Also records its v0.50.16 Steam/Windows address and the version-relocation caveat.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Detailed writeup of DF's job-assignment mechanism, reverse engineered on v0.53.16 Steam via external memory sampling:
doc/job-assignment.rst— the auction model (units pushjob_applicationstbids intoworld.jobs.job_application_heap, a 2000-slot max-heap drained within each assignment pass), observedvaluebands (~9.5k baseline, ~1.05M continuation, ~20M fordo_nowand unit-sourced need jobs), the task-interruption taxonomy (need thresholds, claim races, outbidding, killjob exceptions), reproduction methodology, and open questions. Field offsets and constants are documented with how-to-re-derive instructions.job_heap_sampler.py— a standaloneReadProcessMemorysampler that catches the heap mid-pass (it lives sub-millisecond inside the tick, invisible to in-process Lua polling). Prints each batch with resolved unit ids and posting→job metadata. Constants are parameterized per-build.Related: DFHack/df-structures#908 adds the field-level comments to
df.job.xml; DFHack/dfhack#3245 is the motivating issue (do_nowgranularity).Test plan