document job application heap and posting semantics

[Alistair-Afton]

Summary

Comments-only documentation update in df.job.xml from runtime reverse engineering of DF's job-assignment pass (v0.53.16 Steam, verified by external memory sampling of the live process):

• job_application_binary_heapst/job_application_heap: confirmed it is a max-heap priority queue; entries are transient — the heap is populated and drained within a single assignment pass (~sub-millisecond), which is why it usually reads empty.
• job_applicationst.value: it is a bid score; the highest-valued application wins the posting. job.flags.do_now adds ~20,000,000; ~1,000,000 observed when a unit re-bids the job-type it is already doing (continuation bonus). Unit-sourced special need jobs (Sleep/Drink/Eat) also reach the ~20M band.
• job_postingst: posting_index indexes job_handlerst.postings; dead entries are recycled for new jobs, so an index can point to a different job between passes (existing 'entries never removed' comment extended).
• rough_number_of_applications: incremented as units apply during passes.

No layout/codegen changes — comments and changelog only.

Test plan

• XML parses cleanly (xml.etree + local codegen unchanged)
• Verified empirically in-game on v0.53.16 (heap sampling + do_now toggle experiment)

[ab9rf]

Note that there's nothing here that we haven't known for years. We know exactly where the code that implements the auction is (it's called worldst::handle_job_applications) and have a fairly good idea of how it functions

[Alistair-Afton]

Good to know — thanks. Is documented anywhere I missed? If the exact scoring is already understood I'm happy to tighten these comments to the real numbers (e.g. whether do_now is exactly +20000000 or a multiplier, and what the ~1000000 continuation term actually is) — the XML previously only said 'a priority queue of some sort', so my goal was just to write down what's confirmed. Happy to close this if it's redundant with existing knowledge.

[Alistair-Afton]

(fixing mangled backtick above — meant worldst::handle_job_applications)

[ab9rf]

Good to know — thanks. Is documented anywhere I missed? If the exact scoring is already understood I'm happy to tighten these comments to the real numbers (e.g. whether do_now is exactly +20000000 or a multiplier, and what the ~1000000 continuation term actually is) — the XML previously only said 'a priority queue of some sort', so my goal was just to write down what's confirmed. Happy to close this if it's redundant with existing knowledge.

The do-job-now priority boost is, indeed, exactly 20,000,000. There are a bunch of other adjustments, some of which are of variable size. It takes a good deal of work to get this code to decompile into a readily comprehendible form and I haven't actually put in that work in quite a while so I don't have a solid description of all the various adjustments being done for the current version; looking at 53.16 they're at least superficially different from the last version I looked at closely, but some of this may be due to the compiler inlining code differently (e.g. in one older version I analyzed closely some time ago, the 20,000,000 constant appeared only once, while in the current version it appears seven times, so there's clearly some sort of branch unfolding or other inlining taking place that didn't use to). I've known about this code since Putnam dropped enough hints back in 2023 for me to find it via static reverse engineering techniques, and since then our systematic reverse engineering of first DF 47.05 and later DF 52.05 have made it possible for us to reliably correlate a great many functions in the disassembly with the partial DF source that Bay12 shares with the DFHack core team. We use these systematically reversed-engineered baseline versions as a reference to identify analogous functions in each new release of Dwarf Fortress using static reverse engineering tools. The databases used for this are held privately by the DFHack core team.

We have been far less aggressive in updating the XML with the knowledge acquired via this process since we started getting the header dumps from Bay12. Since we now get DF structure information directly from the authoritative source, we no longer have to tediously work these out and realign them with each release; instead the process is now "diff last release's headers against this release's, and update the XML accordingly". As a result, these XML files are used far less as a place to store notes associated with a process we no longer do, and are now instead used nearly exclusively as the means by which we recast the information in the Bay12-provided header files in order to drive our code generation processes. While we still do a good deal of reverse engineering, the focus of that these days is not "identify what these bytes of memory are used for" (because we now get a great deal of that that information in an authoritative drop from Bay12) but instead "determine how the code manipulates the data we already know about", and that is overwhelmingly being accomplished using static, rather than dynamic, reverse engineering techniques, and the XML structure files are not really the best mechanism for recording knowledge which arises from that process.

[Alistair-Afton]

Thanks — updated in 51dfb9c: the do_now comment and the value comment now state exactly 20000000, and I dropped the speculative ~1M continuation figure since it's only empirical sampling data, not confirmed semantics.

Also understood on the scoping — field-semantics comments kept minimal here; the process-level notes (auction lifecycle, sampler methodology, the 0xf minidump marker you mentioned) live in the df_misc notes instead, which is a better home for them.