Index historical ledger reply consumers

[Amaury Chamayou (achamayou)]

Motivation

Retaining historical requests currently makes each ledger reply traverse unrelated handles. #8525 removes the cache-maintenance scans; this separate upper layer removes the two reply-path scans. It was developed stacked on #8525, which has since merged (as have #8527 and #8549), so this PR now targets main directly. The branch was last updated by merging main at 9b0d37030 (merge commit 695e3b27c, no textual conflicts; #8549's receipt/TxID changes and this index coexist in handle_ledger_entry, with the consumer snapshot still taken after every rejection check).

This does not claim constant whole-process CPU or resolution of the load-test incident. Matched load measurements are coordinated separately.

Implementation summary

A private non-owning index maps each ledger seqno to ordered CompoundHandle consumers with independent direct-entry, supporting-signature, and awaited-secret roles. Mutations register before publishing forward dependencies, roll back failed insertions, remove only the affected role, and erase empty buckets. Byte-accounting references remain separate.

Public-only deserialisation reads the bucket's secret-waiter count without visiting handles. Reply processing takes an ordered snapshot before publishing recovered secrets or stores, visits each original consumer once, and preserves secret-waiter precedence and global store trust even without consumers. Protected last_reply_work.requests_visited records actual examined handles and resets on every entry attempt, including rejected entries; there are no production clocks or extra logs.

Missing-entry handling follows the base's #8527 contract: handle_no_entry drops only the requests interested in that seqno, via erase_request, which also removes their routing roles. Request::is_interested_in from #8527 has the same membership as the routing index, and the test oracle asserts their agreement at every indexed seqno after each operation. That missing-entry path itself still iterates retained requests; routing it through the index is a possible follow-up, not part of this change.

Functional coverage includes shared supporting-only wrappers, overlapping roles and namespaces, sparse/range and receipt-mode retargeting, successive secret recovery, partial exceptions, rejected/fork/duplicate/late replies, and ownership/cleanup. An independent forward-field oracle runs full-secret and latest-secret-only scenarios across 3,200 randomized operations, including null direct dependencies. The existing historical-cache e2e runner also exercises shared receipt/sparse consumers and receipt-handle retargeting.

Safety and compatibility

No public API or ledger-format change. Existing decryption, signature/seqno/view/fork checks, handle namespaces, store/returned-state ownership, expiry, LRU retention, byte accounting, fetch coalescing/retries, and scheduling are preserved; missing-entry handling is the base's #8527 behaviour. Existing allocation-hardening gaps in deadline/release/accounting bookkeeping are not folded into this change; routing remains synchronized with committed forward fields independently of caller accounting deltas.

The optimization avoids unrelated retained requests, not real fan-out or each affected request's range work. The index and transient snapshots add memory outside estimated ledger-entry bytes. At 29,200 retained three-entry requests, the benchmark has 87,600 buckets/memberships and 7,708,848 bytes of index value storage alone, excluding tree links, allocator overhead, request linkage, and snapshots. Whole-cache RSS was 79,200 KB versus 12,960 KB with no retention; this is not index-only RSS attribution. Retained requests in the unit benchmarks use metadata-only trusted wrappers; active replies are genuinely deserialised. Timing samples are descriptive, not statistical confidence claims.

Local validation (updated head 695e3b27c, merged with main at 9b0d37030): historical unit suite passes (45 cases, including the #8527 and #8549 cases from main, 439,222 assertions); both opt-in scaling benchmarks pass (4,301,445 and 1,347,301 assertions) with unchanged exact visit counts: at retained counts 0/7,300/14,600/29,200, four receipt or secret consumers always produce exactly four examined handles, and actual receipt and secret fan-out 1/10/100/1,000 produces exactly the matching handle count. The three clang-tidy findings from CI (modernize-use-nodiscard on has_secret_waiters/snapshot, modernize-pass-by-value on the request handle) are fixed and verified locally with clang-tidy 21.1.8 on the tidied historical_queries_utils.cpp/historical_queries_adapter.cpp translation units; remaining local findings are pre-existing base patterns accepted by CI's clang-tidy version. All 15 scripts/ci-checks.sh checks pass on the updated head. Combined logging/js_generic binaries build, and the registered historical_query_cache_test passes in 99.92s, including shared-consumer and retargeting coverage. The full warning-failing Sphinx build was run on the original stacked head; later updates change no documentation in this PR. Partial-exception regressions exercise post-publication std::bad_alloc and logic_error boundaries, not exhaustive global allocator fault injection.

Matched network sample (2026-10-08): measured at production pin 217189ef4 on base 3fcc0e6f1 (publication head 3124778b7), and not rerun for any later head. Since that pin, production changes only by the clang-tidy fixes above and by merging main, including #8527 and #8549. A fresh maintenance-only control and the indexed build used the original CCF7.0.17 release, identical frozen diagnostic/driver sources and build settings, 500 clients, 60s load plus 65s observation, and three nodes limited to 1 CPU/1GiB each. These are sequential shared-host samples, not randomized runs or equal achieved rates/retained shapes. Control run: 20261008T172123Z-baseline; indexed run: 20261008T172559Z-baseline.

Settled early is 10-20s; late is the last 10s of traffic. CPU values below are OS measurements per core per node, not phase-clock attribution.

Metric	Maintenance-only control, early -> late	With reply index, early -> late
Dispatch CPU (%)	8.219 -> 27.376	5.821 -> 5.636
Other-thread CPU (%)	15.413 -> 15.105	15.861 -> 15.168
Retained requests/node, mean	1395.8 -> 6027.1	1153.5 -> 5073.8
Consumer-stage actual handle visits/call	1327.965 -> 5961.369	4.556 -> 5.122
Secret-selection actual handle visits/call	1327.965 -> 5961.369	0 -> 0
Whole-reply handler mean WALL time (us)	111.351 -> 778.822	56.695 -> 57.139

The dispatch-CPU/reply-cost retention ramp seen in the control is removed in this short indexed sample, with bounded observed consumer work rather than unrelated retained-request traversal. Late genuine fan-out reached 77 handles; this is an observed maximum, not a cap. Legacy secret-selection visits were counted independently and matched its consumer-loop visits; indexed secret selection visited zero handles in every observed window. The protected counter remains the single latest-attempt requests_visited counter, not separate target/signature counters. Handler/phase WALL clocks include preemption and do not establish exact CPU attribution. Late cache-tick means were 8.810us (control) and 7.262us (index); both already include #8525 and avoid unrelated retained-state scans.

No throughput-win claim: successful write completions were 19,768 (control) versus 16,958 (index), a 14.2% lower total count. Measured load durations were 60.35741996765137s and 60.1469931602478s respectively; late successful writes/s were 344.576 and 329.338. Achieved traffic and retained shapes differ, so these are not equal-work or throughput-improvement results.

Startup/error caveat: the indexed run recorded 42 write-request TimeoutError outcomes in load.jsonl, all in the startup response window ending around t=5.980s (load.jsonl:7), with durations approximately 5.37-5.46s. There were zero commit-wait timeouts, workers continued, and the cause was not isolated or attributed to the index. All sampled health probes succeeded; no OOM, election, or deserialisation errors were observed.

Per-node peak RSS was 175.94-178.59MiB (control) versus 160.00-162.81MiB (index). Lower network RSS is not isolated index-memory attribution: achieved retention differs, and the unit metadata/memory qualifications above still apply. Report validation independently reconciled 774 reply windows, CPU/rate/visit/time totals, 222 evidence records, and canonical raw unit CSVs. These short-run findings make no conclusion about the October incident or the long 1800s expiry envelope.

[Copilot]

🔵 Needs a closer look

Core historical-ledger routing and secret-recovery bookkeeping warrant final human review despite comprehensive coverage.

0 open findings

What changed in this PR

Adds indexed routing for historical ledger replies, avoiding scans of unrelated cached requests while preserving receipt and ledger-secret handling.

Changes:

• Tracks direct, signature, and secret consumers per sequence number.
• Adds extensive unit, randomized, benchmark, and end-to-end coverage.
• Documents behavior, memory overhead, and the 7.0.19 change.

Custom instructions used

• .github/copilot-instructions.md
• .github/instructions/reviewing.instructions.md
• .github/instructions/changelog.instructions.md
• .github/skills/testing/SKILL.md
• .github/skills/formatting-and-linting/SKILL.md

File	Description
src/​node/​historical_queries.h	Implements indexed reply-consumer routing.
src/​node/​test/​historical_queries.cpp	Tests invariants, failures, fan-out, and scaling.
tests/​historical_query_cache.py	Adds shared-consumer and retargeting e2e coverage.
doc/​build_apps/​api.rst	Documents routing and memory behavior.
CHANGELOG.md	Records the optimization for 7.0.19.

🧠 Review effort: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.