usb: pipelined register writes for the bring-up; Jaguar3 stage timing - #417
Conversation
PR Summary by QodoPipeline Jaguar3 USB bring-up writes and add transfer timing
AI Description
Diagram
High-Level Assessment
Files changed (13)
|
…e timing Jaguar3 InitWrite is ~14k synchronous EP0 round trips and nothing else. InitTimer now reports the control-transfer count per stage, and IRtlTransport::write_batch_begin/end pipelines the bring-up's register writes behind EP0's in-order completion: warm InitWrite 1.30 -> 0.65 s, cold 2.04 -> ~0.7 s on one drone-side unit. The RX-only Init path opens no batch yet and is unchanged. Submitted upstream as OpenIPC#417. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JiAS32956Z3cmTbnmcXYkT
Code Review by Qodo
1. USB transfers outlive their context
|
josephnef
left a comment
There was a problem hiding this comment.
The mechanism is sound and the reasoning behind it is the strongest part of the PR — never zeroing _aw_inflight by hand, retiring unreapable slots rather than handing libusb a dangling transfer, and WriteBatchScope's destructor closing the batch on a throw are all correct, and those comments earn their length. EP0 submission ordering does make the read-behind-writes trick safe, and the tx_async/tx_sync/write_bytes flushes cover the bulk paths.
I'd still hold it. Two of the four substantive issues are cross-variant validation gaps rather than logic errors, which on a bring-up-critical path is the expensive kind: every number in this PR is one 8812EU, and two of the changes ship to 8822C/8812CU as well.
1. The 8822C settle delays don't flush (inline on Halrf8822e.cpp). Halrf8822c::delay_ms is still static and untouched, while the batch scope covers dac_calibrate() and run_iqk() on both variants.
2. The write-only RF-table load is variant-agnostic (inline on HalJaguar3.cpp). The argument is airtight given readback 0 — an RMW that reads 0 writes exactly data, so the two are bit-identical — so the entire risk is whether 8822C's direct window behaves the same, and that's unmeasured.
3. _aw_abandoned is set but never acted on mid-batch (inline on UsbTransport.cpp).
4. libusb_alloc_transfer(0) isn't null-checked (inline).
The rebase is not mechanical. The PR is CONFLICTING. Post-#415 there is no IRtlTransport — RtlAdapter forwards to devourer::ITransport (src/Transport.h), so the three virtuals have to land there, and the IRtlTransport::write_batch_begin doc references in RtlAdapter.h, HalJaguar3.cpp and src/jaguar3/CLAUDE.md need retargeting.
Validation, by the repo's own bar. Neither the pipelining nor the RF-table rewrite has an on-air or regress.py run, and #417 changes how the RF radio tables are programmed on both dies. I'd want a 2x2 on an 8822EU and an 8822CU, plus tests/tx_teardown_asan.sh over the new drain/cancel paths, before this merges — the description already flags that those haven't been run, which is the right call, but it's also the reason to hold.
Everything else is inline and minor.
InitWrite on the RTL8812EU is ~14k synchronous EP0 round trips and nothing else. InitTimer now brackets every Jaguar3 rtw_hal_init / InitWrite stage and reports the control-transfer count per stage (UsbXferCount.h), which is the unit the bring-up is actually paid in. init.timing gains an `xfers` field; docs/logging.md carries the schema. A transfer costs 76-80 us synchronous on an embedded host (ssc338q) and ~27 us pipelined 8-deep. EP0 completes URBs in submission order, so IRtlTransport::write_batch_begin/end lets UsbTransport queue writes as async URBs and wait only on reads (submitted behind the queue), bulk transfers and flush_writes. Jaguar3 InitWrite runs its whole bring-up in one batch (RAII scope, closed before the coex thread starts); the ms-scale settle delays flush first. The three methods default to no-ops, so PCIe is unaffected. Batches are single-threaded by contract: open one only while no other thread touches the transport, and close it before any worker starts. A drain that times out cancels what is still submitted and keeps pumping for the cancellations rather than declaring the queue empty: an in-flight count zeroed by hand goes negative on the late callback, which silently disables every later drain, returns a slot to the free list twice, and leaves the destructor freeing a transfer libusb still owns. Slots that genuinely cannot be reaped (dead event loop, yanked device) are retired for the session and leaked at teardown instead. Separately, the RF radio-table load is write-only: bits [31:20] of the direct window read back 0 for all 1540 entries, cold and warm, so the vendor's MASK20BITS read-modify-write preserved nothing while paying a synchronous read per entry -- about half the RF-table stage. Measured on one drone-side unit: warm InitWrite 1.30 -> 0.65 s, cold 2.04 -> ~0.7 s. Init (RX-only) opens no batch yet and is unmeasured on a ground-station card, so the RX bring-up path is unchanged by this commit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JiAS32956Z3cmTbnmcXYkT
…sh, per-transport xfer count Addresses the review on the pipelined-write bring-up: - Halrf8822c::delay_ms is no longer static and drains the queue before sleeping, like the 8822E one: WriteBatchScope covers DACK/IQK on both dies, and a settle that sleeps over queued writes is no settle. - The write-only RF-table load is now measured on both dies: tests/j3_rf_window_readback.sh dumps both path windows after a bring-up (chipstate --init --peek ...:4) and histograms bits [31:20]; 8812CU and 8812EU both read 0 for all 512 words, so the plain write is bit-identical to the vendor RMW there. - flush_writes closes the batch when it retires slots: with those slots never returning, the in-flight count could not reach zero again and every later register access walked take-slot -> wait -> drain for the rest of the bring-up. Synchronous from that point instead. - The completion callback touches only an AsyncPool shared (shared_ptr) between the transport and every slot, never the transport: a slot the destructor had to leak can outlive the transport, and its callback may still fire through a libusb context another adapter keeps pumping. - write_batch_begin builds its pool transactionally: a null libusb_alloc_transfer tears the partial pool down and leaves the session synchronous, so no null transfer reaches fill/submit. - The transfer counter is per transport instance (ITransport::ctrl_xfers, RtlAdapter::ctrl_xfers), not process-wide; InitTimer takes it as an optional counter and emits `xfers` only when given one, so two adapters in one process no longer cross-attribute, and PCIe timers stay silent rather than reporting 0. UsbXferCount.h is gone. - A failed pipelined read32_wide is logged rather than silently returned as the all-ones sentinel; the drain-timeout comment names the real cause (an unpumped event loop, since USB_TIMEOUT is 500 ms). - Own header first in HalJaguar3.cpp / RtlJaguar3Device.cpp; the IRtlTransport references are retargeted at ITransport (src/Transport.h) after the rename; src/jaguar3/CLAUDE.md keeps only the Jaguar3-specific facts and points at the transport header for the batching contract. - chipstate: --init followed by --peek/--poke runs the ops on the configured chip, and --peek ...:4 reads aligned 32-bit words (the BB/RF windows answer 32-bit reads only). - tests/regress.py learns 0bda:b812 (CF-924AC V2), the bench's recommended ground station, which its DUT table did not list. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ne Jaguar3 DUT Per rep and per tree: time to first TX submit, bulk-OUT failures and failed synchronous register reads from a timed txdemo flood. Trees alternate rep by rep so a drift in the unit lands on both sides. Used to show the 8812EU's 5 GHz bulk-OUT timeouts are present on master at the same rate as on the pipelined bring-up. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
/review |
38e17e1 to
ecce836
Compare
|
Code review by qodo was updated up to the latest commit ecce836 |
… it tests - Every settle drains the pipelined queue first, the microsecond ones included (delay_us on both Halrf8822c and Halrf8822e, now instance methods; the write_bb / rf_writer µs table markers). The µs sites include 2 ms and 10 ms waits, so the ms/µs split was not a safe line. The drain is free on an empty queue and bounded by its depth otherwise. - tests/j3_rf_window_readback.sh gains the leg that decides: after the histogram it pokes a sample of window words with bits [31:20] SET (low 20 bits unchanged, so the RF register keeps its value), reads them back and restores them. Bits that are storage read back set; bits the MASK20BITS RMW could never have preserved read back 0. This does not depend on which bring-up ran first, unlike the histogram alone, and the exit code now carries the verdict. - chipstate --init + ops releases the device object before the ops, so they cannot interleave with the coex thread's register writes; 0xb812 joins its discovery list. - tests/j3_tx_flood_ab.sh captures txdemo's real exit status instead of the status of a shell negation. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
/review |
|
Code review by qodo was updated up to the latest commit 47e741c |
…circuit, one-shot scope - A pipelined write that cannot be submitted (no usable slot, submit rejected) now falls through to the synchronous control transfer. EP0 keeps submission order, so it lands behind whatever is still queued and no register write is dropped just because the caller ignores the bool. - flush_writes returns at once after a drain has retired slots: the in-flight count stays positive for good by design, and re-draining it on every later flush (bulk sends, batch close, destruction) would only repeat the timeout + cancel turns. - async_take_slot hands out nothing after a recovery flush that abandoned the queue or closed the batch, even when some cancellations did return a slot, so the caller takes the synchronous path rather than queueing another transfer behind the stuck ones. - WriteBatchScope closes once: end() disarms the destructor, so nothing touches the transport's batch state after the coex thread that shares it has started, on the normal path and when unwinding. - The j3init timing closes before the coex thread starts; it shares the adapter's transfer counter, so the final stage and total no longer count that thread's register and H2C traffic. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
/review |
|
Code review by qodo was updated up to the latest commit cc74109 |
… trigger settle - The completion callback normally runs on the submitting thread (this transport pumps its own events while it waits), but a second adapter sharing the libusb context can pump it from another thread and run the callback the moment libusb has the transfer. The pool counters are now atomic, the free list sits under a mutex, and a slot is accounted as in flight BEFORE it is submitted (rolled back if libusb refuses it), so the callback can never observe a completed slot the submitter has not yet counted. `done` is published after status/actual. - efuse_phys_read_8822e drains the queue between the EFC trigger write and its 50 µs settle; the CW-tone arm retry drains before its back-off. The remaining sleeps in the batched bring-up (power-on, H2C box, DLFW polls) read before they sleep, and a read is ordered behind the queued writes on EP0, so they need no drain. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
/review |
|
Code review by qodo was updated up to the latest commit 796047b |
…harness exit codes - async_write_cb finalizes the result and all accounting before the slot is pushed onto the free list, so a taker can never reset a slot the callback is still writing to. - async_wait_progress waits on a real 2 s steady_clock deadline for this pool's completion counter, not on a count of event-loop turns: on a shared libusb context another adapter's completions make every handle_events return at once, and counting turns would declare a healthy queue stuck and cancel it. - chipstate refuses to print a poke the chip did not take (exit 4), so a following peek cannot read as a verdict about bits never written. - j3_tx_flood_ab.sh exits non-zero after any unexpected txdemo exit, while still reporting every rep. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
/review |
|
Code review by qodo was updated up to the latest commit bb8bbe3 |
…ot; chipstate rejects --no-claim with --init - Each slot carries cb_busy: set on callback entry, cleared as its very last store. `inflight` has to clear before the free-list push so a taker sees a finished slot, so it cannot double as the destructor's "safe to free" signal; the destructor now waits out a callback that another adapter's pump thread is still running before freeing. - chipstate refuses --no-claim together with --init: the raw-adapter path cannot bring the chip up, so the combination would have run the ops on an uninitialised device while looking like it asked for a bring-up. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
/review |
|
Code review by qodo was updated up to the latest commit 77b777c |
write_batch_begin opens the caller's batch (and resets its verdict) before deciding whether pipelined submission is possible; a failed pool allocation or a queue retired in an earlier session leaves the writes synchronous, and a failed one still counts for write_batch_end. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
/review |
|
Code review by qodo was updated up to the latest commit 04c0cd2 |
Each write_batch_begin bumps a generation; a slot records the generation it was submitted under, and the callback counts a failed write only for the current one. A slot retired by an earlier drain was counted when it was retired; if its completion arrives during a later batch it changes nothing. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
/review |
|
Code review by qodo was updated up to the latest commit 6275f80 |
…l mutex; re-init joins coex - write_batch_begin/end keep a depth: the outermost pair owns the verdict and the close, an inner end leaves pipelining on. - The callback's busy-clear and free-list push are one critical section under the pool mutex, and the destructor decides under the same mutex (pulling a completed slot off the free list before freeing it), so it can neither free a slot mid-handoff nor leave a dangling pointer on the list. - A second InitWrite on a live device stops and joins the previous coex thread before opening its batch. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
/review |
|
Code review by qodo was updated up to the latest commit 814cb6c |
The rollback guard is installed right after the previous coex thread is joined, before any bring-up step can throw, so a failed re-init cannot leave the readiness flag from the previous successful one in place. The flag is set provisionally where it was and committed after a clean batch close, as before. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
/review |
|
Code review by qodo was updated up to the latest commit 2375a45 |
A PCIe transport's timer omits the xfers field instead of reporting 0, matching the documented schema. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
/review |
|
Code review by qodo was updated up to the latest commit dd47acb |
… atomic A bring-up while the RX loop runs would open a single-threaded batch over a transport that loop's phydm worker still uses, and the loop belongs to the caller's thread, so InitWrite throws instead. _coex_stop is written by Stop, the destructor and now the re-init path while the coex loop reads it: std::atomic<bool>, not volatile. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
/review |
|
Code review by qodo was updated up to the latest commit 7ff9657 |
Superseded: every finding is addressed on the current head (rebased, fixed, validated on both Jaguar3 dies); merging under the skip-qodo-gate escape hatch by maintainer decision.
Jaguar3
InitWriteon an RTL8812EU is ~14k synchronous EP0 round trips andessentially nothing else — no waiting on hardware, just USB control-transfer
latency. This measures that, then removes most of it.
Measuring it
InitTimeralready bracketed init stages with a duration. It now also reportsthe number of USB vendor control transfers each stage spent
(
ITransport::ctrl_xfers, a per-transport-instance counter kept byUsbTransportand handed toInitTimer), which is the unit the bring-up isactually paid in.
init.timinggains anxfersfield where a counter isattached — the Jaguar3 stages;
docs/logging.mdcarries the schema.Every Jaguar3
rtw_hal_init/InitWritestage is bracketed.Removing it
A control transfer costs 76–80 µs synchronous on an embedded host (ssc338q)
and ~27 µs pipelined 8-deep. EP0 completes URBs in submission order, so a
queue of pending writes followed by a read behaves exactly like the
synchronous sequence — the host just doesn't sit through a round trip per
write.
ITransport(src/Transport.h) gainswrite_batch_begin/write_batch_end/flush_writes. Inside a batch,UsbTransportsubmits register writes asasync URBs; a read (submitted behind the queue and waited on its own
completion), a bulk transfer, or an explicit flush is what waits. Jaguar3
InitWriteruns its whole bring-up in one RAII scope, ended before the coexthread starts. Every settle delay flushes first, µs ones included, on both
dies (
delay_us/delay_msonHalrf8822candHalrf8822e, the tabledelay markers, the efuse power-cut): the drain is free on an empty queue
and bounded by its depth otherwise.
All three methods default to no-ops, so
PcieTransportand everygeneration other than Jaguar3 are untouched.
Contract
Batches are single-threaded: open one only while no other thread touches the
transport, and close it before any worker starts. This is enforced by
convention, not by an assert — worth a reviewer's attention.
Failure paths
A drain that times out cancels what is still submitted and keeps pumping for
the cancellations rather than declaring the queue empty. Zeroing the in-flight
count by hand would let a late callback drive it negative (silently disabling
every later drain), return a slot to the free list twice, and leave the
destructor calling
libusb_free_transferon a transfer libusb still owns.Slots that genuinely cannot be reaped — dead event loop, yanked device — are
retired for the session (the batch closes, so the rest of the session runs
synchronously) and leaked at teardown instead, which is the lesser evil
against handing libusb a dangling transfer. The completion callback writes
only to an
AsyncPoolshared between the transport and every slot, so aleaked slot whose callback fires later through a still-pumped libusb
context touches the pool it keeps alive, never a freed transport.
Separately: the RF radio-table load is write-only
Bits [31:20] of the direct window are not storage.
tests/j3_rf_window_readback.shmeasures it two ways per die, both through
chipstate --init: thepost-bring-up histogram (512/512 words read 0 on 8812CU and 8812EU), and
the leg that decides — 16 window words per die poked with the high 12 bits
set (low 20 bits unchanged, then restored) all read back 0. Bits that
were storage would read back set; these do not, on either die, regardless
of which bring-up ran first. So the vendor's
MASK20BITSread-modify-write preserved nothing while paying a synchronous read per
entry, about half the RF-table stage, and the plain write is bit-identical.
Results and limits
Measured on one drone-side 8812EU (ssc338q host): warm
InitWrite1.30 → 0.65 s, cold 2.04 → ~0.7 s. On the x86 bench, alternating
master-vs-branch floods (
tests/j3_tx_flood_ab.sh, 3 reps per tree, timefrom exec to first TX submit):
0bda:c812), ch360bda:a81a), ch36(Final head, with the µs-settle flushes in. An earlier round without them
read CU 846–874 / EU 633–1081 ms on the branch — the extra drains cost
nothing measurable.)
One unit of each die, one host — the transfer-count reduction is
deterministic, the wall-clock figure is not replicated across parts.
Init(RX-only) opens no batch yet and is unmeasured on a ground-stationcard, so the RX bring-up path is unchanged by this PR and gets none of
the speedup.
Validation
tests/regress.py2x2, devourer→devourer cells (kernel cells read 0 onthis rig: no vendor module is built for the receivers today):
8100 / 8000.
run read 2500/15 s and did not reproduce). ch6 reads 0 on this module
with either tree — the documented 8822E 2.4 GHz TX kernel-parity cell
(
docs/8822e-quirks.md).bulk_send EP 5 FAIL rc=-7and one failed coexBTC-window read per 15 s flood on 5 GHz. Same on master (12/12/16 vs
6/8/13 bulk failures across the alternating reps of the final round;
9/10/11 vs 10/18/12 in the earlier one), so it is the unit's known 5 GHz
NAK behaviour, not this change.
tests/tx_teardown_asan.sh(ASan build, max-duty + aggregation + gap-2000,3 reps each) on the 8812CU: 9/9 clean exits. 8812EU (ch36): 9/9 clean exits.
cmake --build+ctestgreen (66/66).Also folded in:
tests/regress.pylearns0bda:b812(the CF-924AC V2 thebench notes recommend as ground station, which its DUT table did not list),
and
chipstate --initfollowed by--peek/--pokeruns the ops on theconfigured chip after releasing the device object (its destructor joins the
coex thread and does not de-init the chip), with
:4for 32-bit-word reads(the BB/RF windows answer 32-bit reads only).
chipstatealso discovers0xb812without--pid.🤖 Generated with Claude Code