Repository navigation
(16) capture: in-dataplane pcapng packet capture - #1905
Draft
daniel-noland wants to merge 7 commits into
Draft
daniel-noland wants to merge 7 commits into
daniel-noland wants to merge 7 commits into
Conversation
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Comment |
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
This was referenced Oct 12, 2026
daniel-noland
force-pushed
the
pr/daniel-noland/capture
branch
from
October 16, 2026 05:47
ef57775 to
c7bc0e6
Compare
daniel-noland
force-pushed
the
split/dpdk-core/07-flow
branch
from
October 16, 2026 05:47
403fe04 to
7f06ff9
Compare
DPDK owns the NICs, so host tools never see dataplane traffic. Add a `capture` crate that records packets from inside the pipeline: - `CaptureRecord` copies a packet's parsed headers and the metadata a capture wants (direction, interfaces, VNIs, done reason, flow key, thread). It holds no buffer and no `FlowInfo`, so it can cross cores and cannot keep a flow alive. - `CaptureStage` is a network function that sends records to a `CaptureSink`, asking the sink first so sampling and filtering happen before anything is copied. - `CaptureWriter` writes records as pcapng via `pcap-file` (pinned to 3.0.0-rc.3): one interface per dataplane interface, the direction in `epb_flags`, and the metadata as a `key=value` comment that stock Wireshark can filter on. Frames carry only the rebuilt headers, so the captured length is shorter than the original length. - `CaptureNames` supplies interface and VPC names to the writer. A capture shows interfaces by name rather than by kernel ifindex, which means nothing once a device is unbound from the kernel. It reports VNIs as VNIs and adds a VPC name only where one is known, since a VNI is not a VPC. A property test checks that every written frame is the prefix of the packet's wire bytes and that each block carries its record's interface, timestamp, direction and comment. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Daniel Noland <daniel@githedgehog.com>
loom 0.7's channel is unbounded only, never reports hang-up (a recv after the last sender drops blocks forever), treats a send to a dropped receiver as a leaked message, and leaves recv_timeout unimplemented. So code using the facade's `mpsc::sync_channel` did not compile under the loom backend at all. Replace the loom backend's `mpsc` with a channel built on loom's Mutex and Condvar, which loom models completely, and add `sync_channel` to it. `recv_timeout` waits like `recv`, matching shuttle, so under either model checker a timeout never fires. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Daniel Noland <daniel@githedgehog.com>
Add `CaptureController`, which starts one capture at a time and hands each worker's capture stages a `ControlledSink`: - A worker with no capture running pays one relaxed load per burst (`CaptureSink::begin_burst`, new, default true). - A running capture admits packets by direction, samples one in N per stage (no shared counter), and takes a unit of a shared packet budget before anything is copied. - Admitted records go to a bounded queue with `try_send`; a full queue drops the record and counts it, so workers never block. - A dedicated `capture-writer` thread owns the pcapng writer and is the only thing that touches the output. A capture ends on its budget, its deadline, `stop`, or a write failure. Ending swaps the remaining budget to zero, which closes admission atomically; the writer then drains until received + dropped equals admitted, so no record a worker was mid-sending is lost or uncounted. `stop` waits for the file to be complete, and the report (admitted, written, dropped, why it ended) is also written into the file as an interface statistics comment, since pcapng has no per-file place for it once the section header is out. Shuttle tests race stop against workers with a one-slot queue and check written + dropped == admitted <= budget and that the file holds exactly the written packets; real-time tests cover sampling, the deadline, and an idle controller. The controller tests are not run under loom: an exhaustive search over the writer thread costs far more CI time than shuttle leaves for it to find. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Daniel Noland <daniel@githedgehog.com>
Create a `CaptureController` with the router and put a capture stage first in every worker's pipeline (the packet as received) and one right after `Egress` (the packet as it will leave), so the two views of a packet can be compared. With no capture running each stage costs one relaxed load per burst. Nothing can start a capture yet; that arrives with the CLI. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Daniel Noland <daniel@githedgehog.com>
Add the commands:
capture start [direction=ingress|egress|both] [packets=N]
[seconds=N] [sample=N]
capture stop
show capture
`capture start` defaults to both directions, 10000 packets, 60 seconds
and every packet. Captures go to /var/run/dataplane/capture (a stopgap
until the pod has a persisted location), one file per capture named by
its UTC start time; `capture stop` waits for the file to be complete and
reports it with the admitted/written/dropped counts.
- capture: `CaptureDirectory` creates the file (never overwriting one)
and removes it again if the capture cannot start. It refuses a second
capture before creating anything.
- routing: the CLI handler names interfaces from the router's interface
table and VPCs from the applied gateway config, so a capture shows
`iif=<name>` and `src_vni=<vni> src_vpc=<name>` rather than kernel
ifindexes or a VNI posing as a VPC.
- cli: new actions, `CaptureDirection`, `capture_*` request args, and a
`CliError::Capture`. The client rejects zero counts; the server
checks again.
An end-to-end routing test drives a real router over its CLI socket:
start, refuse a second start, record a burst through a capture stage,
stop, and read the file back to check the interface carries the router's
name for it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Daniel Noland <daniel@githedgehog.com>
The parser stops at the VXLAN header, and encapsulation writes the inner headers into the payload, so at both capture points an encapsulated packet's `headers` are the outer ones and its inner frame is in the payload. A headers-only record therefore kept exactly the part nobody wants: outer headers that repeat almost unchanged from packet to packet. For a VXLAN packet, parse the inner frame out of the payload and record those headers instead; the record's original length is the inner frame's. This happens on the packet's thread, but only for packets a capture has admitted. The outer VNI, the one outer field that matters (on ingress it is the only thing identifying the VPC), goes into the comment as `vxlan_vni=`, beside `frame=inner|outer`. An inner frame that does not parse falls back to the outer headers, marked `frame=outer`. The round-trip property now wraps a share of its generated packets in VXLAN and checks the written frame against the inner frame's wire bytes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Daniel Noland <daniel@githedgehog.com>
capture start [ingress=*|VPC] [egress=*|VPC] [packets=N]
[seconds=N] [sample=N]
- no filter: both directions, every packet (as before)
- `ingress=V`: packets entering from V, matched on the VXLAN VNI, since
nothing else identifies the VPC before decapsulation
- `egress=W`: packets leaving towards W, matched on `dst_vpcd`
- both: the V -> W conversation, matched at egress on `src_vpcd` and
`dst_vpcd`, since only egress knows both ends
`*` matches any VPC, including packets with none; a named VPC never
matches a packet without one. The CLI resolves names against the
applied config and refuses an unknown one, listing the known ones. The
filter runs before sampling, so `sample=N` picks among matching packets.
This replaces the `direction=` argument, which never left this branch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Daniel Noland <daniel@githedgehog.com>
daniel-noland
force-pushed
the
split/dpdk-core/07-flow
branch
from
October 16, 2026 08:43
7f06ff9 to
c2d4969
Compare
daniel-noland
force-pushed
the
pr/daniel-noland/capture
branch
from
October 16, 2026 08:43
c7bc0e6 to
4207243
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
DPDK owns the NICs, so
tcpdumpon the host sees nothing of the dataplane's traffic. This adds packet capture from inside the pipeline, started from the CLI and written as pcapng.ingress=Vegress=Wdst_vpcdsrc_vpcdanddst_vpcdDefaults: 10000 packets, 60 s, every matching packet.
*matches any VPC, including none. Files go to/var/run/dataplane/capture/capture-<UTC start>.pcapng. That location is a stopgap until the pod has a persisted one.Design
Mbufstays!Send). A single writer thread rebuilds the frames, so a capture has no payload: pcapng shows each packet with captured length < original length.key=valuecomment that stock Wireshark can filter on (frame.comment contains "done=AclDropped"):direction= frame= vxlan_vni= thread= done= iif= oif= src_vni= src_vpc= dst_vni= dst_vpc= flow=try_sendon a bounded queue. A full queue drops the record and counts it.capture stopreturns once the file is complete. The report is also written into the file.Commits
capture: record, pcapng writer (pcap-filepinned to=3.0.0-rc.3), pipeline stageconcurrency: the loom backend gets a boundedsync_channelbuilt on loom'sMutex/Condvar. loom's own channel has no bounded flavour and doesn't model hang-up. Without this the workspace's loom build doesn't compile.capture: controller and writer threaddataplane: capture stages first in the pipeline and right afterEgresscli,routing: the commandscapture: inner frames of VXLAN packetscapture: VPC filterTesting
pcap-file. It checks every frame byte-for-byte against the packet's (inner) wire bytes, plus interface, timestamp, direction and comment.stopagainst workers with a one-slot queue: written + dropped = admitted ≤ budget, and the file holds exactly the written packets. I checked that the schedules aren't vacuous: they cover stops before, during and after admission, with drops. The controller tests are deliberately not run under loom (CI time).Unrelated:
test_cli_handling(routing) failed 4× in a row on its FRR-config check at exactly 3.018 s under load average ~11. It then passed repeatedly on identical code, so it looks like a timeout flake.🤖 Generated with Claude Code