Skip to content

(16) capture: in-dataplane pcapng packet capture - #1905

Draft
daniel-noland wants to merge 7 commits into
split/dpdk-core/07-flowfrom
pr/daniel-noland/capture
Draft

daniel-noland wants to merge 7 commits into
split/dpdk-core/07-flowfrom
pr/daniel-noland/capture

Conversation

@daniel-noland

Copy link
Copy Markdown
Collaborator

DPDK owns the NICs, so tcpdump on the host sees nothing of the dataplane's traffic. This adds packet capture from inside the pipeline, started from the CLI and written as pcapng.

capture start [ingress=*|VPC] [egress=*|VPC] [packets=N] [seconds=N] [sample=N]
capture stop
show capture
Given Captures Matched on
neither both directions, everything —
ingress=V packets entering from V outer VXLAN VNI (nothing else identifies the VPC before decap)
egress=W packets leaving towards W dst_vpcd
both the V → W conversation, at egress only src_vpcd and dst_vpcd

Defaults: 10000 packets, 60 s, every matching packet. * matches any VPC, including none. Files go to /var/run/dataplane/capture/capture-<UTC start>.pcapng. That location is a stopgap until the pod has a persisted one.

Design

  • Headers, not buffers. A capture stage copies a packet's parsed headers and selected metadata into a record. Buffers never leave their core (Mbuf stays !Send). A single writer thread rebuilds the frames, so a capture has no payload: pcapng shows each packet with captured length < original length.
  • Inner frames. For a VXLAN packet the record keeps the inner headers. The outer ones repeat almost unchanged from packet to packet. The outer VNI goes into the comment.
  • Metadata as text. Each packet carries a key=value comment that stock Wireshark can filter on (frame.comment contains "done=AclDropped"):
    direction= frame= vxlan_vni= thread= done= iif= oif= src_vni= src_vpc= dst_vni= dst_vpc= flow=
  • Names, not numbers. Interfaces are named as the router names them, not by kernel ifindex, which means nothing once a device is unbound. A VNI is reported as a VNI, and a VPC name is added only where the config defines one.
  • Workers never block. With no capture running, a stage costs one relaxed load per burst. A running capture filters, then samples (one counter per stage, nothing shared), then takes a unit of a shared packet budget, and only then copies. Records go out with try_send on a bounded queue. A full queue drops the record and counts it.
  • Nothing lost silently. Ending a capture swaps the remaining budget to zero, which closes admission atomically. The writer then drains until written + dropped = admitted. capture stop returns once the file is complete. The report is also written into the file.

Commits

  1. capture: record, pcapng writer (pcap-file pinned to =3.0.0-rc.3), pipeline stage
  2. concurrency: the loom backend gets a bounded sync_channel built on loom's Mutex/Condvar. loom's own channel has no bounded flavour and doesn't model hang-up. Without this the workspace's loom build doesn't compile.
  3. capture: controller and writer thread
  4. dataplane: capture stages first in the pipeline and right after Egress
  5. cli, routing: the commands
  6. capture: inner frames of VXLAN packets
  7. capture: VPC filter

Testing

  • A property test (bolero) writes generated packets, some wrapped in VXLAN, and reads them back with pcap-file. It checks every frame byte-for-byte against the packet's (inner) wire bytes, plus interface, timestamp, direction and comment.
  • Shuttle tests race stop against workers with a one-slot queue: written + dropped = admitted ≤ budget, and the file holds exactly the written packets. I checked that the schedules aren't vacuous: they cover stops before, during and after admission, with drops. The controller tests are deliberately not run under loom (CI time).
  • An end-to-end routing test drives a real router over its CLI socket. It starts a capture, checks that a second start and an unknown VPC are refused, records a burst, stops, and reads the file back to check the interface name.
  • Not yet run against real traffic. It needs the test bench.

Unrelated: test_cli_handling (routing) failed 4× in a row on its FRR-config check at exactly 3.018 s under load average ~11. It then passed repeatedly on identical code, so it looks like a timeout flake.

🤖 Generated with Claude Code

@coderabbitai

coderabbitai Bot commented Oct 11, 2026 •

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@codecov

codecov Bot commented Oct 11, 2026

Copy link
Copy Markdown

daniel-noland and others added 7 commits October 16, 2026 01:58
DPDK owns the NICs, so host tools never see dataplane traffic.  Add a
`capture` crate that records packets from inside the pipeline:

- `CaptureRecord` copies a packet's parsed headers and the metadata a
  capture wants (direction, interfaces, VNIs, done reason, flow key,
  thread).  It holds no buffer and no `FlowInfo`, so it can cross cores
  and cannot keep a flow alive.
- `CaptureStage` is a network function that sends records to a
  `CaptureSink`, asking the sink first so sampling and filtering happen
  before anything is copied.
- `CaptureWriter` writes records as pcapng via `pcap-file` (pinned to
  3.0.0-rc.3): one interface per dataplane interface, the direction in
  `epb_flags`, and the metadata as a `key=value` comment that stock
  Wireshark can filter on.  Frames carry only the rebuilt headers, so
  the captured length is shorter than the original length.
- `CaptureNames` supplies interface and VPC names to the writer.  A
  capture shows interfaces by name rather than by kernel ifindex, which
  means nothing once a device is unbound from the kernel.  It reports
  VNIs as VNIs and adds a VPC name only where one is known, since a VNI
  is not a VPC.

A property test checks that every written frame is the prefix of the
packet's wire bytes and that each block carries its record's interface,
timestamp, direction and comment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Daniel Noland <daniel@githedgehog.com>
loom 0.7's channel is unbounded only, never reports hang-up (a recv
after the last sender drops blocks forever), treats a send to a dropped
receiver as a leaked message, and leaves recv_timeout unimplemented.
So code using the facade's `mpsc::sync_channel` did not compile under
the loom backend at all.

Replace the loom backend's `mpsc` with a channel built on loom's Mutex
and Condvar, which loom models completely, and add `sync_channel` to it.
`recv_timeout` waits like `recv`, matching shuttle, so under either
model checker a timeout never fires.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Daniel Noland <daniel@githedgehog.com>
Add `CaptureController`, which starts one capture at a time and hands
each worker's capture stages a `ControlledSink`:

- A worker with no capture running pays one relaxed load per burst
  (`CaptureSink::begin_burst`, new, default true).
- A running capture admits packets by direction, samples one in N per
  stage (no shared counter), and takes a unit of a shared packet budget
  before anything is copied.
- Admitted records go to a bounded queue with `try_send`; a full queue
  drops the record and counts it, so workers never block.
- A dedicated `capture-writer` thread owns the pcapng writer and is the
  only thing that touches the output.

A capture ends on its budget, its deadline, `stop`, or a write failure.
Ending swaps the remaining budget to zero, which closes admission
atomically; the writer then drains until received + dropped equals
admitted, so no record a worker was mid-sending is lost or uncounted.
`stop` waits for the file to be complete, and the report (admitted,
written, dropped, why it ended) is also written into the file as an
interface statistics comment, since pcapng has no per-file place for it
once the section header is out.

Shuttle tests race stop against workers with a one-slot queue and check
written + dropped == admitted <= budget and that the file holds exactly
the written packets; real-time tests cover sampling, the deadline, and
an idle controller.  The controller tests are not run under loom: an
exhaustive search over the writer thread costs far more CI time than
shuttle leaves for it to find.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Daniel Noland <daniel@githedgehog.com>
Create a `CaptureController` with the router and put a capture stage
first in every worker's pipeline (the packet as received) and one right
after `Egress` (the packet as it will leave), so the two views of a
packet can be compared.

With no capture running each stage costs one relaxed load per burst.
Nothing can start a capture yet; that arrives with the CLI.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Daniel Noland <daniel@githedgehog.com>
Add the commands:

    capture start [direction=ingress|egress|both] [packets=N]
                  [seconds=N] [sample=N]
    capture stop
    show capture

`capture start` defaults to both directions, 10000 packets, 60 seconds
and every packet.  Captures go to /var/run/dataplane/capture (a stopgap
until the pod has a persisted location), one file per capture named by
its UTC start time; `capture stop` waits for the file to be complete and
reports it with the admitted/written/dropped counts.

- capture: `CaptureDirectory` creates the file (never overwriting one)
  and removes it again if the capture cannot start.  It refuses a second
  capture before creating anything.
- routing: the CLI handler names interfaces from the router's interface
  table and VPCs from the applied gateway config, so a capture shows
  `iif=<name>` and `src_vni=<vni> src_vpc=<name>` rather than kernel
  ifindexes or a VNI posing as a VPC.
- cli: new actions, `CaptureDirection`, `capture_*` request args, and a
  `CliError::Capture`.  The client rejects zero counts; the server
  checks again.

An end-to-end routing test drives a real router over its CLI socket:
start, refuse a second start, record a burst through a capture stage,
stop, and read the file back to check the interface carries the router's
name for it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Daniel Noland <daniel@githedgehog.com>
The parser stops at the VXLAN header, and encapsulation writes the inner
headers into the payload, so at both capture points an encapsulated
packet's `headers` are the outer ones and its inner frame is in the
payload.  A headers-only record therefore kept exactly the part nobody
wants: outer headers that repeat almost unchanged from packet to packet.

For a VXLAN packet, parse the inner frame out of the payload and record
those headers instead; the record's original length is the inner
frame's.  This happens on the packet's thread, but only for packets a
capture has admitted.  The outer VNI, the one outer field that matters
(on ingress it is the only thing identifying the VPC), goes into the
comment as `vxlan_vni=`, beside `frame=inner|outer`.  An inner frame
that does not parse falls back to the outer headers, marked
`frame=outer`.

The round-trip property now wraps a share of its generated packets in
VXLAN and checks the written frame against the inner frame's wire bytes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Daniel Noland <daniel@githedgehog.com>
    capture start [ingress=*|VPC] [egress=*|VPC] [packets=N]
                  [seconds=N] [sample=N]

- no filter: both directions, every packet (as before)
- `ingress=V`: packets entering from V, matched on the VXLAN VNI, since
  nothing else identifies the VPC before decapsulation
- `egress=W`: packets leaving towards W, matched on `dst_vpcd`
- both: the V -> W conversation, matched at egress on `src_vpcd` and
  `dst_vpcd`, since only egress knows both ends

`*` matches any VPC, including packets with none; a named VPC never
matches a packet without one.  The CLI resolves names against the
applied config and refuses an unknown one, listing the known ones.  The
filter runs before sampling, so `sample=N` picks among matching packets.
This replaces the `direction=` argument, which never left this branch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Daniel Noland <daniel@githedgehog.com>
@daniel-noland
daniel-noland force-pushed the split/dpdk-core/07-flow branch from 7f06ff9 to c2d4969 Compare October 16, 2026 08:43
@daniel-noland
daniel-noland force-pushed the pr/daniel-noland/capture branch from c7bc0e6 to 4207243 Compare October 16, 2026 08:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant