perf(recording): capture X11 frames via MIT-SHM with GetImage fallback

[shubhook]

Summary

1080p recording on X11 goes from ~41 fps to ~59 fps at a 60 fps target, measured under Xvfb on a 2-vCPU aarch64 VM. Each individual frame grab is 3.3–4.9× faster, measured from 720p to 4K.

Today, Cap's X11 capture calls GetImage for every frame, which copies the whole frame through the X socket. This PR uses ShmGetImage with a reused SysV shared-memory segment instead, so the X server writes frames straight into memory Cap can read. Plain GetImage stays as an automatic fallback.

Wayland, macOS and Windows aren't affected. All new code is behind cfg(target_os = "linux").

How it works

• Choosing the backend: at startup the grabber uses SHM unless one of these applies, in which case it uses GetImage:
  • CAP_X11_DISABLE_SHM is set to 1, true, yes or on
  • the MIT-SHM extension is missing, or older than 1.1
  • the server can't attach the segment, as with remote or TCP displays
  • creating or mapping the segment fails
  • the server's pixel format isn't 32 bits per pixel
• Runtime fallback: if an SHM grab fails, that frame is retried with GetImage. The grabber only switches to GetImage for good, with a single warning, if the retry succeeds. If both fail, the problem is the window or region rather than SHM, so SHM stays on and the error is returned.
• Resizes: the segment is reallocated when the capture size changes, including when a captured window is resized.
• Memory safety: all unsafe code is in screen_capture/linux_shm.rs, with a SAFETY comment on each block. The segment is marked for removal (IPC_RMID) as soon as the X server attaches it, so the kernel frees it even if Cap crashes. On drop, Cap detaches it from the server, waits for the server to confirm, then unmaps it locally.

Changes

• crates/recording/src/sources/screen_capture/linux_shm.rs (new): the SysV segment lifecycle, the MIT-SHM version check, and the stride and size calculations.
• crates/recording/src/sources/screen_capture/linux.rs:
  • X11Grabber now has a GrabBackend enum with Shm and GetImage variants.
  • The original GetImage code moved, unchanged, into a helper.
  • X11Grabber and X11InputConfig went from pub(crate) to #[doc(hidden)] pub. That's the narrowest visibility that lets the benchmark example and the integration tests (both separate crates) call the real grabber. They aren't meant as a supported API.
  • New methods: with_shm(config, allow_shm), so callers can force GetImage, and uses_shm().
• crates/recording/Cargo.toml: enables x11rb's shm feature. libc was already a dependency, and Cargo.lock is unchanged.
• crates/recording/tests/x11_shm_capture.rs (new): integration tests, each against its own Xvfb.
• crates/recording/examples/x11-capture-benchmark.rs and scripts/bench-x11.sh (new): a benchmark of real X11Grabber::grab() calls on each backend.

Benchmark

Xvfb on aarch64 (OCI Ampere)

Neoverse-N1 with 2 vCPUs, X.Org Xvfb 21.1.11 with MIT-SHM 1.2. Each sample is one full X11Grabber::grab(), including the copy into a new ffmpeg BGRZ frame. Each backend gets 30 warmup grabs and 300 timed grabs.

Resolution	Backend	Mean (ms)	p50 (ms)	p95 (ms)	p99 (ms)	Mpix/s	Speedup
1280x720	GetImage	15.227	12.844	26.842	38.135	60.5	1.00x
1280x720	ShmGetImage	3.670	3.101	6.357	9.411	251.1	4.15x
1920x1080	GetImage	22.415	20.827	36.934	47.526	92.5	1.00x
1920x1080	ShmGetImage	6.876	6.742	7.193	9.225	301.6	3.26x
2560x1440	GetImage	49.274	44.398	78.332	103.342	74.8	1.00x
2560x1440	ShmGetImage	13.362	11.928	20.926	26.554	275.9	3.69x
3840x2160	GetImage	131.927	123.994	210.345	298.168	62.9	1.00x
3840x2160	ShmGetImage	26.707	26.159	30.503	33.580	310.6	4.94x

This is a small shared VM, so absolute times are higher than on desktop hardware. GetImage timings were noisy on the 2-vCPU VM: its p95 and p99 varied a lot between runs, and its 4K mean ranged from about 99 ms to 132 ms. SHM timings stayed steady. Compare the speedup ratios, not the absolute times.

End-to-end recording

recording-benchmark --mode screen --duration 15 under a 1920x1080 Xvfb, with a 60 fps target, two runs per backend. "Before" is CAP_X11_DISABLE_SHM=1, which runs the original GetImage code. Frame counts are from the capture thread's sent=… dropped=… log line.

Backend	Run	Frames sent in 15 s	fps	Dropped
GetImage	1	608	40.5	0
GetImage	2	642	42.8	0
MIT-SHM	1	879	58.6	0
MIT-SHM	2	891	59.4	0

Nothing is dropped with GetImage. Each grab just takes too long, so fewer frames get captured: a 60 fps target over 15 s would be about 900 frames. User CPU was about the same: 22.1–23.3 s for GetImage and 23.3–23.7 s for SHM. SHM spends that time encoding about 45% more frames.

How to test

1. Benchmark: run scripts/bench-x11.sh. It builds the example in release mode and runs it under xvfb-run -s "-screen 0 3840x2160x24" at 720p, 1080p, 1440p and 4K. Add --json for machine-readable output. It exits with an error if:

   • the SHM pixels differ from GetImage's
   • the SHM grabber didn't pick SHM
   • the grabber fell back during the run

   FFmpeg needs to be available first, e.g. through bun run cap-setup.

2. Real session: record on an X11 desktop. The logs should show X11 capture using MIT-SHM.

3. Opt-out: record again with CAP_X11_DISABLE_SHM=1. The logs should show X11 MIT-SHM capture disabled by CAP_X11_DISABLE_SHM, and the recording should work the same apart from slower grabs.

Testing

• cargo fmt --all --check
• cargo clippy -p cap-recording --lib --tests --example x11-capture-benchmark --no-deps -- -D warnings
• cargo test -p cap-recording --lib screen_capture: 57 passed. These include stride and size calculations, and checks that dropping a segment frees it.
• cargo test -p cap-recording --test x11_shm_capture -- --include-ignored: 8 passed. Each test starts its own Xvfb and checks one of:
  • SHM and GetImage return identical pixels for a known pattern
  • unaligned and odd-sized regions crop correctly
  • the CAP_X11_DISABLE_SHM opt-out works
  • a server without MIT-SHM falls back to GetImage
  • a TCP display, where the server can't attach the segment, falls back to GetImage
  • window capture keeps using SHM when the window grows and shrinks
  • no segments leak over 40 create/drop cycles
  • a 100k-frame soak finishes in about 6 s and leaves no segments behind

Runtime fallback: there is no committed test for switching from SHM to GetImage mid-run. I checked it by hand with a throwaway test that detached the segment on the server side and then grabbed 50 frames. Exactly one warning was logged, and every later frame used GetImage.

Pre-existing on main, unrelated: an aarch64-only clippy lint in cap-utils (disk_space.rs:186, which is why --no-deps is used above) and the real-device-test-runner example not building on Linux.

Not measured, and follow-ups

• x86 numbers: every number here comes from Xvfb on a small ARM VM. Happy to add x86 numbers if a maintainer can run scripts/bench-x11.sh, or I'll add them once I have hardware.
• XWayland wasn't measured.
• Possible follow-ups:
  • attach the segment through memfd (MIT-SHM 1.2 AttachFd)
  • two-slot pipelining to overlap one grab with the copy of the previous frame
  • a test that makes SHM fail partway through a run

Confidence Score: 5/5

The PR appears safe to merge; no actionable issues remain.

Summary

Adds MIT-SHM capture for Linux X11, with GetImage as the fallback.

• X11 recordings grab frames through shared memory when it is available.
• A benchmark compares X11 shared-memory grabs with socket-based grabs.

The previous and current heads have different merge bases with the target commit, so this review covered the full PR diff. No new actionable issues were found.

_{Reviews (2) · Last reviewed commit: "chore(recording): trim narrating comment..." · Reviewed by Greptile}

[vercel]

@shubhook is attempting to deploy a commit to the Cap Software Inc Team on Vercel.

A member of the Team first needs to authorize it.

[richiemcilroy]

Thanks for the PR. Greptile currently has this at 4/5 (reviewed before your latest commit), and we only merge PRs once Greptile scores the current head 5/5. Please comment @greptileai to request a re-review and address anything remaining until it reaches 5/5.

This also overlaps #2501, which adds MIT-SHM capture to the same X11Grabber path, and it now conflicts with main in .github/workflows/ci.yml (#2553 removed the Linux Build Desktop job; the X11 test step needs to move into the rust-tests job). Please rebase as well.

[shubhook]

@greptileai please re-review the latest head (9771355). Rebased onto main, moved the x11_shm_capture CI step into rust-tests, and addressed both earlier findings.

[shubhook]

Thanks @richiemcilroy. I rebased onto main. The CI step moved into rust-tests.

I also asked Greptile for a re-review.

I checked PR #2501 and here's how this PR differs from that:

• Fallback. This PR only switches to GetImage permanently if the GetImage retry succeeds, so a transient error during a resize doesn't lose SHM for good.
• Cleanup. On drop, it detaches the segment from the server, waits for confirmation, then unmaps. The segment is marked IPC_RMID right after attach, so a crash can't leak it.
• Off switch. CAP_X11_DISABLE_SHM turns SHM off for users who hit problems.
• Performance. 1080p recording goes from ~41 to ~59 fps at a 60 fps target.
• Tests. 8 Xvfb integration tests run in CI. They cover pixel equality, TCP and missing-extension fallback, resize, and leak checks.

If you'd rather land #2501 because it's smaller, I would love to send the integration tests, benchmark script and fallback fix as a follow-up on top of it.

Either works for me.

[shubhook]

Hey @richiemcilroy, just bumping this one. CI still needs approval for the fork whenever you get a sec.