feat(research): memory probe for browser push listening and the resident bridge [S6]

[tunjayoff]

Item S6: research tooling that measures ways to lower the memory of browser-based push listening (the page source) and of the resident bridge. Nothing in this PR touches SofaScore. Everything was built and tested offline against a local fake site on a .test host, reached through the browser's proxy, with a fake NATS-over-WebSocket server. The orchestrator runs the live matrix afterwards.

What is added

• scripts/push_memory_probe.py: runs one configuration per call, for --minutes of steady state after the last tab opens. It writes research/push_memory/<run-id>/{samples.jsonl,events.jsonl,summary.json}.
• scripts/push_memory_matrix.py: runs a list of configurations one after another, with a cool-down between runs.
  • Before each run it checks that no browser from the previous run is still alive. It waits --settle seconds, then terminates any leftover and records it.
  • On timeout it sends SIGTERM. The probe then closes its browsers and still writes its summary.
  • It appends one row per run to research/push_memory/matrix-<id>.jsonl.
• scripts/push_memory_engines.sh: installs the optional engines with pinned versions under ~/.cache/s6-engines only.
• tests/test_push_memory_probe.py: 38 offline unit tests and 5 browser tests marked browser, which run in the "browser bridge (offline)" CI job.

Usage

python scripts/push_memory_probe.py --config baseline --minutes 30
python scripts/push_memory_probe.py --config shared-browser+multi-sport --minutes 30
python scripts/push_memory_probe.py --config flags --flags renderer-limit=1,no-site-isolation --minutes 30
python scripts/push_memory_probe.py --config flags --dry-run          # resolved plan, no browser
python scripts/push_memory_matrix.py --minutes 30 --cooldown 120     # full matrix, sequential
python scripts/push_memory_matrix.py --configs "baseline;flags;engine-camoufox" -- --flags disable-gpu
scripts/push_memory_engines.sh all                                    # optional engines -> ~/.cache/s6-engines

Useful options:

• Timing: --interval 10, --warmup 60 (per idle phase), --tab-gap 120.
• Scope: --sports, --no-bridge, --flags-target all|live|bridge.
• Periodic configs: --reload-minutes 10, --gc-minutes 5.
• Hidden config: --hidden-mode window|js|both, --hidden-allow-backgrounding.
• Live filter: --live-filter-selector.
• Frame gate: --max-silence 300.
• Lightpanda: --page-counter, --cdp-driver scrapling|playwright.
• Output and profiles: --keep-profiles, --run-id.

What is recorded

Every sample (10 s by default):

• Per browser group (bridge, live, Playwright driver, other): PSS, USS and RSS summed from /proc/<pid>/smaps_rollup. RSS is kept to compare with the old README numbers. USS is Private_Clean + Private_Dirty + Private_Hugetlb.
• Process count, renderer count, and processes by kind.
• PSS, USS and RSS of the probe's own Python process.
• Per tab: MSG frames per subject (sport.x, event.{id}, odds.{id}…), the number of distinct event.* subjects, whether the NATS connection is open, and the JS heap.
• Per browser: page API requests sent and blocked, and how many slots were taken from the shared lock.

Processes are attributed to a browser through the profile path in their command line (--user-data-dir= for Chromium, -profile for Firefox) or through the PID (Lightpanda). Chromium rewrites its process title (setproctitle), so the reader also handles a command line packed into one string.

Events:

• Each browser launch, with the effective root switches read back from /proc (only the relevant ones, no paths) and the applied launch options.
• Challenge gate per browser and API-read gate (start and end).
• NATS handshake, close and reconnect, with the gap in seconds; -ERR, reported only as a known NATS error text.
• Phases (bridge, live-idle, tabs-N), reloads (time until the next handshake), GC rounds, hidden marking, live-filter clicks, and page crashes.

Summary:

• Per phase: median and peak PSS/USS/RSS, and medians per group.
• Marginal cost between consecutive phases (live browser, each extra tab).
• Steady phase: median and peak, growth per 10 min (least-squares slope), frames/min per subject, longest silence.
• API: total requests, rate, maximum in any 10 s window, minimum gap.
• Gates and a status: ok / failed (with reasons) / skipped.

What fails a run:

• A tab with no sport.{sport} frames at all.
• A tab with no MSG for longer than --max-silence.
• A challenge that was not solved, or an API read that was not 200.
• Requests closer together than the shared gap.

Rate: every SofaScore request goes through the research lock in scripts/_research_common.py (≤ 1 req/s across research processes). This covers the pages' own API calls, which are throttled inside the load window and blocked after it, exactly like the product. The bridge's reads go through it too, because the bridge's _wait_for_slot is pointed at the same lock. The product's throttle file under ~/.cache/sofascore_scraper is never written.

Privacy:

• The INFO body is never parsed; the product's NatsReader drops it.
• Outgoing frames, which carry the CONNECT credential, are not listened to.
• No WebSocket URL, proxy address or server address is written.
• Error texts pass through the product's redaction and the research redact (IPs and geo keys) and are truncated.
• The browser tests check this end to end: a fake INFO carrying a public client_ip and connect_urls, and a CONNECT credential, never appear in the output, and tests/test_research_privacy.scan is clean.

Profiles: each run uses fresh temporary profiles under ~/.cache/sofascore_research/push_memory_probe/<run-id>/{bridge,bridge-live} and deletes them at the end. A lock in that root keeps two probes from ever running at the same time.

Configurations and what each measures

config	what it does / measures
baseline	The product's page source as it is: the resident bridge (quiet robots.txt tab), a separate <profile>-live browser and one tab per sport, with context-level route_action rules. Phases: bridge → live-idle → tabs.
shared-browser	The sport tabs open in the bridge's own browser. Reports the marginal cost per tab. Rules are installed on the context but act per page: the bridge page and Scrapling's captcha pages fall through untouched, and service-worker requests are treated as out of the load window. The bridge API read at the end checks that the rules do not break it.
multi-sport	Football, then basketball, then tennis, --tab-gap apart. Gives the marginal cost per extra sport. Combines with shared-browser.
flags	Chromium presets: renderer-limit=N, no-site-isolation (--disable-site-isolation-trials + IsolateOrigins,site-per-process,SitePerProcess), no-bfcache, lite-mode, max-old-space=N, disable-gpu, no-images (imagesEnabled=false), small-viewport (400x300 viewport; the screen fingerprint is unchanged). The challenge gate is recorded for every browser that gets the flags.
block-third-party	Stricter rules: everything except SofaScore hosts and challenges.cloudflare.com, plus SofaScore's CSS, font, media, image, beacon and manifest requests. The challenge host is treated as in the product.
live-filter	Opens the sport page and clicks "Canlı" (the click gets a new load window). Records which subjects arrive and how many distinct event.* subjects, to answer whether it subscribes to sport.* and whether it is lighter.
reload-every	page.reload() every N minutes. Records memory and the gap until the next handshake (reload_gaps_s).
gc-pressure	Every N minutes, on every page (tabs and the bridge page): CDP HeapProfiler.collectGarbage + Memory.simulatePressureNotification(critical).
hidden	Minimises the window (CDP Browser.setWindowBounds) and/or overrides the page's visibilityState/hidden and fires visibilitychange. It never freezes the page. Each sample records the visibility the browser itself reports. --hidden-allow-backgrounding also removes the switches that keep background renderers at full priority, from Scrapling's args and from Playwright's defaults (ignore_default_args).
engine-headless-shell	Chromium's chrome-headless-shell, matching patchright's revision 1243 (channel=chromium-headless-shell + executable_path).
engine-camoufox	Camoufox through a Scrapling session whose start() opens Camoufox's persistent context. Scrapling's own Turnstile solver runs on the Firefox pages. Runs with its own virtualenv Python (the matrix picks it automatically).
engine-lightpanda	Feasibility only. Connects through Scrapling's cdp_url; if that fails, through plain Playwright connect_over_cdp. Also counts frames inside the page (see the limits).

Which browser Scrapling 0.4.15 / patchright 1.63 launch today

Scrapling sets channel="chromium". Patchright resolves that to the full Chromium binary (Chrome for Testing 153.0.8010.12, rev 1243) in new headless mode. It would only pick chrome-headless-shell with no channel and headless=True. The engine-headless-shell config makes that choice explicit and records the executable that actually ran.

Scrapling stealth flags are kept

Scrapling 0.4.15 has a pitfall: passing extra_flags replaces its stealth flags (STEALTH_ARGS and the WebRTC policy), because of config.extra_flags or extra_flags. So the probe never uses extra_flags.

Instead, the session is built with exactly the product's arguments and the flags are merged into session._browser_options["args"] before start(). --disable-features, --js-flags and --blink-settings are merged into a single switch that carries Scrapling's values plus ours. This matters because Playwright also adds its own --disable-features and headless --blink-settings earlier on the command line, and the merged switch comes last.

A unit test checks that every stealth switch survives. Another checks that the baseline session is byte-for-byte the product's. A test also runs the product's own BrowserBridge._launch and checks it receives the same constructor arguments.

Optional engines (pinned; the probe marks a missing engine skipped)

engine	pinned version	license	x86_64 Linux	arm64 Linux
chrome-headless-shell	rev 1243 (153.0.8010.12), matching patchright 1.63.0	BSD-3-Clause (Chromium)	yes	yes (Chrome for Testing linux-arm64)
Camoufox	python camoufox==0.5.8 (MIT); browser v156.0.1-beta.36 (MPL-2.0), pinned by the package; playwright==1.62.0 in its own venv (camoufox needs <1.63, the repo pins 1.63.0)	MPL-2.0 / MIT	yes (camoufox-156.0.1-beta.36-lin.x86_64.zip)	yes (…-lin.arm64.zip, sha256 8b5fd083…)
Lightpanda	1.0.0 (sha256 pinned per arch in the script)	AGPL-3.0	yes (lightpanda-x86_64-linux)	yes (lightpanda-aarch64-linux)

Scrapling 0.4.15 no longer ships a Camoufox session. Camoufox comes from the camoufox package, installed in ~/.cache/s6-engines/camoufox-venv together with the repo's requirements, with playwright overridden to 1.62.0. The browser archive (1.3 GB) is fetched with XDG_CACHE_HOME=~/.cache/s6-engines/xdg. The probe points Camoufox's cache there only while it imports Camoufox. Lightpanda's telemetry is always disabled (LIGHTPANDA_DISABLE_TELEMETRY=true).

Offline results (fake site, mechanics only; not a memory verdict)

run	result
baseline	Both challenge gates passed. API reads returned 200. Frames were counted per subject. The fake server dropped the first connection, and the probe recorded the reconnect. Page API requests were spaced ≥ the lock gap and blocked after the load window. Third-party requests and images never reached the network.
shared + multi-sport + flags + hidden + gc + reload + strict + live-filter (one Chrome)	Every flag reached the root process and Scrapling's stealth switches were kept. The bridge API read through the shared context rules returned 200. GC ran on 3 pages with no errors. Reload gaps were recorded and the frames continued.
headless-shell	The executable that ran was chrome-headless-shell. Status ok.
Camoufox	Ran in its venv. Status ok: token present, API read 200, frames arrived through Playwright's Firefox websocket events.
Lightpanda 1.0.0	Feasibility details below.

Lightpanda 1.0.0 feasibility:

• Scrapling (patchright) connect_over_cdp + new_context fails ("Connection closed while reading from the driver").
• Plain Playwright connects. The captcha-page cookie and the bridge-page API read worked against the fake site.
• Only one page can be open per context (Target.createTarget: TargetAlreadyLoaded). In the separate layout the live browser's own page becomes the tab.
• The page's WebSocket works and receives frames, but Lightpanda sends no CDP WebSocket events, so the product's way of listening cannot see the frames. The probe's in-page counter does see them, and such a run is marked failed with that reason.
• WebSocket over an HTTP proxy did not complete a handshake: CONNECT opened, then closed without an upgrade.

Known limits

• The bridge tab waits on the quiet robots.txt page, as in the earlier probe and the README's 1.1 GB baseline. The product's resident bridge sits on /tr, whose own API calls cannot be told apart from the bridge's fetch(), so /tr is not measured.
• The probe infers subscriptions from the subjects that arrive. It never reads outgoing SUB frames (no outgoing listener at all).
• Frame rates depend on how many matches are live at the time. Configs run at different hours are not strictly comparable. Each run records live_events from its API reads so frames per live event can be normalised.
• hidden: in headless Chromium, minimising the window left the browser-reported visibilityState as visible (offline). The page's JS still sees hidden. The live run will show whether SofaScore's page does less work.
• The --disable-features / --blink-settings duplicates come from Playwright, earlier on the command line. Chromium takes the last value, so the merged switch is the one in effect. This was already true of Scrapling's own switch.
• small-viewport may move the Turnstile widget outside the 400x300 viewport. The challenge gate will show it live.
• Lightpanda over a proxy: WebSockets did not work through an HTTP proxy offline.
• gc-pressure and the window mode of hidden need CDP (Chromium engines only). The probe rejects them for Camoufox and Lightpanda.
• The live gates (a real Turnstile, a real NATS server) are for the orchestrator's run. Offline, Scrapling's solver finds no challenge on the fake captcha page.

Checks

• ruff check . passes.
• The full suite passes with STORE_SHADOW_CHECK=1: 9826 passed, 39 skipped, 27 deselected (8 min).
• The browser tests in tests/test_push_memory_probe.py pass: 4 passed in the repo venv, plus the Camoufox test in the Camoufox venv, which is skipped where Camoufox is not installed (CI).
• CHANGELOG.md and docs/design/** are untouched.