research: soak recorder for the live push sources (S2)

[tunjayoff]

Item S2: a soak recorder for the live push path: it measures the live service as it is over hours. Research tooling only; no product code changes. Nothing in this PR contacts SofaScore: all tests run offline.

Usage

python scripts/soak_live.py --sports football,basketball,tennis --hours 6 \
    [--source direct|page] [--run-id ID] [--max-requests N] [--rate 1.0] [--list-share 0.5] \
    [--joined-ids FILE] [--joined-random-soccer 20] [--kill-every MIN] [--kill-at MIN] \
    [--compare-credential-sports] [--out DIR]

• Output goes to research/soak/<run-id>/. The run refuses a run id it has already used. A summary is written every 10 minutes and at the end.
• The service's store goes to ~/.cache/sofascore_research/soak_store/<run-id>. The throttle dir defaults to ~/.cache/sofascore_research/throttle and the browser profile is chrome_soak. The live profile is chrome_soak-live.
• Budget: --max-requests defaults to hours x 3600 x rate, so it averages at most 1 req/s. With --rate-schedule, the default is the sum of the phases. Every REST read is paced at --rate, then takes the shared research lock. Once the budget is spent, no more REST reads are made, but push recording continues.

What runs

It is the service as it is. RecordingLiveService (a LiveService subclass) runs with the chosen push source, the real arbiter and the real PollSource. Its scope covers every live match of each sport, so the poll source's rounds work exactly as they do today. Its confirmation read is turned off, because the recorder makes that read instead. The hooks are:

• a DirectConnection subclass
• a wrapper around the connect function, which also kills the connection
• a wrapper around the page opener (page), which closes the pages
• a tap on each sport's PushFeed
• a wrapper around the credential reader
• the service's _system and fetch

What is recorded

• frames.jsonl: one row per MSG frame. Each row has receive time (wall and monotonic), sport, event id, the changed paths as keys only, values for status.code/status.type and score fields only, changes.changeTimestamp, and the triggers.

• connection.jsonl:

  • session start and end, with outcome, reason and duration
  • handshake time, and handshake-to-INFO time
  • CONNECT sent, recorded as an event only (never the body)
  • SUB and re-SUB, UNSUB
  • client PING, server PING, PONG with RTT and gaps
  • -ERR, with the known NATS text only
  • retries and unhealthy state
  • credential reads: the sport page read, duration, peak RSS, and same/different compared with the previous read
  • --compare-credential-sports: the result for other sports' pages (same/different only)
  • kills
  • for page: page open and gone, and INFO/PING/PONG/-ERR per page socket

• leader.jsonl: arbiter switches (from, to, reason) with fallback start/stop, plus blocked/recovered.

• reads.jsonl: every REST read, with who made it (service, recorder or fallback), kind, event id, status, latency, and live count for lists. For recorder reads it also has the trigger (kinds, source frame/list, time), trigger_to_read_s, and whether the read reflects the change. A CDN-stale read is retried after CONFIRM_RETRY_SECONDS, up to CONFIRM_ATTEMPTS.

• requests.jsonl: one row per minute, aligned to the run start. Each row has:

  • requests by kind and by path, and the total
  • the peak in any one-second window
  • status counts
  • 429s and 403s, split into curl, curl challenge, browser challenge and browser
  • Turnstile attempts, solves and failures
  • latency p50 and p95
  • the rate phase

  403, 429 and Turnstile are counted from the client's own log lines (the Utils and ChallengeSolver loggers); the text of those lines is never written. While the counter is installed, only WARNING and above go on to the root handlers.

• fallback.jsonl: one row per minute and sport while the list leads. Each row has live matches, joined live, reads made (joined, listed, other), skipped (joined, listed) and the cut reason. It also records joined-set changes (join, leave).

• resources.jsonl: every 30 s, and every 5 s during a credential read. It has RSS of this process, the main bridge's browser, the live profile's browser and other children, plus the total and the socket count. Browsers are told apart by their --user-data-dir, which is never written.

• summary.json, per sport:

  • frames, triggers, finish push lag and frame→event-read latency
  • downtime windows with causes, fallback time share and time to first push
  • the longest MSG-less gaps while the live list shows more than 0 matches, with the leader share, open share, PING/PONG counts and longest silence
  • fallback skip rate and worst minute

  Overall, it also has connection stats, requests by who/kind/status, peak RSS by bucket, and notes. One note states that this is the first live run of the shipped stdlib WebSocket client; the 2026-10-01 probe used aiohttp.

Fallback rule (owner's rule)

While the list leads (push down, silent or killed), the arbiter lists every 30 s and the poll source makes its own near-end, dropped, stuck and overdue reads. These are unchanged. On top of that:

• Matches in the joined set are urgent. Each one is read every 60 s, with reason joined.
• All other live matches form a backlog. They are read as the rate allows, longest unread first, with rank listed.
• Each minute allows 60 x --rate x (1 - --list-share) reads; the rest of the rate is kept for discovery and list reads. A read that does not fit its minute is skipped and counted per sport. The queue is rebuilt from the latest list every minute, so it never grows without bound.
• The joined set is made of --joined-ids FILE (shape {"ids": [...]}; a malformed file or one with more than 50 000 ids is ignored with a warning) plus --joined-random-soccer N (default 20). The random part picks N random live football matches at the start and replaces each one when it leaves the live list. There is no external "joined" signal here.
• While push leads, these minute reads stop. Only the arbiter's safety poll remains: a list every 120 s and one round per (re)connect.

The summary adds:

• skip counts and rates (joined, listed, all) per sport and overall, and the worst minute
• read-gap percentiles: per joined match, pooled for joined matches and pooled for the others. A gap is two consecutive reads by any path within one interval in which the list leads.
• the list rate achieved while the list leads, and requests per second achieved

The product's poll_source is unchanged.

Rate phases

--rate-schedule "60:1,120:5" gives 60 minutes at 1 req/s, then 5 req/s; the last phase lasts until the end of the run. The current rate sets both the pacing and each minute's fallback share.

In a phase above 1 req/s, the 1 req/s research lock is not taken. If the schedule goes above the app's client.rate (5/s by default), client.rate is raised to match, unless it is set explicitly.

summary.json has a rate_phases entry per phase with the same statistics as the minute rows, plus the rate achieved.

How the kill works

• direct: the recorder calls shutdown(SHUT_RDWR) on the current socket. The client sees an ordinary drop. The feed closes, so the arbiter switches direct>poll (push_disconnected) and runs a poll round. The client reconnects and subscribes again, the feed opens, and the arbiter switches back poll>direct.
• page: the recorder closes the sport pages. The page's close event makes the feed go away (push_unavailable). PageSource reopens the page after its back-off, and the arbiter switches back.
• Where it is recorded: in connection.jsonl (kill, and the session end or page close carrying the cause), in leader.jsonl, and in the summary downtime windows.

Privacy

• Every written row passes through sofascore_scraper/redact.py, which also covers the runtime credential, and through the research redact() (IPs, geo keys).

• Key names avoid the redactor's secret-key pattern, so values like "different" are not masked.

• test_a_recorded_run_holds_no_credential_address_or_info_body and the page test scan a recorded run. They check that there is:

  • no IPv4/IPv6 literal, with even loopback and documentation ranges disallowed
  • no user/pass/auth_token key
  • no fake credential, server URL or port, CONNECT {, INFO fields or client_ip
  • no *** (nothing was ever about to leak)

  The same tests also run test_research_privacy.scan over the run.

Tests

tests/test_soak_live.py has 27 tests. It uses the in-process fake NATS server and recorded frames from test_live_direct_source.py, the fake page from test_live_push_source.py, and FakeApi. test_research_settings.py now includes the script's profile. The full suite passed with STORE_SHADOW_CHECK=1: 9807 passed, 39 skipped. ruff check . is clean.