Skip to content

fix(envoy-client): ignore pings from earlier connections in the silence check - #5838

Open
eersnington wants to merge 1 commit into
stack/fix-envoy-client-declare-actors-lost-before-the-engine-can-reallocate-them-nkyokownfrom
stack/fix-envoy-client-ignore-pings-from-earlier-connections-in-the-silence-check-uxuuvwzu
Open

eersnington wants to merge 1 commit into
stack/fix-envoy-client-declare-actors-lost-before-the-engine-can-reallocate-them-nkyokownfrom
stack/fix-envoy-client-ignore-pings-from-earlier-connections-in-the-silence-check-uxuuvwzu

Conversation

@eersnington

@eersnington eersnington commented Oct 5, 2026 •

Copy link
Copy Markdown
Member

No description provided.

@claude

claude Bot commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Review

Small, well-reasoned fix. The new integration test drives a real envoy through a reconnect.

What it does: install_connection_with_http resets last_ping_ts to 0 on each new connection, so a ping from the old connection can't trip the silence deadline before the new connection's first ping. The first ping is forwarded to the envoy loop so it re-arms the silence timer.

Notes

  • Reset vs. stale-session ping race. forward_to_envoy checks the session before storing the ping timestamp. A late ping from the old session could land after the reset and restore a stale timestamp. The window is tiny and the failure is the old behavior, but doing the session check and store under one guard would close it. I didn't verify whether the surrounding code already does.
  • is_ping_healthy side effect. It now returns false from reconnect until the first ping. That seems right, but worth confirming upstream health checks tolerate a short unhealthy window after each reconnect.
  • Waking the loop by re-forwarding the first ping. It works, and the comment in envoy.rs explains it. A dedicated wake message would be more explicit.
  • Test timing. It relies on real wall-clock sleeps (2s deadline, 200ms and 300ms margins). The early.is_err() check could flake on a loaded CI runner. CLAUDE.md prefers deterministic tests, though real sockets make that awkward.
  • PR body is empty. A short bullet list would help.

Style matches CLAUDE.md (hard tabs, tests under tests/, no em dashes). No security concerns. Looks good to merge, with the optional race hardening above.

@the-company-company the-company-company Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 1 high-severity finding

Reviewed commit 831507a.

Comment on lines +67 to +69
// A ping from an earlier connection says nothing about this one. Clearing it keeps the engine
// ping silence check disabled until this connection receives its first ping.
shared.last_ping_ts.store(0, Ordering::Release);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 High · Keep a deadline before the first ping

Clearing this to the sentinel disables the only ping-silence deadline for the new session. ToEnvoyInit subsequently clears lost_timeout, so if the engine sends Init and commands and then crashes or the link becomes half-open before its first ping, engine_ping_silence_deadline_ms returns None forever while the newly started actors keep running; the engine can meanwhile declare this envoy lost and reallocate them, violating the single-writer invariant. Track a separate per-session/Init baseline (or equivalent first-ping grace deadline) so the old ping is ignored without leaving the pre-first-ping state unbounded.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

slop? the engine sends Init b4 it refreshes the envoys liveness

conn.rs:146   send Init                       ← envoy would reset its silence baseline here
conn.rs:278   claim_registration
conn.rs:423     write LastPingTsKey           ← engine resets its clock here (or not, if the claim fails)
conn.rs:293   send missed commands
lib.rs:249    ping task → update_ping, then ping

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@claude blast him

@abcxff
abcxff force-pushed the stack/fix-envoy-client-declare-actors-lost-before-the-engine-can-reallocate-them-nkyokown branch 2 times, most recently from 6a97303 to 33530e0 Compare October 5, 2026 15:28
@abcxff
abcxff self-requested a review October 5, 2026 15:28
@abcxff
abcxff force-pushed the stack/fix-envoy-client-declare-actors-lost-before-the-engine-can-reallocate-them-nkyokown branch from 33530e0 to 6a97303 Compare October 6, 2026 00:42
@abcxff
abcxff force-pushed the stack/fix-envoy-client-ignore-pings-from-earlier-connections-in-the-silence-check-uxuuvwzu branch from 831507a to 4ef21a9 Compare October 6, 2026 01:29
@abcxff
abcxff force-pushed the stack/fix-envoy-client-declare-actors-lost-before-the-engine-can-reallocate-them-nkyokown branch from 6a97303 to 27a48b9 Compare October 6, 2026 01:29

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants