Skip to content

fix(envoy-client): keep the ping silence deadline bounded across reconnects - #5843

Open
abcxff wants to merge 1 commit into
stack/fix-envoy-client-ignore-pings-from-earlier-connections-in-the-silence-check-uxuuvwzufrom
stack/fix-envoy-client-keep-the-ping-silence-deadline-bounded-across-reconnects-wpskpplp
Open

abcxff wants to merge 1 commit into
stack/fix-envoy-client-ignore-pings-from-earlier-connections-in-the-silence-check-uxuuvwzufrom
stack/fix-envoy-client-keep-the-ping-silence-deadline-bounded-across-reconnects-wpskpplp

Conversation

@abcxff

@abcxff abcxff commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

No description provided.

@claude

claude Bot commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Review: keep the ping silence deadline bounded across reconnects

The design is sound. A monotonic Instant baseline replaces wall-clock last_ping_ts for the silence check, and a reconnect no longer silently disables or resets the check. Per-generation deadlines also stop a freshly started actor from being killed against a stale baseline. The docs update and the new tests (including partial-expiry) are good. I have not built or run the tests. Notes below.

Possible issues

  1. is_ping_healthy behavior change (connection/mod.rs, handle.rs:88). install_connection_with_http no longer zeroes last_ping_ts. After a reconnect, is_ping_healthy() can return true for up to 20s based on a ping from the previous connection. Before, it returned false until the new connection got its first ping. If upstream health checks rely on "healthy means this connection is live", this is a regression. Either keep the reset for the health signal only, or document the new semantics.

  2. Partial expiry leaves in-flight requests pending (envoy.rs, declare_silent_actors_lost). The full-expiry path goes through declare_actors_lost, which drains kv_requests and fails SQLite and remote SQLite requests with EnvoyShutdownError. The partial path only cancels lost and removes the entries. Requests belonging to the stopped generations stay pending until they time out, or forever if nothing sweeps them. Please check that the per-generation case fails those requests, or that something else cleans them up.

  3. Ignored commands_received() return value (connection/mod.rs). Probably fine because the envoy loop takes a turn when it handles the commands, but a short comment would help, since the ping path explicitly wakes the loop.

Minor

  • declare_silent_actors_lost duplicates the per-entry loss logic (cancel, send Lost, remove) from declare_actors_lost. Consider a shared helper so the two paths cannot drift.
  • (threshold - margin).max(0) as u64 is safe, but a negative threshold from protocol metadata would make the deadline equal to the baseline and stop all actors immediately. Worth a guard or comment.
  • Tests: add coverage for the reconnect-without-claim path (baseline retained across connection_installed) and the commands_received claim path. EngineLiveness is a small pure state machine and easy to unit test.

No security concerns. Style matches CLAUDE.md.

@the-company-company the-company-company Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 1 high-severity finding

Reviewed commit baa6a32.

Comment on lines +99 to +101
pub fn connection_installed(&self, now: Instant) {
let mut state = self.0.lock();
state.connection_installed_at = Some(now);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 High · Preserve command-proven liveness across reconnects

When a connection is claimed by commands before its first ping, commands_received() proves that the engine refreshed its persisted liveness timestamp, but that evidence is represented only by connection_claimed. The next connection_installed() overwrites the install time and clears the flag, while last_ping is still None, so baseline() jumps forward to the replacement connection's install time. If that replacement never gets claimed, the engine continues counting from the previous claim and can declare/move the actors before this client stops them. Retain the prior claimed baseline (the prior install time when no ping exists) across installs, and replace it only after the new connection is proven claimed; add the sequence install -> commands -> reconnect -> no commands/ping to the liveness tests.

@abcxff
abcxff force-pushed the stack/fix-envoy-client-keep-the-ping-silence-deadline-bounded-across-reconnects-wpskpplp branch from baa6a32 to 86507ea Compare October 6, 2026 01:29
@abcxff
abcxff force-pushed the stack/fix-envoy-client-ignore-pings-from-earlier-connections-in-the-silence-check-uxuuvwzu branch from 831507a to 4ef21a9 Compare October 6, 2026 01:29
@abcxff
abcxff force-pushed the stack/fix-envoy-client-keep-the-ping-silence-deadline-bounded-across-reconnects-wpskpplp branch from 86507ea to 57602c5 Compare October 6, 2026 01:53

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant