Repository navigation
fix(pullsync): release inbound streams of abandoned sync requests - #5657
Draft
gacevicljubisa wants to merge 8 commits into
Draft
gacevicljubisa wants to merge 8 commits into
gacevicljubisa wants to merge 8 commits into
Conversation
Member
Author
|
Problem:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Checklist
Description
Full nodes reach the libp2p limit of 5,000 inbound streams and then refuse every new inbound stream. Peers see
stream reset (remote) ... error code: 4098(StreamResourceLimitExceeded) on handshake, pushsync, retrieval and pullsync.Actual error:
Cause: a pullsync handler waiting for the first chunk of an empty interval never returns if the client abandons the stream. The handler context is canceled only when the peer disconnects, not when the stream is reset, and
collectAddrshad no time limit for the first chunk. A goroutine profile of an affected node showed 3,204 of 3,209 pullsync handlers waiting incollectAddrs. 2,176 had waited more than 24h, and three leaders had waited for the full pod uptime (11.6 days).Motivation and Context (Optional)
Observed on 2.8.2 (version unchanged for 35+ days): in one cluster 154 of 574 nodes were at the limit, with ~1.49M inbound pullsync streams in total. Affected nodes climb to 5,000 over a few days and drop back only on restart or mass peer disconnect.
Screenshots (if appropriate):
Inbound streams vs limit (one affected node, 14 days): inbound streams climb steadily to the 5,000 limit (dashed red) and stay there. Pullsync accounts for nearly all of them.
https://grafana.proxy.ethswarm.org/goto/szmtt5?orgId=default
Inbound streams by protocol (same node): pullsync holds ~5,000 inbound streams. Every other protocol stays near zero.
https://grafana.proxy.ethswarm.org/goto/sq9rkc?orgId=default
Open API Spec Version Changes (if applicable)
Motivation and Context (Optional)
Related Issue (Optional)
Screenshots (if appropriate):
AI Disclosure