Skip to content

fix(pullsync): release inbound streams of abandoned sync requests - #5657

Draft
gacevicljubisa wants to merge 8 commits into
masterfrom
fix/pullsync-stream-leak
Draft

gacevicljubisa wants to merge 8 commits into
masterfrom
fix/pullsync-stream-leak

Conversation

@gacevicljubisa

@gacevicljubisa gacevicljubisa commented Oct 9, 2026 •

Copy link
Copy Markdown
Member

Checklist

  • I have read the coding guide.
  • My change requires a documentation update, and I have done it.
  • I have added tests to cover my changes.
  • I have added or updated fuzz targets for any code handling untrusted input.
  • I have filled out the description and linked the related issues.

Description

Full nodes reach the libp2p limit of 5,000 inbound streams and then refuse every new inbound stream. Peers see stream reset (remote) ... error code: 4098 (StreamResourceLimitExceeded) on handshake, pushsync, retrieval and pullsync.

Actual error:

"time"="2026-10-09 17:21:43.116410" "level"="info" "logger"="node/kademlia" "msg"="could not connect to peer" "peer_address"=***" "error"="network status unknown: connect new stream: create stream /swarm/handshake/15.0.0/handshake to ***: failed to negotiate protocol: stream reset (remote): code: 0x1002: transport error: stream reset by remote, error code: 4098"

Cause: a pullsync handler waiting for the first chunk of an empty interval never returns if the client abandons the stream. The handler context is canceled only when the peer disconnects, not when the stream is reset, and collectAddrs had no time limit for the first chunk. A goroutine profile of an affected node showed 3,204 of 3,209 pullsync handlers waiting in collectAddrs. 2,176 had waited more than 24h, and three leaders had waited for the full pod uptime (11.6 days).

Motivation and Context (Optional)

Observed on 2.8.2 (version unchanged for 35+ days): in one cluster 154 of 574 nodes were at the limit, with ~1.49M inbound pullsync streams in total. Affected nodes climb to 5,000 over a few days and drop back only on restart or mass peer disconnect.

Screenshots (if appropriate):

Inbound streams vs limit (one affected node, 14 days): inbound streams climb steadily to the 5,000 limit (dashed red) and stay there. Pullsync accounts for nearly all of them.

https://grafana.proxy.ethswarm.org/goto/szmtt5?orgId=default

image

Inbound streams by protocol (same node): pullsync holds ~5,000 inbound streams. Every other protocol stays near zero.

https://grafana.proxy.ethswarm.org/goto/sq9rkc?orgId=default

image

Open API Spec Version Changes (if applicable)

Motivation and Context (Optional)

Related Issue (Optional)

Screenshots (if appropriate):

AI Disclosure

  • This PR contains code that has been generated by an LLM.
  • I have reviewed the AI generated code thoroughly.
  • I possess the technical expertise to responsibly review the code generated in this PR.

@gacevicljubisa

Copy link
Copy Markdown
Member Author

Problem:

  1. Peer A asks peer B for chunks. The request gives a bin and a start BinID.
  2. Peer B starts a handler for this request. The handler holds one inbound stream.
  3. The handler waits for the first chunk in this interval. This wait has no time limit.
  4. If the interval has no chunks, the handler can wait for a long time, sometimes for days.
  5. During this time, peer A can stop the request. Peer A then resets the stream.
  6. The handler on peer B does not see the reset. The handler reads from and writes to the stream only after a chunk arrives.
  7. The handler context is canceled only when the peer disconnects. A stream reset does not cancel the context.
  8. Thus, the handler continues to wait, and the stream stays in use.
  9. After the reset, peer A sends a new request. The new request opens a new stream and starts a new handler.
  10. Each cycle leaves one stream in use. The number of streams in use increases.
  11. The limit for inbound streams is 5,000. After some days, the node gets to this limit.
  12. At the limit, the node refuses all new inbound streams. This applies to all protocols: handshake, pushsync, retrieval and pullsync.
  13. Other peers get error 4098 (StreamResourceLimitExceeded). Kademlia cannot connect to new peers.
  14. The peers then send their requests again without a pause. This increases the load.

@gacevicljubisa
gacevicljubisa restored the fix/pullsync-stream-leak branch October 12, 2026 09:11
@gacevicljubisa gacevicljubisa changed the title fix(pullsync): bound first-chunk wait and back off puller on sync errors fix(pullsync): release inbound streams of abandoned sync requests Oct 12, 2026
@gacevicljubisa
gacevicljubisa requested a review from acud October 12, 2026 14:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant