Heavy sync init work (e.g. local ML models) starves stdio initialize/tool calls even with threadpool offload -- Windows GIL contention #3089

Activity

1. IgorGanapolsky commented on Jul 14, 2026

   

   IgorGanapolsky

   on Jul 14, 2026

   More actions

   This matches what I've hit putting local models behind stdio MCP servers, and your table is the right diagnosis. The reason the threadpool offload doesn't save you: anyio.to_thread.run_sync still runs in-process, and a torch / sentence-transformers load is GIL-bound work that holds the GIL for most of those 8s. anyio's Windows stdio reader is itself a thread that needs the GIL to pump the pipe, so the loader and the transport reader serialize against each other no matter which thread the load sits on. That's why only the subprocess — separate interpreter, separate GIL — keeps both initialize and the first call responsive.

   A few things that made the subprocess pattern actually reliable for me, beyond just isolating it:

   • Warm inside the child before you accept work, and model the warming state explicitly. Have the child load the model, then flip a ready flag; while it isn't ready, tool calls return a typed unavailable/retry result instead of blocking. That keeps the ~8–9s first-call cost from landing on whichever unlucky request happens to arrive first.
   • Supervise the child. A subprocess turns a deadlock into a crash you can recover from: the parent watches the pipe and, on EOF/exit, respawns and re-warms. Treat the pipe as at-least-once and put an id on every message so a mid-flight restart re-correlates instead of dropping or double-running a call.
   • On Windows, spawn (not fork) re-imports everything in the child, so keep the child entrypoint tiny and import torch only inside it — otherwise the parent pays the import cost too.

   If you'd rather not hand-roll the pipe protocol, running the embedding model behind a tiny local HTTP service and calling it from the tool gets you the same GIL isolation plus a real readiness/health endpoint for free — the MCP server stays pure-async and never imports torch. A little more plumbing, but every failure mode becomes observable instead of surfacing as a mysterious 70s stall.
2. scotthibbs commented on Jul 15, 2026

   

   scotthibbs

   on Jul 15, 2026

   Author

   More actions

   I gave my buddy Claude Code your response and he offered his thanks below:

   Thanks — this is the clearest explanation of why the threadpool offload can't help that I've seen: the load holds the GIL, and anyio's Windows stdio reader is itself a thread that needs the GIL to pump the pipe, so the two serialize regardless of which thread the load sits on. That matches our measurements exactly (8s standalone → ~70s in-process under the live server).

   For anyone finding this later, how our subprocess implementation lines up with your hardening points:

   • Import isolation: the parent never imports torch — sentence_transformers is imported lazily inside the embed helper, and only the child calls it, so Windows spawn re-import never taxes the MCP process.
   • Supervision: the parent treats EOF on the response pipe as worker death and respawns on the next call. We skipped message IDs deliberately: the protocol is strictly serialized under a lock (one request in flight at a time), so there's no correlation ambiguity — a mid-flight crash fails that one call with a clean error instead of dropping or double-running anything.
   • Readiness flag: we let the first call block instead. The child warms eagerly at spawn (overlapping the client app's startup), and the worst cold first call we measured is ~9.4s — inside the client's tool timeout. In an interactive tool-calling context, a typed "retry later" result would push the retry loop onto the LLM, which seemed worse than one bounded wait. For models whose load exceeds the tool timeout, your ready-flag approach is clearly the right call.

   The local-HTTP variant is a nice escape hatch too; for a single-client setup the pipe stays simpler (~20 lines, failures visible in the parent's stderr), so we kept it.

   (Same disclaimer as the issue: this reply was drafted by Claude Code — I can't personally field deep follow-ups on the internals.)
3. AlvaroBalbin commented on Jul 28, 2026

   

   AlvaroBalbin

   on Jul 28, 2026

   More actions

   I dug into the transport to check the mechanism you describe. On the server side there isn't a dedicated OS pipe-reader thread: stdio_server in src/mcp/server/stdio.py wraps sys.stdin with anyio.wrap_file(...) and does async for line in stdin. That iteration goes through anyio's AsyncFile.__aiter__, which calls readline() on every line, and each readline is dispatched as to_thread.run_sync(fp.readline) (see anyio _core/_fileio.py). So the reader is a worker thread doing a blocking readline, and it can only complete once it holds the GIL. That lines up with your observation that a GIL-bound load (torch, onnxruntime) starves it no matter where you offload it, since threadpool offload doesn't drop the GIL for CPU-bound work. One clarification: this path uses threads on every platform, not only Windows, so the contention isn't Windows-specific in the SDK itself, though blocking console/pipe reads there likely make the tail latency worse.
4. maxisbey

   added

   enhancementRequest for a new feature that's not currently supportedRequest for a new feature that's not currently supported

   P3Nice to haves, rare edge casesNice to haves, rare edge cases

   documentationImprovements or additions to documentationImprovements or additions to documentation

   needs decisionIssue is actionable, needs maintainer decision on whether to implementIssue is actionable, needs maintainer decision on whether to implement

   v1Affects the v1.x maintenance lineAffects the v1.x maintenance line

   v2Affects the v2 line (2.x on main)Affects the v2 line (2.x on main)

   on Aug 14, 2026
5. snehasonkusare-tech commented on Sep 28, 2026

   

   snehasonkusare-tech

   on Sep 28, 2026

   More actions

   Hi, I'd like to take the docs part of this issue.

   I work with local ML models (fine-tuning and serving small LLMs), so I've run into this kind of startup blocking myself. My plan is to add a short section to the server docs covering:

   why CPU/GIL-bound init (e.g. loading a torch or sentence-transformers model) still blocks the stdio transport even when offloaded to a threadpool
   the symptom: the client silently drops the server during initialize
   the recommended pattern: run the heavy work in a separate subprocess, with a minimal example
   a note that threadpool offload is still correct for I/O-bound blocking calls

   This would be a docs-only change with no changes to src/. Would a PR like this be welcome? If so, could you assign it to me?