Fix lost pool timers and the py:call reentrancy deadlock - #86
Merged
Merged
Conversation
Every event loop pool loop drove a Python loop attached to the default loop's capsule, so its timers went to the default worker, its pending queue was the default loop's, and the per-loop callback ids collided there. With several tasks sleeping at once only one ever woke up. Give each pool loop a capsule over its own resource, send a timer to the loop that set it and let the worker dispatch the expiry there, key timer refs process-wide, and release the Python loop before a pool loop is destroyed. A worker no longer builds a Python loop for a loop that has no tasks, so raw pending events stay pollable.
A function called with py:call that called erlang.call blocked the context thread on the thread worker pipe, and the callback's nested py:call waited on a busy context until the request timeout. The context thread now sends the callback to its py_context process and serves its own request queue while it waits for the reply, so the callback's py:call runs on the same thread at any depth and the Python code still runs exactly once. Inside a running asyncio loop the thread path stays. The callback process is bound to that context and inherits the caller's process-local env, so the README example runs as written. A resume of a suspended py:eval now replays in the namespace of the original request instead of the context globals. The shared buffer flow control and the sync erlang.sleep use a blocking call that never suspends, so a py:eval around them is not replayed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hornbeam hit two bugs in the embedded contexts.
Tasks on the event loop pool that awaited
asyncio.sleepfor more than a few milliseconds never completed when several ran at once: with 24 concurrent 50 ms sleeps, 23 timed out. Every pool loop drove a Python loop attached to the default loop's capsule, so its timers went to the default worker and its callback ids collided in the default queue. Each pool loop now drives a loop bound to its own resource, a timer expiry is dispatched to the loop that set it, and a loop with no tasks leaves its pending events to whoever polls it.A Python function called with
py:callthat callederlang.call, where the callback calledpy:callagain, hung until the request timeout and then failed with "callback synchronisation lost; retry". The context thread blocked on the thread worker pipe and the nested call waited on that context. The thread now serves its own request queue while it waits for the callback, so the nestedpy:callruns inline on the same context at any depth, and the Python code still runs exactly once. Suspension with replay stays apy:evalmechanism, since the shared buffer flow control and the documentederlang.callcontract depend on exactly-once execution frompy:call.Checking the README reentrant example turned up two more defects on that path: a resumed
py:evalreplayed against the context globals instead of the request's namespace, and the callback process could not see the caller's__main__functions. Both are fixed, and the README example is now a test.