gh-158097: Make test_staggered_race_with_eager_tasks deterministic - #158469
Open
abhinav-phi wants to merge 1 commit into
Open
abhinav-phi wants to merge 1 commit into
abhinav-phi wants to merge 1 commit into
Conversation
staggered_race() awaits the coroutines it is given inline, so the "await asyncio.sleep(0)" in the test's fail() helper suspended the run_one_coro() task running it and left a scheduled resumption behind. If the asyncio.sleep(1) winner (started 0.25s late by the stagger delay) completes before that resumption runs, the winner's cancel() sets _must_cancel and the resumption throws CancelledError into fail(), so excs[2] reports a CancelledError instead of a ValueError. Pending timers are only moved onto the ready queue at the top of BaseEventLoop._run_once() and are appended after the handles already queued, so the t=0.50s stagger timer and the t=1.25s sleep(1) timer run in due-time order. When both are due in the same _run_once() - which is what a busy CI machine causes - the stagger timer starts fail(), which immediately suspends again, and the very next handle is the winner. Raise straight away instead, so the ValueError is stored in excs[2] in the same step that starts the coroutine and there is no longer a scheduled resumption for a cancellation to overtake. The outcome no longer depends on the event loop getting another iteration.
abhinav-phi
requested review from
1st1,
asvetlov,
kumaraditya303 and
willingc
as code owners
September 30, 2026 02:11
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #158097.
Root cause
asyncio.staggered.staggered_race()awaits the coroutines it is given inline —result = await coro_fn()inrun_one_coro()(Lib/asyncio/staggered.py:121) — so each one runs inside the frame of therun_one_coro()task that awaits it, not in a task of its own.The failing helper in the test yielded once before raising:
That left it still unfinished at the moment the race was won, and
staggered_race()cancels every coroutine that has not finished as soon as one of them wins. Concretely,await asyncio.sleep(0)suspends therun_one_coro()task that is runningfail()and leaves a scheduled resumption behind. If the winner completes before that resumption runs, itscancel()sets_must_cancel, the pending resumption throwsCancelledErrorintofail(), andrun_one_coro()stores that inexcs[2]instead of theValueError.The schedule the test silently depends on, traced on a
--with-pydebugbuild:blocked()starts (index 0, never finishes)asyncio.sleep(1)starts (index 1), due at 1.25 sfail()starts (index 2) and suspends on itssleep(0)asyncio.sleep(1)finishes → winner → every unfinished coroutine is cancelledPending timers are only moved onto the ready queue at the top of
BaseEventLoop._run_once(), and they are appended after the handles that are already queued. So the t=0.50 s stagger timer and the t=1.25 sasyncio.sleep(1)timer run in due-time order, and when both come due in the same_run_once()the stagger timer startsfail(), which suspends again immediately, and the very next handle is the winner. Anything that deschedules the process for more than ~0.75 s does it — routine on a busy CI machine, which is where the OpenEmbedded report came from.This is not an asyncio bug: a loser that has not finished is documented to be cancelled. The test was asserting on an exception that
fail()had not yet been given a chance to raise.The fix
Raise immediately, without suspending first. Because the coroutine is awaited inline, the
ValueErroris then raised and stored inexcs[2]in the very same step ofrun_one_coro()that started it, so there is no scheduled resumption left for a cancellation to overtake, and the outcome no longer depends on the event loop getting another iteration.blocked()still suspends and is still cancelled by the winner, andasyncio.sleep(1)still suspends and still wins, so the assertions onexcs[0]andexcs[index]are unchanged. The siblingtest_staggered_race_with_eager_tasks_no_delayalready uses an immediately-raisingfail(), andtest_asyncio/test_staggered.pycoversstaggered_race()'s own cancellation semantics with fully deterministic timing. No asyncio library code is touched.Verification
Built
main(00307b0, 3.16.0a0) with./configure --with-pydebug.The failure is real and reproducible. Blocking the event loop while it waits for the stagger timer reproduces the issue's traceback verbatim —
line 240, in run / self.assertIsInstance(excs[2], ValueError) / AssertionError: CancelledError() is not an instance of <class 'ValueError'>— and the threshold is exactly the ~0.75 s the test depends on:mainThe test now passes consistently, with and without that injection:
setUp/tearDownper run, 10 workers × 500 iterationsregrtest, fresh interpreter per run, 10 workers × 150 runs./python -m test -j10 test_asyncioAI tools were used assistively; I reviewed and tested every change and take responsibility for it.