Fix Flaky Tests

.ai/PERFORMANCE.md

@@ -88,8 +88,11 @@ production bundle, runs the harness against `baseline.json`, and:
   over its absolute `fail_ms`, or `> 2x` the baseline p90 *after normalizing out
   machine speed* (see "Machine-independent gating" above);
 - **warns** (without failing) on a smaller drift - over `warn_ms`, or `> 1.3x`
-  the normalized baseline - and always upserts a single sticky **PR comment**
-  with the table (and the machine scale) so the numbers are visible on every run;
+  the normalized baseline - and always writes the table (and the machine
+  scale) to the **job summary** so the numbers are visible on every run;
+- upserts a single sticky **PR comment** with the same table, for PRs from
+  branches in this repo only. Fork PRs get a read-only `GITHUB_TOKEN`, so the
+  comment is skipped and the job summary is the place to look;
 - uploads `results.json` + `summary.md` as artifacts.
 
 Thresholds live per-scenario in `scenarios.py` (`warn_ms` / `fail_ms`, keyed by

.github/workflows/benchmarks.yml

@@ -100,8 +100,14 @@ jobs:
             benchmarks/summary.md
           retention-days: 30
 
+      - name: Write results to the job summary
+        if: always()
+        run: cat benchmarks/summary.md >> "$GITHUB_STEP_SUMMARY" || true
+
       - name: Comment results on the PR
-        if: always() && github.event_name == 'pull_request'
+        if: >-
+          always() && github.event_name == 'pull_request'
+          && github.event.pull_request.head.repo.full_name == github.repository
         uses: actions/github-script@v7
         with:
           script: |
@@ -127,7 +133,7 @@ jobs:
         if: always()
         run: |
           if [ "${{ steps.bench.outputs.status }}" != "0" ]; then
-            echo "Benchmarks exceeded a hard (fail) threshold. See the PR comment."
+            echo "Benchmarks exceeded a hard (fail) threshold. See the job summary."
             exit 1
           fi
 

CHANGELOG.md

@@ -25,6 +25,7 @@ This project adheres to [Semantic Versioning](https://semver.org/).
 - [#3986](https://github.com/plotly/dash/pull/3986) Adjust `_run_before_hooks` in the `fastapi` backend to honor a response returned by a `before_request` function, matching the `flask` backend's behavior.
 
 ### Fixed
+- [#4041](https://github.com/plotly/dash/pull/4041) Fix flaky integration tests by replacing fixed sleeps and immediate asserts with waits, and remove their `@flaky` retry markers.
 - [#3980](https://github.com/plotly/dash/pull/3980) Fix the three `before_request` hooks (`Dash._setup_server` and the pages `router_sync` / `router_async`) publishing their "already done" guard flag before the setup work behind it had run. Under a multi-threaded WSGI worker such as `gunicorn -k gthread` (or under an ASGI worker for the async router), a second request arriving mid-setup could observe the flag already set, skip setup, then read `registered_paths` / `callback_map` / the pages router callback while they were still being registered - causing the first burst of component bundle requests after a restart to 500 with `Error loading dependency. "<lib>" is not a registered library`, or the pages router to hit `DuplicateCallback` when two workers raced past the guard. Each hook body now runs under a lock (`threading.Lock` for the two sync hooks, an `asyncio.Lock` bound to the running loop for the async router) and only publishes the flag after all work completes. Fixes [#3971](https://github.com/plotly/dash/issues/3971).
 - [#3944](https://github.com/plotly/dash/pull/3944) Fix `dash.testing` runner backend detection for wrapped FastAPI/Quart servers so threaded Flask-only options are not passed to ASGI runners.
 - [#3955](https://github.com/plotly/dash/pull/3955) Unpin `selenium` in the testing requirements (was capped at `<=4.2.0` from 2022) and require `>=4.11.0`, so it can drive current stable Chrome via Selenium Manager and stop the widespread CI flakiness.

components/dash-core-components/src/components/Input.tsx

@@ -208,9 +208,9 @@ function Input({
 
             input.current.value = roundedValue.toString();
             setValue(roundedValue.toString());
-            onEvent();
+            setPropValue(props.value, roundedValue);
         },
-        [step, props.min, props.max, onEvent]
+        [step, props.min, props.max, props.value, setPropValue]
     );
 
     useEffect(() => {

components/dash-core-components/tests/integration/dropdown/test_a11y.py

@@ -1,6 +1,6 @@
 import pytest
-from flaky import flaky
 from dash import Dash, Input, Output
+from dash.testing import wait
 from dash.dcc import Dropdown
 from dash.html import Div, Label, P, Span
 from selenium.common.exceptions import TimeoutException
@@ -261,9 +261,6 @@ def test_a11y005_selection_visibility_multi(dash_duo):
     assert dash_duo.get_logs() == []
 
 
-# ActionChains keyboard nav depends on the menu input having focus, which can
-# lag menu-open under CI load and drop a keystroke; retry the test when it does.
-@flaky(max_runs=3)
 def test_a11y006_multi_select_keyboard_focus_retention(dash_duo):
     def send_keys(key):
         actions = ActionChains(dash_duo.driver)
@@ -296,20 +293,21 @@ def update_output(value):
     dropdown = dash_duo.find_element("#dropdown")
     dropdown.click()
     dash_duo.wait_for_element(".dash-dropdown-options")
+    wait_for_focus(dash_duo, "search")
 
     # Select 3 items by alternating ArrowDown and Spacebar
     send_keys(Keys.ARROW_DOWN)  # Move to first option
-    sleep(0.05)
+    wait_for_focus(dash_duo, "Option 0")
     send_keys(Keys.SPACE)  # Select Option 0
     dash_duo.wait_for_text_to_equal("#output", "Selected: ['Option 0']")
 
     send_keys(Keys.ARROW_DOWN)  # Move to second option
-    sleep(0.05)
+    wait_for_focus(dash_duo, "Option 1")
     send_keys(Keys.SPACE)  # Select Option 1
     dash_duo.wait_for_text_to_equal("#output", "Selected: ['Option 0', 'Option 1']")
 
     send_keys(Keys.ARROW_DOWN)  # Move to third option
-    sleep(0.05)
+    wait_for_focus(dash_duo, "Option 2")
     send_keys(Keys.SPACE)  # Select Option 2
     dash_duo.wait_for_text_to_equal(
         "#output", "Selected: ['Option 0', 'Option 1', 'Option 2']"
@@ -400,24 +398,12 @@ def update_output(value):
     assert dash_duo.get_logs() == []
 
 
-@flaky(max_runs=3)
 def test_a11y008_home_end_pageup_pagedown_navigation(dash_duo):
     def send_keys(key):
         actions = ActionChains(dash_duo.driver)
         actions.send_keys(key)
         actions.perform()
 
-    def get_focused_option_text():
-        return dash_duo.driver.execute_script(
-            """
-            const focused = document.activeElement;
-            if (focused && focused.closest('.dash-options-list-option')) {
-                return focused.closest('.dash-options-list-option').textContent.trim();
-            }
-            return null;
-            """
-        )
-
     app = Dash(__name__)
     app.layout = Div(
         [
@@ -434,6 +420,7 @@ def get_focused_option_text():
     dropdown = dash_duo.find_element("#dropdown")
     dropdown.send_keys(Keys.ENTER)  # Open with Enter key
     dash_duo.wait_for_element(".dash-dropdown-options")
+    wait_for_focus(dash_duo, "search")
 
     # Navigate from search input to options
     send_keys(Keys.ARROW_DOWN)  # Move from search to first option
@@ -443,8 +430,7 @@ def get_focused_option_text():
     send_keys(Keys.ARROW_DOWN)  # Move to third option
     sleep(0.05)
     send_keys(Keys.ARROW_DOWN)  # Move to fourth option
-    sleep(0.05)
-    assert get_focused_option_text() == "Option 3"
+    wait_for_focus(dash_duo, "Option 3")
 
     send_keys(Keys.HOME)  # Should go back to search input (index 0)
     # Verify we're back at search input
@@ -454,35 +440,29 @@ def get_focused_option_text():
 
     # Now arrow down to first option
     send_keys(Keys.ARROW_DOWN)
-    sleep(0.1)
-    assert get_focused_option_text() == "Option 0"
+    wait_for_focus(dash_duo, "Option 0")
 
     # Test End key - should go to last option
     send_keys(Keys.END)
-    sleep(0.1)
-    assert get_focused_option_text() == "Option 49"
+    wait_for_focus(dash_duo, "Option 49")
 
     # Test PageUp - should jump up by 10
     send_keys(Keys.PAGE_UP)
-    sleep(0.1)
-    assert get_focused_option_text() == "Option 39"
+    wait_for_focus(dash_duo, "Option 39")
 
     # Test PageDown - should jump down by 10
     send_keys(Keys.PAGE_DOWN)
-    sleep(0.1)
-    assert get_focused_option_text() == "Option 49"
+    wait_for_focus(dash_duo, "Option 49")
 
     # Test PageUp from middle
     send_keys(Keys.HOME)  # Back to search input (index 0)
     send_keys(Keys.PAGE_DOWN)  # Jump to index 10 (Option 9)
     sleep(0.1)
     send_keys(Keys.PAGE_DOWN)  # Jump to index 20 (Option 19)
-    sleep(0.1)
-    assert get_focused_option_text() == "Option 19"
+    wait_for_focus(dash_duo, "Option 19")
 
     send_keys(Keys.PAGE_UP)  # Jump to index 10 (Option 9)
-    sleep(0.1)
-    assert get_focused_option_text() == "Option 9"
+    wait_for_focus(dash_duo, "Option 9")
 
     assert dash_duo.get_logs() == []
 
@@ -515,10 +495,11 @@ def update_output(value):
     dropdown = dash_duo.find_element("#dropdown")
     dropdown.click()
     dash_duo.wait_for_element(".dash-dropdown-search")
+    wait_for_focus(dash_duo, "search")
 
     # Type to filter, then Enter selects the first visible option
     send_keys("a")
-    sleep(0.1)
+    wait_for_option_texts(dash_duo, ["Apple", "Banana"])
     send_keys(Keys.ENTER)
     dash_duo.wait_for_text_to_equal("#output", "Selected: ['Apple']")
     assert dash_duo.driver.execute_script(
@@ -535,7 +516,7 @@ def update_output(value):
     # Filtering to a different option selects that one
     send_keys(Keys.BACKSPACE)
     send_keys("b")
-    sleep(0.1)
+    wait_for_option_texts(dash_duo, ["Banana"])
     send_keys(Keys.ENTER)
     dash_duo.wait_for_text_to_equal("#output", "Selected: ['Banana']")
 
@@ -570,9 +551,10 @@ def update_output(value):
     dropdown = dash_duo.find_element("#dropdown")
     dropdown.click()
     dash_duo.wait_for_element(".dash-dropdown-search")
+    wait_for_focus(dash_duo, "search")
 
     send_keys("a")
-    sleep(0.1)
+    wait_for_option_texts(dash_duo, ["Apple", "Banana"])
     send_keys(Keys.ENTER)
     dash_duo.wait_for_text_to_equal("#output", "Selected: Apple")
 
@@ -862,3 +844,31 @@ def test_a11y009_dropdown_component_labels_render_correctly(dash_duo):
     assert rendered_labels == ["red", "yellow", "blue"]
 
     assert dash_duo.get_logs() == []
+
+
+def wait_for_focus(dash_duo, target):
+    # The dropdown moves focus in a requestAnimationFrame, so a key sent before
+    # focus lands goes to the wrong element. target is "search" for the search
+    # input, or the text of an option.
+    wait.until(
+        lambda: dash_duo.driver.execute_script(
+            "const el = document.activeElement;"
+            "const option = el.closest('.dash-options-list-option');"
+            "return option ? option.textContent.trim() : el.type;"
+        )
+        == target,
+        3,
+    )
+
+
+def wait_for_option_texts(dash_duo, texts):
+    # Enter acts on the search value from the last render, so let the filter
+    # render first.
+    wait.until(
+        lambda: dash_duo.driver.execute_script(
+            "return [...document.querySelectorAll('.dash-dropdown-option')]"
+            ".map(o => o.textContent.trim());"
+        )
+        == texts,
+        3,
+    )

components/dash-core-components/tests/integration/dropdown/test_dropdown_search_order.py

@@ -1,5 +1,5 @@
 from dash import Dash, html, dcc, Input, Output
-from flaky import flaky
+from dash.testing import wait
 from selenium.webdriver.common.keys import Keys
 from selenium.webdriver.common.action_chains import ActionChains
 from time import sleep
@@ -39,14 +39,23 @@ def test_ddso001_search_preserves_custom_order(dash_duo):
     assert dash_duo.get_logs() == []
 
 
-# Keyboard nav here goes through ActionChains (document.activeElement); the
-# selection depends on the menu input having focus, which can lag menu-open under
-# CI load and drop a keystroke. Retry the whole test when that happens.
-@flaky(max_runs=3)
 def test_ddso002_multi_search_preserves_custom_order(dash_duo):
     def send_keys(key):
         ActionChains(dash_duo.driver).send_keys(key).perform()
 
+    def wait_for_focus(target):
+        # The dropdown moves focus in a requestAnimationFrame, so a key sent
+        # before focus lands goes to the wrong element.
+        wait.until(
+            lambda: dash_duo.driver.execute_script(
+                "const el = document.activeElement;"
+                "const option = el.closest('.dash-options-list-option');"
+                "return option ? option.textContent.trim() : el.type;"
+            )
+            == target,
+            3,
+        )
+
     app = Dash(__name__)
     app.layout = html.Div(
         [
@@ -70,12 +79,13 @@ def update_output(value):
     dropdown = dash_duo.find_element("#dropdown")
     dropdown.click()
     dash_duo.wait_for_element(".dash-dropdown-options")
+    wait_for_focus("search")
 
     # Select '12' (second option)
     send_keys(Keys.ARROW_DOWN)
     sleep(0.2)
     send_keys(Keys.ARROW_DOWN)
-    sleep(0.2)
+    wait_for_focus("12")
     send_keys(Keys.SPACE)
     dash_duo.wait_for_text_to_equal("#output", "Selected: ['12']")
     sleep(0.2)
@@ -84,7 +94,7 @@ def update_output(value):
     send_keys(Keys.ARROW_DOWN)
     sleep(0.2)
     send_keys(Keys.ARROW_DOWN)
-    sleep(0.2)
+    wait_for_focus("111")
     send_keys(Keys.SPACE)
     dash_duo.wait_for_text_to_equal("#output", "Selected: ['12', '111']")
     sleep(0.2)
@@ -93,12 +103,13 @@ def update_output(value):
     send_keys(Keys.HOME)
     sleep(0.2)
     send_keys("1")
-    sleep(0.2)
 
     # Presents selected options first and rest in original order
-    options = dash_duo.find_elements(".dash-dropdown-option")
-    assert len(options) == 5
-    assert [opt.text for opt in options] == ["12", "111", "11 Text", "112", "110"]
+    wait.until(
+        lambda: [opt.text for opt in dash_duo.find_elements(".dash-dropdown-option")]
+        == ["12", "111", "11 Text", "112", "110"],
+        3,
+    )
 
     assert dash_duo.get_logs() == []
 

components/dash-core-components/tests/integration/input/test_number_input.py

@@ -53,7 +53,7 @@ def test_inni003_invalid_numbers_range(dash_dcc, input_range_app):
 
     elem_range = dash_dcc.find_element("#range")
     elem_range.send_keys("1999")
-    assert dash_dcc.find_element("#out").text == "1999"
+    dash_dcc.wait_for_text_to_equal("#out", "1999")
 
     for invalid_number in ("0.0", "12", "10e10"):
         elem_range.send_keys(invalid_number)
@@ -283,3 +283,27 @@ def update_output(val):
     dash_dcc.wait_for_text_to_equal("#output", "16")
 
     assert dash_dcc.get_logs() == []
+
+
+def test_inni012_stepper_clicks_during_debounce(dash_dcc, debounce_number_app):
+    dash_dcc.start_server(debounce_number_app)
+
+    input_elem = dash_dcc.find_element("#input-fast")
+    increment_btn = dash_dcc.find_element("#input-fast~.dash-stepper-increment")
+    decrement_btn = dash_dcc.find_element("#input-fast~.dash-stepper-decrement")
+
+    input_elem.send_keys("100")
+    dash_dcc.wait_for_text_to_equal("#div-fast", "100")
+
+    # Each click lands inside the 0.25s debounce started by the previous one.
+    for _ in range(5):
+        increment_btn.click()
+        dash_dcc.wait_for_text_to_equal("#div-fast", "103")
+        decrement_btn.click()
+        dash_dcc.wait_for_text_to_equal("#div-fast", "100")
+
+    time.sleep(0.5)
+    assert input_elem.get_attribute("value") == "100"
+    assert dash_dcc.find_element("#div-fast").text == "100"
+
+    assert dash_dcc.get_logs() == []