[Bug] ADK 2.8.0: repeated HITL deadlocks on replay within one invocation (no LLM) #7027

Activity

1. chelsealong commented on Sep 5, 2026

   

   chelsealong

   on Sep 5, 2026

   Contributor

   More actions

   Picking this one up now — opening a PR shortly. Flagging it here so nobody duplicates the work; if someone is already on it, say so and I will drop mine.
2. surajksharma07

   self-assigned this

   on Sep 7, 2026
3. sanketpatil06

   added the

   Bug

   issue type on Sep 7, 2026
4. sanketpatil06

   added

   agent engine[Component] This issue is related to Vertex AI Agent Engine[Component] This issue is related to Vertex AI Agent Engine

   on Sep 7, 2026
5. markns commented on Sep 7, 2026

   

   markns

   on Sep 7, 2026

   More actions

   Confirming this on 2.8.0 and on main (b0180620f), and adding a second shape that #7028 as currently written does not cover.

   Second shape: LlmAgent nodes with output_schema in a clarify loop

   Our workflow walks a product down three levels. Each level is an LlmAgent node with an output_schema (a "resolved" pick or a "needs clarification" question), followed by a rerun_on_resume=True function node that either routes onward or yields RequestInput and, on the answer, routes back to the classifier:

   START -> classify_a -> resolve_a --reclassify--> classify_a
                                   --resolved----> classify_b -> resolve_b --reclassify--> classify_b
                                                                           --resolved----> classify_c -> resolve_c -> done
   

   The third resume deadlocks:

   ADK	as Workflow root	as NodeTool under an LlmAgent
   2.8.0	Timed out waiting for sequence key 'classify_a@1'	never resumes (separate issue, fixed by 6d1451806)
   main b0180620f	'classify_a@1'	'classify_a@1', surfaced as the tool's error result
   main + #7028	'classify_a@2'	'classify_a@2'
   main + #7028 + the change below	completes	completes, parent gets the FunctionResponse

   Why #7028 misses it

   #7028 fixes the position of a run id at its first genuine terminal event (output, route or error_code) and ignores later echoes. That works for the issue's HITL node, which yields Event(output=..., route=...) itself. An LlmAgent node with output_schema never persists an event carrying output or route of its own, so the first terminal event _scan_sequence sees for classify_a@2 is the echo from _maybe_reemit_replayed_output on the next resume. That echo lands after resolve_a@2's genuine route event, so the barrier expects resolve_a@2 before classify_a@2, which can never happen.

   Instrumented barrier sequence on main + #7028, third resume:

   [SEQ] ['classify_a@1', 'resolve_a@1', 'resolve_a@2', 'classify_a@2', 'classify_b@1', ...]
   [BARRIER wait classify_a@2] next_expected=resolve_a@2
   RuntimeError: Replay divergence detected: Timed out waiting for sequence key 'classify_a@2' to be unblocked.
   

   A change that fixes both shapes

   Fix each segment's position at its first terminal event and never move it. A node cannot legitimately move behind a successor that depends on it: the interrupted-then-completed case keeps its original position, and the re-emitted echo becomes a no-op for ordering. On top of #7028's _scan_sequence:

         if is_terminal_event(event):
           # First-seen ordering: a child's position is fixed by its FIRST
           # terminal event. Later terminal events for the same run id are either
           # the re-emitted replay echo or an interrupted node completing after
           # its resume; neither can move the node behind a dependent successor.
           if segment not in sequence:
             sequence.append(segment)

   With that, tests/unittests/workflow/utils/test_replay_manager.py (including the new test from #7028), test_node_tool.py, test_workflow_hitl.py and utils/test_rehydration_utils.py pass (134 passed, 1 xfailed), and the reproduction below completes as root and as a NodeTool.

   Reproduction (no API key; scripted BaseLlm)

   NODETOOL=1 wraps the same workflow as a NodeTool under an LlmAgent root.

   """Three-level clarify loop with LlmAgent nodes (output_schema) deadlocks on the 3rd resume."""
   
   from __future__ import annotations
   
   import asyncio
   import hashlib
   import json
   import os
   from typing import Any, AsyncGenerator, Generator, Literal
   
   from google.adk import Agent, Context, Event, Workflow
   from google.adk.apps import App, ResumabilityConfig
   from google.adk.events import RequestInput
   from google.adk.models.base_llm import BaseLlm
   from google.adk.models.llm_response import LlmResponse
   from google.adk.runners import InMemoryRunner
   from google.adk.workflow import node
   from google.genai import types
   from pydantic import BaseModel, model_validator
   
   
   class ClassifyArgs(BaseModel):
       description: str
   
       @model_validator(mode="before")
       @classmethod
       def _accept_bare_string(cls, data):
           return {"description": data} if isinstance(data, str) else data
   
   
   class Pick(BaseModel):
       status: Literal["resolved", "needs_clarification"]
       value: str | None = None
       question: str | None = None
   
   
   class PickLlm(BaseLlm):
       """Clarification on the first call, resolved on the second."""
   
       model: str = "scripted"
       level: str = ""
       calls: int = 0
   
       async def generate_content_async(self, llm_request, stream: bool = False):
           self.calls += 1
           body = (
               {"status": "needs_clarification", "question": f"Clarify {self.level}?"}
               if self.calls == 1
               else {"status": "resolved", "value": f"{self.level}-ok"}
           )
           yield LlmResponse(content=types.Content(role="model", parts=[types.Part(text=json.dumps(body))]))
   
   
   def make_level(level: str):
       classify = Agent(
           name=f"classify_{level}",
           model=PickLlm(level=level),
           instruction="Classify {description}",
           output_schema=Pick,
           output_key=f"pick_{level}",
       )
   
       @node(name=f"resolve_{level}", rerun_on_resume=True)
       def resolve(ctx: Context) -> Generator[Any, None, None]:
           desc = ctx.state.get("description", "")
           pick = ctx.state.get(f"pick_{level}") or {}
           if hasattr(pick, "model_dump"):
               pick = pick.model_dump()
           iid = f"clarify_{level}_{hashlib.blake2b(desc.encode(), digest_size=4).hexdigest()}"
           answer = ctx.resume_inputs.get(iid)
           if answer is None:
               if pick.get("status") == "resolved":
                   yield Event(route="resolved")
                   return
               yield RequestInput(interrupt_id=iid, message=pick.get("question", "?"))
               return
           yield Event(state={"description": f"{desc}; {level}={answer}"}, route="reclassify")
   
       return classify, resolve
   
   
   def seed(node_input: ClassifyArgs | str | dict | None) -> Generator[Any, None, None]:
       d = node_input.description if isinstance(node_input, ClassifyArgs) else (node_input.get("description") if isinstance(node_input, dict) else (node_input or ""))
       yield Event(state={"description": str(d)})
   
   
   def done(ctx: Context) -> Generator[Any, None, None]:
       yield Event(output=f"classified: {ctx.state.get('description')}")
   
   
   classify_a, resolve_a = make_level("a")
   classify_b, resolve_b = make_level("b")
   classify_c, resolve_c = make_level("c")
   
   classify = Workflow(
       name="classify",
       description="Three-level clarify loop.",
       input_schema=ClassifyArgs,
       edges=[
           ("START", seed, classify_a, resolve_a),
           (resolve_a, {"resolved": classify_b, "reclassify": classify_a}),
           (classify_b, resolve_b),
           (resolve_b, {"resolved": classify_c, "reclassify": classify_b}),
           (classify_c, resolve_c),
           (resolve_c, {"resolved": done, "reclassify": classify_c}),
       ],
   )
   
   
   class RootLlm(BaseLlm):
       model: str = "scripted"
       calls: int = 0
   
       async def generate_content_async(self, llm_request: Any, stream: bool = False) -> AsyncGenerator[LlmResponse, None]:
           self.calls += 1
           part = (
               types.Part(function_call=types.FunctionCall(id="fc-1", name="classify", args={"description": "widget"}))
               if self.calls == 1
               else types.Part(text="done.")
           )
           yield LlmResponse(content=types.Content(role="model", parts=[part]))
   
   
   if os.environ.get("NODETOOL") == "1":
       root = Agent(name="root", model=RootLlm(), instruction="Call classify.", tools=[classify])
   else:
       root = classify
   app = App(name="repro", root_agent=root, resumability_config=ResumabilityConfig(is_resumable=True))
   
   
   def _interrupt_ids(events):
       return [p.function_call.id for e in events for p in ((e.content.parts if e.content else None) or []) if p.function_call and p.function_call.name == "adk_request_input"]
   
   
   def _answer(fc_id: str) -> types.Content:
       return types.Content(role="user", parts=[types.Part(function_response=types.FunctionResponse(id=fc_id, name="adk_request_input", response={"result": "x"}))])
   
   
   async def main() -> None:
       print(f"root = {type(root).__name__} {root.name}")
       runner = InMemoryRunner(app=app, app_name="repro")
       session = await runner.session_service.create_session(app_name="repro", user_id="u")
   
       async def turn(msg, label):
           print(f"--- {label} ---")
           events = []
           try:
               async for ev in runner.run_async(user_id="u", session_id=session.id, new_message=msg):
                   events.append(ev)
                   for p in ((ev.content.parts if ev.content else None) or []):
                       if p.function_call:
                           print(f"  FC {p.function_call.name} id={p.function_call.id}")
                       elif p.function_response:
                           print(f"  FR {p.function_response.name} -> {str(p.function_response.response)[:60]}")
                   if isinstance(ev.output, str):
                       print(f"  OUT {ev.output!r}")
           except BaseException as exc:  # noqa: BLE001
               print(f"  !! {type(exc).__name__}: {str(exc)[:120]}")
           return events
   
       evs = await turn(types.Content(role="user", parts=[types.Part(text="classify a widget")]), "turn 1")
       n = 1
       while (ids := _interrupt_ids(evs)) and n < 6:
           n += 1
           evs = await turn(_answer(ids[-1]), f"turn {n}: answer {ids[-1]}")
   
   
   if __name__ == "__main__":
       asyncio.run(main())

   Output on main (b0180620f), root:

   --- turn 1 ---
     FC adk_request_input id=clarify_a_97c9093e
   --- turn 2: answer clarify_a_97c9093e ---
     FC adk_request_input id=clarify_b_5a1ebffa
   --- turn 3: answer clarify_b_5a1ebffa ---
     FC adk_request_input id=clarify_c_6ec1c453
   --- turn 4: answer clarify_c_6ec1c453 ---
     !! RuntimeError: Replay divergence detected: Timed out waiting for sequence key 'classify_a@1' to be unblocked.
   

   With main + #7028 + the first-seen change, turn 4 prints OUT 'classified: classify a widget; a=x; b=x; c=x' instead, and with NODETOOL=1 the parent additionally receives FR classify -> {'result': ...}.

   Happy to open a PR with the change and a test for this shape if that helps, or to fold it into #7028.
6. surajksharma07 commented on Sep 7, 2026

   

   surajksharma07

   on Sep 7, 2026

   Collaborator

   More actions

   HITL@1 gets bumped behind Revise@1 once the fast-forward echo fires and the barrier deadlocks on the third resume, your root cause call was spot on. #7028's first-genuine-terminal-event fix is the right direction and clears the original script but markns's follow-up is real too: main + #7028 still hangs on the LlmAgent/output_schema shape since that node never has a genuine terminal event to anchor on. Tried the stricter first-seen-only tweak markns proposed on top of #7028's _scan_sequence and both repros pass clean. Worth folding that in before merge rather than closing this on the narrower fix alone.
7. copybara-service

   closed this as completedin 4b819abon Sep 9, 2026
8. pull

   added a commit that references this issue on Sep 10, 2026

   fix: stop a re-emitted output echo from reordering the replay sequence

   ...

   This issue will close once commit 4b819ab is merged into the 'main' branch.

   4b819ab