diff --git a/.claude/commands/coder-eval-code-review-full.md b/.claude/commands/coder-eval-code-review-full.md index bb5538b3..afae25ca 100644 --- a/.claude/commands/coder-eval-code-review-full.md +++ b/.claude/commands/coder-eval-code-review-full.md @@ -347,7 +347,7 @@ in a value that doesn't match the formula. coder_eval has several pairs of structures that *must* stay in sync. When one is changed, check the other: - Models with the same field across types (e.g. `RunSummary` and `VariantAggregate`, `TaskDefinition` and `ResolvedTask`, `EvaluationResult` and the per-row `CriterionResult`): verify type, default, validator, and field description match. - Parallel orchestration code paths: `orchestration/batch.py` ↔ `orchestration/experiment.py`. A bug fixed in one routinely needs to be fixed in the other (precedent in this codebase: dataset fan-out, run_limits merging, lineage tracking). - - Parallel agent paths: `Orchestrator` ↔ any new driver (e.g. `isolation/docker_runner.py`) — does the driver preserve the `pending_turn` / `crashed=True TurnRecord` contract documented in CLAUDE.md? + - Parallel agent paths: `Orchestrator` ↔ any new driver (e.g. `isolation/docker_runner.py`) — does the driver preserve the `TurnOutcome` / `crashed=True TurnRecord` contract (a failed turn is an outcome; a cancelled turn is ended before it propagates)? - Parallel renderers: `reports/markdown.py` ↔ `reports/experiment.py` ↔ `reports/html.py` ↔ `reports/helpers.py` — if a new field is added to `EvaluationResult`, do all four render it (and if not, is that deliberate)? Flag any divergence as a finding even if the unchanged side is technically still correct in isolation — the divergence itself is the bug, and silent drift between parallel paths is one of the most expensive defects to debug later. @@ -368,11 +368,11 @@ in a value that doesn't match the formula. 6. **Verify conformance to extension-point contracts (agents, criteria, backends, drivers, renderers).** coder_eval is a plugin-based, agnostic, multi-agent core (Claude / Codex / NoOp agents via the BYOA SPI; auto-discovered criteria; Bedrock / Anthropic backends; in-process / docker drivers). For every registered member of one of these extension points, confirm it honors the documented contract — a member that *registers* but silently *violates* the contract is a high-severity defect that a "read the code" pass misses because the code looks locally fine. - - **Agents** (every `Agent` subclass in `agents/`): `communicate()` calls `self._begin_turn()` at the top and `self._end_turn_ok()` on the success path; `stop()` calls `self._mark_stopped()`; it does NOT override `discard_pending_turn()` / `get_state()`. It emits one `AgentStartEvent` at the top and a matching `AgentEndEvent` on EVERY exit path (success / crash / timeout — from a `finally`), with `TurnStart`/`TurnEnd` per turn and `ToolStart`/`ToolEnd` per tool (orphaned tools closed `status=unresolved`). Before any mid-turn `raise AgentCrashError` / `TurnTimeoutError`, `self.pending_turn` is set to a `crashed=True` `TurnRecord`. The returned `TurnRecord` is built ONLY by the internal `EventCollector` — flag any `TurnRecord(` hand-assembled outside the synthetic-crash path. If the agent shells out / holds OS resources, `stop()` / `kill()` / `kill_sync()` are real, and `kill_sync()` is synchronous (no `await` — it runs on the watchdog's non-asyncio thread). It registers via `registry.register("kind", Config)(Agent)` in a `register(registry)` hook on a `coder_eval.plugins` entry point with its own `type: Literal["kind"]` config — and it does NOT wire itself in by editing the `AgentKind` enum or `Orchestrator._create_agent` (which delegates to the registry's `create_agent()` factory); registration is via the SPI hook only. + - **Agents** (every `Agent` subclass in `agents/`): `communicate(..., iteration=)` opens one `TurnEmitter` via `self._open_emitter(...)`, writes the whole turn through it, and returns `finalize(...)` / `fail(...)` — a crash or timeout is a `TurnOutcome` with a `crashed=True` record, never a raised `AgentCrashError` / `TurnTimeoutError`; on `CancelledError` it calls `fail(CRASHED, "turn cancelled")` before re-raising. `stop()` calls `self._mark_stopped()`; it does NOT override `get_state()`. Flag any event, `AssistantMessage`, `EventCollector` or `TurnRecord(` built in an adapter. If the agent shells out / holds OS resources, `stop()` / `kill()` / `kill_sync()` are real, and `kill_sync()` is synchronous (no `await` — it runs on the watchdog's non-asyncio thread). It registers via `registry.register("kind", Config)(Agent)` in a `register(registry)` hook on a `coder_eval.plugins` entry point with its own `type: Literal["kind"]` config — and it does NOT wire itself in by editing the `AgentKind` enum or `Orchestrator._create_agent` (which delegates to the registry's `create_agent()` factory); registration is via the SPI hook only. - **Per-agent coverage when a new agent is added:** `Settings.validate_api_keys` has a branch for it (don't let it fall through silently — a recurring gap); it supports the run's backends (Bedrock / Anthropic / Azure-OpenAI) or fails with a clear error; it surfaces per-turn `total_cost_usd` so the `max_usd` budget gate can fire; and the token-bucket reconciliation invariant (Σ buckets across `TurnRecord.messages` == `token_usage`) holds, with a test. Agnostic-core litmus: `grep -ri src/coder_eval/` outside the agent's own package + the registry should be ~zero. - **Criteria** (every file in `criteria/`): carries `@register_criterion`, implements `_check_impl`, exposes `aggregate()`, is a member of the `SuccessCriterion` union, AND is re-exported from `coder_eval.models`. - **Backends / drivers / renderers:** every `ApiBackend` is handled in judge routing + pricing + `validate_api_keys`; every sandbox driver / preservation mode preserves the stale-artifact-clear, synthetic-`task.json`-on-death, and env-scrub contracts; every `reports*.py` renderer covers each `EvaluationResult` field / `FinalStatus`. - Several of these are statically enforceable — when you find a violation whose shape is grep-/AST-detectable (an `Agent` subclass missing `_begin_turn`, a bare `raise AgentCrashError` with no preceding `self.pending_turn =`, an `async def kill_sync`, a `TurnRecord(` built outside `EventCollector`, a criterion missing from the `SuccessCriterion` union), propose it as a `CEnnn` lint rule in the Harness & Lint pass. + Several of these are statically enforceable — when you find a violation whose shape is grep-/AST-detectable (an `Agent` subclass that never calls `_open_emitter`, a `raise AgentCrashError` inside `communicate`, an `async def kill_sync`, a `TurnRecord(` built outside `EventCollector`, a criterion missing from the `SuccessCriterion` union), propose it as a `CEnnn` lint rule in the Harness & Lint pass. Apply these techniques while reading. Findings produced this way go into the same output as ordinary findings, tagged with the appropriate axis and severity. ``` diff --git a/.claude/commands/coder-eval-create-plan.md b/.claude/commands/coder-eval-create-plan.md index 6061b29d..9b994538 100644 --- a/.claude/commands/coder-eval-create-plan.md +++ b/.claude/commands/coder-eval-create-plan.md @@ -47,7 +47,7 @@ Follow these steps: - Does this change touch the evaluation flow? (CLI → ExperimentRunner → run_batch → Orchestrator → Sandbox + Agent + SuccessChecker) - Does this affect the 5-layer config merge? (default.yaml → experiment defaults → task YAML → variant → CLI flags). Each list/dict field must declare its `MergeField` strategy (lint rule CE014). - If adding a new criterion: does it fit `BaseCriterion` / `@register_criterion` / the `SuccessCriterion` discriminated union? Does it need a custom `aggregate()` for suite thresholds? - - If adding a new agent: does it follow the plugin SPI (a `BaseAgentConfig` subclass + `Agent` ABC + a `register(registry)` hook exposed via the `coder_eval.plugins` entry-point group)? Does it use the shared turn lifecycle (`_begin_turn`/`_end_turn_ok`/`_mark_stopped`) and emit the standardized event protocol? + - If adding a new agent: does it follow the plugin SPI (a `BaseAgentConfig` subclass + `Agent` ABC + a `register(registry)` hook exposed via the `coder_eval.plugins` entry-point group)? Does `communicate(..., iteration=)` write the turn through one `TurnEmitter` (`_open_emitter`) and return a `TurnOutcome`, and does `stop()` call `_mark_stopped`? - Does this change the task YAML schema? If so, what happens to existing task files in `tasks/`? - Are there edge cases in sandbox isolation, agent lifecycle, retry/crash recovery, or token accounting? - Does this introduce new dependencies? Prefer what's already in the project (pydantic, typer, rich, anyio, anthropic). diff --git a/.claude/commands/coder-eval-implement-plan.md b/.claude/commands/coder-eval-implement-plan.md index 8ae4f326..21d30095 100644 --- a/.claude/commands/coder-eval-implement-plan.md +++ b/.claude/commands/coder-eval-implement-plan.md @@ -57,10 +57,10 @@ The plan's Master Acceptance Checklist and the **Review Criteria** below are the - **All models import from `coder_eval.models`** — never from submodules (lint-guarded). New models are exported from `models/__init__.py`. - **New criterion → two edits.** The `@register_criterion` checker in `criteria/` **and** the `SuccessCriterion` discriminated union in `models/criteria.py`. Discriminated unions use `Field(discriminator="type")` — a bare `A | B` union silently coerces. -- **New agent → plugin SPI, not enum dispatch.** Register via a `register(registry)` hook exposed through the `coder_eval.plugins` entry-point group; do **not** edit `Orchestrator._create_agent` (it already delegates to the registry's `create_agent()` factory) or the `AgentKind` enum (known built-in kinds only). Use the shared turn lifecycle (`_begin_turn`/`_end_turn_ok`/`_mark_stopped`) and emit the standardized event protocol through `EventCollector`. +- **New agent → plugin SPI, not enum dispatch.** Register via a `register(registry)` hook exposed through the `coder_eval.plugins` entry-point group; do **not** edit `Orchestrator._create_agent` (it already delegates to the registry's `create_agent()` factory) or the `AgentKind` enum (known built-in kinds only). Write the turn through one `TurnEmitter` (`_open_emitter`) and return its `TurnOutcome`; `stop()` calls `_mark_stopped`. - **Ripple completeness.** Adding/removing/renaming a model field, config key, or CLI flag means tracing every reference — task YAMLs in `tasks/`, experiment YAMLs in `experiments/`, `experiments/default.yaml`, `.claude/commands/`, docs, and `models/__init__.py`. - **Config merge.** New list/dict fields declare their `MergeField` strategy (CE014). New `ResolvedTask`/`AgentConfig` fields need coverage across all 5 layers and a matching `-D` override path. -- **Crash/retry hygiene.** On `AgentCrashError` / `TurnTimeoutError`, set the partial `crashed=True` TurnRecord on `pending_turn`, then raise bare; reset `_session_id`, `pending_turn`, watchdog refs, streaming `ContextVar`s, and iteration counters before the next attempt. +- **Crash/retry hygiene.** A failed turn returns an outcome with `record.crashed=True`; cancellation ends the turn with `fail(CRASHED, ...)` before it propagates; no cross-attempt state lives on the agent (reset `_session_id`, watchdog refs and streaming `ContextVar`s before the next attempt). - **`extra="forbid"`** on config models that consume YAML/CLI; **Haiku/Sonnet, never Opus** in tests (cost). ## Reference blocks diff --git a/.claude/harness-candidates.md b/.claude/harness-candidates.md index 4fc94fc1..78a3d3d8 100644 --- a/.claude/harness-candidates.md +++ b/.claude/harness-candidates.md @@ -647,6 +647,8 @@ divergences, so the deferred-work record is one place. Measurements in kwarg rule ever lands, extract `tests/lint/rules/_message_calls.py` at that point rather than sooner. Caught in: the CE060 / antigravity `message_id` run. + UPDATE: CE059 and CE060 are retired. CE072 bans an `AssistantMessage` call in + `agents/`, alias included, so only CE058's name list is still open. - [ ] **Nothing pins that `message_id` is only ever a WITHIN-TURN identity.** Ids repeat across retry attempts of one turn on every synthetic-id harness — @@ -808,8 +810,8 @@ re-derive from scratch. PRESENT at the call site (`ce059_generation_window_is_two_reads.py:68`), so removing the kwarg makes `claims_a_window` true at the three legitimate placeholder sites and forces a rule REWRITE rather than a retirement. Net - cost: five reducers, a regeneration of every golden, and a CE059 rework; net - benefit: SSOT alone. **Deferring it is safe because the seam assertion in + cost: five reducers, a regeneration of every golden, and a CE059 rework (CE059 + is now retired, so that part of the cost is gone); net benefit: SSOT alone. **Deferring it is safe because the seam assertion in `timing.subtract_tool_time` now checks the property at runtime** — a group's raw total must equal the span its own bounds describe — which also covers a third-party agent registered through the `coder_eval.plugins` SPI, where no @@ -826,7 +828,8 @@ re-derive from scratch. case was never about the clamp but about the head being measured against the wrong instant. REVISIT IF: an inversion is observed on a live run after CE064, which would mean a basis is still mixed somewhere the rule cannot see - (the plugin SPI, or a harness whose spans come from a CLI). + (the plugin SPI, or a harness whose spans come from a CLI). CE064 is now + retired: `TurnEmitter` stamps the bracket from the turn's one clock. - [ ] **`test_codex_golden[a_agent_message_only]` is FLAKY, ~5% — measured, and pre-existing.** Forty consecutive runs on an unmodified tree (`-n 0`): 2 @@ -967,6 +970,16 @@ re-derive from scratch. - [ ] OpenCode: warn when an inherited `OPENCODE_CONFIG_CONTENT` `permission` / `instructions` value is not a dict / list and is replaced — today it is dropped silently; small, but needs a decision on warn vs. keep — caught in the harness-contract Phase 3 review. - [ ] CE070 blind spot: an adapter that counts `ToolEndEvent`s (or tokens) under a new name to cap or stop a run itself — the rule matches identifiers only; needs a data-flow check that a counter in `agents/` feeds a break or an end status — caught in the central-enforcement plan (Phase 5). - [ ] CE070 blind spot: an adapter that re-grows a skill scanner through `glob("*.md")`, `rglob`, or a file name built from parts — the rule matches the literal `"SKILL.md"` only; needs a filesystem-walk classifier scoped to `agents/` — caught in the central-enforcement plan (Phase 5). -- [ ] Every harness's `TurnEndEvent.tokens` must be a per-report DELTA: over a turn, their sum per bucket must not exceed `AgentEndEvent.usage` (the TurnMonitor latches budgets on the sum) — nothing checks it; the golden-stream runners return only the TurnRecord, so each of the five `run_*_scenario` helpers needs an event sink first — caught in the central-enforcement final review (Claude re-reported an interleaved message id's tokens). -- [ ] Live tests (`-m live`) are neither run nor type-checked in `make verify`, so an SPI signature change (`communicate(max_turns=)`, bool `should_stop`) leaves them broken until someone runs them with credentials — needs pyright over `tests/*_live.py` or an import-time signature smoke test — caught in the central-enforcement live verification (Phase 6). - +- [x] ~~Every harness's `TurnEndEvent.tokens` must be a per-report DELTA: over a turn, their sum per bucket must not exceed `AgentEndEvent.usage` (the TurnMonitor latches budgets on the sum) — nothing checks it; the golden-stream runners return only the TurnRecord, so each of the five `run_*_scenario` helpers needs an event sink first — caught in the central-enforcement final review (Claude re-reported an interleaved message id's tokens).~~ **DONE.** Closed by the emitter plus `assert_stream_balanced`: `TurnEmitter` is the one per-turn accumulator on every harness and logs a WARNING when a bucket of the summed `TurnEndEvent.tokens` exceeds the published usage, and `coder_eval.testing.assert_stream_balanced` fails on the same condition over a replayed or live event stream. +- [x] ~~Live tests (`-m live`) are neither run nor type-checked in `make verify`, so an SPI signature change (`communicate(max_turns=)`, bool `should_stop`) leaves them broken until someone runs them with credentials — needs pyright over `tests/*_live.py` or an import-time signature smoke test — caught in the central-enforcement live verification (Phase 6).~~ **DONE.** Closed by pyright over live tests in `make verify` (a second pass whose generated config includes `tests/*_live.py` and the byoa demo fixture) plus `tests/test_harness_live.py`, which runs one tiny turn per installed harness through `communicate` and checks it with `assert_stream_balanced` and the bucket sums. + +- [ ] `SubprocessJsonlAgent` drains stderr with one unbounded `read()`: a CLI that floods stderr grows evaluator memory without limit — needs a bounded tail that keeps the crash message useful — caught in the turn-emitter final review (pre-existing in Pi/OpenCode). +- [ ] `TurnMonitor._resolved_tool_ids` and `EventCollector._commands` key on the raw `tool_id` for the whole task, while adapters mint fallback ids per invocation (Pi `call_N`): a reused id across dialog turns or retries is counted once and overwrites the earlier command — needs per-invocation scoping and a decision on what a retry counts — caught in the turn-emitter final review (pre-existing). +- [ ] `coder_eval.testing.assert_stream_balanced` tracks ONE open inner turn across threads, so a Claude sub-agent turn interleaved with a main-thread turn in a live stream would read as unbalanced — needs per-`parent_thread_id` tracking plus a replay fixture of the interleaving — caught in the turn-emitter Phase 9 review. +- [ ] CE072 matches a banned class by attribute name alone (`sdk_types.AssistantMessage(...)` false positive) and misses `model_validate` / `model_construct` / star imports — needs module-binding resolution for the attribute form — caught in the turn-emitter Phase 9 review. +- [ ] The budget smoke fixtures (`smoke_budget_exceeded.yaml`, `smoke_cost_budget_exceeded.yaml`) carry Claude-only `permission_mode` / `allowed_tools`, so a cross-harness run needs `-D` overrides (Codex cannot unset `permission_mode` at all) — needs a decision to make them multi-harness fixtures under the `tests/test_harness_contract.py` retype check — caught in the turn-emitter live verification (Phase 10). +- [ ] Claude Code: a tool result for an unknown id is synthesized as a main-thread call even when its user message carries `parent_tool_use_id`, so it counts toward `max_tool_calls` — needs the parent read from the result message — caught in the turn-emitter final review. +- [ ] A symlink fallback that catches a broad `OSError` around `symlink_to` and then copies hides EEXIST/ENOENT and can write into a source tree (the staging copy wrote plugin B into plugin A on a case-folding filesystem) — `link_or_copy` is the one site today and now re-raises; a rule needs an except-clause classifier around filesystem link calls — caught in the max-turns plan Phase 5 review. +- [ ] A config-derived name used as a directory entry must be one path segment and unique ignoring case — `plugin_staging._claim_name` is the one check for plugin and skill names; nothing flags a new `staging_dir / ` join — needs taint tracking from YAML/manifest/frontmatter values to `Path` joins — caught in the max-turns plan Phase 5 review. +- [ ] A plugin root that contains the run directory (`path: .`) is linked whole under `/plugin_root/plugins/`, which makes a directory cycle; today's run-dir walkers do not follow symlinks, but nothing guards a new `copytree(symlinks=False)` / `os.walk(followlinks=True)` / uploader over a run dir — needs a run-dir walker rule or a refusal decision — caught in the max-turns plan final review. +- [ ] `TurnMonitor.on_event` catches every exception (collector, cap counters, `_commit` pricing), not only the armed criteria its docstring names, so a raising collector silently under-counts a cap and a raising price mid-`_commit` can double-count tokens; `_tool_call_index` also advances on a re-emitted resolved `ToolEndEvent` that `tool_calls` dedupes — needs a narrower fail-open boundary plus tests — caught in the max-turns plan final review (pre-existing). diff --git a/.claude/notes/agents.md b/.claude/notes/agents.md index 75f36ed3..d03e3f24 100644 --- a/.claude/notes/agents.md +++ b/.claude/notes/agents.md @@ -10,7 +10,7 @@ on that id (the evalboard's `aggregateSubAgentUsage` does exactly this). Claude bubbles its sub-agent's intermediate generations into the parent stream natively, and the **terminal** generation (delivered as the Agent tool result, never streamed) is - synthesized into one via `_synthesize_subagent_terminal_message` from + synthesized into one via `_subagent_terminal_part` from `tool_use_result.usage`. Codex reconstructs all child generations from the child rollout; both harnesses' turn totals already include sub-agent cost. `CommandTelemetry.result_summary` is stored **untruncated** (no 200-char cap) so @@ -67,15 +67,39 @@ TurnMonitor); CE070 keeps adapters from counting one again. ## Shared turn lifecycle -Every adapter drives the same skeleton, on the base class: `_begin_turn()` resets the -pending slot and bumps the iteration counter, `_end_turn_ok()` marks the turn clean, and -`_mark_stopped()` closes the agent. Before raising on a mid-turn failure an adapter sets -`pending_turn` to a `crashed=True` `TurnRecord` and raises bare, which is what lets the -orchestrator drain the partial record and un-bump the iteration. - -The record is BUILT before `_end_turn_ok()` on every harness: a failure inside the -reduction is a failed turn, and `_end_turn_ok` would already have cleared the rollback -flag `discard_pending_turn` needs. +`communicate(..., iteration=...)` returns a `TurnOutcome` (Appendix C of the harness +target design). A crash or a timeout is an outcome with a `crashed=True` record, not an +exception, and the CALLER owns the iteration: a retry of the same turn passes the same +number. A side channel on the agent (a parked partial record, plus an iteration counter +rolled back once per failed turn) is cross-attempt state that every harness would have to +set correctly on every failure branch. + +The orchestrator maps the status in one place: the clean statuses (an explicit allowlist) +return the record; `CRASHED` / `TIMEOUT` append the record to the result and then raise +through `TurnOutcome.record_or_raise`, so the retry categorisation (a crash retries, a +timeout does not) is unchanged; anything else raises `RuntimeError`. + +A crash retries only when the crashed attempt made no tool call. The retry runs in the +same sandbox, which is not reset, so a crash after the agent wrote files would grade a +different experiment than the one authored. `record_or_raise` puts the tool-call count on +`AgentCrashError`; `execute_with_retry` does not retry an `AGENT_CRASH` that carries one. +A message that categorises as rate limit or API error keeps its own retry policy. + +Cancellation cannot return a value. A `CancelledError` from the task watchdog, the +orchestrator's `wait_for` backstop or the task timeout must keep propagating, or +`task_timeout` stops working. So an adapter ends the turn FIRST +(`fail(CRASHED, "turn cancelled")`) and re-raises, and the orchestrator reads the partial +record from a per-attempt `EventCollector` it attaches to the callback chain itself +(`_attempt_collector`). The agent-side and orchestrator-side records are the same events +through the same reducer, so they cannot differ. The attempt clears that collector on +every exit except a cancel, so a task timeout that fires later (during grading, between +retries) never appends a finished attempt twice. + +A clean turn's `result_summary.result` is the agent's final reply: the text that follows the +last tool call in the last main-thread message. After, not "a message with no tool call": +a live Antigravity turn writes its closing text in the same generation as its last tool +call, and Pi writes text BEFORE a tool call it then makes, which is not a reply. A failed +turn carries no summary; its failure is `crash_reason`. Three exit paths converge on `finalize`, and it is idempotent on all of them, because the protocol allows EXACTLY ONE `AgentEndEvent` per `communicate()`: the clean return, the @@ -221,10 +245,18 @@ wiped the span before `step_finish` could subtract it, a 100% overstatement of t An exit code of 0 with no telemetry is indistinguishable from a real pass in every aggregate, and file-based criteria can still score it SUCCESS. Worse, a turn with no tokens is one whose `max_total_tokens` / `max_usd` gates could never have tripped no -matter how much the run actually billed. So the CLI harnesses crash rather than score: - -- **Vocabulary drift** — a clean exit that recognized NO event from the harness's known - set. This has happened: OpenCode once parsed the `session.next.*` server vocabulary +matter how much the run actually billed. So the harnesses crash rather than score: + +- **An empty turn** (every harness) — `TurnEmitter.finalize(COMPLETED)` on a turn that + wrote nothing after `begin` (no inner turn, tool, text, generation, usage or + `agent_output`) ends `CRASHED`. It lives in the emitter, not in the transport base, + so Claude Code, Codex, Antigravity and plugins get it too. A requested stop is exempt, + because a cut can land before the first event. `noop` opens an inner turn, so it is + never empty. + +- **Vocabulary drift** (the JSONL transport) — a clean exit that recognized NO event from + the harness's known set. The empty-turn arm also catches it; the transport keeps its + own check for the message, which names the unrecognized event types. This has happened: OpenCode once parsed the `session.next.*` server vocabulary instead of the CLI's own and scored SUCCESS 1.0 with zero turns and zero tokens. - **Finished steps with no tokens** (OpenCode) — the same outcome one layer down. Keying on "recognized nothing" alone left it reachable: a `step_finish` carrying no `tokens` @@ -297,6 +329,10 @@ them a CLI upgrade silently zeroes the run's tokens and cost and blinds the budg ## Cost: the stream versus the rate card +`pricing.price_turn(usage, models)` is the one rule for a turn's cost. Every adapter and +`TurnMonitor` call it, and CE071 keeps `calculate_cost` out of both, because five copies +of the rule once let the `max_usd` stop and the persisted cost disagree on one turn. + A non-zero cost the CLI reported always wins — it is the provider's own accounting, and on OpenRouter per-request routing makes it strictly better than a static headline rate. The rate card fills two gaps that would otherwise book tokens with no money: @@ -309,13 +345,34 @@ rate card fills two gaps that would otherwise book tokens with no money: and understating cost silently defeats `max_usd`, which is the worse failure. A genuinely free model has an all-zero rate entry (or none), so it still resolves to 0. +Empty usage returns the reported cost unchanged, `None` included: `EventCollector` +publishes `token_usage=None` only for empty usage with no cost, so pricing an empty turn +at `0.0` would publish a zero-cost usage row for a turn that spent nothing. So an empty +turn reads `token_usage: null` on every harness (Codex, Antigravity and the LiteLLM route +used to price it at `0.0` on a priced model). The monitor adds its own "an empty turn +costs 0" in front, because it sums. A non-finite reported cost counts as unreported. + +The ORDER of `models` is the caller's decision. Adapters pass their one model. The monitor +passes `agent.model`, then the model the agent resolved at start, then the last model a +message reported: the configured model wins so that a sub-agent's model on the stream +cannot reprice the run. + +`max_usd` is never "accepted and ignored". `HarnessContract.reports_cost` says whether a +harness prices every finished turn itself (Claude Code; `noop` has no usage). Pi and +OpenCode do not count: they report $0 for a model their own registry does not price. On a +harness without `reports_cost`, resolution rejects `max_usd` unless `agent.model` has a +rate, and the monitor latches `USD_BUDGET` (then raises `BudgetUnenforceableError`) at the +first in-flight usage it cannot price, instead of counting it as $0 until the turn ends. +On a harness with `reports_cost`, in-flight usage without a rate still counts $0, because +the turn's end brings the harness's own cost. + The Claude SDK's own `costUSD` is a client-side estimate assuming Anthropic pricing, so it is wrong for an open-weight model behind LiteLLM and is repriced from the token buckets at -the model's real rate. The buckets are untouched, so the reconciliation invariant holds — -only the cost scalar changes. An unpriced model sets the cost to `None` (an honest N/A) -**and warns**. When the task sets `max_usd`, the `TurnMonitor` then raises -`BudgetUnenforceableError` at the turn end, so the row finishes `ERROR` and is never a -silent skip. +the model's real rate (`price_turn` with the report cleared). The buckets are untouched, so +the reconciliation invariant holds — only the cost scalar changes. An unpriced model sets +the cost to `None` (an honest N/A) **and warns**. When the task sets `max_usd`, the +`TurnMonitor` then raises `BudgetUnenforceableError` at the turn end, so the row finishes +`ERROR` and is never a silent skip. ## Codex rollout rebuild @@ -357,6 +414,35 @@ On a crash the SDK total never arrives and the per-generation tokens on the flus messages are used instead — but the baseline must still advance past them, or the next turn's delta re-books everything the crashed turn already reported. +## One inner turn per generation on Codex and Antigravity + +Both streams already carried a per-generation boundary — Codex's +`thread/tokenUsage/updated`, Antigravity's `usage_metadata` Step — and both adapters cut +one message there, yet each opened ONE inner turn per `communicate()`, so the +`TurnMonitor` would have counted calls, and `max_turns` was rejected at resolution. Now +the cut also closes an inner turn, carrying that generation's delta as +`TurnEndEvent.tokens`, and the contract declares `usage_granularity=GENERATION`: the +model-turn limits are accepted, and the token and USD budgets overshoot by one +generation instead of one whole turn. + +The inner turn opens LAZILY, at the first evidence of the generation — an item start, a +content item, a text delta, or a billed usage report with nothing else — and never +eagerly after a cut. Eager opening would count a turn that never happens when +`turn/completed` follows the last cut, and, worse, a tool RESULT landing after its +generation's cut (Codex patches `is_error` cross-flush; an Antigravity background job +resolves on a later poll) would open a turn of its own. So a Codex `item/completed` for a +call already open and an Antigravity DONE Step whose calls are all seen open nothing; +an unseen call does, because a call the model just made is the model speaking. The cap +therefore latches exactly where it does on Claude Code: when response N+1 arrives. + +The inner turn id is the message id it will cut to (`-msg-`), so the two line +up in the record. A billed cut with no content (a placeholder reasoning block that was +removed) still closes the turn and still advances the counter, or the next turn would +reuse a closed id and the monitor, which counts each id once per `communicate()`, would +miss it. The safety flush at the end of the pump (`last=None`, or Antigravity's +`end()` with trailing blocks) adds the message but leaves the turn for the emitter to +close with the turn's own end status, so a turn cut short by a stop reads as such. + ## Why the generation is split into sub-messages Codex flushes one generation as up to two `AssistantMessage`s — thinking and action — @@ -446,7 +532,7 @@ instant the sweep runs is not a completion. Stamping it manufactures both an a measured span that the central subtraction takes back out of a generation window it never occupied. `execution_started_at` IS kept: the harness really did emit that start, and one bound alone forms no span. Unknown status and unknown duration are one fact -(CE058) — claude-code's `_finalize_commands` leaves the same field `None` for the same +(CE058) — the emitter's sweep leaves the same field `None` on every harness for the same reason, rather than coercing it to `0.0`, which put an invented measurement on both sides of `avg_command_time_ms`. @@ -581,6 +667,27 @@ profiles and loses the prepend again. Nested zsh keeps it, because `ZDOTDIR` sta exported. No-op on Windows, where Codex shells through PowerShell (`-NoProfile`) or `cmd /c`, neither of which re-sources a profile chain. +## Why a CLI never inherits stdin + +`pi` (`readPipedStdin()`) and `opencode` (`process.stdin.isTTY ? void 0 : await +Bun.stdin.text()`) both read stdin TO EOF when it is not a TTY, before they emit anything. +A CLI spawned without `stdin=` inherits the parent's stdin, so when `coder-eval` itself runs +with stdin on a pipe that stays open (a backgrounded or tool-spawned batch), every CLI +blocks with zero events until the 300 s `turn_timeout`. Measured on 2026-09-16: + +| command | result | +|---|---| +| `(sleep 25) \| timeout 15 pi -p --mode json … "Reply PONG"` | 0 lines, killed at 25 s | +| `pi -p --mode json … "Reply PONG" < /dev/null` | 24 lines, exit 0 in 1 s | +| `(sleep 25) \| timeout 15 opencode run --format json … "Reply PONG"` | 0 lines, killed at 25 s | +| `coder-eval run tasks/pi_smoke_test.yaml -D run_limits.turn_timeout=40 < <(sleep 170)` | `ERROR` after 40 s, 0 commands | +| the same with `< /dev/null` | `SUCCESS` in 10 s | + +So every CLI spawn passes `stdin=asyncio.subprocess.DEVNULL`, which gives an immediate EOF. +The same inheritance reached the task's `pre_run`/`post_run` shell commands (an authored +`read` hung the task) and the `docker run` CLI, so those pass it too, and CE073 requires +every asyncio subprocess spawn under `src/` to decide its stdin. + ## Reaping the CLI harnesses `opencode run` leaves a local server child alive after the CLI exits, and it INHERITS the @@ -589,7 +696,19 @@ deadline, and signalling only the CLI pid orphans the child. Each invocation the in its own session, so its pgid is the CLI's pid and the group holds only what that invocation spawned; each read races against process exit, and a bounded drain collects the tail. Sessions are persisted on disk, so killing a turn's server does not lose `--session` -continuity. +continuity. The group is swept at the end of EVERY turn, a clean one too: a child left +alive until `stop()` keeps running against the sandbox, and its pgid can be reused by +another task's CLI by the time `stop()` signals it. + +Exit is detected by polling `returncode`, never by `Process.wait()` alone. On CPython 3.13 +`wait()` resolves only once every pipe closes, so a child that holds stdout keeps it +pending and the bounded drain never starts: a clean exit waited for the child, or became a +TIMEOUT. + +`KILL_GRACE_SECONDS` (SIGTERM to SIGKILL) must stay below the orchestrator's +`_WAIT_FOR_GRACE_SECONDS`. When it was 5 s against a 2 s backstop, a CLI slow on SIGTERM +was cancelled by the backstop, and its only end event was `CRASHED "turn cancelled"`, not +TIMEOUT. `test_a_cli_that_ignores_sigterm_times_out_inside_the_orchestrator_backstop` pins it. stderr is drained CONCURRENTLY from the moment the CLI starts. Reading it only after exit deadlocks the pair: a child that fills the ~64 KiB stderr pipe blocks on write, stops @@ -615,9 +734,11 @@ orchestrator's mid-turn backstop calls `kill()`, and dropping the dir there woul resume across a retried turn. `_cleanup` always calls `stop()` after any `kill()`, so the tempdir is still reclaimed. -`_TERM_GRACE_SECONDS` is re-declared at the same value in both nd-JSON harnesses rather -than shared: the CLI-driver hoist that would unify their teardown constants and reducers is -a tracked follow-up. `STDOUT_LINE_LIMIT_BYTES`, which IS canonical, is imported. +Both nd-JSON harnesses run on `agents/_transport/subprocess_jsonl.py::SubprocessJsonlAgent`, +which owns this whole transport once: the spawn, the stderr drain, the read loop, the settle, +`kill` / `kill_sync` / the reap, and `KILL_GRACE_SECONDS`, `_EXIT_GRACE_SECONDS`, `_DRAIN_SECONDS`, `_SIGKILL` and +`_MAX_UNRECOGNIZED_TYPES`. A subclass keeps its argv, environment, session handling and its +decoder. ## The system_prompt_semantics marker @@ -641,9 +762,10 @@ cannot disagree with what was sent. ## Skills, per harness -`orchestration/plugin_staging.py` stages every `plugins:` entry into one canonical root, -`/plugin_root`, before `Agent.start`. Each harness then receives the SAME layout: -`.claude-plugin/plugin.json` and `skills/` links. The staging exists because each +`orchestration/plugin_staging.py` stages every `plugins:` entry into one root, +`/plugin_root`, before `Agent.start`. Each harness receives the SAME layout: +`skills/` links (read by every harness) and `plugins/` (each entry whole, +read by Claude Code). The staging exists because each adapter used to scan the authored path its own way. claude-code loaded nothing from a bare skills directory, with no error, so an activation suite scored recall 0.0 and read exactly like a skill that never triggers. @@ -657,11 +779,29 @@ like a skill that never triggers. skill that loads. Confirmed by the plugins reference ("Adds to the default: `skills`") and a CLI 2.1.273 spike on 2026-09-16; the moved reader had treated the manifest as a REPLACEMENT, which dropped the default `skills/` of any plugin that declared extras. -- **Only skills are staged.** A plugin's agents, hooks, commands and MCP servers are - dropped on every harness, claude-code included. That also removes a confound: a project - subagent beside `skills/` can no longer answer the request the skill should answer. -- **The staged manifest is `{"name": "coder-eval-plugins"}` and nothing else.** A - 2026-09-17 spike with `claude -p --plugin-dir` showed a staged root whose manifest declared +- **Claude Code loads each entry as a whole plugin.** On `main` each `plugins:` path went + to the SDK as its own plugin, so its agents, commands, hooks, MCP servers and + `${CLAUDE_PLUGIN_ROOT}` files loaded under the plugin's own name. Staging once reduced + every entry to its skills under one merged manifest name, which dropped + all of that and renamed the skills. It was restored on 2026-09-16: the author chooses the + scope by choosing the path, so a suite that must not load project agents points `path` + at the skills directory. That reverses the earlier "removes a confound" argument. + Codex, OpenCode, Pi and Antigravity still receive skills only, as on `main`. +- **What Claude Code loads from a `--plugin-dir` (CLI 2.1.274 spike, 2026-09-16).** The + manifest `name` wins over the directory name; with no manifest the directory name is the + plugin name; a symlinked plugin root loads whole; a bare skills directory loads nothing; + a manifest plugin whose skills sit only at `//SKILL.md` loads nothing; a + manifest `skills` path outside the root loads nothing. So staging links a plugin root + whole under `plugins/`, wraps a root Claude Code would load no skill from (a bare + skills directory, or that manifest layout, which then loads skills only) in + `plugins//.claude-plugin/plugin.json` plus a `skills` link, and refuses an + out-of-root manifest path and two entries with one plugin name at resolution. Plugin and + skill names become directory entries, so both must be one path segment and are compared + ignoring case: on a case-folding filesystem a second `Foo` link failed with EEXIST and the + copy fallback wrote plugin B into plugin A's source (review, 2026-09-16). The fallback now + runs only when symlinks cannot be created at all, and copies symlinks as links. +- **The wrapper manifest is `{"name": ""}` and nothing else.** A 2026-09-17 spike + with `claude -p --plugin-dir` showed a staged root whose manifest declared `"skills": ["skills"]` load no skill; a 2026-09-16 spike on CLI 2.1.273 loaded a real `["./skills"]` fine. The name-only manifest loads the `skills/` default either way. - **Refusal is at resolution.** `validate_plugins` runs in `validate_resolved_task`, so a @@ -677,7 +817,8 @@ like a skill that never triggers. Delivery, per harness: -- **Claude Code** takes the root as an SDK `{"type": "local", "path": plugin_root}` plugin. +- **Claude Code** takes one SDK `{"type": "local", "path": ...}` plugin per + `staged_plugin_dirs(plugin_root)`, never the staged root itself. - **OpenCode** appends `/skills` to `skills.paths` via `OPENCODE_CONFIG_CONTENT`, which the CLI merges as a final local-scope layer. That was chosen over writing `/.opencode/skills/` because it writes nothing into the @@ -711,6 +852,60 @@ one-way — the plugin loader and the models layer import the registry, never th `create_agent` deliberately does not import `coder_eval.plugins` itself for the same reason; callers reach a config through `parse_agent_config`, which loads them. +## Why plugin loading fails fast, and registration checks the SPI version + +A plugin whose `register` hook raises stops the load with `PluginLoadError`. Logging and +skipping it hides the cause: the run later fails with "No agent registered for type ..." +or, worse, resolves a task against a different kind than the author meant. A load-error +record for an unused plugin can come later if the need is proven. + +`AgentRegistry.register` requires `spi_version`. An assert inside the plugin's own hook +was the only check before, and core never read the number. The plugin passes the +literal version it was written for, not `SPI_VERSION` imported from core, because the +imported constant always matches. `SPI_VERSION` lives in `agents/registry.py` so the +check needs no import from `coder_eval.spi`, which imports the built-in agents. + +## Transport bases + +There is ONE transport base, `SubprocessJsonlAgent`, under Pi and OpenCode: both spawn a +CLI per turn and read nd-JSON from its stdout, so the spawn, settle, crash, timeout, +process-group sweep and reap are the same code. + +Decided 2026-09-16, after the Antigravity, Codex and Claude Code ports: there is no second +base (`HostAgent`). The three hold different lifecycles — a harness process spawned in +`start()` and held across turns (Antigravity), an app-server client held across turns with +rollout recovery (Codex), and an in-process SDK async iterator whose transport a threaded +watchdog kills (Claude Code). After the ports, what they still share is the bracket around +the turn body: open the emitter, run the body under `run_with_watchdog`, map +`WatchdogFired` and a late `timeout_hit` to `TIMEOUT`, map an SDK-raised `CancelledError` +(caller `cancelling() == 0`) to `CRASHED`, end an external cancel then re-raise, and pick +the clean status from the stop reason. That is about 30 lines per adapter (`communicate` +is 68, 85 and 96 lines), and the lines between the shared ones differ per harness: the +exception classification (Claude's `ProcessError` and max-turns short circuit, Codex's +post-stop exception), the values a clean return commits, and the kill target. A base +would have to take each of those as a hook, which moves the same lines rather than +removing them. The kernel (`TurnEmitter`, `run_with_watchdog`) already holds what is +genuinely common. + +Revisit when a fourth host-style adapter (Delegate, out of tree) is ported: four copies of +the same bracket with the same hooks is the point where a base pays for itself. + +## Why the watchdog cancels a child task + +An adapter that returns a `TIMEOUT` outcome when its OWN watchdog cancels the turn cannot +cancel the turn's own task. Measured on Python 3.13.11 (2026-09-16): a handler that catches +that cancel and returns leaves the task's `cancelling()` at 1, so an enclosing +`asyncio.timeout` later raises `CancelledError` instead of `TimeoutError`, and a cancel +that lands just after the body finished hits caller code. `uncancel()` would fix the count +but can also erase a real task-timeout cancel that arrived in the same iteration. + +`run_with_watchdog` runs the body as a CHILD task and arms the watchdog on the child. The +caller's count stays 0, an external cancel of the caller still propagates (and cancels the +child), and a late cancel lands on a finished child, where it does nothing. The watchdog +is unchanged. Verified with plain asyncio, and live with the Claude SDK (anyio) and the +Codex SDK (a threaded iterator). A `ContextVar` set inside the body is not visible to the +caller afterwards; the only one in `src/` is the logging task id, which the child inherits. + ## The threaded watchdog `asyncio.wait_for` is not enough for these harnesses: the Claude SDK wraps its subprocess diff --git a/.claude/notes/isolation.md b/.claude/notes/isolation.md index 5afbdff9..e6b5b2fa 100644 --- a/.claude/notes/isolation.md +++ b/.claude/notes/isolation.md @@ -51,10 +51,7 @@ writes `graded_by_api_routing` / `graded_by_eval_routing` and leaves the run's `api_routing` alone — writing in place contradicted the "prior wins" contract and left a self-contradictory record (a direct route named beside the run's stale - `aws_region`/`bedrock_model`). Two other parity fixes: `command_base_path` is now - persisted into `environment_info` by `_sync_sandbox_command_path_with_agent` and - restored in the evaluate-only branch (closing the PATH gap that method's docstring - already named), and `_join_litellm_actual_cost` **skips** when `prior_result` is set + `aws_region`/`bedrock_model`). One other parity fix: `_join_litellm_actual_cost` **skips** when `prior_result` is set (its join keys on a per-Orchestrator nonce the prior turns never carried, so it would clobber already-correct costs). The verdict is written back into the run's `task.json`, with the pre-grade record kept as `task.execute.json` — that in-place @@ -62,7 +59,7 @@ with **zero** new code. **`Sandbox.adopt(workspace)`** is the grade-in-place primitive: it reuses `setup`'s adoption half but skips every *materializing* step (`_setup_template`, `_generate_cli_recorders`, venv/package installs, the destructive - `$HOME` remediation), running only non-mutating derivation (mock-dir `+x`, venv + `$HOME` remediation), running only non-mutating derivation (mock-dir `+x` and criterion PATH prefix, venv *discovery*, plugin-tools pin); `_cleanup_on_exit` stays False so an adopted tree is never moved or deleted, and `Sandbox.was_adopted` is set — the Orchestrator reads it to SKIP the `pre_run` hook (`run()` calls it unconditionally with `cwd = sandbox_dir`, @@ -218,9 +215,7 @@ explicitly (`evaluate `) also bypasses it, since that config came from the operator. The workspace fallback `artifacts / prior.task_id` is containment-checked like its `sandbox_path` sibling (`task_id` is an unvalidated - string, and `"../../.."` joins to a real directory `is_dir()` confirms), - `_sanitize_restored_path` drops relative entries (they resolve against the grader's - cwd) and anything inside the run dir rather than only the workspace, and + string, and `"../../.."` joins to a real directory `is_dir()` confirms), and `write_text_atomic` opens its temp file `O_EXCL|O_NOFOLLOW` — a pre-planted `task.json.tmp` symlink otherwise bypassed the write-back's destination symlink guard entirely. The record must also describe the task as AUTHORED, not as executed: @@ -796,14 +791,16 @@ under, and would let an agent shadow binaries by writing `.venv/bin/` into its o Each layer is independent — none breaks if another is absent. 1. Inherit the parent environment, so agent tools and credentials remain reachable. -2. If the orchestrator captured the agent's SDK PATH, **prepend** it ahead of the host PATH - rather than replacing it: the agent's PATH only needs to win the lookup race for its - bundled toolchain, and system binaries must stay reachable to criteria. Prepending also - stays symmetric with the venv and node_bin prepends below. -3. Activate the sandbox virtualenv, first-hit-wins. If the agent's PATH already contains the - venv scripts dir — likely, since it inherits this process's environment — the prepend - duplicates the entry, which is harmless everywhere and left explicit so the order does not - depend on what the agent SDK injects. +2. **Prepend** the resolved mock dirs (`record_cli` recorders, then `mock_path_dirs`) ahead + of the host PATH rather than replacing it, so system binaries stay reachable to criteria. + The sandbox sets this prefix itself, at `setup` and at `adopt`, from the same + `resolved_mock_path_dirs` list the orchestrator passes to the agent as `env_path_prepend`. + It used to be read back from the agent after each successful turn, through + `get_sdk_options()`, which only Claude Code implements: on every other harness, and on + Claude Code after a crash, a timeout or in a detached grade, a criterion did not see the + mocks the agent saw. The prefix is not persisted in the run record: every entry is inside + the workspace, which a restored PATH must never trust. +3. Activate the sandbox virtualenv, first-hit-wins. 4. Prepend `/node_modules/.bin`. 5. Pin `NODE_PATH=""` so Node's fallback search cannot pick up contaminated parent-dir installs. This does NOT disable parent-walking from cwd — that is hard-wired in Node — but diff --git a/.claude/notes/orchestration.md b/.claude/notes/orchestration.md index 121099ee..809145a7 100644 --- a/.claude/notes/orchestration.md +++ b/.claude/notes/orchestration.md @@ -462,19 +462,62 @@ nothing to defer for and fail-stops on the first misfire. `EarlyStopWatcher` answered one question on the `should_stop` channel. Every harness also counted its own turn cap in its own unit, and the budgets were checked by the -orchestrator after a turn had already spent the money. `TurnMonitor` answers all four -reasons (`EARLY_CRITERION`, `TOOL_CALL_CAP`, `TOKEN_BUDGET`, `USD_BUDGET`) from ONE -collector, so a cap means the same number of resolved tool calls on every harness and a -budget stops the agent at its next poll. It is cumulative because one instance serves +orchestrator after a turn had already spent the money. `TurnMonitor` answers all five +reasons (`EARLY_CRITERION`, `TOOL_CALL_CAP`, `MODEL_TURN_CAP`, `TOKEN_BUDGET`, +`USD_BUDGET`) from ONE collector, so a cap means the same count on every harness that +accepts it and a budget stops the agent at its next poll. It is cumulative because one instance serves every retry attempt and every dialog turn of a task: the cap, the budgets and `expected_tool_calls` all measure the task, not an attempt. On one round the armed stop -wins, then the cap, then the token budgets, then USD, and the first latched reason is +wins, then the tool-call cap, then the token budgets, then USD (the model-turn cap latches +on a turn start, which carries no tool and no tokens, so it never competes), and the first latched reason is final, so the status an adapter finalizes with cannot flip after the fact. Fail-open covers only the armed criteria: a raising `live_verdict` is agent-output-dependent code, while the cap and budgets read counters and must keep running on a run that has lost its criteria. `result.tool_calls_exhausted` still comes from the turn's end status, not the latch, because a cap latched after the agent's last poll stopped nothing. +### The model-turn cap counts turn starts + +`run_limits.max_turns` and `expected_turns` came back (decisions 2026-09-16) with one unit: +main-thread model turns across the whole task. `MODEL_TURN_CAP` reuses `AgentEndStatus.TOOL_CALLS_EXHAUSTED` so that no status, +resume rule or report label forks. The counter is the monitor's existing `_sdk_turn_index`: +a main-thread `TurnStartEvent` counts once per turn id per `communicate()` (the id set +clears on each main-thread `AgentStartEvent`), cumulatively across attempts and dialog +turns, and the cap latches when turn N+1 STARTS, so a run of exactly N turns is not +capped. It counts starts, not ends: Claude Code closes main turn A when a sub-agent +message arrives and re-opens A for its next block, so an end-based rule latched with cap 1 +before A's second tool was dispatched. On Claude Code the stop lands when response N+1 +arrives: that message is recorded and its tools stay unresolved. Pi counts its own +provider-error retry as a turn (it opens a new `turn_{n}` id); Claude Code and OpenCode +retry below the stream, so their retries count zero. + +Which harnesses accept the fields is derived, not declared: +`HarnessContract.counts_model_turns` is `cooperative_stop` and a `usage_granularity` +other than `turn`. A harness that reports once per `communicate()` would count calls, a +different meaning, so it rejects both fields at resolution. Codex and Antigravity were +that harness until 2026-09-17; each now opens one inner turn per generation +(agents.md § One inner turn per generation on Codex and Antigravity), so only `none` +rejects. The count is persisted as +`EvaluationResult.model_turns` because reports are rebuilt from `task.json` and a detached +grade has an inert monitor; re-deriving it from `TurnRecord.messages` over-counts on +Claude Code (a spike stream showed 5 for 3 main responses). The dialog loop writes it +before the budget gate, so a budget abort keeps it. `main`-era records used +`expected_turns` for visible entries, so the row reports `expected_turns` only beside a +`model_turns` count, and the evalboard reads `expected_turns` as the tool-call target only +on a row with no `expected_tool_calls` key. + +### The monitor scopes to the main thread + +A sub-agent's events arrive tagged with `parent_thread_id`. The cap and the armed criteria +count and evaluate MAIN-THREAD resolved tool calls only, and a nested `TurnStartEvent` +never becomes the reported model. The cap once tripped on a Claude sub-agent's `Bash` +call before the main thread wrote its answer (decision 2026-09-16): the author capped the +agent they configured, and a sub-agent's calls are already covered by the spawning `Agent` +call. A nested `ToolEndEvent` still reaches the monitor's collector, so its command set +equals the authoritative one, and `TurnRecord.commands` keeps sub-agent calls. Budgets +still add a nested `TurnEndEvent`'s tokens: money a sub-agent spends is spent. +`expected_tool_calls` is a post-run warning over `TurnRecord.commands` and is unchanged. + ### Inert triggers are by design, and the watcher fails open A trigger whose polarity an instance can never decide is INERT, not an error — one @@ -594,37 +637,6 @@ silently drops out of the run). The interrupt is caught, the full teardown runs, re-raised at the end so callers observe the same exception. The watchdog cancels exactly once, so the awaits after the catch run normally. -## Restoring a PATH from a run directory - -A run dir is a shareable artifact — that is the whole point of detached grading — and -under `driver: docker` it is bind-mounted writable into the container the agent runs in. -The PATH recorded in its own `task.json` is PREPENDED ahead of the host PATH, so taken -verbatim it lets a run dir decide which binary `pytest` resolves to on the grader's host. - -Four filters, all about what PATH parity actually needs: - -- **Absolute only.** A relative entry resolves against the grader's current working - directory, which has nothing to do with the run, so `evilbin` becomes `$PWD/evilbin` at - the front of every criterion subprocess's PATH. It also cannot be the toolchain location - it claims to be, since the run resolved it somewhere else. -- Drop anything that is not an existing directory — a dead entry buys no parity. -- Drop any entry inside the WORKSPACE being graded: that tree is agent-writable, so a shim - dropped there would shadow a real tool. -- Drop any entry inside the RUN DIRECTORY as a whole. The workspace is only part of it; - `artifacts/`, a sibling replicate's tree and the run root all travel in the same shared - artifact and are equally attacker-chosen. - -The PATH is captured only on the per-turn happy path, after a successful turn, which -leaves three gaps: an agent crash or turn timeout (the sync never runs, and a crashed -agent's SDK PATH may itself be unreliable), evaluate-only mode, and the window before the -first turn. Persisting it is what closes the evaluate-only gap for a LATER detached grade, -which would otherwise resolve `run_command` criteria against ambient PATH and could reach -a different verdict than the run it claims to be grading. - -A sandbox-setup-time sync was considered and rejected: the agent SDK's effective PATH is -only knowable after the SDK initializes, so it would capture the configured prepends -rather than the full agent environment. - ## The dialog loop (The per-site mechanics stay as comments in `_simulation_dialog_loop`; this is only the diff --git a/.claude/notes/reporting.md b/.claude/notes/reporting.md index 823d940a..507edc49 100644 --- a/.claude/notes/reporting.md +++ b/.claude/notes/reporting.md @@ -15,7 +15,8 @@ - **Run-time caps (non-criterion enforcement)**: `TaskDefinition.run_limits` (`RunLimits` model) is the single namespace for all *task-level* run-time caps — - `max_tool_calls` / `task_timeout` / `turn_timeout` (structural) and `max_input_tokens` / + `max_tool_calls` / `max_turns` / `task_timeout` / `turn_timeout` (structural, with the + soft targets `expected_tool_calls` / `expected_turns`) and `max_input_tokens` / `max_output_tokens` / `max_total_tokens` / `max_usd` (cumulative budget). Token/USD breaches abort with `FinalStatus.TOKEN_BUDGET_EXCEEDED` or `COST_BUDGET_EXCEEDED` (both `category == "failed"`). Structural caps are set from the CLI via `-D @@ -75,12 +76,10 @@ authoring walkthrough is [docs/EXTENDING.md](../../docs/EXTENDING.md) and the fi numbered lifecycle requirements are in CLAUDE.md § Adding a New Agent; what follows is why the seams are shaped the way they are. -The turn-lifecycle bookkeeping lives on the BASE class as class-level defaults, so a -subclass gets the behaviour without re-declaring it. `_iteration_was_incremented` is set -right after the counter bump at the top of `communicate()` and consumed by -`discard_pending_turn()`, which rolls the counter back exactly once per failed turn — even -when partial-record assembly leaves `pending_turn` at None. That is why rollback is the -caller's move, not the agent's: only the caller knows a turn failed. +`communicate` returns a `TurnOutcome` and takes the `iteration` from its caller, so no +turn bookkeeping lives on the agent between attempts: only the caller knows a turn failed +and whether it retries it. A cancelled turn ends through `fail(CRASHED, "turn cancelled")` +before the cancel propagates. Why: [agents.md](agents.md) § Shared turn lifecycle. Capabilities are declared rather than probed, on the agent's `HarnessContract`. `contract.cooperative_stop` gates arming early-stop, so arming it on an agent that ignores diff --git a/.claude/notes/timing.md b/.claude/notes/timing.md index 9d58a81c..18845980 100644 --- a/.claude/notes/timing.md +++ b/.claude/notes/timing.md @@ -28,14 +28,15 @@ each turn re-anchors. That is intended — do not "fix" it by re-reading the wal which is the property being removed. One per turn, never module-level and never reused: a long run would accumulate drift -between the pair and real wall time. The turn-state constructors take it as an argument -so the lifetime is visible in the signature, and so a unit test can pass a fake straight -in. An end-to-end test driving `communicate()` cannot — the state is built inside it, out -of the caller's reach — so those replace the class through the agent module instead -(`tests/_bracket_clock.py`). Both reach the same object. - -Which harnesses use it — for their window bounds and, since **CE064**, for their turn -bracket — is stated in `docs/agents/HARNESS_PARITY.md` (the `clock basis for recorded +between the pair and real wall time. `Agent._open_emitter` is the one place an agent +constructs it: it gives a fresh `TurnClock` to the turn's `TurnEmitter` under +`TimingBasis.TURN_CLOCK`, and the host wall clock under `CLI_EPOCH_MS`. The emitter stamps +every event from that clock, the `AgentStartEvent` / `AgentEndEvent` bracket included, so +an adapter cannot put the bracket on a different basis from its window bounds. A +decoder-level test passes a `ScriptedClock` through `coder_eval.testing.replay`; a test +driving `communicate()` replaces the class through `coder_eval.agent.TurnClock`. + +Which harnesses use it — for their window bounds and their turn bracket — is stated in `docs/agents/HARNESS_PARITY.md` (the `clock basis for recorded stamps` and `turn bracket` rows), the designated SSOT for per-harness composition. Asserting it anywhere else is the drift that put a wrong OpenCode row in that table for months. @@ -109,9 +110,12 @@ never push the window start past the first item and invert the span. claude-code none — its stream carries no per-emission item start — so its window opens exactly at the mark. -It deliberately does not return `completed`. The window always ends at `now`, which the -caller passed in, so handing it back would be an argument returned unchanged — -redundancy dressed as symmetry. +It returns a `Window`: a frozen dataclass that holds the two bounds and nothing else. +`duration_ms` is a property computed from them, `completed_at - started_at` clamped at +`0.0`, so an inverted window keeps its real bounds and reads as a measured zero. A caller +cannot set a duration apart from the bounds, because there is no field to set. +`TurnEmitter.add_generation` accepts only a `Window`, and in-tree only `close_window` returns +one, so every measured `generation_duration_ms` comes from this shape. ## decompose_turn @@ -183,7 +187,7 @@ itself: when to reset a span list, when to clear a start stamp, when to advance reducer now publishes the RAW window and keeps only the genuinely harness-shaped decision, which is where its window opens. -Non-mutating for aliasing reasons rather than repeated calls. Every agent builds its +Non-mutating for aliasing reasons rather than repeated calls. `TurnEmitter` builds the terminal event as `AgentEndEvent(messages=list(...))` — that copies the LIST, not the message objects — so writing in place would reach back into the agent's own live state from the collector, which is exactly the layering "the collector is the sole capture seam" @@ -202,20 +206,19 @@ keying on the id would silently collapse every id-less message of a turn into on That equality is what lets `generation_duration_ms` stay a PUBLISHED field rather than one the collector derives from the bounds. Deriving it instead was considered and cut — it -would cost five reducers, a regeneration of every golden and a rewrite of CE059, whose -exemption keys on the kwarg being present at the call site — and the assertion is the +would cost five reducers and a regeneration of every golden — and the assertion is the sensor that makes deferring that safe. A mismatch means a reducer narrowed or widened a window without moving its bounds, which is the drift `tests/_fixtures/golden_streams/_scrub.py::assert_timing_captured`'s "bounds that span it" check catches one replay at a time. -It OVERLAPS with CE061 and is kept anyway. All five reducers build the window with -`close_window(mark=…, now=…)` and write `started_at=started, completed_at=now`, and CE061 -— now exemption-free — forces that shape statically, so the equality is largely true by -construction. What the runtime check adds is the half an import-level check cannot see: a -reducer that bypasses `close_window`, and a third-party agent registered through the -`coder_eval.plugins` SPI, which lives outside `src/coder_eval/agents/` where no lint rule -reaches it. It is not load-bearing on its own. +It OVERLAPS with `TurnEmitter` and is kept anyway. All five reducers pass +`TurnEmitter.add_generation` a `Window` from `close_window(mark=…, now=…)`, and the emitter +writes the bounds and the duration from that one object, so the equality is largely true by +construction. What the runtime check adds is the half the emitter cannot see: a +third-party agent registered through the `coder_eval.plugins` SPI that builds an +`AssistantMessage` itself, outside `src/coder_eval/agents/` where CE072 does not reach. It +is not load-bearing on its own. Raising kills the turn, and that is accepted — the same trade `_require_same_awareness` makes at this seam. The condition is unreachable without a reducer bug; all five are @@ -223,8 +226,8 @@ exercised by the golden corpus and by the ms-exact identity contract. ### Why the zero-total skip runs before the equality check -`close_window` clamps an inverted window — `now` before `mark`, two clocks disagreeing — -to `0.0` while the bounds it writes still say `completed_at < started_at`, so the bounds +`Window.duration_ms` clamps an inverted window — `now` before `mark`, two clocks disagreeing — +to `0.0` while its bounds still say `completed_at < started_at`, so the bounds span is NEGATIVE and the equality fails. That is a measured inversion, the case `decompose_turn` deliberately clamps because both ends were observed; raising on it would kill turns on exactly the shape the clamp exists to tolerate. The cost is that a `0.0` @@ -247,13 +250,13 @@ carries no timestamps at all, so indexing the raw list would measure the wrong t raise. A message whose `generation_duration_ms` is `None` is skipped. That field is the -codebase's own marker for "no window was measurable here", and every producer of one -stamps `started_at == completed_at == datetime.now()` at *append* time as an admitted -placeholder — Codex's rollout rebuild (`_messages_from_items`), both Codex sub-agent -recovery builders, and Claude's `_synthesize_subagent_terminal_message`. Reading those +codebase's own marker for "no window was measurable here", and its only in-tree writer is +`TurnEmitter.add_unmeasured_generation`, which stamps `started_at == completed_at == +clock.now()` at *append* time as an admitted placeholder. Its callers are Codex's rollout rebuild (`_messages_from_items`), both Codex sub-agent +recovery builders, and Claude's synthesized sub-agent terminal (`_subagent_terminal_part`). Reading those stamps as window bounds turns a placeholder into a measurement: a Codex turn rebuilt from its rollout stamps every message at turn END, which would book the entire turn as harness -startup. It is the same exemption CE059 makes for the same reason. +startup. `min` / `max` rather than the first and last list entries, because the list is not ordered by time — Codex appends recovered sub-agent messages after the parent's last flush. diff --git a/.claude/shared/review-rubric.md b/.claude/shared/review-rubric.md index e6e0ba00..68305c70 100644 --- a/.claude/shared/review-rubric.md +++ b/.claude/shared/review-rubric.md @@ -52,7 +52,7 @@ The coder_eval-specific quality checklist. Check every item: 10. **Layer-merge coverage**: new fields on `ResolvedTask` / `AgentConfig` / `BatchRunConfig` have explicit coverage in `test_experiment_resolver.py` exercising all 5 merge layers (default → exp defaults → task → variant → CLI), and a matching `-D` override path. New list/dict fields declare a `MergeField` strategy (CE014). 11. **Pydantic round-trip integrity**: changes to layered configs or polymorphic `CriterionResult` subclasses preserve `model_fields_set` and the discriminator across `model_dump(exclude_unset=True)` → `model_validate()`. Round-trip tests exist for new variants. 12. **Discriminated unions**: new or modified Pydantic unions use `Annotated[..., Field(discriminator="type")]`. Bare `A | B | C` unions silently coerce to the first variant on a missing or typo'd `type`. -13. **Cross-retry state hygiene**: after `AgentCrashError` / `TurnTimeoutError` / `is_error=True` SDK message, the agent resets `_session_id`, `pending_turn`, watchdog references, streaming-event `ContextVar`s, and iteration counters before the next attempt. Test covers a crashing turn followed by a successful turn in the same `Orchestrator` instance. +13. **Cross-retry state hygiene**: a failed turn returns an outcome with `record.crashed=True`; cancellation ends the turn with `fail(CRASHED, ...)` before it propagates; no cross-attempt state lives on the agent (after a crash or an `is_error=True` SDK message it resets `_session_id`, watchdog references and streaming-event `ContextVar`s). Test covers a crashing turn followed by a successful turn in the same `Orchestrator` instance. 14. **Untrusted text in evaluator prompts**: strings derived from agent output (tool-call args, stdout, file contents, dialog history) injected into a judge / simulator / reviewer prompt are wrapped in a fenced block with explicit untrusted-data framing; the system prompt instructs the model to treat that block as adversarial. 15. **NaN / non-finite guards**: score and threshold clamps via `max(lo, min(hi, x))` are preceded by `math.isfinite(x)`. Bad parses fail explicitly instead of silently returning the upper bound (`max(0.0, min(1.0, nan)) == 1.0`). 16. **Registry over hardcoded dispatch**: new agent / criterion / template / route variants go through the existing registry (`@register_criterion`, the `coder_eval.plugins` SPI, etc.). `if x.type == ...` / `isinstance(...)` ladders in `orchestrator.py`, `simulation/`, or `evaluation/` are rejected. diff --git a/.claude/shared/run-layout.md b/.claude/shared/run-layout.md index c006e935..6b064c4f 100644 --- a/.claude/shared/run-layout.md +++ b/.claude/shared/run-layout.md @@ -16,7 +16,7 @@ runs/////{task.json, task.log, artifacts/} - `task.json.graded` — present only after `coder-eval execute --driver docker` refused a container's verdict: the runtime image predated `execute` and graded anyway, so the runner quarantines the graded record here rather than leaving it readable as `task.json`, where a later `--resume` / `aggregate` would fold in exactly the row it declined to publish. Diagnostic-only; `rglob("task.json")` consumers do not match it. - `grade.log` — present only after a DETACHED grade over this directory (`coder-eval run --resume`). The grading pass's own log. It is a separate file because the log handler truncates whatever file it opens, so writing to `task.log` would destroy the agent trajectory log the run already paid for. - `task.log` — the human-readable task log; `artifacts/` — files the agent produced. -- `plugin_root/` — the staged plugin root the agent was handed; symlinks into the authored plugin; present only when the task sets `agent.plugins`. +- `plugin_root/` — the staged plugin root the agent was handed: `skills/` links into the authored skills and `plugins/` (each authored plugin whole); present only when the task sets `agent.plugins`. **Scope-marker files** (used to detect what a given path represents): diff --git a/.github/workflows/pr-checks.yml b/.github/workflows/pr-checks.yml index 935c3e4d..11a279ae 100644 --- a/.github/workflows/pr-checks.yml +++ b/.github/workflows/pr-checks.yml @@ -101,11 +101,11 @@ jobs: - name: Type check with pyright run: .venv/bin/pyright - # The CE036 contract engine lives under tests/, which [tool.pyright] excludes + # The CE036 contract engine and the live tests live under tests/, which [tool.pyright] excludes # -- and `exclude` beats both a CLI file arg and an `include` entry, so it can # only be reached through a config of its own, derived from [tool.pyright] so # the two passes cannot drift. Mirrors `make typecheck`. - - name: Type check the CE036 contract engine + - name: Type check the CE036 contract engine and the live tests run: | .venv/bin/python -m tests.lint.pyright_config .pyright-tests.json .venv/bin/pyright -p .pyright-tests.json @@ -405,7 +405,7 @@ jobs: - name: Type check with pyright run: .venv/Scripts/pyright - - name: Type check the CE036 contract engine + - name: Type check the CE036 contract engine and the live tests run: | .venv/Scripts/python -m tests.lint.pyright_config .pyright-tests.json .venv/Scripts/pyright -p .pyright-tests.json diff --git a/CLAUDE.md b/CLAUDE.md index aebcd9e0..e43ef010 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -33,7 +33,8 @@ data-driven analysis. import from `coder_eval.models`, never from its submodules. - **`criteria/`** auto-discovers one checker per type via `pkgutil`. - **`cli/`** holds Typer commands; each has a plain-Python twin (CE048). -- **`timing.py`** owns the single subtraction seam (CE063). +- **`timing.py`** owns the single subtraction seam and `Window`, the bounds of one + generation window. - **`argv_match.py`** is a STDLIB-ONLY sidecar copied beside the recorder (CE057). - **`fs_permissions.py`** is `set_permissions`, the stacked chmod window. - **`path_utils.py`** owns run ids, atomic writes and tree digests — and every run-record @@ -50,7 +51,11 @@ data-driven analysis. - **`durations.py`** is `format_ms`, split from `formatting.py` so the reports layer does not reach through an SDK-shaped module for it. - **`isolation/`** is `driver: docker`, one container per task. -- **`streaming/`** is the event protocol and `EventCollector`. +- **`streaming/`** is the event protocol and `EventCollector`; **`streaming/emitter.py`** + is `TurnEmitter`, the per-turn kernel every agent writes its turn through. +- **`testing.py`** is `coder_eval.testing`, the adapter test sensors (`replay`, + `assert_identity_closes`, `assert_stream_balanced`, `conformance`, `stop_conformance`). In-tree suites and + plugins use the same module. Outside the package: `tasks/`, `experiments/`, `templates/`, `tests/`, `docs/`, `evalboard/`, `plugins/coder-eval/` (the published plugin), `action.yml` (the published @@ -65,8 +70,8 @@ Each entry is a pointer. Full rationale: `.claude/notes/` (index: `.claude/notes - **Strategy pattern**: `Agent` ABC, implementations in `agents/`. - **Separation of concerns**: `models/` is pure Pydantic; logic lives in `criteria/`, `evaluation/`, `orchestration/`. -- **Callback streaming**: the agent is the sole emitter of the event protocol; - `EventCollector` reduces the stream into a `TurnRecord`. Never hand-assemble one. +- **Callback streaming**: `TurnEmitter` is the sole writer of the event protocol (CE072); + its `EventCollector` reduces the stream into a `TurnRecord`. Never hand-assemble one. - **All core models import from `coder_eval.models`** — never from submodules. - **Single declarative merge resolver**: all five config layers (default → experiment defaults → task → variant → CLI) merge through `orchestration/config_merge.py`. @@ -80,18 +85,19 @@ Each entry is a pointer. Full rationale: `.claude/notes/` (index: `.claude/notes `aggregate()`; classification criteria layer accuracy / P/R/F1. - **Reconciliation message**: summing token buckets across `TurnRecord.messages` equals `token_usage` exactly, on every backend. `EventCollector` is the single writer. -- **Timing has one subtraction seam** (`timing.py`); agents must not do their own - (CE063). An unmeasured duration is `None`, never `0.0` (CE058). +- **Timing has one subtraction seam** (`timing.py`); agents must not do their own. + `TurnEmitter` stamps the turn from one clock. An unmeasured duration is `None`, never `0.0` (CE058). - **Reference solutions are directory-only** and chmod-shielded during `communicate`. Defense-in-depth, not a boundary — the known gaps are documented in the notes. Authoring reference: [Reference Solutions](docs/TASK_DEFINITION_GUIDE.md#reference-solutions). - **Harness run-limit parity**: every structural cap and budget is one `TurnMonitor` - answer on the `should_stop` channel, in tool calls or tokens, on every harness; + answer on the `should_stop` channel, in tool calls, model turns or tokens, on every harness; `run_limits` and agent-field meanings are both generated tables in [Run-Limit Parity](docs/agents/HARNESS_PARITY.md). Every agent declares a `HarnessContract`; a base field the harness marks unsupported is rejected at resolution. Caps are authored under [Run Limits](docs/TASK_DEFINITION_GUIDE.md#run-limits). -- **Plugin staging**: `stage_plugins` hands every harness one canonical plugin root; +- **Plugin staging**: `stage_plugins` hands every harness one staged plugin root (a skills + index, plus each plugin whole for Claude Code); `skills_offered` is the positive control `skill_triggered` checks. - **Execute vs. run**: `execute` is `run` with grading off — rows finalize as `NOT_GRADED` and leave both sides of every rate. Per-command behaviour: @@ -136,9 +142,9 @@ CLI → ExperimentRunner (task × variant, 5-layer merge) → run_batch → Orch Per-task (single iteration; simulation mode runs a multi-turn dialog): 1. Orchestrator._communicate_with_retry(prompt, iteration) → TurnRecord - (wraps agent.communicate with retry, per-attempt turn_timeout, the task's - TurnMonitor as the should_stop poll, and on_attempt_error → preserves - crashed=True partial TurnRecords) + (wraps agent.communicate(..., iteration=) → TurnOutcome with retry, per-attempt + turn_timeout, the task's TurnMonitor as the should_stop poll; a CRASHED/TIMEOUT + outcome's crashed=True record is appended before it is raised) 2. SuccessChecker.check_all_async() → List[CriterionResult] Cleanup: stop agent, save EvaluationResult, generate reports. @@ -228,6 +234,13 @@ A few rules constrain routine edits, so they are worth knowing before you start: - **CE070** keeps agent adapters from counting caps (`max_tool_calls`, `RunLimits`, `tool_calls_exhausted`, …) or scanning for `SKILL.md`: the `TurnMonitor` owns caps and `orchestration/plugin_staging.py` owns skill discovery. +- **CE071** keeps `calculate_cost` out of `agents/` and `orchestration/turn_monitor.py`. + Call `pricing.price_turn`: one rule for the cost of a turn. +- **CE072** keeps agent adapters from constructing the events (`AgentStartEvent`, + `ToolEndEvent`, …), an `AssistantMessage` (an import alias too) or an `EventCollector`. + Call `TurnEmitter` instead. +- **CE073** requires every asyncio subprocess spawn in `src/` to pass `stdin=`. An + inherited stdin stopped a CLI turn with no events until its timeout. **Docs index SSOT.** `nav:` plus `extra.docs_index` in `mkdocs.yml` are the single source of truth for `README.md`'s Documentation table, `docs/index.md`'s "Where to go @@ -266,7 +279,10 @@ A live criterion also needs `ContractCase`s (CE036) and `make plugin-reference`. **A new agent**: agents register through the plugin SPI (entry-point group `coder_eval.plugins`) — there is no closed enum or dispatch to edit, and in-tree and third-party agents take the same path. It declares a `HarnessContract` (registration -fails without one) and imports from `coder_eval.spi`. A new agent must be named on every +fails without one) and imports from `coder_eval.spi`. It writes each turn through one +`TurnEmitter` from `Agent._open_emitter` and returns its `TurnOutcome`; a JSONL CLI +subclasses `SubprocessJsonlAgent`. Test it with the `coder_eval.testing` sensors. A new +agent must be named on every onboarding surface CE047 tracks, and its run-limit behaviour recorded in [Run-Limit Parity](docs/agents/HARNESS_PARITY.md). diff --git a/Makefile b/Makefile index 514890d6..50707e8b 100644 --- a/Makefile +++ b/Makefile @@ -47,13 +47,13 @@ docs-budget: ## Report the docstring/comment prose budget and check it against typecheck: ## Run type checking with pyright uv run pyright - # The CE036 contract engine executes checker code and feeds the early-stop - # design; it is the one tests/ surface worth type-checking. It needs its own - # config: pyproject.toml excludes "tests", and pyright's `exclude` beats BOTH + # The second pass type-checks the tests/ surfaces that break silently: the CE036 + # contract engine and the live tests (an SPI change fails here, without + # credentials). It needs its own config: pyproject.toml excludes "tests", and pyright's `exclude` beats BOTH # an explicitly-passed CLI file arg AND an `include` entry naming the file -- # either shortcut analyzes ZERO files and exits 0, a gate that checks nothing. - # The config below is DERIVED from [tool.pyright] (same rules, only - # include/exclude swapped), so the two passes cannot drift apart. + # The config below is DERIVED from [tool.pyright] (same rules; only + # include/exclude and extraPaths differ), so the two passes cannot drift apart. uv run python -m tests.lint.pyright_config .pyright-tests.json uv run pyright -p .pyright-tests.json @@ -72,6 +72,8 @@ verify: ## Run all verification steps (CI equivalent) uv run ruff format --check $(LINT_PATHS) uv run ruff check $(LINT_PATHS) uv run pyright + uv run python -m tests.lint.pyright_config .pyright-tests.json + uv run pyright -p .pyright-tests.json uv run pytest tests/test_custom_lint.py -v --tb=short --no-header -p no:warnings uv run python -m tests.lint.prose_budget # uv run pip-audit --desc --skip-editable diff --git a/README.md b/README.md index fc350c39..05fd5e60 100644 --- a/README.md +++ b/README.md @@ -19,8 +19,9 @@ agents and their skills** — built for benchmark authors, CLI builders, and ski builders — with sandboxing, reproducibility, and data-driven analysis. It runs a real agent — **Claude Code**, **OpenAI Codex**, **Google Antigravity (Gemini)**, **OpenCode**, or **Pi** — in a sandbox against declarative YAML tasks, then scores the files and -commands it actually produced. Changing harness is one field (`agent.type`); the -tasks, criteria, scoring, telemetry, and reports stay the same. +commands it actually produced. Changing harness is one field (`agent.type`); a field +the target harness cannot honor fails at load, and every harness records its turn +through one kernel, so the telemetry means the same thing on every harness. Reach for it when you want to **benchmark agents on your own domain tasks**, **test whether a skill triggers** in the agent you ship for, **A/B-test Claude Code diff --git a/docs/DIALOG_MODE.md b/docs/DIALOG_MODE.md index 98c6d4c3..b10714dc 100644 --- a/docs/DIALOG_MODE.md +++ b/docs/DIALOG_MODE.md @@ -123,10 +123,10 @@ After each exchange the driver evaluates the stop conditions **in this order**, per-turn checking (`check_criteria: every_turn` or `both`); pairing it with the default `end_of_dialog` is rejected at load time, since there would be nothing to check against. 3. **`max_turns`** (`max_turns`) — the hard cap on exchanges. The agent reaching - [`run_limits.max_tool_calls`](TASK_DEFINITION_GUIDE.md#run-limits) mid-exchange ends the dialog - with its own reason, `tool_call_cap`, which wins over `max_turns`, the budget and the stop token + [`run_limits.max_tool_calls`](TASK_DEFINITION_GUIDE.md#run-limits) or `run_limits.max_turns` + mid-exchange ends the dialog with its own reason, `tool_call_cap`, which wins over `max_turns`, the budget and the stop token on that turn (only a criteria pass outranks it). That cap is cumulative across every dialog turn: - it counts the agent's resolved tool calls over the whole dialog, not per exchange. + it counts the agent's resolved tool calls (or model turns) over the whole dialog, not per exchange. 4. **`max_total_tokens`** (`budget`) — the dialog-wide budget across simulator **and** agent. The dialog ends and the task is **still scored** — unlike [`run_limits.max_total_tokens`](TASK_DEFINITION_GUIDE.md#run-limits), which covers the subject diff --git a/docs/EXTENDING.md b/docs/EXTENDING.md index 995f6d85..8f0fd412 100644 --- a/docs/EXTENDING.md +++ b/docs/EXTENDING.md @@ -35,31 +35,32 @@ my_plugin = "my_plugin:register" ``` At CLI init, `load_plugins()` imports each entry point and calls it with the -`AgentRegistry` **class** (not an instance). A third-party hook that raises is logged -and skipped; only a failing *built-in* registration is fatal. +`AgentRegistry` **class** (not an instance). A hook that raises stops the load with a +`PluginLoadError` that names the entry point, so every command fails until the plugin is +fixed or uninstalled. A broken plugin is never skipped. ### The `register` hook -Import everything from `coder_eval.spi`, the stable plugin surface, and check its -version in the hook. `SPI_VERSION` changes whenever an exported name changes its -signature. +Import everything from `coder_eval.spi`, the stable plugin surface. `SPI_VERSION` +changes whenever an exported name changes its signature. Every `register` call must +pass the SPI version the agent was written against, as the literal number: registration +raises `TypeError` when it is not the version this coder_eval provides. ```python -from coder_eval.spi import SPI_VERSION, AgentRegistry +from coder_eval.spi import AgentRegistry def register(registry: type[AgentRegistry]) -> None: - assert SPI_VERSION == 2, f"my-agent supports coder_eval SPI 2, not {SPI_VERSION}" - # Bind type string → config class → agent class. - registry.register("my-agent", MyAgentConfig)(MyAgent) + # Bind type string → config class → agent class, for SPI 1. + registry.register("my-agent", MyAgentConfig, spi_version=1)(MyAgent) # Optionally contribute pricing here too (see §3): # register_pricing(MY_RATES) ``` -`AgentRegistry.register(agent_kind, config_class)` returns a decorator, so the -decorator form works too: +`AgentRegistry.register(agent_kind, config_class, *, spi_version)` returns a decorator, +so the decorator form works too: ```python -@AgentRegistry.register("my-agent", MyAgentConfig) +@AgentRegistry.register("my-agent", MyAgentConfig, spi_version=1) class MyAgent(Agent[MyAgentConfig]): ... ``` @@ -99,7 +100,15 @@ at resolution, so `coder-eval plan` fails before any run. This is a JSONL CLI ag that appends a system prompt and honors `plan` and tool lists natively: ```python -from coder_eval.spi import Agent, Enforcement, HarnessContract, PermissionMode, ToolNameMap, UsageGranularity +from coder_eval.spi import ( + Agent, + Enforcement, + HarnessContract, + PermissionMode, + TimingBasis, + ToolNameMap, + UsageGranularity, +) # native tool name -> canonical (Claude) name; also used for telemetry _TOOL_NAME_MAP = {"bash": "Bash", "read": "Read", "write": "Write", "edit": "Edit", "task": "Agent"} @@ -115,6 +124,7 @@ class MyAgent(Agent[MyAgentConfig]): disallowed_tools=Enforcement.ENFORCED, cooperative_stop=True, usage_granularity=UsageGranularity.STEP, + timing_basis=TimingBasis.TURN_CLOCK, ) tool_names = ToolNameMap.from_inverse( _TOOL_NAME_MAP, @@ -129,6 +139,9 @@ class MyAgent(Agent[MyAgentConfig]): meaning (`plan` is read-only, `bypassPermissions` runs every permitted tool). - `tool_names` is required exactly when a tool-list row is `ENFORCED`. It must map every canonical name; list a name your harness has no tool for in `no_equivalent`. +- `timing_basis` says who stamps the turn. `TURN_CLOCK`: the `TurnEmitter` stamps every + tool and the turn bracket from one clock. `CLI_EPOCH_MS`: your harness reports its own + stamps, and you pass them for every main-thread tool and window. - Set `cooperative_stop=True` only if your `communicate()` honors `should_stop` (needed for criterion-level `stop_early:` arming and for `run_limits.max_tool_calls` to cut a turn). `False` means early stop is rejected at resolution for your agent. @@ -142,11 +155,12 @@ it on every LiteLLM route. Implement these three abstract methods: - [ ] `async def start(self, working_directory, *, env_path_prepend=None, plugin_tools_dir=None, plugin_root: Path | None = None) -> None` -- [ ] `async def communicate(self, user_input, *, stream_callback=None, timeout=None, should_stop: Callable[[], StopReason | None] | None = None) -> TurnRecord` +- [ ] `async def communicate(self, user_input, *, iteration: int, stream_callback=None, timeout=None, should_stop: Callable[[], StopReason | None] | None = None) -> TurnOutcome` - [ ] `async def stop(self) -> None` -`plugin_root` is the staged plugin root (`/skills//SKILL.md`), or `None` when -the task sets no plugins. Deliver it the harness's native way; do not scan for skills. +`plugin_root` is the staged plugin root, or `None` when the task sets no plugins. It holds +`/skills//SKILL.md` (every harness) and `/plugins/` (each authored +plugin whole, for a harness that loads full plugins). Deliver it the harness's native way; do not scan for skills. `should_stop` is the run's single stop poll. The `TurnMonitor` owns it: it reads your event stream and decides every stop (armed criteria, the tool-call cap, the token and USD @@ -158,28 +172,186 @@ often you report them as `usage_granularity`. With `cooperative_stop=True`: - [ ] Call `should_stop()` at each safe boundary (for example, after each resolved tool call, before you pull the next unit of work). - [ ] When it returns a `StopReason`, stop pulling work and remember the reason. -- [ ] Finalize the turn with `AgentEndStatus` `end_status_for(reason)` (both names - come from `coder_eval.spi`), with `crashed=False`. Do not raise. +- [ ] End the turn with `emitter.finalize(end_status_for(reason))` (both names come + from `coder_eval.spi`). Do not raise. Optional overrides (sensible defaults exist): `kill()`, `kill_sync()` (called from a -non-asyncio watchdog thread — must **not** await), `discard_pending_turn()`. +non-asyncio watchdog thread — must **not** await). + +Write the turn through one `TurnEmitter` (do **not** build events, messages or a +`TurnRecord` yourself): + +- [ ] Open it with `emitter = self._open_emitter(prompt=user_input, iteration=iteration, + model=..., task_id=..., stream_callback=stream_callback)` and call `emitter.begin()`. +- [ ] Report what the harness did: `begin_inner_turn` / `end_inner_turn(tokens=delta)`, + `text`, `open_tool` / `close_tool`, and `add_generation(message_id=..., window=close_window(...), parts=[Generation(...)])`. +- [ ] Return `emitter.finalize(status, ...)` for a clean end, or + `emitter.fail(AgentEndStatus.CRASHED | TIMEOUT, reason)` for a failed one. A crash or + timeout is an outcome, not an exception; an exception out of `communicate` is a bug. +- [ ] On an `asyncio.CancelledError` from outside (`asyncio.current_task().cancelling()` is + not 0), call `emitter.fail(AgentEndStatus.CRASHED, "turn cancelled")`, then re-raise: the + orchestrator recovers the record from its own collector. A `CancelledError` your SDK + raised inside the turn (`cancelling()` is 0) is a failure of the turn: return + `emitter.fail(AgentEndStatus.CRASHED, reason)` and do not re-raise, or the task row is lost. +- [ ] Run an SDK turn body under `run_with_watchdog(...)`, and return + `emitter.fail(AgentEndStatus.TIMEOUT, format_timeout_reason(timeout))` on `WatchdogFired`. +- [ ] Call `self._mark_stopped()` in `stop()` after your own teardown. -Follow the shared turn lifecycle (do **not** hand-assemble a `TurnRecord`): +The emitter owns the event protocol: one `AgentStartEvent`, one `AgentEndEvent` on every +exit, balanced inner turns and tool calls (orphans closed `unresolved`), and the record. +That is why `coder_eval.spi` exports no event class and no `EventCollector`. What a turn +needs from it: -- [ ] Call `self._begin_turn()` at the top of `communicate()`. -- [ ] Call `self._end_turn_ok()` on the success path. -- [ ] Call `self._mark_stopped()` in `stop()` after your own teardown. -- [ ] Before raising on a mid-turn failure, set `self.pending_turn` to a - `crashed=True` `TurnRecord` (built from an `EventCollector`), then raise - `AgentCrashError` / `TurnTimeoutError` (bare — no payload). The orchestrator - drains it and calls `discard_pending_turn()`. - -Emit the standardized event protocol (you are the **sole emitter**): one -`AgentStartEvent` at the top of `communicate()` and one matching `AgentEndEvent` on -**every** exit path (emit from `finally`), a `TurnStart`/`TurnEnd` pair per inner -turn, and `ToolStart`/`ToolEnd` per tool call (close orphaned tools with -`status=unresolved`). Fan events through an internal `EventCollector` — it builds the -returned `TurnRecord`, the single agent-agnostic capture path. +```python +from coder_eval.spi import ( + AgentEndStatus, # the status you pass to finalize / fail + Generation, # one part of a model generation: blocks + its own token delta + JsonlDecoder, # the per-turn reducer of a SubprocessJsonlAgent + StopReason, + SubprocessJsonlAgent, # the base for a CLI that streams nd-JSON on stdout + TokenUsage, + ToolEndStatus, + TurnClock, + TurnEmitter, + TurnEndStatus, + TurnOutcome, + WatchdogFired, + Window, # the bounds of one generation window; close_window returns it + close_window, + end_status_for, + run_with_watchdog, +) +``` + +### A JSONL CLI agent: `SubprocessJsonlAgent` + +If your harness is a CLI that runs one process per turn and prints nd-JSON events on +stdout, subclass `SubprocessJsonlAgent`. The base owns the transport: the spawn (with +`stdin` on `/dev/null`), the stderr drain, the read loop against the turn deadline, the +cooperative stop, the crash and timeout outcomes, and the reap. You supply the argv, the +environment, and a `JsonlDecoder` that turns one event into emitter calls: + +```python +import os +from typing import Any + +from coder_eval.spi import ( + AgentEndStatus, + Generation, + JsonlDecoder, + SubprocessJsonlAgent, + TimingBasis, + TokenUsage, + ToolEndStatus, + TurnEmitter, + TurnOutcome, + close_window, +) + + +class MyDecoder(JsonlDecoder): + def __init__(self, emitter: TurnEmitter) -> None: + super().__init__(emitter) + self.mark = emitter.now() # where the next generation window opens + + def __call__(self, event: dict[str, Any]) -> None: + kind = event.get("type") + if kind == "message": + now = self.emitter.now() + tokens = TokenUsage(output_tokens=int(event.get("output_tokens", 0))) + self.emitter.add_generation( + message_id=event.get("id"), + window=close_window(mark=self.mark, now=now), + parts=[Generation(blocks=[], tokens=tokens)], + ) + self.mark = now + elif kind == "text": + self.emitter.text(str(event.get("text", ""))) + elif kind == "tool_start": + self.emitter.open_tool(str(event["id"]), str(event["name"]), event.get("args") or {}) + elif kind == "tool_end": + status = ToolEndStatus.ERROR if event.get("is_error") else ToolEndStatus.OK + self.emitter.close_tool(str(event["id"]), status=status, summary=event.get("output")) + elif kind == "error": + self.error = str(event.get("message")) # the base crashes the turn on it + + def end(self, status: AgentEndStatus, *, reason: str | None = None) -> TurnOutcome: + if status is AgentEndStatus.CRASHED or status is AgentEndStatus.TIMEOUT: + return self.emitter.fail(status, reason or status.value) + return self.emitter.finalize(status) + + +class MyAgent(SubprocessJsonlAgent[MyAgentConfig]): + contract = HarnessContract(..., timing_basis=TimingBasis.TURN_CLOCK) # the emitter stamps the tools + cli_name = "MyCli" + docs_page = "docs/agents/MY_CLI.md" + recognized_events = frozenset({"message", "text", "tool_start", "tool_end", "error"}) + decoder = MyDecoder + + def argv(self, prompt: str) -> list[str]: + return ["my-cli", "--json", "--model", self.config.model or "default", prompt] + + def env(self) -> dict[str, str]: + return dict(os.environ) + + async def start(self, working_directory: str, **_: Any) -> None: + self.working_directory = working_directory + + async def stop(self) -> None: + await self.kill() + self._mark_stopped() +``` + +A clean exit that produced no event named in `recognized_events` crashes the turn as +format drift. The in-tree example is `src/coder_eval/agents/pi_agent.py`. + +### The sixth-harness checklist + +- [ ] A `HarnessContract` (every field, `timing_basis` included). +- [ ] A config class and its registration. +- [ ] A translation from config to the harness's native call that delivers the staged + `plugin_root`. +- [ ] A decoder: one object per turn that takes the harness's events and calls the emitter. +- [ ] `async def harness_version(self)`: the CLI or SDK version the agent drives, recorded as + `environment_info.harness_version`. A `SubprocessJsonlAgent` gets ` --version` + from its `executable` class attribute. +- [ ] The `coder_eval.testing` sensors in your own tests: `replay` your decoder over a + recorded stream, `assert_identity_closes` on the replay, `assert_stream_balanced` on + its events, `conformance(kind, probes)` for the contract, and + `stop_conformance(kind, probe)` when the contract declares `cooperative_stop`. + +### Test your adapter: `coder_eval.testing` + +The in-tree suites and a plugin's tests call the same module. It does not import +`pytest`: each check raises `AssertionError`. + +| Sensor | What it checks | +|---|---| +| `replay(stream, make_decoder, clock=ScriptedClock(origin), end=...)` | Drives your decoder over a recorded stream through a real `TurnEmitter`. A `Tick(at_ms)` element moves the clock. Returns the record, the events and the bracket stamps. | +| `assert_identity_closes(record, started_at=..., ended_at=...)` | Head + generation + tool union + tail equals the turn's span. | +| `assert_stream_balanced(events)` | Every opened inner turn and tool call closes, and one turn has one start and one end. | +| `await conformance(kind, probes)` | Your agent rejects every field its contract marks unsupported, and `probes` has one check for each enforced cell. | +| `await stop_conformance(kind, probe)` | For every `StopReason`, your agent ends the turn with that reason's status at the first boundary. `probe(stop, reason)` runs one `communicate()` over a scripted harness that calls `FIRST_TOOL_ID` then `SECOND_TOOL_ID`, passes `stop` (a `StopAfterFirstTool`) as both `stream_callback` and `should_stop`, and returns the tool ids your agent pulled. | + +```python +from datetime import datetime + +from coder_eval.spi import AgentEndStatus +from coder_eval.testing import ScriptedClock, Tick, assert_identity_closes, assert_stream_balanced, replay + + +def test_a_recorded_turn_balances_and_closes(): + stream = [ + Tick(10), {"type": "tool_start", "id": "t1", "name": "Bash"}, + Tick(50), {"type": "tool_end", "id": "t1"}, + Tick(80), {"type": "message", "id": "m1", "output_tokens": 12}, + Tick(90), + ] + result = replay(stream, MyDecoder, clock=ScriptedClock(datetime(2026, 1, 1)), + end=lambda d: d.end(AgentEndStatus.COMPLETED)) + assert_stream_balanced(result.events) + assert_identity_closes(result.record, started_at=result.started_at, ended_at=result.ended_at) +``` ### Worked example @@ -327,7 +499,7 @@ MY_RATES = { } def register(registry): - registry.register("my-agent", MyAgentConfig)(MyAgent) + registry.register("my-agent", MyAgentConfig, spi_version=1)(MyAgent) register_pricing(MY_RATES) ``` diff --git a/docs/REPORT_SCHEMA.md b/docs/REPORT_SCHEMA.md index 57aedd04..0f04421e 100644 --- a/docs/REPORT_SCHEMA.md +++ b/docs/REPORT_SCHEMA.md @@ -84,26 +84,31 @@ including: `task_id`, `replicate_index`, `variant_id`, `status` `expected_commands`, `actual_commands`, `commands_efficiency`, `agent_config`, `sdk_options`, `installed_tools`, turn accounting (`total_turns`, `visible_turns`, `expected_tool_calls`, -`expected_tool_calls_overage`, `tool_calls_exhausted`, `has_final_reply`), and early-stop fields (`stopped_early`, +`expected_tool_calls_overage`, `tool_calls_exhausted`, `has_final_reply`, `model_turns`, +`expected_turns`, `expected_turns_overage`), and early-stop fields (`stopped_early`, `early_stop_reason`, `tool_calls_remaining_at_stop`). `iterations` here is a **reduced** turn digest (`{iteration, duration_seconds, command_count, assistant_turn_count, crashed, crash_reason}`) — the full transcript is in `task.json`. -> **Historical spellings.** Runs written before the tool-call rename carry -> `max_turns_exhausted`, `expected_turns`, `expected_turns_overage` and the status -> `MAX_TURNS_EXHAUSTED` instead of `tool_calls_exhausted`, `expected_tool_calls`, -> `expected_tool_calls_overage` and `TOOL_CALLS_EXHAUSTED`. The evalboard reads both. -> The Python side does not: there is no alias. A `task.json` with the old flag loads with -> the fact `false`; one whose `final_status` is `MAX_TURNS_EXHAUSTED`, or whose recorded -> config sets `run_limits.expected_turns`, does not load. `run --resume` then runs that -> row again, and `evaluate ` cannot re-grade it from its recorded config. +> **Historical spellings.** Runs from older releases carry `run_limits.max_turns` +> (Claude Code SDK turns per `communicate()` call) and `run_limits.expected_turns` plus a +> row `expected_turns` / `expected_turns_overage` that count visible entries (tool calls +> and the final reply). Current runs use those names for model turns, and use +> `expected_tool_calls` / `expected_tool_calls_overage` for visible entries. An older row +> has no `expected_tool_calls` key and no `model_turns`, which is how a reader tells the +> two apart; the evalboard reads `expected_turns` as the tool-call target only on such a +> row. Older runs also carry `max_turns_exhausted` and the status `MAX_TURNS_EXHAUSTED` +> instead of `tool_calls_exhausted` and `TOOL_CALLS_EXHAUSTED`. The Python side has no +> alias: a `task.json` with the old flag loads with the fact `false`, and one whose +> `final_status` is `MAX_TURNS_EXHAUSTED` does not load, so `run --resume` runs that row +> again and `evaluate ` cannot re-grade it from its recorded config. A recorded +> `run_limits.max_turns` or `run_limits.expected_turns` loads, so `evaluate ` +> re-grades from the recorded config; no `expected_turns` overage is computed for a record +> without `model_turns`. > -> Runs written before the tool-call cap replaced the turn cap carry -> `turns_remaining_at_stop` instead of `tool_calls_remaining_at_stop`, in both -> `EarlyStopInfo` and the `run.json` row. No reader maps the old key. Their recorded -> config also sets `run_limits.max_turns`, which no longer validates, so -> `evaluate ` re-grades such a run from the source task YAML and prints its -> fallback warning. +> Older runs also carry `turns_remaining_at_stop` instead of +> `tool_calls_remaining_at_stop`, in both `EarlyStopInfo` and the `run.json` row. No +> reader maps the old key. ### Missing cost is never fatal @@ -144,7 +149,8 @@ The authoritative per-replicate record. | --- | --- | --- | | `final_status` | [`FinalStatus`](#finalstatus) | Terminal status. | | `weighted_score` | `float \| null` | Weighted average of criterion scores, 0.0–1.0. | -| `tool_calls_exhausted` | `bool` | The tool-call cap ended an iteration before the agent completed on its own. | +| `tool_calls_exhausted` | `bool` | A structural cap (`max_tool_calls` or `max_turns`) ended an iteration before the agent completed on its own. | +| `model_turns` | `int \| null` | Main-thread model turns counted by the TurnMonitor; null where the harness does not count them, when no turn finished, or on older runs. | | `iteration_count` | `int` | Number of turns. | | `success_criteria_results` | `list[CriterionResult]` | Per-criterion results — see [below](#criterionresult). | | `post_failure_criteria_results` | `list[CriterionResult]` | Diagnostic artifact evidence collected after a terminal agent failure. It does not affect `final_status`, `weighted_score`, gating, or suite aggregation. | @@ -174,6 +180,11 @@ each of `system_prompt`, `plugin_skills`, `permission_mode`, `allowed_tools` and `disallowed_tools`, `"enforced"` or `"unsupported"`, plus `system_prompt_semantics` (the class default), `cooperative_stop`, and `permission_modes` (the sorted `permission_mode` values the harness honors, or `null`). +`environment_info.harness_version` is the version of the CLI or SDK the agent drove, read +where the agent ran (inside the container under `driver: docker`), for example +`"pi 0.84.4"` or `"claude-agent-sdk 0.2.124; Claude Code 2.1.216"`. It is `null` when the +agent cannot report one (the `none` agent, or a plugin agent that does not implement +`Agent.harness_version`), and absent on runs recorded before the field. `environment_info.skills_offered` is the list of skill names the staged plugin root offered to the agent. It is absent when the task sets no `agent.plugins`. `sdk_options.system_prompt` is a `SystemPromptPreset` dict @@ -230,7 +241,8 @@ canonical score remains 0.0. cache buckets, captured proxy-side on the LiteLLM open-weight backend and rendered by the evalboard as a per-call table; empty on every other backend), `num_turns`, `tool_calls_exhausted`, -`result_summary` (`{is_error, subtype, stop_reason, result}`), `crashed`, +`result_summary` (`{is_error, subtype, stop_reason, result}`: how a clean turn ended, `result` being +the agent's final reply; `null` on a crashed or timed-out turn), `crashed`, `crash_reason`. > **Token invariant.** Summing the four token buckets across `messages` @@ -250,7 +262,8 @@ Fields: `reason` (`criterion_passed` / `criterion_failed` / criterion timed out undecided past its `stop_early.decide_within`; it gates through the same weighted armed gate as a native fail), `deciding_criterion_type`, `deciding_criterion_description`, `armed_criteria`, -`sdk_turn_index`, `tool_call_index` (1-based, includes the in-flight call), +`sdk_turn_index` (main-thread model turns started at the stop, each turn id once per +`communicate()`), `tool_call_index` (1-based, includes the in-flight call), `elapsed_seconds`, `tool_calls_remaining_at_stop` (`max_tool_calls − tool_call_index`, floored at `0`; `null` when `run_limits.max_tool_calls` is unset), `gate_threshold` (the `run_limits.stop_early_gate_threshold` in effect for this stop; default `1.0`). diff --git a/docs/TASK_DEFINITION_GUIDE.md b/docs/TASK_DEFINITION_GUIDE.md index 3013dee0..e101d4f1 100644 --- a/docs/TASK_DEFINITION_GUIDE.md +++ b/docs/TASK_DEFINITION_GUIDE.md @@ -259,7 +259,9 @@ valid and an empty block is legal — every field defaults to "no limit". run_limits: # Structural caps max_tool_calls: 20 # hard cap on resolved tool calls across the whole task + max_turns: 15 # hard cap on model turns (Claude Code, OpenCode, Pi) expected_tool_calls: 8 # SOFT efficiency budget (visible tool calls) — never aborts + expected_turns: 8 # SOFT target on model turns — never aborts task_timeout: 300 # wall-clock cap for the full run envelope, seconds turn_timeout: 300 # per-communicate() timeout, seconds @@ -274,13 +276,15 @@ run_limits: | Field | Default | Constraint | Description | |-------|---------|------------|-------------| | `max_tool_calls` | *unset* | `> 0` | Hard cap on resolved tool calls across the whole task: every retry attempt and every dialog turn count. The TurnMonitor enforces it at the agent's next poll boundary, on every harness. The round that reaches the cap is processed whole, so tool calls already in flight can still land after it. The run finalizes cleanly as `tool_calls_exhausted`, and the criteria are still checked. Unset means no cap. | +| `max_turns` | *unset* | `> 0` | Hard cap on model turns (main-thread model responses) across the whole task: every retry attempt and every dialog turn count; a sub-agent's turns do not. The TurnMonitor stops the agent at its next poll once turn N+1 starts, so part of that turn can still land; a run that ends at exactly N turns is not capped. Every built-in harness accepts it; a harness that reports one turn per `communicate()` call rejects it at resolution (see [Run-Limit Parity](agents/HARNESS_PARITY.md)). The run finalizes as `tool_calls_exhausted`, like `max_tool_calls`. Unset means no cap. | | `expected_tool_calls` | *unset* | `>= 1` | **Soft** target for cumulative visible tool calls. Exceeding it warns and badges the report; it never aborts. See [`expected_tool_calls`](#expected_tool_calls-soft-efficiency-budget). | +| `expected_turns` | *unset* | `>= 1` | **Soft** target for cumulative model turns, counted like `max_turns`. Exceeding it warns and badges the report; it never aborts. Every built-in harness accepts it. For a target on visible tool calls, use `expected_tool_calls`. | | `task_timeout` | *unset* | `>= 30` | Max seconds for the full run envelope, including agent work, grading, and post-run work. | | `turn_timeout` | *unset* | `>= 10` | Max seconds for the agent's single `communicate()` iteration. | | `max_input_tokens` | *unset* | `>= 1` | Max cumulative input (prompt) tokens. | | `max_output_tokens` | *unset* | `>= 1` | Max cumulative output (completion) tokens. | | `max_total_tokens` | *unset* | `>= 1` | Max cumulative input + output tokens. Distinct from [`simulation.max_total_tokens`](#simulation-multi-turn-user-dialog) — see the note below. | -| `max_usd` | *unset* | `> 0.0` | Max cumulative cost in USD. Requires per-turn SDK cost reporting. | +| `max_usd` | *unset* | `> 0.0` | Max cumulative cost in USD. On a harness that does not report its own cost, `agent.model` must have a rate (checked at resolution). | | `count_cached_input` | `false` | — | Count `cache_read_input_tokens` toward the input/total budgets. Off by default — cached reads are typically free. | | `count_cache_creation` | `false` | — | Count `cache_creation_input_tokens` toward the input/total budgets. Off by default. | | `stop_early` | *unset* | `false` or unset | Run-level early-stop **kill switch** — there is no master arm. Unset: the criteria's own `stop_early:` blocks decide. `false`: force-disarm every block for this run. `true` (the removed master arm) is rejected at resolution. See [`stop_early`](#stop_early-opt-in-early-stop). | @@ -308,11 +312,17 @@ model. `FinalStatus.COST_BUDGET_EXCEEDED` (`max_usd`). Both categorize as `failed` — see [Report Schema](REPORT_SCHEMA.md). - **`max_usd` is priced from the harness's reported cost**, else from the rate card in - `coder_eval.pricing` for the model the harness reports (then `agent.model`). A turn with no usage - costs nothing. A run that can price a turn neither way finishes **`ERROR`** at that turn's end with - the message "run_limits.max_usd could not be enforced". It is never skipped. Add a rate with - `register_pricing`, pin a priced model, or remove `max_usd`. Mid-turn usage reports rarely carry a - cost, so when the model has no rate the USD cap is checked once the turn's reported cost arrives. + `coder_eval.pricing` for `agent.model` (then the model the harness reports). A turn with no usage + costs nothing. It is never skipped: + - On a harness whose contract does not set `reports_cost` (see + [Run-Limit Parity](agents/HARNESS_PARITY.md)), a task with `max_usd` and no priced + `agent.model` is **rejected at resolution**, so `coder-eval plan` fails. + - A run that still cannot price a turn finishes **`ERROR`** with the message + "run_limits.max_usd could not be enforced". On a harness that does not report cost, this + happens at the first unpriced usage report, mid-turn. + - Add a rate with `register_pricing`, pin a priced model, or remove `max_usd`. On a harness + that reports cost, mid-turn usage reports rarely carry a cost, so when the model has no rate + the USD cap is checked once the turn's reported cost arrives. - **Cached-read and cache-creation tokens are excluded by default.** `count_cache_creation: true` is what makes an input-token budget meaningful for **Codex**, which buckets its fresh (full-price) prompt slice into `cache_creation`; with the default `false`, a Codex token budget effectively @@ -338,9 +348,9 @@ coder-eval run task.yaml -D run_limits.max_usd=2.50 -D run_limits.max_total_toke > the agent model's `extra="forbid"` raises a clear validation error. > `turn_timeout` and `task_timeout` must live under `run_limits:`. (A > deprecation shim hoisted them automatically until it was removed on -> 2026-06-01.) `max_turns` under `run_limits:` is rejected too: use -> `run_limits.max_tool_calls`, which counts resolved tool calls, not agent -> inner-loop turns. +> 2026-06-01.) Under `run_limits:`, `max_turns` now counts model turns +> across the whole task, not SDK turns per call; to cap tool calls instead, +> use `max_tool_calls`. ### `expected_tool_calls` (soft efficiency budget) @@ -1565,7 +1575,9 @@ pre_run: | `timeout` | 30 | Maximum seconds to wait (1–300) | | `fail_on_error` | `true` | When true, failure aborts evaluation with `FinalStatus.ERROR` | -Commands run sequentially with `cwd` set to the sandbox directory. stdout and stderr are +Commands run sequentially with `cwd` set to the sandbox directory, with stdin on +`/dev/null`: a command that reads stdin gets end-of-file at once instead of waiting for +input nobody can type. stdout and stderr are captured in `pre_run_results` on the evaluation result (truncated to 100KB each). When a command fails with `fail_on_error: true`, remaining commands are skipped. diff --git a/docs/USER_GUIDE.md b/docs/USER_GUIDE.md index 0867b5b4..73fb4b67 100644 --- a/docs/USER_GUIDE.md +++ b/docs/USER_GUIDE.md @@ -81,7 +81,7 @@ you want to iterate on afterwards. Grade the results later with budget breach still reports `ERROR` / `TIMEOUT` / `TOKEN_BUDGET_EXCEEDED` and still exits non-zero, exactly as under `run`. -Exhausting `max_tool_calls` is the one fact that does *not* become a status here. Under +Exhausting `max_tool_calls` or `max_turns` is the one fact that does *not* become a status here. Under `run` it decides the outcome only when the criteria fail — a capped trajectory whose criteria pass is `SUCCESS` — so it is not knowable without grading. `execute` records `tool_calls_exhausted: true` on the row and finalizes `NOT_GRADED`; the later @@ -189,9 +189,9 @@ expansion are already baked into `resolved`, so re-loading the source would silently grade a *different* task. The run's trajectory is restored too, so criteria that read the agent's tool calls (`command_executed`, `skill_triggered`, judges with trajectory) score exactly as they would have during the run. -A run recorded before `run_limits.max_turns` became `run_limits.max_tool_calls` carries -the old key, which no longer validates, so `evaluate` re-grades it from the source task -YAML and prints its fallback warning. +A run recorded by an older release with `run_limits.max_turns` validates again, so +`evaluate` re-grades it from its recorded config. The re-grade skips the harness checks, +so a recorded Codex or Antigravity run with that key re-grades too. It writes the verdict back into the run's `task.json` and keeps the pre-grade record beside it as `task.execute.json`. Writing back in place is what makes diff --git a/docs/agents/CLAUDE_CODE.md b/docs/agents/CLAUDE_CODE.md index 78bd948a..12fb1140 100644 --- a/docs/agents/CLAUDE_CODE.md +++ b/docs/agents/CLAUDE_CODE.md @@ -116,6 +116,11 @@ agent: > trips the turn ends `COMPLETED` with `result_summary.subtype == "error_max_turns"`, > while `run_limits.max_tool_calls` is the framework's cap on resolved tool calls > (the `TurnMonitor` enforces it, as on every harness); both apply. +> +> `run_limits.max_turns` counts main-thread API responses across the whole task, not SDK +> turns per call. A sub-agent's responses do not count, and a response interrupted by its +> sub-agent counts once. The stop lands when response N+1 arrives, so that response is +> recorded in part and its tools do not run. `sdk_options.max_turns` is unrelated. > **System-prompt reproducibility.** In `append` mode the preset's *dynamic > sections* (working directory, git status, auto-memory) are excluded so the system @@ -189,9 +194,11 @@ simulator force `[]` for the same reason.) - **Plugins** are supplied as `plugins: [{type: local, path: …}]`. Point `path` at a plugin root (`/skills//SKILL.md`) or at a bare skills directory (`//SKILL.md`). coder-eval stages both into `/plugin_root` and - hands that root to the SDK. A path with no skill fails `plan`. Only skills are - staged: a plugin's `agents/`, `commands/` and `hooks/` do not reach the evaluated - agent. See [Plugin staging](HARNESS_PARITY.md#plugin-staging). + hands the SDK one local plugin per entry. A plugin root loads whole under its own name + (`:`), with its agents, commands, hooks and MCP servers; a bare skills + directory loads its skills only. Plugins that offer no skill at all fail `plan`. To measure a skill + alone, point `path` at the skills directory. See + [Plugin staging](HARNESS_PARITY.md#plugin-staging). - **`PLUGIN_TOOLS_DIR`** pins the canonical `node_modules/@uipath` directory for UiPath CLI plugin discovery; when unset the sandbox derives it from the resolved `uip` binary. See [User Guide → Environment Variables](../USER_GUIDE.md#environment-variables). @@ -237,10 +244,11 @@ event to the task log. advances on clean (non-error) turns. - **Timeouts** are enforced by a `ThreadedWatchdog` (an OS-thread timer immune to event-loop stalls) plus an in-loop wall-clock guard; on breach it SIGKILLs the CLI - subprocess and raises `TurnTimeoutError` with a partial `TurnRecord` preserved. -- **Crashes** raise `AgentCrashError` with assembled stderr; the orchestrator drains - the partial turn and rolls back. Cost is backfilled from the rate card when a run - is killed before a terminal result message arrives. + subprocess and `communicate` returns a `TIMEOUT` outcome whose record is the + `crashed=True` partial turn. +- **Crashes** return a `CRASHED` outcome with assembled stderr as the reason; the + orchestrator appends the partial record and retries. Cost is backfilled from the rate + card when a run is killed before a terminal result message arrives. ## References diff --git a/docs/agents/CODEX.md b/docs/agents/CODEX.md index e00b8818..7da001cf 100644 --- a/docs/agents/CODEX.md +++ b/docs/agents/CODEX.md @@ -133,14 +133,13 @@ Agent (ABC) ### Key Methods - **`start(working_directory)`** - Initialize Codex client and set working directory -- **`communicate(user_input, timeout, stream_callback)`** - Execute one turn with Codex +- **`communicate(user_input, iteration, timeout, stream_callback)`** - Execute one turn with Codex and return its `TurnOutcome` - **`stop()`** - Clean up resources - **`get_state()`** - Return current agent state -- **`discard_pending_turn()`** - Rollback on failure ### TurnRecord Format -Each turn returns a `TurnRecord` with: +Each turn's outcome carries a `TurnRecord` with: - `iteration` - Turn number - `user_input` - The prompt sent - `agent_output` - assembled from the streamed `agentMessage` deltas @@ -154,14 +153,13 @@ Each turn returns a `TurnRecord` with: ### Timeout Handling -The agent uses a `ThreadedWatchdog` to enforce wall-clock timeouts. If a turn exceeds the deadline, a `TurnTimeoutError` is raised with a partial `TurnRecord` preserved in `pending_turn`. +The agent uses a `ThreadedWatchdog` to enforce wall-clock timeouts. If a turn exceeds the deadline, `communicate` returns a `TIMEOUT` outcome whose record is the `crashed=True` partial turn. ### Error Recovery -On failure, the agent: -1. Sets `pending_turn` to a `crashed=True` TurnRecord with captured telemetry -2. Raises `AgentCrashError` or `TurnTimeoutError` -3. The orchestrator reads `pending_turn` and calls `discard_pending_turn()` to roll back state +On failure, the agent returns a `CRASHED` or `TIMEOUT` outcome whose record is a +`crashed=True` `TurnRecord` with the captured telemetry. The orchestrator appends that +record to the result, then retries a crash and ends the task on a timeout. ### Permission and Tool Mapping diff --git a/docs/agents/HARNESS_PARITY.md b/docs/agents/HARNESS_PARITY.md index 84b193e5..fcfc5631 100644 --- a/docs/agents/HARNESS_PARITY.md +++ b/docs/agents/HARNESS_PARITY.md @@ -3,7 +3,9 @@ One task file, run on any harness, must be the same task. `max_turns` broke that promise hardest: Claude Code enforced it, and Codex and Antigravity accepted it and never read it, so `max_turns: 6` ran capped on one backend and unbounded on the other -two. It is now `max_tool_calls`, one counter on every harness. +two. Now `max_tool_calls` is one counter on every harness, and `max_turns` counts model +turns on every harness that opens one inner turn per model response: all five built-in +agents. Only a harness that reports once per `communicate()` rejects it. This page is the contract for what each run limit means per harness, plus what each shared `agent` field means on each harness. Both tables are generated. @@ -17,14 +19,16 @@ CE069 fails the build on drift. | limit | claude-code | codex | antigravity | opencode | pi | none | | --- | --- | --- | --- | --- | --- | --- | -| `max_tool_calls` | TurnMonitor at the should_stop poll, resolved tool calls | TurnMonitor at the should_stop poll, resolved tool calls | TurnMonitor at the should_stop poll, resolved tool calls | TurnMonitor at the should_stop poll, resolved tool calls | TurnMonitor at the should_stop poll, resolved tool calls | not polled (never fires) | +| `max_tool_calls` | TurnMonitor at the should_stop poll, main-thread resolved tool calls | TurnMonitor at the should_stop poll, main-thread resolved tool calls | TurnMonitor at the should_stop poll, main-thread resolved tool calls | TurnMonitor at the should_stop poll, main-thread resolved tool calls | TurnMonitor at the should_stop poll, main-thread resolved tool calls | not polled (never fires) | +| `max_turns` | TurnMonitor at the should_stop poll, when main-thread model turn N+1 starts | TurnMonitor at the should_stop poll, when main-thread model turn N+1 starts | TurnMonitor at the should_stop poll, when main-thread model turn N+1 starts | TurnMonitor at the should_stop poll, when main-thread model turn N+1 starts | TurnMonitor at the should_stop poll, when main-thread model turn N+1 starts | rejected at resolution | | `expected_tool_calls` | orchestrator, cumulative visible tool calls, warns only | orchestrator, cumulative visible tool calls, warns only | orchestrator, cumulative visible tool calls, warns only | orchestrator, cumulative visible tool calls, warns only | orchestrator, cumulative visible tool calls, warns only | orchestrator, cumulative visible tool calls, warns only | +| `expected_turns` | orchestrator, cumulative model turns (TurnMonitor count), warns only | orchestrator, cumulative model turns (TurnMonitor count), warns only | orchestrator, cumulative model turns (TurnMonitor count), warns only | orchestrator, cumulative model turns (TurnMonitor count), warns only | orchestrator, cumulative model turns (TurnMonitor count), warns only | rejected at resolution | | `task_timeout` | orchestrator, agent-agnostic | orchestrator, agent-agnostic | orchestrator, agent-agnostic | orchestrator, agent-agnostic | orchestrator, agent-agnostic | orchestrator, agent-agnostic | | `turn_timeout` | agent watchdog (see Timeouts) | agent watchdog (see Timeouts) | agent watchdog (see Timeouts) | agent watchdog (see Timeouts) | agent watchdog (see Timeouts) | agent watchdog (see Timeouts) | -| `max_input_tokens` | TurnMonitor; usage reported per model generation; overshoot ≤ one model generation + calls in flight | TurnMonitor; usage reported per communicate() call; overshoot ≤ one communicate() call + calls in flight | TurnMonitor; usage reported per communicate() call; overshoot ≤ one communicate() call + calls in flight | TurnMonitor; usage reported per agent-loop step; overshoot ≤ one agent-loop step + calls in flight | TurnMonitor; usage reported per agent-loop step; overshoot ≤ one agent-loop step + calls in flight | TurnMonitor; usage reported per communicate() call; overshoot ≤ one communicate() call + calls in flight | -| `max_output_tokens` | TurnMonitor; usage reported per model generation; overshoot ≤ one model generation + calls in flight | TurnMonitor; usage reported per communicate() call; overshoot ≤ one communicate() call + calls in flight | TurnMonitor; usage reported per communicate() call; overshoot ≤ one communicate() call + calls in flight | TurnMonitor; usage reported per agent-loop step; overshoot ≤ one agent-loop step + calls in flight | TurnMonitor; usage reported per agent-loop step; overshoot ≤ one agent-loop step + calls in flight | TurnMonitor; usage reported per communicate() call; overshoot ≤ one communicate() call + calls in flight | -| `max_total_tokens` | TurnMonitor; usage reported per model generation; overshoot ≤ one model generation + calls in flight | TurnMonitor; usage reported per communicate() call; overshoot ≤ one communicate() call + calls in flight | TurnMonitor; usage reported per communicate() call; overshoot ≤ one communicate() call + calls in flight | TurnMonitor; usage reported per agent-loop step; overshoot ≤ one agent-loop step + calls in flight | TurnMonitor; usage reported per agent-loop step; overshoot ≤ one agent-loop step + calls in flight | TurnMonitor; usage reported per communicate() call; overshoot ≤ one communicate() call + calls in flight | -| `max_usd` | TurnMonitor; usage reported per model generation; overshoot ≤ one model generation + calls in flight | TurnMonitor; usage reported per communicate() call; overshoot ≤ one communicate() call + calls in flight | TurnMonitor; usage reported per communicate() call; overshoot ≤ one communicate() call + calls in flight | TurnMonitor; usage reported per agent-loop step; overshoot ≤ one agent-loop step + calls in flight | TurnMonitor; usage reported per agent-loop step; overshoot ≤ one agent-loop step + calls in flight | TurnMonitor; usage reported per communicate() call; overshoot ≤ one communicate() call + calls in flight | +| `max_input_tokens` | TurnMonitor; usage reported per model generation; overshoot ≤ one model generation + calls in flight | TurnMonitor; usage reported per model generation; overshoot ≤ one model generation + calls in flight | TurnMonitor; usage reported per model generation; overshoot ≤ one model generation + calls in flight | TurnMonitor; usage reported per agent-loop step; overshoot ≤ one agent-loop step + calls in flight | TurnMonitor; usage reported per agent-loop step; overshoot ≤ one agent-loop step + calls in flight | TurnMonitor; usage reported per communicate() call; overshoot ≤ one communicate() call + calls in flight | +| `max_output_tokens` | TurnMonitor; usage reported per model generation; overshoot ≤ one model generation + calls in flight | TurnMonitor; usage reported per model generation; overshoot ≤ one model generation + calls in flight | TurnMonitor; usage reported per model generation; overshoot ≤ one model generation + calls in flight | TurnMonitor; usage reported per agent-loop step; overshoot ≤ one agent-loop step + calls in flight | TurnMonitor; usage reported per agent-loop step; overshoot ≤ one agent-loop step + calls in flight | TurnMonitor; usage reported per communicate() call; overshoot ≤ one communicate() call + calls in flight | +| `max_total_tokens` | TurnMonitor; usage reported per model generation; overshoot ≤ one model generation + calls in flight | TurnMonitor; usage reported per model generation; overshoot ≤ one model generation + calls in flight | TurnMonitor; usage reported per model generation; overshoot ≤ one model generation + calls in flight | TurnMonitor; usage reported per agent-loop step; overshoot ≤ one agent-loop step + calls in flight | TurnMonitor; usage reported per agent-loop step; overshoot ≤ one agent-loop step + calls in flight | TurnMonitor; usage reported per communicate() call; overshoot ≤ one communicate() call + calls in flight | +| `max_usd` | TurnMonitor; usage reported per model generation; overshoot ≤ one model generation + calls in flight; priced by the harness | TurnMonitor; usage reported per model generation; overshoot ≤ one model generation + calls in flight; needs a priced agent.model (checked at resolution) | TurnMonitor; usage reported per model generation; overshoot ≤ one model generation + calls in flight; needs a priced agent.model (checked at resolution) | TurnMonitor; usage reported per agent-loop step; overshoot ≤ one agent-loop step + calls in flight; needs a priced agent.model (checked at resolution) | TurnMonitor; usage reported per agent-loop step; overshoot ≤ one agent-loop step + calls in flight; needs a priced agent.model (checked at resolution) | TurnMonitor; usage reported per communicate() call; overshoot ≤ one communicate() call + calls in flight; priced by the harness | | `count_cached_input` | TurnMonitor bucket rule | TurnMonitor bucket rule | TurnMonitor bucket rule | TurnMonitor bucket rule | TurnMonitor bucket rule | TurnMonitor bucket rule | | `count_cache_creation` | TurnMonitor bucket rule | TurnMonitor bucket rule | TurnMonitor bucket rule | TurnMonitor bucket rule | TurnMonitor bucket rule | TurnMonitor bucket rule | | `stop_early` | cooperative should_stop | cooperative should_stop | cooperative should_stop | cooperative should_stop | cooperative should_stop | rejected at resolution | @@ -39,6 +43,10 @@ A capped run is an ordinary end of run, never `ERROR` and never retried: - Calls of the round that reached the cap can still resolve after it. A call in flight is recorded with `result_status: unknown`, and a Codex sub-agent's recovered calls still reach the record. +- Only main-thread calls count. A sub-agent's calls (Claude's `Task` sub-agent, Codex's + recovered sub-agent calls) are nested under the call that spawned them: they reach + `TurnRecord.commands` but never the cap, and a sub-agent's model never becomes + `model_used`. `tasks/run_limits/subagent_cap.yaml` is the live check. ## Agent-field contract @@ -54,7 +62,9 @@ Generated from each agent class's `contract` by `make parity-table`; CE069 fails | `allowed_tools` | enforced | unsupported | enforced | enforced | enforced | unsupported | | `disallowed_tools` | enforced | unsupported | enforced | enforced | enforced | unsupported | | `cooperative_stop` | yes | yes | yes | yes | yes | no | -| `usage_granularity` | generation | turn | turn | step | step | turn | +| `usage_granularity` | generation | generation | generation | step | step | turn | +| `timing_basis` | turn_clock | cli_epoch_ms | turn_clock | cli_epoch_ms | turn_clock | turn_clock | +| `reports_cost` | yes | no | no | no | no | yes | | `permission_modes` | acceptEdits, bypassPermissions, default, plan | — | bypassPermissions, plan | bypassPermissions, plan | bypassPermissions, plan | — | @@ -65,6 +75,19 @@ developer instruction channel of the model request, never the user turn. `usage_granularity` is how often a harness reports token usage on the stream (per model generation, per agent-loop step, or once per `communicate()`); a token or USD budget can overshoot by one such report. +`usage_granularity` also decides the model-turn limits (`max_turns`, +`expected_turns`): a harness that reports per generation or per step opens one inner turn +per model response, so the TurnMonitor can count them; a harness that reports once per +`communicate()` rejects them at resolution. Codex opens a turn at the first item of a +generation and closes it at that generation's `thread/tokenUsage/updated`; Antigravity +opens one at the first MODEL Step that carries new content or an unseen tool call and +closes it at the Step that carries `usage_metadata`. A tool result that lands after the +cut is not a new turn on either. +`reports_cost` is whether every finished turn carries a cost the harness computed. On a +harness that does not, `max_usd` is priced from `coder_eval.pricing`, so a task that sets +`max_usd` must pin an `agent.model` with a rate, or it is rejected at resolution. Pi and +OpenCode report a cost for the models they know, but $0 for the others, so they count as +not reporting. ### Tool names @@ -99,26 +122,26 @@ wall clock its numbers account for. | Field | claude-code | codex | antigravity | opencode | pi | |---|---|---|---|---|---| -| `generation_duration_ms` RAW window (the reducer's part) | harness clock: previous SDK event → this message | SDK item stamps | harness clock: previous flush → this flush | harness clock: previous `step_finish` → this one | harness clock: previous `turn_end` → this one | +| `generation_duration_ms` RAW window (the reducer's part) | harness clock: previous SDK event → this message | SDK item stamps | harness clock: previous flush → this flush | CLI envelope `timestamp`: previous `step_finish` → this one | harness clock: previous `turn_end` → this one | | tool time subtracted from it | centrally | centrally | centrally | centrally | centrally | | what the **first** window covers | the first `message_start`, so CLI boot + TTFT are OUTSIDE it | the first SDK item's own start, so CLI boot + TTFT are OUTSIDE it | the first MODEL-source `Step`, so dispatch + TTFT are OUTSIDE it | the first `step_start`, so CLI boot + TTFT are OUTSIDE it | the first `turn_start`, so CLI boot + TTFT are OUTSIDE it | | `harness_startup_ms` (turn head) | ~3.6 s — CLI boot fused with TTFT | ~3.1 s — CLI boot fused with TTFT | ~4.7 s — dispatch fused with TTFT (its harness process is spawned once at startup, not per turn) | ~2.5 s — CLI boot fused with TTFT | ~0.23 s — CLI boot fused with TTFT | | `harness_teardown_ms` (turn tail) | ~1.3 s | ~13 ms | ~7 ms | ~26 ms | ~19 ms | -| tool `duration_ms` source | measured around the tool result | SDK `completed_at_ms − started_at_ms`; the item's own `duration_ms` only as a fallback | measured ACTIVE → DONE | measured around the tool event | measured around the tool event | -| `execution_started_at` / `execution_completed_at` | derived from the measured duration | SDK stamps (both, or neither) | measured at ACTIVE / DONE | measured | measured | +| tool `duration_ms` source | measured on the turn clock: `tool_use` arrival → result | SDK `completed_at_ms − started_at_ms`; the item's own `duration_ms` only as a fallback | measured ACTIVE → DONE | CLI `state.time.end − state.time.start` | measured around the tool event | +| `execution_started_at` / `execution_completed_at` | measured on the turn clock | SDK stamps (both, or neither) | measured at ACTIVE / DONE | CLI `state.time` stamps (none when absent) | measured | | `generation_completed_at` | set | `None` — see below | `None` | `None` | `None` | | `message_id` source | SDK `message_id`; `None` when the stream carries none; `subagent-` for a synthesized sub-agent terminal | synthetic `turn_id-msg-N`, shared across the sub-messages of one generation; `turn_id-subagent-N` for recovered sub-agent generations | synthetic `turn_id-msg-N`, one per generation | CLI `messageID`; `None` when absent | CLI `responseId`; `None` when absent | | `Σ generation + ∪ tool + head + tail ≈ turn duration` | yes [^identity] | yes [^identity] | yes [^identity] | yes [^identity] | yes [^identity] | -| clock basis for recorded stamps | one `TurnClock` per turn | SDK epoch ms (`_ms_to_dt`) — the subprocess's own clock, unreachable from the host, for BOTH window bounds and tool spans | one `TurnClock` per turn | **MIXED**: window bounds on the host `datetime.now()` (`:362`, `:696`); tool spans on CLI epoch ms (`_epoch_ms_to_dt`, `:406`/`:462`) | one `TurnClock` per turn | -| turn bracket (`AgentStartEvent` / `AgentEndEvent`) stamp | the same `TurnClock` (**CE064**) | raw `datetime.now()` — consistent with its epoch-ms bounds | the same `TurnClock` (**CE064**) | raw `datetime.now()` — consistent with its epoch-ms tool spans | the same `TurnClock` (**CE064**) | +| clock basis for recorded stamps | one `TurnClock` per turn | SDK epoch ms (`_ms_to_dt`) — the subprocess's own clock, unreachable from the host, for BOTH window bounds and tool spans | one `TurnClock` per turn | CLI epoch ms (`timing_basis` `cli_epoch_ms`): envelope `timestamp` for window bounds, `state.time` for tool spans; the host clock only for a window bound whose event carries no stamp | one `TurnClock` per turn | +| turn bracket (`AgentStartEvent` / `AgentEndEvent`) stamp | the same `TurnClock`, stamped by `TurnEmitter` | the host wall clock, stamped by `TurnEmitter` — consistent with its epoch-ms bounds | the same `TurnClock`, stamped by `TurnEmitter` | the host wall clock, stamped by `TurnEmitter` — consistent with its epoch-ms stamps | the same `TurnClock`, stamped by `TurnEmitter` | | window built by `timing.py::close_window` | yes | yes | yes | yes | yes | [^identity]: "yes" is load-bearing, and THREE sensors check it, each seeing something the others cannot. `tests/test_timing_identity_contract.py` is the committed two-sided one: it -drives every built-in reducer off a scripted clock, through a real -`EventCollector`, and asserts the four buckets tile the turn to the +drives every built-in reducer off a scripted clock, through +`coder_eval.testing.replay` and a real `TurnEmitter`, and asserts the four buckets tile the turn to the MILLISECOND. Magnitudes are only real where a scripted clock makes them real, which is why it is not in the golden corpus. @@ -167,10 +190,12 @@ window's own geometry: tile from the mark, keep a stamp that went backwards from inverting the span, clamp at zero. It had been copy-pasted four times, and Pi shipped a variant that measured from its own turn start — so every inter-turn gap fell into no bucket, and nothing failed, because the identity -above is asserted on one side only. **CE061** requires any module in `agents/` -publishing a measured `generation_duration_ms` to import the helper, and is now -**exemption-free**: claude-code was its one permanent `# noqa` and no longer -needs it. +above is asserted on one side only. The helper returns a `timing.Window`, a +frozen pair of bounds whose `duration_ms` clamps at zero, and +`TurnEmitter.add_generation` takes only a `Window`: an adapter cannot publish a +measured `generation_duration_ms` any other way. A generation with no window +goes through `TurnEmitter.add_unmeasured_generation`, which records +`generation_duration_ms=None`. **Tool execution comes out of the windows ONCE, at the collector.** `timing.py::subtract_tool_time` takes the union of the main-thread @@ -185,8 +210,9 @@ before the flush could subtract it — a 100% overstatement of that window), whe to clear a spent start stamp (a second flush with no intervening start republished the previous span — 3000 ms of generation for a 2000 ms turn), when to advance the mark. Those three lists, their reset rules, and the bounding of -still-open calls are all deleted. **CE063** stops a sixth harness rebuilding -them: no module in `agents/` may import `busy_ms`. +still-open calls are all deleted. A sixth harness has nowhere to rebuild them: +an adapter holds no spans, it calls `TurnEmitter.open_tool` / `close_tool`, and +**CE072** bans it from constructing the events or messages itself. Two consequences worth stating, because both are behaviour changes: @@ -234,38 +260,27 @@ instant", for a harness whose real tail is ~0.1 ms; the same task now records 0.035 ms. It surfaced only here because the drift between the two clocks is tens of microseconds and antigravity holds its process across turns, so nothing happens between its last flush and its end event; every other harness books a -tail of 7-543 ms, where the drift is invisible rather than absent. **CE064** -keeps a sixth harness from reintroducing it: a module under `agents/` that -imports `TurnClock` must pass an explicit `timestamp=` on both brackets. Codex -and OpenCode have no `TurnClock`, so the rule does not see them and their raw -`datetime.now()` bracket stays — which is *consistent* with their own CLI-epoch -bounds rather than a gap. - -claude-code has exactly one raw `datetime.now()` left, on the synthesized -sub-agent terminal message. Those bounds are an admitted placeholder for a -generation that arrives as a tool result and is never streamed -(`generation_duration_ms is None`, `parent_tool_use_id` set), which is what -excludes the message from `subtract_tool_time` and from the head/tail bracket. -A stamp no bucket reads has no basis to share. - -Codex and OpenCode are **not** converted, and their reasons are DIFFERENT — they -were stated as one, and that reading described a state OpenCode is already in. - -**Codex** is genuinely single-basis: both its window bounds and its tool spans -come from `_ms_to_dt` over the CLI's own epoch milliseconds, which cannot be -re-derived host-side. Converting only the window bounds would put two bases -inside one `busy_ms` subtraction — relocating the defect instead of removing it — -so it stays whole, and keeps the naive-local exposure. - -**OpenCode is already mixed, today.** Its window bounds are host -`datetime.now()` (`opencode_agent.py:362` at `step_start`, `:696` at -`step_finish`) while its tool spans are CLI epoch ms (`:406`, assigned to -`execution_started_at` at `:420`, and `:462`), so the two bases already meet -inside one subtraction. The argument for leaving it is therefore not the Codex -one: it is that a monotonic-derived anchor would trade a narrow NTP exposure on -the window bounds for intra-turn drift against the CLI's own tool stamps, which -is the larger of the two. The mixed basis is recorded here rather than defended -as uniform. +tail of 7-543 ms, where the drift is invisible rather than absent. The fix is +now structural: `TurnEmitter` stamps both brackets, and every other event, from +the one clock `Agent._open_emitter` gives it — a `TurnClock` under +`timing_basis` `turn_clock`, the host wall clock under `cli_epoch_ms`. An +adapter never stamps a bracket. For Codex and OpenCode the wall-clock bracket +is *consistent* with their own CLI-epoch bounds rather than a gap. + +The synthesized claude-code sub-agent terminal message has equal placeholder +bounds from the emitter's clock, for a generation that arrives as a tool result +and is never streamed (`generation_duration_ms is None`, `parent_tool_use_id` +set), which is what excludes the message from `subtract_tool_time` and from the +head/tail bracket. No bucket reads those stamps. + +Codex and OpenCode are **not** converted to a `TurnClock`: both are single-basis on the +CLI's own clock (`timing_basis` `cli_epoch_ms`). Codex takes its window bounds and tool +spans from `_ms_to_dt` over the SDK's epoch milliseconds; OpenCode takes its window bounds +from each event's envelope `timestamp` and its tool spans from `state.time`. Neither can be +re-derived host-side, and converting only the window bounds would put two bases inside one +`busy_ms` subtraction — relocating the defect instead of removing it. Both keep the +naive-local exposure. OpenCode falls back to the host clock only for a window bound whose +event carries no envelope stamp, and warns when it does. Deadlines on every harness stay on raw `time.monotonic()` and must — a deadline may not move when the wall clock steps. @@ -299,15 +314,12 @@ those WALL bounds. A monotonic-measured duration would have had the two disagreeing inside one subtraction, which is the defect that let Antigravity's window go negative. Sharing raw `datetime.now()` fixed the disagreement and left both sides naive-local; deriving both from the turn's monotonic anchor -removes that too. The clock is INJECTED into `_ClaudeTurnState` rather than -read from a module global, because a derived stamp escapes a monkeypatched -`datetime` — a test that patched one would quietly measure the real clock and -pass. `_resolve_pending_command` takes the reading as an argument for the same -reason: it stamps the tool span that is clipped against those bounds, so a -second basis at that one call site would put two clocks inside one subtraction. -`turn_start_time` stays raw monotonic and is untouched: `duration_seconds` and -the turn deadline read it, and a deadline must not move when the wall clock -steps. +removes that too. The tool span is now the emitter's own `open_tool` / +`close_tool` stamps on that same clock, so no duration is measured on a second +basis at all. The clock is the emitter's, injected per turn, because a derived +stamp escapes a monkeypatched `datetime` — a test that patched one would quietly +measure the real clock and pass. The turn deadline stays raw monotonic: it must +not move when the wall clock steps. **The head and tail are measured, not normalized.** Generation and tool are only two of the four buckets. The turn's **head** (turn start → first @@ -352,7 +364,7 @@ interval is removed centrally by `subtract_tool_time`, exactly as pi does with Why it survived so long is the more useful half. A tool-heavy shape cannot see it — three concurrent `sleep 3` calls make the tool union absorb the interval and the residual reads 0.05%. Neither can a single-tool-result fixture: -claude-code reconstructs `execution_started_at` by subtracting the measured +claude-code then reconstructed `execution_started_at` by subtracting the measured duration from the resolve instant, so with one message the discarded interval and the tool's own span are the SAME milliseconds and the identity closes either way. It takes a FAST tool plus a SECOND user message carrying no tool @@ -490,8 +502,8 @@ lacks one. Antigravity's `Step` stream carries no message id, so the harness synthesizes one — and it must, because this harness's generation windows are *contiguous* by construction: each opens exactly where the previous one closed, so the gap between two of them is always 0 ms and the fallback would fold a -whole turn's generations into a single row. CE060 makes the kwarg mandatory in -`src/coder_eval/agents/` for that reason. +whole turn's generations into a single row. `TurnEmitter.add_generation` makes +`message_id` a required keyword-only argument for that reason. The collapse is a *display* defect, not an accounting one — the consumer SUMS a group's token buckets and durations, so every total, percentage and cost is @@ -513,8 +525,8 @@ Antigravity's are all distinct, because it emits one message per generation with every block inside it. Runs recorded before a harness captured the field still carry `null` and still depend on the gap fallback, which is why it stays — and so does a current OpenCode or Pi message whose payload omitted the id, -which is the case CE060 cannot see (it requires the kwarg to be present, not -non-`None` at runtime). OpenCode tiles its windows contiguously too, so it is +which is the case the required keyword cannot see (it requires the argument to +be present, not non-`None` at runtime). OpenCode tiles its windows contiguously too, so it is the other harness where a missing id can still collapse a turn. ### Time to first token is not measured @@ -623,8 +635,8 @@ How each harness enforces `run_limits.turn_timeout` (the meaning is the same eve ### What a timeout looks like On Claude Code and Codex a `turn_timeout` breach is a *failure*: the watchdog fires -at the deadline, the partial turn is preserved on `pending_turn`, and the turn is -marked `crashed`. +at the deadline, the agent returns a `TIMEOUT` outcome, and its partial turn is kept as a +`crashed` record. Antigravity stops earlier and more gently, for the reason in the next section. @@ -665,24 +677,31 @@ Each `agent.plugins[].path` names a plugin root or a bare skills directory declared path may be one skill), or the root itself when it holds `SKILL.md`. A skill's name is its `SKILL.md` frontmatter `name`, else its directory name. Every layout works on every harness. Before the -agent starts, coder-eval stages the skills into one root, `/plugin_root`: a -`.claude-plugin/plugin.json` that names `coder-eval-plugins`, and one `skills/` symlink -per skill. Each harness receives that root in its native way. Only skills are staged: a -plugin's `agents/`, `commands/`, `hooks/` and `.mcp.json` do not reach any harness, Claude Code -included. Claude Code names staged skills `coder-eval-plugins:`, not `:`. -Files beside the skills also stay behind: a skill that reads `${CLAUDE_PLUGIN_ROOT}/scripts/` -or a shared `references/` directory at the plugin root cannot find it. Keep a skill's files -inside its own `/` directory. -A path that offers no skill, two paths that offer the same skill name, or a `skill_triggered` +agent starts, coder-eval stages every entry into `/plugin_root`: one +`skills/` symlink per skill, and one `plugins/` per entry. + +Claude Code loads each entry as a whole plugin under its own name (`:`): +its agents, commands, hooks, MCP servers and the files beside its skills load too, so +`${CLAUDE_PLUGIN_ROOT}` works. The plugin name is the manifest `name`, else the directory +name. A bare skills directory becomes a plugin named after the directory that holds its +skills only. Codex, OpenCode, Pi and Antigravity receive the skills only. + +Pointing `path` at a project `.claude` directory also loads its agents and commands on +Claude Code. To load the skills alone, point `path` at the skills directory itself. Two +entries with the same plugin name, or a manifest `skills` path outside its plugin root, +fail `coder-eval plan`. Under `driver: docker` the authored plugin roots are mounted +read-only, so a hook or MCP server that writes into its plugin root, or needs a binary the +image does not have, fails. +Plugins that offer no skill at all, two paths that offer the same skill name, or a `skill_triggered` criterion whose `skill_name` the plugins do not offer fail `coder-eval plan`, before the run is paid for. `environment_info.skills_offered` records the staged skill names; re-grading a recorded run whose `skill_name` is not in that list finishes `ERROR`, not 0.0. ## Reproducing -`tasks/run_limits/` holds one fixture per limit: `max_tool_calls_cap.yaml` asks for more -sequential work than its cap allows, and `turn_timeout.yaml` runs a command that -outlives its watchdog. Run either with `--type claude-code` / `--type codex` / +`tasks/run_limits/` holds one fixture per limit: `max_tool_calls_cap.yaml` and +`max_turns_cap.yaml` ask for more sequential work than their caps allow, and +`turn_timeout.yaml` runs a command that outlives its watchdog. Run either with `--type claude-code` / `--type codex` / `--type antigravity` / `--type opencode` / `--type pi` to check a backend against the contract above. diff --git a/docs/agents/OPENCODE.md b/docs/agents/OPENCODE.md index 58bf66c7..ddaafbc7 100644 --- a/docs/agents/OPENCODE.md +++ b/docs/agents/OPENCODE.md @@ -183,8 +183,8 @@ keys are coarser than its tools: `edit` governs `write`, `edit`, `patch`, no OpenCode equivalent restricts nothing; an allowlist of only such names denies every tool. An empty `allowed_tools: []` restricts nothing, as on Claude Code. Our rules are placed after every inherited rule, because OpenCode applies the last -matching rule. A host rule for `external_directory` or `doom_loop` is kept, so a tool -allowlist never loosens it. A string rule such as `read: "allow"` replaces the CLI's default +matching rule. A host rule for `external_directory` or `doom_loop` is kept and placed +after `"*": "deny"`, so a tool allowlist neither loosens nor hides it. A string rule such as `read: "allow"` replaces the CLI's default `.env` read deny, which is acceptable inside a sandbox. `system_prompt` is written to a temporary file outside the sandbox and listed in @@ -201,7 +201,14 @@ Mapping from the CLI's event vocabulary onto `TurnRecord`: | `text` | `TextChunkEvent` + `agent_output` | | `tool_use` | `ToolStartEvent` + `ToolEndEvent` (one terminal event carries both) | | `step_finish` | `TurnEndEvent` + per-step tokens/cost, one `AssistantMessage` | -| `error` | `AgentCrashError` with the partial turn preserved | +| `error` | a `CRASHED` turn, its partial record kept | + +Timing uses the CLI's own clock (`timing_basis` `cli_epoch_ms`): every event's envelope +`timestamp` (epoch ms) bounds the generation windows, and a tool's `state.time.start` / +`.end` is its execution span. A tool with no `state.time.end` gets no completion stamp and +no duration. An event with no envelope `timestamp` bounds its window on the host clock, +with one warning per turn. The CLI runs with stdin on `/dev/null`: it reads a non-TTY +stdin to EOF before it emits anything, so an inherited open stdin would stall the turn. Token buckets come from `step_finish.tokens`. Two conventions for `tokens.input` exist in the wild, and the stream's own `total` arbitrates **per step**: @@ -226,13 +233,13 @@ reconciliation invariant exact: summing the four buckets across Real per-call cost rides on `step_finish.cost` and lands on `token_usage.total_cost_usd`, so runs are costed from the provider's own accounting rather than the static rate card. The rate card -(`calculate_cost` over the captured buckets) fills two gaps so the run total +(`pricing.price_turn` over the captured buckets) fills two gaps so the run total never books tokens with no money: a stream that reports **no** cost at all (a provider or auth mode that omits it, or a turn that died before its first `step_finish`), and a stream that reports **`cost: 0`** for tokens the rate card prices above zero — OpenCode reports 0 when its own model registry has no price for the model, or under subscription-style auth, and neither means the -tokens were free (the fallback logs a warning naming the substituted amount). A +tokens were free. A *non-zero* cost the CLI reported always wins, and a genuinely free model still resolves to $0 because its rate entry is absent or all-zero. diff --git a/docs/agents/PI.md b/docs/agents/PI.md index 0fc1bf01..5a7456f0 100644 --- a/docs/agents/PI.md +++ b/docs/agents/PI.md @@ -231,8 +231,8 @@ docker` whenever the task prompt or workspace is not fully trusted. directory both work. Pi therefore **can run activation suites**: `skill_triggered` detects Pi's engagement agent-agnostically (the agent `read`s the full `SKILL.md`, a `read`→`Read` call whose `path` matches `skills//`). - Only skills are staged; a plugin's agents, hooks, commands and MCP servers are not - wired. See [Plugin staging](HARNESS_PARITY.md#plugin-staging). + Only skills reach Pi; a plugin's agents, hooks, commands and MCP servers are loaded + on Claude Code only. See [Plugin staging](HARNESS_PARITY.md#plugin-staging). - **`system_prompt_file` is not read by the adapter.** Use `system_prompt` (inline) instead — it is enforced via `--append-system-prompt`. - **`max_tool_calls` counts resolved tool calls, not Pi turns.** The adapter counts @@ -242,6 +242,10 @@ docker` whenever the task prompt or workspace is not fully trusted. - **No sub-agent attribution.** Pi's CLI stream does not expose nested agent generations, so per-sub-agent token grouping (available for Claude and Codex) is not derivable. +- **The CLI never inherits stdin.** `pi -p` reads a non-TTY stdin to EOF before it + emits anything, so a CLI that inherited an open stdin (a backgrounded or + tool-spawned `coder-eval run`) would stall with zero events until `turn_timeout`. + The adapter spawns it with stdin on `/dev/null`. - **Cooperative stop is at event granularity.** `should_stop` is polled between events and honored by terminating the CLI, so `stop_early` works, but the cut lands on an event boundary rather than mid-tool. Pi streams incrementally, so diff --git a/docs/index.md b/docs/index.md index 439a8ad8..1c9c9e22 100644 --- a/docs/index.md +++ b/docs/index.md @@ -15,7 +15,8 @@ skills** — built for benchmark authors, CLI builders, and skill builders — w sandboxing, reproducibility, and data-driven analysis. It runs a real agent — **Claude Code**, **OpenAI Codex**, **Google Antigravity (Gemini)**, **OpenCode**, or **Pi** — against declarative YAML tasks in a sandbox. Changing harness is one field -(`agent.type`); the tasks, criteria, scoring, telemetry, and reports stay the same. +(`agent.type`); a field the target harness cannot honor fails at load, and every harness +records its turn through one kernel, so the telemetry means the same thing on every harness. It is **not a fixed leaderboard**: you bring the tasks and you bring the scoring. If you have ever asked *"how do I benchmark coding agents on my own domain tasks?"*, diff --git a/docs/tutorials/04-writing-a-task.md b/docs/tutorials/04-writing-a-task.md index a472ed8c..57fd5074 100644 --- a/docs/tutorials/04-writing-a-task.md +++ b/docs/tutorials/04-writing-a-task.md @@ -64,7 +64,8 @@ What each block does: only when the task needs your MCP servers. - **`run_limits.expected_tool_calls`** — an efficiency target, not a cap: exceeding it logs a warning and adds a report badge but never aborts (use - `run_limits.max_tool_calls` for a hard cap). + `run_limits.max_tool_calls` for a hard cap). `run_limits.expected_turns` is the + same kind of target on model turns (Claude Code, OpenCode and Pi only). - **`success_criteria`** — each criterion scores 0.0–1.0 and supports `weight` (default 1.0) and `pass_threshold` (default 0.9). `run_command` here only checks the exit code; it can also match stdout (`expected_stdout`) or read a diff --git a/evalboard/lib/__tests__/runs.test.ts b/evalboard/lib/__tests__/runs.test.ts index a293ed18..b47a799e 100644 --- a/evalboard/lib/__tests__/runs.test.ts +++ b/evalboard/lib/__tests__/runs.test.ts @@ -79,6 +79,16 @@ describe("toTaskRow", () => { expect(row.expectedTurns).toBe(4); }); + test("a model-turn expected_turns never becomes the tool-call target", () => { + const row = toTaskRow({ task_id: "x", expected_tool_calls: null, expected_turns: 7 }); + expect(row.expectedTurns).toBeNull(); + }); + + test("expected_tool_calls wins over expected_turns when both are set", () => { + const row = toTaskRow({ task_id: "x", expected_tool_calls: 5, expected_turns: 7 }); + expect(row.expectedTurns).toBe(5); + }); + test("legacy raw shape (no new fields) yields null", () => { const row = toTaskRow({ task_id: "x" }); expect(row.totalTurns).toBeNull(); diff --git a/evalboard/lib/runs.ts b/evalboard/lib/runs.ts index 82e9f689..874b027d 100644 --- a/evalboard/lib/runs.ts +++ b/evalboard/lib/runs.ts @@ -514,7 +514,8 @@ export interface RawTaskResult { // null-fallback through the cell helpers in lib/turns.ts. total_turns?: number; expected_tool_calls?: number | null; - // Historical spelling of expected_tool_calls, on runs written before the rename. + // Model-turn target on current rows; the historical spelling of expected_tool_calls + // on rows without an expected_tool_calls key (see expectedToolCallsFromRaw). expected_turns?: number | null; // Derived expected wall clock for this task, stamped by the eval runner // (see eval_runner/skills/timing.py). Absent on unscored tasks and on every @@ -896,7 +897,7 @@ export function toTaskRow(t: RawTaskResult): TaskResultSummary { totalCostUsd: t.total_cost_usd ?? null, actualCommands: t.actual_commands ?? null, totalTurns: t.total_turns ?? null, - expectedTurns: t.expected_tool_calls ?? t.expected_turns ?? null, + expectedTurns: expectedToolCallsFromRaw(t), expectedSeconds: t.expected_seconds ?? null, hasFinalReply: t.has_final_reply ?? false, inputTokens: t.input_tokens ?? null, @@ -1190,6 +1191,14 @@ export interface RunOverview { timePerPassedTask?: number | null; } +// expected_turns is the historical spelling only on rows that predate expected_tool_calls; +// newer rows always carry expected_tool_calls, and their expected_turns counts model turns. +export function expectedToolCallsFromRaw( + t: Pick, +): number | null { + return t.expected_tool_calls !== undefined ? t.expected_tool_calls : (t.expected_turns ?? null); +} + // Visible-turn count for a task row: the persisted `visible_turns` field when // present, else reconstructed as actual_commands + (1 if final reply). That // reconstruction is the documented turn rule (tool calls + final reply) and is @@ -1280,7 +1289,7 @@ export async function readRunOverview( weightedScore: t.weighted_score ?? null, actualCommands: t.actual_commands ?? null, totalTurns: t.total_turns ?? null, - expectedTurns: t.expected_tool_calls ?? t.expected_turns ?? null, + expectedTurns: expectedToolCallsFromRaw(t), expectedSeconds: t.expected_seconds ?? null, visibleTurns: visibleTurnsFromRaw(t), hasFinalReply: t.has_final_reply ?? false, diff --git a/experiments/plugin-comparison.yaml b/experiments/plugin-comparison.yaml index 561ce4f2..07f0bad7 100644 --- a/experiments/plugin-comparison.yaml +++ b/experiments/plugin-comparison.yaml @@ -35,7 +35,8 @@ variants: - variant_id: with-plugin agent: plugins: - # $PLUGIN_PATH names a plugin root or a bare skills directory. Both are staged; - # a path with no skill fails `plan`. + # $PLUGIN_PATH names a plugin root or a bare skills directory. A plugin root loads + # whole on Claude Code; a bare skills directory loads its skills. A path with no + # skill fails `plan`. - type: "local" path: "$PLUGIN_PATH" diff --git a/plugins/coder-eval/reference/run-layout.md b/plugins/coder-eval/reference/run-layout.md index da111e9f..33925ac3 100644 --- a/plugins/coder-eval/reference/run-layout.md +++ b/plugins/coder-eval/reference/run-layout.md @@ -15,7 +15,7 @@ runs/////{task.json, task.log, artifacts/} - `task.json.graded` — present only after `coder-eval execute --driver docker` refused a container's verdict: the runtime image predated `execute` and graded anyway, so the runner quarantines the graded record here rather than leaving it readable as `task.json`, where a later `--resume` / `aggregate` would fold in exactly the row it declined to publish. Diagnostic-only; `rglob("task.json")` consumers do not match it. - `grade.log` — present only after a DETACHED grade over this directory (`coder-eval run --resume`). The grading pass's own log. It is a separate file because the log handler truncates whatever file it opens, so writing to `task.log` would destroy the agent trajectory log the run already paid for. - `task.log` — the human-readable task log; `artifacts/` — files the agent produced. -- `plugin_root/` — the staged plugin root the agent was handed; symlinks into the authored plugin; present only when the task sets `agent.plugins`. +- `plugin_root/` — the staged plugin root the agent was handed: `skills/` links into the authored skills and `plugins/` (each authored plugin whole); present only when the task sets `agent.plugins`. **Scope-marker files** (used to detect what a given path represents): diff --git a/plugins/coder-eval/reference/templates/activation.yaml b/plugins/coder-eval/reference/templates/activation.yaml index 1d699815..90e8c4cf 100644 --- a/plugins/coder-eval/reference/templates/activation.yaml +++ b/plugins/coder-eval/reference/templates/activation.yaml @@ -10,12 +10,14 @@ tags: [activation] # REPLACE: the skill under test must be REACHABLE by the sandboxed agent. `path` names a # plugin root (`/skills//SKILL.md`) or a bare skills directory -# (`//SKILL.md`); both are staged. Only skills are staged. For -# `.claude/skills/my-skill/SKILL.md`, `.claude` works and `.claude/skills` works too. -# Supply it through an environment variable so the committed suite stays portable -# across machines and CI: +# (`//SKILL.md`); both are staged. Claude Code loads the whole plugin +# (agents, commands, hooks); other harnesses get its skills. To measure the skill alone, +# point `path` at the skills directory: for `.claude/skills/my-skill/SKILL.md`, use +# `.claude/skills`. `.claude` also works, but on Claude Code it also loads the project's +# agents and commands. Supply it through an environment variable so the committed suite +# stays portable across machines and CI: # -# export SKILL_SOURCE_PATH=/abs/path/to/.claude +# export SKILL_SOURCE_PATH=/abs/path/to/.claude/skills # # An unset variable, or a path with no skill, fails `coder-eval plan` with a config error. agent: diff --git a/plugins/coder-eval/skills/analyze/SKILL.md b/plugins/coder-eval/skills/analyze/SKILL.md index 7527b727..ce9d084d 100644 --- a/plugins/coder-eval/skills/analyze/SKILL.md +++ b/plugins/coder-eval/skills/analyze/SKILL.md @@ -55,6 +55,8 @@ summary per task with `jq` (or `python3` if `jq` is missing): total_tokens: (.total_token_usage.input_tokens + .total_token_usage.output_tokens), assistant_turns: .total_assistant_turns, max_tool_calls: .task_config.resolved.run_limits.max_tool_calls, + max_turns: .task_config.resolved.run_limits.max_turns, + model_turns, criteria_count: (.success_criteria_results | length), all_criteria_perfect: (.success_criteria_results | length > 0 and all(.[]; .score == 1.0)), @@ -83,7 +85,11 @@ Two names *did* change between generations, which is what the `keys` check is fo | Current runs | Older runs | Where | | --- | --- | --- | | `iterations` | `turns` | top-level record key | -| `task_config.resolved.run_limits.max_tool_calls` | `task_config.resolved.run_limits.max_turns` (a turn cap, not a tool-call cap), and before that `task_config.resolved.max_iterations` | inside the free-form `task_config` dict | +| `task_config.resolved.run_limits.max_tool_calls` | `task_config.resolved.run_limits.max_turns` (a Claude Code SDK turn cap, not a tool-call cap), and before that `task_config.resolved.max_iterations` | inside the free-form `task_config` dict | + +A current run can ALSO set `run_limits.max_turns`: there it is a cap on main-thread model +turns, beside `max_tool_calls`. Read it as the older turn cap only on a record that has no +`max_tool_calls` key and no `model_turns`. Extract whichever the file actually has. The loader still accepts the older top-level name when reading, so an old run is not broken — but current runs do not write it, and @@ -159,7 +165,7 @@ passes: variant/run scope. 4. **Criteria** — sensitivity `weight × (threshold − score)`; fragile passes sitting on the threshold; redundant criteria and coverage gaps. -5. **Configuration** — lineage conflicts (`source != "task"`), tool-call cap (`max_tool_calls`) hit or +5. **Configuration** — lineage conflicts (`source != "task"`), tool-call cap (`max_tool_calls`) or model-turn cap (`max_turns`) hit or wildly excessive, model fit, `allowed_tools` alignment with what the task needs. 6. **Environment** — infrastructure errors, missing services, expired credentials, CLI tool errors. Also **idempotency and cross-run contamination**: a criterion that passed on diff --git a/plugins/coder-eval/skills/check-skill/SKILL.md b/plugins/coder-eval/skills/check-skill/SKILL.md index 1b786f59..e6368e52 100644 --- a/plugins/coder-eval/skills/check-skill/SKILL.md +++ b/plugins/coder-eval/skills/check-skill/SKILL.md @@ -166,12 +166,14 @@ template's copy rather than writing a second declaration. Otherwise, fill it in. **`path` names a plugin root or a bare skills directory**: the skill sits at `/skills//SKILL.md` or at `//SKILL.md`. Both are -staged. Only skills are staged, so a sibling `agents/`, `commands/` or `hooks/` directory -does not reach the evaluated agent. For `.claude/skills/pdf-forms/SKILL.md`, `.claude` works -and `.claude/skills` works too: +staged. Claude Code loads the whole plugin (agents, commands, hooks); other harnesses get its +skills. To measure the skill alone, point `path` at the skills directory. For +`.claude/skills/pdf-forms/SKILL.md`, use `.claude/skills`. `.claude` also works, but on +Claude Code it also loads the project's agents and commands, which can answer the request +instead of the skill: ```bash -export SKILL_SOURCE_PATH="$(pwd)/.claude" +export SKILL_SOURCE_PATH="$(pwd)/.claude/skills" ``` Keep it an environment variable rather than baking an absolute path into the YAML — the diff --git a/pyproject.toml b/pyproject.toml index 53e42040..ccc6705c 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -294,16 +294,14 @@ external = [ "CE056", "CE057", "CE058", - "CE059", - "CE060", - "CE061", - "CE063", - "CE064", "CE065", "CE066", "CE068", "CE069", "CE070", + "CE071", + "CE072", + "CE073", ] # custom architectural lint rules (tests/lint/) [tool.ruff.lint.pylint] diff --git a/src/coder_eval/agent.py b/src/coder_eval/agent.py index 60bdd494..c0442454 100644 --- a/src/coder_eval/agent.py +++ b/src/coder_eval/agent.py @@ -3,40 +3,20 @@ # by-design model-hub ↔ registry type-level cycle; runtime imports are lazy per CE017 # pyright: reportImportCycles=false -import logging +import asyncio +import contextlib from abc import ABC, abstractmethod from collections.abc import Callable +from datetime import datetime from pathlib import Path -from typing import Any, ClassVar, NoReturn, Protocol +from typing import Any, ClassVar -from .errors import AgentCrashError, TurnTimeoutError -from .errors.agent import format_timeout_reason, truncate_crash_message from .models import AgentState as AgentState -from .models import ApiRoute, BaseAgentConfig, HarnessContract, ToolNameMap, TurnRecord +from .models import ApiRoute, BaseAgentConfig, HarnessContract, TimingBasis, ToolNameMap from .streaming.callbacks import StreamCallback -from .streaming.collector import EventCollector -from .streaming.events import AgentEndStatus, StopReason - - -logger = logging.getLogger(__name__) - - -class _FinalizeFn(Protocol): - """The per-turn ``finalize`` callback shared by every agent's turn-state. - - Pinning the exact keyword-only signature here (instead of a loose - ``Callable[..., None]``) lets pyright catch a future ``Agent`` subclass that - wires an incompatible ``finalize`` into the shared mid-turn failure kernels. - """ - - def __call__( - self, - status: AgentEndStatus, - *, - crashed: bool = ..., - crash_reason: str | None = ..., - ) -> None: - """Finalize the current turn with the given end status.""" +from .streaming.emitter import Clock, TurnEmitter, TurnOutcome +from .streaming.events import StopReason +from .timing import TurnClock class Agent[ConfigT: BaseAgentConfig](ABC): @@ -53,23 +33,7 @@ def __init__(self, config: ClaudeCodeAgentConfig, ...): This ensures mypy enforces the correct config type for each agent. """ - pending_turn: TurnRecord | None = None - """Side-channel for partial turn records from failed ``communicate()`` calls. - - Implementations must set this to a ``crashed=True`` TurnRecord before - raising any mid-turn exception that carries captured telemetry. Callers - must read this slot after every failed ``communicate()`` call, then call - ``discard_pending_turn()`` to clear it. Outside ``communicate()``, this - slot is always None. - """ - - # Class-level defaults so a subclass gets the behaviour without re-declaring it. - # `_iteration_was_incremented` is consumed by `discard_pending_turn()`, which - # rolls the counter back exactly once per failed turn. - # Rationale: .claude/notes/reporting.md § The Agent ABC contract _state: AgentState = AgentState.WORKING - _iteration: int = 0 - _iteration_was_incremented: bool = False # Which uniform config fields this harness honors. No default: registration # rejects a class that does not declare one. @@ -95,96 +59,10 @@ def __init__( self.route = route self.cost_log_tags = cost_log_tags - def _begin_turn(self) -> None: - """Mark the start of a ``communicate()`` turn: reset the pending slot and - bump the iteration counter so a mid-turn failure can be rolled back. - - Call once at the top of every ``communicate()`` implementation. - """ - self.pending_turn = None - self._iteration += 1 - self._iteration_was_incremented = True - - def _end_turn_ok(self) -> None: - """Mark a turn as cleanly completed so its iteration bump stands. - - Call on the success path of ``communicate()`` (before returning). - """ - self._iteration_was_incremented = False - def _mark_stopped(self) -> None: - """Common ``stop()`` tail: clear the pending slot and enter FINISHED. - - Subclasses call this after their own resource teardown. - """ - self.pending_turn = None + """Common ``stop()`` tail: enter FINISHED. Subclasses call this after their own teardown.""" self._state = AgentState.FINISHED - # --- Shared mid-turn failure kernels -------------------------------------- - # - # Each agent keeps its OWN outer try/except/finally bracket -- the brackets - # genuinely differ -- and calls these from inside its existing branches. They - # take the agent's own ``finalize`` callable, so the helper never needs to know - # how each agent assembles its end-event payload. - - def _finalize_and_raise_timeout( - self, finalize: _FinalizeFn, timeout: float, *, cause: BaseException | None = None - ) -> NoReturn: - """Mark ERROR, finalize the turn as a timed-out crash, raise TurnTimeoutError. - - Reproduces the per-branch ``_state=ERROR -> finalize(TIMEOUT) -> raise`` triple - that appears three times in Claude plus once in Codex. When called from inside - an ``except ... as e`` block, pass ``cause=e`` to preserve the explicit - ``__cause__`` link; otherwise Python's implicit ``__context__`` chaining stands. - """ - self._state = AgentState.ERROR - finalize(AgentEndStatus.TIMEOUT, crashed=True, crash_reason=format_timeout_reason(timeout)) - if cause is not None: - raise TurnTimeoutError(timeout, iteration=self._iteration) from cause - raise TurnTimeoutError(timeout, iteration=self._iteration) - - def _finalize_and_raise_crash( - self, finalize: _FinalizeFn, message: str, *, cause: BaseException | None = None - ) -> NoReturn: - """Mark ERROR, finalize the turn as a crash, raise AgentCrashError. - - ``message`` is the agent-built error string (the helper does NOT construct - it). ``crash_reason`` is truncated for storage while the raised - ``AgentCrashError`` carries ``message`` as passed (truncation is idempotent, - so an already-truncated message round-trips unchanged). When called from - inside an ``except ... as e`` block, pass ``cause=e`` to preserve the explicit - ``__cause__`` link; otherwise Python's implicit ``__context__`` chaining stands. - """ - self._state = AgentState.ERROR - finalize(AgentEndStatus.CRASHED, crashed=True, crash_reason=truncate_crash_message(message)) - if cause is not None: - raise AgentCrashError(message) from cause - raise AgentCrashError(message) - - def _finalize_external_cancel(self, finalize: _FinalizeFn) -> None: - """Finalize a turn cancelled from outside (the task watchdog) as a crash. Does NOT raise. - - Only the ``crashed`` branch parks the record on ``pending_turn``; finalizing - as ``COMPLETED`` drops it, and the unwinding frame takes the return value - with it, so a killed turn's telemetry survives only via this path. The caller - re-raises the ``CancelledError`` afterwards. - """ - self._state = AgentState.ERROR - finalize(AgentEndStatus.CRASHED, crashed=True, crash_reason="turn cancelled") - - def _capture_partial_turn(self, collector: EventCollector) -> None: - """Build the crashed partial ``TurnRecord`` into ``pending_turn`` (best-effort). - - Shared crash-tail of each agent's ``finalize``: if assembling the partial - record itself raises, swallow it and leave ``pending_turn`` None rather than - masking the original mid-turn failure. - """ - try: - self.pending_turn = collector.build_turn_record() - except Exception: - logger.exception("Failed to build partial turn record") - self.pending_turn = None - @abstractmethod async def start( self, @@ -206,8 +84,10 @@ async def start( instead of walking up from CWD. An external ``PLUGIN_TOOLS_DIR`` in the process environment still wins. Implementations that don't shell out may ignore this argument. - plugin_root: The staged canonical plugin root (``/skills//SKILL.md``), - or None when the task sets no plugins. Deliver it the harness's native way. + plugin_root: The staged plugin root, or None when the task sets no plugins. It holds + ``/skills//SKILL.md`` (every harness) and ``/plugins/`` (each + authored plugin whole, for a harness that loads full plugins). Deliver it the + harness's native way. """ pass @@ -216,51 +96,65 @@ async def communicate( self, user_input: str, *, + iteration: int, stream_callback: StreamCallback | None = None, timeout: float | None = None, should_stop: Callable[[], StopReason | None] | None = None, - ) -> TurnRecord: - """Send a message to the agent and receive its response. + ) -> TurnOutcome: + """Run one turn and return its outcome; a crash or timeout is an outcome, not an exception. Args: user_input: The message/prompt to send to the agent + iteration: The caller's turn number, stamped on the record; a retry of + the same turn passes the same number. stream_callback: Optional callback for real-time event streaming - timeout: Hard wall-clock deadline in seconds. When exceeded the - agent must force-terminate any in-flight subprocess and raise - TurnTimeoutError. Do not rely solely on asyncio cancellation -- - some SDKs swallow it. + timeout: Hard wall-clock deadline in seconds. When exceeded the agent + force-terminates any in-flight subprocess and returns a ``TIMEOUT`` + outcome. Do not rely solely on asyncio cancellation -- some SDKs + swallow it. should_stop: The run's single stop poll. An implementation with ``contract.cooperative_stop`` calls it at each safe boundary; a - non-None reason means stop pulling work, remember the reason, and - finalize with ``end_status_for(reason)`` (``crashed=False``, no - raise). Agents that do not support it accept and ignore it. + non-None reason means stop pulling work and finalize with + ``end_status_for(reason)``. Agents that do not support it ignore it. Returns: - TurnRecord containing the complete interaction + The ``TurnOutcome`` from the turn's ``TurnEmitter``: ``finalize(...)`` for a + clean status, ``fail(...)`` for ``CRASHED`` / ``TIMEOUT`` (its record is + ``crashed=True``). Raises: - RuntimeError: If agent is not started or communication fails. - TurnTimeoutError: Timeout elapsed; implementations must set - ``self.pending_turn`` to a ``crashed=True`` partial TurnRecord - before raising if telemetry was captured. - AgentCrashError: Agent failed mid-turn; same ``pending_turn`` contract. - - On success ``pending_turn`` must be None. On failure it holds the partial - record, and only ``discard_pending_turn`` — which the caller invokes after - every failed call — rolls back per-turn bookkeeping. - - The agent is the SOLE emitter of the event protocol. Emit exactly one - ``AgentStartEvent`` at entry and one matching ``AgentEndEvent`` from - ``finally`` on every exit path, one ``TurnStartEvent`` / ``TurnEndEvent`` - pair per inner turn, and a ``ToolStartEvent`` closed by a ``ToolEndEvent`` - for every tool call (``status=unresolved`` when a crash orphans one). Fan - every event through an internal ``EventCollector``, which builds the - returned ``TurnRecord``, and through the caller's ``stream_callback``. + asyncio.CancelledError: the turn was cancelled from outside. The agent + ends the turn first with ``fail(CRASHED, "turn cancelled")``, then + re-raises. Any other exception is a harness bug. + + Open one ``TurnEmitter`` per turn with ``_open_emitter``; it is the sole writer + of the event protocol. Rationale: .claude/notes/agents.md § Shared turn lifecycle """ pass + def _open_emitter( + self, + *, + prompt: str, + iteration: int, + model: str | None, + task_id: str, + stream_callback: StreamCallback | None, + ) -> TurnEmitter: + """The turn's emitter, on a fresh ``TurnClock`` or the wall clock per ``contract.timing_basis``.""" + clock: Clock = TurnClock() if self.contract.timing_basis is TimingBasis.TURN_CLOCK else datetime + return TurnEmitter( + task_id=task_id, + iteration=iteration, + prompt=prompt, + model=model, + basis=self.contract.timing_basis, + clock=clock, + sinks=[stream_callback] if stream_callback is not None else [], + ) + @abstractmethod async def stop(self) -> None: """Stop the agent and clean up resources.""" @@ -275,24 +169,6 @@ async def kill(self) -> None: """ return None - async def discard_pending_turn(self) -> None: - """Clear ``pending_turn`` and roll back the iteration counter. - - Rolls back when either signal says a turn was attempted: the - ``_iteration_was_incremented`` flag (survives partial-record assembly - swallowing an exception, which leaves ``pending_turn=None`` — so the - flag, not ``pending_turn``, is the reliable signal) or a non-None - ``pending_turn`` (for callers, e.g. tests, that set it directly). - - Idempotent: after the first call both signals are cleared. Call only - after a failed ``communicate()``; never after a success. - """ - should_rollback = self._iteration_was_incremented or self.pending_turn is not None - self.pending_turn = None - self._iteration_was_incremented = False - if should_rollback and self._iteration > 0: - self._iteration -= 1 - def kill_sync(self) -> None: """Synchronous variant of ``kill`` for callers on non-asyncio threads. @@ -319,6 +195,14 @@ def get_sdk_options(self) -> dict[str, Any] | None: """ return None + async def harness_version(self) -> str | None: + """The version of the CLI or SDK this agent drives, read after ``start``; ``None`` when unknown. + + The orchestrator records it as ``environment_info.harness_version``. It runs where + the agent runs, so under ``driver: docker`` it reads the container's harness. + """ + return None + def get_environment_info(self) -> dict[str, Any]: """Agent-specific routing/environment details to persist into the run's ``EvaluationResult.environment_info``. @@ -341,3 +225,31 @@ def get_environment_info(self) -> dict[str, Any]: "system_prompt_semantics": self.contract.system_prompt_semantics or "unknown", "harness_contract": self.contract.model_dump(mode="json"), } + + +_VERSION_PROBE_SECONDS = 10.0 +_VERSION_PROBE_LIMIT_BYTES = 1024 * 1024 + + +async def command_version(argv: list[str], env: dict[str, str] | None = None) -> str | None: + """The first non-empty stdout line of ``argv`` (a ``--version`` call), or ``None`` on any failure.""" + try: + proc = await asyncio.create_subprocess_exec( + *argv, + stdin=asyncio.subprocess.DEVNULL, + stdout=asyncio.subprocess.PIPE, + stderr=asyncio.subprocess.DEVNULL, + env=env, + limit=_VERSION_PROBE_LIMIT_BYTES, + ) + except OSError: + return None + try: + stdout, _ = await asyncio.wait_for(proc.communicate(), timeout=_VERSION_PROBE_SECONDS) + except TimeoutError: + with contextlib.suppress(ProcessLookupError): + proc.kill() + return None + if proc.returncode != 0: + return None + return next((line.strip() for line in stdout.decode("utf-8", "replace").splitlines() if line.strip()), None) diff --git a/src/coder_eval/agents/_transport/__init__.py b/src/coder_eval/agents/_transport/__init__.py new file mode 100644 index 00000000..293d4599 --- /dev/null +++ b/src/coder_eval/agents/_transport/__init__.py @@ -0,0 +1,6 @@ +"""Transport bases shared by more than one harness adapter.""" + +from coder_eval.agents._transport.subprocess_jsonl import JsonlDecoder, SubprocessJsonlAgent + + +__all__ = ["JsonlDecoder", "SubprocessJsonlAgent"] diff --git a/src/coder_eval/agents/_transport/subprocess_jsonl.py b/src/coder_eval/agents/_transport/subprocess_jsonl.py new file mode 100644 index 00000000..6bf71c40 --- /dev/null +++ b/src/coder_eval/agents/_transport/subprocess_jsonl.py @@ -0,0 +1,379 @@ +"""``SubprocessJsonlAgent``: one CLI invocation per turn, its stdout read as nd-JSON events. + +The base owns the transport: the spawn, the concurrent stderr drain, the read loop +racing each line against exit and the turn deadline, the cooperative stop, the +settle that decides how the turn ended, and every reap. A subclass supplies the +argv, the environment and a ``JsonlDecoder`` that turns one event into +``TurnEmitter`` calls. + +Rationale: .claude/notes/agents.md § Reaping the CLI harnesses +""" + +from __future__ import annotations + +import asyncio +import contextlib +import json +import logging +import os +import signal +import time +from abc import ABC, abstractmethod +from collections.abc import Callable +from typing import Any, ClassVar + +from coder_eval.agent import Agent, command_version +from coder_eval.errors.agent import format_timeout_reason +from coder_eval.isolation.docker_runner import STDOUT_LINE_LIMIT_BYTES +from coder_eval.models import AgentState, ApiRoute, BaseAgentConfig +from coder_eval.streaming.callbacks import StreamCallback +from coder_eval.streaming.emitter import TurnEmitter, TurnOutcome +from coder_eval.streaming.events import AgentEndStatus, StopReason, end_status_for + + +logger = logging.getLogger(__name__) + +# Grace between SIGTERM and SIGKILL. It must stay below the orchestrator's backstop grace, +# or a slow SIGTERM turns a TIMEOUT into a cancelled crash. +KILL_GRACE_SECONDS = 1.0 + +# How long a CLI that closed its event stream may take to exit when no deadline is set. +_EXIT_GRACE_SECONDS = 5.0 + +# How often exit is polled: `Process.wait()` resolves only once every pipe closes. +_EXIT_POLL_SECONDS = 0.05 + +# SIGKILL does not exist on Windows (where the process-group sweep is a no-op). +_SIGKILL: signal.Signals = getattr(signal, "SIGKILL", signal.SIGTERM) + +# How long to keep reading after the CLI exited: a child can hold the pipes open. +_DRAIN_SECONDS = 2.0 + +# How many distinct unrecognized event types the drift crash names. +_MAX_UNRECOGNIZED_TYPES = 8 + + +class JsonlDecoder(ABC): + """One turn's reducer: one decoded nd-JSON event in, ``TurnEmitter`` calls out. + + ``error`` is a terminal CLI or provider error the stream reported; the base + crashes the turn on it. When ``error_survives_stop`` is False, a requested stop + wins: the stream can clear its error later, so a cut may land on a stale one. + """ + + error_survives_stop: ClassVar[bool] = False + + def __init__(self, emitter: TurnEmitter) -> None: + self.emitter = emitter + self.error: str | None = None + + @abstractmethod + def __call__(self, event: dict[str, Any]) -> None: + """Reduce one event; never raises on unexpected payload shapes.""" + + @abstractmethod + def end(self, status: AgentEndStatus, *, reason: str | None = None) -> TurnOutcome: + """End the turn: ``emitter.fail(status, reason)`` for CRASHED / TIMEOUT, else ``emitter.finalize``.""" + + +class SubprocessJsonlAgent[ConfigT: BaseAgentConfig](Agent[ConfigT]): + """An adapter whose turn is one CLI process streaming nd-JSON on stdout.""" + + cli_name: ClassVar[str] + executable: ClassVar[str] + docs_page: ClassVar[str] + recognized_events: ClassVar[frozenset[str]] + decoder: ClassVar[type[JsonlDecoder]] + + def __init__( + self, + config: ConfigT, + route: ApiRoute | None = None, + *, + task_id: str = "unknown", + cost_log_tags: dict[str, str] | None = None, + ) -> None: + super().__init__(config, route, cost_log_tags=cost_log_tags) + self.task_id = task_id + self.working_directory: str | None = None + self._process: asyncio.subprocess.Process | None = None + # Process-group ids of every invocation this agent spawned, swept on + # kill()/kill_sync()/stop(). + self._spawned_pgids: list[int] = [] + self._state = AgentState.WORKING + + @abstractmethod + def argv(self, prompt: str) -> list[str]: + """The CLI command line for one turn; ``prompt`` is a distinct argv element.""" + + @abstractmethod + def env(self) -> dict[str, str]: + """The CLI's whole environment.""" + + async def harness_version(self) -> str | None: + """`` --version`` under the turn's environment.""" + return await command_version([self.executable, "--version"], env=self.env()) + + def observe(self, event: dict[str, Any]) -> None: + """See every decoded event before the decoder does (session ids, for example).""" + return None + + def clean_exit_problem(self, decoder: JsonlDecoder) -> str | None: + """Why a clean, uncut exit that recognized events must still crash, or None.""" + return None + + async def communicate( + self, + user_input: str, + *, + iteration: int, + stream_callback: StreamCallback | None = None, + timeout: float | None = None, + should_stop: Callable[[], StopReason | None] | None = None, + ) -> TurnOutcome: + """Run one CLI invocation as one turn; see ``Agent.communicate``.""" + if self.working_directory is None: + raise RuntimeError(f"{type(self).__name__}.start() must be called before communicate()") + + emitter = self._open_emitter( + prompt=user_input, + iteration=iteration, + model=self.config.model, + task_id=self.task_id, + stream_callback=stream_callback, + ) + emitter.begin() + decoder = self.decoder(emitter) + vocabulary = _Vocabulary() + # Deadlines stay on `time.monotonic()`: a deadline must not move when the wall clock steps. + deadline = None if timeout is None else time.monotonic() + timeout + requested_stop: StopReason | None = None + stderr_drain: asyncio.Future[bytes] | None = None + # Bound OUTSIDE the try so `finally` can tell "never spawned" from "spawned". + proc: asyncio.subprocess.Process | None = None + try: + proc = await asyncio.create_subprocess_exec( + *self.argv(user_input), + # A CLI that reads a non-TTY stdin to EOF stalls on an inherited open one. + # Rationale: .claude/notes/agents.md § Why a CLI never inherits stdin + stdin=asyncio.subprocess.DEVNULL, + stdout=asyncio.subprocess.PIPE, + stderr=asyncio.subprocess.PIPE, + cwd=self.working_directory, + env=self.env(), + # One nd-JSON event can carry a whole tool result, past the 64 KiB default. + limit=STDOUT_LINE_LIMIT_BYTES, + # Own process group, so teardown can killpg a lingering child. + start_new_session=os.name == "posix", + ) + self._process = proc + if os.name == "posix": + self._spawned_pgids.append(proc.pid) + assert proc.stdout is not None + # Drained CONCURRENTLY, or a child that fills the pipe hangs the turn. + if proc.stderr is not None: + stderr_drain = asyncio.ensure_future(proc.stderr.read()) + + exit_waiter = asyncio.ensure_future(_exited(proc)) + read_task: asyncio.Future[bytes] | None = None + try: + while True: + remaining = None if deadline is None else deadline - time.monotonic() + if remaining is not None and remaining <= 0: + return await self._time_out(decoder, timeout or 0.0) + if read_task is None: + read_task = asyncio.ensure_future(proc.stdout.readline()) + done, _pending = await asyncio.wait( + {read_task, exit_waiter}, timeout=remaining, return_when=asyncio.FIRST_COMPLETED + ) + if not done: + return await self._time_out(decoder, timeout or 0.0) + if not read_task.done(): + # Exited with the read pending: bound the tail, a child may hold the pipe. + drain = _DRAIN_SECONDS if deadline is None else min(_DRAIN_SECONDS, deadline - time.monotonic()) + try: + await asyncio.wait_for(asyncio.shield(read_task), max(0.0, drain)) + except TimeoutError: + break + line = read_task.result() + read_task = None + if not line: + break + self._handle_line(line, decoder, vocabulary) + requested_stop = should_stop() if should_stop is not None else None + if requested_stop is not None: + await self.kill() + break + finally: + if read_task is not None: + read_task.cancel() + exit_waiter.cancel() + + return await self._settle( + proc, + decoder, + vocabulary, + stderr_drain, + requested_stop=requested_stop, + deadline=deadline, + timeout=timeout, + ) + except asyncio.CancelledError: + self._state = AgentState.ERROR + decoder.end(AgentEndStatus.CRASHED, reason="turn cancelled") + raise + except Exception as e: + # A spawn failure, a StreamReader ValueError past `limit`, a malformed payload. + logger.warning("%s: turn failed", self.cli_name.lower(), exc_info=True) + return self._crash(decoder, f"{self.cli_name} turn failed: {e!s}") + finally: + if stderr_drain is not None: + stderr_drain.cancel() + self._reap(proc) + self._process = None + + async def _settle( + self, + proc: asyncio.subprocess.Process, + decoder: JsonlDecoder, + vocabulary: _Vocabulary, + stderr_drain: asyncio.Future[bytes] | None, + *, + requested_stop: StopReason | None, + deadline: float | None, + timeout: float | None, + ) -> TurnOutcome: + """Reap the CLI once the read loop is done and end the turn. + + In order: no exit by the deadline is TIMEOUT (without one, CRASHED after the + grace); a stream error is CRASHED (unless a stop was requested and the decoder's + error does not survive one); a requested stop ends with its status; a non-zero + exit, no recognized event, or a subclass's ``clean_exit_problem`` is CRASHED; + else COMPLETED. + + Rationale: .claude/notes/agents.md § Why a clean exit can still be a crash + """ + remaining = None if deadline is None else max(0.0, deadline - time.monotonic()) + try: + if proc.returncode is None: + await asyncio.wait_for(_exited(proc), timeout=_EXIT_GRACE_SECONDS if remaining is None else remaining) + except TimeoutError: + if remaining is not None: + return await self._time_out(decoder, timeout or 0.0) + await self.kill() + return self._crash( + decoder, f"{self.cli_name} closed its event stream but did not exit within {_EXIT_GRACE_SECONDS:.0f}s" + ) + stderr_bytes = b"" + if stderr_drain is not None: + with contextlib.suppress(TimeoutError): + stderr_bytes = await asyncio.wait_for(asyncio.shield(stderr_drain), timeout=_DRAIN_SECONDS) + + if decoder.error is not None and (requested_stop is None or decoder.error_survives_stop): + return self._crash(decoder, f"{self.cli_name} error: {decoder.error}") + if requested_stop is not None: + return decoder.end(end_status_for(requested_stop)) + if proc.returncode not in (0, None): + detail = stderr_bytes.decode("utf-8", "replace").strip() or f"exit code {proc.returncode}" + return self._crash(decoder, f"{self.cli_name} exited non-zero: {detail}") + if vocabulary.recognized == 0: + seen = ", ".join(sorted(vocabulary.unrecognized)) or "none (stdout carried no JSON events)" + return self._crash( + decoder, + f"{self.cli_name} exited cleanly but the turn captured no recognized events. Unrecognized event " + + f"types seen: {seen}. The CLI's event schema may have changed — see {self.docs_page} before " + + "trusting any run from this CLI version.", + ) + problem = self.clean_exit_problem(decoder) + if problem is not None: + return self._crash(decoder, problem) + return decoder.end(AgentEndStatus.COMPLETED) + + def _crash(self, decoder: JsonlDecoder, message: str) -> TurnOutcome: + self._state = AgentState.ERROR + return decoder.end(AgentEndStatus.CRASHED, reason=message) + + async def _time_out(self, decoder: JsonlDecoder, timeout: float) -> TurnOutcome: + await self.kill() + self._state = AgentState.ERROR + return decoder.end(AgentEndStatus.TIMEOUT, reason=format_timeout_reason(timeout)) + + def _handle_line(self, line: bytes, decoder: JsonlDecoder, vocabulary: _Vocabulary) -> None: + """Parse one line and hand a JSON object to ``observe`` and the decoder; skip anything else.""" + raw = line.decode("utf-8", "replace").strip() + if not raw: + return + try: + event = json.loads(raw) + except json.JSONDecodeError: + logger.debug("%s: skipping non-JSON stdout line: %s", self.cli_name.lower(), raw[:200]) + return + if not isinstance(event, dict): + return + event_type = str(event.get("type") or "") + if event_type in self.recognized_events: + vocabulary.recognized += 1 + elif len(vocabulary.unrecognized) < _MAX_UNRECOGNIZED_TYPES: + vocabulary.unrecognized.add(event_type or "") + self.observe(event) + decoder(event) + + async def kill(self) -> None: + """SIGTERM the in-flight CLI, SIGKILL it after the grace, then sweep its process groups.""" + proc = self._process + if proc is not None and proc.returncode is None: + with contextlib.suppress(ProcessLookupError): + proc.terminate() + with contextlib.suppress(TimeoutError, asyncio.TimeoutError): + await asyncio.wait_for(_exited(proc), timeout=KILL_GRACE_SECONDS) + if proc.returncode is None: + with contextlib.suppress(ProcessLookupError): + proc.kill() + self._sweep_process_groups() + + def kill_sync(self) -> None: + """SIGKILL the in-flight CLI and its process groups (watchdog thread; must not await).""" + proc = self._process + if proc is not None and proc.returncode is None: + with contextlib.suppress(ProcessLookupError, PermissionError): + os.kill(proc.pid, _SIGKILL) + self._sweep_process_groups() + + def _sweep_process_groups(self) -> None: + """SIGKILL every process group this agent spawned (POSIX only); each holds one invocation's children.""" + if os.name != "posix": + return + for pgid in self._spawned_pgids: + with contextlib.suppress(ProcessLookupError, PermissionError, OSError): + os.killpg(pgid, _SIGKILL) + self._spawned_pgids.clear() + + def _reap(self, proc: asyncio.subprocess.Process | None) -> None: + """Kill a CLI still running as the turn unwinds, then sweep the turn's process groups. + + Synchronous, so it survives a cancel. The sweep runs after a clean exit too: a child + the turn left behind must not outlive it. + """ + if proc is not None and proc.returncode is None: + with contextlib.suppress(ProcessLookupError, PermissionError): + proc.kill() + self._sweep_process_groups() + + +async def _exited(proc: asyncio.subprocess.Process) -> int | None: + """Resolve when the process exits, even while a child still holds its pipes.""" + waiter = asyncio.ensure_future(proc.wait()) + try: + while not waiter.done() and proc.returncode is None: + await asyncio.wait({waiter}, timeout=_EXIT_POLL_SECONDS) + return proc.returncode + finally: + waiter.cancel() + + +class _Vocabulary: + """The drift check's evidence: how many events matched, and a sample of the types that did not.""" + + def __init__(self) -> None: + self.recognized = 0 + self.unrecognized: set[str] = set() diff --git a/src/coder_eval/agents/antigravity_agent.py b/src/coder_eval/agents/antigravity_agent.py index 7e82f417..fdb863bc 100644 --- a/src/coder_eval/agents/antigravity_agent.py +++ b/src/coder_eval/agents/antigravity_agent.py @@ -29,50 +29,30 @@ from coder_eval.agent import Agent, AgentState from coder_eval.agents._logging import PrefixedAdapter -from coder_eval.agents.registry import AgentRegistry -from coder_eval.agents.watchdog import ThreadedWatchdog +from coder_eval.agents.registry import SPI_VERSION, AgentRegistry +from coder_eval.agents.watchdog import WatchdogFired, run_with_watchdog from coder_eval.config import settings -from coder_eval.errors import ( - AgentCrashError, - TurnTimeoutError, - truncate_crash_message, -) +from coder_eval.errors.agent import format_timeout_reason from coder_eval.models import ( READ_ONLY_DENIED_TOOLS, AgentKind, AntigravityAgentConfig, ApiRoute, - AssistantMessage, - CommandTelemetry, ContentBlock, DirectRoute, Enforcement, HarnessContract, PermissionMode, + TimingBasis, TokenUsage, ToolNameMap, - TranscriptMessage, - TurnRecord, UsageGranularity, ) -from coder_eval.pricing import calculate_cost -from coder_eval.streaming.callbacks import CompositeStreamCallback, StreamCallback -from coder_eval.streaming.collector import EventCollector -from coder_eval.streaming.events import ( - AgentEndEvent, - AgentEndStatus, - AgentStartEvent, - StopReason, - TextChunkEvent, - ToolEndEvent, - ToolEndStatus, - ToolStartEvent, - TurnEndEvent, - TurnEndStatus, - TurnStartEvent, - end_status_for, -) -from coder_eval.timing import TurnClock, close_window +from coder_eval.pricing import price_turn +from coder_eval.streaming.callbacks import StreamCallback +from coder_eval.streaming.emitter import Generation, TurnEmitter, TurnOutcome +from coder_eval.streaming.events import AgentEndStatus, StopReason, ToolEndStatus, TurnEndStatus, end_status_for +from coder_eval.timing import close_window logger = logging.getLogger(__name__) @@ -182,23 +162,251 @@ def _to_token_usage(usage: Any, model: str | None) -> TokenUsage: thoughts = getattr(usage, "thoughts_token_count", 0) or 0 uncached_input = max(prompt - cached, 0) output = candidates + thoughts - cost = calculate_cost(model, uncached_input, output, 0, cached) if model else None - return TokenUsage( + tokens = TokenUsage( uncached_input_tokens=uncached_input, output_tokens=output, cache_creation_input_tokens=0, cache_read_input_tokens=cached, - total_cost_usd=cost, ) + tokens.total_cost_usd = price_turn(tokens, (model,)) + return tokens + + +class _AntigravityDecoder: + """One turn's reducer: Antigravity ``Step`` objects in, ``TurnEmitter`` calls out. + + Step-stream shape (observed): each ``step_index`` is yielded repeatedly through + ACTIVE -> DONE transitions; ``usage_metadata`` lands once per generation on a + DONE/terminal step (summing them == the turn total); a tool call carries a stable + ``id`` and its result is folded into expanded ``args`` at DONE. Each generation + is one inner turn: opened at the first MODEL Step that brings new content or an + unseen tool call, closed at its ``usage_metadata`` with that usage as its tokens. + """ + + def __init__(self, emitter: TurnEmitter, *, turn_id: str = "antigravity-1") -> None: + self.emitter = emitter + self.turn_id = turn_id + self.timeout_hit = False + self.stop_reason: StopReason | None = None + self.total_usage = TokenUsage() + self.output_parts: list[str] = [] + self.generations = 0 + self._seen_tools: set[str] = set() + self._closed_tools: set[str] = set() + # Arg keys present when a tool was first seen (its model-supplied inputs), + # used at DONE to tell them from harness-appended result fields. + self._tool_input_keys: dict[str, set[str]] = {} + self._tool_names: dict[str, str] = {} + # Most recently seen StepStatus per tool id, for has_orphaned_tool_call. + self._tool_last_status: dict[str, Any] = {} + # Content blocks accumulated since the last per-generation flush. + self._blocks: list[ContentBlock] = [] + # Where the CURRENT generation started, advanced only by a flush that + # actually added a message. + self._gen_mark: datetime = emitter.now() + # Re-seeded ONCE, at the first observed MODEL Step. + self._first_output_seen = False + + @property + def ended_cleanly(self) -> bool: + """True once the loop broke on a ``should_stop`` reason: a later exception is not a crash.""" + return self.stop_reason is not None + + def _seed_first_generation_window(self, source: Any) -> None: + """Move the first window's mark to the first observed MODEL output, once per turn. + + Gated on ``source``: a turn can open with a SYSTEM or USER Step, and seeding + on one would put the mark before the model spoke. + + Rationale: .claude/notes/agents.md § First-generation window seeding + """ + if self._first_output_seen or _enum_value(source) != _SOURCE_MODEL: + return + self._first_output_seen = True + self._gen_mark = self.emitter.now() + + @staticmethod + def _is_reply(stype: Any, ssource: Any, starget: Any) -> bool: + """The ONE test for "the model talking to the user"; a USER-source prompt echo is not.""" + return stype == _TYPE_TEXT_RESPONSE and ssource == _SOURCE_MODEL and starget == _TARGET_USER + + def _open_generation(self) -> None: + """Open the inner turn for the generation now arriving; a no-op while one is open.""" + if not self.emitter.inner_turn_open: + self.emitter.begin_inner_turn(f"{self.turn_id}-msg-{self.generations}") + + def _starts_generation(self, step: Any, ssource: Any) -> bool: + """True when ``step`` is the model speaking: new content, new usage, or a tool call not seen before. + + A DONE Step for a call already open is its result landing, not a model turn. + """ + if ssource != _SOURCE_MODEL: + return False + if step.tool_calls: + return any(self._call_id(call, step, i) not in self._seen_tools for i, call in enumerate(step.tool_calls)) + return bool( + step.thinking + or step.thinking_delta + or step.content + or step.content_delta + or step.usage_metadata is not None + ) + + def __call__(self, step: Any) -> None: + """Route one streamed ``Step`` to the emitter.""" + stype = _enum_value(step.type) + sstatus = _enum_value(step.status) + ssource = _enum_value(step.source) + self._seed_first_generation_window(ssource) + if self._starts_generation(step, ssource): + self._open_generation() + starget = _enum_value(step.target) + done = sstatus in (_STATUS_DONE, _STATUS_ERROR) + reply = self._is_reply(stype, ssource, starget) + + if step.content_delta and reply: + self.emitter.text(step.content_delta) + + for call_index, call in enumerate(step.tool_calls): + self._handle_tool_call(call, step, done, sstatus, call_index) + + if done: + if stype == _TYPE_THINKING and step.thinking: + self._blocks.append(ContentBlock(block_type="thinking", sequence=0, thinking=step.thinking)) + elif reply and step.content: + self.output_parts.append(step.content) + self._blocks.append(ContentBlock(block_type="text", sequence=0, text=step.content)) + + if step.usage_metadata is not None: + gen = _to_token_usage(step.usage_metadata, self.emitter.model) + self.total_usage = self.total_usage + gen + self._flush_generation(gen, getattr(step.usage_metadata, "thoughts_token_count", 0) or 0) + + @staticmethod + def _call_id(call: Any, step: Any, call_index: int) -> str: + # call.id is usually present but the SDK types it optional. The fallback + # mirrors the SDK's own `trajectory_id:step_index` scheme; call_index + # further disambiguates multiple id-less calls within one step. + # Rationale: .claude/notes/agents.md § Why the tool-call id falls back the way it does + trajectory_id = getattr(step, "trajectory_id", "") or "" + step_key = f"{trajectory_id}:{step.step_index}" if trajectory_id else str(step.step_index) + return call.id or f"{_enum_value(call.name)}_{step_key}_{call_index}" + + def _handle_tool_call(self, call: Any, step: Any, done: bool, sstatus: Any, call_index: int) -> None: + raw_name = _enum_value(call.name) + cid = self._call_id(call, step, call_index) + self._tool_last_status[cid] = sstatus + if cid not in self._seen_tools: + self._seen_tools.add(cid) + tool_name = _ANTIGRAVITY_TO_CLAUDE_TOOL_MAP.get(raw_name, str(raw_name)) + self._tool_names[cid] = tool_name + self._tool_input_keys[cid] = set(call.args) + self.emitter.open_tool(cid, tool_name, self._params(tool_name, call.args, self._tool_input_keys[cid])) + + if done and cid not in self._closed_tools: + self._closed_tools.add(cid) + exit_code = call.args.get("exit_code") + errored = (sstatus == _STATUS_ERROR) or (exit_code not in (None, 0)) + result_text = ( + call.args.get("combined_output") + or call.args.get("diff_block") + or call.args.get("output") + or call.args.get("results") + or call.args.get("summary") + or step.content + or None + ) + tool_name = self._tool_names[cid] + self.emitter.close_tool( + cid, + status=ToolEndStatus.ERROR if errored else ToolEndStatus.OK, + summary=str(result_text) if result_text is not None else None, + error=(step.error or "tool failed") if errored else None, + parameters=self._params(tool_name, call.args, self._tool_input_keys.get(cid)), + ) + self._blocks.append(ContentBlock(block_type="tool_use", sequence=0, tool_use_id=cid)) + + @staticmethod + def _params(tool_name: str, args: dict[str, Any], input_keys: set[str] | None) -> dict[str, Any]: + """Model-supplied inputs only, renamed to canonical cross-agent keys. + + A key is a (dropped) result field when it is in the static + ``_RESULT_ARG_KEYS`` backstop OR first appeared at DONE, given the + input-key snapshot taken at tool start. Survivors are renamed to the + canonical vocabulary. + """ + rename = _ANTIGRAVITY_ARG_RENAME.get(tool_name, {}) + out: dict[str, Any] = {} + for k, v in args.items(): + if k in _RESULT_ARG_KEYS: + continue + if input_keys is not None and k not in input_keys: + continue # appeared only at DONE → harness result payload + out[rename.get(k, k)] = v + return out + + def _flush_generation( + self, gen: TokenUsage, reasoning_tokens: int, *, status: TurnEndStatus = TurnEndStatus.COMPLETED + ) -> None: + """Cut the accumulated blocks into one generation carrying this step's tokens, and close its inner turn.""" + if not self._blocks and gen.is_empty(): + return + self._open_generation() + now = self.emitter.now() + # Do NOT "simplify" this to resetting the mark when a tool ends: this + # harness interleaves a tool INTO a window rather than tiling around it, + # so the RAW window legitimately contains time that is not model time. + # Rationale: .claude/notes/agents.md § Per-harness generation marks + for i, block in enumerate(self._blocks): + block.sequence = i + self.emitter.add_generation( + # The Step stream carries no message id; one per generation. + message_id=f"{self.turn_id}-msg-{self.generations}", + window=close_window(mark=self._gen_mark, now=now), + parts=[Generation(blocks=list(self._blocks), tokens=gen, reasoning_tokens=reasoning_tokens)], + ) + self.generations += 1 + self._blocks = [] + self._gen_mark = now + self.emitter.end_inner_turn(status, tokens=gen) + + def has_orphaned_tool_call(self) -> bool: + """True if any NOT-YET-CLOSED tool call's most recently seen status is + ACTIVE — the structural signature of a backgrounded task the model went + idle on without waiting for. See ``AntigravityAgent._poll_background_work``. + + An ALLOWLIST on ACTIVE, never a denylist on "not yet closed": the SDK also + has WAITING_FOR_USER, CANCELED and UNKNOWN, none of which the poll loop + should wait out. The `not in _closed_tools` guard is layered on top as a + monotonicity backstop, not a substitute. + + Rationale: .claude/notes/agents.md § Antigravity Step interleaving and the background poll + """ + return any(cid not in self._closed_tools and s == _STATUS_ACTIVE for cid, s in self._tool_last_status.items()) + + def end(self, status: AgentEndStatus, *, reason: str | None = None, agent_output: str | None = None) -> TurnOutcome: + """Flush trailing blocks as an unbilled generation and end the turn. + + Every billed generation already closed its own inner turn; the emitter closes + one still open with ``status``. ``agent_output`` is the fallback when the + stream carried no reply text. + """ + if self._blocks: + self._flush_generation(TokenUsage(), 0, status=TurnEndStatus(status.value)) + output = "".join(self.output_parts) if self.output_parts else (agent_output or "") + if status is AgentEndStatus.CRASHED or status is AgentEndStatus.TIMEOUT: + return self.emitter.fail(status, reason or status.value, usage=self.total_usage, agent_output=output) + return self.emitter.finalize(status, usage=self.total_usage, agent_output=output) -@AgentRegistry.register(AgentKind.ANTIGRAVITY, AntigravityAgentConfig) +@AgentRegistry.register(AgentKind.ANTIGRAVITY, AntigravityAgentConfig, spi_version=SPI_VERSION) class AntigravityAgent(Agent[AntigravityAgentConfig]): """Implementation of the Agent interface for Google Antigravity (Gemini).""" # The step loop has a between-steps guard where `should_stop` runs; # TemplatedSystemInstructions wraps system_instructions around the harness's # own prompt, and always has — so runs ARE comparable across the marker. + # Usage is one `usage_metadata` per model generation, each closing one inner turn. # Rationale: .claude/notes/agents.md § The system_prompt_semantics marker contract = HarnessContract( system_prompt=Enforcement.ENFORCED, @@ -208,7 +416,8 @@ class AntigravityAgent(Agent[AntigravityAgentConfig]): allowed_tools=Enforcement.ENFORCED, disallowed_tools=Enforcement.ENFORCED, cooperative_stop=True, - usage_granularity=UsageGranularity.TURN, + usage_granularity=UsageGranularity.GENERATION, + timing_basis=TimingBasis.TURN_CLOCK, permission_modes=frozenset({PermissionMode.PLAN, PermissionMode.BYPASS_PERMISSIONS}), ) tool_names = _TOOL_NAMES @@ -240,7 +449,6 @@ def __init__( # Dirs prepended to PATH so sandbox mock CLIs shadow real ones for the # harness's run_command tool (see _harness_env). self._env_path_prepend: list[str] = [] - # Turn-lifecycle bookkeeping lives on the Agent base class. self._log = PrefixedAdapter(logger, {"prefix": instance_name}) def _effective_model(self) -> str: @@ -372,10 +580,10 @@ async def start( async def _drain( self, conversation: Any, - state: "_AntigravityTurnState", + decoder: _AntigravityDecoder, should_stop: Callable[[], StopReason | None] | None, ) -> None: - """Consume one ``receive_steps()`` cycle onto ``state``, honoring a + """Consume one ``receive_steps()`` cycle into ``decoder``, honoring a cooperative stop mid-stream. Shared by the initial drain and each poll cycle's re-drain, so this shape lives in one place. @@ -394,10 +602,10 @@ async def _drain( async with contextlib.aclosing(conversation.receive_steps()) as steps: async for step in steps: pulled = True - state.process_step(step) + decoder(step) reason = should_stop() if should_stop is not None else None if reason is not None: - state.stop_reason = reason + decoder.stop_reason = reason self._log.debug("Stop requested (%s); ending step loop at this boundary", reason.value) break return @@ -414,181 +622,143 @@ async def communicate( self, user_input: str, *, + iteration: int, stream_callback: StreamCallback | None = None, timeout: float | None = None, should_stop: Callable[[], StopReason | None] | None = None, - ) -> TurnRecord: - """Send a message to the Antigravity agent and receive its response. - - ``should_stop`` is the run's stop poll, called after each processed step. - On a reason the step loop breaks, the conversation is cancelled - (best-effort) and the turn finalizes cleanly with ``end_status_for(reason)`` - (``crashed=False``). - - Drives one logical turn: ``conversation.send(prompt)`` then iterate - ``receive_steps()`` until the turn goes idle. + ) -> TurnOutcome: + """Send the prompt, drain the Step stream until the turn goes idle, and end the turn. - Raises: - RuntimeError: If the agent is not started. - TurnTimeoutError: Timeout elapsed (partial TurnRecord on pending_turn). - AgentCrashError: SDK/harness failed mid-turn (same pending_turn contract). + ``should_stop`` is polled after each processed Step; on a reason the loop + breaks, the conversation is cancelled (best-effort) and the turn ends with + ``end_status_for(reason)``. See ``Agent.communicate``. """ if not self.working_directory or self._sdk_agent is None: raise RuntimeError("Agent not started. Call start() first.") - assert self.config.type is not None, "AntigravityAgent requires AgentConfig.type before communicate()" - self._begin_turn() - # Raw monotonic, deliberately NOT the turn clock: this seeds the poll - # deadline and `duration_seconds`, neither of which may move when the wall - # clock steps. `TurnClock` is for the RECORDED stamps. - turn_start_time = time.monotonic() - # ONE clock per turn, so the window bounds and the tool intervals - # subtracted from them share a basis. - clock = TurnClock() - task_id = str(self.config.type) - model = self._effective_model() - collector = EventCollector() - emit = CompositeStreamCallback([c for c in (collector, stream_callback) if c is not None]) - turn_id = f"antigravity-{self._iteration}" - - state = _AntigravityTurnState( - agent=self, - emit=emit, - task_id=task_id, - turn_id=turn_id, - collector=collector, - user_input=user_input, - iteration=self._iteration, - model=model, - turn_start_time=turn_start_time, - clock=clock, + emitter = self._open_emitter( + prompt=user_input, + iteration=iteration, + model=self._effective_model(), + task_id=str(self.config.type), + stream_callback=stream_callback, ) + emitter.begin() + turn_id = f"antigravity-{iteration}" + decoder = _AntigravityDecoder(emitter, turn_id=turn_id) + + def _on_turn_timeout() -> None: + decoder.timeout_hit = True try: - # From the TURN CLOCK, not the model's raw `datetime.now()` default: - # this bound is subtracted against window bounds the same clock - # produced, and two bases in one subtraction clamped this harness's - # -0.017 ms tail to a measured 0.0 (CE058). - emit.on_event( - AgentStartEvent( - task_id=task_id, - prompt=user_input, - iteration=self._iteration, - model=model, - timestamp=clock.now(), + try: + await run_with_watchdog( + self._run_turn(user_input, decoder, timeout, should_stop), + timeout_seconds=timeout, + on_timeout=_on_turn_timeout, + label=f"Turn timeout ({timeout:g}s)" if timeout else "turn_timeout", ) - ) - - def _on_turn_timeout() -> None: - state.timeout_hit = True - - with ThreadedWatchdog( - timeout_seconds=timeout, - on_timeout=_on_turn_timeout, - asyncio_task_to_cancel=asyncio.current_task(), - label=f"Turn timeout ({timeout:g}s)" if timeout else "turn_timeout", - ): - emit.on_event(TurnStartEvent(task_id=task_id, turn_id=turn_id, model=model)) - conversation = self._sdk_agent.conversation - poll_count = 0 - # Bound the poll loop's OWN exit earlier than the watchdog's, so - # its graceful path reliably wins that race. `timeout=None` has - # nothing to take a fraction of, so the cycle cap is the sole bound. - poll_deadline = turn_start_time + timeout * _POLL_DEADLINE_TIMEOUT_FRACTION if timeout else None - try: - await conversation.send(user_input) - # should_stop runs AFTER process_step (the emission the monitor - # latches on) and BEFORE the next step is pulled. - await self._drain(conversation, state, should_stop) - - # The model may background a run_command and go idle, so - # receive_steps() exhausts with that call still open. Gated on - # the orphaned-tool signal, so a normal turn takes this branch - # zero times. - # Rationale: .claude/notes/agents.md § Antigravity Step interleaving and the background poll - while ( - state.stop_reason is None - and not state.timeout_hit - and state.has_orphaned_tool_call() - and ( - poll_count < _MAX_BACKGROUND_POLLS - if poll_deadline is None - else time.monotonic() < poll_deadline - ) - ): - poll_count += 1 - self._log.debug("Polling for backgrounded work (orphaned tool call); attempt %d", poll_count) - await asyncio.sleep(_BACKGROUND_POLL_INTERVAL_SECONDS) - if state.timeout_hit or (poll_deadline is not None and time.monotonic() >= poll_deadline): - # Skip the re-drain, which could itself await - # indefinitely on genuinely non-idle work. - break - reason = should_stop() if should_stop is not None else None - if reason is not None: - state.stop_reason = reason - break - await self._drain(conversation, state, should_stop) - - if state.has_orphaned_tool_call() and state.stop_reason is None and not state.timeout_hit: - # Exited via this loop's OWN bound, not an external - # stop/timeout: the call is force-closed as unresolved and - # the turn is still graded normally on everything else. - bound = ( - f"poll_deadline ({_POLL_DEADLINE_TIMEOUT_FRACTION:.0%} of {timeout:g}s turn timeout)" - if poll_deadline is not None - else f"_MAX_BACKGROUND_POLLS ({_MAX_BACKGROUND_POLLS})" - ) - msg = "Poll budget exhausted (%s, poll_count=%d) with a tool call still ACTIVE." - self._log.warning(msg, bound, poll_count) - - if state.stop_reason is not None: - # Best-effort server-side cancel. One check point, so it - # fires exactly once whichever drain stopped. - with contextlib.suppress(Exception): - await conversation.cancel() - except asyncio.CancelledError: - if state.timeout_hit: - self._finalize_and_raise_timeout(state.finalize, timeout or 0) - raise - except Exception as e: - if state.timeout_hit: - self._finalize_and_raise_timeout(state.finalize, timeout or 0, cause=e) - if state.ended_cleanly: - # Already stopped on purpose — do not escalate. - # Rationale: .claude/notes/agents.md § Why a post-stop exception is not a crash - self._log.warning("Ignoring post-stop exception; finalizing cleanly: %s", e) - else: - self._finalize_and_raise_crash( - state.finalize, truncate_crash_message(f"Antigravity turn failed: {e!s}"), cause=e - ) - - if state.timeout_hit: - # Watchdog fired but the pump finished before the cancel landed. - assert timeout is not None - self._finalize_and_raise_timeout(state.finalize, timeout) - except (AgentCrashError, TurnTimeoutError): - raise + except WatchdogFired: + return self._fail(decoder, AgentEndStatus.TIMEOUT, format_timeout_reason(timeout or 0)) + except Exception as e: + if decoder.timeout_hit: + return self._fail(decoder, AgentEndStatus.TIMEOUT, format_timeout_reason(timeout or 0)) + if not decoder.ended_cleanly: + return self._fail(decoder, AgentEndStatus.CRASHED, f"Antigravity turn failed: {e!s}") + # Already stopped on purpose — do not escalate. + # Rationale: .claude/notes/agents.md § Why a post-stop exception is not a crash + self._log.warning("Ignoring post-stop exception; finalizing cleanly: %s", e) + if decoder.timeout_hit: + # The watchdog fired but the body finished before the cancel landed. + return self._fail(decoder, AgentEndStatus.TIMEOUT, format_timeout_reason(timeout or 0)) except asyncio.CancelledError: - if not state.finalized: - self._finalize_external_cancel(state.finalize) + caller = asyncio.current_task() + if caller is not None and caller.cancelling() == 0: + # Not a cancel from outside: the SDK raised it inside the turn body. + return self._fail(decoder, AgentEndStatus.CRASHED, "Antigravity turn failed: the SDK was cancelled") + self._state = AgentState.ERROR + decoder.end(AgentEndStatus.CRASHED, reason="turn cancelled", agent_output=self._last_response()) raise - except Exception as e: - if state.ended_cleanly and not state.timeout_hit: - # Same retry-poisoning guard as the inner handler. - self._log.warning("Ignoring post-stop exception; finalizing cleanly: %s", e) - else: - self._finalize_and_raise_crash( - state.finalize, truncate_crash_message(f"Antigravity turn failed: {e!s}"), cause=e - ) self._state = AgentState.WORKING - self._end_turn_ok() - # Precedence: timeout (raised above) > the stop reason > done. - # Rationale: .claude/notes/agents.md § Shared turn lifecycle - status = end_status_for(state.stop_reason) if state.stop_reason is not None else AgentEndStatus.COMPLETED - state.finalize(status, crashed=False, crash_reason=None) - return collector.build_turn_record() + # Precedence: timeout (above) > the stop reason > done. + status = end_status_for(decoder.stop_reason) if decoder.stop_reason is not None else AgentEndStatus.COMPLETED + return decoder.end(status, agent_output=self._last_response()) + + async def _run_turn( + self, + user_input: str, + decoder: _AntigravityDecoder, + timeout: float | None, + should_stop: Callable[[], StopReason | None] | None, + ) -> None: + """Send the prompt, drain, and poll a backgrounded tool call until the turn goes idle.""" + conversation = self._sdk_agent.conversation + turn_start = time.monotonic() + # Bound the poll loop's OWN exit earlier than the watchdog's, so its graceful + # path reliably wins that race. `timeout=None` leaves the cycle cap as the bound. + poll_deadline = turn_start + timeout * _POLL_DEADLINE_TIMEOUT_FRACTION if timeout else None + try: + await conversation.send(user_input) + await self._drain(conversation, decoder, should_stop) + await self._poll_background_work(conversation, decoder, timeout, should_stop, poll_deadline) + finally: + if decoder.stop_reason is not None: + # Best-effort server-side cancel, once, whichever drain stopped. + with contextlib.suppress(Exception): + await conversation.cancel() + + async def _poll_background_work( + self, + conversation: Any, + decoder: _AntigravityDecoder, + timeout: float | None, + should_stop: Callable[[], StopReason | None] | None, + poll_deadline: float | None, + ) -> None: + poll_count = 0 + + # The model may background a run_command and go idle, so receive_steps() + # exhausts with that call still open. Gated on the orphaned-tool signal, so a + # normal turn takes this branch zero times. + # Rationale: .claude/notes/agents.md § Antigravity Step interleaving and the background poll + while ( + decoder.stop_reason is None + and not decoder.timeout_hit + and decoder.has_orphaned_tool_call() + and (poll_count < _MAX_BACKGROUND_POLLS if poll_deadline is None else time.monotonic() < poll_deadline) + ): + poll_count += 1 + self._log.debug("Polling for backgrounded work (orphaned tool call); attempt %d", poll_count) + await asyncio.sleep(_BACKGROUND_POLL_INTERVAL_SECONDS) + if decoder.timeout_hit or (poll_deadline is not None and time.monotonic() >= poll_deadline): + # Skip the re-drain, which could itself await indefinitely. + break + reason = should_stop() if should_stop is not None else None + if reason is not None: + decoder.stop_reason = reason + break + await self._drain(conversation, decoder, should_stop) + + if decoder.has_orphaned_tool_call() and decoder.stop_reason is None and not decoder.timeout_hit: + bound = ( + f"poll_deadline ({_POLL_DEADLINE_TIMEOUT_FRACTION:.0%} of {timeout:g}s turn timeout)" + if poll_deadline is not None + else f"_MAX_BACKGROUND_POLLS ({_MAX_BACKGROUND_POLLS})" + ) + self._log.warning( + "Poll budget exhausted (%s, poll_count=%d) with a tool call still ACTIVE.", bound, poll_count + ) + + def _fail(self, decoder: _AntigravityDecoder, status: AgentEndStatus, reason: str) -> TurnOutcome: + self._state = AgentState.ERROR + return decoder.end(status, reason=reason, agent_output=self._last_response()) + + def _last_response(self) -> str: + with contextlib.suppress(Exception): + return str(self._sdk_agent.conversation.last_response or "") + return "" async def stop(self) -> None: """Stop the agent and tear down the local harness session.""" @@ -612,6 +782,15 @@ def kill_sync(self) -> None: """ self._state = AgentState.ERROR + async def harness_version(self) -> str | None: + """The ``google-antigravity`` SDK version; its localharness ships inside the package.""" + from importlib.metadata import PackageNotFoundError, version + + try: + return f"google-antigravity {version('google-antigravity')}" + except PackageNotFoundError: + return None + def get_environment_info(self) -> dict[str, Any]: """Record the resolved Gemini model + thinking level for auditability.""" return { @@ -645,347 +824,3 @@ async def _teardown(self) -> None: self._log.warning( "Antigravity harness teardown failed; the harness process may still be running", exc_info=True ) - - -class _AntigravityTurnState: - """Per-turn mutable scratch for one ``AntigravityAgent.communicate`` call. - - Maps the Gemini step stream onto the standardized event protocol and - reconstructs the assistant transcript. The same ``messages`` / ``commands`` - accumulate live, so a mid-turn crash keeps the partial transcript (the - agent's shared crash kernel builds ``pending_turn`` from ``collector``). - - Step-stream shape this consumes (observed): each ``step_index`` is yielded - repeatedly through ACTIVE -> DONE transitions; ``usage_metadata`` lands once - per generation on a DONE/terminal step (summing them == the turn total); a - tool call carries a stable ``id`` and its result is folded into expanded - ``args`` at DONE. - """ - - def __init__( - self, - *, - agent: AntigravityAgent, - emit: CompositeStreamCallback, - task_id: str, - turn_id: str, - collector: EventCollector, - user_input: str, - iteration: int, - model: str, - turn_start_time: float, - clock: TurnClock, - ) -> None: - self._agent = agent - self.emit = emit - self.task_id = task_id - self.turn_id = turn_id - self.collector = collector - self.user_input = user_input - self.iteration = iteration - self.model = model - self.turn_start_time = turn_start_time - # Injected, not read from a module global, so a test supplies a fake - # instead of monkeypatching `datetime` out from under the reducer. - self.clock = clock - - self.timeout_hit = False - self.stop_reason: StopReason | None = None - self.finalized = False - - self.total_usage = TokenUsage() - self.messages: list[TranscriptMessage] = [] - self.commands: list[CommandTelemetry] = [] - self._output_parts: list[str] = [] - self._assistant_turns = 0 - - # ToolStart on first sight of an id; ToolEnd at DONE. - self._next_seq = 0 - self._seen_tools: set[str] = set() - self._closed_tools: set[str] = set() - self._open_tools: dict[str, CommandTelemetry] = {} - # Arg keys present when a tool was first seen (its model-supplied - # inputs), used at DONE to tell them from harness-appended result fields. - self._tool_input_keys: dict[str, set[str]] = {} - # Most recently seen StepStatus per tool id, for has_orphaned_tool_call. - # Separate from _closed_tools, which tracks only DONE/ERROR. - self._tool_last_status: dict[str, Any] = {} - # Content blocks accumulated since the last per-generation flush. - self._blocks: list[ContentBlock] = [] - # Where the CURRENT generation started, advanced only by a flush that - # actually emitted a message. - self._gen_mark_wall: datetime = clock.now() - # Re-seeded ONCE, at the first observed Step. See - # `_seed_first_generation_window`. - self._first_output_seen: bool = False - - @property - def ended_cleanly(self) -> bool: - """True once the loop broke on a ``should_stop`` reason. - - A non-crash termination, so a stray exception raised while unwinding the - step generator afterwards must not be escalated. - """ - return self.stop_reason is not None - - def _seed_first_generation_window(self, source: Any) -> None: - """Move the first window's mark to the first observed MODEL output. - - GATED ON ``source``, because ``harness_startup_ms`` is defined as model - output and the SDK streams Steps that are not: a turn can legitimately open - with a SYSTEM or USER Step, and seeding on one would put the mark BEFORE - the model spoke. The same gate guards text streaming below. - - ONCE PER TURN, and that is the whole contract: re-seeding would stop the - windows tiling. The flag needs no reset — a fresh turn state is built per - ``communicate()``. A turn that streams no MODEL Step keeps the turn-entry - mark and clamps to ``0.0``, which is the correct degradation. - - Rationale: .claude/notes/agents.md § First-generation window seeding - """ - if self._first_output_seen or _enum_value(source) != _SOURCE_MODEL: - return - self._first_output_seen = True - self._gen_mark_wall = self.clock.now() - - def process_step(self, step: Any) -> None: - """Route one streamed ``Step`` to events + transcript reconstruction.""" - stype = _enum_value(step.type) - sstatus = _enum_value(step.status) - ssource = _enum_value(step.source) - self._seed_first_generation_window(ssource) - starget = _enum_value(step.target) - done = sstatus in (_STATUS_DONE, _STATUS_ERROR) - - # Stream visible assistant text deltas. - if step.content_delta and ssource == _SOURCE_MODEL and starget == _TARGET_USER and stype == _TYPE_TEXT_RESPONSE: - self.emit.on_event(TextChunkEvent(task_id=self.task_id, turn_id=self.turn_id, text=step.content_delta)) - - # Tool calls: ToolStart on first sight, ToolEnd when the owning step is DONE. - for call_index, call in enumerate(step.tool_calls): - self._handle_tool_call(call, step, done, sstatus, call_index) - - # Capture content blocks on the terminal transition of a step. - if done: - if stype == _TYPE_THINKING and step.thinking: - self._blocks.append(ContentBlock(block_type="thinking", sequence=0, thinking=step.thinking)) - elif stype == _TYPE_TEXT_RESPONSE and step.content: - self._output_parts.append(step.content) - self._blocks.append(ContentBlock(block_type="text", sequence=0, text=step.content)) - - # Per-generation usage: fold into the turn total and cut an AssistantMessage. - if step.usage_metadata is not None: - gen = _to_token_usage(step.usage_metadata, self.model) - self.total_usage = self.total_usage + gen - self._flush_generation(gen, getattr(step.usage_metadata, "thoughts_token_count", 0) or 0) - - def _handle_tool_call(self, call: Any, step: Any, done: bool, sstatus: Any, call_index: int) -> None: - raw_name = _enum_value(call.name) - # call.id is usually present but the SDK types it optional. The fallback - # mirrors the SDK's own `trajectory_id:step_index` scheme; call_index - # further disambiguates multiple id-less calls within one step. - # Rationale: .claude/notes/agents.md § Why the tool-call id falls back the way it does - trajectory_id = getattr(step, "trajectory_id", "") or "" - step_key = f"{trajectory_id}:{step.step_index}" if trajectory_id else str(step.step_index) - cid = call.id or f"{raw_name}_{step_key}_{call_index}" - self._tool_last_status[cid] = sstatus - if cid not in self._seen_tools: - self._seen_tools.add(cid) - seq = self._next_seq - self._next_seq += 1 - tool_name = _ANTIGRAVITY_TO_CLAUDE_TOOL_MAP.get(raw_name, str(raw_name)) - self._tool_input_keys[cid] = set(call.args) - now = self.clock.now() - tel = CommandTelemetry( - tool_name=tool_name, - tool_id=cid, - timestamp=now, - parameters=self._params(tool_name, call.args, self._tool_input_keys[cid]), - sequence_number=seq, - execution_started_at=now, - ) - self._open_tools[cid] = tel - self.emit.on_event(ToolStartEvent(task_id=self.task_id, turn_id=self.turn_id, tool=tel)) - - if done and cid in self._open_tools and cid not in self._closed_tools: - self._closed_tools.add(cid) - start_tel = self._open_tools[cid] - exit_code = call.args.get("exit_code") - errored = (sstatus == _STATUS_ERROR) or (exit_code not in (None, 0)) - result_text = ( - call.args.get("combined_output") - or call.args.get("diff_block") - or call.args.get("output") - or call.args.get("results") - or call.args.get("summary") - or step.content - or None - ) - completed = self.clock.now() - started = start_tel.execution_started_at or completed - tool_ms = max((completed - started).total_seconds() * 1000.0, 0.0) - end_tel = start_tel.model_copy( - update={ - "parameters": self._params(start_tel.tool_name, call.args, self._tool_input_keys.get(cid)), - "result_status": "error" if errored else "success", - "result_summary": str(result_text) if result_text is not None else None, - "error_message": (step.error or "tool failed") if errored else None, - "execution_completed_at": completed, - "duration_ms": tool_ms, - } - ) - self.commands.append(end_tel) - self.emit.on_event( - ToolEndEvent( - task_id=self.task_id, - turn_id=self.turn_id, - tool=end_tel, - status=ToolEndStatus.ERROR if errored else ToolEndStatus.OK, - ) - ) - self._blocks.append(ContentBlock(block_type="tool_use", sequence=0, tool_use_id=cid)) - - @staticmethod - def _params(tool_name: str, args: dict[str, Any], input_keys: set[str] | None) -> dict[str, Any]: - """Model-supplied inputs only, renamed to canonical cross-agent keys. - - A key is a (dropped) result field when it is in the static - ``_RESULT_ARG_KEYS`` backstop OR first appeared at DONE, given the - input-key snapshot taken at tool start. Survivors are renamed to the - canonical vocabulary. - """ - rename = _ANTIGRAVITY_ARG_RENAME.get(tool_name, {}) - out: dict[str, Any] = {} - for k, v in args.items(): - if k in _RESULT_ARG_KEYS: - continue - if input_keys is not None and k not in input_keys: - continue # appeared only at DONE → harness result payload - out[rename.get(k, k)] = v - return out - - def _flush_generation(self, gen: TokenUsage, reasoning_tokens: int) -> None: - """Cut accumulated blocks into one AssistantMessage carrying this gen's tokens. - - Keeping the per-message buckets summing to the turn total means the - collector's reconciliation books a zero residual. - """ - if not self._blocks and gen.is_empty(): - return - now_wall = self.clock.now() - # Do NOT "simplify" this to resetting the mark when a tool ends: this - # harness interleaves a tool INTO a window rather than tiling around it, - # so the RAW window legitimately contains time that is not model time and - # the collector clips the tool union out of it. Resetting instead drops - # the model time around a fast tool. - # Rationale: .claude/notes/agents.md § Per-harness generation marks - _, generation_ms = close_window(mark=self._gen_mark_wall, now=now_wall) - for i, block in enumerate(self._blocks): - block.sequence = i - self.messages.append( - AssistantMessage( - started_at=self._gen_mark_wall, - completed_at=now_wall, - generation_duration_ms=generation_ms, - content_blocks=list(self._blocks), - tool_use_ids=[b.tool_use_id for b in self._blocks if b.block_type == "tool_use" and b.tool_use_id], - input_tokens=gen.uncached_input_tokens, - output_tokens=gen.output_tokens, - cache_creation_tokens=0, - cache_read_tokens=gen.cache_read_input_tokens, - reasoning_tokens=reasoning_tokens, - model=self.model, - # The Step stream carries no message id, and the evalboard's - # gap fallback cannot split contiguous windows. - message_id=f"{self.turn_id}-msg-{self._assistant_turns}", - ) - ) - self._assistant_turns += 1 - self._blocks = [] - # Advance ONLY after a message was appended: a no-op flush leaves the - # window open, so a later real generation still measures from its start. - self._gen_mark_wall = now_wall - - def _agent_output(self) -> str: - if self._output_parts: - return "".join(self._output_parts) - with contextlib.suppress(Exception): - return self._agent._sdk_agent.conversation.last_response # type: ignore[union-attr] - return "" - - def has_orphaned_tool_call(self) -> bool: - """True if any NOT-YET-CLOSED tool call's most recently seen status is - ACTIVE — the structural signature of a backgrounded task the model went - idle on without waiting for. See ``communicate``'s poll loop. - - An ALLOWLIST on ACTIVE, never a denylist on "not yet closed": the SDK also - has WAITING_FOR_USER, CANCELED and UNKNOWN, none of which the poll loop - should wait out. The `not in _closed_tools` guard is layered on top as a - monotonicity backstop, not a substitute. - - Rationale: .claude/notes/agents.md § Antigravity Step interleaving and the background poll - """ - return any(cid not in self._closed_tools and s == _STATUS_ACTIVE for cid, s in self._tool_last_status.items()) - - def finalize(self, status: AgentEndStatus, *, crashed: bool = False, crash_reason: str | None = None) -> None: - """Close orphaned tools, flush leftover blocks, emit TurnEnd + AgentEnd. - - Idempotent. On a crash, also builds the partial ``pending_turn`` from the - collector (the agent base's shared crash kernel). - """ - if self.finalized: - return - self.finalized = True - - # Force-close any tool that emitted ToolStart but never reached DONE. - for cid, tel in self._open_tools.items(): - if cid in self._closed_tools: - continue - orphan = tel.model_copy(update={"result_status": "unknown", "execution_completed_at": self.clock.now()}) - self.emit.on_event( - ToolEndEvent( - task_id=self.task_id, - turn_id=self.turn_id, - tool=orphan, - status=ToolEndStatus.UNRESOLVED, - ) - ) - - # Flush any trailing blocks not yet attached to a generation (no usage). - if self._blocks: - self._flush_generation(TokenUsage(), 0) - - # Parallel by value, so an unmapped future member raises loudly instead - # of silently bucketing to COMPLETED. - turn_status = TurnEndStatus(status.value) - - self.emit.on_event( - TurnEndEvent( - task_id=self.task_id, - turn_id=self.turn_id, - status=turn_status, - tokens=self.total_usage, - ) - ) - self.emit.on_event( - AgentEndEvent( - task_id=self.task_id, - status=status, - usage=self.total_usage, - iteration=self.iteration, - user_input=self.user_input, - agent_output=self._agent_output(), - model_used=self.model, - assistant_turn_count=self._assistant_turns, - messages=self.messages, - num_turns=self._assistant_turns, - crashed=crashed, - crash_reason=crash_reason, - duration_seconds=time.monotonic() - self.turn_start_time, - # One basis with the window bounds — see the AgentStartEvent site. - timestamp=self.clock.now(), - ) - ) - - if crashed: - self._agent._capture_partial_turn(self.collector) diff --git a/src/coder_eval/agents/claude_code_agent.py b/src/coder_eval/agents/claude_code_agent.py index 6f381312..ed25d921 100644 --- a/src/coder_eval/agents/claude_code_agent.py +++ b/src/coder_eval/agents/claude_code_agent.py @@ -8,7 +8,7 @@ import time from collections.abc import Callable, Sequence from contextlib import suppress -from datetime import datetime, timedelta +from datetime import datetime from pathlib import Path from typing import Any @@ -32,15 +32,12 @@ # is the only import route (same treatment as evaluation/verdict_tool.py). from claude_agent_sdk.types import SdkPluginConfig, SystemPromptPreset -from coder_eval.agent import Agent, AgentState +from coder_eval.agent import Agent, AgentState, command_version from coder_eval.agents._logging import PrefixedAdapter, log_raw_sdk_event -from coder_eval.agents.registry import AgentRegistry -from coder_eval.agents.watchdog import ThreadedWatchdog +from coder_eval.agents.registry import SPI_VERSION, AgentRegistry +from coder_eval.agents.watchdog import WatchdogFired, run_with_watchdog from coder_eval.config import settings -from coder_eval.errors import ( - TurnTimeoutError, - format_timeout_reason, -) +from coder_eval.errors import format_timeout_reason from coder_eval.formatting import format_messages, format_payload from coder_eval.models import ( CANONICAL_TOOL_NAMES, @@ -48,7 +45,6 @@ ApiRoute, BedrockRoute, ClaudeCodeAgentConfig, - CommandTelemetry, ContentBlock, DirectRoute, Enforcement, @@ -57,34 +53,28 @@ PermissionMode, ResultSummary, SystemPromptSemantics, + TimingBasis, TokenUsage, ToolNameMap, TranscriptMessage, - TurnRecord, UsageGranularity, to_bedrock_inference_profile, ) from coder_eval.models import ( AssistantMessage as AssistantMessageTelemetry, ) -from coder_eval.pricing import calculate_cost -from coder_eval.streaming.callbacks import CompositeStreamCallback, StreamCallback -from coder_eval.streaming.collector import EventCollector +from coder_eval.orchestration.plugin_staging import staged_plugin_dirs +from coder_eval.pricing import price_turn +from coder_eval.streaming.callbacks import StreamCallback +from coder_eval.streaming.emitter import Generation, TurnEmitter, TurnOutcome from coder_eval.streaming.events import ( - AgentEndEvent, AgentEndStatus, - AgentStartEvent, StopReason, - TextChunkEvent, - ToolEndEvent, ToolEndStatus, - ToolStartEvent, - TurnEndEvent, TurnEndStatus, - TurnStartEvent, end_status_for, ) -from coder_eval.timing import TurnClock, close_window +from coder_eval.timing import close_window from coder_eval.utils import dump_dataclass @@ -189,102 +179,53 @@ def _is_sdk_result_message(message: Any) -> bool: _JSON_START_SEARCH_LIMIT = 200 -class _ClaudeTurnState: - """Per-turn mutable scratch state for one ``ClaudeCodeAgent.communicate`` call. - - Holds every cross-branch local the SDK-stream pump mutates, with one method - per message kind plus ``dispatch`` and ``finalize``. A back-reference to the - agent lets it reuse the agent's helpers. +class _ClaudeDecoder: + """Per-turn decoder: receives SDK messages for one ``communicate`` call and reports them to the emitter. - The two raw lists are DISTINCT and must stay so: ``messages`` holds raw SDK - ``Message`` objects and ``sdk_messages`` holds telemetry - ``TranscriptMessage`` objects, carried on ``AgentEndEvent``. + One inner turn per message id; a message with ``parent_tool_use_id`` is a sub-agent's, so + its turn, generation and tools are nested under that id. ``sdk_model_used`` follows the + main thread only. """ - def __init__( - self, - agent: "ClaudeCodeAgent", - *, - emit: CompositeStreamCallback, - collector: EventCollector, - task_id: str, - user_input: str, - iteration: int, - log: PrefixedAdapter, - turn_start_time: float, - deadline: float | None, - clock: TurnClock | None = None, - ) -> None: + def __init__(self, agent: "ClaudeCodeAgent", emitter: TurnEmitter, *, effective_model: str | None) -> None: self._agent = agent - self.emit = emit - self.collector = collector - self.task_id = task_id - self.user_input = user_input - self.iteration = iteration - self.log = log - self.turn_start_time = turn_start_time - self.deadline = deadline - # Set True by the in-loop deadline break OR the watchdog callback. + self.emitter = emitter + self.effective_model = effective_model + self.log = agent._log + # Set by the in-loop deadline break OR the watchdog callback. self.timeout_hit = False - # Set by the in-loop should_stop break. Distinct from timeout_hit: a clean, - # non-crash stop that must NOT raise. self.stop_reason: StopReason | None = None - # Resolved by _build_claude_query, set on the state before any finalize - # path. Stays None if we crash before setup (finalize reads it for cost - # backfill). - self.effective_model: str | None = None - - # Two distinct lists (do NOT merge): raw SDK objects vs. telemetry. self.messages: list[Message] = [] - self.sdk_messages: list[TranscriptMessage] = [] - # Two-phase command tracking (tool_id -> {telemetry, command_start_time}). - self.pending_commands: dict[str, dict[str, Any]] = {} + self.opened_tools: dict[str, str] = {} + self.transcript: list[AssistantMessageTelemetry] = [] self.processed_results: set[str] = set() - self.sequence_number = 0 - - self.last_assistant_message_index: int | None = None - # ONE clock per turn, and every wall stamp this turn records derives from - # it, so the central subtraction clips WALL tool spans to WALL window - # bounds. Injectable so a test supplies a fake rather than monkeypatching - # this module's `datetime`, which a derived stamp silently escapes. - # `turn_start_time` stays raw monotonic: a deadline must not move when the - # wall clock steps. - self.clock = clock or TurnClock() - self.last_event_wall: datetime = self.clock.now() - # Re-seeded ONCE, at the first observed model output. See - # `_seed_first_generation_window`. - self.first_output_seen: bool = False - - # SDK ResultMessage capture. + + self.last_event_wall: datetime = emitter.now() + self.first_output_seen = False + self.sdk_result_usage: dict[str, Any] | None = None self.sdk_result_model_usage: dict[str, Any] | None = None self.sdk_result_cost: float | None = None self.num_turns: int | None = None self.sdk_result_summary: ResultSummary | None = None - self.sdk_model_used: str | None = None + # The last model each sub-agent streamed, keyed by its spawning tool_use_id. + self.subagent_models: dict[str, str] = {} - # Per-emission output_tokens recovery from raw stream events. self.pending_delta_output_tokens: int | None = None self.current_stream_message_id: str | None = None self.emissions_by_id: dict[str, list[AssistantMessageTelemetry]] = {} self.emission_proxies_by_id: dict[str, list[int]] = {} - - # Dedup of multi-emission API calls sharing a message_id. self.seen_message_ids: set[str] = set() - self.last_message_had_id: bool = False + self.last_message: AssistantMessageTelemetry | None = None + self.last_message_had_id = False self.assistant_turn_count = 0 - - # Turn/tool bracketing (self-describing event tree). self.current_turn_id: str | None = None # Tokens each message id already reported on a TurnEndEvent: an id can # resume after another id's emission, and must report only what is new. self.reported_tokens_by_id: dict[str, TokenUsage] = {} - self.tool_turn_ids: dict[str, str] = {} # tool_id -> spawning turn_id - self.emitted_tool_ends: set[str] = set() # tool_ids already closed - self.finalized = False def turn_tokens(self, turn_id: str) -> TokenUsage | None: """Best-effort per-turn tokens, summed over that call's block emissions.""" @@ -317,20 +258,21 @@ def unreported_turn_tokens(self, turn_id: str) -> TokenUsage | None: cache_read_input_tokens=total.cache_read_input_tokens - previous.cache_read_input_tokens, ) - def dispatch(self, message: Message) -> None: + def __call__(self, message: Message) -> None: """Record the raw message and route it to its per-kind handler. ORDER IS LOAD-BEARING: ``_is_sdk_result_message`` before ``_is_user_message``, and the TaskNotification guard before both. """ self.messages.append(message) - msg_type = type(message).__name__ - log_raw_sdk_event(self.log, repr_target=message, type=msg_type) + log_raw_sdk_event(self.log, repr_target=message, type=type(message).__name__) if _is_assistant_message(message): self.on_assistant_message(message) elif _is_task_notification(message): - self.on_task_notification(message) + # Its per-sub-agent usage is LOSSY and is captured from the Agent tool + # result instead; the branch only keeps it from reading as a result. + pass elif _is_sdk_result_message(message): self.on_result_message(message) elif isinstance(getattr(message, "event", None), dict): @@ -338,211 +280,159 @@ def dispatch(self, message: Message) -> None: elif _is_user_message(message): self.on_user_message(message) - def on_assistant_message(self, message: Message) -> None: - """Capture ToolUseBlocks + build the AssistantMessage telemetry record.""" - message_arrival_wall = self.clock.now() - generation_started_wall = self.last_event_wall + def _switch_inner_turn(self, turn_id: str, model: str | None, parent: str | None) -> None: + if turn_id == self.current_turn_id: + return + if self.current_turn_id is not None and self.emitter.inner_turn_open: + self.emitter.end_inner_turn(tokens=self.unreported_turn_tokens(self.current_turn_id)) + self.current_turn_id = turn_id + self.emitter.begin_inner_turn(turn_id, model, parent_tool_id=parent) - current_turn_index = len(self.sdk_messages) - self.assistant_turn_count += 1 + def on_assistant_message(self, message: Message) -> None: + """Open the message's inner turn and tools, then add its generation.""" + arrival = self.emitter.now() + mark = self.last_event_wall + raw_parent = getattr(message, "parent_tool_use_id", None) + parent = raw_parent if isinstance(raw_parent, str) else None model_attr = getattr(message, "model", None) - if isinstance(model_attr, str): - self.sdk_model_used = model_attr + message_model = model_attr if isinstance(model_attr, str) else None + if message_model is not None: + if parent is None: + self.sdk_model_used = message_model + else: + self.subagent_models[parent] = message_model + model = self.sdk_model_used if parent is None else message_model + self.assistant_turn_count += 1 - # Inner-turn boundary: one TurnStart per new message_id (one API call). raw_mid = getattr(message, "message_id", None) - turn_id = raw_mid if isinstance(raw_mid, str) else f"turn-{self.assistant_turn_count}" - if turn_id != self.current_turn_id: - if self.current_turn_id is not None: - self.emit.on_event( - TurnEndEvent( - task_id=self.task_id, - turn_id=self.current_turn_id, - status=TurnEndStatus.COMPLETED, - tokens=self.unreported_turn_tokens(self.current_turn_id), - ) - ) - self.current_turn_id = turn_id - self.emit.on_event(TurnStartEvent(task_id=self.task_id, turn_id=turn_id, model=self.sdk_model_used)) - - content = getattr(message, "content", None) - turn_content_blocks: list[ContentBlock] = [] - turn_tool_use_ids: list[str] = [] - emission_content_chars = 0 - - if content and isinstance(content, list): - for block in content: - block_seq = len(turn_content_blocks) + message_id = raw_mid if isinstance(raw_mid, str) else None + self._switch_inner_turn(message_id or f"turn-{self.assistant_turn_count}", model, parent) - if _is_tool_use_block(block): - tool_args = block.input if isinstance(block.input, dict) else {"raw": block.input} - emission_content_chars += len(str(getattr(block, "name", "") or "")) + len( - json.dumps(tool_args, default=str) - ) - command_start_time = time.monotonic() - - telemetry = CommandTelemetry( - tool_name=block.name, - tool_id=block.id, - timestamp=message_arrival_wall, - generation_completed_at=message_arrival_wall, - assistant_turn_index=current_turn_index, - parameters=block.input if isinstance(block.input, dict) else {"raw": block.input}, - sequence_number=self.sequence_number, - result_status=None, - duration_ms=None, - ) - - self.pending_commands[block.id] = { - "telemetry": telemetry, - "command_start_time": command_start_time, - } - self.sequence_number += 1 - - turn_content_blocks.append( - ContentBlock(block_type="tool_use", sequence=block_seq, tool_use_id=block.id) - ) - turn_tool_use_ids.append(block.id) - - self.tool_turn_ids[block.id] = self.current_turn_id or "" - self.emit.on_event( - ToolStartEvent(task_id=self.task_id, turn_id=self.current_turn_id or "", tool=telemetry) - ) - elif _is_thinking_block(block): - thinking_text = getattr(block, "thinking", None) - if thinking_text: - emission_content_chars += len(str(thinking_text)) - turn_content_blocks.append( - ContentBlock( - block_type="thinking", - sequence=block_seq, - thinking=str(thinking_text) if thinking_text else None, - signature=getattr(block, "signature", None), - ) - ) - elif _is_text_block(block): - text_value = str(block.text) - emission_content_chars += len(text_value) - turn_content_blocks.append(ContentBlock(block_type="text", sequence=block_seq, text=text_value)) - self.emit.on_event( - TextChunkEvent(task_id=self.task_id, turn_id=self.current_turn_id or "", text=text_value) + blocks, proxy = self._blocks(getattr(message, "content", None), parent) + tokens, reasoning = self._emission_tokens(getattr(message, "usage", None) or {}, message_id) + stop_reason = getattr(message, "stop_reason", None) + # The RAW window, opened at the mark, since this stream carries no + # per-emission item start to pull the window open to. + # Rationale: .claude/notes/agents.md § Per-harness generation marks + (record,) = self.emitter.add_generation( + message_id=message_id, + window=close_window(mark=mark, now=arrival), + parts=[ + Generation( + blocks=blocks, + tokens=tokens, + reasoning_tokens=reasoning, + stop_reason=stop_reason if isinstance(stop_reason, str) else None, + ) + ], + model=model, + parent_tool_id=parent, + ) + self.transcript.append(record) + if message_id is not None: + self.emissions_by_id.setdefault(message_id, []).append(record) + self.emission_proxies_by_id.setdefault(message_id, []).append(proxy) + self.last_message = record + self.last_event_wall = arrival + + def _blocks(self, content: Any, parent: str | None) -> tuple[list[ContentBlock], int]: + """The message's content blocks, opening each tool; also its content-length proxy.""" + blocks: list[ContentBlock] = [] + proxy = 0 + if not isinstance(content, list): + return blocks, proxy + for block in content: + sequence = len(blocks) + if _is_tool_use_block(block): + params = block.input if isinstance(block.input, dict) else {"raw": block.input} + proxy += len(str(getattr(block, "name", "") or "")) + len(json.dumps(params, default=str)) + blocks.append(ContentBlock(block_type="tool_use", sequence=sequence, tool_use_id=block.id)) + self.opened_tools[block.id] = block.name + self.emitter.open_tool(block.id, block.name, params, parent_tool_id=parent, generation_completed=True) + elif _is_thinking_block(block): + thinking = getattr(block, "thinking", None) + if thinking: + proxy += len(str(thinking)) + blocks.append( + ContentBlock( + block_type="thinking", + sequence=sequence, + thinking=str(thinking) if thinking else None, + signature=getattr(block, "signature", None), ) - - msg_usage = getattr(message, "usage", None) or {} - message_id = getattr(message, "message_id", None) - parent_tool_use_id = getattr(message, "parent_tool_use_id", None) - is_duplicate_emission = isinstance(message_id, str) and message_id in self.seen_message_ids - if isinstance(message_id, str): + ) + elif _is_text_block(block): + text = str(block.text) + proxy += len(text) + blocks.append(ContentBlock(block_type="text", sequence=sequence, text=text)) + self.emitter.text(text, parent_tool_id=parent) + return blocks, proxy + + def _emission_tokens(self, usage: dict[str, Any], message_id: str | None) -> tuple[TokenUsage, int]: + """This emission's tokens; a repeated message id's emission carries none.""" + duplicate = message_id is not None and message_id in self.seen_message_ids + self.last_message_had_id = message_id is not None + if message_id is not None: self.seen_message_ids.add(message_id) - self.last_message_had_id = True - else: - self.last_message_had_id = False - - if is_duplicate_emission: - in_tok = out_tok = cw_tok = cr_tok = rt_tok = 0 + if duplicate: + return TokenUsage(), 0 + if self.pending_delta_output_tokens is not None: + output = self.pending_delta_output_tokens else: - in_tok = int(msg_usage.get("input_tokens", 0) or 0) - cw_tok = int(msg_usage.get("cache_creation_input_tokens", 0) or 0) - cr_tok = int(msg_usage.get("cache_read_input_tokens", 0) or 0) - rt_tok = int(msg_usage.get("reasoning_tokens", 0) or 0) - if self.pending_delta_output_tokens is not None: - out_tok = self.pending_delta_output_tokens - else: - out_tok = int(msg_usage.get("output_tokens", 0) or 0) - self.pending_delta_output_tokens = None - - # The RAW window. `started` is the mark, since this stream carries no - # per-emission item start to pull the window open to. - # Rationale: .claude/notes/agents.md § Per-harness generation marks - started, raw_generation_ms = close_window(mark=generation_started_wall, now=message_arrival_wall) - assistant_telemetry = AssistantMessageTelemetry( - started_at=started, - completed_at=message_arrival_wall, - generation_duration_ms=raw_generation_ms, - content_blocks=turn_content_blocks, - tool_use_ids=turn_tool_use_ids, - input_tokens=in_tok, - output_tokens=out_tok, - cache_creation_tokens=cw_tok, - cache_read_tokens=cr_tok, - reasoning_tokens=rt_tok, - stop_reason=( - getattr(message, "stop_reason", None) - if isinstance(getattr(message, "stop_reason", None), str) - else None + output = int(usage.get("output_tokens", 0) or 0) + self.pending_delta_output_tokens = None + return ( + TokenUsage( + uncached_input_tokens=int(usage.get("input_tokens", 0) or 0), + output_tokens=output, + cache_creation_input_tokens=int(usage.get("cache_creation_input_tokens", 0) or 0), + cache_read_input_tokens=int(usage.get("cache_read_input_tokens", 0) or 0), ), - model=self.sdk_model_used, - message_id=message_id if isinstance(message_id, str) else None, - parent_tool_use_id=(parent_tool_use_id if isinstance(parent_tool_use_id, str) else None), + int(usage.get("reasoning_tokens", 0) or 0), ) - self.sdk_messages.append(assistant_telemetry) - if isinstance(message_id, str): - self.emissions_by_id.setdefault(message_id, []).append(assistant_telemetry) - self.emission_proxies_by_id.setdefault(message_id, []).append(emission_content_chars) - self.last_assistant_message_index = len(self.sdk_messages) - 1 - - self.last_event_wall = message_arrival_wall - - def on_task_notification(self, message: Message) -> None: - """TaskNotification carries LOSSY per-sub-agent usage, captured from the - Agent tool-result instead. This exists only to keep - ``_is_sdk_result_message`` from misreading it.""" - pass def on_result_message(self, message: Message) -> None: """Capture the SDK ResultMessage usage/cost/session + the id-less backfill.""" + agent = self._agent self.sdk_result_usage = getattr(message, "usage", None) self.sdk_result_model_usage = getattr(message, "model_usage", None) self.sdk_result_cost = getattr(message, "total_cost_usd", None) self.num_turns = getattr(message, "num_turns", None) - self.sdk_result_summary = self._agent._summarize_result(message) - # Only advance session_id on clean turns. + self.sdk_result_summary = agent._summarize_result(message) new_session_id = getattr(message, "session_id", None) if self.sdk_result_summary is not None and self.sdk_result_summary.is_error: - self.log.debug( - "is_error ResultMessage; not advancing session_id (kept %s)", - self._agent._session_id, - ) + self.log.debug("is_error ResultMessage; not advancing session_id (kept %s)", agent._session_id) else: - if new_session_id != self._agent._session_id: - self.log.debug("session_id changed: %s -> %s", self._agent._session_id, new_session_id) - self._agent._session_id = new_session_id - - # Retro-populate the last AssistantMessage from ResultMessage.usage when - # per-message capture was not in effect for THAT message (no message_id). - if self.last_assistant_message_index is not None and self.sdk_result_usage and not self.last_message_had_id: - last_msg = self.sdk_messages[self.last_assistant_message_index] - if isinstance(last_msg, AssistantMessageTelemetry): - last_msg.input_tokens = int(self.sdk_result_usage.get("input_tokens", 0) or 0) - last_msg.output_tokens = int(self.sdk_result_usage.get("output_tokens", 0) or 0) - last_msg.cache_creation_tokens = int(self.sdk_result_usage.get("cache_creation_input_tokens", 0) or 0) - last_msg.cache_read_tokens = int(self.sdk_result_usage.get("cache_read_input_tokens", 0) or 0) - last_msg.reasoning_tokens = int(self.sdk_result_usage.get("reasoning_tokens", 0) or 0) + if new_session_id != agent._session_id: + self.log.debug("session_id changed: %s -> %s", agent._session_id, new_session_id) + agent._session_id = new_session_id + + # Retro-populate the last generation from ResultMessage.usage when + # per-message capture was not in effect for it (no message_id). + last, usage = self.last_message, self.sdk_result_usage + if last is not None and usage and not self.last_message_had_id: + last.input_tokens = int(usage.get("input_tokens", 0) or 0) + last.output_tokens = int(usage.get("output_tokens", 0) or 0) + last.cache_creation_tokens = int(usage.get("cache_creation_input_tokens", 0) or 0) + last.cache_read_tokens = int(usage.get("cache_read_input_tokens", 0) or 0) + last.reasoning_tokens = int(usage.get("reasoning_tokens", 0) or 0) def _seed_first_generation_window(self) -> None: - """Move the first window's mark to the first observed model output. - - ONCE PER TURN, and that is the whole contract: re-seeding on every - ``message_start`` would stop the windows tiling and drop the gap before - the next emission into no bucket. The flag needs no reset — a fresh - ``_ClaudeTurnState`` is built per ``communicate()``. A turn with no - ``message_start`` keeps the turn-entry mark and clamps to ``0.0``, which is - the correct degradation. + """Move the first window's mark to the first observed model output, ONCE per turn. - One route to that degradation is OPERATOR-REACHABLE: - ``-D agent.sdk_options.include_partial_messages=false`` turns the raw - stream off and with it this re-seed. Nothing warns. + Re-seeding on every ``message_start`` would stop the windows tiling. A turn with + no ``message_start`` (including ``-D agent.sdk_options.include_partial_messages=false``) + keeps the turn-entry mark and clamps to ``0.0``, silently. Rationale: .claude/notes/agents.md § First-generation window seeding """ if self.first_output_seen: return self.first_output_seen = True - self.last_event_wall = self.clock.now() + self.last_event_wall = self.emitter.now() def on_stream_event(self, message: Message) -> None: - """Recover cumulative output_tokens from raw ``message_start`` / - ``message_delta`` stream events.""" + """Recover cumulative output_tokens from raw ``message_start`` / ``message_delta`` events.""" evt: dict[str, Any] = getattr(message, "event", None) or {} evt_type = evt.get("type") if evt_type == "message_start": @@ -566,65 +456,69 @@ def on_stream_event(self, message: Message) -> None: self.pending_delta_output_tokens = ot def on_user_message(self, message: Message) -> None: - """Process tool results (and a sub-agent's terminal generation) from a - tool-result UserMessage. The sub-agent message is appended BEFORE the - tool-result loop — its position in ``sdk_messages`` is observable. + """Add a sub-agent's terminal generation, then close each tool the message resolves. The generation mark is DELIBERATELY NOT advanced here: leaving it where ``on_assistant_message`` put it is what makes the windows tile. Rationale: .claude/notes/agents.md § Per-harness generation marks """ - sub_msg = self._agent._synthesize_subagent_terminal_message(message, self.sdk_model_used) - if sub_msg is not None: - self.sdk_messages.append(sub_msg) - + terminal = self._agent._subagent_terminal_part(message) + if terminal is not None: + tool_use_id, part = terminal + self.transcript.append( + self.emitter.add_unmeasured_generation( + message_id=f"subagent-{tool_use_id}", + part=part, + model=self.subagent_models.get(tool_use_id, self.sdk_model_used), + parent_tool_id=tool_use_id, + ) + ) content = getattr(message, "content", None) - if content and isinstance(content, list): - for block in content: - if _is_tool_result_block(block): - tool_name = "" - if block.tool_use_id in self.pending_commands: - tool_name = self.pending_commands[block.tool_use_id]["telemetry"].tool_name - self._agent._resolve_pending_command( - block.tool_use_id, - getattr(block, "is_error", False) or False, - block.content, - self.pending_commands, - self.processed_results, - now=self.clock.now(), - ) - is_error_flag = getattr(block, "is_error", False) or False - resolved = self.pending_commands.get(block.tool_use_id, {}).get("telemetry") - tool_for_event = resolved or CommandTelemetry( - tool_name=tool_name or "unknown", - tool_id=block.tool_use_id, - timestamp=self.clock.now(), - result_status="error" if is_error_flag else "success", - result_summary=format_payload(block.content), - ) - status = self._agent._tool_end_status(is_error_flag, block.content) - self.emitted_tool_ends.add(block.tool_use_id) - self.emit.on_event( - ToolEndEvent( - task_id=self.task_id, - turn_id=self.tool_turn_ids.get(block.tool_use_id, self.current_turn_id or ""), - tool=tool_for_event, - status=status, - ) - ) + if not isinstance(content, list): + return + for block in content: + if _is_tool_result_block(block): + self._close_tool(block.tool_use_id, bool(getattr(block, "is_error", False)), block.content) - def _finalize_token_usage(self) -> TokenUsage: - """Build the turn's cumulative TokenUsage, repricing for LiteLLM. + def _close_tool(self, tool_use_id: str, is_error: bool, content: Any) -> None: + if tool_use_id in self.processed_results: + self.log.debug("Multiple results for tool_id=%s; the first result stands.", tool_use_id) + return + self.processed_results.add(tool_use_id) + agent = self._agent + content_str = str(content) if content is not None else "" + status = agent._tool_end_status(is_error, content) + if tool_use_id not in self.opened_tools: + self.log.warning( + "Tool result received for unknown tool_use_id=%s. No matching ToolUseBlock found.", tool_use_id + ) + self.emitter.close_tool(tool_use_id, status=status, summary=format_payload(content)) + return + if status is ToolEndStatus.PERMISSION_DENIED: + self.log.warning( + "Tool use blocked: %s (id=%s) - permission denied. Error: %s", + self.opened_tools[tool_use_id], + tool_use_id, + content_str[:200], + ) + self.emitter.close_tool( + tool_use_id, + status=status, + summary=content_str or None, + error=content_str if is_error else None, + result_data=agent._try_parse_json_value(content), + ) - Extracted from ``finalize`` so the LiteLLM repricing *wiring*, not just - the helper, is directly testable. + def _finalize_token_usage(self) -> TokenUsage: + """The turn's cumulative TokenUsage, repriced for LiteLLM. Rationale: .claude/notes/agents.md § Cost: the stream versus the rate card """ + agent = self._agent usage = ( - self._agent._build_token_usage( - self.sdk_messages, + agent._build_token_usage( + self.transcript, self.sdk_result_usage, self.sdk_result_cost, self.sdk_result_model_usage, @@ -632,77 +526,65 @@ def _finalize_token_usage(self) -> TokenUsage: ) or TokenUsage() ) - if isinstance(self._agent.route, LiteLLMRoute): - self._agent._reprice_for_litellm(usage, self.effective_model) + if isinstance(agent.route, LiteLLMRoute): + agent._reprice_for_litellm(usage, self.effective_model) return usage - def finalize(self, status: AgentEndStatus, *, crashed: bool = False, crash_reason: str | None = None) -> None: - """Close orphaned tools + the open turn, emit the terminal AgentEndEvent, - and on a crash build the partial TurnRecord. Idempotent.""" - if self.finalized: - return - self.finalized = True - - commands = self._agent._finalize_commands(self.pending_commands, self.messages) - for cmd in commands: - if cmd.tool_id in self.emitted_tool_ends: - continue - self.emitted_tool_ends.add(cmd.tool_id) - self.emit.on_event( - ToolEndEvent( - task_id=self.task_id, - turn_id=self.tool_turn_ids.get(cmd.tool_id, self.current_turn_id or ""), - tool=cmd, - status=ToolEndStatus.UNRESOLVED, - ) + def end(self, status: AgentEndStatus, *, reason: str | None = None) -> TurnOutcome: + """Close the open inner turn with its unreported tokens and end the turn.""" + if self.current_turn_id is not None and self.emitter.inner_turn_open: + self.emitter.end_inner_turn( + TurnEndStatus(status.value), tokens=self.unreported_turn_tokens(self.current_turn_id) ) - - if self.current_turn_id is not None: - self.emit.on_event( - TurnEndEvent( - task_id=self.task_id, - turn_id=self.current_turn_id, - status=TurnEndStatus(status.value), - tokens=self.unreported_turn_tokens(self.current_turn_id), - ) + unresolved = sorted(set(self.opened_tools) - self.processed_results) + if unresolved: + counts: dict[str, int] = {} + for msg in self.messages: + counts[type(msg).__name__] = counts.get(type(msg).__name__, 0) + 1 + self.log.warning( + "Turn ended with %d tool call(s) without a result (%s). Messages received: [%s].", + len(unresolved), + ", ".join(unresolved), + ", ".join(f"{k}={v}" for k, v in sorted(counts.items())), ) - self.current_turn_id = None - usage = self._finalize_token_usage() - try: agent_output = self._agent._format_messages(self.messages) except Exception as fmt_err: logger.warning("Failed to format messages for AgentEndEvent; using placeholder", exc_info=True) agent_output = f"" - - self.emit.on_event( - AgentEndEvent( - task_id=self.task_id, - status=status, + if status is AgentEndStatus.CRASHED or status is AgentEndStatus.TIMEOUT: + return self.emitter.fail( + status, + reason or status.value, + usage=usage, + agent_output=agent_output, + model_used=self.sdk_model_used, + assistant_turn_count=self.assistant_turn_count, + num_turns=self.num_turns, + ) + if self.sdk_result_summary is None: + # A stop broke the loop before any ResultMessage: the emitter's default summary applies. + return self.emitter.finalize( + status, usage=usage, - iteration=self.iteration, - user_input=self.user_input, agent_output=agent_output, model_used=self.sdk_model_used, assistant_turn_count=self.assistant_turn_count, - messages=list(self.sdk_messages), num_turns=self.num_turns, - result_summary=self.sdk_result_summary, - crashed=crashed, - crash_reason=crash_reason, - duration_seconds=time.monotonic() - self.turn_start_time, - # One basis with the window bounds — see the AgentStartEvent - # site in `communicate`. - timestamp=self.clock.now(), ) + return self.emitter.finalize( + status, + usage=usage, + agent_output=agent_output, + model_used=self.sdk_model_used, + assistant_turn_count=self.assistant_turn_count, + num_turns=self.num_turns, + result_summary=self.sdk_result_summary, ) - if crashed: - self._agent._capture_partial_turn(self.collector) - -@AgentRegistry.register(AgentKind.CLAUDE_CODE, ClaudeCodeAgentConfig) +@AgentRegistry.register(AgentKind.CLAUDE_CODE, ClaudeCodeAgentConfig, spi_version=SPI_VERSION) class ClaudeCodeAgent(Agent[ClaudeCodeAgentConfig]): """Implementation of the Agent interface for Claude Code using the SDK.""" @@ -715,7 +597,9 @@ class ClaudeCodeAgent(Agent[ClaudeCodeAgentConfig]): allowed_tools=Enforcement.ENFORCED, disallowed_tools=Enforcement.ENFORCED, cooperative_stop=True, + reports_cost=True, usage_granularity=UsageGranularity.GENERATION, + timing_basis=TimingBasis.TURN_CLOCK, permission_modes=frozenset(PermissionMode), ) tool_names = ToolNameMap(names={name: (name,) for name in CANONICAL_TOOL_NAMES}, mcp_names=True) @@ -757,7 +641,6 @@ def __init__( self._extra_mcp_servers = extra_mcp_servers or {} self.client: ClaudeSDKClient | None = None self.working_directory: Path | None = None - # Turn-lifecycle bookkeeping lives on the Agent base class. self._sdk_options_dump: dict[str, Any] | None = None self._session_id: str | None = None # Held only while a communicate() call is in flight, so kill() can reach @@ -787,7 +670,7 @@ async def start( subprocess (typically the resolved ``SandboxConfig.mock_path_dirs``). plugin_tools_dir: Canonical ``node_modules/@uipath`` to export as ``PLUGIN_TOOLS_DIR``. An external env-var pin still wins. - plugin_root: The staged plugin root, loaded as one local plugin. + plugin_root: The staged plugin root; each ``plugins/`` is loaded as one local plugin. """ self.working_directory = Path(working_directory) self._env_path_prepend = list(env_path_prepend or []) @@ -937,227 +820,149 @@ async def communicate( self, user_input: str, *, + iteration: int, stream_callback: StreamCallback | None = None, timeout: float | None = None, should_stop: Callable[[], StopReason | None] | None = None, - ) -> TurnRecord: - """Send a message to Claude and receive its response. - - Args: - user_input: The message/prompt to send - stream_callback: Optional callback for real-time event streaming - timeout: Hard wall-clock deadline in seconds. A watchdog force-kills - the CLI subprocess when it elapses — the SDK's anyio task groups - suppress cooperative cancellation, so `asyncio.wait_for` is not - sufficient. - should_stop: The run's stop poll, checked after each dispatched message; - the first reason finalizes cleanly with ``end_status_for(reason)`` - (``crashed=False``, no raise) at that boundary. - - Returns: - TurnRecord containing the complete interaction + ) -> TurnOutcome: + """Run one turn; see ``Agent.communicate``. - Raises: - RuntimeError: If agent is not started. - TurnTimeoutError: Watchdog/wall-clock fired; carries a partial TurnRecord. - AgentCrashError: SDK/CLI failed mid-turn; carries a partial TurnRecord. + ``timeout`` arms a watchdog that force-kills the CLI subprocess: the SDK's anyio + task groups suppress cooperative cancellation. ``should_stop`` is polled after + each dispatched message. """ if not self.working_directory: raise RuntimeError("Agent not started. Call start() first.") - - # Every constructor path sets it; assert so the streaming-event sites - # below can use `str(self.config.type)`. assert self.config.type is not None, "ClaudeCodeAgent requires AgentConfig.type to be set before communicate()" - - # Reset the pending slot + bump the iteration counter (shared lifecycle). - self._begin_turn() - - turn_start_time = time.monotonic() - deadline = turn_start_time + timeout if timeout is not None else None - - # The agent is the SOLE emitter: events fan out to an internal - # EventCollector and the caller's stream_callback. task_id = str(self.config.type) # str() so a plugin subclass with a non-enum kind also works - collector = EventCollector() - emit = CompositeStreamCallback([c for c in (collector, stream_callback) if c is not None]) - - # Built BEFORE the try so except/finally can finalize even when setup - # crashes. `timeout_hit` is written by both the in-loop deadline break and - # the watchdog callback; bool assignment is atomic under the GIL. - state = _ClaudeTurnState( - self, - emit=emit, - collector=collector, - task_id=task_id, - user_input=user_input, - iteration=self._iteration, - log=self._log, - turn_start_time=turn_start_time, - deadline=deadline, - ) - # STAYS a communicate local: it is wired into the SDK options during - # setup, which the state (built first) would order-invert. stderr_lines: list[str] = [] - - def capture_stderr(line: str) -> None: - stderr_lines.append(line) - try: - options, transport, effective_model = self._build_claude_query(user_input, timeout, capture_stderr) - # Set on the state BEFORE the AgentStart emit and any finalize path - # (finalize reads it for cost backfill); stays None if setup crashed. - state.effective_model = effective_model - if transport is not None: - self._active_transport = transport - - # Agent lifecycle opens here (the agent — not the orchestrator — owns it). - emit.on_event( - AgentStartEvent( - task_id=task_id, - prompt=user_input, - iteration=self._iteration, - model=effective_model, - # From the TURN CLOCK, not the model's raw `datetime.now()` - # default: this bound is subtracted against window bounds the - # same clock produced, and two bases in one subtraction publish - # a clamped inversion as a measured 0.0 (CE058). - timestamp=state.clock.now(), - ) + options, transport, effective_model = self._build_claude_query( + user_input, iteration, timeout, stderr_lines.append ) + except Exception as e: + emitter = self._open_emitter( + prompt=user_input, iteration=iteration, model=None, task_id=task_id, stream_callback=stream_callback + ) + emitter.begin() + decoder = _ClaudeDecoder(self, emitter, effective_model=None) + return self._fail(decoder, AgentEndStatus.CRASHED, self._crash_message(e, None, stderr_lines)) - # Captured in the CLOSURE, not read from self._active_transport, so a - # stale watchdog from an earlier turn cannot kill this turn's process. - watchdog_target = transport - - def _on_turn_timeout() -> None: - state.timeout_hit = True - self._kill_transport(watchdog_target) - - # Only when one was built, so mocks with strict (prompt, options) - # signatures keep working on the no-timeout path. - query_kwargs: dict[str, Any] = {"prompt": user_input, "options": options} - if transport is not None: - query_kwargs["transport"] = transport + emitter = self._open_emitter( + prompt=user_input, + iteration=iteration, + model=effective_model, + task_id=task_id, + stream_callback=stream_callback, + ) + emitter.begin() + decoder = _ClaudeDecoder(self, emitter, effective_model=effective_model) + deadline = time.monotonic() + timeout if timeout is not None else None + + def _on_turn_timeout() -> None: + decoder.timeout_hit = True + # The CAPTURED transport, so a stale watchdog cannot kill a later turn's process. + self._kill_transport(transport) + + # Only when one was built, so mocks with strict (prompt, options) signatures keep working. + query_kwargs: dict[str, Any] = {"prompt": user_input, "options": options} + if transport is not None: + query_kwargs["transport"] = transport + self._active_transport = transport + timed_out = format_timeout_reason(timeout or 0) + try: self._log.debug("Starting agent query stream...") - # OS-thread watchdog: fires regardless of event-loop liveness, and is - # immune to anyio cancel-scope suppression. - with ThreadedWatchdog( + await run_with_watchdog( + self._pump_messages(decoder, query_kwargs, deadline, should_stop), timeout_seconds=timeout, on_timeout=_on_turn_timeout, - asyncio_task_to_cancel=asyncio.current_task(), label=f"Turn timeout ({timeout:g}s)" if timeout else "turn_timeout", - ): - await self._pump_messages(state, query_kwargs, deadline, should_stop) - + ) self._log.debug("Agent query stream ended") - + except WatchdogFired: + return self._fail(decoder, AgentEndStatus.TIMEOUT, timed_out) except asyncio.CancelledError: - # A cancel that landed BECAUSE of the timeout re-raises as - # TurnTimeoutError, so the retry system sees a terminal timeout rather - # than a transient cancel. External cancels propagate unchanged. - if self._timed_out(state.timeout_hit, deadline): - assert timeout is not None - self._finalize_and_raise_timeout(state.finalize, timeout) - # Cancelled from outside: park the telemetry on `pending_turn`, or the - # `finally` finalizes as COMPLETED and keeps no record. - if not state.finalized: - self._finalize_external_cancel(state.finalize) + caller = asyncio.current_task() + if caller is not None and caller.cancelling() == 0: + # Not a cancel from outside: the SDK raised it inside the turn body. + if self._timed_out(decoder.timeout_hit, deadline): + return self._fail(decoder, AgentEndStatus.TIMEOUT, timed_out) + return self._fail(decoder, AgentEndStatus.CRASHED, "Communication with agent failed: cancelled") + self._state = AgentState.ERROR + decoder.end(AgentEndStatus.CRASHED, reason="turn cancelled") raise - except ProcessError as e: - # A watchdog SIGKILL surfaces as ProcessError (exit -9); classify it as - # a timeout so the retry system does not treat it as AGENT_CRASH. - if self._timed_out(state.timeout_hit, deadline): - assert timeout is not None - self._finalize_and_raise_timeout(state.finalize, timeout, cause=e) - if not self._max_turns_short_circuit(state.sdk_result_summary, f"ProcessError(exit={e.exit_code})"): - stderr = self._build_stderr_message(e.stderr, stderr_lines) - error_info = self._format_error_summary(state.sdk_result_summary) - detail = error_info or stderr - message = f"CLI process failed (exit code {e.exit_code}): {detail}" - self._finalize_and_raise_crash(state.finalize, message, cause=e) except Exception as e: - # The SDK may re-raise a watchdog kill as a generic Exception. Check - # both the flag AND the wall clock, in case the flip races this catch. - if self._timed_out(state.timeout_hit, deadline): - assert timeout is not None - self._finalize_and_raise_timeout(state.finalize, timeout, cause=e) - if not self._max_turns_short_circuit(state.sdk_result_summary, "Generic Exception"): - # The SDK wraps ProcessError as a generic Exception via the - # message stream; read the ResultMessage summary for context. - error_info = self._format_error_summary(state.sdk_result_summary) - cause_stderr = self._extract_cause_stderr(e) - stderr = self._build_stderr_message(cause_stderr, stderr_lines) - error_details = self._clean_error_message(str(e)) - if error_info: - error_details += f"\nDetails: {error_info}" - elif stderr: - error_details += f"\nStderr output:\n{stderr}" - message = f"Communication with agent failed: {error_details}" - self._finalize_and_raise_crash(state.finalize, message, cause=e) + # A watchdog SIGKILL surfaces as ProcessError (exit -9) or a generic + # Exception; both the flag and the wall clock, in case the flip races this catch. + if self._timed_out(decoder.timeout_hit, deadline): + return self._fail(decoder, AgentEndStatus.TIMEOUT, timed_out) + label = f"ProcessError(exit={e.exit_code})" if isinstance(e, ProcessError) else "Generic Exception" + if not self._max_turns_short_circuit(decoder.sdk_result_summary, label): + return self._fail( + decoder, AgentEndStatus.CRASHED, self._crash_message(e, decoder.sdk_result_summary, stderr_lines) + ) finally: - # Auto-finalize any path the except blocks did not. Idempotent, so - # exactly one AgentEndEvent is produced on every exit path. - if not state.finalized: - if state.timeout_hit: - assert timeout is not None - state.finalize(AgentEndStatus.TIMEOUT, crashed=True, crash_reason=format_timeout_reason(timeout)) - elif state.stop_reason is not None: - # NOT a crash, NOT a timeout. - state.finalize(end_status_for(state.stop_reason), crashed=False, crash_reason=None) - else: - state.finalize(AgentEndStatus.COMPLETED, crashed=False, crash_reason=None) self._active_transport = None # Only the flag here, never the wall clock: a drift during post-loop # cleanup would misclassify a successful turn as a timeout. - if state.timeout_hit: - assert timeout is not None - raise TurnTimeoutError(timeout, iteration=self._iteration) - - self._update_state_from_messages(state.messages) - - # This turn completed successfully — the iteration increment stands. - self._end_turn_ok() - - # The collector's reduction of the events emitted above. - return collector.build_turn_record() + if decoder.timeout_hit: + return self._fail(decoder, AgentEndStatus.TIMEOUT, timed_out) + self._update_state_from_messages(decoder.messages) + status = end_status_for(decoder.stop_reason) if decoder.stop_reason is not None else AgentEndStatus.COMPLETED + return decoder.end(status) + + def _fail(self, decoder: _ClaudeDecoder, status: AgentEndStatus, reason: str) -> TurnOutcome: + self._state = AgentState.ERROR + return decoder.end(status, reason=reason) + + def _crash_message(self, error: Exception, summary: ResultSummary | None, stderr_lines: list[str]) -> str: + """The crash reason for ``error``: the errored result summary, else stderr.""" + error_info = self._format_error_summary(summary) + if isinstance(error, ProcessError): + detail = error_info or self._build_stderr_message(error.stderr, stderr_lines) + return f"CLI process failed (exit code {error.exit_code}): {detail}" + # The SDK wraps ProcessError as a generic Exception via the message stream. + details = self._clean_error_message(str(error)) + if error_info: + details += f"\nDetails: {error_info}" + else: + stderr = self._build_stderr_message(self._extract_cause_stderr(error), stderr_lines) + details += f"\nStderr output:\n{stderr}" + return f"Communication with agent failed: {details}" async def _pump_messages( self, - state: _ClaudeTurnState, + decoder: _ClaudeDecoder, query_kwargs: dict[str, Any], deadline: float | None, should_stop: Callable[[], StopReason | None] | None, ) -> None: - """Drive the SDK message stream for one turn (extracted from ``communicate``). - - Kept separate so the cooperative-stop check keeps ``communicate`` under - ruff's statement cap. ``query`` is still resolved as a module global at - call time, so ``patch("...claude_code_agent.query", ...)`` mocks work. - - Two break conditions, and the ORDER MATTERS: + """Drive the SDK message stream for one turn. - - The wall-clock guard runs at the TOP, so an over-deadline message is - DISCARDED — no append, no events. Do NOT move it to a post-loop check. - - The cooperative stop runs AFTER ``state.dispatch(message)``, so the monitor - can flip its flag on THIS message and the next is never pulled. + ``query`` is resolved as a module global at call time, so + ``patch("...claude_code_agent.query", ...)`` mocks work. The ORDER of the two + breaks matters: the deadline guard runs at the TOP, so an over-deadline message + is DISCARDED; the cooperative stop runs AFTER dispatch, so the monitor can flip + its flag on THIS message and the next is never pulled. """ async for message in query(**query_kwargs): if deadline is not None and time.monotonic() > deadline: - state.timeout_hit = True + decoder.timeout_hit = True self._log.warning("Turn timeout reached mid-stream; breaking out of message loop") break - state.dispatch(message) + decoder(message) reason = should_stop() if should_stop is not None else None if reason is not None: - state.stop_reason = reason + decoder.stop_reason = reason self._log.debug("Stop requested (%s); ending message loop at this boundary", reason.value) break def _build_claude_query( self, user_input: str, + iteration: int, timeout: float | None, stderr_callback: Callable[[str], None], ) -> tuple[ClaudeAgentOptions, SubprocessCLITransport | None, str | None]: @@ -1171,14 +976,16 @@ def _build_claude_query( assert self.working_directory is not None # guaranteed by communicate's guard above plugins: list[SdkPluginConfig] = ( - [{"type": "local", "path": str(self._plugin_root)}] if self._plugin_root is not None else [] + [{"type": "local", "path": str(d)} for d in staged_plugin_dirs(self._plugin_root)] + if self._plugin_root is not None + else [] ) # Per-turn cost-correlation headers (LiteLLM only): the run/task tag plus # this turn's iteration, so the proxy-side cost log joins to the turn. cost_log_tags: dict[str, str] | None = None if self.cost_log_tags is not None: - cost_log_tags = {**self.cost_log_tags, "x-ce-iteration": str(self._iteration)} + cost_log_tags = {**self.cost_log_tags, "x-ce-iteration": str(iteration)} env, route_model = self._build_sdk_env( self.route, path_prepend=self._env_path_prepend, @@ -1267,6 +1074,15 @@ def _resolve_system_prompt(self) -> str | SystemPromptPreset: preset["append"] = self.config.system_prompt return preset + async def harness_version(self) -> str | None: + """The SDK version, and the version of the CLI it spawns: the bundled one, else ``claude`` on PATH.""" + import claude_agent_sdk + from claude_agent_sdk._cli_version import __cli_version__ + + bundled = Path(claude_agent_sdk.__file__).parent / "_bundled" / ("claude.exe" if os.name == "nt" else "claude") + cli = __cli_version__ if bundled.is_file() else await command_version(["claude", "--version"]) + return f"claude-agent-sdk {claude_agent_sdk.__version__}; Claude Code {cli or 'unknown'}" + def get_environment_info(self) -> dict[str, Any]: """Record which system-prompt regime built this run's prompts. @@ -1335,42 +1151,6 @@ def _kill_transport(transport: SubprocessCLITransport | None) -> None: with suppress(OSError): proc.kill() - def _finalize_commands( - self, pending_commands: dict[str, dict[str, Any]], messages: list[Message] - ) -> list[CommandTelemetry]: - """Convert pending commands to a finalized list, marking unresolved ones as unknown.""" - commands: list[CommandTelemetry] = [] - unknown_status_count = 0 - - for tool_id, cmd_data in pending_commands.items(): - cmd = cmd_data["telemetry"] - if cmd.result_status is None: - # Unknown status and unknown duration are one fact: nothing - # resolved this command, so nothing timed it. `duration_ms` is - # deliberately left None (CE058). - # Rationale: .claude/notes/agents.md § Why only a RESOLVED tool is timed - cmd.result_status = "unknown" - unknown_status_count += 1 - self._log.warning( - f"Command {cmd.tool_name}:{tool_id} completed without tool result. " - + "Status set to 'unknown'. This may indicate agent interruption or SDK issue." - ) - commands.append(cmd) - - if unknown_status_count > 0: - msg_type_counts: dict[str, int] = {} - for msg in messages: - type_name = type(msg).__name__ - msg_type_counts[type_name] = msg_type_counts.get(type_name, 0) + 1 - type_summary = ", ".join(f"{k}={v}" for k, v in sorted(msg_type_counts.items())) - self._log.warning( - f"Turn completed with {unknown_status_count} command(s) in 'unknown' status. " - + f"Messages received: [{type_summary}]. " - + "This may indicate an SDK message type mismatch or agent interruption." - ) - - return commands - @staticmethod def _aggregate_model_usage(model_usage: dict[str, Any] | None) -> TokenUsage | None: """Sum the SDK ResultMessage ``model_usage`` into a cumulative TokenUsage. @@ -1474,36 +1254,16 @@ def _build_token_usage( model, ) - @staticmethod - def _price_from_buckets(usage: TokenUsage, model: str | None) -> float | None: - """Price the four token buckets at ``model``'s list rate. - - ``None`` when ``model`` is unset or absent from the rate card. - """ - if not model: - return None - return calculate_cost( - model, - uncached_input_tokens=usage.uncached_input_tokens, - output_tokens=usage.output_tokens, - cache_creation_tokens=usage.cache_creation_input_tokens, - cache_read_tokens=usage.cache_read_input_tokens, - ) - @staticmethod def _backfill_cost(usage: TokenUsage, model: str | None) -> TokenUsage: - """Price the token buckets when the SDK gave no cost (timeout / kill). + """Price the turn in place with ``pricing.price_turn`` and return it. A timed-out or killed turn has no terminal ``ResultMessage``, so the cost - is absent even though the tokens are fully captured. A no-op when the cost - is already set or the model is unpriced. + is absent even though the tokens are fully captured; the rate card fills it. """ - if usage.total_cost_usd is not None or not model: - return usage - cost = ClaudeCodeAgent._price_from_buckets(usage, model) - if cost is not None: - usage.total_cost_usd = cost - else: + reported = usage.total_cost_usd + usage.total_cost_usd = price_turn(usage, (model,)) + if usage.total_cost_usd is None and reported is None and model and not usage.is_empty(): # Not in the rate card, so the turn reverts to a null cost. Surface # it, or a stale pricing table silently reads as "Cost = —". logger.warning("No pricing for model %r; timeout/kill turn cost left unset", model) @@ -1522,9 +1282,9 @@ def _reprice_for_litellm(usage: TokenUsage, model: str | None) -> None: Rationale: .claude/notes/agents.md § Cost: the stream versus the rate card """ - cost = ClaudeCodeAgent._price_from_buckets(usage, model) + cost = price_turn(usage.model_copy(update={"total_cost_usd": None}), (model,)) usage.total_cost_usd = cost - if cost is None: + if cost is None and not usage.is_empty(): logger.warning("No pricing for litellm model %r; turn cost left unset", model) def get_sdk_options(self) -> dict[str, Any] | None: @@ -1585,7 +1345,7 @@ def _try_parse_json_value(content: Any) -> dict[str, Any] | list[Any] | None: @classmethod def _tool_end_status(cls, is_error: bool, content: Any) -> ToolEndStatus: - """Classify a tool result into a ToolEndStatus (promotes the old string-scan).""" + """Classify a tool result into a ToolEndStatus.""" if not is_error: return ToolEndStatus.OK text = str(content).lower() if content is not None else "" @@ -1594,21 +1354,15 @@ def _tool_end_status(cls, is_error: bool, content: Any) -> ToolEndStatus: return ToolEndStatus.ERROR @staticmethod - def _synthesize_subagent_terminal_message(message: Any, model: str | None) -> AssistantMessageTelemetry | None: - """Materialize a sub-agent's TERMINAL generation as an AssistantMessage. + def _subagent_terminal_part(message: Any) -> tuple[str, Generation] | None: + """A sub-agent's TERMINAL generation, as ``(spawning tool_use_id, part)``. A sub-agent's intermediate generations bubble into the parent stream as - ``parent_tool_use_id``-tagged messages, but its terminal one is delivered - as the Agent tool RESULT and never streamed. Synthesizing it puts the - sub-agent's full lifecycle in the transcript, so per-sub-agent usage is - recoverable by grouping on that id. - - ``tool_use_result.usage`` is the terminal call's own breakdown — complete, - and terminal-only, so it does NOT overlap the bubbled intermediates. - Returns None for a non-sub-agent tool result (no ``agentId``). - - The token total is unaffected: it derives from ``model_usage``, which - ignores this transcript, so the synthetic message is purely additive. + ``parent_tool_use_id``-tagged messages, but its terminal one is delivered as the + Agent tool RESULT and never streamed, so it has no window to measure. + ``tool_use_result.usage`` is that call's own breakdown and does not overlap the + bubbled intermediates. None for a non-sub-agent tool result (no ``agentId``). + The token total is unaffected: it derives from ``model_usage``. """ tur = getattr(message, "tool_use_result", None) if not isinstance(tur, dict) or "agentId" not in tur: @@ -1637,98 +1391,16 @@ def _int(value: Any) -> int: except (TypeError, ValueError): return 0 - # Never streamed, so no window exists to measure: None (unknown), not 0.0. - # These bounds are an admitted PLACEHOLDER, which is why they are - # deliberately NOT on the turn's `TurnClock` — the only wall stamp in this - # harness that is not. `generation_duration_ms is None` plus a set - # `parent_tool_use_id` is exactly what excludes this message from the - # subtraction and the head/tail bracket, so the stamp is read by no - # arithmetic and has no basis to share. - now = datetime.now() - return AssistantMessageTelemetry( - started_at=now, - completed_at=now, - generation_duration_ms=None, - content_blocks=([ContentBlock(block_type="text", sequence=0, text=result_text)] if result_text else []), - tool_use_ids=[], - input_tokens=_int(usage.get("input_tokens")), - output_tokens=_int(usage.get("output_tokens")), - cache_creation_tokens=_int(usage.get("cache_creation_input_tokens")), - cache_read_tokens=_int(usage.get("cache_read_input_tokens")), - reasoning_tokens=0, - model=model, - message_id=f"subagent-{tool_use_id}", - parent_tool_use_id=tool_use_id, + return tool_use_id, Generation( + blocks=[ContentBlock(block_type="text", sequence=0, text=result_text)] if result_text else [], + tokens=TokenUsage( + uncached_input_tokens=_int(usage.get("input_tokens")), + output_tokens=_int(usage.get("output_tokens")), + cache_creation_input_tokens=_int(usage.get("cache_creation_input_tokens")), + cache_read_input_tokens=_int(usage.get("cache_read_input_tokens")), + ), ) - def _resolve_pending_command( - self, - tool_use_id: str, - is_error: bool, - content: Any, - pending_commands: dict[str, dict[str, Any]], - processed_results: set[str], - *, - now: datetime, - ) -> None: - """Match a tool result back to its pending command and update status/duration. - - Args: - tool_use_id: The tool use ID from the result - is_error: Whether the tool execution resulted in an error - content: The result content (string or structured) - pending_commands: Map of tool_id -> {telemetry, command_start_time} - processed_results: Set of already-processed tool IDs (for duplicate detection) - now: This turn's ``TurnClock`` reading, passed in rather than read - here: the span stamped below is clipped against window bounds the - same clock produced. - """ - # Normalize content to string for storage - content_str = str(content) if content is not None else "" - - if tool_use_id in pending_commands: - cmd_data = pending_commands[tool_use_id] - cmd = cmd_data["telemetry"] - command_start_time = cmd_data["command_start_time"] - - # Calculate precise duration - command_end_time = time.monotonic() - duration_ms = (command_end_time - command_start_time) * 1000 - - # Update command with actual results - cmd.result_status = "error" if is_error else "success" - cmd.duration_ms = duration_ms - cmd.result_summary = content_str if content_str else None - cmd.result_data = ClaudeCodeAgent._try_parse_json_value(content) - - # `execution_started_at` is RECONSTRUCTED by subtracting the measured - # monotonic duration from the turn clock's reading, which is exact - # because the turn clock is itself monotonic-derived. - cmd.execution_completed_at = now - cmd.execution_started_at = cmd.execution_completed_at - timedelta(milliseconds=duration_ms) - - if is_error: - cmd.error_message = content_str - - # Abnormal flow: warn so it surfaces without DEBUG enabled. - content_lower = content_str.lower() - if any( - phrase in content_lower - for phrase in ("permission", "not allowed", "requires approval", "denied", "blocked") - ): - self._log.warning( - f"Tool use blocked: {cmd.tool_name} (id={tool_use_id}) " - + f"- permission denied. Error: {content_str[:200]}" - ) - - if tool_use_id in processed_results: - self._log.debug(f"Multiple results for tool_id={tool_use_id}. Last result wins.") - processed_results.add(tool_use_id) - else: - self._log.warning( - f"Tool result received for unknown tool_use_id={tool_use_id}. No matching ToolUseBlock found." - ) - @staticmethod def _build_stderr_message(sdk_stderr: str | None, stderr_lines: list[str]) -> str: """Combine SDK stderr with captured stderr lines, filtering out placeholder text. diff --git a/src/coder_eval/agents/codex_agent.py b/src/coder_eval/agents/codex_agent.py index 5e041b41..99d49be6 100644 --- a/src/coder_eval/agents/codex_agent.py +++ b/src/coder_eval/agents/codex_agent.py @@ -8,7 +8,6 @@ import shlex import shutil import tempfile -import time from collections.abc import Callable from datetime import datetime from pathlib import Path @@ -17,47 +16,28 @@ from coder_eval.agent import Agent, AgentState from coder_eval.agents._logging import PrefixedAdapter, log_raw_sdk_event -from coder_eval.agents.registry import AgentRegistry -from coder_eval.agents.watchdog import ThreadedWatchdog +from coder_eval.agents.registry import SPI_VERSION, AgentRegistry +from coder_eval.agents.watchdog import WatchdogFired, run_with_watchdog from coder_eval.config import settings -from coder_eval.errors import ( - AgentCrashError, - TurnTimeoutError, - truncate_crash_message, -) +from coder_eval.errors.agent import format_timeout_reason from coder_eval.models import ( AgentKind, ApiRoute, AssistantMessage, CodexAgentConfig, - CommandTelemetry, ContentBlock, DirectRoute, Enforcement, HarnessContract, + TimingBasis, TokenUsage, - TranscriptMessage, - TurnRecord, UsageGranularity, ) from coder_eval.orchestration.plugin_staging import link_or_copy -from coder_eval.pricing import calculate_cost -from coder_eval.streaming.callbacks import CompositeStreamCallback, StreamCallback -from coder_eval.streaming.collector import EventCollector -from coder_eval.streaming.events import ( - AgentEndEvent, - AgentEndStatus, - AgentStartEvent, - StopReason, - TextChunkEvent, - ToolEndEvent, - ToolEndStatus, - ToolStartEvent, - TurnEndEvent, - TurnEndStatus, - TurnStartEvent, - end_status_for, -) +from coder_eval.pricing import price_turn +from coder_eval.streaming.callbacks import StreamCallback +from coder_eval.streaming.emitter import Generation, TurnEmitter, TurnOutcome +from coder_eval.streaming.events import AgentEndStatus, StopReason, ToolEndStatus, TurnEndStatus, end_status_for from coder_eval.timing import close_window @@ -140,31 +120,34 @@ _STREAM_DONE = object() -def _ms_to_dt(ms: int | None) -> datetime: - """Convert a Codex Unix-millisecond timestamp to a datetime (now() if absent).""" - if ms is None: - return datetime.now() +def _ms_to_dt(ms: int) -> datetime: + """Convert a Codex Unix-millisecond timestamp to a datetime.""" return datetime.fromtimestamp(ms / 1000) class _ItemTiming(NamedTuple): - """When a Codex tool item ran, as the four fields CommandTelemetry records.""" + """When a Codex tool item ran: the CLI bounds, and the duration they or the SDK give.""" - timestamp: datetime execution_started_at: datetime | None execution_completed_at: datetime | None duration_ms: float | None +class _ToolEnd(NamedTuple): + """What a completed Codex tool item tells ``TurnEmitter.close_tool``.""" + + is_error: bool + summary: str | None + error: str | None + result_data: Any + parameters: dict[str, Any] + timing: _ItemTiming + + def _item_timing(started_ms: int | None, completed_ms: int | None, sdk_duration_ms: float | None) -> _ItemTiming: """Resolve a tool item's timing from the SDK's millisecond stamps. - One helper for all three telemetry builders, so a command, a file change and - an MCP call cannot disagree about what a missing stamp means. - - BOTH stamps or neither: pairing a real stamp with ``_ms_to_dt(None)`` — which - is ``datetime.now()`` — fabricates an interval out of one reading and the - current time, so the raw values are checked BEFORE conversion. + BOTH stamps or neither: one stamp is no interval. Without them, the SDK item's own ``duration_ms`` is used only when it reports something. A ``0`` there is an UNREPORTED duration, not an instant command (70 @@ -178,7 +161,6 @@ def _item_timing(started_ms: int | None, completed_ms: int | None, sdk_duration_ if started_ms is not None and completed_ms is not None: started = _ms_to_dt(started_ms) return _ItemTiming( - timestamp=started, execution_started_at=started, execution_completed_at=_ms_to_dt(completed_ms), # Clamped for clock skew; both bounds stay as reported so the @@ -189,7 +171,6 @@ def _item_timing(started_ms: int | None, completed_ms: int | None, sdk_duration_ # CE058 coalesce read backwards, and hides the decision being made. duration = None if sdk_duration_ms is None or sdk_duration_ms <= 0 else float(sdk_duration_ms) return _ItemTiming( - timestamp=datetime.now(), execution_started_at=None, execution_completed_at=None, duration_ms=duration, @@ -241,6 +222,18 @@ def since(self, baseline: "_ThreadTotals") -> "_ThreadTotals": ) +def _generation_tokens(last: Any) -> TokenUsage: + """One generation's usage from the SDK's ``last`` breakdown; empty when ``last`` is None.""" + if not last: + return TokenUsage() + cached = getattr(last, "cached_input_tokens", 0) or 0 + return TokenUsage( + uncached_input_tokens=_fresh_input_tokens(getattr(last, "input_tokens", 0) or 0, cached), + output_tokens=getattr(last, "output_tokens", 0) or 0, + cache_read_input_tokens=cached, + ) + + def _message_uncached_input(m: AssistantMessage) -> int: """A captured generation's fresh (uncached) input. @@ -279,225 +272,58 @@ def _get_item_root(notification: Any) -> Any: return getattr(item, "root", None) -class _CodexTurnState: - """Per-turn mutable scratch state for one ``CodexAgent.communicate`` call. - - Holds the stream-pump locals and the transcript reconstruction buffers, with - one method per notification kind plus ``dispatch`` (True on ``turn/completed`` - to break the pump), ``_flush_message`` and ``finalize``. - - ``commands`` and ``messages`` are the SAME list objects ``communicate`` owns, - held by identity (no copy), so a mid-turn crash keeps the partial transcript. +class _CodexDecoder: + """One turn's reducer: Codex SDK notifications in, ``TurnEmitter`` calls out. - The finalize inputs are COMMITTED by ``communicate`` only after the pump - returns cleanly, defaulting to None/None/"" — so a crashed turn finalizes from - the captured messages, and the live pump scratch is intentionally NOT what - finalize reads. + Timing is the SDK's own epoch milliseconds. A generation is cut at each + ``thread/tokenUsage/updated``; its window runs from the previous cut to the last + item's completion, and it is split into a thinking and an action sub-message. + Each generation is one inner turn: opened at its first item or text, closed at + its cut with the SDK's ``last`` delta as the turn's tokens. """ - def __init__( - self, - agent: "CodexAgent", - *, - emit: CompositeStreamCallback, - task_id: str, - turn_id: str, - collector: EventCollector, - commands: list[CommandTelemetry], - messages: list[TranscriptMessage], - user_input: str, - iteration: int, - turn_start_time: float, - ) -> None: + def __init__(self, agent: "CodexAgent", emitter: TurnEmitter, *, turn_id: str) -> None: self._agent = agent - self.emit = emit - self.task_id = task_id + self.emitter = emitter self.turn_id = turn_id - self.collector = collector - self.commands = commands - self.messages = messages - self.user_input = user_input - self.iteration = iteration - self.turn_start_time = turn_start_time self.timeout_hit = False self.stop_reason: StopReason | None = None - self.finalized = False - - # Live pump scratch (set during streaming). self.turn_result: Any = None self.latest_token_usage: Any = None self.agent_message_chunks: list[str] = [] - # Sequence per executable item, assigned at item/started and reused at - # item/completed via this id->seq map. - self.next_sequence = 0 - self.seq_by_id: dict[str, int] = {} + # Every AssistantMessage this turn added, in order; the crash fallback sums them. + self.messages: list[AssistantMessage] = [] + self.opened_tools: set[str] = set() + # An id-less tool item's id, minted once at item/started, per item type in order. + self._pending_idless: dict[str, list[str]] = {} + self._minted = 0 # Spawned sub-agent child thread id -> spawning Agent call tool_use_id. self.collab_spawn_by_thread: dict[str, str] = {} # (child_thread_id, spawning Agent tool_use_id, spawned model) per spawn. self.spawned_children: list[tuple[str, str, str | None]] = [] # child thread id -> returned message (fallback when the rollout is absent). self.collab_results: dict[str, str] = {} - - # Assistant-transcript reconstruction buffers (one AssistantMessage per gen). self.open_blocks: list[ContentBlock] = [] self.open_start_ms: int | None = None self.open_end_ms: int | None = None # Where the NEXT generation window starts: the previous flush's end. None - # until the first flush, which falls back to its own first item — the SDK - # gives no "turn began" stamp, and inventing one from time.time() would - # mix our clock with the SDK's inside one subtraction. + # until the first flush, which falls back to its own first item. # Rationale: .claude/notes/agents.md § Per-harness generation marks self.gen_mark_ms: int | None = None self.start_ms_by_id: dict[str, int] = {} self.blocks_by_id: dict[str, ContentBlock] = {} - # Tools that emitted item/started but not item/completed; whatever remains - # at turn end is an orphan, force-closed unresolved. - self.open_tools: dict[str, CommandTelemetry] = {} - # Text-less reasoning blocks, resolved at flush once reasoning tokens known. + # Text-less reasoning blocks, resolved at flush once reasoning tokens are known. self.reasoning_placeholders: list[ContentBlock] = [] self.gen_index = 0 - - # Finalize inputs, COMMITTED by communicate after a clean pump return. - # Defaults are the crash values (no terminal usage; format from messages). - self.sdk_token_usage: Any = None - self.result_turn: Any = None - self.result_text: str = "" - - def _record_block(self, block: ContentBlock, item_id: str, completed_ms: int | None) -> None: - self.open_blocks.append(block) - start_ms = self.start_ms_by_id.get(item_id) - if start_ms is not None and (self.open_start_ms is None or start_ms < self.open_start_ms): - self.open_start_ms = start_ms - if completed_ms is not None and (self.open_end_ms is None or completed_ms > self.open_end_ms): - self.open_end_ms = completed_ms - - def _flush_message(self, last: Any) -> None: - """Cut the open buffer into AssistantMessage(s) for one generation. - - ``last`` is the SDK breakdown for the generation that produced these - blocks (None for a safety flush). Emits ONE sub-message per block kind, - all sharing this generation's ``message_id``. - - Rationale: .claude/notes/agents.md § Why the generation is split into sub-messages - """ - if not self.open_blocks: - self.reasoning_placeholders = [] - return - # Per-generation tokens from the matching tokenUsage `last` delta. - cached = (getattr(last, "cached_input_tokens", 0) or 0) if last else 0 - raw_input = (getattr(last, "input_tokens", 0) or 0) if last else 0 - total_output = (getattr(last, "output_tokens", 0) or 0) if last else 0 - # OpenAI bills no separate cache-write fee, so cache_creation is 0. - gen_input = _fresh_input_tokens(raw_input, cached) - gen_cache_write = 0 - reasoning_tok = (getattr(last, "reasoning_output_tokens", 0) or 0) if last else 0 - # A text-less reasoning block becomes a placeholder when reasoning was - # billed, and is dropped otherwise. - if self.reasoning_placeholders: - if reasoning_tok > 0: - for blk in self.reasoning_placeholders: - blk.thinking = "_Reasoning hidden by OpenAI policy_" - else: - for blk in self.reasoning_placeholders: - if blk in self.open_blocks: - self.open_blocks.remove(blk) - self.reasoning_placeholders = [] - if not self.open_blocks: - self.open_start_ms = self.open_end_ms = None - return - - thinking_blocks = [b for b in self.open_blocks if b.block_type == "thinking"] - action_blocks = [b for b in self.open_blocks if b.block_type != "thinking"] - # From the PREVIOUS flush's end, not this generation's first item: the - # SDK stamps an item with the moment it began EXECUTING, so seeding there - # discards the model time that produced it. - mark_ms = self.gen_mark_ms if self.gen_mark_ms is not None else self.open_start_ms - window_end_ms = self.open_end_ms if self.open_end_ms is not None else self.open_start_ms - mark = _ms_to_dt(mark_ms) - completed = _ms_to_dt(window_end_ms) - # The RAW window, extended to the LAST item's completion — so a - # generation containing a tool call already CONTAINS its execution, and - # the collector takes it back out. That is also what makes the sub-message - # split safe: the two specs SHARE these bounds, so the overlap is - # subtracted once rather than once per part. - started, gen_ms = close_window( - mark=mark, - now=completed, - item_start=_ms_to_dt(self.open_start_ms) if self.open_start_ms is not None else None, - ) - message_id = f"{self.turn_id}-msg-{self.gen_index}" - - # Output split: reasoning portion to the thinking row, the remainder to - # the action row. With only one kind present, that kind gets all output. - think_out = reasoning_tok if action_blocks else total_output - action_out = max(total_output - reasoning_tok, 0) if thinking_blocks else total_output - - # Thinking first. The FIRST carries the gen's input/cache: per-CALL - # billing figures that must not be split. Generation TIME is a property - # of the content, so it IS apportioned below. - specs: list[tuple[list[ContentBlock], int, int]] = [] - if thinking_blocks: - specs.append((thinking_blocks, think_out, reasoning_tok)) - if action_blocks: - specs.append((action_blocks, action_out, 0)) - - # By OUTPUT-TOKEN share, the last taking the remainder so the parts - # reconstruct gen_ms to float precision (shares round to 1e-6 ms, so do - # not assert exact equality on a measured window). With no output anywhere, - # split evenly. The evalboard's twin weighs by CONTENT SIZE instead, which - # is deliberate, not an oversight to unify. - out_total = sum(out_tok for _, out_tok, _ in specs) - gen_parts: list[float] = [] - assigned = 0.0 - for idx, (_, out_tok, _) in enumerate(specs): - if idx == len(specs) - 1: - gen_parts.append(gen_ms - assigned) - else: - share = round(gen_ms * (out_tok / out_total if out_total > 0 else 1 / len(specs)), 6) - gen_parts.append(share) - assigned += share - - for idx, (blocks, out_tok, reas_tok) in enumerate(specs): - for i, blk in enumerate(blocks): - blk.sequence = i - first = idx == 0 - self.messages.append( - AssistantMessage( - started_at=started, - completed_at=completed, - generation_duration_ms=gen_parts[idx], - content_blocks=blocks, - tool_use_ids=[b.tool_use_id for b in blocks if b.block_type == "tool_use" and b.tool_use_id], - input_tokens=gen_input if first else 0, - output_tokens=out_tok, - cache_creation_tokens=gen_cache_write if first else 0, - cache_read_tokens=cached if first else 0, - reasoning_tokens=reas_tok, - model=self._agent._effective_model(), - message_id=message_id, - ) - ) - self.gen_index += 1 - # A message was appended, so the next window starts where this one - # ended. Both early returns above leave the mark alone on purpose. - if window_end_ms is not None: - self.gen_mark_ms = window_end_ms - self.open_blocks = [] - self.open_start_ms = None - self.open_end_ms = None + self.subagent_index = 0 @property def ended_cleanly(self) -> bool: - """True once the pump broke on a ``should_stop`` reason. - - A non-crash termination, so an exception raised while tearing the stream - down afterwards must not be escalated into a retry. - """ + """True once the pump broke on a ``should_stop`` reason: a later teardown exception is not a crash.""" return self.stop_reason is not None - def dispatch(self, notification: Any) -> bool: - """Route a notification to its handler. Returns True on ``turn/completed`` - (a valid TurnCompletedNotification) so the pump loop breaks.""" + def __call__(self, notification: Any) -> bool: + """Route a notification; True on a valid ``turn/completed`` so the pump breaks.""" root = _get_item_root(notification) method = notification.method log_raw_sdk_event( @@ -519,83 +345,97 @@ def dispatch(self, notification: Any) -> bool: return self.on_turn_completed(notification) return False + def _open_generation(self) -> None: + """Open the inner turn for the generation now arriving; a no-op while one is open.""" + if not self.emitter.inner_turn_open: + self.emitter.begin_inner_turn(f"{self.turn_id}-msg-{self.gen_index}") + + def _record_block(self, block: ContentBlock, item_id: str, completed_ms: int | None) -> None: + self.open_blocks.append(block) + start_ms = self.start_ms_by_id.get(item_id) + if start_ms is not None and (self.open_start_ms is None or start_ms < self.open_start_ms): + self.open_start_ms = start_ms + if completed_ms is not None and (self.open_end_ms is None or completed_ms > self.open_end_ms): + self.open_end_ms = completed_ms + + def _tool_id(self, root: Any, root_type: str, *, starting: bool) -> str: + item_id = getattr(root, "id", None) + if item_id: + return str(item_id) + pending = self._pending_idless.setdefault(root_type, []) + if not starting and pending: + return pending.pop(0) + minted = f"{root_type}_{self._minted}" + self._minted += 1 + if starting: + pending.append(minted) + return minted + def on_item_started(self, notification: Any) -> None: - """Emit ToolStartEvent + record the tool_use block for every tool-like item.""" + """Open a tool call and record its tool_use block, for every tool-like item.""" root = _get_item_root(notification) if root is None: return - # Record the start time for every item kind so flushed messages get real timing. item_id = getattr(root, "id", None) started_at_ms = getattr(notification.payload, "started_at_ms", None) if item_id is not None and started_at_ms is not None: self.start_ms_by_id[item_id] = started_at_ms root_type = getattr(root, "type", None) - # Any item that isn't transcript content is a tool call (generic capture). - if root_type is not None and root_type not in _CONTENT_ITEM_TYPES: - tool_id = item_id or f"{root_type}_{self.next_sequence}" - self.seq_by_id[tool_id] = self.next_sequence - # Recorded on the START telemetry too: close_open_tools publishes - # this object verbatim for an orphan, and without it an unresolved - # call cannot be placed on a timeline at all. - started_at = _ms_to_dt(started_at_ms) if started_at_ms is not None else None - start_tel = CommandTelemetry( - tool_name=self._agent._tool_name(root_type), - tool_id=tool_id, - timestamp=started_at or datetime.now(), - execution_started_at=started_at, - parameters=self._agent._tool_parameters(root, root_type), - sequence_number=self.next_sequence, - ) - self.open_tools[tool_id] = start_tel - self.emit.on_event(ToolStartEvent(task_id=self.task_id, turn_id=self.turn_id, tool=start_tel)) - self.next_sequence += 1 - # is_error is patched at item/completed even after the message is - # flushed, because the block is held by reference. - block = ContentBlock(block_type="tool_use", sequence=0, tool_use_id=tool_id) - self.blocks_by_id[tool_id] = block - self._record_block(block, tool_id, None) + if root_type is None: + return + self._open_generation() + if root_type in _CONTENT_ITEM_TYPES: + return + tool_id = self._tool_id(root, root_type, starting=True) + if started_at_ms is not None: + self.start_ms_by_id[tool_id] = started_at_ms + self.opened_tools.add(tool_id) + self.emitter.open_tool( + tool_id, + self._agent._tool_name(root_type), + self._agent._tool_parameters(root, root_type), + started_at=_ms_to_dt(started_at_ms) if started_at_ms is not None else None, + ) + # is_error is patched at item/completed even after the message is flushed, + # because the block is held by reference. + block = ContentBlock(block_type="tool_use", sequence=0, tool_use_id=tool_id) + self.blocks_by_id[tool_id] = block + self._record_block(block, tool_id, None) def on_item_completed(self, notification: Any) -> None: - """Emit ToolEndEvent + capture telemetry / sub-agents / transcript blocks.""" + """Close a tool call, or capture a reasoning / agent message block.""" root = _get_item_root(notification) if root is None: return completed_ms = getattr(notification.payload, "completed_at_ms", None) root_type = getattr(root, "type", None) if root_type is not None and root_type not in _CONTENT_ITEM_TYPES: - tool_id = getattr(root, "id", None) or f"{root_type}_{self.next_sequence}" - seq = self.seq_by_id.get(tool_id, self.next_sequence) - # This tool is now resolved — drop it from the orphan set. - self.open_tools.pop(tool_id, None) - - # `on_item_started` banked the start; passing both in is what lets - # the builders record real bounds instead of the SDK's frequent 0. - telemetry, is_error = self._agent._telemetry_for_item( - root, - root_type, - tool_id, - seq, - started_ms=self.start_ms_by_id.get(tool_id), - completed_ms=completed_ms, - ) - if telemetry: - self.commands.append(telemetry) - self.emit.on_event( - ToolEndEvent( - task_id=self.task_id, - turn_id=self.turn_id, - tool=telemetry - or CommandTelemetry( - tool_name=self._agent._tool_name(root_type), - tool_id=tool_id, - timestamp=datetime.now(), - sequence_number=seq, - ), - status=ToolEndStatus.ERROR if is_error else ToolEndStatus.OK, + tool_id = self._tool_id(root, root_type, starting=False) + if tool_id not in self.opened_tools: + # A result landing after the cut is not a new model turn; an unseen call is. + self._open_generation() + self.opened_tools.add(tool_id) + self.emitter.open_tool( + tool_id, + self._agent._tool_name(root_type), + self._agent._tool_parameters(root, root_type), + started_at=None, ) + end = self._agent._tool_end_for_item( + root, root_type, started_ms=self.start_ms_by_id.get(tool_id), completed_ms=completed_ms + ) + is_error = end is not None and end.is_error + self.emitter.close_tool( + tool_id, + status=ToolEndStatus.ERROR if is_error else ToolEndStatus.OK, + summary=end.summary if end else None, + error=end.error if end else None, + result_data=end.result_data if end else None, + parameters=end.parameters if end else None, + completed_at=end.timing.execution_completed_at if end else None, + started_at=end.timing.execution_started_at if end else None, + reported_duration_ms=end.timing.duration_ms if end else None, ) - # Patch the block recorded at item/started, and extend the still-open - # message's end time. if tool_id in self.blocks_by_id: self.blocks_by_id[tool_id].is_error = is_error if ( @@ -604,58 +444,49 @@ def on_item_completed(self, notification: Any) -> None: and (self.open_end_ms is None or completed_ms > self.open_end_ms) ): self.open_end_ms = completed_ms - - # Codex's native multi-agent calls land here as collabAgentToolCall items. if root_type == "collabAgentToolCall": self._agent._handle_collab_completion( - root, - tool_id, - self.collab_spawn_by_thread, - self.spawned_children, - self.collab_results, + root, tool_id, self.collab_spawn_by_thread, self.spawned_children, self.collab_results ) - elif root_type == "reasoning": + self._open_generation() # OpenAI never returns raw CoT, so a text-less item becomes a # placeholder, resolved with its token count at flush. - reasoning_id = getattr(root, "id", f"reasoning_{self.next_sequence}") + reasoning_id = getattr(root, "id", f"reasoning_{len(self.open_blocks)}") parts = getattr(root, "content", None) or getattr(root, "summary", None) or [] text = "\n".join(p for p in parts if p) block = ContentBlock(block_type="thinking", sequence=0, thinking=text or None) self._record_block(block, reasoning_id, completed_ms) if not text: self.reasoning_placeholders.append(block) - elif root_type == "agentMessage": - # The message is cut at the following tokenUsage event (the - # generation boundary), not here. - message_item_id = getattr(root, "id", f"msg_{self.next_sequence}") + self._open_generation() + # Cut at the following tokenUsage event (the generation boundary), not here. + message_item_id = getattr(root, "id", f"msg_{len(self.open_blocks)}") text = getattr(root, "text", "") or "" if text: self._record_block( - ContentBlock(block_type="text", sequence=0, text=text), - message_item_id, - completed_ms, + ContentBlock(block_type="text", sequence=0, text=text), message_item_id, completed_ms ) def on_agent_message_delta(self, notification: Any) -> None: - """Emit TextChunkEvent for streaming assistant text.""" if notification.payload: delta = getattr(notification.payload, "delta", None) if delta: + self._open_generation() self.agent_message_chunks.append(delta) - self.emit.on_event(TextChunkEvent(task_id=self.task_id, turn_id=self.turn_id, text=delta)) + self.emitter.text(delta) def on_token_usage_updated(self, notification: Any) -> None: - """One per generation → cut a message. Carries `total` (cumulative over the - whole THREAD, i.e. every turn so far) and `last` (this generation's delta).""" + """One per generation: cut a message and close its inner turn. ``last`` is this generation's delta.""" if notification.payload: self.latest_token_usage = getattr(notification.payload, "token_usage", None) - self._flush_message(getattr(self.latest_token_usage, "last", None)) + last = getattr(self.latest_token_usage, "last", None) + if not _generation_tokens(last).is_empty(): + self._open_generation() + self.flush(last) def on_turn_completed(self, notification: Any) -> bool: - """Capture the final Turn. Returns True (break the pump) iff the payload is - a valid TurnCompletedNotification.""" from openai_codex.generated.v2_all import TurnCompletedNotification if isinstance(notification.payload, TurnCompletedNotification): @@ -663,85 +494,131 @@ def on_turn_completed(self, notification: Any) -> bool: return True return False - def close_open_tools(self) -> None: - """Force-close any tool that started but never completed (an orphan) as - ``unresolved``, so its transcript block keeps a real tool name + count.""" - for start_tel in sorted(self.open_tools.values(), key=lambda t: getattr(t, "sequence_number", 0)): - start_tel.result_status = "unknown" - self.emit.on_event( - ToolEndEvent( - task_id=self.task_id, - turn_id=self.turn_id, - tool=start_tel, - status=ToolEndStatus.UNRESOLVED, - ) - ) - self.open_tools.clear() + def flush(self, last: Any) -> None: + """Cut the open buffer into one generation and close its inner turn. + + ``last`` is the SDK breakdown for the generation; its delta becomes the inner + turn's tokens. ``None`` is a safety flush at the end of the pump: it adds the + message but leaves the inner turn open for the turn's own end status. - def finalize(self, status: AgentEndStatus, *, crashed: bool = False, crash_reason: str | None = None) -> None: - """Emit the terminal TurnEnd + AgentEnd and, on a crash, build the partial - TurnRecord. Idempotent. Reads the COMMITTED finalize inputs (None/"" on a - crash) so a crashed turn under-reports nothing it didn't actually commit.""" - if self.finalized: + Rationale: .claude/notes/agents.md § Why the generation is split into sub-messages + """ + before = self.gen_index + self._cut(last) + if last is None or not self.emitter.inner_turn_open: return - self.finalized = True + self.emitter.end_inner_turn(TurnEndStatus.COMPLETED, tokens=_generation_tokens(last)) + if self.gen_index == before: + # A billed cut with no message still spends its id, or the next turn would reuse it. + self.gen_index += 1 - # On crash/timeout the SDK total stays None, so fall back to the - # per-generation tokens on the messages — but the thread baseline still - # has to move past them, or the NEXT turn's delta re-books this one. - token_usage = self._agent._token_usage_from_sdk(self.sdk_token_usage) - if token_usage is None: - token_usage = self._agent._token_usage_from_messages(self.messages) - self._agent._advance_usage_baseline(token_usage) - # AFTER the baseline advance: the SDK total covers the parent thread only, - # so child tokens must not shift the parent's baseline. - token_usage = self._agent._fold_subagent_tokens(token_usage, self.messages) - - self.emit.on_event( - TurnEndEvent( - task_id=self.task_id, - turn_id=self.turn_id, - status=TurnEndStatus(status.value), - tokens=token_usage, - ) - ) + def _cut(self, last: Any) -> None: + if not self.open_blocks: + self.reasoning_placeholders = [] + return + cached = (getattr(last, "cached_input_tokens", 0) or 0) if last else 0 + raw_input = (getattr(last, "input_tokens", 0) or 0) if last else 0 + total_output = (getattr(last, "output_tokens", 0) or 0) if last else 0 + reasoning_tok = (getattr(last, "reasoning_output_tokens", 0) or 0) if last else 0 + # A text-less reasoning block becomes a placeholder when reasoning was billed. + if self.reasoning_placeholders: + if reasoning_tok > 0: + for blk in self.reasoning_placeholders: + blk.thinking = "_Reasoning hidden by OpenAI policy_" + else: + for blk in self.reasoning_placeholders: + if blk in self.open_blocks: + self.open_blocks.remove(blk) + self.reasoning_placeholders = [] + if not self.open_blocks: + self.open_start_ms = self.open_end_ms = None + return - model_used = getattr(self.result_turn, "model", None) or self._agent.config.model - usage = token_usage or TokenUsage() - # Real assistant text arrives as agentMessage deltas; fall back to the raw - # Turn dump only when nothing streamed. - agent_output = self.result_text or ( - self._agent._format_turn_result(self.result_turn) if self.result_turn is not None else "" + thinking_blocks = [b for b in self.open_blocks if b.block_type == "thinking"] + action_blocks = [b for b in self.open_blocks if b.block_type != "thinking"] + think_out = reasoning_tok if action_blocks else total_output + action_out = max(total_output - reasoning_tok, 0) if thinking_blocks else total_output + # The FIRST part carries the generation's input/cache: per-call billing that is not split. + billed = TokenUsage( + uncached_input_tokens=_fresh_input_tokens(raw_input, cached), cache_read_input_tokens=cached ) + parts: list[Generation] = [] + for blocks, out_tok, reas_tok in ( + (thinking_blocks, think_out, reasoning_tok), + (action_blocks, action_out, 0), + ): + if not blocks: + continue + for i, blk in enumerate(blocks): + blk.sequence = i + tokens = billed if not parts else TokenUsage() + parts.append( + Generation( + blocks=blocks, + tokens=tokens.model_copy(update={"output_tokens": out_tok}), + reasoning_tokens=reas_tok, + ) + ) - self.emit.on_event( - AgentEndEvent( - task_id=self.task_id, - status=status, - usage=usage, - iteration=self.iteration, - user_input=self.user_input, - agent_output=agent_output, - model_used=model_used, - assistant_turn_count=1, - messages=self.messages, - num_turns=1, - crashed=crashed, - crash_reason=crash_reason, - duration_seconds=time.monotonic() - self.turn_start_time, + message_id = f"{self.turn_id}-msg-{self.gen_index}" + # From the PREVIOUS flush's end, not this generation's first item: the SDK + # stamps an item with the moment it began EXECUTING. The window runs to the + # LAST item's completion, so its tool execution comes back out centrally. + mark_ms = self.gen_mark_ms if self.gen_mark_ms is not None else self.open_start_ms + window_end_ms = self.open_end_ms if self.open_end_ms is not None else self.open_start_ms + if mark_ms is None or window_end_ms is None: + for part in parts: + self.messages.append(self.emitter.add_unmeasured_generation(message_id=message_id, part=part)) + else: + window = close_window( + mark=_ms_to_dt(mark_ms), + now=_ms_to_dt(window_end_ms), + item_start=_ms_to_dt(self.open_start_ms) if self.open_start_ms is not None else None, ) - ) + self.messages.extend(self.emitter.add_generation(message_id=message_id, window=window, parts=parts)) + self.gen_index += 1 + if window_end_ms is not None: + self.gen_mark_ms = window_end_ms + self.open_blocks = [] + self.open_start_ms = None + self.open_end_ms = None - if crashed: - self._agent._capture_partial_turn(self.collector) + def end( + self, + status: AgentEndStatus, + *, + reason: str | None = None, + sdk_token_usage: Any = None, + result_turn: Any = None, + result_text: str = "", + ) -> TurnOutcome: + """Book the turn's tokens once and end it. + + ``sdk_token_usage``, ``result_turn`` and ``result_text`` are what a CLEAN pump + return committed; a crash passes none, so its tokens come from the flushed + messages and the thread baseline still advances past them. + """ + agent = self._agent + token_usage = agent._token_usage_from_sdk(sdk_token_usage) + if token_usage is None: + token_usage = agent._token_usage_from_messages(self.messages) + agent._advance_usage_baseline(token_usage) + # AFTER the baseline advance: the SDK total covers the parent thread only. + token_usage = agent._fold_subagent_tokens(token_usage, self.messages) + usage = token_usage or TokenUsage() + agent_output = result_text or (agent._format_turn_result(result_turn) if result_turn is not None else None) + if status is AgentEndStatus.CRASHED or status is AgentEndStatus.TIMEOUT: + return self.emitter.fail(status, reason or status.value, usage=usage, agent_output=agent_output) + return self.emitter.finalize(status, usage=usage, agent_output=agent_output, model_used=agent.config.model) -@AgentRegistry.register(AgentKind.CODEX, CodexAgentConfig) +@AgentRegistry.register(AgentKind.CODEX, CodexAgentConfig, spi_version=SPI_VERSION) class CodexAgent(Agent[CodexAgentConfig]): """Implementation of the Agent interface for OpenAI Codex using the Codex SDK.""" # The pump has a between-items guard where `should_stop` runs; `system_prompt` - # maps to developer_instructions, ON TOP of the base prompt. + # maps to developer_instructions, ON TOP of the base prompt. Usage is one + # `thread/tokenUsage/updated` per model generation, each closing one inner turn. # Rationale: .claude/notes/agents.md § The system_prompt_semantics marker contract = HarnessContract( system_prompt=Enforcement.ENFORCED, @@ -751,7 +628,8 @@ class CodexAgent(Agent[CodexAgentConfig]): allowed_tools=Enforcement.UNSUPPORTED, disallowed_tools=Enforcement.UNSUPPORTED, cooperative_stop=True, - usage_granularity=UsageGranularity.TURN, + usage_granularity=UsageGranularity.GENERATION, + timing_basis=TimingBasis.CLI_EPOCH_MS, ) def __init__( @@ -779,8 +657,6 @@ def __init__( self.working_directory: Path | None = None self._env_path_prepend: list[str] = [] self._login_shell_home: Path | None = None - # _state / _iteration / _iteration_was_incremented / pending_turn lifecycle - # bookkeeping lives on the Agent base class (shared defaults + helpers). self._log = PrefixedAdapter(logger, {"prefix": instance_name}) # Live handle to the in-flight turn, so kill()/kill_sync() can interrupt a # stuck one: the watchdog's task.cancel() lands only at an await point, @@ -847,151 +723,88 @@ async def communicate( self, user_input: str, *, + iteration: int, stream_callback: StreamCallback | None = None, timeout: float | None = None, should_stop: Callable[[], StopReason | None] | None = None, - ) -> TurnRecord: - """Send a message to Codex and receive its response. + ) -> TurnOutcome: + """Run one Codex turn; see ``Agent.communicate``. - Args: - user_input: The message/prompt to send - stream_callback: Optional callback for real-time event streaming - timeout: Hard wall-clock deadline in seconds - should_stop: The run's stop poll, called after each dispatched - notification. On a reason the pump breaks, the in-flight turn is - interrupted (best-effort) and the turn finalizes cleanly with - ``end_status_for(reason)`` (``crashed=False``). - - Returns: - TurnRecord containing the complete interaction - - Raises: - RuntimeError: If agent is not started - TurnTimeoutError: Timeout elapsed - AgentCrashError: SDK/CLI failed mid-turn + ``should_stop`` is polled after each dispatched notification; on a reason the + pump breaks, the in-flight turn is interrupted (best-effort) and the turn ends + with ``end_status_for(reason)``. """ if not self.working_directory or not self.codex_client: raise RuntimeError("Agent not started. Call start() first.") - assert self.config.type is not None, "CodexAgent requires AgentConfig.type to be set before communicate()" - # Reset the pending slot + bump the iteration counter (shared lifecycle). - self._begin_turn() - - turn_start_time = time.monotonic() - - # The agent is the SOLE emitter: events fan out to an internal - # EventCollector and the caller's stream_callback. - task_id = str(self.config.type) # str() so a plugin subclass with a non-enum kind also works - collector = EventCollector() - emit = CompositeStreamCallback([c for c in (collector, stream_callback) if c is not None]) - - # Codex has no per-API-call boundary: one thread.turn() == one turn_id. - turn_id = f"codex-{self._iteration}" - # The same commands/messages lists flow through the pump and finalize. - # `timeout_hit` is written by the watchdog callback (atomic bool). - state = _CodexTurnState( - self, - emit=emit, - task_id=task_id, - turn_id=turn_id, - collector=collector, - commands=[], - messages=[], - user_input=user_input, - iteration=self._iteration, - turn_start_time=turn_start_time, + emitter = self._open_emitter( + prompt=user_input, + iteration=iteration, + model=self._effective_model(), + task_id=str(self.config.type), # str() so a plugin subclass with a non-enum kind also works + stream_callback=stream_callback, ) + emitter.begin() + decoder = _CodexDecoder(self, emitter, turn_id=f"codex-{iteration}") - try: - emit.on_event( - AgentStartEvent( - task_id=task_id, - prompt=user_input, - iteration=self._iteration, - model=self._effective_model(), - ) - ) + def _on_turn_timeout() -> None: + decoder.timeout_hit = True + try: if self.thread is None: thread_kwargs = self._build_thread_options() - # Add working directory if self.working_directory: thread_kwargs["cwd"] = str(self.working_directory) self.thread = await self._run_async(self.codex_client.thread_start, **thread_kwargs) # A fresh thread counts its cumulative total from zero. self._thread_usage_baseline = _ThreadTotals() - - def _on_turn_timeout() -> None: - state.timeout_hit = True - - with ThreadedWatchdog( + self._log.debug("Starting Codex turn...") + committed = await run_with_watchdog( + self._run_turn_with_streaming(user_input, decoder, should_stop), timeout_seconds=timeout, on_timeout=_on_turn_timeout, - asyncio_task_to_cancel=asyncio.current_task(), label=f"Turn timeout ({timeout:g}s)" if timeout else "turn_timeout", - ): - self._log.debug("Starting Codex turn...") - emit.on_event(TurnStartEvent(task_id=task_id, turn_id=turn_id, model=self._effective_model())) - - try: - # Committed only on a CLEAN return; a crash skips this, so - # finalize reads the defaults and falls back to the messages. - state.result_turn, state.sdk_token_usage, state.result_text = await self._run_turn_with_streaming( - state, should_stop - ) - except asyncio.CancelledError: - if state.timeout_hit: - self._finalize_and_raise_timeout(state.finalize, timeout or 0) - raise - except Exception as e: - if state.timeout_hit: - self._finalize_and_raise_timeout(state.finalize, timeout or 0, cause=e) - if state.ended_cleanly: - # Already stopped on purpose — do not escalate. - # Rationale: .claude/notes/agents.md § Why a post-stop exception is not a crash - self._log.warning("Ignoring post-stop exception; finalizing cleanly: %s", e) - else: - self._finalize_and_raise_crash( - state.finalize, truncate_crash_message(f"Codex turn failed: {e!s}"), cause=e - ) - - if state.timeout_hit: - # Watchdog fired but the pump finished before the cancel landed. - # Routed through the shared kernel so this path sets _state=ERROR - # like every other timeout path. - assert timeout is not None - self._finalize_and_raise_timeout(state.finalize, timeout) - except (AgentCrashError, TurnTimeoutError): - # Already funneled through finalize by the inner handlers. - raise + ) + except WatchdogFired: + return self._fail(decoder, AgentEndStatus.TIMEOUT, format_timeout_reason(timeout or 0)) except asyncio.CancelledError: - # External, or during thread_start before the watchdog block. Close - # the AgentStart so the event tree stays balanced; finalize is - # idempotent, so the timeout case is a no-op here. - if not state.finalized: - self._finalize_external_cancel(state.finalize) + caller = asyncio.current_task() + if caller is not None and caller.cancelling() == 0: + # Not a cancel from outside: the SDK raised it inside the turn body. + return self._fail(decoder, AgentEndStatus.CRASHED, "Codex turn failed: the SDK was cancelled") + self._state = AgentState.ERROR + decoder.end(AgentEndStatus.CRASHED, reason="turn cancelled") raise except Exception as e: - # Failures OUTSIDE the inner turn block, notably thread_start. Without - # this they escape bare: the orchestrator never drains pending_turn - # and _iteration stays incremented. - if state.ended_cleanly and not state.timeout_hit: - # Same retry-poisoning guard as the inner handler. - self._log.warning("Ignoring post-stop exception; finalizing cleanly: %s", e) - else: - self._finalize_and_raise_crash( - state.finalize, truncate_crash_message(f"Codex turn failed: {e!s}"), cause=e - ) + if decoder.timeout_hit: + return self._fail(decoder, AgentEndStatus.TIMEOUT, format_timeout_reason(timeout or 0)) + if not decoder.ended_cleanly: + return self._fail(decoder, AgentEndStatus.CRASHED, f"Codex turn failed: {e!s}") + # Already stopped on purpose — do not escalate. + # Rationale: .claude/notes/agents.md § Why a post-stop exception is not a crash + self._log.warning("Ignoring post-stop exception; finalizing cleanly: %s", e) + committed = (None, None, "") + result_turn, sdk_token_usage, result_text = committed + if decoder.timeout_hit: + # The watchdog fired but the pump finished before the cancel landed. + self._state = AgentState.ERROR + return decoder.end( + AgentEndStatus.TIMEOUT, + reason=format_timeout_reason(timeout or 0), + sdk_token_usage=sdk_token_usage, + result_turn=result_turn, + result_text=result_text, + ) self._state = AgentState.WORKING - self._end_turn_ok() + # Precedence: timeout (above) > the stop reason > done. + status = end_status_for(decoder.stop_reason) if decoder.stop_reason is not None else AgentEndStatus.COMPLETED + return decoder.end(status, sdk_token_usage=sdk_token_usage, result_turn=result_turn, result_text=result_text) - # Precedence: timeout (raised above) > the stop reason > done. - # Rationale: .claude/notes/agents.md § Shared turn lifecycle - status = end_status_for(state.stop_reason) if state.stop_reason is not None else AgentEndStatus.COMPLETED - state.finalize(status, crashed=False, crash_reason=None) - return collector.build_turn_record() + def _fail(self, decoder: _CodexDecoder, status: AgentEndStatus, reason: str) -> TurnOutcome: + self._state = AgentState.ERROR + return decoder.end(status, reason=reason) async def stop(self) -> None: """Stop the agent and tear down the Codex SDK session. @@ -1039,6 +852,17 @@ def _close_client(self) -> None: with contextlib.suppress(Exception): client.close() + async def harness_version(self) -> str | None: + """The ``openai-codex`` SDK version, and the app-server version from its initialize handshake.""" + from importlib.metadata import version + + server = None + if self.codex_client is not None: + with contextlib.suppress(Exception): + info = self.codex_client.metadata.serverInfo + server = info.version if info is not None else None + return f"openai-codex {version('openai-codex')}; codex app-server {server or 'unknown'}" + def get_environment_info(self) -> dict[str, Any]: """Record the resolved Codex routing so runs are auditable/comparable. @@ -1335,20 +1159,14 @@ def _format_turn_result(self, turn_result: Any) -> str: return str(turn_result) async def _run_turn_with_streaming( - self, state: _CodexTurnState, should_stop: Callable[[], StopReason | None] | None = None + self, user_input: str, decoder: _CodexDecoder, should_stop: Callable[[], StopReason | None] | None = None ) -> tuple[Any, Any, str]: - """Drive ``turn.stream()`` through the per-turn state, emitting the standard - event protocol; returns ``(turn_result, latest_token_usage, agent_text)``. - - ``communicate()`` owns the TurnStart/TurnEnd/AgentEnd boundaries; this - drives the inner pump. ``state`` is mutated IN PLACE, so a mid-turn crash - keeps the partial. + """Drive ``turn.stream()`` through the decoder; returns ``(turn_result, latest_token_usage, agent_text)``. - ``should_stop`` runs AFTER ``state.dispatch`` (the emission the monitor - latches on) and BEFORE the next notification is pulled. + ``should_stop`` runs AFTER the decoder handles a notification and BEFORE the + next one is pulled. """ - # Starts the turn without blocking, and opens the event stream. - turn_handle = await self._run_async(self.thread.turn, state.user_input) + turn_handle = await self._run_async(self.thread.turn, user_input) self._active_turn_handle = turn_handle stream = await self._run_async(turn_handle.stream) @@ -1356,85 +1174,69 @@ async def _run_turn_with_streaming( try: while True: # Offloaded so the event loop stays free (parallel agents do not - # serialize) and the watchdog's task.cancel() can land here. + # serialize) and the watchdog's cancel can land here. notification: Any = await asyncio.to_thread(next, stream_iter, _STREAM_DONE) if notification is _STREAM_DONE: break - if state.dispatch(notification): # True on a valid turn/completed + if decoder(notification): # True on a valid turn/completed break reason = should_stop() if should_stop is not None else None if reason is not None: - state.stop_reason = reason + decoder.stop_reason = reason self._log.debug("Stop requested (%s); ending notification pump at this boundary", reason.value) self._interrupt_active_turn() # best-effort; stops server-side spend break finally: self._active_turn_handle = None - # Close orphan tools, flush trailing blocks no tokenUsage event - # closed, then close the stream. Runs on EVERY exit path. - state.close_open_tools() - state._flush_message(None) + # Flush trailing blocks no tokenUsage event closed, then close the stream. + decoder.flush(None) with contextlib.suppress(Exception): await self._run_async(stream.close) - if state.turn_result is None and not state.ended_cleanly: + if decoder.turn_result is None and not decoder.ended_cleanly: raise RuntimeError("Turn did not complete (no turn/completed notification received)") - # If streaming surfaced no transcript, rebuild it from the terminal - # Turn's ordered item list. - if not state.messages: - state.messages.extend(self._messages_from_items(getattr(state.turn_result, "items", None), state.turn_id)) + # If streaming surfaced no transcript, rebuild it from the terminal Turn's items. + if not decoder.messages: + self._messages_from_items(getattr(decoder.turn_result, "items", None), decoder) # RUNS on a cap or budget stop, because recovery is also the only writer of - # the `parent_tool_use_id`-tagged messages `_fold_subagent_tokens` sums — so - # skipping it drops the child threads' spend from the run's cost entirely. - # Still SKIPPED on an early-criterion stop: an armed gate has already - # decided the run, and children may have no rollout yet. + # the child-thread messages `_fold_subagent_tokens` sums. Still SKIPPED on an + # early-criterion stop: an armed gate has already decided the run. # Rationale: .claude/notes/agents.md § Codex rollout rebuild - if state.spawned_children and state.stop_reason is not StopReason.EARLY_CRITERION: - await self._recover_subagent_tool_calls( - state.spawned_children, - state.collab_results, - state.messages, - state.commands, - state.emit, - state.task_id, - state.turn_id, - ) + if decoder.spawned_children and decoder.stop_reason is not StopReason.EARLY_CRITERION: + await self._recover_subagent_tool_calls(decoder) - return state.turn_result, state.latest_token_usage, "".join(state.agent_message_chunks) + return decoder.turn_result, decoder.latest_token_usage, "".join(decoder.agent_message_chunks) - def _messages_from_items(self, items: Any, turn_id: str) -> list[AssistantMessage]: + def _messages_from_items(self, items: Any, decoder: _CodexDecoder) -> None: """Rebuild the assistant transcript from a Turn's ``items`` list (fallback). Same item->block mapping as the streaming path, but Turn items carry no - per-item timestamps, so there is no window to measure: the bounds fall back - to now() and ``generation_duration_ms`` is None, never 0.0 (CE058). + per-item timestamps, so there is no window to measure: each rebuilt message is + an unmeasured generation (CE058). """ if not items: - return [] - - rebuilt: list[AssistantMessage] = [] + return open_blocks: list[ContentBlock] = [] + rebuilt = 0 def _flush() -> None: - nonlocal open_blocks + nonlocal open_blocks, rebuilt if not open_blocks: return for i, blk in enumerate(open_blocks): blk.sequence = i - now = datetime.now() - rebuilt.append( - AssistantMessage( - started_at=now, - completed_at=now, - generation_duration_ms=None, - content_blocks=open_blocks, - tool_use_ids=[b.tool_use_id for b in open_blocks if b.block_type == "tool_use" and b.tool_use_id], - model=self._effective_model(), - message_id=f"{turn_id}-msg-{len(rebuilt)}", + message_id = f"{decoder.turn_id}-msg-{rebuilt}" + if not decoder.emitter.inner_turn_open: + decoder.emitter.begin_inner_turn(message_id) + decoder.messages.append( + decoder.emitter.add_unmeasured_generation( + message_id=message_id, part=Generation(blocks=open_blocks, tokens=TokenUsage()) ) ) + decoder.emitter.end_inner_turn(TurnEndStatus.COMPLETED) + rebuilt += 1 open_blocks = [] for item in items: @@ -1442,7 +1244,6 @@ def _flush() -> None: root_type = getattr(root, "type", None) item_id = getattr(root, "id", "") if root_type is not None and root_type not in _CONTENT_ITEM_TYPES: - # Any tool-like item, mirroring the streaming path's broad capture. status = _status_value(getattr(root, "status", "completed")) exit_code = getattr(root, "exit_code", None) is_error = ( @@ -1466,7 +1267,6 @@ def _flush() -> None: _flush() _flush() - return rebuilt @staticmethod def _tool_name(root_type: str | None) -> str: @@ -1508,75 +1308,50 @@ def _tool_parameters(self, root: Any, root_type: str | None) -> dict[str, Any]: return {"prompt": getattr(root, "revised_prompt", None) or ""} return {} - def _telemetry_for_item( + def _tool_end_for_item( self, root: Any, root_type: str | None, - tool_id: str, - seq: int, *, started_ms: int | None = None, completed_ms: int | None = None, - ) -> tuple[CommandTelemetry | None, bool]: - """Build (telemetry, is_error) for a completed tool item. + ) -> _ToolEnd | None: + """What a completed tool item reports, or None if it cannot be read. commandExecution/fileChange keep their rich extractors; every other kind - routes through the generic builder so it still produces countable - telemetry. The SDK's millisecond stamps arrive as ARGUMENTS rather than - being read back out of the reducer, so each builder stays pure. + routes through the generic one so it still closes a countable call. The + SDK's millisecond stamps arrive as ARGUMENTS, so each extractor stays pure. """ if root_type == "commandExecution": - exit_code = getattr(root, "exit_code", None) - return self._extract_command_telemetry(root, seq, started_ms, completed_ms), exit_code != 0 + return self._command_tool_end(root, started_ms, completed_ms) if root_type == "fileChange": changes = getattr(root, "changes", []) or [] - status_str = _status_value(getattr(root, "status", "completed")) - failed = status_str in _FILE_CHANGE_FAILURE_STATUSES - return ( - self._extract_file_change_telemetry(tool_id, changes, status_str, seq, started_ms, completed_ms), - failed, - ) - return self._extract_generic_telemetry(root, root_type, tool_id, seq, started_ms, completed_ms) + return self._file_change_tool_end(changes, getattr(root, "status", "completed"), started_ms, completed_ms) + return self._generic_tool_end(root, root_type, started_ms, completed_ms) - def _extract_generic_telemetry( + def _generic_tool_end( self, root: Any, root_type: str | None, - tool_id: str, - seq: int, started_ms: int | None = None, completed_ms: int | None = None, - ) -> tuple[CommandTelemetry | None, bool]: - """CommandTelemetry for any tool item without a dedicated extractor. - - Reads status / duration / error generically, so MCP calls, web searches, - collab-agent spawns and future kinds all render and count uniformly. - """ + ) -> _ToolEnd | None: + """A tool item without a dedicated extractor: status, duration and error read generically.""" try: status_str = _status_value(getattr(root, "status", "") or "") err = getattr(root, "error", None) success = getattr(root, "success", None) - is_error = bool(err) or success is False or status_str in _FILE_CHANGE_FAILURE_STATUSES - timing = _item_timing(started_ms, completed_ms, getattr(root, "duration_ms", None)) - return ( - CommandTelemetry( - tool_name=self._tool_name(root_type), - tool_id=tool_id, - timestamp=timing.timestamp, - execution_started_at=timing.execution_started_at, - execution_completed_at=timing.execution_completed_at, - duration_ms=timing.duration_ms, - parameters=self._tool_parameters(root, root_type), - result_status="error" if is_error else ("success" if status_str else "unknown"), - result_summary=self._summarize_tool_item(root, root_type), - error_message=str(err) if err else None, - sequence_number=seq, - ), - is_error, + return _ToolEnd( + is_error=bool(err) or success is False or status_str in _FILE_CHANGE_FAILURE_STATUSES, + summary=self._summarize_tool_item(root, root_type), + error=str(err) if err else None, + result_data=None, + parameters=self._tool_parameters(root, root_type), + timing=_item_timing(started_ms, completed_ms, getattr(root, "duration_ms", None)), ) except Exception as e: self._log.debug(f"Failed to extract generic tool telemetry ({root_type}): {e}") - return None, False + return None @staticmethod def _summarize_tool_item(root: Any, root_type: str | None) -> str: @@ -1628,65 +1403,46 @@ def _handle_collab_completion( if message and thread_id in spawn_by_thread: collab_results[str(thread_id)] = str(message) - async def _recover_subagent_tool_calls( - self, - spawned_children: list[tuple[str, str, str | None]], - collab_results: dict[str, str], - messages: list[TranscriptMessage], - commands: list[CommandTelemetry], - emit: StreamCallback, - task_id: str, - turn_id: str, - ) -> None: - """Recover each spawned sub-agent's INNER tool calls AND token usage. + async def _recover_subagent_tool_calls(self, decoder: _CodexDecoder) -> None: + """Recover each spawned sub-agent's INNER tool calls AND token usage, tagged to its spawning call. - Per inner call, emits one ``CommandTelemetry`` (so the tool row resolves) - plus one nested ``AssistantMessage`` parented to the spawning Agent call - (so the evalboard renders it as an expandable child), carrying that - generation's real tokens. ``finalize`` folds those into the turn total. - - Best-effort: any failure is swallowed, so a recovery hiccup never fails the - turn. + Per inner call: one nested tool call (so the tool row resolves), plus one + nested unmeasured generation carrying that generation's real tokens, which + the turn's end folds into the total. Best-effort: a failure never fails the turn. Rationale: .claude/notes/agents.md § Codex rollout rebuild """ home = self._codex_home() - for thread_id, parent_tool_id, model in spawned_children: + emitter = decoder.emitter + for thread_id, parent_tool_id, model in decoder.spawned_children: try: path = await self._await_rollout_file(home, thread_id) if path is None: - # No rollout to mine: nest just the returned message (if any) so - # the sub-agent's answer still shows, tokenless. + # No rollout to mine: nest just the returned message, tokenless. self._log.debug("CodexAgent: no rollout found for sub-agent thread %s", thread_id) - result = collab_results.get(thread_id) + result = decoder.collab_results.get(thread_id) if result: - messages.append( - self._subagent_text_message(result, parent_tool_id, model, turn_id, len(messages)) + self._add_subagent_message( + decoder, + [ContentBlock(block_type="text", sequence=0, text=result)], + None, + parent_tool_id, + model, ) continue - gens = self._parse_rollout_generations(path) - # Rebuild the sub-agent's generations in order — each a nested - # message parented to the spawn, carrying its real per-generation - # tokens (fresh slice is plain input, cache_creation=0 — Codex has - # no separate cache-write fee) and its blocks. - for gi, gen in enumerate(gens): - blocks, tools = self._subagent_generation_blocks(gen, thread_id) + for gen in self._parse_rollout_generations(path): + blocks, calls = self._subagent_generation_blocks(gen, thread_id) if not blocks: continue - for tel in tools: - commands.append(tel) - emit.on_event(ToolStartEvent(task_id=task_id, turn_id=turn_id, tool=tel)) - emit.on_event( - ToolEndEvent( - task_id=task_id, - turn_id=turn_id, - tool=tel, - status=ToolEndStatus.ERROR if tel.result_status == "error" else ToolEndStatus.OK, - ) + for tool_id, call in calls: + emitter.open_tool(tool_id, call["tool_name"], call["parameters"], parent_tool_id=parent_tool_id) + emitter.close_tool( + tool_id, + status=ToolEndStatus.ERROR if call["is_error"] else ToolEndStatus.OK, + summary=call["result_summary"], ) - messages.append(self._subagent_generation_message(blocks, gen, parent_tool_id, model, turn_id, gi)) + self._add_subagent_message(decoder, blocks, gen["tokens"], parent_tool_id, model) except Exception as exc: - # Best-effort: a recovery hiccup must never fail the turn. self._log.debug("CodexAgent: sub-agent recovery failed for %s: %s", thread_id, exc) @staticmethod @@ -1860,15 +1616,13 @@ def _subagent_output(payload: dict[str, Any]) -> tuple[str, bool]: def _subagent_generation_blocks( self, gen: dict[str, Any], thread_id: str - ) -> tuple[list[ContentBlock], list[CommandTelemetry]]: - """Content blocks + tool telemetry for one recovered sub-agent generation. + ) -> tuple[list[ContentBlock], list[tuple[str, dict[str, Any]]]]: + """Content blocks and ``(tool_id, call)`` pairs for one recovered sub-agent generation. - Each tool call gets a ``tool_use`` block whose id matches a - ``CommandTelemetry``, so the evalboard tool row resolves. Inner ids are - THREAD-PREFIXED to stay unique across the parent's own tools. + Inner ids are THREAD-PREFIXED to stay unique across the parent's own tools. """ blocks: list[ContentBlock] = [] - telemetries: list[CommandTelemetry] = [] + calls: list[tuple[str, dict[str, Any]]] = [] for seq, item in enumerate(gen["items"]): if item["kind"] == "tool": call = item["call"] @@ -1876,99 +1630,53 @@ def _subagent_generation_blocks( blocks.append( ContentBlock(block_type="tool_use", sequence=seq, tool_use_id=tool_id, is_error=call["is_error"]) ) - telemetries.append( - CommandTelemetry( - tool_name=call["tool_name"], - tool_id=tool_id, - timestamp=datetime.now(), - parameters=call["parameters"], - result_status="error" if call["is_error"] else "success", - result_summary=call["result_summary"], - ) - ) + calls.append((tool_id, call)) elif item["kind"] == "text": blocks.append(ContentBlock(block_type="text", sequence=seq, text=item["text"])) - return blocks, telemetries + return blocks, calls - def _subagent_generation_message( + def _add_subagent_message( self, + decoder: _CodexDecoder, blocks: list[ContentBlock], - gen: dict[str, Any], + tokens: tuple[int, int, int, int] | None, parent_tool_use_id: str, model: str | None, - turn_id: str, - index: int, - ) -> AssistantMessage: - """A nested sub-agent generation as an AssistantMessage with real tokens. - - Parented to the spawning Agent call so it nests in the transcript. Tokens - come from the child's per-generation ``token_count``. - """ - raw_input, cached, output, reasoning = gen["tokens"] or (0, 0, 0, 0) - fresh = _fresh_input_tokens(raw_input, cached) - now = datetime.now() - return AssistantMessage( - started_at=now, - completed_at=now, - generation_duration_ms=None, - content_blocks=blocks, - tool_use_ids=[b.tool_use_id for b in blocks if b.block_type == "tool_use" and b.tool_use_id], - input_tokens=fresh, - output_tokens=output, - cache_creation_tokens=0, - cache_read_tokens=cached, - reasoning_tokens=reasoning, - model=model or self._effective_model(), - message_id=f"{turn_id}-subagent-{index}", - parent_tool_use_id=parent_tool_use_id, - ) - - def _subagent_text_message( - self, text: str, parent_tool_use_id: str, model: str | None, turn_id: str, index: int - ) -> AssistantMessage: - """Fallback nested message: just the sub-agent's returned text, tokenless. - - Used only when the child's rollout cannot be found. ``model`` is the - SPAWNED sub-agent's model, not the parent's, matching the other path.""" - now = datetime.now() - return AssistantMessage( - started_at=now, - completed_at=now, - generation_duration_ms=None, - content_blocks=[ContentBlock(block_type="text", sequence=0, text=text)], - tool_use_ids=[], - input_tokens=0, - output_tokens=0, - cache_creation_tokens=0, - cache_read_tokens=0, - model=model or self._effective_model(), - message_id=f"{turn_id}-subagent-{index}", - parent_tool_use_id=parent_tool_use_id, + ) -> None: + """One nested sub-agent generation, parented to its spawning Agent call, with its real tokens.""" + raw_input, cached, output, reasoning = tokens or (0, 0, 0, 0) + decoder.messages.append( + decoder.emitter.add_unmeasured_generation( + message_id=f"{decoder.turn_id}-subagent-{decoder.subagent_index}", + part=Generation( + blocks=blocks, + tokens=TokenUsage( + uncached_input_tokens=_fresh_input_tokens(raw_input, cached), + output_tokens=output, + cache_read_input_tokens=cached, + ), + reasoning_tokens=reasoning, + ), + model=model or self._effective_model(), + parent_tool_id=parent_tool_use_id, + ) ) + decoder.subagent_index += 1 - def _extract_command_telemetry( + def _command_tool_end( self, command_item: Any, - sequence: int, started_ms: int | None = None, completed_ms: int | None = None, - ) -> CommandTelemetry | None: - """Extract CommandTelemetry from a CommandExecutionThreadItem. + ) -> _ToolEnd | None: + """What a CommandExecutionThreadItem reports. Rationale: .claude/notes/agents.md § Tool-name and argument normalization """ - try: - # Extract basic info - command = getattr(command_item, "command", "") - command_id = getattr(command_item, "id", f"cmd_{sequence}") - duration_ms = getattr(command_item, "duration_ms", None) exit_code = getattr(command_item, "exit_code", None) output = getattr(command_item, "aggregated_output", None) - # Determine result status from exit code - result_status = "success" if exit_code == 0 else "error" if exit_code is not None else "unknown" - # Store the output WHOLE: result_summary is the untruncated tool-result # body and its length drives result_tokens, so trimming here # under-reports tool-output size for every command (CE043). Display @@ -1976,73 +1684,51 @@ def _extract_command_telemetry( summary_parts = [f"Exit code: {exit_code}" if exit_code is not None else "Command executed"] if output and len(output.strip()) > 0: summary_parts.append(f"Output: {output}") - result_summary = " | ".join(summary_parts) - # Try to parse output as JSON result_data = None if output: with contextlib.suppress(json.JSONDecodeError, TypeError): result_data = json.loads(output) - # Build parameters from command string - parameters = {"command": command} - - timing = _item_timing(started_ms, completed_ms, duration_ms) - return CommandTelemetry( - tool_name="Bash", - tool_id=command_id, - timestamp=timing.timestamp, - execution_started_at=timing.execution_started_at, - execution_completed_at=timing.execution_completed_at, - duration_ms=timing.duration_ms, - parameters=parameters, - result_status=result_status, - result_summary=result_summary, - error_message=None if exit_code == 0 else output or f"Exit code {exit_code}", + return _ToolEnd( + is_error=exit_code != 0, + summary=" | ".join(summary_parts), + error=None if exit_code == 0 else output or f"Exit code {exit_code}", result_data=result_data, - sequence_number=sequence, + parameters={"command": getattr(command_item, "command", "")}, + timing=_item_timing(started_ms, completed_ms, getattr(command_item, "duration_ms", None)), ) except Exception as e: self._log.debug(f"Failed to extract command telemetry: {e}") return None - def _extract_file_change_telemetry( + def _file_change_tool_end( self, - change_id: str, changes: Any, status: Any, - sequence: int, started_ms: int | None = None, completed_ms: int | None = None, - ) -> CommandTelemetry | None: - """Build CommandTelemetry for a Codex fileChange item. + ) -> _ToolEnd | None: + """What a Codex fileChange item reports; a failed or declined apply_patch is an error. - Recorded as a ``Write`` so cross-agent criteria see the same signal they - get from Claude's Write/Edit calls. A failed or declined apply_patch is an - ``error``, never a successful write. + Recorded as a ``Write`` so cross-agent criteria see the same signal they get + from Claude's Write/Edit calls. """ try: paths = [str(c.path) for c in changes if hasattr(c, "path")] if changes else [] status_str = _status_value(status) failed = status_str in _FILE_CHANGE_FAILURE_STATUSES - timing = _item_timing(started_ms, completed_ms, None) - return CommandTelemetry( - tool_name="Write", - tool_id=change_id, - timestamp=timing.timestamp, - execution_started_at=timing.execution_started_at, - execution_completed_at=timing.execution_completed_at, - duration_ms=timing.duration_ms, - parameters={"paths": paths}, - result_status="error" if failed else "success", - result_summary=( + return _ToolEnd( + is_error=failed, + summary=( f"{len(paths)} file(s) changed" if not failed else f"apply_patch {status_str}: {len(paths)} file(s) not written" ), - error_message=f"apply_patch {status_str}" if failed else None, + error=f"apply_patch {status_str}" if failed else None, result_data=None, - sequence_number=sequence, + parameters={"paths": paths}, + timing=_item_timing(started_ms, completed_ms, None), ) except Exception as e: self._log.debug(f"Failed to extract file-change telemetry: {e}") @@ -2080,18 +1766,13 @@ def _token_usage_from_sdk(self, sdk_token_usage: Any) -> TokenUsage | None: self._thread_usage_baseline = cumulative # Fresh slice = full prompt minus the cached prefix. uncached = _fresh_input_tokens(turn.input, turn.cached) - cost = calculate_cost( - self._effective_model() or "", - uncached_input_tokens=uncached, - output_tokens=turn.output, - cache_read_tokens=turn.cached, - ) - return TokenUsage( + usage = TokenUsage( uncached_input_tokens=uncached, output_tokens=turn.output, cache_read_input_tokens=turn.cached, - total_cost_usd=cost, ) + usage.total_cost_usd = price_turn(usage, (self._effective_model(),)) + return usage def _advance_usage_baseline(self, usage: TokenUsage | None) -> None: """Move the thread baseline past a turn whose SDK total never arrived. @@ -2112,7 +1793,7 @@ def _advance_usage_baseline(self, usage: TokenUsage | None) -> None: cached=base.cached + usage.cache_read_input_tokens, ) - def _fold_subagent_tokens(self, parent: TokenUsage | None, messages: list[TranscriptMessage]) -> TokenUsage | None: + def _fold_subagent_tokens(self, parent: TokenUsage | None, messages: list[AssistantMessage]) -> TokenUsage | None: """Add recovered sub-agent (child-thread) tokens to the parent turn total. Codex bills children on separate threads, so the parent's streamed total @@ -2135,11 +1816,13 @@ def _fold_subagent_tokens(self, parent: TokenUsage | None, messages: list[Transc # Each child generation on its own model, then sum. The total is unpriced # when any priced-from-tokens part is: a partial sum would read as the bill. child_costs = [ - calculate_cost( - m.model or self._effective_model() or "", - uncached_input_tokens=_message_uncached_input(m), - output_tokens=m.output_tokens, - cache_read_tokens=m.cache_read_tokens, + price_turn( + TokenUsage( + uncached_input_tokens=_message_uncached_input(m), + output_tokens=m.output_tokens, + cache_read_input_tokens=m.cache_read_tokens, + ), + (m.model or self._effective_model(),), ) for m in children ] @@ -2153,7 +1836,7 @@ def _fold_subagent_tokens(self, parent: TokenUsage | None, messages: list[Transc total_cost_usd=None if unpriced else (base_cost or 0.0) + sum(c or 0.0 for c in child_costs), ) - def _token_usage_from_messages(self, messages: list[TranscriptMessage]) -> TokenUsage | None: + def _token_usage_from_messages(self, messages: list[AssistantMessage]) -> TokenUsage | None: """Sum per-generation tokens off the captured assistant messages. Crash/timeout fallback: when the stream raises before returning the SDK @@ -2171,18 +1854,13 @@ def _token_usage_from_messages(self, messages: list[TranscriptMessage]) -> Token cache_read = sum(m.cache_read_tokens for m in assistant) if not (uncached or output or cache_read): return None - cost = calculate_cost( - self._effective_model() or "", - uncached_input_tokens=uncached, - output_tokens=output, - cache_read_tokens=cache_read, - ) - return TokenUsage( + usage = TokenUsage( uncached_input_tokens=uncached, output_tokens=output, cache_read_input_tokens=cache_read, - total_cost_usd=cost, ) + usage.total_cost_usd = price_turn(usage, (self._effective_model(),)) + return usage @staticmethod async def _run_async(func: Any, *args: Any, **kwargs: Any) -> Any: diff --git a/src/coder_eval/agents/noop_agent.py b/src/coder_eval/agents/noop_agent.py index e49860fe..b7c7d0cf 100644 --- a/src/coder_eval/agents/noop_agent.py +++ b/src/coder_eval/agents/noop_agent.py @@ -4,8 +4,8 @@ for system / canary checks that reuse the eval infrastructure (sandbox, ``pre_run``, reports, evalboard, ADX) without running a coding agent. Its ``start`` / ``communicate`` / ``stop`` are no-ops and it makes no model API -call; ``communicate`` emits the standardized event protocol for a single empty -turn and returns the ``EventCollector``'s reduction (an empty +call; ``communicate`` writes a single empty turn through its ``TurnEmitter`` and +returns its outcome (an empty :class:`~coder_eval.models.results.TurnRecord`), so the orchestrator's normal lifecycle runs unmodified and then checks the success criteria directly against the sandbox. @@ -19,39 +19,30 @@ from pathlib import Path from coder_eval.agent import Agent, AgentState -from coder_eval.agents.registry import AgentRegistry +from coder_eval.agents.registry import SPI_VERSION, AgentRegistry from coder_eval.models import ( AgentKind, ApiRoute, Enforcement, HarnessContract, NoneAgentConfig, - TurnRecord, + TimingBasis, UsageGranularity, ) -from coder_eval.streaming.callbacks import CompositeStreamCallback, StreamCallback -from coder_eval.streaming.collector import EventCollector -from coder_eval.streaming.events import ( - AgentEndEvent, - AgentEndStatus, - AgentStartEvent, - StopReason, - TurnEndEvent, - TurnEndStatus, - TurnStartEvent, -) +from coder_eval.streaming.callbacks import StreamCallback +from coder_eval.streaming.emitter import TurnOutcome +from coder_eval.streaming.events import AgentEndStatus, StopReason -@AgentRegistry.register(AgentKind.NONE, NoneAgentConfig) +@AgentRegistry.register(AgentKind.NONE, NoneAgentConfig, spi_version=SPI_VERSION) class NoOpAgent(Agent[NoneAgentConfig]): """Agent that does nothing — every lifecycle method is a no-op. Created and driven by the orchestrator exactly like any other agent, so no ``agentless`` branching is needed: the single signal is ``agent.type == - AgentKind.NONE``. ``communicate`` is the SOLE emitter of one clean, balanced - event tree (``AgentStart`` -> ``TurnStart`` -> ``TurnEnd`` -> ``AgentEnd``, - all ``COMPLETED``) and returns the empty turn the ``EventCollector`` reduces - from it. + AgentKind.NONE``. ``communicate`` writes one clean, balanced event tree + (``AgentStart`` -> ``TurnStart`` -> ``TurnEnd`` -> ``AgentEnd``, all + ``COMPLETED``) and returns its outcome. """ contract = HarnessContract( @@ -61,7 +52,9 @@ class NoOpAgent(Agent[NoneAgentConfig]): allowed_tools=Enforcement.UNSUPPORTED, disallowed_tools=Enforcement.UNSUPPORTED, cooperative_stop=False, + reports_cost=True, usage_granularity=UsageGranularity.TURN, + timing_basis=TimingBasis.TURN_CLOCK, ) def __init__( @@ -84,43 +77,27 @@ async def communicate( self, user_input: str, *, + iteration: int, stream_callback: StreamCallback | None = None, timeout: float | None = None, should_stop: Callable[[], StopReason | None] | None = None, - ) -> TurnRecord: - """Return an empty turn without contacting any model. - - ``should_stop`` is accepted for ``Agent.communicate`` override - compatibility and ignored — a no-op turn has nothing to interrupt. + ) -> TurnOutcome: + """Return one empty, completed turn without contacting any model. - Honors the streaming contract — sole emitter of a balanced event tree - (``AgentStart`` -> ``TurnStart`` -> ``TurnEnd`` -> ``AgentEnd``) — so the - task-log handler and renderers see a clean turn boundary. The returned - ``TurnRecord`` is the ``EventCollector``'s reduction of those events. + ``timeout`` and ``should_stop`` are accepted and ignored: a no-op turn has + nothing to interrupt. """ - self._begin_turn() - - task_id = str(self.config.type) # str() so a plugin subclass with a non-enum kind also works - turn_id = f"none-{self._iteration}" - collector = EventCollector() - emit = CompositeStreamCallback([c for c in (collector, stream_callback) if c is not None]) - - emit.on_event(AgentStartEvent(task_id=task_id, prompt=user_input, iteration=self._iteration)) - emit.on_event(TurnStartEvent(task_id=task_id, turn_id=turn_id)) - emit.on_event(TurnEndEvent(task_id=task_id, turn_id=turn_id, status=TurnEndStatus.COMPLETED)) - emit.on_event( - AgentEndEvent( - task_id=task_id, - status=AgentEndStatus.COMPLETED, - iteration=self._iteration, - user_input=user_input, - agent_output="", - assistant_turn_count=0, - ) + emitter = self._open_emitter( + prompt=user_input, + iteration=iteration, + model=None, + task_id=str(self.config.type), # str() so a plugin subclass with a non-enum kind also works + stream_callback=stream_callback, ) - - self._end_turn_ok() - return collector.build_turn_record() + emitter.begin() + emitter.begin_inner_turn(f"none-{iteration}") + emitter.end_inner_turn() + return emitter.finalize(AgentEndStatus.COMPLETED, assistant_turn_count=0, num_turns=None, result_summary=None) async def stop(self) -> None: """No-op: nothing to tear down.""" diff --git a/src/coder_eval/agents/opencode_agent.py b/src/coder_eval/agents/opencode_agent.py index e799cf2a..ff474eb0 100644 --- a/src/coder_eval/agents/opencode_agent.py +++ b/src/coder_eval/agents/opencode_agent.py @@ -1,9 +1,8 @@ """OpenCode agent implementation (the open-source terminal coding agent). Drives the ``opencode`` CLI in non-interactive mode, which streams -newline-delimited JSON events on stdout, and reduces that stream into the -standardized coder_eval event protocol so :class:`EventCollector` builds the -``TurnRecord``. +newline-delimited JSON events on stdout, and reduces that stream through one +``TurnEmitter`` per turn, on the CLI's own epoch-millisecond stamps. The CLI emits TWO envelope shapes on the same stream: the normal form carries its payload under ``part``, while the CLI's own error path emits a flat object @@ -18,81 +17,45 @@ from __future__ import annotations import asyncio -import contextlib import json import logging import os import shutil -import signal import tempfile -import time -from collections.abc import Callable from datetime import datetime from pathlib import Path -from typing import Any, Literal, NoReturn +from typing import Any -from coder_eval.agent import Agent -from coder_eval.errors import AgentCrashError, TurnTimeoutError -from coder_eval.isolation.docker_runner import STDOUT_LINE_LIMIT_BYTES +from coder_eval.agents._transport import JsonlDecoder, SubprocessJsonlAgent from coder_eval.models import ( READ_ONLY_DENIED_TOOLS, AgentKind, AgentState, ApiRoute, - AssistantMessage, - CommandTelemetry, ContentBlock, Enforcement, HarnessContract, OpenCodeAgentConfig, PermissionMode, - ResultSummary, + TimingBasis, TokenUsage, ToolNameMap, - TranscriptMessage, - TurnRecord, UsageGranularity, ) -from coder_eval.pricing import calculate_cost -from coder_eval.streaming.callbacks import StreamCallback, safe_emit -from coder_eval.streaming.collector import EventCollector +from coder_eval.pricing import price_turn +from coder_eval.streaming.emitter import Generation, TurnEmitter, TurnOutcome from coder_eval.streaming.events import ( - AgentEndEvent, AgentEndStatus, - AgentStartEvent, - StopReason, - StreamEvent, - TextChunkEvent, - ToolEndEvent, ToolEndStatus, - ToolStartEvent, - TurnEndEvent, TurnEndStatus, - TurnStartEvent, - end_status_for, ) from coder_eval.timing import close_window -from .registry import AgentRegistry +from .registry import SPI_VERSION, AgentRegistry logger = logging.getLogger(__name__) -# Grace period between SIGTERM and SIGKILL when tearing down the CLI subprocess. -# Doubles as the post-EOF exit grace in _settle_turn when no deadline is set. -_TERM_GRACE_SECONDS = 5.0 - -# SIGKILL does not exist on Windows (where the process-group sweep is a no-op -# anyway); resolve it dynamically so the module imports and typechecks on every -# platform, falling back to SIGTERM for the direct-pid kill_sync path. -_SIGKILL: signal.Signals = getattr(signal, "SIGKILL", signal.SIGTERM) - -# How long to keep draining stdout/stderr after the CLI has been reaped: -# `opencode run` leaves a server child holding the pipes open, so EOF never -# arrives on its own. -# Rationale: .claude/notes/agents.md § Reaping the CLI harnesses -_DRAIN_SECONDS = 2.0 - # The CLI's OWN compact vocabulary, captured from a live run — NOT the # `session.next.*` names in the server's OpenAPI schema, which describe # `opencode serve`'s HTTP/SSE surface. The two are not interchangeable. @@ -107,10 +70,6 @@ # Rationale: .claude/notes/agents.md § Why a clean exit can still be a crash _RECOGNIZED_EVENTS = frozenset({_STEP_START, _STEP_FINISH, _TEXT, _TOOL_USE, _ERROR}) -# How many distinct unrecognized event-type strings to retain for the crash -# message when the vocabulary check fails (diagnosis, not an exhaustive list). -_MAX_UNRECOGNIZED_TYPES = 8 - # OpenCode's native tool names -> the canonical (Claude) vocabulary that every # criterion is written against. Unknown tools pass through unchanged. # Rationale: .claude/notes/agents.md § Tool-name and argument normalization @@ -188,27 +147,21 @@ _PROMPT_FILE_NAME = "system_prompt.md" -# ToolEndStatus -> CommandTelemetry.result_status (the persisted tri-state). -_RESULT_STATUS: dict[ToolEndStatus, Literal["success", "error", "unknown"]] = { - ToolEndStatus.OK: "success", - ToolEndStatus.ERROR: "error", - ToolEndStatus.PERMISSION_DENIED: "error", - ToolEndStatus.UNRESOLVED: "unknown", -} - -def _unwrap(obj: dict[str, Any]) -> tuple[str, dict[str, Any]]: - """Normalize an OpenCode CLI event to ``(event_type, payload)``. +def _unwrap(obj: dict[str, Any]) -> tuple[str, dict[str, Any], datetime | None]: + """Normalize an OpenCode CLI event to ``(event_type, payload, envelope stamp)``. Every line carries its payload under ``part`` except the CLI's own error line, which is flat. Returning the top-level dict for that case is safe: the - accessors read named keys, never iterate. + accessors read named keys, never iterate. The envelope ``timestamp`` (epoch + ms) is the CLI's own stamp for the event; ``None`` when absent. """ event_type = str(obj.get("type") or "") + stamp = _epoch_ms_to_dt(obj.get("timestamp")) part = obj.get("part") if isinstance(part, dict): - return event_type, part - return event_type, obj + return event_type, part, stamp + return event_type, obj, stamp def _epoch_ms_to_dt(value: Any) -> datetime | None: @@ -235,157 +188,125 @@ def _canonical_params(tool_name: str, params: dict[str, Any]) -> dict[str, Any]: return {rename.get(key, key): value for key, value in params.items()} -class _OpenCodeTurnState: - """Per-``communicate()`` accumulator: events in, finalization payload out. +class _OpenCodeDecoder(JsonlDecoder): + """One turn's reducer: OpenCode's nd-JSON events in, ``TurnEmitter`` calls out. - Owns everything the terminal ``AgentEndEvent`` must carry (transcript - messages, cumulative usage, text output) plus the open-tool bookkeeping - needed to force-close orphans when a turn dies mid-flight. + Timing is the CLI's own: window bounds come from each event's envelope + ``timestamp`` and tool spans from ``state.time``. A missing envelope stamp + falls back to the host clock for a window bound, with one warning per turn. + The CLI's ``error`` event is final, so it crashes the turn even after a stop. """ - def __init__(self, *, task_id: str, iteration: int, user_input: str, model: str | None) -> None: - self.task_id = task_id - self.iteration = iteration - self.user_input = user_input - self.model = model + error_survives_stop = True - self.started_at = time.monotonic() - self.session_id: str | None = None - self.thread_id: str | None = None - - # Cumulative turn totals (summed across every inner step). + def __init__(self, emitter: TurnEmitter) -> None: + super().__init__(emitter) self.usage = TokenUsage() self.cost_usd: float = 0.0 self.saw_cost = False - - self.messages: list[TranscriptMessage] = [] - self.text_parts: list[str] = [] + self.stop_reason: str | None = None self.step_count = 0 # Steps the CLI reported as FINISHED, as opposed to `step_count`, which - # counts the ones it started. `_settle_turn` needs the distinction. + # counts the ones it started. `clean_exit_problem` needs the distinction. self.steps_finished = 0 - self.turn_id: str = "" - # True between a step's `step_start` and its `step_finish`. `finalize` - # needs it to close a TurnStartEvent the stream never got to close. - self.step_open = False + self.tool_count = 0 self.step_started_at: datetime | None = None # Where the NEXT generation window starts: the previous step's finish. - # None until the first step finishes, and deliberately so — everything - # before the first `step_start` is CLI process spawn, not model time. + # None until the first step finishes — everything before the first + # `step_start` is CLI process spawn, not model time. # Rationale: .claude/notes/agents.md § Per-harness generation marks self.gen_mark: datetime | None = None self.step_text_parts: list[str] = [] self.step_tool_ids: list[str] = [] - - # callID -> (telemetry, started_at) for tools awaiting a result. - self.open_tools: dict[str, CommandTelemetry] = {} - self.sequence = 0 - self.stop_reason: str | None = None - self.error_message: str | None = None - # Guards the one-terminal-event rule; see finalize(). - self.finalized = False - # Guards _warn_token_shape: one report per turn, not one per step. + # callID -> the latest canonical parameters of a call still awaiting its result. + self.open_tools: dict[str, dict[str, Any]] = {} + self._tool_names: dict[str, str] = {} self.warned_token_shape = False - # Vocabulary drift detection (see _settle_turn): how many events matched - # _RECOGNIZED_EVENTS, and a bounded sample of the types that did not. - self.recognized_events = 0 - self.unrecognized_types: set[str] = set() - - self._emit: Callable[[StreamEvent], None] = lambda _e: None + self.warned_missing_stamp = False - def bind(self, emit: Callable[[StreamEvent], None]) -> None: - self._emit = emit - - def emit(self, event: StreamEvent) -> None: - self._emit(event) - - @property - def agent_output(self) -> str: - return "".join(self.text_parts) - - # --- event handlers ---------------------------------------------------- + def __call__(self, event: dict[str, Any]) -> None: + event_type, part, stamp = _unwrap(event) + if event_type == _STEP_START: + self.on_step_start(part, stamp) + elif event_type == _TEXT: + self.on_text(part) + elif event_type == _TOOL_USE: + self.on_tool_use(part) + elif event_type == _STEP_FINISH: + self.on_step_finish(part, stamp) + elif event_type == _ERROR: + self.on_error(part) + else: + logger.debug("opencode: unhandled event type %r", event_type) - def on_step_start(self, part: dict[str, Any]) -> None: + def end(self, status: AgentEndStatus, *, reason: str | None = None) -> TurnOutcome: + """End the turn: ``fail`` for CRASHED / TIMEOUT (with ``reason``), else ``finalize``.""" + reported = self.usage.model_copy(update={"total_cost_usd": self.cost_usd if self.saw_cost else None}) + usage = self.usage.model_copy(update={"total_cost_usd": price_turn(reported, (self.emitter.model,))}) + if status is AgentEndStatus.CRASHED or status is AgentEndStatus.TIMEOUT: + return self.emitter.fail(status, reason or status.value, usage=usage) + return self.emitter.finalize(status, usage=usage, stop_reason=self.stop_reason) + + def _bound(self, stamp: datetime | None) -> datetime: + """A window bound: the CLI's envelope stamp, else the host clock (warned once per turn).""" + if stamp is not None: + return stamp + if not self.warned_missing_stamp: + self.warned_missing_stamp = True + logger.warning("opencode: an event carried no envelope timestamp; bounding its window on the host clock") + return self.emitter.now() + + def on_step_start(self, part: dict[str, Any], stamp: datetime | None) -> None: + if self.emitter.inner_turn_open: + self.emitter.end_inner_turn(TurnEndStatus.CRASHED) self.step_count += 1 - self.step_open = True - self.turn_id = str(part.get("messageID") or f"step_{self.step_count}") - self.step_started_at = datetime.now() + self.emitter.begin_inner_turn(str(part.get("messageID") or f"step_{self.step_count}")) + self.step_started_at = self._bound(stamp) self.step_text_parts = [] self.step_tool_ids = [] - # No per-step span list to reset here any more: the collector sees every - # span at once and clips each to the window it overlaps. - self.emit( - TurnStartEvent( - task_id=self.task_id, - thread_id=self.thread_id, - turn_id=self.turn_id, - model=self.model, - ) - ) def on_text(self, part: dict[str, Any]) -> None: """``text`` carries a COMPLETE assistant message, not a streaming delta.""" text = part.get("text") if not isinstance(text, str) or not text: return - self.text_parts.append(text) self.step_text_parts.append(text) - self.emit(TextChunkEvent(task_id=self.task_id, thread_id=self.thread_id, turn_id=self.turn_id, text=text)) + self.emitter.text(text) def on_tool_use(self, part: dict[str, Any]) -> None: """A ``tool_use`` event carries the tool's whole state under ``state``. - In practice the CLI emits one already-``completed`` event per call rather - than a call/result pair, so both ``ToolStart`` and ``ToolEnd`` are - synthesized here. A non-terminal state is still handled: the tool is left - open and closed by a later event for the same ``callID``, or force-closed - as ``unresolved``. Execution timestamps come from ``state.time``, so - ``duration_ms`` is the tool's real runtime, not our parse instant. + The CLI usually emits one already-``completed`` event per call. A + non-terminal state leaves the call open, to be closed by a later event for + the same ``callID`` or swept as ``unresolved``. The span is ``state.time``. """ state = part.get("state") state = state if isinstance(state, dict) else {} - call_id = str(part.get("callID") or f"call_{self.sequence + 1}") + call_id = str(part.get("callID") or f"call_{self.tool_count + 1}") time_val = state.get("time") times = time_val if isinstance(time_val, dict) else {} - started = _epoch_ms_to_dt(times.get("start")) params = state.get("input") params = params if isinstance(params, dict) else {} - telemetry = self.open_tools.get(call_id) - if telemetry is None: - self.sequence += 1 + if call_id not in self.open_tools: + self.tool_count += 1 raw_tool = str(part.get("tool") or "unknown") tool_name = _TOOL_NAME_MAP.get(raw_tool.lower(), raw_tool) - telemetry = CommandTelemetry( - tool_name=tool_name, - tool_id=call_id, - assistant_turn_index=self.step_count, - timestamp=started or datetime.now(), - execution_started_at=started, - parameters=_canonical_params(tool_name, params), - sequence_number=self.sequence, - ) - self.open_tools[call_id] = telemetry + canonical = _canonical_params(tool_name, params) + self.emitter.open_tool(call_id, tool_name, canonical, started_at=_epoch_ms_to_dt(times.get("start"))) + self.open_tools[call_id] = canonical self.step_tool_ids.append(call_id) - self.emit( - ToolStartEvent(task_id=self.task_id, thread_id=self.thread_id, turn_id=self.turn_id, tool=telemetry) - ) - else: - # A SECOND event for a call already open. The first routinely carries - # no `input` yet, so freezing its view would leave `parameters` - # permanently `{}` and zero every `command_executed` row while the run - # looked normal. Later evidence wins; absent evidence clears nothing. - if params: - telemetry.parameters = _canonical_params(telemetry.tool_name, params) - if started is not None: - telemetry.execution_started_at = started + self._tool_names[call_id] = tool_name + elif params: + # A SECOND event for an open call: the first routinely carries no + # `input` yet. Later evidence wins; absent evidence clears nothing. + self.open_tools[call_id] = _canonical_params(self._tool_names[call_id], params) status_text = str(state.get("status") or "").lower() + if status_text in ("pending", "running"): + return output = state.get("output") error_text = state.get("error") - if status_text in ("pending", "running"): - return # still in flight; a later event (or the orphan sweep) closes it - if status_text == "error" or error_text: message = str(error_text or output or "tool failed") denied = "permission" in message.lower() or "denied" in message.lower() @@ -393,92 +314,16 @@ def on_tool_use(self, part: dict[str, Any]) -> None: else: message = None status = ToolEndStatus.OK - - # `times` is the SAME dict read at the top: nothing between rebinds or - # mutates `state`. - self._close_tool( + self.emitter.close_tool( call_id, status=status, summary=output if isinstance(output, str) else None, error=message, + parameters=self.open_tools.pop(call_id), completed_at=_epoch_ms_to_dt(times.get("end")), + started_at=_epoch_ms_to_dt(times.get("start")), ) - def _close_tool( - self, - call_id: str, - *, - status: ToolEndStatus, - summary: str | None, - error: str | None, - completed_at: datetime | None = None, - ) -> None: - telemetry = self.open_tools.pop(call_id, None) - if telemetry is None: - # A result with no matching call (shouldn't happen, but never drop it). - self.sequence += 1 - telemetry = CommandTelemetry( - tool_name="unknown", - tool_id=call_id, - assistant_turn_index=self.step_count, - timestamp=datetime.now(), - sequence_number=self.sequence, - ) - completed = completed_at or datetime.now() - telemetry.execution_completed_at = completed - if telemetry.execution_started_at is not None: - telemetry.duration_ms = (completed - telemetry.execution_started_at).total_seconds() * 1000 - telemetry.result_status = _RESULT_STATUS[status] - # Stored untruncated by design (sub-agent returns must survive whole). - telemetry.result_summary = summary - telemetry.error_message = error - self.emit( - ToolEndEvent( - task_id=self.task_id, - thread_id=self.thread_id, - turn_id=self.turn_id, - tool=telemetry, - status=status, - ) - ) - - def _rate_card_cost(self) -> float | None: - """Price the captured buckets from the static rate card. - - ``None`` when the model is unpinned or unpriced. - """ - if not self.model or self.usage.is_empty(): - return None - return calculate_cost( - self.model, - uncached_input_tokens=self.usage.uncached_input_tokens, - output_tokens=self.usage.output_tokens, - cache_creation_tokens=self.usage.cache_creation_input_tokens, - cache_read_tokens=self.usage.cache_read_input_tokens, - ) - - def _resolve_cost(self) -> float | None: - """Decide the turn's cost: the stream's own accounting vs the rate card. - - A non-zero cost the CLI reported always wins. The rate card fills two gaps - that would otherwise book tokens with no money: no ``cost`` field at all, - and ``cost: 0`` for tokens the rate card prices above zero. - - Rationale: .claude/notes/agents.md § Cost: the stream versus the rate card - """ - rate = self._rate_card_cost() - if not self.saw_cost: - return rate - if self.cost_usd == 0.0 and rate: - logger.warning( - "opencode: the stream reported $0 for a turn the rate card prices at $%.6f " - + "(model unpriced in OpenCode's registry, or subscription auth); using the rate card " - + "so the run total is not understated.", - rate, - ) - return rate - return self.cost_usd - def _warn_token_shape(self, message: str, *args: Any) -> None: """Report a token-bucket surprise ONCE per turn (a broken stream repeats it).""" if self.warned_token_shape: @@ -575,9 +420,8 @@ def _fresh_input_slice( ) return raw_in - def on_step_finish(self, part: dict[str, Any]) -> None: + def on_step_finish(self, part: dict[str, Any], stamp: datetime | None) -> None: self.steps_finished += 1 - self.step_open = False tokens = part.get("tokens") tokens = tokens if isinstance(tokens, dict) else {} cache_val = tokens.get("cache") @@ -589,16 +433,14 @@ def on_step_finish(self, part: dict[str, Any]) -> None: step_cr = self._as_int("cache.read", cache.get("read") or 0) step_in = self._fresh_input_slice(tokens, raw_in, raw_out, step_reasoning, step_cw, step_cr) - # Reasoning bills at the output rate but is reported apart from `output`, - # so fold it into the turn total; the per-message record keeps it apart. - step_out = raw_out + step_reasoning - - self.usage = TokenUsage( - uncached_input_tokens=self.usage.uncached_input_tokens + step_in, - output_tokens=self.usage.output_tokens + step_out, - cache_creation_input_tokens=self.usage.cache_creation_input_tokens + step_cw, - cache_read_input_tokens=self.usage.cache_read_input_tokens + step_cr, + # Reasoning bills at the output rate but is reported apart from `output`. + step_delta = TokenUsage( + uncached_input_tokens=step_in, + output_tokens=raw_out + step_reasoning, + cache_creation_input_tokens=step_cw, + cache_read_input_tokens=step_cr, ) + self.usage += step_delta cost = part.get("cost") if isinstance(cost, int | float): self.cost_usd += float(cost) @@ -608,7 +450,7 @@ def on_step_finish(self, part: dict[str, Any]) -> None: if isinstance(finish, str) and finish: self.stop_reason = finish - completed = datetime.now() + completed = self._bound(stamp) step_start = self.step_started_at or completed blocks: list[ContentBlock] = [] step_text = "".join(self.step_text_parts) @@ -618,137 +460,46 @@ def on_step_finish(self, part: dict[str, Any]) -> None: blocks.append(ContentBlock(block_type="tool_use", sequence=i, tool_use_id=tool_id)) # Tile from the previous step's finish. The RAW window only. - started, generation_ms = close_window( - mark=self.gen_mark if self.gen_mark is not None else step_start, - now=completed, - item_start=step_start, - ) - self.messages.append( - AssistantMessage( - started_at=started, - completed_at=completed, - generation_duration_ms=generation_ms, - content_blocks=blocks, - tool_use_ids=list(self.step_tool_ids), - input_tokens=step_in, - output_tokens=step_out, - cache_creation_tokens=step_cw, - cache_read_tokens=step_cr, - reasoning_tokens=step_reasoning, - stop_reason=finish if isinstance(finish, str) else None, - model=self.model, - message_id=str(part.get("messageID") or "") or None, - ) + self.emitter.add_generation( + message_id=str(part.get("messageID") or "") or None, + window=close_window( + mark=self.gen_mark if self.gen_mark is not None else step_start, now=completed, item_start=step_start + ), + parts=[ + Generation( + blocks=blocks, + tokens=step_delta, + reasoning_tokens=step_reasoning, + stop_reason=finish if isinstance(finish, str) else None, + ) + ], ) - # A message was appended, so the next window starts where this one ended. - # Only `step_finish` advances the mark. self.gen_mark = completed # SPENT state, cleared HERE and not only in `on_step_start`: a second # `step_finish` with no intervening start would otherwise republish this - # step's whole span as the next one's. The `min()` in `close_window` still - # defends a genuinely OPEN step against a backwards clock, which is what - # it is for — this reducer's stamps are raw `datetime.now()`. + # step's whole span as the next one's. # Rationale: .claude/notes/agents.md § Per-harness generation marks self.step_started_at = None - self.emit( - TurnEndEvent( - task_id=self.task_id, - thread_id=self.thread_id, - turn_id=self.turn_id, - status=TurnEndStatus.COMPLETED, - tokens=TokenUsage( - uncached_input_tokens=step_in, - output_tokens=step_out, - cache_creation_input_tokens=step_cw, - cache_read_input_tokens=step_cr, - ), - ) - ) + if self.emitter.inner_turn_open: + self.emitter.end_inner_turn(TurnEndStatus.COMPLETED, tokens=step_delta) def on_error(self, part: dict[str, Any]) -> None: - """Record the CLI's own structured error, which ``_settle_turn`` crashes on. + """Record the CLI's own structured error, which the settle crashes on. - The payload is the flat envelope, and its shape varies: a nested - ``error.data.message`` when the CLI has one, otherwise the error's - ``name``. Anything else degrades to its string form rather than raising. + Its shape varies: a nested ``error.data.message`` when the CLI has one, + otherwise the error's ``name``; anything else degrades to its string form. """ error = part.get("error") if isinstance(error, dict): data = error.get("data") message = (data or {}).get("message") if isinstance(data, dict) else None - self.error_message = str(message or error.get("name") or "unknown error") + self.error = str(message or error.get("name") or "unknown error") else: - self.error_message = str(error or "unknown error") + self.error = str(error or "unknown error") - def close_open_tools(self) -> None: - """Force-close every tool still awaiting a result (crash/timeout orphans).""" - for call_id in list(self.open_tools): - self._close_tool(call_id, status=ToolEndStatus.UNRESOLVED, summary=None, error="no result observed") - def finalize( - self, - status: AgentEndStatus, - *, - crashed: bool = False, - crash_reason: str | None = None, - ) -> None: - """Close orphaned tools and emit the terminal ``AgentEndEvent``. - - Idempotent: the protocol allows EXACTLY ONE ``AgentEndEvent`` per - ``communicate()``. - - Rationale: .claude/notes/agents.md § Shared turn lifecycle - """ - if self.finalized: - return - self.finalized = True - self.close_open_tools() - usage = self.usage - cost = self._resolve_cost() - if cost is not None: - usage = usage.model_copy(update={"total_cost_usd": cost}) - # A step still open never received its `step_finish`; close it or the - # one-pair-per-inner-turn contract breaks. Completed steps already closed - # themselves, so this fires ONLY for the straggler. - if self.step_open: - self.step_open = False - self.emit( - TurnEndEvent( - task_id=self.task_id, - thread_id=self.thread_id, - turn_id=self.turn_id, - status=TurnEndStatus(status.value), - tokens=None, - ) - ) - self.emit( - AgentEndEvent( - task_id=self.task_id, - thread_id=self.thread_id, - status=status, - usage=usage, - iteration=self.iteration, - user_input=self.user_input, - agent_output=self.agent_output, - model_used=self.model, - assistant_turn_count=self.step_count, - messages=list(self.messages), - num_turns=self.step_count, - result_summary=ResultSummary( - is_error=crashed, - subtype=status.value, - stop_reason=self.stop_reason, - result=crash_reason or self.error_message, - ), - crashed=crashed, - crash_reason=crash_reason, - duration_seconds=time.monotonic() - self.started_at, - ) - ) - - -@AgentRegistry.register(AgentKind.OPENCODE, OpenCodeAgentConfig) -class OpenCodeAgent(Agent[OpenCodeAgentConfig]): +@AgentRegistry.register(AgentKind.OPENCODE, OpenCodeAgentConfig, spi_version=SPI_VERSION) +class OpenCodeAgent(SubprocessJsonlAgent[OpenCodeAgentConfig]): """Runs the ``opencode`` CLI as a subprocess, one invocation per turn.""" # `should_stop` is polled at every event boundary (tool-call granularity); @@ -763,9 +514,15 @@ class OpenCodeAgent(Agent[OpenCodeAgentConfig]): disallowed_tools=Enforcement.ENFORCED, cooperative_stop=True, usage_granularity=UsageGranularity.STEP, + timing_basis=TimingBasis.CLI_EPOCH_MS, permission_modes=frozenset({PermissionMode.PLAN, PermissionMode.BYPASS_PERMISSIONS}), ) tool_names = _TOOL_NAMES + cli_name = "OpenCode" + executable = "opencode" + docs_page = "docs/agents/OPENCODE.md" + recognized_events = _RECOGNIZED_EVENTS + decoder = _OpenCodeDecoder def __init__( self, @@ -784,9 +541,7 @@ def __init__( Rationale: .claude/notes/agents.md § Why the constructors declare every kwarg """ - super().__init__(config, route, cost_log_tags=cost_log_tags) - self.task_id = task_id - self.working_directory: str | None = None + super().__init__(config, route, task_id=task_id, cost_log_tags=cost_log_tags) self._env_path_prepend: list[str] = [] self._plugin_tools_dir: str | None = None self._skill_dirs: list[str] = [] @@ -794,12 +549,6 @@ def __init__( # removed in stop(). Never inside the sandbox. self._prompt_dir: str | None = None self._session_id: str | None = None - self._process: asyncio.subprocess.Process | None = None - # Process-group ids of every invocation this agent spawned, swept on - # kill()/kill_sync()/stop(): signalling only the CLI pid orphans the - # server child `opencode run` leaves behind. - self._spawned_pgids: list[int] = [] - self._state = AgentState.WORKING # --- lifecycle --------------------------------------------------------- @@ -811,7 +560,7 @@ async def start( plugin_tools_dir: str | None = None, plugin_root: Path | None = None, ) -> None: - if shutil.which("opencode") is None: + if shutil.which(self.executable) is None: raise RuntimeError( "The 'opencode' CLI was not found on PATH." + " Install it with `npm install -g opencode-ai` (or see https://opencode.ai/docs/)." @@ -840,41 +589,6 @@ def _remove_prompt_dir(self) -> None: shutil.rmtree(self._prompt_dir, ignore_errors=True) self._prompt_dir = None - async def kill(self) -> None: - proc = self._process - if proc is not None and proc.returncode is None: - with contextlib.suppress(ProcessLookupError): - proc.terminate() - with contextlib.suppress(TimeoutError, asyncio.TimeoutError): - await asyncio.wait_for(proc.wait(), timeout=_TERM_GRACE_SECONDS) - if proc.returncode is None: - with contextlib.suppress(ProcessLookupError): - proc.kill() - self._sweep_process_groups() - - def kill_sync(self) -> None: - """SIGKILL the in-flight CLI and its process group (watchdog thread; must not await).""" - proc = self._process - if proc is not None and proc.returncode is None: - with contextlib.suppress(ProcessLookupError, PermissionError): - os.kill(proc.pid, _SIGKILL) - self._sweep_process_groups() - - def _sweep_process_groups(self) -> None: - """SIGKILL every process group this agent spawned (POSIX only). - - Each invocation runs in its own session, so its pgid is the CLI's pid and - the group holds ONLY what that invocation spawned. The CLI itself gets - SIGTERM-then-SIGKILL first (see ``kill``); this reaps what survives. - Sessions persist on disk, so this does not lose ``--session`` continuity. - """ - if os.name != "posix": - return - for pgid in self._spawned_pgids: - with contextlib.suppress(ProcessLookupError, PermissionError, OSError): - os.killpg(pgid, _SIGKILL) - self._spawned_pgids.clear() - def get_environment_info(self) -> dict[str, Any]: # Base first so the `system_prompt_semantics` run marker is always # present (an absent marker reads as a pre-marker run). @@ -891,8 +605,8 @@ def get_environment_info(self) -> dict[str, Any]: # --- command construction --------------------------------------------- - def _build_argv(self, user_input: str) -> list[str]: - argv = ["opencode", "run", "--format", "json"] + def argv(self, prompt: str) -> list[str]: + argv = [self.executable, "run", "--format", "json"] if self.config.model: argv += ["-m", self.config.model] if self.working_directory: @@ -907,10 +621,10 @@ def _build_argv(self, user_input: str) -> list[str]: if self._session_id: argv += ["--session", self._session_id] argv.append("--") - argv.append(user_input) + argv.append(prompt) return argv - def _build_env(self) -> dict[str, str]: + def env(self) -> dict[str, str]: """The CLI's full environment: the host's, plus the sandbox's contributions. The PATH prepend is the mock-shadowing contract (``Agent.start``): the @@ -995,332 +709,42 @@ def _inject_config_content(self, env: dict[str, str]) -> None: ] if permission: # OpenCode applies the LAST matching rule, so ours go after every inherited one. - # A host rule for a non-tool key is kept: a tool allowlist must not loosen it. + # A host rule for a non-tool key is kept, and placed after ours so `"*"` does not hide it. inherited_rules = config.get("permission") inherited = inherited_rules if isinstance(inherited_rules, dict) else {} - ours = {k: v for k, v in permission.items() if not (k in _NON_TOOL_PERMISSIONS and k in inherited)} - config["permission"] = {**{k: v for k, v in inherited.items() if k not in ours}, **ours} + kept = {k: v for k, v in inherited.items() if k in _NON_TOOL_PERMISSIONS} + ours = {k: v for k, v in permission.items() if k not in kept} + earlier = {k: v for k, v in inherited.items() if k not in ours and k not in kept} + config["permission"] = {**earlier, **ours, **kept} env[_CONFIG_CONTENT_ENV] = json.dumps(config) # --- the turn ---------------------------------------------------------- - async def communicate( - self, - user_input: str, - *, - stream_callback: StreamCallback | None = None, - timeout: float | None = None, - should_stop: Callable[[], StopReason | None] | None = None, - ) -> TurnRecord: - if self.working_directory is None: - raise RuntimeError("OpenCodeAgent.start() must be called before communicate()") - - self._begin_turn() - collector = EventCollector() - - def emit(event: StreamEvent) -> None: - collector.on_event(event) - if stream_callback is not None: - safe_emit(stream_callback, event) - - state = _OpenCodeTurnState( - task_id=self.task_id, - iteration=self._iteration, - user_input=user_input, - model=self.config.model, - ) - state.bind(emit) - - emit( - AgentStartEvent( - task_id=self.task_id, - prompt=user_input, - iteration=self._iteration, - model=self.config.model, - ) - ) - - deadline = None if timeout is None else time.monotonic() + timeout - requested_stop: StopReason | None = None - stderr_drain: asyncio.Future[bytes] | None = None - # Bound OUTSIDE the try so `finally` can tell "never spawned" from - # "spawned and possibly still running". - proc: asyncio.subprocess.Process | None = None - try: - proc = await asyncio.create_subprocess_exec( - *self._build_argv(user_input), - stdout=asyncio.subprocess.PIPE, - stderr=asyncio.subprocess.PIPE, - cwd=self.working_directory, - env=self._build_env(), - # One nd-JSON event can carry a whole tool result, past - # StreamReader's default 64 KiB cap. - limit=STDOUT_LINE_LIMIT_BYTES, - # Own session/process group, so teardown can killpg the server - # child. POSIX-only knob; harmless False elsewhere. - start_new_session=os.name == "posix", - ) - self._process = proc - if os.name == "posix": - self._spawned_pgids.append(proc.pid) - assert proc.stdout is not None - - # Drain stderr CONCURRENTLY, or a child that fills the pipe blocks on - # write and hangs the turn to its deadline. - # Rationale: .claude/notes/agents.md § Reaping the CLI harnesses - if proc.stderr is not None: - stderr_drain = asyncio.ensure_future(proc.stderr.read()) - - # The server child INHERITS this stdout pipe, so it is not closed when - # the CLI exits: race each read against process exit, then drain. - exit_waiter = asyncio.ensure_future(proc.wait()) - read_task: asyncio.Future[bytes] | None = None - try: - while True: - remaining = None if deadline is None else deadline - time.monotonic() - if remaining is not None and remaining <= 0: - await self._timeout_turn(state, collector, timeout or 0.0) - - if read_task is None: - read_task = asyncio.ensure_future(proc.stdout.readline()) - done, _pending = await asyncio.wait( - {read_task, exit_waiter}, - timeout=remaining, - return_when=asyncio.FIRST_COMPLETED, - ) - if not done: - await self._timeout_turn(state, collector, timeout or 0.0) - if not read_task.done(): - # Exited with the read still pending: bound the tail rather - # than wait on the grandchild's open write end. - try: - await asyncio.wait_for(asyncio.shield(read_task), _DRAIN_SECONDS) - except TimeoutError: - break - line = read_task.result() - read_task = None - if not line: - break - - self._handle_line(line, state) - - requested_stop = should_stop() if should_stop is not None else None - if requested_stop is not None: - await self.kill() - break - finally: - if read_task is not None: - read_task.cancel() - exit_waiter.cancel() - - status = await self._settle_turn( - proc, - state, - collector, - stderr_drain, - requested_stop=requested_stop, - deadline=deadline, - timeout=timeout, - ) - state.finalize(status) - # Build BEFORE marking the turn clean: a failure in the reduction is a - # failed turn, and `_end_turn_ok` clears the rollback flag. - record = collector.build_turn_record() - self._end_turn_ok() - return record - - except (AgentCrashError, TurnTimeoutError): - # Already funneled through finalize by _crash_turn / _timeout_turn. - raise - except asyncio.CancelledError: - self._finalize_external_cancel(state.finalize) - self._capture_partial_turn(collector) - raise - except Exception as e: - # Everything the loop does NOT anticipate. Without this the exception - # escapes raw and breaks the pending-turn contract three ways: no - # AgentEndEvent, the telemetry dropped rather than parked, and - # `_iteration` left incremented. - self._crash_turn(state, collector, f"OpenCode turn failed: {e!s}", cause=e) - raise # unreachable (_crash_turn is NoReturn) — makes the no-fall-through explicit - finally: - if stderr_drain is not None: - stderr_drain.cancel() - self._reap_orphaned_cli(proc) - self._process = None - - def _reap_orphaned_cli(self, proc: asyncio.subprocess.Process | None) -> None: - """Kill a CLI that is still running as the turn unwinds. No-op otherwise. - - Deliberately synchronous: this runs while a ``CancelledError`` is - propagating, where any await can itself be cut short. Skipping the SIGTERM - courtesy is right for a turn that is already lost — :meth:`kill` still - owns every path with something left to flush. ``proc`` is ``None`` when - the spawn itself failed. - - Rationale: .claude/notes/agents.md § Reaping the CLI harnesses - """ - if proc is None or proc.returncode is not None: - return - with contextlib.suppress(ProcessLookupError, PermissionError): - proc.kill() - self._sweep_process_groups() - - async def _settle_turn( - self, - proc: asyncio.subprocess.Process, - state: _OpenCodeTurnState, - collector: EventCollector, - stderr_drain: asyncio.Future[bytes] | None, - *, - requested_stop: StopReason | None, - deadline: float | None, - timeout: float | None, - ) -> AgentEndStatus: - """Reap the CLI once the read loop is done and decide the turn's end status. - - Raises ``AgentCrashError`` (via :meth:`_crash_turn`) on a structured error, - on a death with neither a structured error nor an intentional stop, or on - a clean exit that captured no token telemetry. Raises ``TurnTimeoutError`` - when the deadline elapses while waiting for the exit. - """ - # Bound the reap: the read loop can end at EOF with the CLI still alive, - # and an unbounded wait here would outlive the turn deadline. - remaining = None if deadline is None else max(0.0, deadline - time.monotonic()) - try: - await asyncio.wait_for(proc.wait(), timeout=_TERM_GRACE_SECONDS if remaining is None else remaining) - except TimeoutError: - if remaining is not None: - await self._timeout_turn(state, collector, timeout or 0.0) - await self.kill() - self._crash_turn( - state, - collector, - f"OpenCode closed its event stream but did not exit within {_TERM_GRACE_SECONDS:.0f}s", - ) - # Bounded for the same reason as the read loop: the inherited stderr pipe - # outlives the CLI. Shielded so the timeout doesn't kill it early. - stderr_bytes = b"" - if stderr_drain is not None: - with contextlib.suppress(TimeoutError): - stderr_bytes = await asyncio.wait_for(asyncio.shield(stderr_drain), timeout=_DRAIN_SECONDS) - - if state.error_message is not None: - self._crash_turn(state, collector, f"OpenCode error: {state.error_message}") - - # A non-zero exit with no structured error still means the turn died: - # surface stderr rather than reporting a silent empty success. - if proc.returncode not in (0, None) and requested_stop is None: - detail = stderr_bytes.decode("utf-8", "replace").strip() or f"exit code {proc.returncode}" - self._crash_turn(state, collector, f"OpenCode exited non-zero: {detail}") - - # A clean exit that captured NO token telemetry must not score. Keying on - # the token counts ALONE is what misses the second arm: an exit that - # recognized no events at all reaches the same silent-empty-success - # outcome. Intentional cuts are exempt — either can land before the first - # event, or mid-step. (The two arms are NOT interchangeable downstream; - # see the require_token_telemetry escape hatch below.) - # Rationale: .claude/notes/agents.md § Why a clean exit can still be a crash - nothing_recognized = state.recognized_events == 0 - finished_without_tokens = state.steps_finished > 0 and state.usage.is_empty() - if requested_stop is None and (nothing_recognized or finished_without_tokens): - if nothing_recognized: - seen = ", ".join(sorted(state.unrecognized_types)) or "none (stdout carried no JSON events)" - detail = f"It emitted no recognized events at all. Unrecognized event types seen: {seen}." - else: - detail = ( - f"It reported {state.steps_finished} finished step(s), none of which carried usable " - + f"token counts (cost reported: {'yes' if state.saw_cost else 'no'})." - ) - message = ( - f"OpenCode exited cleanly but the turn captured zero token telemetry. {detail} The CLI's " - + "event or token schema may have changed — see docs/agents/OPENCODE.md (Telemetry) before " - + "trusting any run from this CLI version." - ) - # Escape hatch for a provider/auth mode that reports no usage at all. - # Deliberately does NOT cover `nothing_recognized`: that arm is - # vocabulary drift, which no provider quirk explains. - if not self.config.require_token_telemetry and not nothing_recognized: - logger.warning("opencode: %s Scored anyway — require_token_telemetry is off.", message) - else: - self._crash_turn(state, collector, message) + def observe(self, event: dict[str, Any]) -> None: + """Remember the CLI's session id: it rides on the envelope and is replayed via ``--session``.""" + part = event.get("part") + session_id = event.get("sessionID") or (part.get("sessionID") if isinstance(part, dict) else None) + if isinstance(session_id, str) and session_id: + self._session_id = session_id - return end_status_for(requested_stop) if requested_stop is not None else AgentEndStatus.COMPLETED + def clean_exit_problem(self, decoder: JsonlDecoder) -> str | None: + """A clean exit whose finished steps carried no token counts must not score. - def _crash_turn( - self, - state: _OpenCodeTurnState, - collector: EventCollector, - message: str, - *, - cause: BaseException | None = None, - ) -> NoReturn: - """Park the crashed partial record and raise ``AgentCrashError``. + ``require_token_telemetry: false`` is the escape hatch for a provider or auth + mode that reports no usage at all: the turn is scored with a warning. - ``cause`` preserves the ``__cause__`` link from an ``except ... as e``. + Rationale: .claude/notes/agents.md § Why a clean exit can still be a crash """ - state.close_open_tools() - try: - self._finalize_and_raise_crash(state.finalize, message, cause=cause) - finally: - self._capture_partial_turn(collector) - - async def _timeout_turn( - self, - state: _OpenCodeTurnState, - collector: EventCollector, - timeout: float, - ) -> NoReturn: - """Kill the CLI, park the crashed partial record, raise ``TurnTimeoutError``. - - The partial record is captured immediately after, so ``pending_turn`` - carries everything observed before the deadline. - """ - await self.kill() - state.close_open_tools() - try: - self._finalize_and_raise_timeout(state.finalize, timeout) - finally: - self._capture_partial_turn(collector) - - def _handle_line(self, line: bytes, state: _OpenCodeTurnState) -> None: - """Parse one nd-JSON line and dispatch it. Never raises on bad input.""" - raw = line.decode("utf-8", "replace").strip() - if not raw: - return - try: - obj = json.loads(raw) - except json.JSONDecodeError: - # OpenCode interleaves non-JSON notices (the Bun AVX warning) on - # stdout; a malformed line must not kill the turn. - logger.debug("opencode: skipping non-JSON stdout line: %s", raw[:200]) - return - if not isinstance(obj, dict): - return - - event_type, part = _unwrap(obj) - if event_type in _RECOGNIZED_EVENTS: - state.recognized_events += 1 - elif len(state.unrecognized_types) < _MAX_UNRECOGNIZED_TYPES: - state.unrecognized_types.add(event_type or "") - - # sessionID rides on the envelope, not the part. - session_id = obj.get("sessionID") or part.get("sessionID") - if isinstance(session_id, str) and session_id: - if state.session_id is None: - state.session_id = session_id - state.thread_id = session_id - self._session_id = session_id - - if event_type == _STEP_START: - state.on_step_start(part) - elif event_type == _TEXT: - state.on_text(part) - elif event_type == _TOOL_USE: - state.on_tool_use(part) - elif event_type == _STEP_FINISH: - state.on_step_finish(part) - elif event_type == _ERROR: - state.on_error(part) - else: - logger.debug("opencode: unhandled event type %r", event_type) + assert isinstance(decoder, _OpenCodeDecoder) + if decoder.steps_finished == 0 or not decoder.usage.is_empty(): + return None + message = ( + "OpenCode exited cleanly but the turn captured zero token telemetry. It reported " + + f"{decoder.steps_finished} finished step(s), none of which carried usable token counts " + + f"(cost reported: {'yes' if decoder.saw_cost else 'no'}). The CLI's event or token schema may have " + + "changed — see docs/agents/OPENCODE.md (Telemetry) before trusting any run from this CLI version." + ) + if not self.config.require_token_telemetry: + logger.warning("opencode: %s Scored anyway — require_token_telemetry is off.", message) + return None + return message diff --git a/src/coder_eval/agents/pi_agent.py b/src/coder_eval/agents/pi_agent.py index 6cf26b79..51dd4c78 100644 --- a/src/coder_eval/agents/pi_agent.py +++ b/src/coder_eval/agents/pi_agent.py @@ -1,16 +1,13 @@ """Pi agent implementation (the ``pi`` Node coding agent — https://pi.dev/). Drives the ``pi`` CLI in JSON print mode, which streams newline-delimited JSON -events on stdout, and reduces that stream into the standardized coder_eval event -protocol so :class:`EventCollector` builds the ``TurnRecord``. The design mirrors -:mod:`coder_eval.agents.opencode_agent`. +events on stdout, and reduces that stream through one ``TurnEmitter`` per turn. Three grammar facts that are not obvious from the event names (``pi`` 0.84.4): - ``agent_start`` can appear MORE THAN ONCE per invocation — Pi auto-retries a transient provider error internally — and ``agent_end`` is therefore NOT - terminal. ``agent_settled`` (or EOF) is; the single ``AgentEndEvent`` is - emitted there. + terminal. ``agent_settled`` (or EOF) is; the turn ends there. - ``turn_start`` is one per agent-loop step (``num_turns`` on the record). - ``message_end`` is ignored for token accounting: ``turn_end`` echoes the same assistant usage once per step, so reading both would double-count. @@ -23,89 +20,42 @@ from __future__ import annotations -import asyncio -import contextlib -import json import logging import os import re import shutil -import signal import tempfile -import time -from collections.abc import Callable from datetime import datetime from pathlib import Path -from typing import Any, Literal, NoReturn +from typing import Any from uuid import uuid4 -from coder_eval.agent import Agent -from coder_eval.errors import AgentCrashError, TurnTimeoutError -from coder_eval.isolation.docker_runner import STDOUT_LINE_LIMIT_BYTES +from coder_eval.agents._transport import JsonlDecoder, SubprocessJsonlAgent from coder_eval.models import ( READ_ONLY_DENIED_TOOLS, AgentKind, AgentState, ApiRoute, - AssistantMessage, - CommandTelemetry, ContentBlock, Enforcement, HarnessContract, PermissionMode, PiAgentConfig, - ResultSummary, + TimingBasis, TokenUsage, ToolNameMap, - TranscriptMessage, - TurnRecord, UsageGranularity, ) -from coder_eval.pricing import calculate_cost -from coder_eval.streaming.callbacks import StreamCallback, safe_emit -from coder_eval.streaming.collector import EventCollector -from coder_eval.streaming.events import ( - AgentEndEvent, - AgentEndStatus, - AgentStartEvent, - StopReason, - StreamEvent, - TextChunkEvent, - ToolEndEvent, - ToolEndStatus, - ToolStartEvent, - TurnEndEvent, - TurnEndStatus, - TurnStartEvent, - end_status_for, -) -from coder_eval.timing import TurnClock, close_window +from coder_eval.pricing import price_turn +from coder_eval.streaming.emitter import Generation, TurnEmitter, TurnOutcome +from coder_eval.streaming.events import AgentEndStatus, ToolEndStatus, TurnEndStatus +from coder_eval.timing import close_window -from .registry import AgentRegistry +from .registry import SPI_VERSION, AgentRegistry logger = logging.getLogger(__name__) -# Grace period between SIGTERM and SIGKILL when tearing down the CLI subprocess. -# Doubles as the post-EOF exit grace in _settle_turn when no turn deadline is set. -# Re-declared at OpenCode's value rather than shared — see the notes. -# Rationale: .claude/notes/agents.md § Reaping the CLI harnesses -_TERM_GRACE_SECONDS = 5.0 - -# SIGKILL does not exist on Windows (where the process-group sweep is a no-op -# anyway); resolve it dynamically so the module imports and typechecks on every -# platform, falling back to SIGTERM for the direct-pid kill_sync path. -_SIGKILL: signal.Signals = getattr(signal, "SIGKILL", signal.SIGTERM) - -# How long to keep draining stdout/stderr after the CLI has been reaped: a -# print-mode CLI may leave an inherited pipe open, so every post-exit read is -# bounded. -_DRAIN_SECONDS = 2.0 - -# How many distinct unrecognized event-type strings to retain for the crash -# message when the vocabulary check fails (diagnosis, not an exhaustive list). -_MAX_UNRECOGNIZED_TYPES = 8 - # pi's native tool names -> the canonical (Claude) vocabulary every criterion is # written against. Unknown tools pass through unchanged. # Rationale: .claude/notes/agents.md § Tool-name and argument normalization @@ -165,14 +115,6 @@ } ) -# ToolEndStatus -> CommandTelemetry.result_status (the persisted tri-state). -_RESULT_STATUS: dict[ToolEndStatus, Literal["success", "error", "unknown"]] = { - ToolEndStatus.OK: "success", - ToolEndStatus.ERROR: "error", - ToolEndStatus.PERMISSION_DENIED: "error", - ToolEndStatus.UNRESOLVED: "unknown", -} - def _canonical_params(tool_name: str, params: dict[str, Any]) -> dict[str, Any]: """Rename a tool call's argument keys to the canonical cross-agent vocabulary. @@ -204,50 +146,23 @@ def _result_text(result: Any) -> str | None: return str(result) -class _PiTurnState: - """Per-``communicate()`` accumulator: events in, finalization payload out. +class _PiDecoder(JsonlDecoder): + """One turn's reducer: Pi's nd-JSON events in, ``TurnEmitter`` calls out. - Owns everything the terminal ``AgentEndEvent`` must carry (transcript - messages, summed usage, text output) plus the open-tool bookkeeping needed - to force-close orphans when a turn dies mid-flight. + Holds only what the emitter cannot know: where the next generation window + opens, the current step's text and tool ids, the usage and reported-cost sums, + the last stop reason, and a terminal provider ``error``. """ - def __init__( - self, - *, - task_id: str, - iteration: int, - user_input: str, - model: str | None, - clock: TurnClock | None = None, - ) -> None: - self.task_id = task_id - self.iteration = iteration - self.user_input = user_input - self.model = model - - # ONE clock per turn, so the tool spans and the window bounds they are - # subtracted from share a basis. Injectable so a test supplies a fake - # rather than monkeypatching this module's `datetime` global, which a - # derived stamp would silently escape. - self.clock = clock or TurnClock() - self.started_at = time.monotonic() - self.thread_id: str | None = None - - # Cumulative turn totals (summed across every inner step). + def __init__(self, emitter: TurnEmitter) -> None: + super().__init__(emitter) self.usage = TokenUsage() self.cost_usd: float = 0.0 self.saw_cost = False - - self.messages: list[TranscriptMessage] = [] - self.text_parts: list[str] = [] - - # Pi `turn_start` events counted. + self.stop_reason: str | None = None self.turn_count = 0 - self.turn_id: str = "" - # True between a step's `turn_start` and its `turn_end`. `finalize` needs - # it to close a TurnStartEvent the stream never got to close. - self.turn_open = False + self.tool_count = 0 + self.open_tool_ids: set[str] = set() self.turn_started_at: datetime | None = None self.turn_text_parts: list[str] = [] self.turn_tool_ids: list[str] = [] @@ -256,164 +171,80 @@ def __init__( # before the first `turn_start` is CLI process spawn, not model time. # Rationale: .claude/notes/agents.md § Per-harness generation marks self.gen_mark: datetime | None = None - - # toolCallId -> telemetry for tools awaiting a result. - self.open_tools: dict[str, CommandTelemetry] = {} - self.sequence = 0 - self.stop_reason: str | None = None - self.error_message: str | None = None - # Guards the one-terminal-event rule; see finalize(). - self.finalized = False - # Count of events matched against the recognized Pi vocabulary (drift check), - # plus a bounded sample of the types that did NOT match — so the drift crash - # message can name what it actually saw. - self.recognized_events = 0 - self.unrecognized_types: set[str] = set() - # Warn-once guard for token-accounting drift: the event-vocabulary check - # cannot see inside `usage`. # Rationale: .claude/notes/agents.md § Why token-shape drift warns instead of raising self.warned_token_shape = False - self._emit: Callable[[StreamEvent], None] = lambda _e: None - - def bind(self, emit: Callable[[StreamEvent], None]) -> None: - self._emit = emit - - def emit(self, event: StreamEvent) -> None: - self._emit(event) - - @property - def agent_output(self) -> str: - return "".join(self.text_parts) + def __call__(self, event: dict[str, Any]) -> None: + event_type = event.get("type") + if event_type == "turn_start": + self.on_turn_start() + elif event_type == "message_update": + self.on_message_update(event) + elif event_type == "tool_execution_start": + self.on_tool_execution_start(event) + elif event_type == "tool_execution_end": + self.on_tool_execution_end(event) + elif event_type == "turn_end": + self.on_turn_end(event) + else: + logger.debug("pi: unhandled event type %r", event_type) - # --- event handlers ---------------------------------------------------- + def end(self, status: AgentEndStatus, *, reason: str | None = None) -> TurnOutcome: + """End the turn: ``fail`` for CRASHED / TIMEOUT (with ``reason``), else ``finalize``.""" + reported = self.usage.model_copy(update={"total_cost_usd": self.cost_usd if self.saw_cost else None}) + usage = self.usage.model_copy(update={"total_cost_usd": price_turn(reported, (self.emitter.model,))}) + if status is AgentEndStatus.CRASHED or status is AgentEndStatus.TIMEOUT: + return self.emitter.fail(status, reason or status.value, usage=usage) + return self.emitter.finalize(status, usage=usage, stop_reason=self.stop_reason) def on_turn_start(self) -> None: # A prior step's `turn_start` with no `turn_end` — a generation aborted - # mid-turn (the willRetry case). Close its dangling TurnStartEvent, or the - # stream carries N starts and N-1 ends and breaks the one-pair-per-inner-turn - # contract. `finalize` closes only the LAST open turn, so it cannot cover this. - if self.turn_open: - self.turn_open = False - self.emit( - TurnEndEvent( - task_id=self.task_id, - thread_id=self.thread_id, - turn_id=self.turn_id, - status=TurnEndStatus.CRASHED, - tokens=None, - ) - ) + # mid-turn (the willRetry case). Close its dangling inner turn first. + if self.emitter.inner_turn_open: + self.emitter.end_inner_turn(TurnEndStatus.CRASHED) self.turn_count += 1 - self.turn_open = True - self.turn_id = f"turn_{self.turn_count}" - self.turn_started_at = self.clock.now() + self.emitter.begin_inner_turn(f"turn_{self.turn_count}") + self.turn_started_at = self.emitter.now() self.turn_text_parts = [] self.turn_tool_ids = [] - # No per-turn span list to reset here any more: the collector subtracts - # from final bounds with every span known. - self.emit( - TurnStartEvent( - task_id=self.task_id, - thread_id=self.thread_id, - turn_id=self.turn_id, - model=self.model, - ) - ) - def on_message_update(self, obj: dict[str, Any]) -> None: - """Stream a ``text_delta`` as a ``TextChunkEvent`` (thinking/toolcall ignored).""" - event = obj.get("assistantMessageEvent") - if not isinstance(event, dict) or event.get("type") != "text_delta": + def on_message_update(self, event: dict[str, Any]) -> None: + """Stream a ``text_delta`` (thinking/toolcall ignored).""" + update = event.get("assistantMessageEvent") + if not isinstance(update, dict) or update.get("type") != "text_delta": return - delta = event.get("delta") + delta = update.get("delta") if not isinstance(delta, str) or not delta: return - self.text_parts.append(delta) self.turn_text_parts.append(delta) - self.emit(TextChunkEvent(task_id=self.task_id, thread_id=self.thread_id, turn_id=self.turn_id, text=delta)) + self.emitter.text(delta) - def on_tool_execution_start(self, obj: dict[str, Any]) -> None: - call_id = str(obj.get("toolCallId") or f"call_{self.sequence + 1}") - if call_id in self.open_tools: + def on_tool_execution_start(self, event: dict[str, Any]) -> None: + call_id = str(event.get("toolCallId") or f"call_{self.tool_count + 1}") + if call_id in self.open_tool_ids: return - self.sequence += 1 - raw_tool = str(obj.get("toolName") or "unknown") + self.tool_count += 1 + self.open_tool_ids.add(call_id) + raw_tool = str(event.get("toolName") or "unknown") tool_name = _TOOL_NAME_MAP.get(raw_tool.lower(), raw_tool) - args = obj.get("args") - params = args if isinstance(args, dict) else {} - started = self.clock.now() - telemetry = CommandTelemetry( - tool_name=tool_name, - tool_id=call_id, - assistant_turn_index=self.turn_count, - timestamp=started, - execution_started_at=started, - parameters=_canonical_params(tool_name, params), - sequence_number=self.sequence, - ) - self.open_tools[call_id] = telemetry + args = event.get("args") + self.emitter.open_tool(call_id, tool_name, _canonical_params(tool_name, args if isinstance(args, dict) else {})) self.turn_tool_ids.append(call_id) - self.emit(ToolStartEvent(task_id=self.task_id, thread_id=self.thread_id, turn_id=self.turn_id, tool=telemetry)) - def on_tool_execution_end(self, obj: dict[str, Any]) -> None: - call_id = str(obj.get("toolCallId") or "") - summary = _result_text(obj.get("result")) - is_error = bool(obj.get("isError")) - if is_error: + def on_tool_execution_end(self, event: dict[str, Any]) -> None: + call_id = str(event.get("toolCallId") or "") + summary = _result_text(event.get("result")) + if event.get("isError"): message = summary or "tool failed" # Best-effort: Pi does not tag permission denials, so infer from the - # text. The persisted tri-state folds both to "error", so a - # misclassification is cosmetic. + # text. The persisted tri-state folds both to "error". denied = "permission" in message.lower() or "denied" in message.lower() status = ToolEndStatus.PERMISSION_DENIED if denied else ToolEndStatus.ERROR else: message = None status = ToolEndStatus.OK - self._close_tool(call_id, status=status, summary=summary, error=message) - - def _close_tool( - self, - call_id: str, - *, - status: ToolEndStatus, - summary: str | None, - error: str | None, - ) -> None: - telemetry = self.open_tools.pop(call_id, None) - if telemetry is None: - # A result with no matching call (shouldn't happen, but never drop it). - self.sequence += 1 - telemetry = CommandTelemetry( - tool_name="unknown", - tool_id=call_id, - assistant_turn_index=self.turn_count, - timestamp=self.clock.now(), - sequence_number=self.sequence, - ) - # Only a RESOLVED tool is timed: an orphan was never observed finishing, - # so stamping it would manufacture a span the central subtraction then - # takes out of a window it never occupied. `execution_started_at` IS - # kept — one bound alone forms no span (CE058). - # Rationale: .claude/notes/agents.md § Why only a RESOLVED tool is timed - if status is not ToolEndStatus.UNRESOLVED: - completed = self.clock.now() - telemetry.execution_completed_at = completed - if telemetry.execution_started_at is not None: - telemetry.duration_ms = (completed - telemetry.execution_started_at).total_seconds() * 1000 - telemetry.result_status = _RESULT_STATUS[status] - # Stored untruncated by design (sub-agent returns must survive whole). - telemetry.result_summary = summary - telemetry.error_message = error - self.emit( - ToolEndEvent( - task_id=self.task_id, - thread_id=self.thread_id, - turn_id=self.turn_id, - tool=telemetry, - status=status, - ) - ) + self.open_tool_ids.discard(call_id) + self.emitter.close_tool(call_id, status=status, summary=summary, error=message) def _warn_token_shape(self, message: str, *args: Any) -> None: """Log a token-accounting anomaly at most once per turn (not once per bucket/step).""" @@ -425,10 +256,8 @@ def _warn_token_shape(self, message: str, *args: Any) -> None: def _as_int(self, value: Any) -> int: """Coerce one stream-supplied token count; count a non-number as 0. - A bool is never a token count (``int(True) == 1``). - - ``None`` is a legitimately-absent bucket (silent). Any OTHER unparseable - value is schema drift and warns once. + A bool is never a token count (``int(True) == 1``). ``None`` is a + legitimately-absent bucket (silent); any OTHER unparseable value warns once. Rationale: .claude/notes/agents.md § Why token-shape drift warns instead of raising """ @@ -443,20 +272,16 @@ def _as_int(self, value: Any) -> int: self._warn_token_shape("token count %r was not parseable as an int; counted as 0", value) return 0 - def on_turn_end(self, obj: dict[str, Any]) -> None: - """Accumulate this step's usage and append its assistant message. + def on_turn_end(self, event: dict[str, Any]) -> None: + """Book this step's usage and its generation. Usage is read from ``turn_end`` ONCE per step (not from every - ``message_end``, which echoes the same numbers) so the turn total is the - sum of the per-generation slices. + ``message_end``, which echoes the same numbers). """ - self.turn_open = False - message = obj.get("message") + message = event.get("message") message = message if isinstance(message, dict) else {} raw_usage = message.get("usage") if not isinstance(raw_usage, dict) or not raw_usage: - # A completed step that booked no usage object at all: its tokens and - # cost silently resolve to 0, so say so once. self._warn_token_shape("turn_end carried no usage object; this step's tokens/cost counted as 0") usage = raw_usage if isinstance(raw_usage, dict) else {} @@ -465,28 +290,22 @@ def on_turn_end(self, obj: dict[str, Any]) -> None: step_reasoning = self._as_int(usage.get("reasoning")) step_cw = self._as_int(usage.get("cacheWrite")) step_cr = self._as_int(usage.get("cacheRead")) - # Reasoning bills at the output rate but is reported apart from `output`; - # fold it into the turn total (the per-message record keeps it separately). + # Reasoning bills at the output rate but is reported apart from `output`. step_out = raw_out + step_reasoning - # A usage object whose every bucket resolves to 0 is the drift shape the - # whole-object check cannot see. Warn once; score, don't crash. if raw_usage and step_in == raw_out == step_reasoning == step_cw == step_cr == 0: self._warn_token_shape("turn_end usage object had all-zero token buckets; this step booked 0 tokens/cost") - self.usage = TokenUsage( - uncached_input_tokens=self.usage.uncached_input_tokens + step_in, - output_tokens=self.usage.output_tokens + step_out, - cache_creation_input_tokens=self.usage.cache_creation_input_tokens + step_cw, - cache_read_input_tokens=self.usage.cache_read_input_tokens + step_cr, + tokens = TokenUsage( + uncached_input_tokens=step_in, + output_tokens=step_out, + cache_creation_input_tokens=step_cw, + cache_read_input_tokens=step_cr, ) - # Cross-check the stream's OWN `totalTokens` against the summed buckets. - # Pi's invariant is totalTokens == input + output + cacheRead + cacheWrite - # — reasoning bills at the output rate but is EXCLUDED from this field, so - # compare against raw_out, not step_out. Only when the field is present. + self.usage += tokens + # Pi's invariant is totalTokens == input + output + cacheRead + cacheWrite; + # reasoning is EXCLUDED from it, so compare against raw_out. reported_total = usage.get("totalTokens") - # int OR float: a `123.0`-shaped total is itself a plausible drift, and - # the compare below is exact for whole values. if isinstance(reported_total, int | float) and not isinstance(reported_total, bool): expected_total = step_in + raw_out + step_cw + step_cr if reported_total != expected_total: @@ -507,17 +326,15 @@ def on_turn_end(self, obj: dict[str, Any]) -> None: finish = message.get("stopReason") if isinstance(finish, str) and finish: self.stop_reason = finish - # Capture a terminal provider error so a `pi -p` that exits 0 after - # exhausting retries still surfaces WHY (finalize reads error_message into - # result_summary.result). Reset on a non-error turn so an intermediate - # retry error that a later cycle recovered from never leaks into the result. + # A terminal provider error: `pi -p` exits 0 after exhausting retries. + # Reset on a non-error turn so a recovered retry error never leaks. if finish == "error": err = message.get("errorMessage") - self.error_message = err if isinstance(err, str) and err else "pi reported stopReason=error" + self.error = err if isinstance(err, str) and err else "pi reported stopReason=error" else: - self.error_message = None + self.error = None - completed = self.clock.now() + completed = self.emitter.now() blocks: list[ContentBlock] = [] turn_text = "".join(self.turn_text_parts) if turn_text: @@ -527,151 +344,35 @@ def on_turn_end(self, obj: dict[str, Any]) -> None: # Tile from the previous turn's end. The RAW window only. turn_start = self.turn_started_at if self.turn_started_at is not None else completed - started, generation_ms = close_window( - mark=self.gen_mark if self.gen_mark is not None else turn_start, - now=completed, - item_start=turn_start, - ) - self.messages.append( - AssistantMessage( - started_at=started, - completed_at=completed, - generation_duration_ms=generation_ms, - content_blocks=blocks, - tool_use_ids=list(self.turn_tool_ids), - input_tokens=step_in, - output_tokens=step_out, - cache_creation_tokens=step_cw, - cache_read_tokens=step_cr, - reasoning_tokens=step_reasoning, - stop_reason=finish if isinstance(finish, str) else None, - model=self.model, - message_id=str(message.get("responseId") or "") or None, - ) + self.emitter.add_generation( + message_id=str(message.get("responseId") or "") or None, + window=close_window( + mark=self.gen_mark if self.gen_mark is not None else turn_start, now=completed, item_start=turn_start + ), + parts=[ + Generation( + blocks=blocks, + tokens=tokens, + reasoning_tokens=step_reasoning, + stop_reason=finish if isinstance(finish, str) else None, + ) + ], ) - # A message was appended, so the next window starts where this one ended. - # Only a FINISHED turn advances the mark. self.gen_mark = completed - # SPENT state, reset HERE and not only in `on_turn_start`: a second - # `turn_end` with no intervening start — a duplicate or replayed line, - # which this reducer promises to survive — would otherwise republish this - # turn's span, text and tool ids as the next turn's. + # SPENT state, reset HERE and not only in `on_turn_start`: a duplicate + # `turn_end` with no intervening start would otherwise republish this + # turn's span, text and tool ids as the next turn's. It still books its + # own generation and tokens, but closes no inner turn. # Rationale: .claude/notes/agents.md § Per-harness generation marks self.turn_started_at = None self.turn_text_parts = [] self.turn_tool_ids = [] - self.emit( - TurnEndEvent( - task_id=self.task_id, - thread_id=self.thread_id, - turn_id=self.turn_id, - status=TurnEndStatus.COMPLETED, - tokens=TokenUsage( - uncached_input_tokens=step_in, - output_tokens=step_out, - cache_creation_input_tokens=step_cw, - cache_read_input_tokens=step_cr, - ), - ) - ) - - def _rate_card_cost(self) -> float | None: - if not self.model or self.usage.is_empty(): - return None - return calculate_cost( - self.model, - uncached_input_tokens=self.usage.uncached_input_tokens, - output_tokens=self.usage.output_tokens, - cache_creation_tokens=self.usage.cache_creation_input_tokens, - cache_read_tokens=self.usage.cache_read_input_tokens, - ) - - def _resolve_cost(self) -> float | None: - """Decide the turn's cost: the stream's own accounting vs the rate card. + if self.emitter.inner_turn_open: + self.emitter.end_inner_turn(TurnEndStatus.COMPLETED, tokens=tokens) - Pi reports a real per-call ``cost.total``, which wins for any nonzero - total. It falls back to the rate card when the stream reported no cost at - all, or reported exactly ``$0`` on a model the rate card DOES price. - Rationale: .claude/notes/agents.md § Cost: the stream versus the rate card - """ - rate = self._rate_card_cost() - if not self.saw_cost: - return rate - if self.cost_usd == 0.0 and rate: - logger.debug( - "pi: the stream reported $0 for a turn the rate card prices at $%.6f; using the rate card " - + "so the run total is not understated.", - rate, - ) - return rate - return self.cost_usd - - def close_open_tools(self) -> None: - """Force-close every tool still awaiting a result (crash/timeout orphans).""" - for call_id in list(self.open_tools): - self._close_tool(call_id, status=ToolEndStatus.UNRESOLVED, summary=None, error="no result observed") - - def finalize( - self, - status: AgentEndStatus, - *, - crashed: bool = False, - crash_reason: str | None = None, - ) -> None: - """Close orphaned tools and emit the terminal ``AgentEndEvent`` (idempotent).""" - if self.finalized: - return - self.finalized = True - self.close_open_tools() - usage = self.usage - cost = self._resolve_cost() - if cost is not None: - usage = usage.model_copy(update={"total_cost_usd": cost}) - # A turn still open never received its `turn_end`; close it or the - # one-pair-per-inner-turn contract breaks. - if self.turn_open: - self.turn_open = False - self.emit( - TurnEndEvent( - task_id=self.task_id, - thread_id=self.thread_id, - turn_id=self.turn_id, - status=TurnEndStatus(status.value), - tokens=None, - ) - ) - self.emit( - AgentEndEvent( - task_id=self.task_id, - thread_id=self.thread_id, - status=status, - usage=usage, - iteration=self.iteration, - user_input=self.user_input, - agent_output=self.agent_output, - model_used=self.model, - assistant_turn_count=self.turn_count, - messages=list(self.messages), - num_turns=self.turn_count, - result_summary=ResultSummary( - is_error=crashed, - subtype=status.value, - stop_reason=self.stop_reason, - result=crash_reason or self.error_message, - ), - crashed=crashed, - crash_reason=crash_reason, - duration_seconds=time.monotonic() - self.started_at, - # One basis with the window bounds — see the AgentStartEvent - # site in `communicate`. - timestamp=self.clock.now(), - ) - ) - - -@AgentRegistry.register(AgentKind.PI, PiAgentConfig) -class PiAgent(Agent[PiAgentConfig]): +@AgentRegistry.register(AgentKind.PI, PiAgentConfig, spi_version=SPI_VERSION) +class PiAgent(SubprocessJsonlAgent[PiAgentConfig]): """Runs the ``pi`` CLI as a subprocess, one invocation per turn.""" # `should_stop` is polled at every event boundary (tool-call granularity); @@ -686,9 +387,15 @@ class PiAgent(Agent[PiAgentConfig]): disallowed_tools=Enforcement.ENFORCED, cooperative_stop=True, usage_granularity=UsageGranularity.STEP, + timing_basis=TimingBasis.TURN_CLOCK, permission_modes=frozenset({PermissionMode.PLAN, PermissionMode.BYPASS_PERMISSIONS}), ) tool_names = _TOOL_NAMES + cli_name = "Pi" + executable = "pi" + docs_page = "docs/agents/PI.md" + recognized_events = _RECOGNIZED_EVENTS + decoder = _PiDecoder def __init__( self, @@ -706,9 +413,7 @@ def __init__( Rationale: .claude/notes/agents.md § Why the constructors declare every kwarg """ - super().__init__(config, route, cost_log_tags=cost_log_tags) - self.task_id = task_id - self.working_directory: str | None = None + super().__init__(config, route, task_id=task_id, cost_log_tags=cost_log_tags) self._env_path_prepend: list[str] = [] self._plugin_tools_dir: str | None = None # The staged root's skills dir, passed to `pi --skill`. Assigned in start(). @@ -718,11 +423,6 @@ def __init__( # Rationale: .claude/notes/agents.md § Reaping the CLI harnesses self._session_id: str | None = None self._session_dir: str | None = None - self._process: asyncio.subprocess.Process | None = None - # Process-group ids of every invocation this agent spawned, swept on - # kill()/kill_sync()/stop(). - self._spawned_pgids: list[int] = [] - self._state = AgentState.WORKING # --- lifecycle --------------------------------------------------------- @@ -734,7 +434,7 @@ async def start( plugin_tools_dir: str | None = None, plugin_root: Path | None = None, ) -> None: - if shutil.which("pi") is None: + if shutil.which(self.executable) is None: raise RuntimeError( "The 'pi' CLI was not found on PATH." + " Install it with `npm install -g @earendil-works/pi-coding-agent` (see https://pi.dev/)." @@ -766,39 +466,6 @@ def _cleanup_session_dir(self) -> None: shutil.rmtree(self._session_dir, ignore_errors=True) self._session_dir = None - async def kill(self) -> None: - proc = self._process - if proc is not None and proc.returncode is None: - with contextlib.suppress(ProcessLookupError): - proc.terminate() - with contextlib.suppress(TimeoutError, asyncio.TimeoutError): - await asyncio.wait_for(proc.wait(), timeout=_TERM_GRACE_SECONDS) - if proc.returncode is None: - with contextlib.suppress(ProcessLookupError): - proc.kill() - self._sweep_process_groups() - - def kill_sync(self) -> None: - """SIGKILL the in-flight CLI and its process group (watchdog thread; must not await).""" - proc = self._process - if proc is not None and proc.returncode is None: - with contextlib.suppress(ProcessLookupError, PermissionError): - os.kill(proc.pid, _SIGKILL) - self._sweep_process_groups() - - def _sweep_process_groups(self) -> None: - """SIGKILL every process group this agent spawned (POSIX only). - - Each invocation runs in its own session, so its pgid is the CLI's pid and - the group holds ONLY what that invocation spawned. - """ - if os.name != "posix": - return - for pgid in self._spawned_pgids: - with contextlib.suppress(ProcessLookupError, PermissionError, OSError): - os.killpg(pgid, _SIGKILL) - self._spawned_pgids.clear() - def get_environment_info(self) -> dict[str, Any]: # Spread the base first so the `system_prompt_semantics` run marker is # always present (CE046). @@ -813,14 +480,14 @@ def get_environment_info(self) -> dict[str, Any]: # --- command construction --------------------------------------------- - def _build_argv(self, user_input: str) -> list[str]: + def argv(self, prompt: str) -> list[str]: # -p exits after the run; --no-context-files + --no-approve isolate the # sandbox from host AGENTS.md/CLAUDE.md and project-local trust. # --session-dir + --session-id give cross-communicate() continuity — NOT # --no-session, which would defeat it. No --dir: the working dir is `cwd`. assert self._session_dir is not None and self._session_id is not None argv = [ - "pi", + self.executable, "-p", "--mode", "json", @@ -842,8 +509,8 @@ def _build_argv(self, user_input: str) -> list[str]: argv += self._tool_flags() if self.config.system_prompt: argv += ["--append-system-prompt", self.config.system_prompt] - # user_input is a distinct argv element after `--` (never shell-interpolated). - argv += ["--", user_input] + # The prompt is a distinct argv element after `--` (never shell-interpolated). + argv += ["--", prompt] return argv def _tool_flags(self) -> list[str]: @@ -861,7 +528,7 @@ def _tool_flags(self) -> list[str]: return ["--tools", ",".join(sorted(allow))] if allow else ["--no-tools"] return ["--exclude-tools", ",".join(sorted(deny))] if deny else [] - def _build_env(self) -> dict[str, str]: + def env(self) -> dict[str, str]: """The CLI's full environment: the host's, plus the sandbox's contributions. The PATH prepend is the mock-shadowing contract (``Agent.start``): the @@ -876,294 +543,3 @@ def _build_env(self) -> dict[str, str]: if self._plugin_tools_dir and "PLUGIN_TOOLS_DIR" not in env: env["PLUGIN_TOOLS_DIR"] = self._plugin_tools_dir return env - - # --- the turn ---------------------------------------------------------- - - async def communicate( - self, - user_input: str, - *, - stream_callback: StreamCallback | None = None, - timeout: float | None = None, - should_stop: Callable[[], StopReason | None] | None = None, - ) -> TurnRecord: - if self.working_directory is None: - raise RuntimeError("PiAgent.start() must be called before communicate()") - - self._begin_turn() - collector = EventCollector() - - def emit(event: StreamEvent) -> None: - collector.on_event(event) - if stream_callback is not None: - safe_emit(stream_callback, event) - - state = _PiTurnState( - task_id=self.task_id, - iteration=self._iteration, - user_input=user_input, - model=self.config.model, - ) - state.bind(emit) - - emit( - AgentStartEvent( - task_id=self.task_id, - prompt=user_input, - iteration=self._iteration, - model=self.config.model, - # One basis with the window bounds this is subtracted against; - # the model's raw `datetime.now()` default put two clocks inside - # one subtraction (CE058). - timestamp=state.clock.now(), - ) - ) - - # Deadlines stay on `time.monotonic()`, deliberately NOT the turn clock: - # a deadline must not move when the wall clock steps. - deadline = None if timeout is None else time.monotonic() + timeout - requested_stop: StopReason | None = None - stderr_drain: asyncio.Future[bytes] | None = None - # Bound OUTSIDE the try so `finally` can tell "never spawned" from - # "spawned and possibly still running". - proc: asyncio.subprocess.Process | None = None - try: - proc = await asyncio.create_subprocess_exec( - *self._build_argv(user_input), - stdout=asyncio.subprocess.PIPE, - stderr=asyncio.subprocess.PIPE, - cwd=self.working_directory, - env=self._build_env(), - # One nd-JSON event can carry a whole tool result, past - # StreamReader's default 64 KiB cap. - limit=STDOUT_LINE_LIMIT_BYTES, - # Own session/process group, so teardown can killpg a lingering - # child without touching anything this invocation didn't spawn. - start_new_session=os.name == "posix", - ) - self._process = proc - if os.name == "posix": - self._spawned_pgids.append(proc.pid) - assert proc.stdout is not None - - # Drain stderr CONCURRENTLY, or a child that fills the pipe blocks on - # write and hangs the turn to its deadline. - # Rationale: .claude/notes/agents.md § Reaping the CLI harnesses - if proc.stderr is not None: - stderr_drain = asyncio.ensure_future(proc.stderr.read()) - - # An inherited pipe may never reach EOF, so race each read against - # process exit; a bounded drain then collects the tail. - exit_waiter = asyncio.ensure_future(proc.wait()) - read_task: asyncio.Future[bytes] | None = None - try: - while True: - remaining = None if deadline is None else deadline - time.monotonic() - if remaining is not None and remaining <= 0: - await self._timeout_turn(state, collector, timeout or 0.0) - - if read_task is None: - read_task = asyncio.ensure_future(proc.stdout.readline()) - done, _pending = await asyncio.wait( - {read_task, exit_waiter}, - timeout=remaining, - return_when=asyncio.FIRST_COMPLETED, - ) - if not done: - await self._timeout_turn(state, collector, timeout or 0.0) - if not read_task.done(): - try: - await asyncio.wait_for(asyncio.shield(read_task), _DRAIN_SECONDS) - except TimeoutError: - break - line = read_task.result() - read_task = None - if not line: - break - - self._handle_line(line, state) - - requested_stop = should_stop() if should_stop is not None else None - if requested_stop is not None: - await self.kill() - break - finally: - if read_task is not None: - read_task.cancel() - exit_waiter.cancel() - - status = await self._settle_turn( - proc, - state, - collector, - stderr_drain, - requested_stop=requested_stop, - deadline=deadline, - timeout=timeout, - ) - state.finalize(status) - # Build BEFORE marking the turn clean: a failure in the reduction is a - # failed turn, and `_end_turn_ok` clears the rollback flag. - record = collector.build_turn_record() - self._end_turn_ok() - return record - - except (AgentCrashError, TurnTimeoutError): - # Already funneled through finalize by _crash_turn / _timeout_turn. - raise - except asyncio.CancelledError: - self._finalize_external_cancel(state.finalize) - self._capture_partial_turn(collector) - raise - except Exception as e: - # A spawn failure, a StreamReader ValueError past `limit`, a malformed - # payload, a pydantic error. Funnel to the pending-turn contract. - self._crash_turn(state, collector, f"Pi turn failed: {e!s}", cause=e) - raise # unreachable (_crash_turn is NoReturn) — makes the no-fall-through explicit - finally: - if stderr_drain is not None: - stderr_drain.cancel() - self._reap_orphaned_cli(proc) - self._process = None - - def _reap_orphaned_cli(self, proc: asyncio.subprocess.Process | None) -> None: - """Kill a CLI still running as the turn unwinds. No-op otherwise. - - Synchronous (no await) so it survives a ``CancelledError`` in flight. - ``proc`` is ``None`` when the spawn failed. - - Rationale: .claude/notes/agents.md § Reaping the CLI harnesses - """ - if proc is None or proc.returncode is not None: - return - with contextlib.suppress(ProcessLookupError, PermissionError): - proc.kill() - self._sweep_process_groups() - - async def _settle_turn( - self, - proc: asyncio.subprocess.Process, - state: _PiTurnState, - collector: EventCollector, - stderr_drain: asyncio.Future[bytes] | None, - *, - requested_stop: StopReason | None, - deadline: float | None, - timeout: float | None, - ) -> AgentEndStatus: - """Reap the CLI once the read loop is done and decide the turn's end status. - - Raises ``AgentCrashError`` (via :meth:`_crash_turn`) when the process died - with neither an intentional stop nor a recognized event stream. Raises - ``TurnTimeoutError`` when the deadline elapses while waiting for the exit. - """ - remaining = None if deadline is None else max(0.0, deadline - time.monotonic()) - try: - await asyncio.wait_for(proc.wait(), timeout=_TERM_GRACE_SECONDS if remaining is None else remaining) - except TimeoutError: - if remaining is not None: - await self._timeout_turn(state, collector, timeout or 0.0) - await self.kill() - self._crash_turn( - state, - collector, - f"Pi closed its event stream but did not exit within {_TERM_GRACE_SECONDS:.0f}s", - ) - stderr_bytes = b"" - if stderr_drain is not None: - with contextlib.suppress(TimeoutError): - stderr_bytes = await asyncio.wait_for(asyncio.shield(stderr_drain), timeout=_DRAIN_SECONDS) - - # A terminal provider error is infrastructure failure, not an agent - # failure, and `pi -p` exits 0 after exhausting retries. GATED on - # intentional cuts: a cut can fire before the clearing `turn_end` arrives, - # leaving a stale error from a turn pi was still retrying. - # Rationale: .claude/notes/agents.md § Why a clean exit can still be a crash - if state.error_message is not None and requested_stop is None: - self._crash_turn(state, collector, f"Pi error: {state.error_message}") - - # A non-zero exit with no intentional cut means the turn died. - if proc.returncode not in (0, None) and requested_stop is None: - detail = stderr_bytes.decode("utf-8", "replace").strip() or f"exit code {proc.returncode}" - self._crash_turn(state, collector, f"Pi exited non-zero: {detail}") - - # A clean exit that recognized NO events is vocabulary drift. Intentional - # cuts are exempt: a stop can land before the first event. - if requested_stop is None and state.recognized_events == 0: - seen = ", ".join(sorted(state.unrecognized_types)) or "none (stdout carried no JSON events)" - self._crash_turn( - state, - collector, - "Pi exited cleanly but the turn captured no recognized events. Unrecognized event types seen: " - + f"{seen}. The CLI's event schema may have changed — see docs/agents/PI.md before trusting any " - + "run from this CLI version.", - ) - - return end_status_for(requested_stop) if requested_stop is not None else AgentEndStatus.COMPLETED - - def _crash_turn( - self, - state: _PiTurnState, - collector: EventCollector, - message: str, - *, - cause: BaseException | None = None, - ) -> NoReturn: - """Park the crashed partial record and raise ``AgentCrashError``.""" - state.close_open_tools() - try: - self._finalize_and_raise_crash(state.finalize, message, cause=cause) - finally: - self._capture_partial_turn(collector) - - async def _timeout_turn( - self, - state: _PiTurnState, - collector: EventCollector, - timeout: float, - ) -> NoReturn: - """Kill the CLI, park the crashed partial record, raise ``TurnTimeoutError``.""" - await self.kill() - state.close_open_tools() - try: - self._finalize_and_raise_timeout(state.finalize, timeout) - finally: - self._capture_partial_turn(collector) - - def _handle_line(self, line: bytes, state: _PiTurnState) -> None: - """Parse one nd-JSON line and dispatch it. Never raises on bad input. - - ``agent_end`` is NOT terminal — only ``agent_settled`` / stdout EOF is — so - it is recognized, ignored, and the read loop keeps going. - """ - raw = line.decode("utf-8", "replace").strip() - if not raw: - return - try: - obj = json.loads(raw) - except json.JSONDecodeError: - logger.debug("pi: skipping non-JSON stdout line: %s", raw[:200]) - return - if not isinstance(obj, dict): - return - - event_type = str(obj.get("type") or "") - # `session`, `message_start`, `message_end`, `agent_end` and - # `agent_settled` carry no state we accumulate, but are all recognized. - if event_type in _RECOGNIZED_EVENTS: - state.recognized_events += 1 - elif len(state.unrecognized_types) < _MAX_UNRECOGNIZED_TYPES: - state.unrecognized_types.add(event_type or "") - - if event_type == "turn_start": - state.on_turn_start() - elif event_type == "message_update": - state.on_message_update(obj) - elif event_type == "tool_execution_start": - state.on_tool_execution_start(obj) - elif event_type == "tool_execution_end": - state.on_tool_execution_end(obj) - elif event_type == "turn_end": - state.on_turn_end(obj) - else: - logger.debug("pi: unhandled event type %r", event_type) diff --git a/src/coder_eval/agents/registry.py b/src/coder_eval/agents/registry.py index 177c7e83..be8a93d8 100644 --- a/src/coder_eval/agents/registry.py +++ b/src/coder_eval/agents/registry.py @@ -7,7 +7,7 @@ from collections.abc import Callable from dataclasses import dataclass -from typing import TYPE_CHECKING, Any, ClassVar, TypeVar, cast, get_args +from typing import TYPE_CHECKING, Any, ClassVar, Final, TypeVar, cast, get_args # TYPE_CHECKING-only imports, so this module imports nothing from coder_eval at @@ -17,6 +17,8 @@ from coder_eval.agent import Agent from coder_eval.models import AgentKind, ApiRoute, BaseAgentConfig +SPI_VERSION: Final[int] = 1 + MethodConfigT = TypeVar("MethodConfigT", bound="BaseAgentConfig") AgentClassT = TypeVar("AgentClassT") @@ -87,12 +89,12 @@ class AgentRegistry: @classmethod def register( - cls, agent_kind: str | AgentKind, config_class: type[MethodConfigT] + cls, agent_kind: str | AgentKind, config_class: type[MethodConfigT], *, spi_version: int ) -> Callable[[type[AgentClassT]], type[AgentClassT]]: """Decorator to register an agent class (identity-preserving). Usage: - @AgentRegistry.register(AgentKind.CLAUDE_CODE, ClaudeCodeAgentConfig) + @AgentRegistry.register(AgentKind.CLAUDE_CODE, ClaudeCodeAgentConfig, spi_version=SPI_VERSION) class ClaudeCodeAgent(Agent[ClaudeCodeAgentConfig]): ... @@ -100,10 +102,20 @@ class ClaudeCodeAgent(Agent[ClaudeCodeAgentConfig]): agent_kind: The agent kind this agent implements — an ``AgentKind`` member (built-ins) or a raw kind string (plugins). config_class: The config class this agent expects (e.g., ClaudeCodeAgentConfig) + spi_version: The ``SPI_VERSION`` the agent was written against. Returns: A decorator that registers and returns the agent class unchanged (preserves type) + + Raises: + TypeError: ``spi_version`` is not this core's ``SPI_VERSION``. """ + if spi_version != SPI_VERSION: + raise TypeError( + f"Agent kind {str(agent_kind)!r} was written against coder_eval SPI {spi_version!r}, " + + f"but this coder_eval provides SPI {SPI_VERSION}. Install a plugin version built for " + + f"SPI {SPI_VERSION}, or a coder_eval version that provides SPI {spi_version!r}." + ) def decorator(agent_cls: type[AgentClassT]) -> type[AgentClassT]: kind = str(agent_kind) diff --git a/src/coder_eval/agents/watchdog.py b/src/coder_eval/agents/watchdog.py index 7e3c1038..91c60d89 100644 --- a/src/coder_eval/agents/watchdog.py +++ b/src/coder_eval/agents/watchdog.py @@ -13,8 +13,8 @@ import contextlib import logging import threading -from collections.abc import Callable -from typing import Self +from collections.abc import Callable, Coroutine +from typing import Any, Self logger = logging.getLogger(__name__) @@ -45,7 +45,7 @@ def __init__( *, timeout_seconds: float | None, on_timeout: Callable[[], None], - asyncio_task_to_cancel: asyncio.Task[object] | None = None, + asyncio_task_to_cancel: asyncio.Task[Any] | None = None, label: str = "watchdog", ) -> None: self._timeout = timeout_seconds @@ -55,6 +55,7 @@ def __init__( self._timer: threading.Timer | None = None self._lock = threading.Lock() self._fired = False + self._closed = False @property def fired(self) -> bool: @@ -65,7 +66,8 @@ def fired(self) -> bool: def _fire(self) -> None: """Timer-thread callback. Short, exception-safe.""" with self._lock: - if self._fired: + # A callback that starts after __exit__ is a timer the guarded body already outran. + if self._fired or self._closed: return self._fired = True logger.warning("%s fired after %.1fs — hard-killing subprocess", self._label, self._timeout or 0) @@ -92,6 +94,48 @@ def __enter__(self) -> Self: return self def __exit__(self, exc_type: object, exc: object, tb: object) -> None: + with self._lock: + self._closed = True if self._timer is not None: self._timer.cancel() self._timer = None + + +class WatchdogFired(Exception): # noqa: N818 - a signal, not an error: the plan-named SPI export + """The watchdog cancelled the guarded body at its deadline.""" + + +async def run_with_watchdog[T]( + body: Coroutine[Any, Any, T], + *, + timeout_seconds: float | None, + on_timeout: Callable[[], None], + label: str, +) -> T: + """Run ``body`` as a child task that a ``ThreadedWatchdog`` cancels at ``timeout_seconds``. + + Returns the body's value; any exception the body raises propagates unchanged. + + Raises: + WatchdogFired: the watchdog fired and cancelled the body while the caller + itself was not being cancelled. The caller's cancel count is untouched. + asyncio.CancelledError: the caller was cancelled; the body is cancelled too. + + Rationale: .claude/notes/agents.md § Why the watchdog cancels a child task + """ + child = asyncio.create_task(body) + watchdog = ThreadedWatchdog( + timeout_seconds=timeout_seconds, on_timeout=on_timeout, asyncio_task_to_cancel=child, label=label + ) + try: + with watchdog: + return await child + except asyncio.CancelledError: + caller = asyncio.current_task() + if watchdog.fired and child.cancelled() and (caller is None or caller.cancelling() == 0): + raise WatchdogFired(label) from None + child.cancel() + raise + except BaseException: + child.cancel() + raise diff --git a/src/coder_eval/cli/plan_command.py b/src/coder_eval/cli/plan_command.py index 8039f4e4..318a8a9a 100644 --- a/src/coder_eval/cli/plan_command.py +++ b/src/coder_eval/cli/plan_command.py @@ -72,9 +72,13 @@ def run_plan(*, task_files: list[Path] | None = None, experiment: Path | None = check_api_keys() # Lazy import to avoid circular dependency at module level - from ..orchestration.experiment import DEFAULT_EXPERIMENT_PATH, load_experiment, resolve_task_for_variant + from ..orchestration.experiment import ( + DEFAULT_EXPERIMENT_PATH, + load_experiment, + resolve_variant_prompt_files, + resolve_variant_task, + ) from ..orchestration.harness_contract import TaskResolutionError - from ..orchestration.resolution_checks import validate_resolved_task from ..orchestration.run_limits import validate_run_limits # Always load experiment (defaults to experiments/default.yaml) @@ -87,6 +91,7 @@ def run_plan(*, task_files: list[Path] | None = None, experiment: Path | None = default_exp = load_experiment(DEFAULT_EXPERIMENT_PATH) else: default_exp = exp_def # fall back to custom as its own baseline + resolve_variant_prompt_files(exp_def, exp_path) except Exception as e: console.print(f"[red]Failed to load experiment ({exp_path}): {e}[/red]") raise typer.Exit(1) from e @@ -141,8 +146,9 @@ def run_plan(*, task_files: list[Path] | None = None, experiment: Path | None = # Show resolved agent per variant for variant in exp_def.variants: try: - resolved, _lineage, _ = resolve_task_for_variant(default_exp, task, exp_def, variant) - validate_resolved_task(resolved) + resolved, _lineage, _ = resolve_variant_task( + default_exp, task, exp_def, variant, None, task_file=task_file, experiment_file=exp_path + ) for message in validate_run_limits(resolved): console.print( f" [yellow]⚠[/yellow] [yellow]Variant '{variant.variant_id}': {message}[/yellow]" diff --git a/src/coder_eval/errors/agent.py b/src/coder_eval/errors/agent.py index 9c90e046..e300bac9 100644 --- a/src/coder_eval/errors/agent.py +++ b/src/coder_eval/errors/agent.py @@ -14,7 +14,15 @@ def truncate_crash_message(message: str, *, limit: int = CRASH_REASON_MAX_CHARS) class AgentCrashError(RuntimeError): - """Mid-turn agent failure; routed to AGENT_CRASH by isinstance.""" + """Mid-turn agent failure; routed to AGENT_CRASH by isinstance. + + ``tool_calls`` counts the tool calls the crashed attempt made. An ``AGENT_CRASH`` + with one or more is not retried, because a retry would run on a changed sandbox. + """ + + def __init__(self, message: str = "", tool_calls: int = 0) -> None: + super().__init__(message) + self.tool_calls = tool_calls class AgentConfigError(RuntimeError): diff --git a/src/coder_eval/errors/executor.py b/src/coder_eval/errors/executor.py index 80f3cce9..10028693 100644 --- a/src/coder_eval/errors/executor.py +++ b/src/coder_eval/errors/executor.py @@ -5,7 +5,8 @@ from collections.abc import Awaitable, Callable from typing import Any -from .categories import RETRY_CONFIG, RetryConfig +from .agent import AgentCrashError +from .categories import RETRY_CONFIG, ErrorCategory, RetryConfig from .categorization import categorize_error from .retry import get_error_tip, get_retry_delay, should_retry @@ -18,12 +19,12 @@ async def execute_with_retry( operation_name: str, context: dict[str, Any], max_attempts: int | None = None, - on_attempt_error: Callable[[Exception, int], Awaitable[None]] | None = None, ) -> Any: """Execute an operation with automatic retry on transient errors. Retries only what ``errors/categorization.py`` classifies as retryable; - everything else raises on the first attempt. + everything else raises on the first attempt. An ``AGENT_CRASH`` from an + ``AgentCrashError`` with ``tool_calls`` is never retried. Rationale: .claude/notes/agents.md § Shared turn lifecycle @@ -32,11 +33,6 @@ async def execute_with_retry( operation_name: Human-readable name, for logging only. context: Requires ``task_id``; ``component`` and ``agent_name`` are optional. max_attempts: Overrides the safety limit of 10. - on_attempt_error: Async ``(exception, zero_indexed_attempt) -> None`` invoked - after every failed attempt, including the final non-retryable one, and - before the backoff. Its own exceptions are logged and swallowed so they - cannot mask the original. The orchestrator uses it to drain - ``agent.pending_turn`` and call ``agent.discard_pending_turn()``. Returns: Whatever ``operation`` returned. @@ -46,7 +42,7 @@ async def execute_with_retry( Example: >>> async def flaky_api_call(): - ... return await agent.communicate(prompt) + ... return (await agent.communicate(prompt, iteration=1)).record_or_raise() >>> >>> result = await execute_with_retry( ... operation=flaky_api_call, @@ -73,23 +69,17 @@ async def execute_with_retry( except Exception as e: last_error = e - # Fire callback before the retry decision so partial telemetry - # is captured even on the final non-retryable attempt. - if on_attempt_error is not None: - try: - await on_attempt_error(e, attempt) - except Exception: - logger.exception( - "[%s] on_attempt_error callback raised for %s (attempt %d); ignoring", - task_id, - operation_name, - attempt + 1, - ) - # Categorize error category = categorize_error(e, context) config = RETRY_CONFIG.get(category, RetryConfig()) + if category is ErrorCategory.AGENT_CRASH and isinstance(e, AgentCrashError) and e.tool_calls: + logger.error( + f"[{task_id}] {operation_name} failed (not retried, the attempt made {e.tool_calls} tool calls): " + + f"{category.value} - {e}" + ) + raise + # Check if we should retry (handles both non-retryable categories and exhausted attempts) if not should_retry(category, attempt): if config.max_retries == 0: diff --git a/src/coder_eval/evaluation/sub_agent.py b/src/coder_eval/evaluation/sub_agent.py index e1940a60..5433887c 100644 --- a/src/coder_eval/evaluation/sub_agent.py +++ b/src/coder_eval/evaluation/sub_agent.py @@ -213,7 +213,8 @@ async def _run_agent( """ try: await agent.start(str(judge_dir), plugin_tools_dir=plugin_tools_dir) - return await agent.communicate(user_msg, timeout=turn_timeout) + outcome = await agent.communicate(user_msg, iteration=1, timeout=turn_timeout) + return outcome.record_or_raise(timeout_seconds=turn_timeout) except BaseException: with contextlib.suppress(Exception): await agent.kill() diff --git a/src/coder_eval/isolation/docker_runner.py b/src/coder_eval/isolation/docker_runner.py index 9b0b3764..225bb81a 100644 --- a/src/coder_eval/isolation/docker_runner.py +++ b/src/coder_eval/isolation/docker_runner.py @@ -660,6 +660,7 @@ async def run(self) -> EvaluationResult: heartbeat_task = asyncio.create_task(_heartbeat_loop(heartbeat_path)) proc = await asyncio.create_subprocess_exec( *argv, + stdin=asyncio.subprocess.DEVNULL, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.STDOUT, limit=STDOUT_LINE_LIMIT_BYTES, @@ -1322,7 +1323,7 @@ def _plugin_mount_paths(self) -> list[str]: """Each plugin path as authored, plus each skill source outside every plugin root. Staging links a skill to its RESOLVED source, which can sit outside the root - (a symlinked skill, a manifest ``skills: ../x``), so that source is mounted too. + (a symlinked skill), so that source is mounted too. """ plugins = (self.rt.task.agent.plugins if self.rt.task.agent else None) or [] paths = [plugin["path"] for plugin in plugins] diff --git a/src/coder_eval/models/__init__.py b/src/coder_eval/models/__init__.py index a999658f..1c9bbab1 100644 --- a/src/coder_eval/models/__init__.py +++ b/src/coder_eval/models/__init__.py @@ -94,7 +94,13 @@ ) # Harness contract -from coder_eval.models.harness_contract import Enforcement, HarnessContract, ToolNameMap, UsageGranularity +from coder_eval.models.harness_contract import ( + Enforcement, + HarnessContract, + TimingBasis, + ToolNameMap, + UsageGranularity, +) # Judge from coder_eval.models.judge import JudgeVerdict @@ -253,6 +259,7 @@ "Enforcement", "HarnessContract", "ToolNameMap", + "TimingBasis", "UsageGranularity", # Enums "AgentKind", diff --git a/src/coder_eval/models/agent_config.py b/src/coder_eval/models/agent_config.py index 9ad932cb..a3178137 100644 --- a/src/coder_eval/models/agent_config.py +++ b/src/coder_eval/models/agent_config.py @@ -42,8 +42,8 @@ class LocalPluginConfig(TypedDict): """Vendor-neutral local skills source: a plugin root or a bare skills directory. - Staged by ``orchestration.plugin_staging.stage_plugins`` into one canonical root - before the agent starts. A plain dict at runtime (TypedDict), so the agnostic + Staged by ``orchestration.plugin_staging.stage_plugins`` before the agent starts, into a + root that holds each plugin whole plus a skills index. A plain dict at runtime (TypedDict), so the agnostic ``BaseAgentConfig`` declares ``plugins`` without an SDK type. """ diff --git a/src/coder_eval/models/harness_contract.py b/src/coder_eval/models/harness_contract.py index 4d9e9e07..13d371c9 100644 --- a/src/coder_eval/models/harness_contract.py +++ b/src/coder_eval/models/harness_contract.py @@ -27,6 +27,13 @@ class UsageGranularity(StrEnum): TURN = "turn" +class TimingBasis(StrEnum): + """Where a harness's recorded stamps come from, which decides who stamps a tool and a window.""" + + TURN_CLOCK = "turn_clock" + CLI_EPOCH_MS = "cli_epoch_ms" + + class HarnessContract(BaseModel): """The per-agent declaration of which uniform fields reach the harness. @@ -56,6 +63,20 @@ class HarnessContract(BaseModel): "per communicate() call. A budget can overshoot by one such report." ) ) + timing_basis: TimingBasis = Field( + description=( + "Where recorded stamps come from: turn_clock (the TurnEmitter stamps the turn bracket, every tool " + "and every window from one TurnClock) or cli_epoch_ms (the adapter passes the CLI's own stamps for " + "windows and main-thread tools)." + ) + ) + reports_cost: bool = Field( + default=False, + description=( + "Whether every finished turn with usage carries a cost the harness computed for any model it can " + "run. When False, run_limits.max_usd requires an agent.model that pricing.py prices." + ), + ) permission_modes: frozenset[PermissionMode] | None = Field( default=None, description=( @@ -64,6 +85,11 @@ class HarnessContract(BaseModel): ), ) + @property + def counts_model_turns(self) -> bool: + """Whether the stream opens one inner turn per model response, so run_limits counts model turns.""" + return self.cooperative_stop and self.usage_granularity is not UsageGranularity.TURN + @model_validator(mode="after") def check_semantics_matches_prompt_support(self) -> Self: """Require a semantics value exactly when the system prompt is enforced.""" diff --git a/src/coder_eval/models/limits.py b/src/coder_eval/models/limits.py index e893e148..2514c8ca 100644 --- a/src/coder_eval/models/limits.py +++ b/src/coder_eval/models/limits.py @@ -27,7 +27,7 @@ class RunLimits(BaseModel): """Run-time caps on a task. - Unifies structural caps (max_tool_calls, task_timeout, turn_timeout) and + Unifies structural caps (max_tool_calls, max_turns, task_timeout, turn_timeout) and budget caps (tokens, USD). Structural caps and budget caps stop the task at the agent's next poll boundary; both are cumulative across every turn of the task and apply to the subject agent only. @@ -41,13 +41,25 @@ class RunLimits(BaseModel): default=None, gt=0, description=( - "Hard cap on resolved tool calls across the whole task (every retry attempt and every " + "Hard cap on main-thread resolved tool calls across the whole task (every retry attempt and every " "dialog turn). Enforced by the TurnMonitor at the agent's next poll boundary on every " "harness: the round that reaches the cap is processed whole, so tool calls already in " "flight can still land after it. The run finalizes cleanly as tool_calls_exhausted; " "criteria are still checked. None = no cap." ), ) + max_turns: int | None = Field( + default=None, + gt=0, + description=( + "Hard cap on main-thread model turns (model responses) across the whole task (every retry attempt " + "and every dialog turn); a sub-agent's turns do not count. The TurnMonitor stops the agent at its " + "next poll once turn N+1 starts, so part of that turn can still land. Only harnesses with a " + "per-response turn boundary accept it (usage_granularity generation or step); the others reject it " + "at resolution. The run finalizes cleanly as tool_calls_exhausted; criteria are still checked. " + "None = no cap." + ), + ) expected_tool_calls: int | None = Field( default=None, ge=1, @@ -57,6 +69,15 @@ class RunLimits(BaseModel): "and badges the report; the run is NOT aborted (use max_tool_calls for a hard cap)." ), ) + expected_turns: int | None = Field( + default=None, + ge=1, + description=( + "Soft target for cumulative main-thread model turns across a task, counted like max_turns. Exceeding " + "it logs a one-shot warning and badges the report; the run is NOT aborted (use max_turns for a hard " + "cap). Only harnesses with a per-response turn boundary accept it; the others reject it at resolution." + ), + ) task_timeout: int | None = Field( default=None, ge=30, @@ -88,9 +109,10 @@ class RunLimits(BaseModel): description=( "Max cumulative cost in USD across the task. Enforced live by the TurnMonitor: priced from the " "harness's reported cost when it reports one, else from pricing.py for the reported model or " - "agent.model. A run that can do neither finishes ERROR at that turn's end (register_pricing adds a " - "plugin rate). Overshoot is soft by one usage report (see usage_granularity in " - "docs/agents/HARNESS_PARITY.md) plus any calls in flight." + "agent.model. On a harness that does not report its own cost (reports_cost in " + "docs/agents/HARNESS_PARITY.md), agent.model must be priced, or the task is rejected at resolution " + "(register_pricing adds a plugin rate). A run that still cannot price a turn finishes ERROR. " + "Overshoot is soft by one usage report (see usage_granularity) plus any calls in flight." ), ) count_cached_input: bool = Field( diff --git a/src/coder_eval/models/results.py b/src/coder_eval/models/results.py index 464d643e..ae961d23 100644 --- a/src/coder_eval/models/results.py +++ b/src/coder_eval/models/results.py @@ -281,24 +281,20 @@ def _criterion_result_discriminator(v: Any) -> str: class ResultSummary(BaseModel): - """Diagnostic fields lifted from the SDK's final ResultMessage. - - Powers the agent's debug log and the error-path formatter that - surfaces a useful detail string when the CLI crashes. Persisted on - ``TurnRecord`` for clean turns only — on a crash the agent raises - before the TurnRecord is constructed, so post-mortem persistence on - error turns is out of scope here. - - Mirrors the diagnostic-bearing subset of - ``claude_agent_sdk.ResultMessage``; pure accounting fields - (``num_turns``, ``duration_ms``) live on ``TurnRecord`` / - ``TokenUsage``. + """How a clean turn ended, on every harness. + + Persisted on ``TurnRecord`` for clean turns only; a crashed or timed-out turn + carries none, and its failure is in ``crash_reason``. ``result`` is the agent's + final reply: the text of the last main-thread assistant message when that + message calls no tool. A harness with its own final summary may pass that + instead. Accounting fields (``num_turns``, durations) live on + ``TurnRecord`` / ``TokenUsage``. """ - is_error: bool = Field(description="Whether the SDK reported the turn as errored") - subtype: str = Field(description="Coarse classification (e.g. 'success', 'error_during_execution')") + is_error: bool = Field(description="Whether the harness reported the finished turn as errored") + subtype: str = Field(description="Coarse classification: the end status, or the harness's own subtype") stop_reason: str | None = Field(default=None, description="Why the model stopped, if reported") - result: str | None = Field(default=None, description="Free-form result/error text from the SDK") + result: str | None = Field(default=None, description="The agent's final reply text, or the harness's result text") class TurnRecord(BaseModel): @@ -400,11 +396,12 @@ class TurnRecord(BaseModel): ) tool_calls_exhausted: bool = Field( default=False, - description="Whether the tool-call cap ended this turn before the agent completed on its own", + description="Whether a structural cap (max_tool_calls or max_turns) ended this turn before the agent " + + "completed on its own", ) result_summary: ResultSummary | None = Field( default=None, - description="SDK ResultMessage summary, when one was emitted (clean turns or partials that got one).", + description="How a clean turn ended, including the agent's final reply; None on a crashed or timed-out turn.", ) provider_call_costs: list[ProviderCallCost] = Field( default_factory=list, @@ -523,7 +520,7 @@ class EarlyStopInfo(BaseModel): + "advisory without re-deriving from task_config.", ) sdk_turn_index: int = Field( - description="SDK inner-turn count at the stop (the monitor counts TurnStartEvents). NOT the " + description="Main-thread model turns started at the stop (each turn id once per communicate()). NOT the " + "orchestrator iteration, which is always 1 in single-shot." ) tool_call_index: int = Field( @@ -599,7 +596,8 @@ class EvaluationResult(BaseModel): final_status: FinalStatus = Field(description="Final status of the evaluation") tool_calls_exhausted: bool = Field( default=False, - description="Whether the tool-call cap ended any iteration before the agent completed on its own", + description="Whether a structural cap (max_tool_calls or max_turns) ended any iteration before the agent " + + "completed on its own", ) weighted_score: float | None = Field( default=None, ge=0.0, le=1.0, description="Weighted average of criterion scores (0.0 to 1.0)" @@ -683,6 +681,14 @@ class EvaluationResult(BaseModel): default=None, description="Total assistant turns across all orchestrator iterations", ) + model_turns: int | None = Field( + default=None, + description=( + "Main-thread model turns across the task, counted by the TurnMonitor (each turn id once per " + "communicate()). None when the harness does not count model turns, when no turn finished, or on a " + "run recorded before the field." + ), + ) # Commands efficiency (orchestrator-level tracking) expected_commands: int | None = Field(default=None, description="Expected commands from task definition") diff --git a/src/coder_eval/orchestration/experiment.py b/src/coder_eval/orchestration/experiment.py index 5214167e..8690f527 100644 --- a/src/coder_eval/orchestration/experiment.py +++ b/src/coder_eval/orchestration/experiment.py @@ -585,7 +585,7 @@ def _add_rl(rl: RunLimits | None, source: ConfigSource) -> None: def _apply_cli_overrides( task: TaskDefinition, - config: BatchRunConfig, + config: BatchRunConfig | None, lineage: dict[str, ConfigLineageEntry] | None = None, ) -> None: """Apply CLI overrides (layer 5) to a task definition in-place. @@ -596,7 +596,7 @@ def _apply_cli_overrides( Args: task: The task definition to mutate. - config: Batch run configuration containing CLI overrides. + config: Batch run configuration containing CLI overrides; ``None`` applies none. lineage: Optional lineage dict to update with CLI override entries. """ from .overrides import apply_overrides @@ -604,7 +604,8 @@ def _apply_cli_overrides( # Rationale: .claude/notes/orchestration.md § No-op tasks need no special case anywhere assert task.agent is not None, f"Task '{task.task_id}' has no agent config" - apply_overrides(task, config.overrides, agent_type=config.agent_type, lineage=lineage) + if config is not None: + apply_overrides(task, config.overrides, agent_type=config.agent_type, lineage=lineage) # Final guard: agent.type must be set after all 5 layers have merged. if task.agent.type is None: @@ -641,6 +642,53 @@ def resolve_task_files( resolve_template_source_paths(task.sandbox.template_sources, exp_dir) +def resolve_variant_task( + default_experiment: ExperimentDefinition, + task: TaskDefinition, + experiment: ExperimentDefinition, + variant: ExperimentVariant, + config: BatchRunConfig | None, + *, + task_file: Path, + experiment_file: Path | None, +) -> tuple[TaskDefinition, dict[str, ConfigLineageEntry], int]: + """One (task x variant) through all five layers, its files inlined, then validated. + + ``run`` and ``plan`` both resolve through here, so ``plan`` rejects what ``run`` rejects. + + Raises: + TaskResolutionError: the resolved task fails a resolution check. + ValueError: a layer or a file path does not resolve. + """ + from .resolution_checks import validate_resolved_task + + resolved_task, lineage, effective_repeats = resolve_task_for_variant( + default_experiment, task, experiment, variant, config + ) + resolve_task_files(resolved_task, task_file, experiment_file) + _apply_prompt_overrides(resolved_task, experiment, variant, lineage) + _apply_cli_overrides(resolved_task, config, lineage) + validate_resolved_task(resolved_task) + return resolved_task, lineage, effective_repeats + + +def resolve_variant_prompt_files(experiment: ExperimentDefinition, experiment_file: Path | None) -> None: + """Inline every variant's ``initial_prompt_file``, relative to the experiment file. + + Raises: + ValueError: a variant uses ``initial_prompt_file`` and ``experiment_file`` is None. + """ + exp_dir = experiment_file.parent if experiment_file is not None else None + for variant in experiment.variants: + if variant.initial_prompt_file is not None: + if exp_dir is None: + raise ValueError( + f"variant '{variant.variant_id}' uses initial_prompt_file but no experiment file path " + + "is available for resolving relative paths" + ) + resolve_variant_initial_prompt_file(variant, exp_dir) + + def resolve_all_tasks( task_files: list[Path], experiment: ExperimentDefinition, @@ -674,7 +722,6 @@ def resolve_all_tasks( ValueError: If duplicate task IDs are found after resolution. """ from .harness_contract import TaskResolutionError - from .resolution_checks import validate_resolved_task resolved: list[ResolvedTask] = [] skipped: list[SkippedTask] = [] @@ -683,16 +730,7 @@ def resolve_all_tasks( resolution_errors: list[tuple[Path, Exception]] = [] attempted = 0 - # Resolve variant-level initial_prompt_file paths before the main loop - exp_dir = experiment_file.parent if experiment_file is not None else None - for variant in experiment.variants: - if variant.initial_prompt_file is not None: - if exp_dir is None: - raise ValueError( - f"variant '{variant.variant_id}' uses initial_prompt_file but no experiment file path " - + "is available for resolving relative paths" - ) - resolve_variant_initial_prompt_file(variant, exp_dir) + resolve_variant_prompt_files(experiment, experiment_file) for task_file in task_files: try: @@ -728,24 +766,16 @@ def resolve_all_tasks( try: for expanded_task in expanded_tasks: for variant in experiment.variants: - # Apply layers 1-4 (default → experiment-defaults → task → variant) + resolve repeats - resolved_task, lineage, effective_repeats = resolve_task_for_variant( - default_experiment, expanded_task, experiment, variant, config + resolved_task, lineage, effective_repeats = resolve_variant_task( + default_experiment, + expanded_task, + experiment, + variant, + config, + task_file=task_file, + experiment_file=experiment_file, ) - # Resolve file paths injected by variant overrides - resolve_task_files(resolved_task, task_file, experiment_file) - - # Apply prompt mutations or overrides (between file resolution and CLI overrides) - _apply_prompt_overrides(resolved_task, experiment, variant, lineage) - - # Apply layer 5 (CLI overrides) - _apply_cli_overrides(resolved_task, config, lineage) - - # Once the task is fully resolved, so the -D kill switch is - # already merged. No-op unless armed. - validate_resolved_task(resolved_task) - # Fan-out: simulation n_trials takes precedence over experiment repeats # when simulation is active; otherwise use experiment-level repeats. sim = resolved_task.simulation diff --git a/src/coder_eval/orchestration/harness_contract.py b/src/coder_eval/orchestration/harness_contract.py index 495c06b7..d2b0a929 100644 --- a/src/coder_eval/orchestration/harness_contract.py +++ b/src/coder_eval/orchestration/harness_contract.py @@ -15,6 +15,7 @@ PermissionMode, ToolNameMap, ) +from coder_eval.pricing import is_priced if TYPE_CHECKING: @@ -44,6 +45,8 @@ class HarnessContractError(TaskResolutionError): "disallowed_tools": "disallowed_tools", } +MODEL_TURN_LIMITS: tuple[str, ...] = ("max_turns", "expected_turns") + def registration_for(task: TaskDefinition, *, requirement: str, hint: str = "") -> AgentRegistration[Any]: """The registry entry for the task's resolved agent kind. @@ -78,13 +81,16 @@ def registration_for(task: TaskDefinition, *, requirement: str, hint: str = "") def validate_harness_contract(task: TaskDefinition) -> None: """Reject agent config the task's harness cannot honor with its documented meaning. - Three checks, in order: a gated field set on a harness whose contract marks it + Five checks, in order: a gated field set on a harness whose contract marks it unsupported; a ``permission_mode`` value outside the contract's ``permission_modes``; a tool-list name outside ``CANONICAL_TOOL_NAMES`` (or an - ``mcp__`` name the harness cannot address). A field is set when a config layer - wrote it with a value other than None or an empty tool list (which restricts - nothing). A task without an agent type returns - silently; the layer-5 type guard reports that. + ``mcp__`` name the harness cannot address); a non-None model-turn run limit + (``MODEL_TURN_LIMITS``) on a harness whose contract does not count model turns; + ``run_limits.max_usd`` on a harness that does not report cost, with an + ``agent.model`` the rate card cannot price. + An agent field is set when a config layer wrote it with a value other than None + or an empty tool list (which restricts nothing). A task without an agent type + returns silently; the layer-5 type guard reports that. Raises: HarnessContractError: on the first violation, or an unregistered kind. @@ -110,6 +116,36 @@ def validate_harness_contract(task: TaskDefinition) -> None: for field in ("allowed_tools", "disallowed_tools"): if field in set_fields and tool_names is not None: _check_tool_names(field, getattr(task.agent, field), tool_names, kind) + _check_model_turn_limits(task, contract, kind) + _check_max_usd_priceable(task, contract, kind) + + +def _check_model_turn_limits(task: TaskDefinition, contract: HarnessContract, kind: str) -> None: + limits = task.run_limits + if limits is None or contract.counts_model_turns: + return + for field in MODEL_TURN_LIMITS: + if getattr(limits, field) is not None: + raise HarnessContractError( + f"run_limits.{field} is set but the {kind!r} harness reports no per-response turn boundary " + + f"(usage_granularity={contract.usage_granularity.value}; see docs/agents/HARNESS_PARITY.md). " + + "Remove the field, or set it only in a variant for a harness that counts model turns " + + f"({_honoring_kinds(lambda c: c.counts_model_turns)})." + ) + + +def _check_max_usd_priceable(task: TaskDefinition, contract: HarnessContract, kind: str) -> None: + if task.run_limits is None or task.run_limits.max_usd is None or contract.reports_cost: + return + model = task.agent.model if task.agent is not None else None + if model and is_priced(model): + return + subject = f"agent.model {model!r} has no rate in coder_eval.pricing" if model else "agent.model is not set" + raise HarnessContractError( + f"run_limits.max_usd is set but the {kind!r} harness does not report its own cost and {subject}, " + + "so the budget cannot be priced. Pin a priced agent.model, register a rate with register_pricing, " + + f"or remove max_usd. Harnesses that report their own cost: {_honoring_kinds(lambda c: c.reports_cost)}." + ) def _is_set(agent: BaseAgentConfig, field: str) -> bool: diff --git a/src/coder_eval/orchestration/plugin_staging.py b/src/coder_eval/orchestration/plugin_staging.py index 71274a8c..27a19e7e 100644 --- a/src/coder_eval/orchestration/plugin_staging.py +++ b/src/coder_eval/orchestration/plugin_staging.py @@ -1,12 +1,13 @@ -"""Stage every ``agent.plugins`` entry into one canonical plugin root each harness receives. +"""Stage every ``agent.plugins`` entry into one plugin root each harness receives. -The canonical root is ``/.claude-plugin/plugin.json`` plus -``/skills/`` (a symlink to the authored skill directory, or a copy where -symlinks fail). A plugin root is read the way Claude Code reads it: the default -``skills/`` plus every manifest-declared path, where a path may parent skills or be one -skill, and a root holding ``SKILL.md`` is a single-skill plugin. A bare skills -directory is accepted too. A skill's name is its ``SKILL.md`` frontmatter ``name``, -else its directory name. +The root holds ``/skills/``, a skills index every harness reads (a symlink +to the authored skill directory, or a copy where symlinks fail), and +``/plugins/``, each authored plugin whole for a harness that loads full +plugins (a bare skills directory gets a name-only manifest wrapper). A plugin root is +read the way Claude Code reads it: the default ``skills/`` plus every manifest-declared +path inside the root, where a path may parent skills or be one skill, and a root holding +``SKILL.md`` is a single-skill plugin. A bare skills directory is accepted too. A +skill's name is its ``SKILL.md`` frontmatter ``name``, else its directory name. Rationale: .claude/notes/agents.md § Skills, per harness """ @@ -15,7 +16,7 @@ import json import shutil -from collections.abc import Sequence +from collections.abc import Mapping, Sequence from dataclasses import dataclass from pathlib import Path from typing import TYPE_CHECKING, Any @@ -31,23 +32,22 @@ from coder_eval.models import LocalPluginConfig, TaskDefinition -STAGED_MANIFEST: dict[str, str] = {"name": "coder-eval-plugins"} - _MANIFEST_RELPATH = (".claude-plugin", "plugin.json") _DEFAULT_SKILLS_SUBDIR = "skills" +_PLUGINS_SUBDIR = "plugins" _SKILL_FILE = "SKILL.md" @dataclass(frozen=True) class StagedPlugins: - """The staged root handed to ``Agent.start`` and the skill names it offers.""" + """The staged root handed to ``Agent.start`` (``skills/`` and ``plugins/``) and the skill names it offers.""" root: Path skills_offered: tuple[str, ...] class PluginStagingError(TaskResolutionError): - """A plugins: entry that yields no skill, a duplicate skill name, or an unresolvable path.""" + """A plugins: entry that yields no skill, a duplicate skill or plugin name, or an unresolvable path.""" def resolve_plugin_path(raw: str) -> Path: @@ -66,21 +66,42 @@ def resolve_plugin_path(raw: str) -> Path: return root -def _declared_skill_paths(root: Path) -> list[Path]: +def _read_manifest(root: Path) -> dict[str, Any]: + """The parsed ``.claude-plugin/plugin.json``, or ``{}`` when it is absent, unreadable or not an object.""" manifest = root.joinpath(*_MANIFEST_RELPATH) - declared: list[str] = [] - if manifest.is_file(): - try: - data: Any = json.loads(manifest.read_text(encoding="utf-8")) - except (OSError, UnicodeDecodeError, json.JSONDecodeError): - data = None - if isinstance(data, dict): - value = data.get("skills") - if isinstance(value, str): - declared = [value] - elif isinstance(value, list): - declared = [entry for entry in value if isinstance(entry, str)] - return [(root / relative).resolve() for relative in declared] + if not manifest.is_file(): + return {} + try: + data: Any = json.loads(manifest.read_text(encoding="utf-8")) + except (OSError, UnicodeDecodeError, json.JSONDecodeError): + return {} + return data if isinstance(data, dict) else {} + + +def _declared_skill_paths(root: Path) -> list[Path]: + value = _read_manifest(root).get("skills") + if isinstance(value, str): + declared = [value] + elif isinstance(value, list): + declared = [entry for entry in value if isinstance(entry, str)] + else: + declared = [] + paths: list[Path] = [] + for relative in declared: + path = (root / relative).resolve() + if not path.is_relative_to(root): + raise PluginStagingError( + f"agent.plugins {root}: manifest skills path {relative!r} leaves the plugin root; " + + "Claude Code loads no skill from it" + ) + paths.append(path) + return paths + + +def _plugin_name(root: Path) -> str: + """The manifest ``name`` when it is a non-empty string, else the directory name: Claude Code's rule.""" + name = _read_manifest(root).get("name") + return name if isinstance(name, str) and name else root.name def _skill_name(skill_dir: Path) -> str: @@ -97,8 +118,8 @@ def _skill_name(skill_dir: Path) -> str: return skill_dir.name -def _skill_dirs(root: Path) -> list[Path]: - """Every skill directory one ``plugins:`` root offers, in Claude Code's reading order.""" +def _plugin_skill_dirs(root: Path) -> list[Path]: + """The skill directories Claude Code itself loads from ``root`` given as a plugin, in its reading order.""" candidates = [root / _DEFAULT_SKILLS_SUBDIR, *_declared_skill_paths(root)] found: list[Path] = [] for candidate in candidates: @@ -108,26 +129,40 @@ def _skill_dirs(root: Path) -> list[Path]: found += [skill_file.parent for skill_file in sorted(candidate.glob(f"*/{_SKILL_FILE}"))] if found: return found - if (root / _SKILL_FILE).is_file(): - return [root] - return [skill_file.parent for skill_file in sorted(root.glob(f"*/{_SKILL_FILE}"))] + return [root] if (root / _SKILL_FILE).is_file() else [] + + +def _skill_dirs(root: Path) -> list[Path]: + """Every skill directory one ``plugins:`` root offers: the plugin reading, else a bare skills directory.""" + return _plugin_skill_dirs(root) or [skill_file.parent for skill_file in sorted(root.glob(f"*/{_SKILL_FILE}"))] + + +def _claim_name(kind: str, name: str, source: Path, taken: Mapping[str, Path], hint: str) -> None: + """Refuse a staged name that is not one path segment, or that another source took (ignoring case). + + Case is ignored because the staged names are directory entries on a filesystem that may fold case. + """ + if name in {"", ".", ".."} or any(char in name for char in "/\\\x00"): + raise PluginStagingError(f"agent.plugins {source}: {kind} name {name!r} is not one path segment; {hint}") + for other, previous in taken.items(): + if other.casefold() == name.casefold() and previous != source: + raise PluginStagingError( + f"ambiguous {kind} name {name!r}: agent.plugins offers it from both {previous} and {source}; {hint}" + ) def scan_plugin_skills(plugins: Sequence[LocalPluginConfig]) -> dict[str, Path]: """Skill name -> its directory, over every entry and every accepted layout. Raises: - PluginStagingError: an unresolvable path, a skill name from two sources, or no skill at all. + PluginStagingError: an unresolvable path, a skill name that is not one path segment, a skill + name from two sources, or no skill at all. """ skills: dict[str, Path] = {} for plugin in plugins: for skill_dir in _skill_dirs(resolve_plugin_path(plugin["path"])): name, source = _skill_name(skill_dir), skill_dir.resolve() - previous = skills.get(name) - if previous is not None and previous != source: - raise PluginStagingError( - f"ambiguous skill name {name!r}: agent.plugins offers it from both {previous} and {source}" - ) + _claim_name("skill", name, source, skills, "rename it in its SKILL.md frontmatter") skills[name] = source if not skills: paths = [plugin["path"] for plugin in plugins] @@ -138,6 +173,27 @@ def scan_plugin_skills(plugins: Sequence[LocalPluginConfig]) -> dict[str, Path]: return skills +def scan_plugin_roots(plugins: Sequence[LocalPluginConfig]) -> dict[str, tuple[Path, bool]]: + """Plugin name -> (resolved root, wrapped), where a wrapped root is a bare skills directory. + + A root is wrapped when Claude Code would load no skill from it as a plugin but it + holds ``/SKILL.md`` skills. Every other root, including one with no skill, is + linked whole. + + Raises: + PluginStagingError: an unresolvable path, a plugin name that is not one path segment, + or one plugin name from two roots. + """ + roots: dict[str, tuple[Path, bool]] = {} + for plugin in plugins: + root = resolve_plugin_path(plugin["path"]) + name = _plugin_name(root) + taken = {other: previous for other, (previous, _wrapped) in roots.items()} + _claim_name("plugin", name, root, taken, "set a distinct name in its .claude-plugin/plugin.json") + roots[name] = (root, not _plugin_skill_dirs(root) and bool(_skill_dirs(root))) + return roots + + def validate_plugins(task: TaskDefinition) -> None: """Resolution-time refusal; no-op when plugins is unset or empty. @@ -146,12 +202,14 @@ def validate_plugins(task: TaskDefinition) -> None: placeholder is checked on its expanded row instead. Raises: - PluginStagingError: see ``scan_plugin_skills``, or a ``skill_triggered`` target not offered. + PluginStagingError: see ``scan_plugin_skills`` and ``scan_plugin_roots``, or a ``skill_triggered`` + target not offered. """ plugins = task.agent.plugins if task.agent is not None else None if not plugins: return offered = scan_plugin_skills(plugins) + scan_plugin_roots(plugins) targets = {c.skill_name for c in task.success_criteria if isinstance(c, SkillTriggeredCriterion)} missing = sorted(name for name in targets if "${" not in name and name not in offered) if missing: @@ -163,33 +221,67 @@ def validate_plugins(task: TaskDefinition) -> None: def link_or_copy(source: Path, target: Path) -> None: - """Symlink ``target`` to ``source``, or copy the tree where symlinks are unavailable.""" + """Symlink ``target`` to ``source``, or copy the tree where symlinks are unavailable. + + Only a failure to create symlinks at all falls back: an existing or unreachable ``target`` raises. The copy + follows symlinks, since the host cannot make them, but skips a link to one of its own ancestors and skips + ``target`` and its ancestors, so neither a link loop nor a source that contains the target recurses. + """ try: target.symlink_to(source, target_is_directory=True) + except (FileExistsError, FileNotFoundError): + raise except (OSError, NotImplementedError): - shutil.copytree(source, target, dirs_exist_ok=True) + resolved = target.resolve() + + def _skipped(directory: str, names: list[str]) -> list[str]: + entries = [Path(directory) / name for name in names] + return [ + entry.name + for entry in entries + if resolved.is_relative_to(entry) + or (entry.is_symlink() and entry.resolve() in entry.absolute().parents) + ] + + shutil.copytree(source, target, dirs_exist_ok=True, ignore_dangling_symlinks=True, ignore=_skipped) def stage_plugins(plugins: Sequence[LocalPluginConfig], staging_dir: Path) -> StagedPlugins: - """Write ``/.claude-plugin/plugin.json`` and ``/skills/``. + """Write ``/skills/`` per skill and ``/plugins/`` per entry. - The returned root is absolute: every harness runs with the sandbox as its cwd. An - existing ``staging_dir`` is removed first, so a re-executed row starts clean. + A plugin entry is linked whole; a bare skills directory becomes + ``plugins//.claude-plugin/plugin.json`` (name only) plus a ``skills`` link. The + returned root is absolute: every harness runs with the sandbox as its cwd. An existing + ``staging_dir`` is removed first, so a re-executed row starts clean. Raises: - PluginStagingError: see ``scan_plugin_skills``. + PluginStagingError: see ``scan_plugin_skills`` and ``scan_plugin_roots``. """ skills = scan_plugin_skills(plugins) + roots = scan_plugin_roots(plugins) staging_dir = staging_dir.absolute() if staging_dir.is_symlink() or staging_dir.is_file(): staging_dir.unlink() elif staging_dir.exists(): shutil.rmtree(staging_dir) - manifest = staging_dir.joinpath(*_MANIFEST_RELPATH) - manifest.parent.mkdir(parents=True) - manifest.write_text(json.dumps(STAGED_MANIFEST), encoding="utf-8") skills_dir = staging_dir / _DEFAULT_SKILLS_SUBDIR - skills_dir.mkdir() + skills_dir.mkdir(parents=True) for name, source in sorted(skills.items()): link_or_copy(source, skills_dir / name) + plugins_dir = staging_dir / _PLUGINS_SUBDIR + plugins_dir.mkdir() + for name, (root, wrapped) in sorted(roots.items()): + if wrapped: + manifest = plugins_dir.joinpath(name, *_MANIFEST_RELPATH) + manifest.parent.mkdir(parents=True) + manifest.write_text(json.dumps({"name": name}), encoding="utf-8") + link_or_copy(root, plugins_dir / name / _DEFAULT_SKILLS_SUBDIR) + else: + link_or_copy(root, plugins_dir / name) return StagedPlugins(root=staging_dir, skills_offered=tuple(sorted(skills))) + + +def staged_plugin_dirs(root: Path) -> list[Path]: + """Each ``/plugins/`` a staged root holds, in name order; empty without a ``plugins/``.""" + plugins_dir = root / _PLUGINS_SUBDIR + return sorted(plugins_dir.iterdir()) if plugins_dir.is_dir() else [] diff --git a/src/coder_eval/orchestration/turn_monitor.py b/src/coder_eval/orchestration/turn_monitor.py index 0e30ac7a..172a038f 100644 --- a/src/coder_eval/orchestration/turn_monitor.py +++ b/src/coder_eval/orchestration/turn_monitor.py @@ -1,4 +1,4 @@ -"""The run's single ``should_stop`` answerer: armed early stop, the tool-call cap and the budgets. +"""The run's single ``should_stop`` answerer: armed early stop, the tool-call and model-turn caps and the budgets. ``TurnMonitor`` is a ``StreamCallback`` composed into the agent's callback chain for the whole task. It owns ONE ``EventCollector`` across every retry attempt and @@ -6,13 +6,13 @@ polls ``should_stop`` at its safe boundaries; the first non-None ``StopReason`` is latched and final. -Precedence on one round: ``EARLY_CRITERION``, ``TOOL_CALL_CAP``, ``TOKEN_BUDGET``, -``USD_BUDGET``. A budget breach seen mid-turn latches its reason and its figures, so a +Precedence on one round: ``EARLY_CRITERION``, ``TOOL_CALL_CAP``, ``MODEL_TURN_CAP``, +``TOKEN_BUDGET``, ``USD_BUDGET``. A budget breach seen mid-turn latches its reason and its figures, so a turn that stopped on a budget always finalizes as that budget's status. FAIL-OPEN covers the armed criteria only: any exception while reducing an event or -evaluating them disarms them and the run degrades to a full run. The cap reads -counters and is checked on every resolved call regardless, so it never disarms. +evaluating them disarms them and the run degrades to a full run. The caps read +counters, so they never disarm. Rationale: .claude/notes/orchestration.md § Early stop on criterion """ @@ -20,7 +20,6 @@ from __future__ import annotations import logging -import math import time from typing import TYPE_CHECKING, Any @@ -36,7 +35,7 @@ TokenUsage, ) from coder_eval.orchestration.early_stop import early_stop_active -from coder_eval.pricing import calculate_cost +from coder_eval.pricing import price_turn from coder_eval.streaming.collector import EventCollector from coder_eval.streaming.events import ( AgentEndEvent, @@ -63,6 +62,17 @@ _DISARMED = "armed criteria disarmed, run degrades to a full run" +def _reports_cost(task: TaskDefinition) -> bool: + from coder_eval.agents.registry import AgentRegistry + from coder_eval.plugins import ensure_plugins_loaded + + if task.agent is None or task.agent.type is None: + return False + ensure_plugins_loaded() + registration = AgentRegistry.get(str(task.agent.type)) + return registration is not None and registration.agent_class.contract.reports_cost + + class TurnMonitor: """Observes the agent event stream and answers the cooperative ``should_stop`` poll. @@ -80,6 +90,7 @@ def __init__( *, limits: RunLimits | None, model: str | None = None, + reports_cost: bool = False, gate_threshold: float = DEFAULT_STOP_EARLY_GATE_THRESHOLD, ) -> None: self._task_id = task_id @@ -127,11 +138,14 @@ def __init__( self._collector = EventCollector() self._resolved_tool_ids: set[str] = set() self._sdk_turn_index = 0 + self._call_turn_ids: set[str] = set() self._tool_call_index = 0 self._started_monotonic: float | None = None self._committed = TokenUsage() self._committed_cost = 0.0 self._unpriced_turn = False + self._reports_cost = reports_cost + self._unpriced_in_flight = False self._in_flight = TokenUsage() self._budget_breach: tuple[str, float, float] | None = None # Once an entry leaves "undecided" on a RESOLVED round its checker is @@ -170,7 +184,14 @@ def for_task(cls, task: TaskDefinition, *, arm: bool) -> TurnMonitor: limits = task.run_limits gate_threshold = limits.stop_early_gate_threshold if limits is not None else DEFAULT_STOP_EARLY_GATE_THRESHOLD model = task.agent.model if task.agent is not None else None - return cls(task.task_id, armed, limits=limits, model=model, gate_threshold=gate_threshold) + return cls( + task.task_id, + armed, + limits=limits, + model=model, + reports_cost=_reports_cost(task), + gate_threshold=gate_threshold, + ) def on_event(self, event: StreamEvent) -> None: """Reduce one event; an unexpected exception disarms the criteria and never stops the counters.""" @@ -188,19 +209,29 @@ def _on_event_impl(self, event: StreamEvent) -> None: increments on each resolved end. UNRESOLVED tool ends are RECORDED but never counted or evaluated on — they must still land in the collector, or the monitor would reduce a strictly smaller command set than the authoritative - check. The cap counts distinct resolved tool ids. + check. The cap counts distinct resolved tool ids. A main-thread turn start counts once + per turn id per ``communicate()``; the model-turn cap latches when turn N+1 starts. + + A nested (sub-agent) event is main-thread-scoped out: its tool end is recorded + but never counted or evaluated, its turn start sets no model, and only its + turn-end tokens count, toward the budgets. Rationale: .claude/notes/orchestration.md § Verdicts latch, and the decision happens on the CALL """ if event.parent_thread_id is not None: + self._on_nested_event(event) return if isinstance(event, AgentStartEvent): if self._started_monotonic is None: self._started_monotonic = time.monotonic() self._in_flight = TokenUsage() self._start_model = event.model or self._start_model + self._call_turn_ids = set() elif isinstance(event, TurnStartEvent): - self._sdk_turn_index += 1 + if event.turn_id not in self._call_turn_ids: + self._call_turn_ids.add(event.turn_id) + self._sdk_turn_index += 1 + self._evaluate_model_turn_cap() self._reported_model = event.model or self._reported_model elif isinstance(event, TurnEndEvent): if event.tokens is not None: @@ -224,6 +255,13 @@ def _on_event_impl(self, event: StreamEvent) -> None: return self._collector.on_event(event) + def _on_nested_event(self, event: StreamEvent) -> None: + if isinstance(event, ToolEndEvent): + self._collector.on_event(event) + elif isinstance(event, TurnEndEvent) and event.tokens is not None: + self._in_flight += event.tokens + self._evaluate_budgets() + def should_stop(self) -> StopReason | None: """The cooperative poll the agent calls at each safe boundary.""" return self._stop_reason @@ -253,6 +291,11 @@ def tool_calls(self) -> int: """Distinct resolved tool calls across the whole task.""" return len(self._resolved_tool_ids) + @property + def model_turns(self) -> int: + """Main-thread model turns started across the whole task, each turn id once per communicate().""" + return self._sdk_turn_index + @property def usage(self) -> TokenUsage: """Committed usage from every finished ``communicate()`` plus the in-flight deltas.""" @@ -261,9 +304,9 @@ def usage(self) -> TokenUsage: def cost_usd(self) -> float | None: """Cumulative USD: every finished turn priced on its own, plus the priceable in-flight deltas. - A turn is priced from its reported cost, else from the rate card for the first - priced model of ``agent.model``, the model the agent resolved at start, and the - last model a message reported; a turn with no usage costs 0. + A turn is priced by ``pricing.price_turn`` with the models ``agent.model``, the + model the agent resolved at start, and the last model a message reported, in + that order; a turn with no usage and no reported cost costs 0. ``None`` once any finished turn could be priced none of these ways. """ if self._unpriced_turn: @@ -276,13 +319,18 @@ def raise_if_over_budget(self, *, iteration: int) -> None: Raises: BudgetExceededError: a budget reason latched mid-turn (with the figures from that moment), or the finished turns' totals breach a budget. - BudgetUnenforceableError: ``max_usd`` is set and a finished turn was unpriceable. + BudgetUnenforceableError: ``max_usd`` is set and a finished turn was unpriceable, + or in-flight usage was unpriceable on a harness that does not report cost. """ breach = self._budget_breach if self._budget_breach is not None else self._breach() if breach is not None: name, actual, limit = breach raise BudgetExceededError(name, actual=actual, limit=limit, task_id=self._task_id, iteration=iteration) - if self._limits is not None and self._limits.max_usd is not None and self._unpriced_turn: + if ( + self._limits is not None + and self._limits.max_usd is not None + and (self._unpriced_turn or self._unpriced_in_flight) + ): raise BudgetUnenforceableError( "run_limits.max_usd could not be enforced: the harness reported no cost and " + f"agent.model {self._model!r} (reported {self._reported_model!r}) has no rate in " @@ -300,23 +348,9 @@ def _commit(self, usage: TokenUsage) -> None: self._in_flight = TokenUsage() def _price(self, usage: TokenUsage) -> float | None: - if usage.total_cost_usd is not None: - return usage.total_cost_usd if math.isfinite(usage.total_cost_usd) else None if usage.is_empty(): - return 0.0 - for model in (self._model, self._start_model, self._reported_model): - if model is None: - continue - cost = calculate_cost( - model, - usage.uncached_input_tokens, - usage.output_tokens, - usage.cache_creation_input_tokens, - usage.cache_read_input_tokens, - ) - if cost is not None: - return cost - return None + return price_turn(usage, ()) or 0.0 + return price_turn(usage, (self._model, self._start_model, self._reported_model)) def _breach(self) -> tuple[str, float, float] | None: """The first budget over its cap as ``(budget name, actual, limit)``: input, output, total, usd.""" @@ -341,12 +375,26 @@ def _evaluate_budgets(self) -> None: return breach = self._breach() if breach is None: + self._evaluate_in_flight_priceable() return self._budget_breach = breach name, actual, limit = breach logger.info("[%s] %s budget reached: %g > %g", self._task_id, name, actual, limit) self._latch(StopReason.USD_BUDGET if name == "usd" else StopReason.TOKEN_BUDGET) + def _evaluate_in_flight_priceable(self) -> None: + """Latch ``USD_BUDGET`` when ``max_usd`` is set and in-flight usage has no price. + + A harness that reports cost prices the turn at its end, so its in-flight usage is exempt. + """ + if self._limits is None or self._limits.max_usd is None or self._reports_cost: + return + if self._in_flight.is_empty() or self._price(self._in_flight) is not None: + return + self._unpriced_in_flight = True + logger.error("[%s] usd budget cannot be enforced: the in-flight usage has no price", self._task_id) + self._latch(StopReason.USD_BUDGET) + def _latch(self, reason: StopReason) -> None: if self._stop_reason is None: self._stop_reason = reason @@ -369,6 +417,18 @@ def _evaluate_cap(self) -> None: ) self._latch(StopReason.TOOL_CALL_CAP) + def _evaluate_model_turn_cap(self) -> None: + cap = self._limits.max_turns if self._limits is not None else None + if cap is not None and self._sdk_turn_index > cap: + if self._stop_reason is None: + logger.info( + "[%s] model-turn cap reached: model turn %d started (cap %d)", + self._task_id, + self._sdk_turn_index, + cap, + ) + self._latch(StopReason.MODEL_TURN_CAP) + def _ceiling(self, verdicts: list[LiveVerdict]) -> float: """Best-case weighted score over the WHOLE armed set, given current verdicts. diff --git a/src/coder_eval/orchestrator.py b/src/coder_eval/orchestrator.py index 6506289a..fbbedf90 100644 --- a/src/coder_eval/orchestrator.py +++ b/src/coder_eval/orchestrator.py @@ -11,7 +11,6 @@ from contextlib import suppress from dataclasses import dataclass from datetime import datetime -from inspect import isawaitable from pathlib import Path from typing import Any, NamedTuple from urllib.parse import urlparse @@ -67,6 +66,7 @@ ) from .orchestration.early_stop import early_stop_active, validate_early_stop from .orchestration.evaluation import resolve_reference_dir, stage_reference_dir +from .orchestration.harness_contract import registration_for from .orchestration.plugin_staging import stage_plugins from .orchestration.resolution_checks import validate_resolved_task from .orchestration.run_limits import validate_run_limits @@ -84,7 +84,8 @@ from .sandbox import Sandbox from .simulation import DialogStopReason, SimulatorResult, UserSimulator, evaluate_stop from .streaming.callbacks import CompositeStreamCallback, StreamCallback, TaskScopedCallback, safe_emit -from .streaming.events import CriteriaCheckEvent, CriterionSummary +from .streaming.collector import EventCollector +from .streaming.events import AgentEndStatus, CriteriaCheckEvent, CriterionSummary from .telemetry import Scalar, hash_identifier from .utils import get_version_info, looks_like_version, runtime_uip_versions @@ -93,10 +94,22 @@ logger = logging.getLogger(__name__) -# Grace on outer wait_for so the agent's in-band watchdog (which preserves a partial) -# wins the race against the asyncio cancel path (which doesn't). +# Grace on outer wait_for so the agent's in-band watchdog (a TIMEOUT outcome) wins the +# race against the asyncio cancel path (a CRASHED "turn cancelled" record). _WAIT_FOR_GRACE_SECONDS = 2.0 +# The clean end statuses a turn's outcome is returned for; CRASHED and TIMEOUT raise. +# An allowlist, so a new AgentEndStatus member fails loudly until it is placed. +_RETURNED_END_STATUSES = frozenset( + { + AgentEndStatus.COMPLETED, + AgentEndStatus.STOPPED_EARLY, + AgentEndStatus.TOOL_CALLS_EXHAUSTED, + AgentEndStatus.TOKEN_BUDGET_EXCEEDED, + AgentEndStatus.COST_BUDGET_EXCEEDED, + } +) + def _close_subprocess_transport(proc: asyncio.subprocess.Process | None) -> None: """Release a finished subprocess's pipe transport deterministically. @@ -478,13 +491,20 @@ def __init__( # count it answers the should_stop poll from is cumulative per task. self._monitor: TurnMonitor | None = None + # The in-flight communicate attempt's own collector, so a task timeout can + # recover the turn the agent ended before the cancel propagated. Cleared on + # every other exit, so a finished attempt is never appended twice. + self._attempt_collector: EventCollector | None = None + # The skill names the staged plugin root offered; None when the task sets # no plugins. Read back from the prior result on an evaluate-only grade. self._skills_offered: tuple[str, ...] | None = None - # One-shot flag: emit the expected_tool_calls rollup warning exactly once per - # task run even though _check_expected_tool_calls is called after every turn. + # One-shot flags: emit the expected_tool_calls and expected_turns warnings exactly + # once per task run even though _check_expected_targets is called after every turn. self._expected_tool_calls_warning_emitted: bool = False + self._expected_turns_warning_emitted: bool = False + self._counts_model_turns: bool = False # One-shot flag: a resolved task may be inspected more than once during # setup, but its ineffective timeout relationship should be logged once. @@ -648,8 +668,7 @@ def _kill_agent_subprocess_sync() -> None: logger.error(f"Task timed out: {e}") # Nothing else on this path recovers the in-flight turn: the - # cancel arrives as a BaseException, so it never reaches the retry - # executor's per-attempt hook. + # cancel arrives as a BaseException, so no outcome ever returns. await self._drain_killed_turn() except BudgetExceededError as e: # Map token-budget breaches and cost-budget breaches to distinct @@ -763,6 +782,7 @@ def _seed_from_prior_result(self) -> None: # Execution facts that outlive the agent process. self.result.tool_calls_exhausted = prior.tool_calls_exhausted + self.result.model_turns = prior.model_turns self.result.error_message = prior.error_message self.result.error_details = prior.error_details self.result.error_log_tail = prior.error_log_tail @@ -931,27 +951,20 @@ async def _evaluate_post_failure_criteria(self) -> None: self.result.post_failure_criteria_results = recovered async def _drain_killed_turn(self) -> None: - """Move a hard-killed turn's partial record from the agent onto the result. - - The only reader of ``pending_turn`` on the task-timeout path. Ordering - matters both ways: it must run before ``_cleanup`` (whose ``agent.stop()`` - clears the slot) and before ``_finalize_result``, so the recovered turn - feeds token aggregation and command stats like any other. + """Move a hard-killed turn's record from the in-flight attempt's collector onto the result. - Best-effort: a task killed before its first turn has nothing parked, and - this runs on the way to a saved row, so it must not raise. + Runs before ``_cleanup`` and ``_finalize_result``, so the recovered turn feeds + token aggregation and command stats like any other. Best-effort: a task + killed before its turn ended has nothing to recover, and this must not raise. """ - if self.agent is None or self.result is None: + if self.result is None: return try: - partial = self.agent.pending_turn - # `pending_turn` is a slot any agent implementation fills, so a non-record - # here would fail validation during teardown and take the row down with it. - if not isinstance(partial, TurnRecord): + collector = self._attempt_collector + partial = self._append_attempt_record(collector) if collector is not None else None + if partial is None: logger.debug("[%s] Hard-killed task preserved no partial turn", self.task.task_id) return - self.result.iterations.append(partial) - await self.agent.discard_pending_turn() usage = partial.token_usage logger.info( "[%s] Recovered the hard-killed turn: %d tokens, %s", @@ -964,6 +977,17 @@ async def _drain_killed_turn(self) -> None: except Exception: logger.warning("[%s] Could not recover the hard-killed turn", self.task.task_id, exc_info=True) + def _append_attempt_record(self, collector: EventCollector) -> TurnRecord | None: + """Append the attempt's record when its turn ended, once; the collector is then spent.""" + assert self.result is not None + self._attempt_collector = None + if not collector.ended: + logger.debug("[%s] The killed attempt never ended its turn; nothing appended", self.task.task_id) + return None + record = collector.build_turn_record() + self.result.iterations.append(record) + return record + def _finalize_weighted_score(self) -> None: """Write ``weighted_score``, or ``None`` when this run was not graded. @@ -1155,33 +1179,50 @@ def _finalize_result(self, start_time: float) -> None: write_task_html(self.result, self.html_report_path) - def _check_expected_tool_calls(self, *, iteration: int) -> None: - """Emit a one-shot warning if visible tool calls exceed expected_tool_calls. + def _check_expected_targets(self, *, iteration: int) -> None: + """One-shot warnings when visible tool calls exceed ``expected_tool_calls`` or model turns exceed + ``expected_turns``; never aborts. - Soft sibling of the hard tool-call cap: never aborts the run. The count is - one timeline entry per tool call plus the final reply when present — the - same metric evalboard renders. Cumulative across iterations so dialog tasks - compare against the budget the user set. + Soft siblings of the hard caps. Visible tool calls are one timeline entry per tool + call plus the final reply when present — the same metric evalboard renders. Model + turns are ``result.model_turns``, the TurnMonitor's count. Both are cumulative across + iterations so dialog tasks compare against the target the user set. """ if self.result is None: return limits = self.task.run_limits - if limits is None or limits.expected_tool_calls is None: - return - if self._expected_tool_calls_warning_emitted: + if limits is None: return - total = visible_turn_count(self.result) - if total > limits.expected_tool_calls: + if limits.expected_tool_calls is not None and not self._expected_tool_calls_warning_emitted: + total = visible_turn_count(self.result) + if total > limits.expected_tool_calls: + logger.warning( + "Visible tool calls (%d) exceeded expected_tool_calls (%d) at iteration %d " + + "for task %s. Run continues — this target never aborts.", + total, + limits.expected_tool_calls, + iteration, + self.task.task_id, + ) + self._expected_tool_calls_warning_emitted = True + + model_turns = self.result.model_turns + if ( + limits.expected_turns is not None + and not self._expected_turns_warning_emitted + and model_turns is not None + and model_turns > limits.expected_turns + ): logger.warning( - "Visible tool calls (%d) exceeded expected_tool_calls (%d) at iteration %d " - + "for task %s. Run continues — this target never aborts.", - total, - limits.expected_tool_calls, + "Model turns (%d) exceeded expected_turns (%d) at iteration %d for task %s. " + + "Run continues — this target never aborts.", + model_turns, + limits.expected_turns, iteration, self.task.task_id, ) - self._expected_tool_calls_warning_emitted = True + self._expected_turns_warning_emitted = True def _warn_on_ineffective_task_timeout(self) -> None: """Log resolved cross-field run-limit warnings once per task run.""" @@ -1372,21 +1413,6 @@ def _build_monitor(self) -> None: + "the full trajectory is the deliverable." ) - def _restore_recorded_command_path(self) -> None: - """Re-apply the graded run's own PATH before its criteria run. - - PATH parity with the run being graded. `_sync_sandbox_command_path_with_ - agent` recorded the agent's effective PATH; no agent runs on the - evaluate-only path, so restore it explicitly or `run_command` criteria - resolve binaries against ambient PATH and can disagree with the original - verdict. - """ - assert self.result is not None - assert self.sandbox is not None - restored_path = self.result.environment_info.get("command_base_path") - if isinstance(restored_path, str) and restored_path: - self.sandbox.set_command_base_path(self._sanitize_restored_path(restored_path)) - async def _setup(self) -> None: """Set up all components for evaluation. @@ -1414,7 +1440,6 @@ async def _setup(self) -> None: self.sandbox.reference_dir = self._reference_dir self.result.sandbox_path = str(self.sandbox.sandbox_dir) - self._restore_recorded_command_path() recorded_skills = self.prior_result.environment_info.get("skills_offered") if self.prior_result else None if isinstance(recorded_skills, list): self._skills_offered = tuple(str(name) for name in recorded_skills) @@ -1426,6 +1451,9 @@ async def _setup(self) -> None: # After the evaluate-only return: a re-grade builds no agent, so a recorded # config from before the contract existed stays gradable. validate_resolved_task(self.task) + self._counts_model_turns = registration_for( + self.task, requirement="model-turn accounting" + ).agent_class.contract.counts_model_turns # validate_api_keys exempts the no-op agent internally — it makes no API # call, so it needs no agent keys. @@ -1518,6 +1546,12 @@ async def _start_agent() -> None: context={"task_id": self.task.task_id, "component": "agent", "agent_name": self._agent_name}, ) + try: + self.result.environment_info["harness_version"] = await self.agent.harness_version() + except Exception: + logger.warning("[%s] agent.harness_version() raised", self.task.task_id, exc_info=True) + self.result.environment_info["harness_version"] = None + # Save agent config on result (copy to prevent mutation of shared reference) self.result.agent_config = self.task.agent.model_copy(deep=True) @@ -1539,55 +1573,6 @@ async def _start_agent() -> None: if self.sandbox and self.sandbox.installed_tool_versions: self.result.environment_info["installed_tools"] = self.sandbox.installed_tool_versions - def _sync_sandbox_command_path_with_agent(self) -> None: - """Align criteria command PATH with the PATH used for the last agent query. - - Called from the per-turn happy path AFTER a successful - ``_communicate_with_retry``, which leaves three gaps: an agent crash or - turn timeout, evaluate-only mode, and the window before the first turn. In - each, criteria fall back to ambient ``os.environ['PATH']``. - - ``Agent.get_sdk_options()`` is declared synchronous on the ABC, but - ``AsyncMock`` fixtures return a coroutine for ANY attribute access, so it - is closed rather than awaited — a test-fixture concern, logged at DEBUG. A - non-dict, non-None return IS a production contract violation and warns. - - Rationale: .claude/notes/orchestration.md § Restoring a PATH from a run directory - """ - if self.agent is None or self.sandbox is None: - return - sdk_options = self.agent.get_sdk_options() - if sdk_options is None: - return - if isawaitable(sdk_options): - close = getattr(sdk_options, "close", None) - if callable(close): - # `close()` only documents RuntimeError, which cannot apply here, - # so narrow the suppress and let real exceptions propagate. - with suppress(RuntimeError): - close() - logger.debug( - "Agent.get_sdk_options() returned an awaitable; skipping PATH sync." - + " (Typical when tests stub the agent with AsyncMock.)" - ) - return - if not isinstance(sdk_options, dict): - logger.warning( - "Agent.get_sdk_options() returned non-dict %r; skipping PATH sync.", - type(sdk_options).__name__, - ) - return - sdk_env = sdk_options.get("env") - if not isinstance(sdk_env, dict): - return - path = sdk_env.get("PATH") - if isinstance(path, str) and path: - self.sandbox.set_command_base_path(path) - # Persisted so a LATER detached grade can restore the same PATH; - # otherwise it resolves run_command criteria against ambient PATH. - if self.result is not None: - self.result.environment_info["command_base_path"] = path - def _eval_route_overrides(self) -> EvalRouteOverrides: """The ``(backend, model)`` pair from ``task.checker_context.api_route``, if any. @@ -1812,13 +1797,12 @@ async def _communicate_with_retry( ) -> TurnRecord: """Run ``agent.communicate`` with retry, partial-preservation, and a per-attempt timeout. - Shared by the criteria-feedback and simulation loops. Crashed partials - from ``AgentCrashError`` / ``TurnTimeoutError`` are appended to - ``self.result.iterations`` via the ``on_attempt_error`` hook (terminal - failures included) so observational criteria still see them. Each - attempt gets a fresh ``turn_timeout``; ``TurnTimeoutError`` is - ``AGENT_TIMEOUT`` (``max_retries=0``) so it still terminates after - one attempt. + Shared by the criteria-feedback and simulation loops. A ``CRASHED`` or + ``TIMEOUT`` outcome's record is appended to ``self.result.iterations`` + before it is raised as ``AgentCrashError`` / ``TurnTimeoutError`` (terminal + failures included), so observational criteria still see it. Each attempt + gets a fresh ``turn_timeout``; ``TurnTimeoutError`` is ``AGENT_TIMEOUT`` + (``max_retries=0``) so it still terminates after one attempt. """ assert self.agent is not None assert self.task.agent is not None @@ -1832,82 +1816,59 @@ async def _communicate_with_retry( monitor = self._monitor assert monitor is not None, "TurnMonitor not built" - # The sole callback when --stream is off, else alongside the - # TaskScopedCallback. The same instance persists across retry attempts and - # dialog turns, so its counters and wall-clock origin accumulate. - agent_callback: StreamCallback = monitor - if self.stream_callback is not None: - agent_callback = CompositeStreamCallback( - [monitor, TaskScopedCallback(self.stream_callback, self._log_task_id)] - ) - - def _drain_pending_turn(*, attempt: int) -> None: - """Read agent.pending_turn and, if set, append it to result.iterations.""" - partial = agent.pending_turn - if partial is not None: - result.iterations.append(partial) - logger.debug( - "[%s] Drained partial turn record (attempt %d, iteration %d): %d commands", - self.task.task_id, - attempt + 1, - iteration, - len(partial.commands), - ) - else: - logger.debug( - "[%s] No pending_turn to drain on attempt %d (iteration %d)", - self.task.task_id, - attempt + 1, - iteration, - ) - - async def _on_attempt_failure( - err: Exception, - attempt: int, - ) -> None: - if not isinstance(err, (AgentCrashError, TurnTimeoutError)): - return - _drain_pending_turn(attempt=attempt) - try: - await agent.discard_pending_turn() - except Exception: - logger.warning( - "[%s] discard_pending_turn raised on attempt %d", - self.task.task_id, - attempt + 1, - exc_info=True, - ) + unhandled: list[AgentEndStatus] = [] async def _communicate_attempt() -> TurnRecord: - coro = agent.communicate( - prompt, - stream_callback=agent_callback, - timeout=turn_timeout, - should_stop=monitor.should_stop, - ) - if turn_timeout is None: - return await coro - # Grace buffer: agent's in-band watchdog (sets pending_turn) must beat - # wait_for cancel so the slot is populated before we give up. - outer_timeout = turn_timeout + _WAIT_FOR_GRACE_SECONDS + # A fresh collector per attempt is how a cancelled turn is recovered: + # the agent ends the turn before the cancel propagates, and + # `_drain_killed_turn` reads the record from here. + attempt_collector = EventCollector() + self._attempt_collector = attempt_collector + callbacks: list[StreamCallback] = [monitor, attempt_collector] + if self.stream_callback is not None: + callbacks.append(TaskScopedCallback(self.stream_callback, self._log_task_id)) + cancelled = False try: - return await asyncio.wait_for(coro, timeout=outer_timeout) - except TimeoutError: - # Watchdog wedged or too slow. Kill only — drain + discard happen - # in _on_attempt_failure when this TurnTimeoutError propagates up. - try: - await agent.kill() - except Exception: - logger.warning( - "[%s] agent.kill() raised on wait_for backstop path", - self.task.task_id, - exc_info=True, - ) - raise TurnTimeoutError( - turn_timeout, - task_id=self.task.task_id, + coro = agent.communicate( + prompt, iteration=iteration, - ) from None + stream_callback=CompositeStreamCallback(callbacks), + timeout=turn_timeout, + should_stop=monitor.should_stop, + ) + if turn_timeout is None: + outcome = await coro + else: + try: + # Grace buffer: the agent's own watchdog must beat this backstop. + outcome = await asyncio.wait_for(coro, timeout=turn_timeout + _WAIT_FOR_GRACE_SECONDS) + except TimeoutError: + try: + await agent.kill() + except Exception: + logger.warning( + "[%s] agent.kill() raised on wait_for backstop path", + self.task.task_id, + exc_info=True, + ) + self._append_attempt_record(attempt_collector) + raise TurnTimeoutError(turn_timeout, task_id=self.task.task_id, iteration=iteration) from None + if outcome.status in _RETURNED_END_STATUSES: + return outcome.record + if outcome.status in (AgentEndStatus.CRASHED, AgentEndStatus.TIMEOUT): + result.iterations.append(outcome.record) + return outcome.record_or_raise( + timeout_seconds=turn_timeout, task_id=self.task.task_id, iteration=iteration + ) + # Raised after the retry executor, which would retry a RuntimeError. + unhandled.append(outcome.status) + return outcome.record + except asyncio.CancelledError: + cancelled = True + raise + finally: + if not cancelled: + self._attempt_collector = None # ANTI-CHEAT WINDOW. Both the reference and the task dir sit at mode 000 # for the whole of every communicate attempt — retries included, since this @@ -1927,9 +1888,10 @@ async def _communicate_attempt() -> TurnRecord: "component": "agent", "agent_name": self._agent_name, }, - on_attempt_error=_on_attempt_failure, ) assert turn_record is not None # execute_with_retry returns the turn or raises + if unhandled: + raise RuntimeError(f"unhandled end status {unhandled[0]}") return turn_record @staticmethod @@ -1970,49 +1932,6 @@ def _accumulate_judge_usage( # forward so it isn't dropped from the latest results list. r.token_usage = prior - def _sanitize_restored_path(self, recorded: str) -> str: - """Filter a PATH restored from a run's own ``task.json`` before prepending it. - - The restored value arrives from inside the directory being graded — a - shareable artifact, bind-mounted writable into the agent's container under - ``driver: docker`` — so verbatim it lets a run dir decide which binary - ``pytest`` resolves to on the grader's host. - - Four filters: absolute paths only, existing directories only, nothing - inside the workspace, nothing inside the run directory. What remains is the - run's genuine toolchain locations. - - Rationale: .claude/notes/orchestration.md § Restoring a PATH from a run directory - """ - workspace = self.sandbox.sandbox_dir.resolve() if self.sandbox and self.sandbox.sandbox_dir else None - run_root = self.run_dir.resolve() - blocked = [p for p in (workspace, run_root) if p is not None] - kept: list[str] = [] - for entry in recorded.split(os.pathsep): - if not entry: - continue - candidate = Path(entry) - if not candidate.is_absolute(): - logger.warning( - "Dropping recorded PATH entry %r: it is relative, so it would resolve against " - + "the grader's working directory rather than the run's toolchain.", - entry, - ) - continue - if not candidate.is_dir(): - logger.debug("Dropping recorded PATH entry %s: not a directory here.", entry) - continue - resolved = candidate.resolve() - if any(resolved == root or root in resolved.parents for root in blocked): - logger.warning( - "Dropping recorded PATH entry %s: it lies inside the run being graded, " - + "so a binary there could shadow a real tool on the grader's host.", - entry, - ) - continue - kept.append(str(resolved)) - return os.pathsep.join(kept) - def _select_gate(self) -> bool: """Apply the verdict gate to the criteria results already on ``self.result``. @@ -2129,7 +2048,6 @@ async def _evaluation_loop(self) -> bool: operation_label="Agent communication", ) self.result.iterations.append(turn_record) - self._sync_sandbox_command_path_with_agent() # Record early-stop info (if the monitor tripped) BEFORE check_all_async, so it # survives even if a checker raises. None on a full run or when unarmed. @@ -2141,7 +2059,7 @@ async def _evaluation_loop(self) -> bool: # Facts about the RUN, recorded BEFORE the grading switch: `execute` # withholds the verdict, never the facts. Recording the fact is not - # finalizing on it — the tool-call cap decides the status only when the criteria + # finalizing on it — a structural cap decides the status only when the criteria # fail, so under grade=False this is carried into task.json for the # detached grade rather than turned into a terminal status. Read from the # turn's end status, not the monitor's latch: a cap latched after the @@ -2150,11 +2068,14 @@ async def _evaluation_loop(self) -> bool: if turn_record.tool_calls_exhausted: self.result.tool_calls_exhausted = True logger.warning( - "Agent reached the tool-call cap (%d resolved tool calls).", + "Agent reached a run cap (%s): %d resolved tool calls, %d model turns.", + monitor.stop_reason, monitor.tool_calls, + monitor.model_turns, ) - # Soft cumulative-turn check (logs once; never aborts). - self._check_expected_tool_calls(iteration=iteration) + self.result.model_turns = monitor.model_turns if self._counts_model_turns else None + # Soft cumulative targets (log once; never abort). + self._check_expected_targets(iteration=iteration) # Grading site 2 of 4. The trajectory is captured and persisted exactly as # on a graded run, but nothing is scored; returning False keeps FinalStatus @@ -2468,9 +2389,8 @@ async def _simulation_dialog_loop(self, initial_prompt: str | None, sandbox_dir: # Keyed by (position, criterion_type) — a stable criterion identity. judge_usage_accum: dict[tuple[int, str], TokenUsage] = {} - # In lockstep with the agent's _iteration — one - # _communicate_with_retry per sim turn — so a partial turn and its - # successful retry share an iteration number. + # One _communicate_with_retry per sim turn, passed this turn's number, + # so a partial turn and its successful retry share an iteration number. while True: turns_completed += 1 self.result.iteration_count = turns_completed @@ -2488,7 +2408,6 @@ async def _simulation_dialog_loop(self, initial_prompt: str | None, sandbox_dir: turn_record.messages.insert(0, pending_user_turn) pending_user_turn = None self.result.iterations.append(turn_record) - self._sync_sandbox_command_path_with_agent() dialog_pairs.append((current_prompt, _extract_utterance(turn_record.agent_output or ""))) agent_meta_parts = [] if turn_record.duration_seconds is not None: @@ -2524,6 +2443,7 @@ async def _simulation_dialog_loop(self, initial_prompt: str | None, sandbox_dir: # Budget gate: aborts the dialog with a dedicated stop reason and # ensures end-of-dialog criteria still run for partial credit. assert self._monitor is not None + self.result.model_turns = self._monitor.model_turns if self._counts_model_turns else None try: self._monitor.raise_if_over_budget(iteration=turns_completed) except BudgetExceededError: @@ -2536,7 +2456,7 @@ async def _simulation_dialog_loop(self, initial_prompt: str | None, sandbox_dir: # Soft check (logs once, never aborts), then the cap fact, both BEFORE # any stop decision, so a turn that also ends the dialog keeps them. - self._check_expected_tool_calls(iteration=turns_completed) + self._check_expected_targets(iteration=turns_completed) if turn_record.tool_calls_exhausted: self.result.tool_calls_exhausted = True @@ -2549,7 +2469,8 @@ async def _simulation_dialog_loop(self, initial_prompt: str | None, sandbox_dir: if turn_record.tool_calls_exhausted and stop_decision.reason is not DialogStopReason.CRITERIA_PASSED: stop_reason = DialogStopReason.TOOL_CALL_CAP logger.warning( - "Agent reached the tool-call cap during simulation turn %s; ending dialog.", + "Agent reached a run cap (%s) during simulation turn %s; ending dialog.", + self._monitor.stop_reason, turns_completed, ) break @@ -2741,6 +2662,9 @@ async def _run_command_list( proc = await asyncio.create_subprocess_shell( cmd.command, cwd=str(sandbox_dir), + # An authored command that reads stdin must not stall the task. + # Rationale: .claude/notes/agents.md § Why a CLI never inherits stdin + stdin=asyncio.subprocess.DEVNULL, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE, limit=self._POST_RUN_STREAM_LIMIT, diff --git a/src/coder_eval/plugins.py b/src/coder_eval/plugins.py index e8dc7a1d..333c9c0c 100644 --- a/src/coder_eval/plugins.py +++ b/src/coder_eval/plugins.py @@ -15,22 +15,19 @@ Discovery is idempotent and re-entrancy-safe (the ``_loaded`` flag is set before the scan, so a plugin that imports back into coder-eval during its own -registration does not recurse). A plugin whose ``register`` raises is logged and -skipped — one broken plugin never aborts startup. +registration does not recurse). A plugin whose ``register`` raises stops the load +with a ``PluginLoadError`` that names it: a broken plugin is never skipped. """ from __future__ import annotations -import logging +PLUGIN_ENTRY_POINT_GROUP = "coder_eval.plugins" -logger = logging.getLogger(__name__) -PLUGIN_ENTRY_POINT_GROUP = "coder_eval.plugins" +class PluginLoadError(RuntimeError): + """A ``coder_eval.plugins`` entry point failed to import or register.""" -# The built-in agents register through this same entry point, so a failure here is -# a real breakage (empty registry), not a skippable third-party plugin error. -BUILTIN_PLUGIN_NAME = "coder_eval" _loaded = False @@ -41,6 +38,10 @@ def load_plugins(*, force: bool = False) -> None: Idempotent: a second call is a no-op unless ``force=True``. The ``_loaded`` flag is set *before* iterating so a plugin re-entering via :func:`ensure_plugins_loaded` during its own import does not recurse. + + Raises: + PluginLoadError: an entry point failed to load or its ``register`` raised. + The flag is cleared, so a retry re-runs the scan. """ global _loaded if _loaded and not force: @@ -55,15 +56,12 @@ def load_plugins(*, force: bool = False) -> None: try: register = ep.load() register(AgentRegistry) - except Exception: - # Fatal: otherwise it surfaces later as a misleading "No agent - # registered for 'claude-code'" instead of the real cause. - if ep.name == BUILTIN_PLUGIN_NAME: - # Clear the flag so a caller that catches and retries re-runs the - # scan instead of getting a no-op against an empty registry. - _loaded = False - raise - logger.exception("Failed to load coder_eval plugin %r (%s); skipping", ep.name, ep.value) + except Exception as e: + _loaded = False + raise PluginLoadError( + f"coder_eval plugin {ep.name!r} ({ep.value}) failed to load: {e}. " + + "Fix or uninstall the package that provides it." + ) from e def ensure_plugins_loaded() -> None: diff --git a/src/coder_eval/pricing.py b/src/coder_eval/pricing.py index 0e4d2fc1..1665ca4a 100644 --- a/src/coder_eval/pricing.py +++ b/src/coder_eval/pricing.py @@ -13,9 +13,27 @@ generated file. """ -from collections.abc import Iterable, Mapping +import logging +import math +from collections.abc import Iterable, Mapping, Sequence from dataclasses import dataclass from types import MappingProxyType +from typing import Protocol + + +class _TurnUsage(Protocol): + """The ``TokenUsage`` shape ``price_turn`` reads; a protocol, because models import this module.""" + + uncached_input_tokens: int + output_tokens: int + cache_creation_input_tokens: int + cache_read_input_tokens: int + total_cost_usd: float | None + + def is_empty(self) -> bool: ... + + +logger = logging.getLogger(__name__) @dataclass(frozen=True) @@ -244,3 +262,39 @@ def calculate_cost( + cache_creation_tokens * pricing.cache_write_per_mtok + cache_read_tokens * pricing.cache_read_per_mtok ) / 1_000_000 + + +def price_turn(usage: _TurnUsage, models: Sequence[str | None]) -> float | None: + """The cost of one turn: ``usage.total_cost_usd`` is what the harness reported. + + In order: a finite, non-zero reported cost wins; empty usage returns the reported + cost unchanged (``None`` when nothing finite was reported); else the rate card for + the first model in ``models`` that it prices (falsy entries skipped); else a + reported ``0.0``; else ``None``. A non-finite reported cost counts as unreported. + + Rationale: .claude/notes/agents.md § Cost: the stream versus the rate card + """ + reported = usage.total_cost_usd + if reported is not None and not math.isfinite(reported): + reported = None + if reported: + return reported + if usage.is_empty(): + return reported + for model in models: + if not model: + continue + cost = calculate_cost( + model, + usage.uncached_input_tokens, + usage.output_tokens, + usage.cache_creation_input_tokens, + usage.cache_read_input_tokens, + ) + if cost is not None: + if reported == 0.0 and cost: + logger.debug( + "a turn reported $0 that the rate card prices at $%.6f for %r; using the rate card", cost, model + ) + return cost + return reported diff --git a/src/coder_eval/reports/html.py b/src/coder_eval/reports/html.py index ddb2de28..286abc30 100644 --- a/src/coder_eval/reports/html.py +++ b/src/coder_eval/reports/html.py @@ -24,7 +24,7 @@ from ..analysis import calculate_command_statistics from ..durations import format_ms from ..models import FinalStatus, eval_result_total_cost, sum_costs -from ..result_metrics import expected_tool_calls_overage, turn_time_buckets +from ..result_metrics import expected_tool_calls_overage, expected_turns_overage, turn_time_buckets from ..stats import stddev, welch_t_test from .helpers import ( collect_variant_series, @@ -356,6 +356,11 @@ def _render_header(result: EvaluationResult) -> str: expected_tool_calls_badge = ( f'expected_tool_calls exceeded ({actual}/{expected})' ) + expected_turns_badge = "" + turns_overage = expected_turns_overage(result) + if turns_overage is not None: + actual, expected = turns_overage + expected_turns_badge = f'expected_turns exceeded ({actual}/{expected})' early_stop_badge = "" if result.early_stop is not None: title = early_stop_gate_note(result.early_stop.reason.value) @@ -376,6 +381,7 @@ def _render_header(result: EvaluationResult) -> str: {_esc(duration)} {cost_badge} {expected_tool_calls_badge} + {expected_turns_badge} {early_stop_badge} Toggle theme diff --git a/src/coder_eval/reports/markdown.py b/src/coder_eval/reports/markdown.py index 11860bcd..0d1e8f35 100644 --- a/src/coder_eval/reports/markdown.py +++ b/src/coder_eval/reports/markdown.py @@ -515,8 +515,8 @@ def _task_details_lines(summary: RunSummary) -> list[str]: @staticmethod def _runtime_notes_lines(summary: RunSummary) -> list[str]: - """The ``## Run-time Notes`` blockquotes (tool-call cap + expected_tool_calls - overage). Returns ``[]`` when there are no notes so the caller adds nothing — + """The ``## Run-time Notes`` blockquotes (tool-call cap, expected_tool_calls and + expected_turns overage). Returns ``[]`` when there are no notes so the caller adds nothing — preserving the "only render the section when notes exist" behavior. """ # Surface per-task signals as plain blockquote one-liners. Same surface for @@ -526,17 +526,18 @@ def _runtime_notes_lines(summary: RunSummary) -> list[str]: task_id = t.get("task_id", "?") if t.get("tool_calls_exhausted"): notes.append(f"> **WARNING:** [{task_id}] tool-call cap reached") - overage_field = t.get("expected_tool_calls_overage") - if ( - isinstance(overage_field, (list, tuple)) - and len(overage_field) == 2 - and all(isinstance(x, int) for x in overage_field) + for key, unit in ( + ("expected_tool_calls", "cumulative visible tool calls"), + ("expected_turns", "cumulative model turns"), ): - actual, expected = overage_field - notes.append( - f"> **WARNING:** [{task_id}] expected_tool_calls exceeded: {actual}/{expected}" - + " (cumulative visible tool calls)" - ) + overage_field = t.get(f"{key}_overage") + if ( + isinstance(overage_field, (list, tuple)) + and len(overage_field) == 2 + and all(isinstance(x, int) for x in overage_field) + ): + actual, expected = overage_field + notes.append(f"> **WARNING:** [{task_id}] {key} exceeded: {actual}/{expected} ({unit})") if t.get("stopped_early"): reason = t.get("early_stop_reason") or "unknown" # No "N turn(s) avoided" claim: on harnesses where one diff --git a/src/coder_eval/resources/tags.yaml b/src/coder_eval/resources/tags.yaml index 3306f056..2ba456eb 100644 --- a/src/coder_eval/resources/tags.yaml +++ b/src/coder_eval/resources/tags.yaml @@ -50,6 +50,6 @@ tags: examples: - "300s turn_timeout exceeded at 80 turns; analysis recommends 600s" - name: max-turns-too-low - definition: Task reached the tool-call cap before completing; max_tool_calls is below what the task realistically needs. + definition: Task reached a structural cap (max_tool_calls or max_turns) before completing; the cap is below what the task realistically needs. examples: - "TOOL_CALLS_EXHAUSTED: cap of 50 tool calls reached" diff --git a/src/coder_eval/result_metrics.py b/src/coder_eval/result_metrics.py index fbc29fba..17db91d2 100644 --- a/src/coder_eval/result_metrics.py +++ b/src/coder_eval/result_metrics.py @@ -153,23 +153,30 @@ def visible_turn_count(result: EvaluationResult) -> int: return commands + (1 if has_final_reply(result) else 0) -def expected_tool_calls_overage(result: EvaluationResult) -> tuple[int, int] | None: - """Return ``(visible_turns, expected)`` when the visible-events turn - count strictly exceeds ``run_limits.expected_tool_calls``; else ``None``. +def recorded_run_limit(result: EvaluationResult, name: str) -> int | None: + """The positive int ``run_limits.`` from the recorded resolved config, else None.""" + task_cfg = result.task_config + run_limits = (task_cfg.resolved or {}).get("run_limits") if task_cfg is not None else None + value = run_limits.get(name) if isinstance(run_limits, dict) else None + return value if isinstance(value, int) and value >= 1 else None + - Safe against missing ``task_config``, missing ``run_limits``, and - non-int ``expected_tool_calls`` values. +def expected_tool_calls_overage(result: EvaluationResult) -> tuple[int, int] | None: + """``(visible_turns, expected)`` when the visible-events count strictly exceeds + ``run_limits.expected_tool_calls``; else ``None``. """ - task_cfg = result.task_config - if task_cfg is None: - return None - run_limits = (task_cfg.resolved or {}).get("run_limits") or {} - if not isinstance(run_limits, dict): - return None - expected = run_limits.get("expected_tool_calls") - if not isinstance(expected, int) or expected < 1: + expected = recorded_run_limit(result, "expected_tool_calls") + if expected is None: return None actual = visible_turn_count(result) if actual > expected: return actual, expected return None + + +def expected_turns_overage(result: EvaluationResult) -> tuple[int, int] | None: + """``(model_turns, expected)`` when recorded model turns strictly exceed ``run_limits.expected_turns``.""" + expected = recorded_run_limit(result, "expected_turns") + if expected is None or result.model_turns is None or result.model_turns <= expected: + return None + return result.model_turns, expected diff --git a/src/coder_eval/run_record.py b/src/coder_eval/run_record.py index 5234bb6e..43f1bbc3 100644 --- a/src/coder_eval/run_record.py +++ b/src/coder_eval/run_record.py @@ -17,7 +17,13 @@ from coder_eval.errors import truncate_crash_message from coder_eval.models import EvaluationResult, FinalStatus, judge_cost_usd, simulator_cost_usd, sum_costs -from coder_eval.result_metrics import expected_tool_calls_overage, turn_time_buckets, visible_turn_count +from coder_eval.result_metrics import ( + expected_tool_calls_overage, + expected_turns_overage, + recorded_run_limit, + turn_time_buckets, + visible_turn_count, +) from coder_eval.result_metrics import has_final_reply as _has_final_reply @@ -85,6 +91,7 @@ def eval_result_to_task_dict( break overage = expected_tool_calls_overage(result) + turns_overage = expected_turns_overage(result) total_turns = sum((t.num_turns or 0) for t in result.iterations) @@ -97,14 +104,6 @@ def eval_result_to_task_dict( simulator_cost = simulator_cost_usd(result) row_total_cost = sum_costs(agent_cost, judge_cost, simulator_cost) - expected_tool_calls_value: int | None = None - if result.task_config is not None: - rl = (result.task_config.resolved or {}).get("run_limits") or {} - if isinstance(rl, dict): - raw = rl.get("expected_tool_calls") - if isinstance(raw, int) and raw >= 1: - expected_tool_calls_value = raw - _buckets = turn_time_buckets(result) d: dict[str, Any] = { @@ -181,7 +180,11 @@ def eval_result_to_task_dict( # "Visible turns" (tool calls + final reply) -- what the "within expected # tool calls" metric compares against. Distinct from total_turns (SDK num_turns). "visible_turns": visible_turn_count(result), - "expected_tool_calls": expected_tool_calls_value, + "expected_tool_calls": recorded_run_limit(result, "expected_tool_calls"), + "model_turns": result.model_turns, + # expected_turns is reported only beside the model-turn count it targets. + "expected_turns": recorded_run_limit(result, "expected_turns") if result.model_turns is not None else None, + "expected_turns_overage": list(turns_overage) if turns_overage is not None else None, "has_final_reply": has_reply, # None/False on the default path, so downstream analysis never confuses a # truncated run with a full one. diff --git a/src/coder_eval/sandbox.py b/src/coder_eval/sandbox.py index ce62c526..5fff2a5f 100644 --- a/src/coder_eval/sandbox.py +++ b/src/coder_eval/sandbox.py @@ -560,21 +560,22 @@ def _apply_template_dir_source(self, source: TemplateDirSource) -> None: logger.debug(f"Overwrote {len(overwrites)} files from {source.path}") def _prepare_mock_path_dirs(self) -> None: - """Apply +x to plain files in each ``mock_path_dirs`` entry. - - Resolves each configured directory against the sandbox root and, for every - plain file directly under it, ORs in the user/group/other execute bits. - Required on NTFS and after copies that drop the +x bit; a no-op when the - bit is already set. Missing entries and non-files (e.g. fixture - subdirectories) are skipped silently. PATH wiring happens in the agent -- - this method only owns the filesystem side. + """Make each mock dir executable and put the dirs in front of the criterion PATH. + + For every plain file directly under a resolved mock dir, ORs in the + user/group/other execute bits (required on NTFS and after copies that drop + the +x bit). Then sets :attr:`command_base_path` from the same list the + orchestrator passes to the agent as ``env_path_prepend``, so a + ``run_command`` criterion resolves the binaries the agent saw. """ assert self.sandbox_dir is not None, "Sandbox directory not initialized" - for dir_path in self.resolved_mock_path_dirs: + mock_dirs = self.resolved_mock_path_dirs + for dir_path in mock_dirs: for entry in dir_path.iterdir(): if entry.is_file(): entry.chmod(entry.stat().st_mode | 0o111) + self._command_base_path = os.pathsep.join(str(d) for d in mock_dirs) or None @property def resolved_mock_path_dirs(self) -> list[Path]: @@ -903,44 +904,17 @@ def _capture_node_tool_versions(self) -> None: exc, ) - def set_command_base_path(self, path: str | None) -> None: - """Set the parent PATH used by sandbox command checks. - - The orchestrator uses this to align success-criteria commands with the - PATH passed to the agent SDK. Sandbox-local venv and node bin entries - are still prepended by ``run_command``. - - Also re-derives the canonical ``PLUGIN_TOOLS_DIR`` (MST-9795): the - resolved ``uip`` binary depends on PATH, and the path-aligned criterion - is the canonical lookup. Failures are swallowed — the env var simply - stays unset and the CLI falls back to its walk-based discovery. - - Passing ``None`` clears the agent-aligned PATH prefix and re-derives - ``PLUGIN_TOOLS_DIR`` from ``os.environ['PATH']`` alone. The new pin - may differ from the previous one if the parent PATH resolves ``uip`` - to a different install — by design, since dropping the agent - alignment means the criterion subprocess should now match the parent - environment. - """ - self._command_base_path = path or None - self._refresh_plugin_tools_dir() - @property def command_base_path(self) -> str | None: - """Read-only view of the configured base PATH (or ``None`` when unset). - - Tests can observe orchestrator-set overrides without touching the - underlying private slot. Mutate via :meth:`set_command_base_path`. - """ + """The resolved mock dirs joined as a PATH prefix, or ``None`` when there are none.""" return self._command_base_path @property def plugin_tools_dir(self) -> str | None: """Canonical ``node_modules/@uipath`` derived from the resolved ``uip``. - Populated by :meth:`_refresh_plugin_tools_dir` after the agent's PATH - is captured. When non-None, ``_build_run_command_env`` exports it as - ``PLUGIN_TOOLS_DIR`` so the UiPath CLI pins plugin discovery instead + Populated by :meth:`_refresh_plugin_tools_dir` at setup. When non-None, + ``_build_run_command_env`` exports it as ``PLUGIN_TOOLS_DIR`` so the UiPath CLI pins plugin discovery instead of walking up from CWD — eliminating MST-9795's host-pollution asymmetry between authoring-time and criterion-time validation. @@ -952,7 +926,7 @@ def plugin_tools_dir(self) -> str | None: @property def uip_search_path(self) -> str: - """The PATH used to resolve ``uip`` — agent-aligned prefix + process PATH. + """The PATH used to resolve ``uip`` — mock-dir prefix + process PATH. The same PATH ``run_command`` subprocesses and the agent SDK env see, so a binary resolved against it is the one task commands actually @@ -979,9 +953,7 @@ def _refresh_plugin_tools_dir(self) -> None: ``uip_search_path`` (``command_base_path + os.environ['PATH']`` — the same PATH ``run_command`` and the agent SDK will see), then stores the result as a string on ``self._plugin_tools_dir`` (or ``None`` if no - usable ``uip`` is on PATH). Idempotent across calls; safe to call from - both ``setup`` (initial value when no command_base_path yet) and - ``set_command_base_path`` (re-derive after PATH alignment). + usable ``uip`` is on PATH). Idempotent across calls. """ from .utils import resolve_uipath_plugin_dir @@ -1081,7 +1053,7 @@ def _build_run_command_env(self) -> dict[str, str]: """Build the environment for ``run_command``. Each layer is independent -- none breaks if another is absent: the parent - env, the agent's captured SDK PATH (PREPENDED, so system binaries stay + env, the mock dirs the agent also got (PREPENDED, so system binaries stay reachable), the sandbox venv, ``/node_modules/.bin``, ``NODE_PATH=""``, a sandbox-scoped ``NPM_CONFIG_PREFIX``, ``TASK_DIR``, ``REFERENCE_DIR``, and ``PLUGIN_TOOLS_DIR`` (which defers to an inherited diff --git a/src/coder_eval/simulation/user_simulator.py b/src/coder_eval/simulation/user_simulator.py index c119c759..9a42b5cc 100644 --- a/src/coder_eval/simulation/user_simulator.py +++ b/src/coder_eval/simulation/user_simulator.py @@ -186,6 +186,7 @@ def __init__( self._route: ApiRoute | None = route self._agent_override: Agent[Any] | None = agent_override self._agent: Agent[Any] | None = None + self._turns = 0 self._scratch_dir: Path | None = None # PINNED from config, not inherited from the route: leaving it None let @@ -332,7 +333,8 @@ async def next_user_message(self, dialog_pairs: list[tuple[str, str]]) -> Simula assert self._agent is not None, "UserSimulator.start() must be called before next_user_message()" prompt = dialog_pairs[-1][1] if dialog_pairs else _OPENER_NUDGE - turn = await self._agent.communicate(prompt) + self._turns += 1 + turn = (await self._agent.communicate(prompt, iteration=self._turns)).record_or_raise() raw = turn.agent_output or "" usage = turn.token_usage input_tokens = usage.uncached_input_tokens if usage is not None else None diff --git a/src/coder_eval/spi.py b/src/coder_eval/spi.py index 6082ac39..455c9536 100644 --- a/src/coder_eval/spi.py +++ b/src/coder_eval/spi.py @@ -1,15 +1,16 @@ """The stable import surface for plugin agents. -A plugin imports only from this module and checks ``SPI_VERSION`` in its -``register(registry)`` hook. Any signature change to a name exported here bumps -``SPI_VERSION``; adding a name does not. +A plugin imports only from this module and passes ``SPI_VERSION`` to every +``AgentRegistry.register`` call, which rejects a version other than this core's. +Any signature change to a name exported here bumps ``SPI_VERSION``; adding a +name does not. """ -from typing import Final - from coder_eval.agent import Agent -from coder_eval.agents.registry import AgentRegistry -from coder_eval.errors import AgentConfigError, AgentCrashError, TurnTimeoutError +from coder_eval.agents._transport import JsonlDecoder, SubprocessJsonlAgent +from coder_eval.agents.registry import SPI_VERSION, AgentRegistry +from coder_eval.agents.watchdog import WatchdogFired, run_with_watchdog +from coder_eval.errors import AgentConfigError, AgentCrashError, TurnTimeoutError, format_timeout_reason from coder_eval.models import ( CANONICAL_TOOL_NAMES, READ_ONLY_DENIED_TOOLS, @@ -17,57 +18,49 @@ ApiRoute, BaseAgentConfig, CommandTelemetry, + ContentBlock, Enforcement, HarnessContract, LocalPluginConfig, PermissionMode, ResultSummary, SystemPromptMode, + TimingBasis, TokenUsage, ToolNameMap, TranscriptMessage, TurnRecord, UsageGranularity, ) -from coder_eval.pricing import ModelPricing, register_pricing -from coder_eval.streaming.callbacks import CompositeStreamCallback, StreamCallback -from coder_eval.streaming.collector import EventCollector +from coder_eval.pricing import ModelPricing, price_turn, register_pricing +from coder_eval.streaming.callbacks import StreamCallback +from coder_eval.streaming.emitter import Generation, TurnEmitter, TurnOutcome from coder_eval.streaming.events import ( - AgentEndEvent, AgentEndStatus, - AgentStartEvent, StopReason, - TextChunkEvent, - ToolEndEvent, ToolEndStatus, - ToolStartEvent, - TurnEndEvent, TurnEndStatus, - TurnStartEvent, end_status_for, ) -from coder_eval.timing import TurnClock, close_window - +from coder_eval.timing import TurnClock, Window, close_window -SPI_VERSION: Final[int] = 2 __all__ = [ # noqa: RUF022 - plain sort, pinned by tests/test_spi.py "Agent", "AgentConfigError", "AgentCrashError", - "AgentEndEvent", "AgentEndStatus", "AgentRegistry", - "AgentStartEvent", "AgentState", "ApiRoute", "BaseAgentConfig", "CANONICAL_TOOL_NAMES", "CommandTelemetry", - "CompositeStreamCallback", + "ContentBlock", "Enforcement", - "EventCollector", + "Generation", "HarnessContract", + "JsonlDecoder", "LocalPluginConfig", "ModelPricing", "PermissionMode", @@ -76,22 +69,26 @@ "SPI_VERSION", "StopReason", "StreamCallback", + "SubprocessJsonlAgent", "SystemPromptMode", - "TextChunkEvent", + "TimingBasis", "TokenUsage", - "ToolEndEvent", "ToolEndStatus", "ToolNameMap", - "ToolStartEvent", "TranscriptMessage", "TurnClock", - "TurnEndEvent", + "TurnEmitter", "TurnEndStatus", + "TurnOutcome", "TurnRecord", - "TurnStartEvent", "TurnTimeoutError", "UsageGranularity", + "WatchdogFired", + "Window", "close_window", "end_status_for", + "format_timeout_reason", + "price_turn", "register_pricing", + "run_with_watchdog", ] diff --git a/src/coder_eval/streaming/collector.py b/src/coder_eval/streaming/collector.py index 27971d63..d4cea82b 100644 --- a/src/coder_eval/streaming/collector.py +++ b/src/coder_eval/streaming/collector.py @@ -4,8 +4,8 @@ therefore ``task.json``) is assembled — so a new agent emits the standard events and gets capture for free. -An agent attaches its own collector alongside the caller's ``stream_callback`` -and returns ``build_turn_record()`` from ``communicate()``. +``TurnEmitter`` feeds one per turn and returns its record in the ``TurnOutcome``; +the orchestrator attaches a second one per attempt to recover a killed turn. ``commands`` are derived from the ``ToolEndEvent`` stream (crash-orphaned calls included, force-closed as ``unresolved``), ordered by ``sequence_number``. The @@ -42,8 +42,9 @@ class EventCollector: Tolerant by construction: ``build_turn_record()`` can be called at any point (including mid-stream after a crash) and returns the best record derivable from the events seen so far. Sub-agent activity is captured as - ``parent_tool_use_id``-tagged messages in the transcript; per-sub-agent - attribution is derived by grouping those messages, not from a separate field. + ``parent_tool_use_id``-tagged messages in the transcript. A nested event + (``parent_thread_id`` set) contributes only its ``ToolEndEvent`` to ``commands``; + it sets no model and counts no turn. """ def __init__(self) -> None: @@ -53,14 +54,19 @@ def __init__(self) -> None: self._turn_starts: int = 0 # Stamped by AgentStartEvent; the head is measured from it. self._agent_start_at: datetime | None = None - # tool_id -> finalized telemetry (last ToolEnd wins, mirroring last-result-wins). + # tool_id -> finalized telemetry (the last ToolEnd for an id wins). self._commands: dict[str, CommandTelemetry] = {} self._agent_end: AgentEndEvent | None = None + @property + def ended(self) -> bool: + """True once the current attempt's ``AgentEndEvent`` has been seen.""" + return self._agent_end is not None + def on_event(self, event: StreamEvent) -> None: - # Only the main agent's own events shape its TurnRecord. Forward-looking: - # no agent emits nested sub-agent events yet, so this never fires today. if event.parent_thread_id is not None: + if isinstance(event, ToolEndEvent): + self._commands[event.tool.tool_id] = event.tool return if isinstance(event, AgentStartEvent): @@ -82,8 +88,20 @@ def on_event(self, event: StreamEvent) -> None: elif isinstance(event, AgentEndEvent): self._agent_end = event - def _ordered_commands(self) -> list[CommandTelemetry]: - return sorted(self._commands.values(), key=lambda c: c.sequence_number) + def _ordered_commands(self, messages: list[TranscriptMessage]) -> list[CommandTelemetry]: + """Commands by ``sequence_number``, each a copy carrying its derived ``assistant_turn_index``. + + The index is the position, among the ``AssistantMessage`` entries, of the first + message whose ``tool_use_ids`` names the command; ``None`` when none does. + """ + owner: dict[str, int] = {} + for index, message in enumerate(m for m in messages if isinstance(m, AssistantMessage)): + for tool_id in message.tool_use_ids: + owner.setdefault(tool_id, index) + return [ + command.model_copy(update={"assistant_turn_index": owner.get(command.tool_id)}) + for command in sorted(self._commands.values(), key=lambda c: c.sequence_number) + ] def _overhead_ms( self, messages: list[TranscriptMessage], tool_spans: list[tuple[datetime, datetime]] @@ -94,7 +112,7 @@ def _overhead_ms( ``generation_duration_ms`` is ``None`` — that field is the codebase's marker for "no window was measurable here", and its producers stamp a placeholder ``started_at == completed_at`` that would otherwise read as a - measurement (the same exemption CE059 makes). + measurement (``TurnEmitter.add_unmeasured_generation`` writes exactly that shape). ``min``/``max``, not the first and last entries: the list is not ordered by time. MAIN THREAD ONLY, so all four buckets measure one thread. @@ -175,7 +193,6 @@ def _reconciled_messages(messages: list[TranscriptMessage], usage: TokenUsage) - def build_turn_record(self) -> TurnRecord: """Assemble the ``TurnRecord`` from the events observed so far.""" end = self._agent_end - commands = self._ordered_commands() if end is None: # No terminal event yet (mid-stream snapshot): minimal record. @@ -183,7 +200,7 @@ def build_turn_record(self) -> TurnRecord: iteration=self._iteration, user_input=self._user_input, agent_output="", - commands=commands, + commands=self._ordered_commands([]), token_usage=None, model_used=self._model, assistant_turn_count=self._turn_starts, @@ -216,7 +233,7 @@ def build_turn_record(self) -> TurnRecord: iteration=end.iteration or self._iteration, user_input=end.user_input or self._user_input, agent_output=end.agent_output, - commands=commands, + commands=self._ordered_commands(messages), duration_seconds=end.duration_seconds, token_usage=token_usage, model_used=end.model_used or self._model, diff --git a/src/coder_eval/streaming/emitter.py b/src/coder_eval/streaming/emitter.py new file mode 100644 index 00000000..8b0ea7c1 --- /dev/null +++ b/src/coder_eval/streaming/emitter.py @@ -0,0 +1,655 @@ +"""TurnEmitter: the one writer of the event protocol for one ``communicate()`` turn. + +An adapter opens one per turn (``Agent._open_emitter``), calls ``begin``, reports what +its harness did through the write methods, and returns ``finalize(...)`` or +``fail(...)``. The emitter owns every per-turn value: open tools, sequence numbers, +the transcript, reported usage, text output, the open inner turn and the end. + +Rationale: .claude/notes/agents.md § Shared turn lifecycle +""" + +from __future__ import annotations + +import logging +import time +from collections.abc import Sequence +from dataclasses import dataclass +from datetime import datetime +from typing import Any, Literal, Protocol + +from coder_eval.errors import AgentCrashError, TurnTimeoutError +from coder_eval.errors.agent import truncate_crash_message +from coder_eval.models import ( + AssistantMessage, + CommandTelemetry, + ContentBlock, + ResultSummary, + TimingBasis, + TokenUsage, + TurnRecord, +) +from coder_eval.streaming.callbacks import StreamCallback, safe_emit +from coder_eval.streaming.collector import EventCollector +from coder_eval.streaming.events import ( + AgentEndEvent, + AgentEndStatus, + AgentStartEvent, + StreamEvent, + TextChunkEvent, + ToolEndEvent, + ToolEndStatus, + ToolStartEvent, + TurnEndEvent, + TurnEndStatus, + TurnStartEvent, +) +from coder_eval.timing import Window + + +logger = logging.getLogger(__name__) + +EMPTY_TURN_REASON = "the harness ended the turn cleanly but reported no model turn, tool call, text or usage" +_UNSET: Any = object() +_FAILED = (AgentEndStatus.CRASHED, AgentEndStatus.TIMEOUT) +_BUCKETS = ("uncached_input_tokens", "output_tokens", "cache_creation_input_tokens", "cache_read_input_tokens") +_RESULT_STATUS: dict[ToolEndStatus, Literal["success", "error", "unknown"]] = { + ToolEndStatus.OK: "success", + ToolEndStatus.ERROR: "error", + ToolEndStatus.PERMISSION_DENIED: "error", + ToolEndStatus.UNRESOLVED: "unknown", +} + + +class Clock(Protocol): + """The source of every stamp the emitter writes.""" + + def now(self) -> datetime: ... + + +@dataclass(frozen=True) +class Generation: + """One sub-message of one model generation; ``tokens`` is this part's own delta.""" + + blocks: list[ContentBlock] + tokens: TokenUsage + reasoning_tokens: int = 0 + stop_reason: str | None = None + + +@dataclass(frozen=True) +class TurnOutcome: + """What a ``communicate()`` turn produced. + + ``error`` is the untruncated failure message, set only for ``CRASHED`` and ``TIMEOUT``. + """ + + record: TurnRecord + status: AgentEndStatus + error: str | None + + def record_or_raise( + self, *, timeout_seconds: float | None = None, task_id: str | None = None, iteration: int | None = None + ) -> TurnRecord: + """The record, or the exception the orchestrator's retry policy classifies. + + Raises: + AgentCrashError: the status is ``CRASHED``. + TurnTimeoutError: the status is ``TIMEOUT``. + """ + if self.status is AgentEndStatus.CRASHED: + raise AgentCrashError(self.error or "agent turn crashed", len(self.record.commands)) + if self.status is AgentEndStatus.TIMEOUT: + raise TurnTimeoutError(timeout_seconds or 0.0, task_id=task_id, iteration=iteration) + return self.record + + +@dataclass +class _OpenTool: + telemetry: CommandTelemetry + turn_id: str + parent_tool_id: str | None + + +class TurnEmitter: + """The sole writer of the event protocol for one turn. + + Every event goes to an internal ``EventCollector`` first, then to each sink through + ``safe_emit``, stamped ``clock.now()``. After ``finalize`` or ``fail`` every write + is dropped. ``RuntimeError`` and ``TypeError`` from a write mean a harness bug. + + Rationale: .claude/notes/agents.md § Shared turn lifecycle + """ + + def __init__( + self, + *, + task_id: str, + iteration: int, + prompt: str, + model: str | None, + basis: TimingBasis, + clock: Clock, + sinks: Sequence[StreamCallback], + ) -> None: + self._task_id = task_id + self._iteration = iteration + self._prompt = prompt + self._model = model + self._basis = basis + self._clock = clock + self._sinks = list(sinks) + self._collector = EventCollector() + self._began = False + self._began_at: datetime | None = None + self._began_monotonic = 0.0 + self._outcome: TurnOutcome | None = None + self._ending = False + self._dropped_logged = False + self._events = 0 + self._open_tools: dict[str, _OpenTool] = {} + self._closed_tool_ids: set[str] = set() + self._sequence = 0 + self._messages: list[AssistantMessage] = [] + self._text: list[str] = [] + self._turn_id: str | None = None + self._turn_parent: str | None = None + self._main_turns = 0 + self._reported = TokenUsage() + + @property + def iteration(self) -> int: + return self._iteration + + @property + def model(self) -> str | None: + return self._model + + @property + def inner_turn_open(self) -> bool: + return self._turn_id is not None + + def now(self) -> datetime: + return self._clock.now() + + def begin(self) -> None: + """Emit the ``AgentStartEvent``; once per emitter, before any other call.""" + if self._began: + raise RuntimeError("TurnEmitter.begin() called twice") + self._began = True + self._began_at = self.now() + self._began_monotonic = time.monotonic() + self._emit( + AgentStartEvent(task_id=self._task_id, prompt=self._prompt, iteration=self._iteration, model=self._model), + stamp=self._began_at, + ) + + def begin_inner_turn(self, turn_id: str, model: str | None = None, *, parent_tool_id: str | None = None) -> None: + """Open one inner turn; raises ``RuntimeError`` while another is open.""" + self._require_begun("begin_inner_turn") + if self._ended(): + return + if self._turn_id is not None: + raise RuntimeError(f"inner turn {turn_id!r} begun while {self._turn_id!r} is still open") + self._turn_id = turn_id + self._turn_parent = parent_tool_id + if parent_tool_id is None: + self._main_turns += 1 + model = model or self._model + self._emit(TurnStartEvent(task_id=self._task_id, turn_id=turn_id, model=model), parent_tool_id) + + def end_inner_turn( + self, status: TurnEndStatus = TurnEndStatus.COMPLETED, *, tokens: TokenUsage | None = None + ) -> None: + """Close the open inner turn, adding ``tokens`` (a delta) to the reported usage.""" + self._require_begun("end_inner_turn") + if self._ended(): + return + if self._turn_id is None: + raise RuntimeError("end_inner_turn() with no inner turn open") + if tokens is not None: + self._reported += tokens + turn_id, parent = self._turn_id, self._turn_parent + self._turn_id = self._turn_parent = None + self._emit(TurnEndEvent(task_id=self._task_id, turn_id=turn_id, status=status, tokens=tokens), parent) + + def text(self, chunk: str, *, parent_tool_id: str | None = None) -> None: + """Stream visible assistant text; main-thread chunks form the default ``agent_output``.""" + self._require_begun("text") + if self._ended(): + return + if parent_tool_id is None: + self._text.append(chunk) + self._emit(TextChunkEvent(task_id=self._task_id, turn_id=self._turn_id or "", text=chunk), parent_tool_id) + + def open_tool( + self, + tool_id: str, + name: str, + params: dict[str, Any], + *, + parent_tool_id: str | None = None, + started_at: datetime | None = _UNSET, + generation_completed: bool = False, + ) -> None: + """Record a tool call's start; ``timestamp`` is the execution start, else the clock. + + An id already closed in this turn is ignored: a call is opened and closed once. + + Raises: + TypeError: on a main-thread tool, ``started_at`` passed under ``TURN_CLOCK`` or + omitted under ``CLI_EPOCH_MS``. A nested tool takes either. + """ + self._require_begun("open_tool") + if self._ended() or self._already_closed(tool_id): + return + now = self.now() + execution_started_at = self._stamp("started_at", started_at, now, parent_tool_id) + telemetry = CommandTelemetry( + tool_name=name, + tool_id=tool_id, + timestamp=execution_started_at or now, + parameters=params, + sequence_number=self._next_sequence(), + execution_started_at=execution_started_at, + generation_completed_at=now if generation_completed else None, + ) + turn_id = self._turn_id or "" + self._open_tools[tool_id] = _OpenTool(telemetry, turn_id, parent_tool_id) + self._emit(ToolStartEvent(task_id=self._task_id, turn_id=turn_id, tool=telemetry), parent_tool_id) + + def close_tool( + self, + tool_id: str, + *, + status: ToolEndStatus, + summary: str | None = None, + error: str | None = None, + result_data: dict[str, Any] | list[Any] | None = None, + parameters: dict[str, Any] | None = None, + completed_at: datetime | None = _UNSET, + started_at: datetime | None = None, + reported_duration_ms: float | None = None, + ) -> None: + """Record a tool call's end; a never-opened id synthesizes a ``tool_name="unknown"`` call. + + An id already closed in this turn is ignored, so a repeated close cannot replace the record. + + Only a resolved call is timed: ``UNRESOLVED`` keeps ``execution_started_at`` + and sets no completion stamp and no duration. ``started_at`` is a CLI start + stamp that arrived only with the result (``CLI_EPOCH_MS``); it fills a call + opened without one and never replaces an existing start. ``reported_duration_ms`` + is a duration the CLI reported for a call with no stamps; a positive value is + used only when the stamps measure none. + + Raises: + TypeError: the same basis rule as ``open_tool``, for ``completed_at``; or + ``started_at`` / ``reported_duration_ms`` given under ``TURN_CLOCK``. + """ + self._require_begun("close_tool") + if self._ended(): + return + if self._basis is TimingBasis.TURN_CLOCK and (started_at is not None or reported_duration_ms is not None): + raise TypeError( + "started_at= and reported_duration_ms= are not accepted under TimingBasis.TURN_CLOCK: " + + "the emitter stamps the clock" + ) + if self._already_closed(tool_id): + return + opened = self._open_tools.get(tool_id) + stamp = self._stamp("completed_at", completed_at, self.now(), opened.parent_tool_id if opened else None) + if opened is not None and started_at is not None and opened.telemetry.execution_started_at is None: + opened.telemetry.execution_started_at = started_at + opened.telemetry.timestamp = started_at + self._close(tool_id, status, summary, error, result_data, parameters, stamp, reported_duration_ms) + + def add_generation( + self, + *, + message_id: str | None, + window: Window, + parts: Sequence[Generation], + model: str | None = None, + parent_tool_id: str | None = None, + ) -> list[AssistantMessage]: + """Add one measured generation, one ``AssistantMessage`` per part, sharing ``window``. + + ``window.duration_ms`` is apportioned by each part's output tokens (evenly when + none has output), each share but the last rounded to 1e-6 ms. The returned + messages and the passed blocks are the live objects in the transcript; after + the turn ended they are detached and change nothing. + + Raises: + ValueError: ``parts`` is empty. + """ + self._require_begun("add_generation") + if not parts: + raise ValueError("add_generation() needs at least one part") + total_ms = window.duration_ms + output = sum(part.tokens.output_tokens for part in parts) + messages: list[AssistantMessage] = [] + assigned = 0.0 + for index, part in enumerate(parts): + if index == len(parts) - 1: + share = total_ms - assigned + else: + share = round(total_ms * (part.tokens.output_tokens / output if output > 0 else 1 / len(parts)), 6) + assigned += share + messages.append( + self._message(part, window.started_at, window.completed_at, share, message_id, model, parent_tool_id) + ) + if not self._ended(): + self._messages.extend(messages) + return messages + + def add_unmeasured_generation( + self, + *, + message_id: str | None, + part: Generation, + model: str | None = None, + parent_tool_id: str | None = None, + ) -> AssistantMessage: + """Add a generation with no measurable window: equal bounds at ``now()``, no duration.""" + self._require_begun("add_unmeasured_generation") + now = self.now() + message = self._message(part, now, now, None, message_id, model, parent_tool_id) + if not self._ended(): + self._messages.append(message) + return message + + def finalize( + self, + status: AgentEndStatus, + *, + usage: TokenUsage | None = None, + stop_reason: str | None = None, + agent_output: str | None = None, + model_used: str | None = None, + assistant_turn_count: int | None = None, + num_turns: int | None = _UNSET, + result_summary: ResultSummary | None = _UNSET, + ) -> TurnOutcome: + """End a clean turn; a second call returns the first outcome and emits nothing. + + A ``COMPLETED`` turn that wrote nothing after ``begin`` (no inner turn, tool, text, + generation, usage or ``agent_output``) ends ``CRASHED`` instead: an empty turn is + never graded as agent behavior. A requested stop is exempt. + + ``usage`` defaults to the sum of ``end_inner_turn`` tokens. ``result_summary`` + defaults to the final reply: the text that follows the last tool call in the + last main-thread message. ``num_turns`` defaults to the main-thread inner turns; + an explicit ``None`` records that the harness reported none. + + Raises: + ValueError: ``status`` is ``CRASHED`` or ``TIMEOUT`` (use ``fail``). + """ + self._require_begun("finalize") + if status in _FAILED: + raise ValueError(f"finalize({status.value}): a failed turn ends with fail()") + if self._outcome is not None: + return self._outcome + if self._ending: + raise RuntimeError("the turn already ended, but its record could not be built") + if status is AgentEndStatus.COMPLETED and self._wrote_nothing(usage, agent_output): + return self._end( + AgentEndStatus.CRASHED, + reason=EMPTY_TURN_REASON, + usage=usage, + agent_output=agent_output, + model_used=model_used, + assistant_turn_count=assistant_turn_count, + num_turns=num_turns, + result_summary=None, + ) + if result_summary is _UNSET: + result_summary = ResultSummary( + is_error=False, subtype=status.value, stop_reason=stop_reason, result=self._final_reply() + ) + return self._end( + status, + reason=None, + usage=usage, + agent_output=agent_output, + model_used=model_used, + assistant_turn_count=assistant_turn_count, + num_turns=num_turns, + result_summary=result_summary, + ) + + def fail( + self, + status: Literal[AgentEndStatus.CRASHED, AgentEndStatus.TIMEOUT], + reason: str, + *, + usage: TokenUsage | None = None, + agent_output: str | None = None, + model_used: str | None = None, + assistant_turn_count: int | None = None, + num_turns: int | None = _UNSET, + ) -> TurnOutcome: + """End a failed turn with the full ``reason``; a second call returns the first outcome. + + The payload keywords default as in ``finalize``. + + Raises: + ValueError: ``status`` is not ``CRASHED`` or ``TIMEOUT``. + """ + self._require_begun("fail") + if status not in _FAILED: + raise ValueError(f"fail({status.value}): a clean turn ends with finalize()") + if self._outcome is not None: + return self._outcome + if self._ending: + raise RuntimeError("the turn already ended, but its record could not be built") + return self._end( + status, + reason=reason, + usage=usage, + agent_output=agent_output, + model_used=model_used, + assistant_turn_count=assistant_turn_count, + num_turns=num_turns, + result_summary=None, + ) + + def _require_begun(self, method: str) -> None: + if not self._began: + raise RuntimeError(f"TurnEmitter.{method}() before begin(): the turn has no AgentStartEvent") + + def _wrote_nothing(self, usage: TokenUsage | None, agent_output: str | None) -> bool: + return ( + self._events <= 1 + and not self._messages + and self._reported.is_empty() + and (usage is None or usage.is_empty()) + and not agent_output + ) + + def _ended(self) -> bool: + if self._outcome is None and not self._ending: + return False + if not self._dropped_logged: + self._dropped_logged = True + logger.debug("[%s] a write after the turn ended was dropped", self._task_id) + return True + + def _emit(self, event: StreamEvent, parent_tool_id: str | None = None, *, stamp: datetime | None = None) -> None: + event.timestamp = stamp if stamp is not None else self.now() + event.thread_id = event.parent_thread_id = parent_tool_id + self._events += 1 + self._collector.on_event(event) + for sink in self._sinks: + safe_emit(sink, event) + + def _next_sequence(self) -> int: + sequence = self._sequence + self._sequence += 1 + return sequence + + def _stamp( + self, keyword: str, value: datetime | None, now: datetime, parent_tool_id: str | None + ) -> datetime | None: + if parent_tool_id is not None: + if value is not _UNSET: + return value + return now if self._basis is TimingBasis.TURN_CLOCK else None + if self._basis is TimingBasis.TURN_CLOCK: + if value is not _UNSET: + raise TypeError( + f"{keyword}= is not accepted under TimingBasis.TURN_CLOCK: the emitter stamps the clock" + ) + return now + if value is _UNSET: + raise TypeError(f"{keyword}= is required under TimingBasis.{self._basis.name} (None means no CLI stamp)") + return value + + def _already_closed(self, tool_id: str) -> bool: + if tool_id not in self._closed_tool_ids: + return False + logger.debug("ignoring a repeated open or close for tool id %s, already closed in this turn", tool_id) + return True + + def _close( + self, + tool_id: str, + status: ToolEndStatus, + summary: str | None, + error: str | None, + result_data: dict[str, Any] | list[Any] | None, + parameters: dict[str, Any] | None, + stamp: datetime | None, + reported_duration_ms: float | None = None, + ) -> None: + now = self.now() + self._closed_tool_ids.add(tool_id) + opened = self._open_tools.pop(tool_id, None) + if opened is None: + opened = _OpenTool( + CommandTelemetry( + tool_name="unknown", tool_id=tool_id, timestamp=now, sequence_number=self._next_sequence() + ), + self._turn_id or "", + None, + ) + telemetry = opened.telemetry + if status is not ToolEndStatus.UNRESOLVED and stamp is not None: + telemetry.execution_completed_at = stamp + if telemetry.execution_started_at is not None: + telemetry.duration_ms = max(0.0, (stamp - telemetry.execution_started_at).total_seconds() * 1000) + if ( + status is not ToolEndStatus.UNRESOLVED + and telemetry.duration_ms is None + and reported_duration_ms is not None + and reported_duration_ms > 0 + ): + telemetry.duration_ms = reported_duration_ms + telemetry.result_status = _RESULT_STATUS[status] + telemetry.result_summary = summary + telemetry.error_message = error + telemetry.result_data = result_data + if parameters is not None: + telemetry.parameters = parameters + self._emit( + ToolEndEvent(task_id=self._task_id, turn_id=opened.turn_id, tool=telemetry, status=status), + opened.parent_tool_id, + ) + + def _message( + self, + part: Generation, + started_at: datetime, + completed_at: datetime, + duration_ms: float | None, + message_id: str | None, + model: str | None, + parent_tool_id: str | None, + ) -> AssistantMessage: + tokens = part.tokens + return AssistantMessage( + started_at=started_at, + completed_at=completed_at, + generation_duration_ms=duration_ms, + content_blocks=part.blocks, + tool_use_ids=[b.tool_use_id for b in part.blocks if b.block_type == "tool_use" and b.tool_use_id], + input_tokens=tokens.uncached_input_tokens, + output_tokens=tokens.output_tokens, + cache_creation_tokens=tokens.cache_creation_input_tokens, + cache_read_tokens=tokens.cache_read_input_tokens, + reasoning_tokens=part.reasoning_tokens, + stop_reason=part.stop_reason, + model=model or self._model, + message_id=message_id, + parent_tool_use_id=parent_tool_id, + ) + + def _final_reply(self) -> str | None: + main = [m for m in self._messages if m.parent_tool_use_id is None] + if not main: + return None + blocks = main[-1].content_blocks + last_tool = max((i for i, b in enumerate(blocks) if b.block_type == "tool_use"), default=-1) + return "".join(b.text or "" for b in blocks[last_tool + 1 :] if b.block_type == "text") or None + + def _end( + self, + status: AgentEndStatus, + *, + reason: str | None, + usage: TokenUsage | None, + agent_output: str | None, + model_used: str | None, + assistant_turn_count: int | None, + num_turns: int | None, + result_summary: ResultSummary | None, + ) -> TurnOutcome: + for tool_id in list(self._open_tools): + self._close(tool_id, ToolEndStatus.UNRESOLVED, None, None, None, None, None) + if self._turn_id is not None: + self.end_inner_turn(TurnEndStatus(status.value)) + published = usage if usage is not None else self._reported + self._warn_on_delta_overshoot(published) + crashed = status in _FAILED + ended_at = self.now() + self._emit( + AgentEndEvent( + task_id=self._task_id, + status=status, + usage=published, + iteration=self._iteration, + user_input=self._prompt, + agent_output=agent_output if agent_output is not None else "".join(self._text), + model_used=model_used if model_used is not None else self._model, + assistant_turn_count=assistant_turn_count if assistant_turn_count is not None else self._main_turns, + messages=list(self._messages), + num_turns=self._main_turns if num_turns is _UNSET else num_turns, + result_summary=result_summary, + crashed=crashed, + crash_reason=truncate_crash_message(reason) if reason is not None else None, + duration_seconds=self._duration_seconds(ended_at), + ), + stamp=ended_at, + ) + self._ending = True + self._outcome = TurnOutcome(record=self._collector.build_turn_record(), status=status, error=reason) + return self._outcome + + def _duration_seconds(self, ended_at: datetime) -> float: + """The bracket's own span; monotonic when the clock is the wall clock, which can step.""" + if self._began_at is None: + return 0.0 + if self._basis is TimingBasis.TURN_CLOCK: + return (ended_at - self._began_at).total_seconds() + return time.monotonic() - self._began_monotonic + + def _warn_on_delta_overshoot(self, published: TokenUsage) -> None: + over = [ + f"{bucket} {getattr(self._reported, bucket)} > {getattr(published, bucket)}" + for bucket in _BUCKETS + if getattr(self._reported, bucket) > getattr(published, bucket) + ] + if over: + logger.warning( + "[%s] the inner-turn token deltas exceed the published turn usage (%s); a harness double-counts", + self._task_id, + "; ".join(over), + ) diff --git a/src/coder_eval/streaming/events.py b/src/coder_eval/streaming/events.py index 93a92e9f..9dfe131b 100644 --- a/src/coder_eval/streaming/events.py +++ b/src/coder_eval/streaming/events.py @@ -82,6 +82,7 @@ class StopReason(StrEnum): EARLY_CRITERION = "early_criterion" TOOL_CALL_CAP = "tool_call_cap" + MODEL_TURN_CAP = "model_turn_cap" TOKEN_BUDGET = "token_budget" USD_BUDGET = "usd_budget" @@ -89,6 +90,7 @@ class StopReason(StrEnum): _END_STATUS_FOR_STOP: dict[StopReason, AgentEndStatus] = { StopReason.EARLY_CRITERION: AgentEndStatus.STOPPED_EARLY, StopReason.TOOL_CALL_CAP: AgentEndStatus.TOOL_CALLS_EXHAUSTED, + StopReason.MODEL_TURN_CAP: AgentEndStatus.TOOL_CALLS_EXHAUSTED, StopReason.TOKEN_BUDGET: AgentEndStatus.TOKEN_BUDGET_EXCEEDED, StopReason.USD_BUDGET: AgentEndStatus.COST_BUDGET_EXCEEDED, } diff --git a/src/coder_eval/testing.py b/src/coder_eval/testing.py new file mode 100644 index 00000000..27386dcb --- /dev/null +++ b/src/coder_eval/testing.py @@ -0,0 +1,416 @@ +"""The test harness a harness adapter shares with the in-tree suites: replay, identity, balance, conformance. + +No ``pytest`` import: every check raises ``AssertionError``, and callers parametrize. +A plugin calls these from its own tests exactly as ``tests/`` does. +""" + +from __future__ import annotations + +import math +from collections import Counter +from collections.abc import Awaitable, Callable, Iterable, Mapping, Sequence +from dataclasses import dataclass +from datetime import datetime, timedelta +from typing import TYPE_CHECKING, Any + +from coder_eval.models import AssistantMessage, Enforcement, HarnessContract, PermissionMode, TimingBasis, TurnRecord +from coder_eval.streaming.emitter import TurnEmitter, TurnOutcome +from coder_eval.streaming.events import ( + AgentEndEvent, + AgentEndStatus, + AgentStartEvent, + StopReason, + StreamEvent, + ToolEndEvent, + ToolStartEvent, + TurnEndEvent, + TurnStartEvent, + end_status_for, +) +from coder_eval.timing import main_thread_tool_spans, union_ms + + +if TYPE_CHECKING: + from coder_eval.models import RunLimits, TaskDefinition + + +@dataclass(frozen=True) +class Tick: + """A stream element that moves the replay's ``ScriptedClock`` to ``at_ms`` after its origin.""" + + at_ms: float + + +class ScriptedClock: + """A clock that reads ``origin + at_ms`` and moves only on a ``Tick``.""" + + def __init__(self, origin: datetime) -> None: + self._origin = origin + self._at_ms = 0.0 + + def now(self) -> datetime: + return self._origin + timedelta(milliseconds=self._at_ms) + + def _move_to(self, at_ms: float) -> None: + self._at_ms = at_ms + + +@dataclass(frozen=True) +class Replay: + """What a replayed turn produced; ``started_at`` / ``ended_at`` are its bracket stamps.""" + + record: TurnRecord + events: list[StreamEvent] + outcome: TurnOutcome + started_at: datetime + ended_at: datetime + + +class _Recorder: + def __init__(self) -> None: + self.events: list[StreamEvent] = [] + + def on_event(self, event: StreamEvent) -> None: + self.events.append(event) + + +def replay[D: Callable[[Any], None]]( + stream: Iterable[Any], + make_decoder: Callable[[TurnEmitter], D], + *, + clock: ScriptedClock, + basis: TimingBasis = TimingBasis.TURN_CLOCK, + model: str | None = "m", + end: Callable[[D], TurnOutcome] | None = None, +) -> Replay: + """Drive a decoder over ``stream`` through a real ``TurnEmitter`` on ``clock``. + + A ``Tick`` moves the clock; every other element is passed to the decoder. The turn + ends with ``end(decoder)`` when given, else ``emitter.finalize(COMPLETED)``. An + exception from the decoder propagates. + """ + recorder = _Recorder() + emitter = TurnEmitter( + task_id="replay", iteration=1, prompt="go", model=model, basis=basis, clock=clock, sinks=[recorder] + ) + emitter.begin() + decoder = make_decoder(emitter) + for element in stream: + if isinstance(element, Tick): + clock._move_to(element.at_ms) # pyright: ignore[reportPrivateUsage] + else: + decoder(element) + outcome = end(decoder) if end is not None else emitter.finalize(AgentEndStatus.COMPLETED) + starts = [e for e in recorder.events if isinstance(e, AgentStartEvent)] + ends = [e for e in recorder.events if isinstance(e, AgentEndEvent)] + return Replay( + record=outcome.record, + events=recorder.events, + outcome=outcome, + started_at=starts[0].timestamp, + ended_at=ends[-1].timestamp, + ) + + +def assert_identity_closes(record: TurnRecord, *, started_at: datetime, ended_at: datetime) -> None: + """Assert head + Σ generation + UNION(tool) + tail equals the turn's span, to float precision. + + Main thread only on both sides, through production's own span selector. Also + asserts the stored ``tool_union_ms`` equals the union computed here. + + Raises: + AssertionError: a bucket is missing, the stored union disagrees, or the buckets + do not tile the span. + """ + span_ms = (ended_at - started_at).total_seconds() * 1000.0 + generation_ms = sum( + m.generation_duration_ms + for m in record.messages + if isinstance(m, AssistantMessage) and m.parent_tool_use_id is None and m.generation_duration_ms is not None + ) + tool_ms = union_ms(main_thread_tool_spans(record.messages, record.commands)) + head, tail = record.harness_startup_ms, record.harness_teardown_ms + if head is None or tail is None: + raise AssertionError(f"a turn that generated has a measured head and tail (head={head}, tail={tail})") + stored = record.tool_union_ms + if (stored is None and tool_ms > 0) or (stored is not None and not math.isclose(stored, tool_ms, abs_tol=1e-6)): + raise AssertionError( + f"TurnRecord.tool_union_ms is {record.tool_union_ms}, but the main-thread command spans union to " + + f"{tool_ms:.4f} ms: the stored value and the selection rule have come apart" + ) + bucket_sum = head + generation_ms + tool_ms + tail + if not math.isclose(bucket_sum, span_ms, abs_tol=1e-6): + raise AssertionError( + f"the four buckets sum to {bucket_sum:.4f} ms against a {span_ms:.4f} ms turn " + + f"(off by {bucket_sum - span_ms:+.4f} ms): head={head:.4f}, generation={generation_ms:.4f}, " + + f"tool_union={tool_ms:.4f}, tail={tail:.4f}. A sum UNDER the turn means some interval is " + + "booked nowhere; a sum OVER it means one is booked twice." + ) + + +_BUCKETS = ("uncached_input_tokens", "output_tokens", "cache_creation_input_tokens", "cache_read_input_tokens") + + +def assert_stream_balanced(events: Sequence[StreamEvent]) -> None: + """Assert one turn's event stream is well formed. + + Exactly one ``AgentStartEvent``, first, and one ``AgentEndEvent``, last; every + ``TurnStartEvent`` closed by a ``TurnEndEvent`` of the same ``turn_id`` before the + next start and before the end, and no end without its start; every tool id started + once is ended exactly once, and none ends unstarted; per token bucket, the sum of + ``TurnEndEvent.tokens`` is at most ``AgentEndEvent.usage``. + + Raises: + AssertionError: naming every violation found. + """ + problems: list[str] = [] + if not events: + raise AssertionError("the stream is empty") + starts = [e for e in events if isinstance(e, AgentStartEvent)] + ends = [e for e in events if isinstance(e, AgentEndEvent)] + if len(starts) != 1 or not isinstance(events[0], AgentStartEvent): + problems.append(f"{len(starts)} AgentStartEvent(s), first event is {type(events[0]).__name__}") + if len(ends) != 1 or not isinstance(events[-1], AgentEndEvent): + problems.append(f"{len(ends)} AgentEndEvent(s), last event is {type(events[-1]).__name__}") + open_turn: str | None = None + for event in events: + if isinstance(event, TurnStartEvent): + if open_turn is not None: + problems.append(f"turn {event.turn_id!r} started while {open_turn!r} is open") + open_turn = event.turn_id + elif isinstance(event, TurnEndEvent): + if open_turn != event.turn_id: + problems.append(f"turn {event.turn_id!r} ended while the open turn is {open_turn!r}") + open_turn = None + elif isinstance(event, AgentEndEvent) and open_turn is not None: + problems.append(f"turn {open_turn!r} is still open at the AgentEndEvent") + open_turn = None + tool_starts = Counter(e.tool.tool_id for e in events if isinstance(e, ToolStartEvent)) + tool_ends = Counter(e.tool.tool_id for e in events if isinstance(e, ToolEndEvent)) + problems += [f"tool {tid!r} started {n} times" for tid, n in tool_starts.items() if n != 1] + problems += [f"tool {tid!r} ended {tool_ends[tid]} times" for tid in tool_starts if tool_ends[tid] != 1] + problems += [f"tool {tid!r} ended without a start" for tid in tool_ends if tid not in tool_starts] + if ends: + usage = ends[-1].usage + for bucket in _BUCKETS: + reported = sum(getattr(e.tokens, bucket) for e in events if isinstance(e, TurnEndEvent) and e.tokens) + if reported > getattr(usage, bucket): + problems.append(f"TurnEndEvent {bucket} sum {reported} > AgentEndEvent.usage {getattr(usage, bucket)}") + if problems: + raise AssertionError("unbalanced event stream: " + "; ".join(problems)) + + +_FIELDS = ("system_prompt", "plugin_skills", "permission_mode", "allowed_tools", "disallowed_tools") +_CONFIG_FIELD = {"plugin_skills": "plugins"} +_GATED_VALUES: dict[str, Any] = { + "system_prompt": "CONFORMANCE-MARKER-7f3a", + "plugins": [{"type": "local", "path": "/plugins/p"}], + "permission_mode": "plan", + "allowed_tools": ["Bash"], + "disallowed_tools": ["Bash"], +} + + +def enforced_cells(contract: HarnessContract, kind: str) -> set[tuple[str, str]]: + """``(kind, cell)`` for every ENFORCED field; ``permission_mode`` gives one ``permission_mode=`` per mode.""" + cells: set[tuple[str, str]] = set() + for field in _FIELDS: + if getattr(contract, field) is not Enforcement.ENFORCED: + continue + if field == "permission_mode": + cells |= {(kind, f"permission_mode={mode.value}") for mode in contract.permission_modes or ()} + else: + cells.add((kind, field)) + return cells + + +def _task(kind: str, *, run_limits: RunLimits | None = None, **agent: Any) -> TaskDefinition: + from coder_eval.models import AgentKind, FileExistsCriterion, SandboxConfig, TaskDefinition, parse_agent_config + + return TaskDefinition( + task_id="t", + description="d", + initial_prompt=None if kind == AgentKind.NONE.value else "do the task", + agent=parse_agent_config(type=kind, **agent), + sandbox=SandboxConfig(driver="tempdir"), + success_criteria=[FileExistsCriterion(description="c", path="out.txt")], + run_limits=run_limits, + ) + + +def _expect_rejected(task: Callable[[], TaskDefinition], match: str) -> None: + import re + + from coder_eval.orchestration.harness_contract import HarnessContractError, validate_harness_contract + + try: + validate_harness_contract(task()) + except HarnessContractError as error: + if not re.search(match, str(error)): + raise AssertionError(f"rejected, but {str(error)!r} does not match {match!r}") from error + return + raise AssertionError(f"expected a HarnessContractError matching {match!r}; the task resolved") + + +def rejections(kind: str) -> list[tuple[str, Callable[[], None]]]: + """The resolution-time rejections ``kind``'s contract implies, as named checks that raise ``AssertionError``.""" + from coder_eval.agents.registry import AgentRegistry + from coder_eval.plugins import ensure_plugins_loaded + + ensure_plugins_loaded() + registration = AgentRegistry.get(kind) + if registration is None: + raise AssertionError(f"agent kind {kind!r} is not registered") + contract = registration.agent_class.contract + checks: list[tuple[str, Callable[[], None]]] = [] + for field in _FIELDS: + if getattr(contract, field) is Enforcement.UNSUPPORTED: + config_field = _CONFIG_FIELD.get(field, field) + value = _GATED_VALUES[config_field] + checks.append( + ( + f"unsupported {config_field}", + lambda c=config_field, v=value: _expect_rejected( + lambda: _task(kind, **{c: v}), rf"agent\.{c}.*{kind!r}" + ), + ) + ) + if contract.permission_mode is Enforcement.ENFORCED: + for mode in PermissionMode: + if mode not in (contract.permission_modes or frozenset()): + checks.append( + ( + f"undeclared permission_mode={mode.value}", + lambda m=mode: _expect_rejected( + lambda: _task(kind, permission_mode=m), "has no documented meaning" + ), + ) + ) + if Enforcement.ENFORCED in (contract.allowed_tools, contract.disallowed_tools): + checks.append( + ( + "misspelled tool name", + lambda: _expect_rejected(lambda: _task(kind, allowed_tools=["Bassh"]), "did you mean 'Bash'"), + ) + ) + if not contract.reports_cost: + from coder_eval.models import RunLimits + + checks.append( + ( + "max_usd without a priced agent.model", + lambda: _expect_rejected( + lambda: _task(kind, run_limits=RunLimits(max_usd=1.0), model="conformance/not-on-the-rate-card"), + r"run_limits\.max_usd is set", + ), + ) + ) + if not contract.counts_model_turns: + from coder_eval.models import RunLimits + from coder_eval.orchestration.harness_contract import MODEL_TURN_LIMITS + + for field in MODEL_TURN_LIMITS: + checks.append( + ( + f"unsupported run_limits.{field}", + lambda f=field: _expect_rejected( + lambda: _task(kind, run_limits=RunLimits.model_validate({f: 1})), rf"run_limits\.{f}" + ), + ) + ) + return checks + + +async def conformance(kind: str, probes: Mapping[tuple[str, str], Callable[[], Awaitable[None]]]) -> None: + """Assert ``kind`` rejects what its contract marks unsupported and honors every enforced cell. + + Runs every check from ``rejections(kind)``, asserts ``probes`` covers exactly + ``enforced_cells(contract, kind)``, then awaits every probe. + + Raises: + AssertionError: a rejection is missing, a probe is missing or extra, or a probe fails. + """ + from coder_eval.agents.registry import AgentRegistry + + for _name, check in rejections(kind): + check() + registration = AgentRegistry.get(kind) + if registration is None: + raise AssertionError(f"no agent is registered for {kind!r}") + expected = enforced_cells(registration.agent_class.contract, kind) + if set(probes) != expected: + raise AssertionError( + f"probes for {kind!r} do not match its enforced cells: missing {sorted(expected - set(probes))}, " + + f"extra {sorted(set(probes) - expected)}" + ) + for cell in sorted(probes): + await probes[cell]() + + +FIRST_TOOL_ID = "first" +SECOND_TOOL_ID = "second-call" + + +class StopAfterFirstTool: + """A ``should_stop`` poll and stream callback in one: ``reason`` once one tool call has resolved. + + ``end`` holds the turn's ``AgentEndEvent`` once it arrives. + """ + + def __init__(self, reason: StopReason) -> None: + self.reason = reason + self.tool_ends = 0 + self.end: AgentEndEvent | None = None + + def on_event(self, event: StreamEvent) -> None: + if isinstance(event, ToolEndEvent): + self.tool_ends += 1 + elif isinstance(event, AgentEndEvent): + self.end = event + + def __call__(self) -> StopReason | None: + return self.reason if self.tool_ends else None + + +type StopProbe = Callable[[StopAfterFirstTool, StopReason], Awaitable[Sequence[str]]] + + +async def stop_conformance(kind: str, probe: StopProbe) -> None: + """Assert ``kind`` ends the turn at the first boundary for every ``StopReason``. + + For each reason, ``probe(stop, reason)`` runs one ``communicate()`` whose scripted + harness makes the tool calls ``FIRST_TOOL_ID`` then ``SECOND_TOOL_ID``, passing + ``stop`` as both ``stream_callback`` and ``should_stop``. It returns the tool ids the + adapter pulled from the harness. The turn must end with ``end_status_for(reason)``, + not crashed, after pulling the first call and before pulling the second. + + Raises: + AssertionError: ``kind`` does not declare ``cooperative_stop``, or a reason is not honored. + """ + from coder_eval.agents.registry import AgentRegistry + from coder_eval.plugins import ensure_plugins_loaded + + ensure_plugins_loaded() + registration = AgentRegistry.get(kind) + if registration is None: + raise AssertionError(f"agent kind {kind!r} is not registered") + if not registration.agent_class.contract.cooperative_stop: + raise AssertionError(f"agent kind {kind!r} does not declare cooperative_stop, so it has no stop to check") + for reason in StopReason: + stop = StopAfterFirstTool(reason) + pulled = await probe(stop, reason) + problems: list[str] = [] + expected = end_status_for(reason) + if stop.end is None: + problems.append("no AgentEndEvent") + else: + if stop.end.status is not expected: + problems.append(f"ended {stop.end.status.value}, not {expected.value}") + if stop.end.crashed: + problems.append(f"crashed: {stop.end.crash_reason}") + if FIRST_TOOL_ID not in pulled: + problems.append(f"never pulled {FIRST_TOOL_ID!r}") + if SECOND_TOOL_ID in pulled: + problems.append(f"pulled {SECOND_TOOL_ID!r} after the stop") + if problems: + raise AssertionError(f"{kind!r} did not honor StopReason.{reason.name}: " + "; ".join(problems)) diff --git a/src/coder_eval/timing.py b/src/coder_eval/timing.py index 25c7c77f..04fea023 100644 --- a/src/coder_eval/timing.py +++ b/src/coder_eval/timing.py @@ -21,6 +21,7 @@ import math import time from collections.abc import Iterable +from dataclasses import dataclass from datetime import datetime, timedelta from coder_eval.models import AssistantMessage, CommandTelemetry, TranscriptMessage @@ -127,8 +128,21 @@ def union_ms(spans: list[tuple[datetime, datetime]]) -> float: return busy_ms(spans, min(s for s, _ in spans), max(e for _, e in spans)) -def close_window(*, mark: datetime, now: datetime, item_start: datetime | None = None) -> tuple[datetime, float]: - """Open one generation window at ``mark`` and close it at ``now``: its ``(started, span_ms)``. +@dataclass(frozen=True, slots=True) +class Window: + """One generation window: its two bounds, and nothing a caller could set apart from them.""" + + started_at: datetime + completed_at: datetime + + @property + def duration_ms(self) -> float: + """``completed_at - started_at`` in ms, clamped at ``0.0``: an inverted window is a measured zero.""" + return max(0.0, (self.completed_at - self.started_at).total_seconds() * 1000.0) + + +def close_window(*, mark: datetime, now: datetime, item_start: datetime | None = None) -> Window: + """Open one generation window at ``mark`` and close it at ``now``. The shape all five reducers share, and what it returns is the RAW window — tool execution comes back out centrally, in ``subtract_tool_time``. @@ -141,14 +155,12 @@ def close_window(*, mark: datetime, now: datetime, item_start: datetime | None = ``item_start`` is this emission's own first stamp, when the harness has one; the ``min()`` against ``mark`` stops a backwards stamp inverting the span. - The span is clamped at ``0.0``: an inverted window is a measured zero, not a - negative generation. ``completed`` is deliberately not returned — it is - always ``now``, which the caller already has. + An inverted window keeps its bounds and its duration clamps to ``0.0``. Rationale: .claude/notes/timing.md § close_window """ started = min(mark, item_start) if item_start is not None else mark - return started, max(0.0, (now - started).total_seconds() * 1000.0) + return Window(started_at=started, completed_at=now) def decompose_turn( @@ -261,7 +273,7 @@ def subtract_tool_time( Raises ``ValueError`` if a group's published total does not equal the span its bounds describe. That equality is what lets ``generation_duration_ms`` - stay a PUBLISHED field; CE061 forces the same shape statically. Raising + stay a PUBLISHED field; ``TurnEmitter.add_generation`` builds it from one ``Window``. Raising kills the turn, which is accepted. Rationale: .claude/notes/timing.md § subtract_tool_time @@ -300,8 +312,8 @@ def subtract_tool_time( + "into, sums to it) — tool execution comes back out HERE, once, for every harness. " + "A disagreement means the reducer narrowed or widened a window without moving its " + "bounds, which makes the duration and the bounds two answers to one question and " - + "breaks the four-bucket identity. Build the window with `timing.close_window` and " - + "write `completed_at=now` (CE061), rather than adjusting the duration in place." + + "breaks the four-bucket identity. Pass `TurnEmitter.add_generation` a `Window` from " + + "`timing.close_window`, rather than adjusting the duration in place." ) net = max(raw_total - busy_ms(spans, started, completed), 0.0) assigned = 0.0 diff --git a/tasks/mock_path_dirs_smoke.yaml b/tasks/mock_path_dirs_smoke.yaml index 22b80d44..a3230f1e 100644 --- a/tasks/mock_path_dirs_smoke.yaml +++ b/tasks/mock_path_dirs_smoke.yaml @@ -31,3 +31,7 @@ success_criteria: path: "out.txt" includes: ["MOCK_PATH_OK from say_hello"] description: "out.txt must contain the mock binary's signature line." + - type: "run_command" + command: "say_hello criterion" + timeout: 10 + description: "A criterion command resolves the mock binary from PATH, as the agent did." diff --git a/tasks/run_limits/max_turns_cap.yaml b/tasks/run_limits/max_turns_cap.yaml new file mode 100644 index 00000000..7a6b22e0 --- /dev/null +++ b/tasks/run_limits/max_turns_cap.yaml @@ -0,0 +1,49 @@ +task_id: run-limits-max-turns-cap +description: >- + Parity fixture for run_limits.max_turns. The prompt asks for far more + sequential model turns than the cap allows, so every harness that counts model + turns must stop at the cap rather than running the prompt to completion. Run it + with --type claude-code / codex / antigravity / opencode / pi and compare: the cap + must produce a CLEAN stop (`tool_calls_exhausted`, criteria still checked), never + a crash. + +tags: + - run-limits + - max-turns + - parity + +initial_prompt: | + Create 12 files in the current directory named step-01.txt through step-12.txt. + step-01.txt must contain just its own name. Every later file must contain the + contents of the PREVIOUS file, then its own name on a new line — so you have to + read step-N before you can write step-N+1. + + Create them ONE AT A TIME. Run a separate shell command for each file. Do not + use a loop, do not combine several files into one command, and do not batch + multiple tool calls together. Work strictly in order, starting at step-01.txt. + +run_limits: + # Far below the 12 the prompt asks for, so an agent that follows the prompt always meets the cap. + max_turns: 4 + # Generous: this fixture must fail on the cap, never on the clock. + turn_timeout: 300 + task_timeout: 600 + +success_criteria: + # The early files prove the agent really was working when the cap cut it off, + # which distinguishes "capped" from "never started". + - type: file_exists + path: "step-01.txt" + description: "First file was created before the cap fired" + weight: 1.0 + + # And this is the half that actually tests the cap. Without it the fixture + # passes on a harness that ignores the cap entirely — the exact bug it exists + # to catch — because step-01.txt gets written either way. The chained contents + # in the prompt make each step depend on reading the one before it, so no amount + # of batching can reach step 12 inside a cap of 4 model turns; a run that + # produced the last file therefore ran uncapped. + - type: run_command + command: "test ! -f step-12.txt" + description: "The cap bound the run: the agent never reached the last file" + weight: 1.0 diff --git a/tasks/run_limits/subagent_cap.yaml b/tasks/run_limits/subagent_cap.yaml new file mode 100644 index 00000000..96b6824c --- /dev/null +++ b/tasks/run_limits/subagent_cap.yaml @@ -0,0 +1,36 @@ +task_id: run-limits-subagent-cap +description: >- + Proves run_limits.max_tool_calls counts main-thread tool calls only. The main + thread makes exactly two calls (Task, then Write) and the sub-agent it spawns + makes at least one Bash call. With a cap of 2 the task must end SUCCESS with + answer.txt written and model_used equal to the main model. If sub-agent calls + counted, the cap would trip at Task + Bash and answer.txt would never exist. + +tags: + - run-limits + - sub-agent + - claude + +agent: + type: claude-code + +initial_prompt: | + Delegate this to a SUB-AGENT (use the Task tool to spawn one). Instruct the + sub-agent to compute the sum of the integers from 1 to 100 inclusive by RUNNING + A SHELL COMMAND with its Bash tool (for example `python3 -c "print(sum(range(1,101)))"` + or `seq 1 100 | paste -sd+ | bc`) — it must execute a command, not reason the + number out in its head. When the sub-agent returns the result, write just that + number to a file named answer.txt in the working directory, and stop. Do not + compute the sum yourself — have the sub-agent do it via Bash. + Make exactly two tool calls yourself: the Task call, then the Write call. + +success_criteria: + - type: file_contains + path: answer.txt + includes: ["5050"] + description: "answer.txt must contain the sub-agent's computed sum (5050)." + +run_limits: + max_tool_calls: 2 + turn_timeout: 180 + task_timeout: 300 diff --git a/tasks/run_limits/turn_timeout.yaml b/tasks/run_limits/turn_timeout.yaml index 184798a8..bbd818e9 100644 --- a/tasks/run_limits/turn_timeout.yaml +++ b/tasks/run_limits/turn_timeout.yaml @@ -11,7 +11,7 @@ tags: - parity initial_prompt: | - Run the shell command `sleep 240` and wait for it to finish. When it returns, + Run the shell command `python3 -c "import time; time.sleep(240)"` and wait for it to finish. When it returns, report its exit code. Do not run it in the background, and do not shorten the sleep. diff --git a/tasks/smoke_task_timeout.yaml b/tasks/smoke_task_timeout.yaml index 4617e5fe..80b9861d 100644 --- a/tasks/smoke_task_timeout.yaml +++ b/tasks/smoke_task_timeout.yaml @@ -7,13 +7,15 @@ description: | would let the runner hang up to the GitHub-Actions job timeout (10 min) and burn a real budget — this guards against that silently happening. initial_prompt: | - Run this Bash command and wait for it to finish: `sleep 300` + Run this Bash command and wait for it to finish: `python3 -c "import time; time.sleep(300)"` # task_timeout enforced by an orchestrator watchdog. Minimum allowed value # per the TaskDefinition validator (ge=30) is 30s — anything shorter would # constitute a misuse warning at load time. Wall-clock cost in CI is ~30s. # Sleep is 300s (10× task_timeout) so a regression that lets the agent run # through but kills it on the SDK-level turn timeout would also miscount. +# The sleep is spelled through python3 because Claude Code refuses a standalone +# `sleep N` and the turn would then end before any timeout fires. # max_tool_calls stays out of reach: a retried or backgrounded `sleep` must not # end the run on the cap before the watchdog fires. run_limits: diff --git a/tests/_bracket_clock.py b/tests/_bracket_clock.py index 46faa1ea..31921a7b 100644 --- a/tests/_bracket_clock.py +++ b/tests/_bracket_clock.py @@ -1,10 +1,8 @@ -"""A ``TurnClock`` stand-in anchored far from real time, for the CE064 tests. +"""A ``TurnClock`` stand-in anchored far from real time, for the turn-bracket tests. -CE064 checks only that ``timestamp=`` is PRESENT on an ``AgentStartEvent`` / -``AgentEndEvent`` emit — its own declared blind spot is that it cannot tell -``self.clock.now()`` from a ``datetime.now()`` written out at the call site. -This is the guard for the SOURCE of that stamp, on the three harnesses that own -a clock. +``TurnEmitter`` stamps the ``AgentStartEvent`` / ``AgentEndEvent`` bracket from the +turn's clock. This is the guard for the SOURCE of that stamp, on the three harnesses +that run on a ``TurnClock``. ANCHORED FAR FROM NOW, and that is the whole trick. A bracket left on ``StreamEvent.timestamp``'s ``default_factory=datetime.now`` lands within @@ -17,7 +15,7 @@ what lets the same fixture assert the second half: with the bracket and the window bounds finally on one basis, ``decompose_turn``'s head and tail come out as small positive measurements rather than as the clamped ``0.0`` a cross-basis -subtraction produced (see ``ce064_turn_bracket_on_the_clock``'s measured probe). +subtraction produced (the measured probe is in ``docs/agents/HARNESS_PARITY.md``). """ import time @@ -60,7 +58,7 @@ def assert_bracket_on_the_clock(events: list[StreamEvent]) -> None: f"{type(event).__name__}.timestamp is {event.timestamp}, which is not from the turn's " f"clock (anchored at {ANCHOR}). It fell back to StreamEvent's default_factory=datetime.now, " "so the turn bracket and the generation-window bounds sit on two bases inside one " - "`decompose_turn` subtraction — see CE064." + "`decompose_turn` subtraction." ) assert ends[0].timestamp >= starts[0].timestamp @@ -93,5 +91,5 @@ def assert_overhead_is_measured(record: TurnRecord) -> None: ) assert 0.0 < record.harness_teardown_ms < 60_000.0, ( f"harness_teardown_ms is {record.harness_teardown_ms} ms — a 0.0 here is the clamped " - "inversion CE064 exists to remove, not an instant teardown" + "inversion of a bracket off the turn clock, not an instant teardown" ) diff --git a/tests/_fixtures/golden_streams/__init__.py b/tests/_fixtures/golden_streams/__init__.py index dbec503a..d7363b77 100644 --- a/tests/_fixtures/golden_streams/__init__.py +++ b/tests/_fixtures/golden_streams/__init__.py @@ -3,7 +3,7 @@ A safety net for the ``ClaudeCodeAgent.communicate`` / ``CodexAgent`` turn-loop decomposition: each scenario replays a recorded SDK event stream through ``communicate()`` and snapshots the resulting ``TurnRecord`` (or, on a -crash/timeout, the ``pending_turn`` partial) as canonical JSON. The decomposition +crash/timeout, the outcome's crashed partial) as canonical JSON. The decomposition must keep these snapshots byte-identical post-scrub. The scrubber masks only the inherently per-run fields (timestamps, durations, diff --git a/tests/_fixtures/golden_streams/_recorder.py b/tests/_fixtures/golden_streams/_recorder.py new file mode 100644 index 00000000..0ef18d14 --- /dev/null +++ b/tests/_fixtures/golden_streams/_recorder.py @@ -0,0 +1,11 @@ +"""A stream callback that keeps every event a golden replay emitted.""" + +from coder_eval.streaming.events import StreamEvent + + +class EventRecorder: + def __init__(self) -> None: + self.events: list[StreamEvent] = [] + + def on_event(self, event: StreamEvent) -> None: + self.events.append(event) diff --git a/tests/_fixtures/golden_streams/_scrub.py b/tests/_fixtures/golden_streams/_scrub.py index 0c19a806..5a8a7907 100644 --- a/tests/_fixtures/golden_streams/_scrub.py +++ b/tests/_fixtures/golden_streams/_scrub.py @@ -133,7 +133,7 @@ def _tool_union_ms(record: dict[str, Any]) -> float: What is shared with production is the SELECTION and ``union_ms``. What is NOT shared is the bookkeeping around them — this still builds its own span set and computes its own union, which is where every timing defect on this - branch actually lived (see CE063's docstring). Do not "simplify" it into + branch actually lived. Do not "simplify" it into reading ``tool_union_ms``: that would make the sensor a restatement of the producer's answer, and the cross-check below is what verifies that field. """ @@ -175,9 +175,7 @@ def assert_timing_captured( The bounds half is not redundant. Two harnesses derive the duration from a MONOTONIC clock and the bounds from the wall clock, so the two can disagree: a reducer could report a healthy duration beside two stamps that - collapsed to one instant. CE059 catches that statically only when both - bounds are the same ``ast.Name``; when they are two different names - holding the same value it cannot, and this is the check that does. + collapsed to one instant, and this is the check that catches it. **Unconditional, and keyed on the messages rather than on the flag.** A turn's head and tail (``harness_startup_ms`` / ``harness_teardown_ms``) are diff --git a/tests/_fixtures/golden_streams/antigravity_fixtures.py b/tests/_fixtures/golden_streams/antigravity_fixtures.py index 17f4ccf8..2aa2a1ad 100644 --- a/tests/_fixtures/golden_streams/antigravity_fixtures.py +++ b/tests/_fixtures/golden_streams/antigravity_fixtures.py @@ -22,6 +22,8 @@ from coder_eval.agents import antigravity_agent from coder_eval.agents.antigravity_agent import AntigravityAgent from coder_eval.models import parse_agent_config +from coder_eval.streaming.events import StreamEvent +from tests._fixtures.golden_streams._recorder import EventRecorder def _usage(prompt: int, cached: int, candidates: int, thoughts: int) -> SimpleNamespace: @@ -135,8 +137,11 @@ class AntigravityScenario: steps: list[Any] -async def run_antigravity_scenario(scenario: AntigravityScenario, working_dir: str) -> dict[str, Any]: +async def run_antigravity_scenario( + scenario: AntigravityScenario, working_dir: str +) -> tuple[dict[str, Any], list[StreamEvent]]: """Replay one scenario and return the resulting record as a plain dump.""" + recorder = EventRecorder() agent = _agent_with_steps(scenario.steps) agent.working_directory = pathlib.Path(working_dir) # Neutralize the orphan poll loop's real 5s sleeps. `d_orphaned_tool` @@ -144,8 +149,8 @@ async def run_antigravity_scenario(scenario: AntigravityScenario, working_dir: s # up to 120 cycles, i.e. ten minutes of wall clock in a unit test. The # loop's LOGIC is what the scenario records; the waiting is not. with patch.object(antigravity_agent.asyncio, "sleep", _no_sleep): - record = await agent.communicate("do it") - return record.model_dump(mode="json") + record = (await agent.communicate("do it", iteration=1, stream_callback=recorder)).record + return record.model_dump(mode="json"), recorder.events def _build_catalogue() -> list[AntigravityScenario]: @@ -273,6 +278,26 @@ def _build_catalogue() -> list[AntigravityScenario]: ) ) + # (f) the user's prompt arrives as a TEXT_RESPONSE Step with source USER (captured + # live, 2026-09-16). It is not the model talking to the user, so it must never + # become assistant text or `agent_output`; the model's reply after it must. + scenarios.append( + AntigravityScenario( + name="f_user_prompt_step", + steps=[ + _step("TEXT_RESPONSE", "DONE", source="USER", target="UNKNOWN", content="do it"), + _step( + "TEXT_RESPONSE", + "DONE", + content="DONE.", + content_delta="DONE.", + complete=True, + usage=_usage(100, 0, 5, 0), + ), + ], + ) + ) + return scenarios diff --git a/tests/_fixtures/golden_streams/claude_fixtures.py b/tests/_fixtures/golden_streams/claude_fixtures.py index 2340bc27..d1ed88bd 100644 --- a/tests/_fixtures/golden_streams/claude_fixtures.py +++ b/tests/_fixtures/golden_streams/claude_fixtures.py @@ -19,6 +19,9 @@ import coder_eval.agents.claude_code_agent as claude_module from coder_eval.models import AgentKind, parse_agent_config +from coder_eval.streaming import events as protocol +from coder_eval.streaming.events import AgentEndStatus +from tests._fixtures.golden_streams._recorder import EventRecorder ClaudeCodeAgent = claude_module.ClaudeCodeAgent @@ -178,7 +181,7 @@ class ClaudeScenario: name: str build_query: Callable[[], Callable[..., Any]] timeout: float | None = None - expects: type[BaseException] | None = None + expects: AgentEndStatus | None = None # When set, ``ClaudeCodeAgent._timed_out`` is patched to this constant so the # timeout-vs-crash classification is deterministic without real wall-clock. timed_out: bool | None = None @@ -362,7 +365,7 @@ def _scenario_g() -> ClaudeScenario: return ClaudeScenario( name="g_crash_format_placeholder", build_query=lambda: _raising_query(events, RuntimeError("crash after poison")), - expects=None, # set below to AgentCrashError + expects=None, # set below to CRASHED ) @@ -377,7 +380,7 @@ def _scenario_h1() -> ClaudeScenario: def _scenario_h2() -> ClaudeScenario: - """Non-timeout ProcessError -> AgentCrashError.""" + """Non-timeout ProcessError -> a CRASHED outcome.""" return ClaudeScenario( name="h2_process_error_crash", build_query=lambda: _raising_query([], ProcessError("boom", exit_code=1, stderr="bad config")), @@ -385,7 +388,6 @@ def _scenario_h2() -> ClaudeScenario: def _build_catalogue() -> list[ClaudeScenario]: - from coder_eval.errors import AgentCrashError, TurnTimeoutError scenarios = [ _scenario_a(), @@ -397,19 +399,19 @@ def _build_catalogue() -> list[ClaudeScenario]: ] g = _scenario_g() - g.expects = AgentCrashError + g.expects = AgentEndStatus.CRASHED scenarios.append(g) h1 = _scenario_h1() - h1.expects = TurnTimeoutError + h1.expects = AgentEndStatus.TIMEOUT scenarios.append(h1) h2 = _scenario_h2() - h2.expects = AgentCrashError + h2.expects = AgentEndStatus.CRASHED scenarios.append(h2) i = _build_deadline_break_scenario() - i.expects = TurnTimeoutError + i.expects = AgentEndStatus.TIMEOUT scenarios.append(i) return scenarios @@ -432,14 +434,16 @@ def _patches(scenario: ClaudeScenario) -> Iterator[Any]: yield patch.object(claude_module.time, "monotonic", scenario.monotonic) -async def run_claude_scenario(scenario: ClaudeScenario, working_dir: str) -> dict[str, Any]: - """Run ``scenario`` and return the ``TurnRecord``/``pending_turn`` model_dump. +async def run_claude_scenario( + scenario: ClaudeScenario, working_dir: str +) -> tuple[dict[str, Any], list[protocol.StreamEvent]]: + """Run ``scenario`` and return its outcome record's model_dump and the events. Raises ``AssertionError`` if a crash/timeout scenario fails to raise its expected exception (so a refactor that silently swallows the failure is caught). """ - import pytest + recorder = EventRecorder() config = parse_agent_config(type=AgentKind.CLAUDE_CODE, permission_mode="acceptEdits") agent = ClaudeCodeAgent(config) @@ -448,12 +452,11 @@ async def run_claude_scenario(scenario: ClaudeScenario, working_dir: str) -> dic with contextlib.ExitStack() as stack: for ctx in _patches(scenario): stack.enter_context(ctx) - if scenario.expects is not None: - with pytest.raises(scenario.expects): - await agent.communicate(scenario.prompt, timeout=scenario.timeout) - record = agent.pending_turn - assert record is not None, f"{scenario.name}: pending_turn was not set on the failure path" - else: - record = await agent.communicate(scenario.prompt, timeout=scenario.timeout) - - return record.model_dump(mode="json") + outcome = await agent.communicate( + scenario.prompt, iteration=1, timeout=scenario.timeout, stream_callback=recorder + ) + expected = scenario.expects or AgentEndStatus.COMPLETED + assert outcome.status is expected, f"{scenario.name}: ended {outcome.status}, expected {expected}" + record = outcome.record + + return record.model_dump(mode="json"), recorder.events diff --git a/tests/_fixtures/golden_streams/codex_fixtures.py b/tests/_fixtures/golden_streams/codex_fixtures.py index e0a157fd..6747b577 100644 --- a/tests/_fixtures/golden_streams/codex_fixtures.py +++ b/tests/_fixtures/golden_streams/codex_fixtures.py @@ -23,6 +23,8 @@ from coder_eval.agents.codex_agent import CodexAgent from coder_eval.models import AgentKind, parse_agent_config +from coder_eval.streaming.events import AgentEndStatus, StreamEvent +from tests._fixtures.golden_streams._recorder import EventRecorder CODEX_MODEL = "gpt-5-codex" @@ -170,11 +172,10 @@ def _collab( class CodexScenario: name: str notifications: list[Any] - expects: type[BaseException] | None = None + expects: AgentEndStatus | None = None def _build_catalogue() -> list[CodexScenario]: - from coder_eval.errors import AgentCrashError scenarios: list[CodexScenario] = [] @@ -221,21 +222,13 @@ def _build_catalogue() -> list[CodexScenario]: CodexScenario( name="c_reasoning_placeholder", notifications=[ - # Real bounds, and they are load-bearing rather than decorative: - # with none, `_flush_message` takes `_ms_to_dt(None)` for BOTH - # ends, which is two adjacent `datetime.now()` reads. Those - # collide at microsecond resolution often enough that this - # scenario failed `assert_timing_captured`'s - # `completed_at > started_at` roughly one run in twenty under - # parallel load, naming a different scenario each time. - _item("item/completed", _reasoning(text=""), started_at_ms=_T0_MS, completed_at_ms=_T0_MS + 40), + # Real bounds: without a start stamp the generation is unmeasured. + # The SDK carries `started_at_ms` only on item/started. + _item("item/started", _reasoning(text=""), started_at_ms=_T0_MS), + _item("item/completed", _reasoning(text=""), completed_at_ms=_T0_MS + 40), _delta("final answer"), - _item( - "item/completed", - _agent_message("final answer"), - started_at_ms=_T0_MS + 40, - completed_at_ms=_T0_MS + 300, - ), + _item("item/started", _agent_message("final answer"), started_at_ms=_T0_MS + 40), + _item("item/completed", _agent_message("final answer"), completed_at_ms=_T0_MS + 300), _token_usage(inp=100, out=50, cached=8, reasoning=20), _turn_completed(), ], @@ -307,15 +300,11 @@ def _build_catalogue() -> list[CodexScenario]: notifications=[ _delta("partial"), # Bounded for the same reason as (c) above. - _item( - "item/completed", - _agent_message("partial"), - started_at_ms=_T0_MS, - completed_at_ms=_T0_MS + 200, - ), + _item("item/started", _agent_message("partial"), started_at_ms=_T0_MS), + _item("item/completed", _agent_message("partial"), completed_at_ms=_T0_MS + 200), _token_usage(inp=100, out=40, cached=8), ], - expects=AgentCrashError, + expects=AgentEndStatus.CRASHED, ) ) @@ -378,9 +367,9 @@ def _rebase_notifications(notifications: list[Any]) -> list[Any]: return rebased -async def run_codex_scenario(scenario: CodexScenario, working_dir: str) -> dict[str, Any]: - """Run ``scenario`` with fakes and return the TurnRecord/pending_turn dump.""" - import pytest +async def run_codex_scenario(scenario: CodexScenario, working_dir: str) -> tuple[dict[str, Any], list[StreamEvent]]: + """Run ``scenario`` with fakes and return the outcome record's dump and the events.""" + recorder = EventRecorder() config = parse_agent_config(type=AgentKind.CODEX, model=CODEX_MODEL) agent = CodexAgent(config) @@ -391,12 +380,9 @@ async def run_codex_scenario(scenario: CodexScenario, working_dir: str) -> dict[ # Point CODEX_HOME at a sessions-less dir so sub-agent rollout recovery # short-circuits instead of polling the real ~/.codex. with patch.dict(os.environ, {"CODEX_HOME": working_dir}): - if scenario.expects is not None: - with pytest.raises(scenario.expects): - await agent.communicate("do it") - record = agent.pending_turn - assert record is not None, f"{scenario.name}: pending_turn was not set on the failure path" - else: - record = await agent.communicate("do it") - - return record.model_dump(mode="json") + outcome = await agent.communicate("do it", iteration=1, stream_callback=recorder) + expected = scenario.expects or AgentEndStatus.COMPLETED + assert outcome.status is expected, f"{scenario.name}: ended {outcome.status}, expected {expected}" + record = outcome.record + + return record.model_dump(mode="json"), recorder.events diff --git a/tests/_fixtures/golden_streams/expected/antigravity_a_single_text_turn.json b/tests/_fixtures/golden_streams/expected/antigravity_a_single_text_turn.json index af711bd0..055f20af 100644 --- a/tests/_fixtures/golden_streams/expected/antigravity_a_single_text_turn.json +++ b/tests/_fixtures/golden_streams/expected/antigravity_a_single_text_turn.json @@ -39,7 +39,12 @@ ], "model_used": "gemini-3.5-flash", "num_turns": 1, - "result_summary": null, + "result_summary": { + "is_error": false, + "result": "All done.", + "stop_reason": null, + "subtype": "completed" + }, "timestamp": "", "token_usage": { "cache_creation_input_tokens": 0, diff --git a/tests/_fixtures/golden_streams/expected/antigravity_b_tool_call_resolved.json b/tests/_fixtures/golden_streams/expected/antigravity_b_tool_call_resolved.json index 9b697ba3..d6fe5a1e 100644 --- a/tests/_fixtures/golden_streams/expected/antigravity_b_tool_call_resolved.json +++ b/tests/_fixtures/golden_streams/expected/antigravity_b_tool_call_resolved.json @@ -3,7 +3,7 @@ "assistant_turn_count": 1, "commands": [ { - "assistant_turn_index": null, + "assistant_turn_index": 0, "duration_ms": "", "error_message": null, "execution_completed_at": "", @@ -70,7 +70,12 @@ ], "model_used": "gemini-3.5-flash", "num_turns": 1, - "result_summary": null, + "result_summary": { + "is_error": false, + "result": "done", + "stop_reason": null, + "subtype": "completed" + }, "timestamp": "", "token_usage": { "cache_creation_input_tokens": 0, diff --git a/tests/_fixtures/golden_streams/expected/antigravity_c_thinking_and_tool_same_generation.json b/tests/_fixtures/golden_streams/expected/antigravity_c_thinking_and_tool_same_generation.json index 0ddd04e5..c3b1175a 100644 --- a/tests/_fixtures/golden_streams/expected/antigravity_c_thinking_and_tool_same_generation.json +++ b/tests/_fixtures/golden_streams/expected/antigravity_c_thinking_and_tool_same_generation.json @@ -3,7 +3,7 @@ "assistant_turn_count": 1, "commands": [ { - "assistant_turn_index": null, + "assistant_turn_index": 0, "duration_ms": "", "error_message": null, "execution_completed_at": "", @@ -79,7 +79,12 @@ ], "model_used": "gemini-3.5-flash", "num_turns": 1, - "result_summary": null, + "result_summary": { + "is_error": false, + "result": "read it", + "stop_reason": null, + "subtype": "completed" + }, "timestamp": "", "token_usage": { "cache_creation_input_tokens": 0, diff --git a/tests/_fixtures/golden_streams/expected/antigravity_d_orphaned_tool.json b/tests/_fixtures/golden_streams/expected/antigravity_d_orphaned_tool.json index 48cb5dac..7d96767d 100644 --- a/tests/_fixtures/golden_streams/expected/antigravity_d_orphaned_tool.json +++ b/tests/_fixtures/golden_streams/expected/antigravity_d_orphaned_tool.json @@ -6,7 +6,7 @@ "assistant_turn_index": null, "duration_ms": null, "error_message": null, - "execution_completed_at": "", + "execution_completed_at": null, "execution_started_at": "", "generation_completed_at": null, "parameters": { @@ -59,7 +59,12 @@ ], "model_used": "gemini-3.5-flash", "num_turns": 1, - "result_summary": null, + "result_summary": { + "is_error": false, + "result": "backgrounded", + "stop_reason": null, + "subtype": "completed" + }, "timestamp": "", "token_usage": { "cache_creation_input_tokens": 0, @@ -70,6 +75,6 @@ "uncached_input_tokens": 90 }, "tool_calls_exhausted": false, - "tool_union_ms": "", + "tool_union_ms": null, "user_input": "do it" } diff --git a/tests/_fixtures/golden_streams/expected/antigravity_e_multi_generation.json b/tests/_fixtures/golden_streams/expected/antigravity_e_multi_generation.json index 8a5f8253..58111780 100644 --- a/tests/_fixtures/golden_streams/expected/antigravity_e_multi_generation.json +++ b/tests/_fixtures/golden_streams/expected/antigravity_e_multi_generation.json @@ -93,7 +93,12 @@ ], "model_used": "gemini-3.5-flash", "num_turns": 3, - "result_summary": null, + "result_summary": { + "is_error": false, + "result": "third", + "stop_reason": null, + "subtype": "completed" + }, "timestamp": "", "token_usage": { "cache_creation_input_tokens": 0, diff --git a/tests/_fixtures/golden_streams/expected/antigravity_f_user_prompt_step.json b/tests/_fixtures/golden_streams/expected/antigravity_f_user_prompt_step.json new file mode 100644 index 00000000..906a0ba7 --- /dev/null +++ b/tests/_fixtures/golden_streams/expected/antigravity_f_user_prompt_step.json @@ -0,0 +1,60 @@ +{ + "agent_output": "DONE.", + "assistant_turn_count": 1, + "commands": [], + "crash_reason": null, + "crashed": false, + "duration_seconds": "", + "harness_startup_ms": "", + "harness_teardown_ms": "", + "iteration": 1, + "messages": [ + { + "cache_creation_tokens": 0, + "cache_read_tokens": 0, + "completed_at": "", + "content_blocks": [ + { + "block_type": "text", + "is_error": false, + "sequence": 0, + "signature": null, + "text": "DONE.", + "thinking": null, + "tool_use_id": null + } + ], + "generation_duration_ms": "", + "input_tokens": 100, + "message_id": "antigravity-1-msg-0", + "model": "gemini-3.5-flash", + "output_tokens": 5, + "parent_tool_use_id": null, + "reasoning_tokens": 0, + "role": "assistant", + "started_at": "", + "stop_reason": null, + "tool_use_ids": [] + } + ], + "model_used": "gemini-3.5-flash", + "num_turns": 1, + "result_summary": { + "is_error": false, + "result": "DONE.", + "stop_reason": null, + "subtype": "completed" + }, + "timestamp": "", + "token_usage": { + "cache_creation_input_tokens": 0, + "cache_read_input_tokens": 0, + "input_tokens": 100, + "output_tokens": 5, + "total_cost_usd": "", + "uncached_input_tokens": 100 + }, + "tool_calls_exhausted": false, + "tool_union_ms": null, + "user_input": "do it" +} diff --git a/tests/_fixtures/golden_streams/expected/claude_f_orphaned_tool.json b/tests/_fixtures/golden_streams/expected/claude_f_orphaned_tool.json index ab23e17c..6642bec5 100644 --- a/tests/_fixtures/golden_streams/expected/claude_f_orphaned_tool.json +++ b/tests/_fixtures/golden_streams/expected/claude_f_orphaned_tool.json @@ -7,7 +7,7 @@ "duration_ms": null, "error_message": null, "execution_completed_at": null, - "execution_started_at": null, + "execution_started_at": "", "generation_completed_at": "", "parameters": { "content": "code", diff --git a/tests/_fixtures/golden_streams/expected/codex_a_agent_message_only.json b/tests/_fixtures/golden_streams/expected/codex_a_agent_message_only.json index 2b610394..b32cc7c2 100644 --- a/tests/_fixtures/golden_streams/expected/codex_a_agent_message_only.json +++ b/tests/_fixtures/golden_streams/expected/codex_a_agent_message_only.json @@ -5,8 +5,8 @@ "crash_reason": null, "crashed": false, "duration_seconds": "", - "harness_startup_ms": "", - "harness_teardown_ms": "", + "harness_startup_ms": null, + "harness_teardown_ms": null, "iteration": 1, "messages": [ { @@ -24,7 +24,7 @@ "tool_use_id": null } ], - "generation_duration_ms": "", + "generation_duration_ms": null, "input_tokens": 92, "message_id": "codex-1-msg-0", "model": "gpt-5-codex", @@ -39,7 +39,12 @@ ], "model_used": "gpt-5-codex", "num_turns": 1, - "result_summary": null, + "result_summary": { + "is_error": false, + "result": "Hello world", + "stop_reason": null, + "subtype": "completed" + }, "timestamp": "", "token_usage": { "cache_creation_input_tokens": 0, diff --git a/tests/_fixtures/golden_streams/expected/codex_b_command_execution.json b/tests/_fixtures/golden_streams/expected/codex_b_command_execution.json index 1d172aff..70d81f21 100644 --- a/tests/_fixtures/golden_streams/expected/codex_b_command_execution.json +++ b/tests/_fixtures/golden_streams/expected/codex_b_command_execution.json @@ -3,7 +3,7 @@ "assistant_turn_count": 1, "commands": [ { - "assistant_turn_index": null, + "assistant_turn_index": 0, "duration_ms": "", "error_message": null, "execution_completed_at": "", @@ -70,7 +70,12 @@ ], "model_used": "gpt-5-codex", "num_turns": 1, - "result_summary": null, + "result_summary": { + "is_error": false, + "result": "done", + "stop_reason": null, + "subtype": "completed" + }, "timestamp": "", "token_usage": { "cache_creation_input_tokens": 0, diff --git a/tests/_fixtures/golden_streams/expected/codex_c_reasoning_placeholder.json b/tests/_fixtures/golden_streams/expected/codex_c_reasoning_placeholder.json index 3f76c93a..cb52aa08 100644 --- a/tests/_fixtures/golden_streams/expected/codex_c_reasoning_placeholder.json +++ b/tests/_fixtures/golden_streams/expected/codex_c_reasoning_placeholder.json @@ -66,7 +66,12 @@ ], "model_used": "gpt-5-codex", "num_turns": 1, - "result_summary": null, + "result_summary": { + "is_error": false, + "result": "final answer", + "stop_reason": null, + "subtype": "completed" + }, "timestamp": "", "token_usage": { "cache_creation_input_tokens": 0, diff --git a/tests/_fixtures/golden_streams/expected/codex_d_cross_flush_is_error.json b/tests/_fixtures/golden_streams/expected/codex_d_cross_flush_is_error.json index 8371a9a8..dc64a8a2 100644 --- a/tests/_fixtures/golden_streams/expected/codex_d_cross_flush_is_error.json +++ b/tests/_fixtures/golden_streams/expected/codex_d_cross_flush_is_error.json @@ -3,7 +3,7 @@ "assistant_turn_count": 1, "commands": [ { - "assistant_turn_index": null, + "assistant_turn_index": 0, "duration_ms": "", "error_message": "hi\n", "execution_completed_at": "", @@ -61,7 +61,12 @@ ], "model_used": "gpt-5-codex", "num_turns": 1, - "result_summary": null, + "result_summary": { + "is_error": false, + "result": null, + "stop_reason": null, + "subtype": "completed" + }, "timestamp": "", "token_usage": { "cache_creation_input_tokens": 0, diff --git a/tests/_fixtures/golden_streams/expected/codex_e_orphan_tool.json b/tests/_fixtures/golden_streams/expected/codex_e_orphan_tool.json index 663e1e36..498de75b 100644 --- a/tests/_fixtures/golden_streams/expected/codex_e_orphan_tool.json +++ b/tests/_fixtures/golden_streams/expected/codex_e_orphan_tool.json @@ -3,7 +3,7 @@ "assistant_turn_count": 1, "commands": [ { - "assistant_turn_index": null, + "assistant_turn_index": 0, "duration_ms": null, "error_message": null, "execution_completed_at": null, @@ -61,7 +61,12 @@ ], "model_used": "gpt-5-codex", "num_turns": 1, - "result_summary": null, + "result_summary": { + "is_error": false, + "result": null, + "stop_reason": null, + "subtype": "completed" + }, "timestamp": "", "token_usage": null, "tool_calls_exhausted": false, diff --git a/tests/_fixtures/golden_streams/expected/codex_f_collab_fallback.json b/tests/_fixtures/golden_streams/expected/codex_f_collab_fallback.json index 7ca68b57..f80a1ea7 100644 --- a/tests/_fixtures/golden_streams/expected/codex_f_collab_fallback.json +++ b/tests/_fixtures/golden_streams/expected/codex_f_collab_fallback.json @@ -3,7 +3,7 @@ "assistant_turn_count": 1, "commands": [ { - "assistant_turn_index": null, + "assistant_turn_index": 0, "duration_ms": "", "error_message": null, "execution_completed_at": "", @@ -24,7 +24,7 @@ "tool_name": "Agent" }, { - "assistant_turn_index": null, + "assistant_turn_index": 0, "duration_ms": "", "error_message": null, "execution_completed_at": "", @@ -106,7 +106,7 @@ ], "generation_duration_ms": null, "input_tokens": 0, - "message_id": "codex-1-subagent-1", + "message_id": "codex-1-subagent-0", "model": "gpt-5.5", "output_tokens": 0, "parent_tool_use_id": "call_spawn", @@ -119,7 +119,12 @@ ], "model_used": "gpt-5-codex", "num_turns": 1, - "result_summary": null, + "result_summary": { + "is_error": false, + "result": null, + "stop_reason": null, + "subtype": "completed" + }, "timestamp": "", "token_usage": null, "tool_calls_exhausted": false, diff --git a/tests/_fixtures/golden_streams/expected/codex_g_items_rebuild.json b/tests/_fixtures/golden_streams/expected/codex_g_items_rebuild.json index 7a6f5df8..3861ee95 100644 --- a/tests/_fixtures/golden_streams/expected/codex_g_items_rebuild.json +++ b/tests/_fixtures/golden_streams/expected/codex_g_items_rebuild.json @@ -39,7 +39,12 @@ ], "model_used": "gpt-5-codex", "num_turns": 1, - "result_summary": null, + "result_summary": { + "is_error": false, + "result": "rebuilt from items", + "stop_reason": null, + "subtype": "completed" + }, "timestamp": "", "token_usage": null, "tool_calls_exhausted": false, diff --git a/tests/_fixtures/golden_streams/expected/codex_h_no_turn_completed_crash.json b/tests/_fixtures/golden_streams/expected/codex_h_no_turn_completed_crash.json index c286da2b..83072e91 100644 --- a/tests/_fixtures/golden_streams/expected/codex_h_no_turn_completed_crash.json +++ b/tests/_fixtures/golden_streams/expected/codex_h_no_turn_completed_crash.json @@ -1,5 +1,5 @@ { - "agent_output": "", + "agent_output": "partial", "assistant_turn_count": 1, "commands": [], "crash_reason": "Codex turn failed: Turn did not complete (no turn/completed notification received)", diff --git a/tests/_fixtures/golden_streams/expected/opencode_a_single_text_turn.json b/tests/_fixtures/golden_streams/expected/opencode_a_single_text_turn.json index 594b2e72..2b1d35fa 100644 --- a/tests/_fixtures/golden_streams/expected/opencode_a_single_text_turn.json +++ b/tests/_fixtures/golden_streams/expected/opencode_a_single_text_turn.json @@ -41,7 +41,7 @@ "num_turns": 1, "result_summary": { "is_error": false, - "result": null, + "result": "All done.", "stop_reason": "stop", "subtype": "completed" }, diff --git a/tests/_fixtures/golden_streams/expected/opencode_b_tool_call_resolved.json b/tests/_fixtures/golden_streams/expected/opencode_b_tool_call_resolved.json index 0b4faf28..aa5f6b5a 100644 --- a/tests/_fixtures/golden_streams/expected/opencode_b_tool_call_resolved.json +++ b/tests/_fixtures/golden_streams/expected/opencode_b_tool_call_resolved.json @@ -3,7 +3,7 @@ "assistant_turn_count": 2, "commands": [ { - "assistant_turn_index": 1, + "assistant_turn_index": 0, "duration_ms": "", "error_message": null, "execution_completed_at": "", @@ -16,7 +16,7 @@ "result_status": "success", "result_summary": "print('hi')", "result_tokens": 3, - "sequence_number": 1, + "sequence_number": 0, "timestamp": "", "tool_id": "call_1", "tool_name": "Read" @@ -90,7 +90,7 @@ "num_turns": 2, "result_summary": { "is_error": false, - "result": null, + "result": "Created the file.", "stop_reason": "stop", "subtype": "completed" }, diff --git a/tests/_fixtures/golden_streams/expected/opencode_c_multi_step_tiling.json b/tests/_fixtures/golden_streams/expected/opencode_c_multi_step_tiling.json index c486f259..d237c144 100644 --- a/tests/_fixtures/golden_streams/expected/opencode_c_multi_step_tiling.json +++ b/tests/_fixtures/golden_streams/expected/opencode_c_multi_step_tiling.json @@ -3,7 +3,7 @@ "assistant_turn_count": 2, "commands": [ { - "assistant_turn_index": 1, + "assistant_turn_index": 0, "duration_ms": "", "error_message": null, "execution_completed_at": "", @@ -16,7 +16,7 @@ "result_status": "success", "result_summary": "main.py", "result_tokens": 2, - "sequence_number": 1, + "sequence_number": 0, "timestamp": "", "tool_id": "call_1", "tool_name": "Bash" @@ -90,7 +90,7 @@ "num_turns": 2, "result_summary": { "is_error": false, - "result": null, + "result": "Listed it.", "stop_reason": "stop", "subtype": "completed" }, diff --git a/tests/_fixtures/golden_streams/expected/opencode_d_orphaned_tool.json b/tests/_fixtures/golden_streams/expected/opencode_d_orphaned_tool.json index 9a2d261f..1422c542 100644 --- a/tests/_fixtures/golden_streams/expected/opencode_d_orphaned_tool.json +++ b/tests/_fixtures/golden_streams/expected/opencode_d_orphaned_tool.json @@ -3,10 +3,10 @@ "assistant_turn_count": 1, "commands": [ { - "assistant_turn_index": 1, + "assistant_turn_index": 0, "duration_ms": null, - "error_message": "no result observed", - "execution_completed_at": "", + "error_message": null, + "execution_completed_at": null, "execution_started_at": null, "generation_completed_at": null, "parameters": { @@ -16,7 +16,7 @@ "result_status": "unknown", "result_summary": null, "result_tokens": 0, - "sequence_number": 1, + "sequence_number": 0, "timestamp": "", "tool_id": "call_1", "tool_name": "Bash" diff --git a/tests/_fixtures/golden_streams/expected/opencode_e_error_after_generation.json b/tests/_fixtures/golden_streams/expected/opencode_e_error_after_generation.json index 65a062a5..f89b5a28 100644 --- a/tests/_fixtures/golden_streams/expected/opencode_e_error_after_generation.json +++ b/tests/_fixtures/golden_streams/expected/opencode_e_error_after_generation.json @@ -39,12 +39,7 @@ ], "model_used": "deepseek/deepseek-v4-pro", "num_turns": 1, - "result_summary": { - "is_error": true, - "result": "OpenCode error: 401 from the provider", - "stop_reason": "stop", - "subtype": "crashed" - }, + "result_summary": null, "timestamp": "", "token_usage": { "cache_creation_input_tokens": 0, diff --git a/tests/_fixtures/golden_streams/expected/opencode_f_captured_stream.json b/tests/_fixtures/golden_streams/expected/opencode_f_captured_stream.json new file mode 100644 index 00000000..63709697 --- /dev/null +++ b/tests/_fixtures/golden_streams/expected/opencode_f_captured_stream.json @@ -0,0 +1,148 @@ +{ + "agent_output": "I'll help you create the file and list the directory.DONE", + "assistant_turn_count": 2, + "commands": [ + { + "assistant_turn_index": 0, + "duration_ms": "", + "error_message": null, + "execution_completed_at": "", + "execution_started_at": "", + "generation_completed_at": null, + "parameters": { + "content": "hi", + "file_path": "/work/hello.txt" + }, + "result_data": null, + "result_status": "success", + "result_summary": "Wrote file successfully.", + "result_tokens": 6, + "sequence_number": 0, + "timestamp": "", + "tool_id": "toolu_016ZZZbGJz51bwQrFrxN4Sv2", + "tool_name": "Write" + }, + { + "assistant_turn_index": 0, + "duration_ms": "", + "error_message": null, + "execution_completed_at": "", + "execution_started_at": "", + "generation_completed_at": null, + "parameters": { + "command": "ls -la /work/" + }, + "result_data": null, + "result_status": "success", + "result_summary": "total 8\ndrwx------ 3 user staff 96 Sep 16 15:57 .\ndrwx------@ 39029 user staff 1248928 Sep 16 15:57 ..\n-rw-r--r-- 1 user staff 2 Sep 16 15:57 hello.txt\n", + "result_tokens": 45, + "sequence_number": 1, + "timestamp": "", + "tool_id": "toolu_01DYyaQT59GsS3Qyj9smkbrb", + "tool_name": "Bash" + } + ], + "crash_reason": null, + "crashed": false, + "duration_seconds": "", + "harness_startup_ms": "", + "harness_teardown_ms": "", + "iteration": 1, + "messages": [ + { + "cache_creation_tokens": 16336, + "cache_read_tokens": 0, + "completed_at": "", + "content_blocks": [ + { + "block_type": "text", + "is_error": false, + "sequence": 0, + "signature": null, + "text": "I'll help you create the file and list the directory.", + "thinking": null, + "tool_use_id": null + }, + { + "block_type": "tool_use", + "is_error": false, + "sequence": 1, + "signature": null, + "text": null, + "thinking": null, + "tool_use_id": "toolu_016ZZZbGJz51bwQrFrxN4Sv2" + }, + { + "block_type": "tool_use", + "is_error": false, + "sequence": 2, + "signature": null, + "text": null, + "thinking": null, + "tool_use_id": "toolu_01DYyaQT59GsS3Qyj9smkbrb" + } + ], + "generation_duration_ms": "", + "input_tokens": 3, + "message_id": "msg_0ac700f2b001SSGFoj9fVWpcYA", + "model": "deepseek/deepseek-v4-pro", + "output_tokens": 210, + "parent_tool_use_id": null, + "reasoning_tokens": 0, + "role": "assistant", + "started_at": "", + "stop_reason": "tool-calls", + "tool_use_ids": [ + "toolu_016ZZZbGJz51bwQrFrxN4Sv2", + "toolu_01DYyaQT59GsS3Qyj9smkbrb" + ] + }, + { + "cache_creation_tokens": 355, + "cache_read_tokens": 16336, + "completed_at": "", + "content_blocks": [ + { + "block_type": "text", + "is_error": false, + "sequence": 0, + "signature": null, + "text": "DONE", + "thinking": null, + "tool_use_id": null + } + ], + "generation_duration_ms": "", + "input_tokens": 7, + "message_id": "msg_0ac701b6a001f4VpkH4NL51mAe", + "model": "deepseek/deepseek-v4-pro", + "output_tokens": 5, + "parent_tool_use_id": null, + "reasoning_tokens": 0, + "role": "assistant", + "started_at": "", + "stop_reason": "stop", + "tool_use_ids": [] + } + ], + "model_used": "deepseek/deepseek-v4-pro", + "num_turns": 2, + "result_summary": { + "is_error": false, + "result": "DONE", + "stop_reason": "stop", + "subtype": "completed" + }, + "timestamp": "", + "token_usage": { + "cache_creation_input_tokens": 16691, + "cache_read_input_tokens": 16336, + "input_tokens": 33037, + "output_tokens": 215, + "total_cost_usd": "", + "uncached_input_tokens": 10 + }, + "tool_calls_exhausted": false, + "tool_union_ms": "", + "user_input": "do it" +} diff --git a/tests/_fixtures/golden_streams/expected/pi_a_single_text_turn.json b/tests/_fixtures/golden_streams/expected/pi_a_single_text_turn.json index 94ef4899..af1cce36 100644 --- a/tests/_fixtures/golden_streams/expected/pi_a_single_text_turn.json +++ b/tests/_fixtures/golden_streams/expected/pi_a_single_text_turn.json @@ -41,7 +41,7 @@ "num_turns": 1, "result_summary": { "is_error": false, - "result": null, + "result": "All done.", "stop_reason": "stop", "subtype": "completed" }, diff --git a/tests/_fixtures/golden_streams/expected/pi_b_tool_call_resolved.json b/tests/_fixtures/golden_streams/expected/pi_b_tool_call_resolved.json index da349e69..9d13e456 100644 --- a/tests/_fixtures/golden_streams/expected/pi_b_tool_call_resolved.json +++ b/tests/_fixtures/golden_streams/expected/pi_b_tool_call_resolved.json @@ -3,7 +3,7 @@ "assistant_turn_count": 3, "commands": [ { - "assistant_turn_index": 1, + "assistant_turn_index": 0, "duration_ms": "", "error_message": null, "execution_completed_at": "", @@ -17,13 +17,13 @@ "result_status": "success", "result_summary": "Successfully wrote 2 bytes to hello.txt", "result_tokens": 10, - "sequence_number": 1, + "sequence_number": 0, "timestamp": "", "tool_id": "write:0", "tool_name": "Write" }, { - "assistant_turn_index": 2, + "assistant_turn_index": 1, "duration_ms": "", "error_message": null, "execution_completed_at": "", @@ -36,7 +36,7 @@ "result_status": "success", "result_summary": "hi", "result_tokens": 1, - "sequence_number": 2, + "sequence_number": 1, "timestamp": "", "tool_id": "read:1", "tool_name": "Read" @@ -139,7 +139,7 @@ "num_turns": 3, "result_summary": { "is_error": false, - "result": null, + "result": "Created `hello.txt` and read it back. Contents: **hi**", "stop_reason": "stop", "subtype": "completed" }, diff --git a/tests/_fixtures/golden_streams/expected/pi_c_multi_turn_tiling.json b/tests/_fixtures/golden_streams/expected/pi_c_multi_turn_tiling.json index d70291ca..21db5cb1 100644 --- a/tests/_fixtures/golden_streams/expected/pi_c_multi_turn_tiling.json +++ b/tests/_fixtures/golden_streams/expected/pi_c_multi_turn_tiling.json @@ -3,7 +3,7 @@ "assistant_turn_count": 2, "commands": [ { - "assistant_turn_index": 1, + "assistant_turn_index": 0, "duration_ms": "", "error_message": null, "execution_completed_at": "", @@ -16,7 +16,7 @@ "result_status": "success", "result_summary": "main.py", "result_tokens": 2, - "sequence_number": 1, + "sequence_number": 0, "timestamp": "", "tool_id": "call_1", "tool_name": "Bash" @@ -90,7 +90,7 @@ "num_turns": 2, "result_summary": { "is_error": false, - "result": null, + "result": "Listed it.", "stop_reason": "stop", "subtype": "completed" }, diff --git a/tests/_fixtures/golden_streams/expected/pi_d_orphaned_tool.json b/tests/_fixtures/golden_streams/expected/pi_d_orphaned_tool.json index 04b07999..2fe522b7 100644 --- a/tests/_fixtures/golden_streams/expected/pi_d_orphaned_tool.json +++ b/tests/_fixtures/golden_streams/expected/pi_d_orphaned_tool.json @@ -3,9 +3,9 @@ "assistant_turn_count": 1, "commands": [ { - "assistant_turn_index": 1, + "assistant_turn_index": 0, "duration_ms": null, - "error_message": "no result observed", + "error_message": null, "execution_completed_at": null, "execution_started_at": "", "generation_completed_at": null, @@ -16,7 +16,7 @@ "result_status": "unknown", "result_summary": null, "result_tokens": 0, - "sequence_number": 1, + "sequence_number": 0, "timestamp": "", "tool_id": "call_1", "tool_name": "Bash" diff --git a/tests/_fixtures/golden_streams/expected/pi_e_error_after_generation.json b/tests/_fixtures/golden_streams/expected/pi_e_error_after_generation.json index 4bec8274..a7edec74 100644 --- a/tests/_fixtures/golden_streams/expected/pi_e_error_after_generation.json +++ b/tests/_fixtures/golden_streams/expected/pi_e_error_after_generation.json @@ -56,12 +56,7 @@ ], "model_used": "openrouter/moonshotai/kimi-k3", "num_turns": 2, - "result_summary": { - "is_error": true, - "result": "Pi error: provider returned 529 after 5 retries", - "stop_reason": "error", - "subtype": "crashed" - }, + "result_summary": null, "timestamp": "", "token_usage": { "cache_creation_input_tokens": 0, diff --git a/tests/_fixtures/golden_streams/opencode_fixtures.py b/tests/_fixtures/golden_streams/opencode_fixtures.py index 5f138147..8a01710c 100644 --- a/tests/_fixtures/golden_streams/opencode_fixtures.py +++ b/tests/_fixtures/golden_streams/opencode_fixtures.py @@ -26,12 +26,14 @@ import os from dataclasses import dataclass from datetime import datetime +from pathlib import Path from typing import Any from unittest.mock import patch from coder_eval.agents.opencode_agent import OpenCodeAgent -from coder_eval.errors import AgentCrashError from coder_eval.models import OpenCodeAgentConfig +from coder_eval.streaming.events import AgentEndStatus, StreamEvent +from tests._fixtures.golden_streams._recorder import EventRecorder SESSION = "ses_test123" @@ -49,10 +51,20 @@ _REPLAY_LEAD_MS = 2 -def _evt(event_type: str, part: dict[str, Any]) -> str: - """One CLI event line: payload under ``part``, sessionID on the envelope.""" +def _evt(event_type: str, part: dict[str, Any], *, at_ms: int = 0) -> str: + """One CLI event line: payload under ``part``; sessionID and the CLI's own stamp on the envelope. + + ``at_ms`` places the event on the recorded timeline: the envelope stamp is what + bounds a generation window on this harness, so a scenario that should measure + one gives its events increasing stamps. + """ return json.dumps( - {"type": event_type, "timestamp": _T0_MS, "sessionID": SESSION, "part": {"sessionID": SESSION, **part}} + { + "type": event_type, + "timestamp": _T0_MS + at_ms, + "sessionID": SESSION, + "part": {"sessionID": SESSION, **part}, + } ) @@ -80,6 +92,27 @@ def shift(node: Any) -> Any: _STAMP_KEYS = frozenset({"timestamp", "start", "end"}) +def _starting_at_t0(lines: list[str]) -> list[str]: + """Shift a captured stream's stamps so its first envelope ``timestamp`` is ``_T0_MS``.""" + first = json.loads(lines[0])["timestamp"] + + def shift(node: Any) -> Any: + if isinstance(node, dict): + return { + k: (v - first + _T0_MS if k in _STAMP_KEYS and isinstance(v, int) else shift(v)) + for k, v in node.items() + } + if isinstance(node, list): + return [shift(v) for v in node] + return node + + return [json.dumps(shift(json.loads(line))) for line in lines] + + +_CAPTURED = Path(__file__).resolve().parents[2] / "fixtures" / "opencode_happy_stream.jsonl" +CAPTURED_STREAM = _CAPTURED.read_text(encoding="utf-8").splitlines() + + def _tokens(inp: int, out: int, *, write: int = 0, read: int = 0, reasoning: int = 0) -> dict[str, Any]: """Token payload in the NESTED convention (total = input+output+reasoning, cache counted inside `input`); see TestTokenShapeIsObservable for the flat one.""" @@ -93,7 +126,7 @@ def _tokens(inp: int, out: int, *, write: int = 0, read: int = 0, reasoning: int HAPPY_STREAM = [ - _evt("step_start", {"id": "prt_1", "messageID": "msg_1", "type": "step-start"}), + _evt("step_start", {"id": "prt_1", "messageID": "msg_1", "type": "step-start"}, at_ms=1400), _evt( "tool_use", { @@ -109,6 +142,7 @@ def _tokens(inp: int, out: int, *, write: int = 0, read: int = 0, reasoning: int "time": {"start": 1786663018214, "end": 1786663018231}, }, }, + at_ms=1435, ), _evt( "step_finish", @@ -119,9 +153,10 @@ def _tokens(inp: int, out: int, *, write: int = 0, read: int = 0, reasoning: int "cost": 0.001, "tokens": _tokens(100, 20, write=5, read=10), }, + at_ms=1440, ), - _evt("step_start", {"id": "prt_4", "messageID": "msg_2", "type": "step-start"}), - _evt("text", {"id": "prt_5", "messageID": "msg_2", "type": "text", "text": "Created the file."}), + _evt("step_start", {"id": "prt_4", "messageID": "msg_2", "type": "step-start"}, at_ms=1450), + _evt("text", {"id": "prt_5", "messageID": "msg_2", "type": "text", "text": "Created the file."}, at_ms=1460), _evt( "step_finish", { @@ -131,6 +166,7 @@ def _tokens(inp: int, out: int, *, write: int = 0, read: int = 0, reasoning: int "cost": 0.002, "tokens": _tokens(50, 30, read=40, reasoning=7), }, + at_ms=1470, ), ] @@ -202,8 +238,8 @@ def _agent() -> OpenCodeAgent: class OpenCodeScenario: """One recorded CLI event stream. - ``expects`` names the exception a scenario is supposed to raise, and the - runner then snapshots ``pending_turn`` instead of the returned record — + ``expects`` names the failed end status a scenario is supposed to reach, and the + runner asserts that end status and snapshots the crashed record — the same knob ``ClaudeScenario`` carries, for the same reason: the partial a crash preserves is a real capture path, and one nobody was comparing against a snapshot on this harness. @@ -211,12 +247,14 @@ class OpenCodeScenario: name: str lines: list[str] - expects: type[BaseException] | None = None + expects: AgentEndStatus | None = None -async def run_opencode_scenario(scenario: OpenCodeScenario, working_dir: str) -> dict[str, Any]: +async def run_opencode_scenario( + scenario: OpenCodeScenario, working_dir: str +) -> tuple[dict[str, Any], list[StreamEvent]]: """Replay one scenario and return the resulting record as a plain dump.""" - import pytest + recorder = EventRecorder() proc = _FakeProcess(_rebase_lines(scenario.lines)) @@ -231,14 +269,11 @@ async def fake_exec(*_argv: str, **_kwargs: Any) -> _FakeProcess: patch.object(os, "killpg", lambda _pgid, _sig: None, create=True), ): await agent.start(working_dir) - if scenario.expects is not None: - with pytest.raises(scenario.expects): - await agent.communicate("do it") - record = agent.pending_turn - assert record is not None, f"{scenario.name}: pending_turn was not set on the failure path" - else: - record = await agent.communicate("do it") - return record.model_dump(mode="json") + outcome = await agent.communicate("do it", iteration=1, stream_callback=recorder) + expected = scenario.expects or AgentEndStatus.COMPLETED + assert outcome.status is expected, f"{scenario.name}: ended {outcome.status}, expected {expected}" + record = outcome.record + return record.model_dump(mode="json"), recorder.events def _build_catalogue() -> list[OpenCodeScenario]: @@ -249,8 +284,8 @@ def _build_catalogue() -> list[OpenCodeScenario]: OpenCodeScenario( name="a_single_text_turn", lines=[ - _evt("step_start", {"id": "prt_1", "messageID": "msg_1"}), - _evt("text", {"id": "prt_2", "messageID": "msg_1", "text": "All done."}), + _evt("step_start", {"id": "prt_1", "messageID": "msg_1"}, at_ms=0), + _evt("text", {"id": "prt_2", "messageID": "msg_1", "text": "All done."}, at_ms=1), _evt( "step_finish", { @@ -260,6 +295,7 @@ def _build_catalogue() -> list[OpenCodeScenario]: "cost": 0.001, "tokens": _tokens(100, 20), }, + at_ms=3, ), ], ) @@ -280,7 +316,7 @@ def _build_catalogue() -> list[OpenCodeScenario]: OpenCodeScenario( name="c_multi_step_tiling", lines=[ - _evt("step_start", {"id": "prt_1", "messageID": "msg_1"}), + _evt("step_start", {"id": "prt_1", "messageID": "msg_1"}, at_ms=0), _evt( "tool_use", { @@ -295,16 +331,19 @@ def _build_catalogue() -> list[OpenCodeScenario]: "time": {"start": _T0_MS, "end": _T0_MS + 5}, }, }, + at_ms=5, ), _evt( "step_finish", {"id": "prt_3", "messageID": "msg_1", "reason": "tool-calls", "tokens": _tokens(100, 20)}, + at_ms=6, ), - _evt("step_start", {"id": "prt_4", "messageID": "msg_2"}), - _evt("text", {"id": "prt_5", "messageID": "msg_2", "text": "Listed it."}), + _evt("step_start", {"id": "prt_4", "messageID": "msg_2"}, at_ms=8), + _evt("text", {"id": "prt_5", "messageID": "msg_2", "text": "Listed it."}, at_ms=9), _evt( "step_finish", {"id": "prt_6", "messageID": "msg_2", "reason": "stop", "tokens": _tokens(50, 30)}, + at_ms=12, ), ], ) @@ -323,7 +362,7 @@ def _build_catalogue() -> list[OpenCodeScenario]: OpenCodeScenario( name="d_orphaned_tool", lines=[ - _evt("step_start", {"id": "prt_1", "messageID": "msg_1"}), + _evt("step_start", {"id": "prt_1", "messageID": "msg_1"}, at_ms=0), _evt( "tool_use", { @@ -333,42 +372,52 @@ def _build_catalogue() -> list[OpenCodeScenario]: "callID": "call_1", "state": {"status": "pending", "input": {"command": "sleep 600"}}, }, + at_ms=1, ), - _evt("text", {"id": "prt_3", "messageID": "msg_1", "text": "Waiting."}), + _evt("text", {"id": "prt_3", "messageID": "msg_1", "text": "Waiting."}, at_ms=2), _evt( "step_finish", {"id": "prt_4", "messageID": "msg_1", "reason": "stop", "tokens": _tokens(100, 20)}, + at_ms=4, ), ], ) ) # (e) the CLI's own structured error AFTER a complete generation. `_settle_turn` - # crashes on it, and the partial `pending_turn` must still carry that + # crashes on it, and the crashed record must still carry that # generation and its head/tail — a crash does not un-measure what was # measured before it. scenarios.append( OpenCodeScenario( name="e_error_after_generation", lines=[ - _evt("step_start", {"id": "prt_1", "messageID": "msg_1"}), - _evt("text", {"id": "prt_2", "messageID": "msg_1", "text": "Starting."}), + _evt("step_start", {"id": "prt_1", "messageID": "msg_1"}, at_ms=0), + _evt("text", {"id": "prt_2", "messageID": "msg_1", "text": "Starting."}, at_ms=1), _evt( "step_finish", {"id": "prt_3", "messageID": "msg_1", "reason": "stop", "tokens": _tokens(100, 20)}, + at_ms=3, ), json.dumps( { "type": "error", + "timestamp": _T0_MS + 4, "sessionID": SESSION, "error": {"name": "ProviderAuthError", "data": {"message": "401 from the provider"}}, } ), ], - expects=AgentCrashError, + expects=AgentEndStatus.CRASHED, ) ) + # (f) a real stream captured from `opencode run --format json` (1.18.30, Haiku via + # OpenRouter): two steps, a `write` and a `bash` call, the CLI's own envelope and + # `state.time` stamps. Session id and paths are scrubbed. The stamps are moved so + # the stream starts at `_T0_MS`, which `_rebase_lines` puts on the replay clock. + scenarios.append(OpenCodeScenario(name="f_captured_stream", lines=_starting_at_t0(CAPTURED_STREAM))) + return scenarios diff --git a/tests/_fixtures/golden_streams/pi_fixtures.py b/tests/_fixtures/golden_streams/pi_fixtures.py index 2d26d7d6..c4e80f15 100644 --- a/tests/_fixtures/golden_streams/pi_fixtures.py +++ b/tests/_fixtures/golden_streams/pi_fixtures.py @@ -24,8 +24,9 @@ from unittest.mock import patch from coder_eval.agents.pi_agent import PiAgent -from coder_eval.errors import AgentCrashError from coder_eval.models import PiAgentConfig +from coder_eval.streaming.events import AgentEndStatus, StreamEvent +from tests._fixtures.golden_streams._recorder import EventRecorder # tests/_fixtures/golden_streams/ -> tests/fixtures/ (this module moved two @@ -191,8 +192,8 @@ def _agent() -> PiAgent: class PiScenario: """One recorded CLI event stream. - ``expects`` names the exception a scenario is supposed to raise, and the - runner then snapshots ``pending_turn`` instead of the returned record — + ``expects`` names the failed end status a scenario is supposed to reach, and the + runner asserts that end status and snapshots the crashed record — the same knob ``ClaudeScenario`` carries, for the same reason: the partial a crash preserves is a real capture path, and one nobody was comparing against a snapshot on this harness. @@ -200,12 +201,12 @@ class PiScenario: name: str lines: list[str] - expects: type[BaseException] | None = None + expects: AgentEndStatus | None = None -async def run_pi_scenario(scenario: PiScenario, working_dir: str) -> dict[str, Any]: +async def run_pi_scenario(scenario: PiScenario, working_dir: str) -> tuple[dict[str, Any], list[StreamEvent]]: """Replay one scenario and return the resulting record as a plain dump.""" - import pytest + recorder = EventRecorder() proc = _FakeProcess(scenario.lines) @@ -220,14 +221,11 @@ async def fake_exec(*_argv: str, **_kwargs: Any) -> _FakeProcess: patch.object(os, "killpg", lambda _pgid, _sig: None, create=True), ): await agent.start(working_dir) - if scenario.expects is not None: - with pytest.raises(scenario.expects): - await agent.communicate("do it") - record = agent.pending_turn - assert record is not None, f"{scenario.name}: pending_turn was not set on the failure path" - else: - record = await agent.communicate("do it") - return record.model_dump(mode="json") + outcome = await agent.communicate("do it", iteration=1, stream_callback=recorder) + expected = scenario.expects or AgentEndStatus.COMPLETED + assert outcome.status is expected, f"{scenario.name}: ended {outcome.status}, expected {expected}" + record = outcome.record + return record.model_dump(mode="json"), recorder.events def _build_catalogue() -> list[PiScenario]: @@ -280,7 +278,7 @@ def _build_catalogue() -> list[PiScenario]: # the pair then read as a measured span that the collector subtracted from # a generation window the tool never occupied. One bound alone forms no # span (`main_thread_tool_spans` requires both), so the window is left - # whole. Same rule as claude-code's `_finalize_commands`: unknown status + # whole. The emitter's sweep applies it on every harness: unknown status # and unknown duration are one fact (CE058). scenarios.append( PiScenario( @@ -296,7 +294,7 @@ def _build_catalogue() -> list[PiScenario]: # (e) the provider error pi's internal retries could not clear, AFTER a # complete generation. The CLI still exits 0, so `_settle_turn` crashes on - # `stopReason=error` alone — and the partial `pending_turn` must still carry + # `stopReason=error` alone — and the crashed record must still carry # that generation and its head/tail. A crash does not un-measure what was # measured before it. scenarios.append( @@ -309,7 +307,7 @@ def _build_catalogue() -> list[PiScenario]: _turn_start(), _turn_end_error("provider returned 529 after 5 retries"), ], - expects=AgentCrashError, + expects=AgentEndStatus.CRASHED, ) ) diff --git a/tests/fixtures/byoa_demo_plugin/byoa_demo.py b/tests/fixtures/byoa_demo_plugin/byoa_demo.py index d5ff8ca4..99328997 100644 --- a/tests/fixtures/byoa_demo_plugin/byoa_demo.py +++ b/tests/fixtures/byoa_demo_plugin/byoa_demo.py @@ -22,7 +22,6 @@ from coder_eval.agents.claude_code_agent import ClaudeCodeAgent from coder_eval.agents.registry import AgentRegistry from coder_eval.models import ClaudeCodeAgentConfig -from coder_eval.spi import SPI_VERSION DEMO_KIND = "byoa-demo" @@ -44,5 +43,4 @@ def register(registry: type[AgentRegistry]) -> None: ``registry`` is the ``AgentRegistry`` class (not an instance). """ - assert SPI_VERSION == 2, f"byoa_demo supports coder_eval SPI 2, not {SPI_VERSION}" - registry.register(DEMO_KIND, DemoAgentConfig)(DemoAgent) + registry.register(DEMO_KIND, DemoAgentConfig, spi_version=1)(DemoAgent) diff --git a/tests/fixtures/harness_stubs.py b/tests/fixtures/harness_stubs.py index 6cd5be52..fdeca425 100644 --- a/tests/fixtures/harness_stubs.py +++ b/tests/fixtures/harness_stubs.py @@ -6,7 +6,7 @@ from pydantic import create_model -from coder_eval.models import BaseAgentConfig, Enforcement, HarnessContract, UsageGranularity +from coder_eval.models import BaseAgentConfig, Enforcement, HarnessContract, TimingBasis, UsageGranularity def stub_contract(*, cooperative_stop: bool = True) -> HarnessContract: @@ -20,6 +20,7 @@ def stub_contract(*, cooperative_stop: bool = True) -> HarnessContract: disallowed_tools=Enforcement.UNSUPPORTED, cooperative_stop=cooperative_stop, usage_granularity=UsageGranularity.TURN, + timing_basis=TimingBasis.TURN_CLOCK, ) diff --git a/tests/fixtures/mock_agent.py b/tests/fixtures/mock_agent.py index 0966cba9..d42b1cbd 100644 --- a/tests/fixtures/mock_agent.py +++ b/tests/fixtures/mock_agent.py @@ -9,6 +9,8 @@ from coder_eval.agent import Agent, AgentState from coder_eval.models import TaskDefinition, TurnRecord +from coder_eval.streaming.emitter import TurnOutcome +from coder_eval.streaming.events import AgentEndStatus from tests.fixtures.harness_stubs import stub_contract @@ -70,28 +72,29 @@ def get_state(self) -> AgentState: """ return self.state - async def communicate(self, user_input: str, **kwargs) -> TurnRecord: + async def communicate(self, user_input: str, *, iteration: int, **kwargs) -> TurnOutcome: """Simulate agent turn based on configured scenario. Args: user_input: Prompt from orchestrator Returns: - TurnRecord with simulated agent response and file changes + A completed outcome carrying the simulated agent response and file changes Raises: ValueError: If scenario is unknown """ - self._iteration += 1 # Increment iteration count + self._iteration = iteration if self.scenario == "success": - return self._success_turn(user_input) + record = self._success_turn(user_input) elif self.scenario == "failure": - return self._failure_turn(user_input) + record = self._failure_turn(user_input) elif self.scenario == "partial": - return self._partial_turn(user_input) + record = self._partial_turn(user_input) else: raise ValueError(f"Unknown scenario: {self.scenario}") + return TurnOutcome(record=record, status=AgentEndStatus.COMPLETED, error=None) def _success_turn(self, user_input: str) -> TurnRecord: """Simulate successful task completion. diff --git a/tests/fixtures/opencode_happy_stream.jsonl b/tests/fixtures/opencode_happy_stream.jsonl new file mode 100644 index 00000000..b14c28ae --- /dev/null +++ b/tests/fixtures/opencode_happy_stream.jsonl @@ -0,0 +1,8 @@ +{"type":"step_start","timestamp":1789599421824,"sessionID":"ses_captured","part":{"id":"prt_0ac70157a001ZT0k7RkMOES5fK","messageID":"msg_0ac700f2b001SSGFoj9fVWpcYA","sessionID":"ses_captured","type":"step-start"}} +{"type":"tool_use","timestamp":1789599422997,"sessionID":"ses_captured","part":{"type":"tool","tool":"write","callID":"toolu_016ZZZbGJz51bwQrFrxN4Sv2","state":{"status":"completed","input":{"filePath":"/work/hello.txt","content":"hi"},"output":"Wrote file successfully.","metadata":{"diagnostics":{},"filepath":"/work/hello.txt","exists":false,"truncated":false},"title":"work/hello.txt","time":{"start":1789599422988,"end":1789599422995}},"metadata":{"openrouter":{"reasoning_details":[]}},"id":"prt_0ac701686001Qw5kHqmnUNaw0i","sessionID":"ses_captured","messageID":"msg_0ac700f2b001SSGFoj9fVWpcYA"}} +{"type":"text","timestamp":1789599423294,"sessionID":"ses_captured","part":{"id":"prt_0ac70157e001dUrObdNiWMqhff","messageID":"msg_0ac700f2b001SSGFoj9fVWpcYA","sessionID":"ses_captured","type":"text","text":"I'll help you create the file and list the directory.","time":{"start":1789599421822,"end":1789599423291}}} +{"type":"tool_use","timestamp":1789599423340,"sessionID":"ses_captured","part":{"type":"tool","tool":"bash","callID":"toolu_01DYyaQT59GsS3Qyj9smkbrb","state":{"status":"completed","input":{"command":"ls -la /work/"},"output":"total 8\ndrwx------ 3 user staff 96 Sep 16 15:57 .\ndrwx------@ 39029 user staff 1248928 Sep 16 15:57 ..\n-rw-r--r-- 1 user staff 2 Sep 16 15:57 hello.txt\n","metadata":{"output":"total 8\ndrwx------ 3 user staff 96 Sep 16 15:57 .\ndrwx------@ 39029 user staff 1248928 Sep 16 15:57 ..\n-rw-r--r-- 1 user staff 2 Sep 16 15:57 hello.txt\n","exit":0,"truncated":false},"title":"ls -la /work/","time":{"start":1789599423281,"end":1789599423331}},"id":"prt_0ac701a0d0016U67B5JGdiRx7R","sessionID":"ses_captured","messageID":"msg_0ac700f2b001SSGFoj9fVWpcYA"}} +{"type":"step_finish","timestamp":1789599423340,"sessionID":"ses_captured","part":{"id":"prt_0ac701b65001oV8lzL5tJ354vF","reason":"tool-calls","messageID":"msg_0ac700f2b001SSGFoj9fVWpcYA","sessionID":"ses_captured","type":"step-finish","tokens":{"total":16549,"input":3,"output":210,"reasoning":0,"cache":{"write":16336,"read":0}},"cost":0.021473}} +{"type":"step_start","timestamp":1789599424051,"sessionID":"ses_captured","part":{"id":"prt_0ac701e2c001eq21OKMX4VsR60","messageID":"msg_0ac701b6a001f4VpkH4NL51mAe","sessionID":"ses_captured","type":"step-start"}} +{"type":"text","timestamp":1789599424098,"sessionID":"ses_captured","part":{"id":"prt_0ac701e30001wThYZ7I3muX1xY","messageID":"msg_0ac701b6a001f4VpkH4NL51mAe","sessionID":"ses_captured","type":"text","text":"DONE","time":{"start":1789599424048,"end":1789599424088}}} +{"type":"step_finish","timestamp":1789599424098,"sessionID":"ses_captured","part":{"id":"prt_0ac701e5b001GJolKGDIGkU7ZI","reason":"stop","messageID":"msg_0ac701b6a001f4VpkH4NL51mAe","sessionID":"ses_captured","type":"step-finish","tokens":{"total":16703,"input":7,"output":5,"reasoning":0,"cache":{"write":355,"read":16336}},"cost":0.00210935}} diff --git a/tests/fixtures/text_stub_agent.py b/tests/fixtures/text_stub_agent.py index 90f8e310..fa294588 100644 --- a/tests/fixtures/text_stub_agent.py +++ b/tests/fixtures/text_stub_agent.py @@ -11,6 +11,8 @@ from coder_eval.agent import Agent, AgentState from coder_eval.models import TurnRecord +from coder_eval.streaming.emitter import TurnOutcome +from coder_eval.streaming.events import AgentEndStatus from tests.fixtures.harness_stubs import stub_contract @@ -47,12 +49,8 @@ async def stop(self) -> None: def get_state(self) -> AgentState: return self._state - async def communicate(self, user_input: str, **kwargs: object) -> TurnRecord: - self._iteration += 1 + async def communicate(self, user_input: str, *, iteration: int, **kwargs: object) -> TurnOutcome: self.calls.append(user_input) text = self._responses.pop(0) if self._responses else "" - return TurnRecord( - iteration=self._iteration, - user_input=user_input, - agent_output=text, - ) + record = TurnRecord(iteration=iteration, user_input=user_input, agent_output=text) + return TurnOutcome(record=record, status=AgentEndStatus.COMPLETED, error=None) diff --git a/tests/lint/harness_parity.py b/tests/lint/harness_parity.py index 07bdf0a1..1af18d3b 100644 --- a/tests/lint/harness_parity.py +++ b/tests/lint/harness_parity.py @@ -87,15 +87,30 @@ def _budget_cell(contract: HarnessContract) -> str: _RUN_LIMIT_CELLS: dict[str, Callable[[HarnessContract], str]] = { "max_tool_calls": lambda c: ( - "TurnMonitor at the should_stop poll, resolved tool calls" if c.cooperative_stop else "not polled (never fires)" + "TurnMonitor at the should_stop poll, main-thread resolved tool calls" + if c.cooperative_stop + else "not polled (never fires)" + ), + "max_turns": lambda c: ( + "TurnMonitor at the should_stop poll, when main-thread model turn N+1 starts" + if c.counts_model_turns + else "rejected at resolution" ), "expected_tool_calls": lambda _c: "orchestrator, cumulative visible tool calls, warns only", + "expected_turns": lambda c: ( + "orchestrator, cumulative model turns (TurnMonitor count), warns only" + if c.counts_model_turns + else "rejected at resolution" + ), "task_timeout": lambda _c: "orchestrator, agent-agnostic", "turn_timeout": lambda _c: "agent watchdog (see Timeouts)", "max_input_tokens": _budget_cell, "max_output_tokens": _budget_cell, "max_total_tokens": _budget_cell, - "max_usd": _budget_cell, + "max_usd": lambda c: ( + _budget_cell(c) + + ("; priced by the harness" if c.reports_cost else "; needs a priced agent.model (checked at resolution)") + ), "count_cached_input": lambda _c: "TurnMonitor bucket rule", "count_cache_creation": lambda _c: "TurnMonitor bucket rule", "stop_early": lambda c: "cooperative should_stop" if c.cooperative_stop else "rejected at resolution", diff --git a/tests/lint/pyright_config.py b/tests/lint/pyright_config.py index f472c45e..9cd143f0 100644 --- a/tests/lint/pyright_config.py +++ b/tests/lint/pyright_config.py @@ -1,19 +1,17 @@ -"""Emit a pyright config that type-checks the CE036 contract engine under `tests/`. +"""Emit a pyright config that type-checks the `tests/` modules worth checking. `make typecheck`'s main pass cannot reach those modules. `pyproject.toml`'s -`[tool.pyright]` excludes `"tests"`, and pyright's `exclude` beats BOTH of the -obvious shortcuts (verified against a probe file carrying a deliberate error): +`[tool.pyright]` excludes `"tests"`, and pyright's `exclude` beats both an +explicitly-passed CLI file arg and an `include` entry naming the file: either +shortcut analyzes zero files and exits 0. -* `pyright tests/lint/live_verdict_contract.py` — an explicitly-passed CLI file - arg is still excluded: `filesAnalyzed: 0`, exit 0. A gate that checks nothing. -* adding the path to `include` — likewise dropped; the probe never appears in - the analyzed set. +So the second pass needs its own config, DERIVED from `[tool.pyright]`: every +rule setting is copied verbatim, and only `include`, `exclude` (minus `"tests"`) +and `extraPaths` change, so the two passes cannot drift. -So the second pass needs its own config. This script DERIVES it from -`[tool.pyright]` — every rule setting is copied verbatim, and only `include` -(the modules below) and `exclude` (minus `"tests"`) are swapped. That is the -point of generating it instead of checking in a hand-written twin: a rule tuned -in `pyproject.toml` applies to both passes, and the two can never drift. +The pass covers CE036's contract engine and every `tests/*_live.py`. Live tests +need credentials to run, so nobody runs them on a routine change; checking their +types statically makes an SPI signature change fail the build instead. Usage: `python -m tests.lint.pyright_config ` """ @@ -28,22 +26,26 @@ REPO_ROOT = Path(__file__).resolve().parents[2] -# The `tests/` modules worth type-checking: CE036's contract engine executes real -# checker code and encodes the early-stop design, so a type error there is a bug in -# the gate itself. Add a path here only for a tests/ module with that character — -# this is deliberately not "all of tests/". INCLUDE = [ "tests/lint/live_verdict_contract.py", "tests/_fixtures/live_criteria.py", ] +BYOA_DEMO_DIR = "tests/fixtures/byoa_demo_plugin" +BYOA_DEMO = f"{BYOA_DEMO_DIR}/byoa_demo.py" + + +def live_tests() -> list[str]: + return sorted(path.relative_to(REPO_ROOT).as_posix() for path in (REPO_ROOT / "tests").glob("*_live.py")) + def build_config() -> dict[str, object]: with (REPO_ROOT / "pyproject.toml").open("rb") as handle: settings = dict(tomllib.load(handle)["tool"]["pyright"]) - settings["include"] = list(INCLUDE) + settings["include"] = [*INCLUDE, *live_tests(), BYOA_DEMO] settings["exclude"] = [pattern for pattern in settings.get("exclude", []) if pattern != "tests"] + settings["extraPaths"] = [*settings.get("extraPaths", []), BYOA_DEMO_DIR] return settings diff --git a/tests/lint/rules/_model_ctor.py b/tests/lint/rules/_model_ctor.py index da355c9d..39ecb039 100644 --- a/tests/lint/rules/_model_ctor.py +++ b/tests/lint/rules/_model_ctor.py @@ -1,36 +1,22 @@ -"""Resolve `coder_eval.models` constructor calls inside one module's AST. +"""Resolve constructor calls to a known class inside one module's AST. -CE060 and CE061 ask the same first question — *is this call building an -`AssistantMessage`?* — and answering it takes more than matching a name: a -module may bind the class under any alias, reach it through a relative import, -or never bind it at all and spell it `models.AssistantMessage(...)`. CE060 -worked that out once; duplicating it into CE061 would mean a model rename or a -new import spelling needs two fixes in two rules, and the second one is the one -that gets missed. So it lives here and both rules consume it. +Answering *is this call building a ``coder_eval.models.AssistantMessage``?* takes more +than matching a name: a module may bind the class under any alias, reach it through a +relative import, or spell it ``models.AssistantMessage(...)``. CE072 asks that question +for every class it bans, so the resolution lives here once. -The class name is taken from the model itself rather than written as a string, -the way CE056 imports `IN_CONTAINER_ENV` and CE057 derives its target set from -`SIDECAR_MODULES`: renaming the model moves both rules with it. - -BLIND SPOT, inherited by every consumer: a re-export through an intermediate -module (`from .sibling import AssistantMessage`) is invisible, because -resolving it means following imports across files and no rule in this package -does that. +BLIND SPOT, inherited by every consumer: a re-export through an intermediate module +(``from .sibling import AssistantMessage``) is invisible, because resolving it means +following imports across files and no rule in this package does that. """ import ast import re -from coder_eval.models import AssistantMessage - -# Reducers live here; nothing outside it builds a generation window. +# The adapters: every file under src/coder_eval/agents/. AGENTS_ROOT = re.compile(r"(?:^|[/\\])src[/\\]coder_eval[/\\]agents[/\\]") -_MODELS_MODULE = "coder_eval.models" -# Taken from the model, never spelled here: a rename then moves the rules too. -ASSISTANT_MESSAGE = AssistantMessage.__name__ - def reaches_module(node: ast.ImportFrom, module_path: str) -> bool: """True if this `from ... import` reaches `module_path`. @@ -58,23 +44,11 @@ def reaches_module(node: ast.ImportFrom, module_path: str) -> bool: return any(spelled[: len(segments) - i] == segments[i:] for i in range(1, len(segments))) -def reaches_models_module(node: ast.ImportFrom) -> bool: - """`reaches_module` pinned to `coder_eval.models` — CE060/CE061's question.""" - return reaches_module(node, _MODELS_MODULE) - - def bindings_from(tree: ast.AST, class_name: str, module_path: str) -> set[str]: """Every local name this module binds `.` to. Built per file: caching it across files would leak one module's alias into another's matching. - - Parameterized on the module because CE064 asks the identical question about - `coder_eval.streaming.events` and `coder_eval.timing` rather than about - `coder_eval.models`. Copying the resolver into it would mean a new import - spelling needs three fixes in three rules, and the third is the one that - gets missed — which is the argument this file already makes for CE060 and - CE061 sharing it. """ names: set[str] = set() for node in ast.walk(tree): @@ -83,11 +57,6 @@ def bindings_from(tree: ast.AST, class_name: str, module_path: str) -> set[str]: return names -def local_bindings(tree: ast.AST, class_name: str) -> set[str]: - """`bindings_from` pinned to `coder_eval.models` — CE060/CE061's question.""" - return bindings_from(tree, class_name, _MODELS_MODULE) - - def constructor_name(func: ast.expr, names: set[str], class_name: str) -> str | None: """The spelling this call used to name the model, or None if it did not. @@ -101,16 +70,3 @@ def constructor_name(func: ast.expr, names: set[str], class_name: str) -> str | if isinstance(func, ast.Attribute) and func.attr == class_name: return func.attr return None - - -def keywords_of(node: ast.Call) -> dict[str, ast.expr]: - """The call's named arguments. A `**`-expansion contributes nothing. - - That is deliberate rather than an oversight: such a call has not declared - the field AT THE SITE, which is what these rules are about. - """ - return {kw.arg: kw.value for kw in node.keywords if kw.arg is not None} - - -def is_none(node: ast.expr | None) -> bool: - return isinstance(node, ast.Constant) and node.value is None diff --git a/tests/lint/rules/ce058_no_timing_literal.py b/tests/lint/rules/ce058_no_timing_literal.py index d1364c5e..8f66ed4a 100644 --- a/tests/lint/rules/ce058_no_timing_literal.py +++ b/tests/lint/rules/ce058_no_timing_literal.py @@ -34,7 +34,7 @@ 3. ``x if x is not None else 0.0`` (and the ``is None`` mirror); 4. ``if x.duration_ms is None: x.duration_ms = 0.0`` — the form no existing rule shape covers, and where the live Claude instance was hiding - (``_finalize_commands`` set it on every command force-closed without a + (its former command finalizer set it on every command force-closed without a tool result, in the one harness a timing audit had called healthy); 5. ``model_copy(update={"duration_ms": 0.0})`` — a keyword rule is blind to a dict, and the dict is how ``CommandTelemetry.duration_ms`` is actually @@ -45,7 +45,7 @@ Form 6 exists because form 4 was passing the live defect by coincidence. Form 4 keys on the ``if`` test naming a timing attribute, and the shipped -``_finalize_commands`` bug happened to spell it that way +Claude command-finalizer bug happened to spell it that way (``if cmd.duration_ms is None:``) — but the assignment sat inside an outer ``if cmd.result_status is None:`` block, and rewriting it to set the literal under THAT guard instead, which reads just as naturally and books the identical diff --git a/tests/lint/rules/ce059_generation_window_is_two_reads.py b/tests/lint/rules/ce059_generation_window_is_two_reads.py deleted file mode 100644 index 254de2b3..00000000 --- a/tests/lint/rules/ce059_generation_window_is_two_reads.py +++ /dev/null @@ -1,79 +0,0 @@ -"""CE059: one clock read cannot measure a window. - -An ``AssistantMessage`` that receives the SAME name for both ``started_at`` and -``completed_at`` records a zero-length generation window — whatever -``generation_duration_ms`` happens to say beside it. The Antigravity reducer -read ``datetime.now()`` once and passed it as both bounds, so -``started_at == completed_at`` on 368 of 368 sampled messages and every -consumer that derives a window from the two stamps saw nothing at all. A window -needs two reads at two moments. - -Separate id from CE058 deliberately: this is a different invariant (a -zero-length window, regardless of what the duration field says), and one -invariant per id is what makes a ``# noqa`` mean one thing. - -WHAT IT DOES NOT FIRE ON, and why that is the rule rather than a stack of -suppressions: a call that passes ``generation_duration_ms=None`` in the same -breath is not claiming a window — it is saying, in the field built to say it, -that none was measurable. Three sites are legitimately like that (Codex's -rollout rebuild, and both sub-agent syntheses on Codex and Claude: the -generation arrives as a tool result and is never streamed), and collapsing -their bounds to one ``now()`` is then a formatting choice, not a false -measurement. Exempting them here — rather than through four permanent -``# noqa`` lines — keeps the rule pointed at the case that actually misleads: -a duration asserted beside two stamps that cannot support it. - -Scoped to ``src/coder_eval/agents/``, the layer that measures. The check is -skipped unless BOTH bounds are a bare ``ast.Name`` — comparing attribute or -call expressions (``self.a`` vs ``self.b``) would be guesswork. - -BLIND SPOT: two DIFFERENT names that hold the same instant at runtime. Codex -already produces that shape — ``started = _ms_to_dt(self.open_start_ms)`` and -``completed = _ms_to_dt(self.open_end_ms if ... is not None else -self.open_start_ms)`` collapse to one instant whenever ``open_end_ms`` is -None. No AST rule can see it. The catch for that case is the replay-based -``assert_timing_captured`` golden invariant, which runs the real reducer and -asserts a non-zero window actually came out. -""" - -import ast -import re - -from tests.lint.rules.base import BaseRule - - -_AGENTS_ROOT = re.compile(r"(?:^|[/\\])src[/\\]coder_eval[/\\]agents[/\\]") - -_MESSAGE_CONSTRUCTORS = frozenset({"AssistantMessage", "AssistantMessageTelemetry"}) - - -def _is_none(node: ast.expr | None) -> bool: - return isinstance(node, ast.Constant) and node.value is None - - -class GenerationWindowIsTwoReads(BaseRule): - id = "CE059" - - def __init__(self, filepath: str) -> None: - super().__init__(filepath) - self._in_scope = bool(_AGENTS_ROOT.search(filepath)) - - def visit_Call(self, node: ast.Call) -> None: - if self._in_scope: - func = node.func - name = func.id if isinstance(func, ast.Name) else func.attr if isinstance(func, ast.Attribute) else None - if name in _MESSAGE_CONSTRUCTORS: - kwargs = {kw.arg: kw.value for kw in node.keywords if kw.arg is not None} - start, end = kwargs.get("started_at"), kwargs.get("completed_at") - claims_a_window = not _is_none(kwargs.get("generation_duration_ms")) - if claims_a_window and isinstance(start, ast.Name) and isinstance(end, ast.Name) and start.id == end.id: - self.violation( - node, - f"'started_at' and 'completed_at' are both {start.id!r}, so the generation " - "window has zero length while generation_duration_ms claims one beside it. " - "Antigravity shipped this on 368 of 368 sampled messages. Read the clock " - "twice — mark the end of the previous SDK event, and read again when this " - "message arrives — or pass generation_duration_ms=None if no window is " - "measurable.", - ) - self.generic_visit(node) diff --git a/tests/lint/rules/ce060_message_id_declared.py b/tests/lint/rules/ce060_message_id_declared.py deleted file mode 100644 index fbdb4820..00000000 --- a/tests/lint/rules/ce060_message_id_declared.py +++ /dev/null @@ -1,112 +0,0 @@ -"""CE060: an assistant message must declare its identity. - -``AssistantMessage.message_id`` is what lets a consumer tell two generations -apart. Antigravity simply omitted the kwarg, so the field defaulted to ``None`` -on every message it ever recorded, and the evalboard — which groups assistant -emissions by ``message_id`` and falls back to a wall-clock ``SAME_EMISSION_GAP_MS`` -threshold when either side lacks one — folded a whole turn's generations into a -single timeline row once the harness's windows became contiguous. Nothing -failed: the consumer sums a group, so every total came out right, and the -golden snapshots had ratified the ``null`` the day they were written. It was -not confined to the timeline either — a grouped emission is one API call to the -thinking-cost simulator, so its whole cache cascade was computed from one call -per turn. The mechanism and the blast radius live in -``docs/agents/HARNESS_PARITY.md`` § Timing capture; neither is restated here. - -Separate id from CE058 and CE059 deliberately: those two are about *timing* -(an unknown duration published as a literal, a window built from one clock -read), this one is about *identity*. One invariant per id is what makes a -``# noqa`` mean one thing. - -WHY IT RESOLVES ALIASES where CE058 and CE059 hardcode constructor names: -CE058's own docstring already concedes that spelling-based matching dies on a -rename, and the weakness is live — ``claude_code_agent.py`` binds *only* -``AssistantMessage as AssistantMessageTelemetry`` and never the bare name, so a -name list guards that file's two construction sites purely because somebody -wrote the current alias into a different rule. CE060 instead derives its -constructor set from each module's own ``coder_eval.models`` imports, which -removes the gap rather than documenting it and catches an arbitrary -``AssistantMessage as Msg`` besides. Widening the other two rules the same way -is recorded in ``.claude/harness-candidates.md``; it is a change to two shipped -rules and needs its own mutation checks. - -What it removes is the *local binding* spelling, not every rename: the class's -own name still has to be known, so it is taken from the model itself -(``AssistantMessage.__name__``) rather than written here as a string, the way -CE056 imports ``IN_CONTAINER_ENV`` and CE057 derives its target set from -``SIDECAR_MODULES``. Renaming the model therefore moves this rule with it. - -That resolution lives in ``_model_ctor.py`` and is shared with CE061, which -needs the identical answer to a different question. Keeping two copies would -mean a new import spelling needs two fixes in two rules. - -BLIND SPOT 1: the runtime ``None``. The rule requires the kwarg to be -*present*, not non-``None`` when it runs. ``opencode_agent.py`` passes -``str(part.get("messageID") or "") or None`` and ``pi_agent.py`` the same shape -for ``responseId``, so either records ``None`` whenever the id is missing from -the payload (`pi_a_single_text_turn.json` is a snapshot of that shape, though -its null comes from a fixture that emits no ``responseId`` rather than from a -live CLI omission). No AST rule can see it, and demanding a statically -non-``None`` value would be wrong: passing a fallback expression *is* deciding -what the id is. The sensor for that case is the golden corpus, and only -partially — a snapshot is written from whatever the code currently does, so it -catches a later change, never an initial omission. - -BLIND SPOT 2: a binding the resolver cannot follow. It reads one module's own -imports, so it sees the direct forms — absolute or relative ``from ... import -AssistantMessage``, under any alias — and the attribute spelling -``.AssistantMessage(...)``. What remains invisible is a re-export -through an intermediate module (``from .sibling import AssistantMessage``); see -``_model_ctor.py``. - -A ``**``-expanded call fires: such a call has not declared the field at the -site. There is no carve-out because no site in ``src/coder_eval/agents/`` uses -``**`` expansion for these constructors; if one is ever added, pass -``message_id=`` explicitly beside it. -""" - -import ast - -from tests.lint.rules._model_ctor import ( - AGENTS_ROOT, - ASSISTANT_MESSAGE, - constructor_name, - is_none, - keywords_of, - local_bindings, -) -from tests.lint.rules.base import BaseRule -from tests.lint.violation import Violation - - -class MessageIdDeclared(BaseRule): - id = "CE060" - - def __init__(self, filepath: str) -> None: - super().__init__(filepath) - self._in_scope = bool(AGENTS_ROOT.search(filepath)) - self._names: set[str] = set() - - def check(self, tree: ast.AST) -> list[Violation]: - self._names = local_bindings(tree, ASSISTANT_MESSAGE) - return super().check(tree) - - def visit_Call(self, node: ast.Call) -> None: - if self._in_scope: - name = constructor_name(node.func, self._names, ASSISTANT_MESSAGE) - if name is not None: - kwargs = keywords_of(node) - if "message_id" not in kwargs or is_none(kwargs["message_id"]): - self.violation( - node, - f"{name}(...) leaves 'message_id' undeclared — absent, or an explicit None — " - "so every message it builds shares one empty identity. Pass the field " - "with a real value: the CLI's own " - "id where the stream carries one, else synthesize it the way Codex does " - "(f'{turn_id}-msg-{gen_index}'). Antigravity shipped without it: the " - "evalboard then falls back to its SAME_EMISSION_GAP_MS wall-clock gap to " - "group emissions, and a harness whose generation windows are contiguous " - "has every one of a turn's generations collapse into one row. See " - "docs/agents/HARNESS_PARITY.md.", - ) - self.generic_visit(node) diff --git a/tests/lint/rules/ce061_window_via_close_window.py b/tests/lint/rules/ce061_window_via_close_window.py deleted file mode 100644 index 47e49a3e..00000000 --- a/tests/lint/rules/ce061_window_via_close_window.py +++ /dev/null @@ -1,141 +0,0 @@ -"""CE061: a generation window must come from the shared helper. - -Pi shipped measuring its window from its own ``turn_start`` while four sibling -reducers tiled from a mark, so the wall clock between one turn's end and the -next turn's start — the model time that PRODUCED that turn — fell into no -bucket at all. Nothing failed. ``docs/agents/HARNESS_PARITY.md`` asserted the -four-bucket identity, and the only sensor for it -(``tests/_fixtures/golden_streams/_scrub.py``) checks ONE side: it catches a -bucket claiming more time than the turn contains and says nothing about one -claiming less. Pi's own tests passed because they were written against Pi's -own arithmetic. - -That is the shape this rule guards against: not a reducer that computes the -window wrongly, but a reducer that computes it AT ALL instead of asking -``coder_eval.timing.close_window``. A new harness whose author reimplements the -arithmetic inline arrives with a green test suite by construction. - -Separate id from CE058, CE059 and CE060 deliberately. Those three are about the -VALUES a message carries — an unknown duration published as a literal, a window -built from one clock read, a missing identity. This one is about PROVENANCE: -where the arithmetic came from. One invariant per id is what makes a ``# noqa`` -mean one thing. - -NOTE what this rule no longer covers, and deliberately: the tool SUBTRACTION is -not part of a window's geometry any more, so "did this reducer subtract -correctly" is not a question here. CE063 owns it — no module in ``agents/`` may -import ``busy_ms`` at all. - -BLIND SPOT, and it is the whole weakness of the chosen shape: this proves the -module IMPORTS the helper, never that any particular call used it. The value -passed to ``generation_duration_ms=`` is always a local (``generation_ms``, -``gen_parts[idx]``), so no AST rule can trace it back to a call. The sensors for -the arithmetic itself are ``tests/test_timing_close_window.py`` and the -per-reducer window tests; this rule adds only the cheap structural half that -neither can reach — a sixth harness rolling its own. - -It costs NO suppression. It used to cost exactly one: ``claude_code_agent.py`` -computed its window from a monotonic delta and subtracted tool time once at -finalization, because a call issued by an earlier emission is still running when -the next window closes — and forcing that into ``close_window`` would have meant -a mode flag on a helper whose whole value is having one shape. Moving the -subtraction to ``timing.subtract_tool_time`` dissolved the exception: -the collector is already the place where every span is known, so claude-code -needs no separate pass and calls the same shrunken helper as the other four. -``tests/test_custom_lint.py::TestCE061WindowViaCloseWindow::test_the_rule_is_now_exemption_free`` -pins the suppression set EMPTY, so a new exemption has to be argued for. - -EXEMPT, because both are honest claims that no window was measured: an explicit -``generation_duration_ms=None`` (codex's rollout rebuild, claude-code's -sub-agent synthesis) and the kwarg absent altogether, which defaults to -``None``. Not matched: ``**``-expansion and ``model_copy(update={...})`` — CE058 -already covers the ``model_copy`` dict shape for timing literals. - -Alias resolution, and its blind spot, live in ``_model_ctor.py``, shared with -CE060. The helper's own name is taken from the function object rather than -written here as a string, so renaming it moves this rule too. -""" - -import ast - -from coder_eval.timing import close_window -from tests.lint.rules._model_ctor import ( - AGENTS_ROOT, - ASSISTANT_MESSAGE, - constructor_name, - is_none, - keywords_of, - local_bindings, -) -from tests.lint.rules.base import BaseRule -from tests.lint.violation import Violation - - -_TIMING_MODULE = "coder_eval.timing" -_TIMING_TAIL = _TIMING_MODULE.rpartition(".")[2] - -# Taken from the function, never spelled here: a rename then moves the rule too. -_HELPER = close_window.__name__ - - -def _imports_the_helper(tree: ast.AST) -> bool: - """True if this module can reach `close_window` under any spelling. - - Both the `from`-import (under any alias) and the module import that makes - `timing.close_window(...)` possible count — a rule that recognized only the - first would tell an author to change a working call site. - """ - for node in ast.walk(tree): - if isinstance(node, ast.ImportFrom): - module = node.module or "" - reaches = module.startswith(_TIMING_MODULE) or ( - bool(node.level) and (module == _TIMING_TAIL or module.startswith(f"{_TIMING_TAIL}.")) - ) - if reaches and any(a.name == _HELPER for a in node.names): - return True - # `from coder_eval import timing` / `from .. import timing`. The - # package is checked too: `from anywhere import timing` is not this - # module, and accepting it would let an unrelated name disarm the - # rule for a whole file. - package = module == _TIMING_MODULE.rpartition(".")[0] or (bool(node.level) and not module) - if package and any(a.name == _TIMING_TAIL for a in node.names): - return True - elif isinstance(node, ast.Import): - if any(a.name == _TIMING_MODULE for a in node.names): - return True - return False - - -class WindowViaCloseWindow(BaseRule): - id = "CE061" - - def __init__(self, filepath: str) -> None: - super().__init__(filepath) - self._in_scope = bool(AGENTS_ROOT.search(filepath)) - self._names: set[str] = set() - self._has_helper = False - - def check(self, tree: ast.AST) -> list[Violation]: - self._names = local_bindings(tree, ASSISTANT_MESSAGE) - self._has_helper = _imports_the_helper(tree) - return super().check(tree) - - def visit_Call(self, node: ast.Call) -> None: - if self._in_scope and not self._has_helper: - name = constructor_name(node.func, self._names, ASSISTANT_MESSAGE) - duration = keywords_of(node).get("generation_duration_ms") - if name is not None and duration is not None and not is_none(duration): - self.violation( - node, - f"{name}(...) publishes a measured 'generation_duration_ms' but this module " - f"never imports {_TIMING_MODULE}.{_HELPER} — so it is computing a generation " - "window of its own. Every window is the same geometry: tile from the mark, and " - "keep a backwards item stamp from inverting the span. Publish that RAW span; do " - "NOT subtract tool time here — coder_eval.timing.subtract_tool_time does it once, " - "for every harness, and doing it in the reducer too takes it out twice (CE063 " - "guards that half). Pi got the mark wrong by measuring from its own turn start, " - "and nothing caught it because the golden identity check is one-sided; " - "tests/test_timing_identity_contract.py is the two-sided one. " - f"Call {_HELPER} instead.", - ) - self.generic_visit(node) diff --git a/tests/lint/rules/ce063_no_busy_ms_in_agents.py b/tests/lint/rules/ce063_no_busy_ms_in_agents.py deleted file mode 100644 index 18d0e1b6..00000000 --- a/tests/lint/rules/ce063_no_busy_ms_in_agents.py +++ /dev/null @@ -1,105 +0,0 @@ -"""CE063: a reducer may not compute its own tool subtraction. - -Tool execution comes out of a generation window in exactly ONE place: -``coder_eval.timing.subtract_tool_time``. Before that, five -reducers each did it themselves — four through ``close_window`` as they -flushed, claude-code once at finalization — while the head and the tail were -already computed centrally at the collector seam. That asymmetry is where every -timing defect on this branch actually lived, and none of them was in the -arithmetic: they were in the bookkeeping AROUND it. When to reset a per-step -span list (clearing it at ``step_start`` wiped a span before the flush could -subtract it — a 100% overstatement of that window). When to clear a spent start -stamp (a second flush with no intervening start republished the previous span — -3000 ms of generation for a 2000 ms turn). When to advance the mark. - -A sixth harness whose author reaches for ``busy_ms`` is rebuilding exactly that -bookkeeping, and its tool time would then be subtracted TWICE: once by the -reducer and once by the collector, which subtracts from every window it is -handed. The result is a silently under-reported generation figure on one -harness only — the shape that takes a corpus comparison to notice. - -Separate id from CE061 deliberately, and CE061 is NOT rebodied into this. -CE061 asks where a window's ARITHMETIC came from, and four reducers still call -``close_window``, so its property is still live and still worth guarding — it -is not superseded. This one asks a different question: whether a reducer -subtracts tool time at all. One invariant per id is what makes a ``# noqa`` -mean one thing. (Phase 5 did make CE061 exemption-free: claude-code now calls -the shrunken ``close_window`` like the other four, so its one permanent -suppression is gone.) - -WHY NOT ``_imports_the_helper``, which CE061 uses. That function deliberately -returns True for a bare module import (``from coder_eval import timing``), so -that ``timing.close_window(...)`` counts as reaching the helper — its own -comment says a rule that missed it "would tell an author to change a working -call site." Inverted into a BAN that branch flags any reducer importing the -module and calling ``timing.close_window(...)``, which after Phase 5 is four of -them. So this rule keys on the ``busy_ms`` NAME binding plus an -``ast.Attribute`` match for the ``timing.busy_ms`` spelling, and leaves the -module import alone. - -The name is taken from the function object rather than written here as a -string, the way CE061 takes ``close_window``: renaming it moves this rule too. - -BLIND SPOT: a reducer that re-implements the union inline, without importing -anything, is invisible — as is one reaching ``busy_ms`` through a re-export. -The sensor for the arithmetic itself is -``tests/test_timing_identity_contract.py``, which drives every harness off a -scripted clock and asserts the four buckets tile the turn to the millisecond; -this rule adds only the cheap structural half that a static check can reach. -""" - -import ast - -from coder_eval.timing import busy_ms -from tests.lint.rules._model_ctor import AGENTS_ROOT -from tests.lint.rules.base import BaseRule - - -_TIMING_MODULE = "coder_eval.timing" -_TIMING_TAIL = _TIMING_MODULE.rpartition(".")[2] - -# Taken from the function, never spelled here: a rename then moves the rule too. -_BANNED = busy_ms.__name__ - -_MESSAGE = ( - f"imports '{_BANNED}', but a reducer does not subtract tool time any more — " - "coder_eval.timing.subtract_tool_time does it once, for every harness, " - "at the single capture seam. Publish the RAW window (close_window gives you its bounds " - "and span) and let the collector clip the tool union out of it. Subtracting here too " - "takes it out twice and silently under-reports generation on this harness alone." -) - - -class NoBusyMsInAgents(BaseRule): - id = "CE063" - - def __init__(self, filepath: str) -> None: - super().__init__(filepath) - self._in_scope = bool(AGENTS_ROOT.search(filepath)) - - def visit_ImportFrom(self, node: ast.ImportFrom) -> None: - """`from coder_eval.timing import busy_ms`, under any alias. - - Relative forms (`from ..timing import busy_ms`) count too: the module - is the same one whatever the path to it looks like. - """ - if not self._in_scope: - return - module = node.module or "" - reaches = module.startswith(_TIMING_MODULE) or ( - bool(node.level) and (module == _TIMING_TAIL or module.startswith(f"{_TIMING_TAIL}.")) - ) - if reaches and any(alias.name == _BANNED for alias in node.names): - self.violation(node, _MESSAGE) - - def visit_Attribute(self, node: ast.Attribute) -> None: - """The `timing.busy_ms` spelling. - - Defensive: no reducer uses it today (all five import plain names), but - a name-binding check alone would let it through, and it is one arm. - """ - if not self._in_scope: - return - if node.attr == _BANNED and isinstance(node.value, ast.Name) and node.value.id == _TIMING_TAIL: - self.violation(node, _MESSAGE) - self.generic_visit(node) diff --git a/tests/lint/rules/ce064_turn_bracket_on_the_clock.py b/tests/lint/rules/ce064_turn_bracket_on_the_clock.py deleted file mode 100644 index 713f823f..00000000 --- a/tests/lint/rules/ce064_turn_bracket_on_the_clock.py +++ /dev/null @@ -1,116 +0,0 @@ -"""CE064: a clocked harness must stamp its turn BRACKET off that same clock. - -``decompose_turn`` computes the head and the tail by subtracting a generation -window bound from an ``AgentStartEvent`` / ``AgentEndEvent`` timestamp. Those -two stamps therefore have to share a basis, and a reducer that derives its -window bounds from a ``TurnClock`` while letting the bracket fall back to -``StreamEvent.timestamp``'s ``default_factory=datetime.now`` puts a -monotonic-derived stamp and a raw wall stamp inside one subtraction — the exact -split ``timing.TurnClock`` exists to remove, reintroduced at the one seam the -clock does not own. - -MEASURED, not hypothetical. Instrumenting ``decompose_turn`` on a live -antigravity turn printed:: - - PROBE tail: elapsed=-0.017000ms busy=0.000000ms raw=-0.017000ms - last_completed = 09:05:22.033099 - agent_end = 09:05:22.033082 - -an ``AgentEndEvent`` stamped 17 us BEFORE its own last message finished, which -cannot happen: the event is constructed strictly after the final flush. -``decompose_turn`` then clamps the negative to ``0.0`` and publishes it, which -is "measured, and instant" — the CE058 confusion, arrived at from the other -direction. The published ``harness_teardown_ms`` was ``0.0`` for a harness -whose real tail is ~0.1 ms. - -WHY IT ONLY SHOWED ON ONE HARNESS, and why the rule is not scoped to that one: -the drift between the two clocks is tens of microseconds, so it can only flip a -sign where the true interval is itself that small. Antigravity is the only -harness that spawns its process ONCE in ``start()`` and holds it across turns, -so nothing happens between its last flush and its ``AgentEndEvent``; every -other harness books a head of 0.2-6 s and a tail of 7-543 ms, where the drift -is invisible. Invisible is not absent. The fix belongs at every clocked site -because that is what makes the subtraction single-basis rather than -usually-close, and "usually-close" is not a property a millisecond field can -rest on. - -SCOPE IS DERIVED, never listed. The rule applies to a module under -``agents/`` that imports ``TurnClock`` — antigravity, pi and claude-code today. -Codex and OpenCode take their spans from the CLI's own epoch stamps and -deliberately have no ``TurnClock`` (see that class's docstring), so a raw -``datetime.now()`` bracket is CONSISTENT with their bounds and the rule must -not fire on them; the noop agent has no windows at all. The day one of them -adopts a clock, this rule starts applying to it with no edit here — which is -the half a hardcoded harness list would get wrong. - -Separate id from CE058/CE059/CE060/CE061 for the reason CE060 states: one -invariant per id, so a ``# noqa`` means one thing. CE058 is about publishing a -literal for an unknown duration, CE059 about a window built from a single clock -read, CE060 about identity, CE061 about where a window's arithmetic comes from. -This one is about the turn's OUTER bounds, which no other rule looks at — they -all scope to ``AssistantMessage``, and the bracket is not one. - -BLIND SPOT: presence, not correctness. The rule requires ``timestamp=`` to be -passed; it cannot tell ``self.clock.now()`` from ``datetime.now()`` written out -at the call site, because an agent may legitimately reach its clock through any -expression (a local ``clock`` in ``communicate``, ``state.clock`` from the -caller, ``self.clock`` inside the state). Demanding a specific spelling would -make the rule a syntax check on three harnesses' internal structure. What it -removes is the SILENT case — a default nobody chose — which is the one that -shipped. -""" - -import ast - -from coder_eval.streaming.events import AgentEndEvent, AgentStartEvent -from coder_eval.timing import TurnClock -from tests.lint.rules._model_ctor import AGENTS_ROOT, bindings_from, constructor_name, keywords_of -from tests.lint.rules.base import BaseRule -from tests.lint.violation import Violation - - -_EVENTS_MODULE = "coder_eval.streaming.events" -_TIMING_MODULE = "coder_eval.timing" - -# Taken from the classes themselves, never spelled here: a rename moves the -# rule with them, the way CE056 imports IN_CONTAINER_ENV. -_BRACKETS = (AgentStartEvent.__name__, AgentEndEvent.__name__) -_CLOCK = TurnClock.__name__ - - -class TurnBracketOnTheClock(BaseRule): - id = "CE064" - - def __init__(self, filepath: str) -> None: - super().__init__(filepath) - self._in_scope = bool(AGENTS_ROOT.search(filepath)) - self._clocked = False - self._names: dict[str, set[str]] = {} - - def check(self, tree: ast.AST) -> list[Violation]: - if not self._in_scope: - return [] - self._clocked = bool(bindings_from(tree, _CLOCK, _TIMING_MODULE)) - if not self._clocked: - return [] - self._names = {name: bindings_from(tree, name, _EVENTS_MODULE) for name in _BRACKETS} - return super().check(tree) - - def visit_Call(self, node: ast.Call) -> None: - for bracket in _BRACKETS: - name = constructor_name(node.func, self._names[bracket], bracket) - if name is None: - continue - if "timestamp" not in keywords_of(node): - self.violation( - node, - f"{name}(...) leaves 'timestamp' to StreamEvent's default_factory " - "(a raw datetime.now()), but this harness derives its generation-window " - f"bounds from a {_CLOCK}. `timing.decompose_turn` subtracts one from the " - "other to get harness_startup_ms / harness_teardown_ms, so the two bases " - "meet inside one subtraction — measured at -0.017 ms on antigravity, an " - "agent end stamped BEFORE its own last message finished, which " - "decompose_turn then clamped to the 0.0 that means 'measured, and " - "instant' (CE058). Pass timestamp=.now().", - ) - self.generic_visit(node) diff --git a/tests/lint/rules/ce070_no_cap_or_skill_scan_in_adapters.py b/tests/lint/rules/ce070_no_cap_or_skill_scan_in_adapters.py index e0097aa2..32e87516 100644 --- a/tests/lint/rules/ce070_no_cap_or_skill_scan_in_adapters.py +++ b/tests/lint/rules/ce070_no_cap_or_skill_scan_in_adapters.py @@ -12,8 +12,8 @@ Fires, in files under ``src/coder_eval/agents/``, on any name, attribute, keyword, parameter or ``from``-import alias spelled ``max_turns``, ``max_tool_calls``, -``max_turns_reached``, ``max_turns_hit``, ``tool_calls_exhausted``, ``RunLimits`` or -``expand_env_vars``, and on the string literal ``"SKILL.md"``. The identifiers are the +``max_turns_reached``, ``max_turns_hit``, ``expected_turns``, ``tool_calls_exhausted``, +``RunLimits`` or ``expand_env_vars``, and on the string literal ``"SKILL.md"``. The identifiers are the sensor because the cap has one owner, the flag one writer and staging one scanner: an adapter that needs any of them is re-growing a copy. A substring is not a match, so ``_is_max_turns_result`` and the SDK's ``"error_max_turns"`` stay legal. @@ -35,6 +35,7 @@ "max_tool_calls", "max_turns_reached", "max_turns_hit", + "expected_turns", "tool_calls_exhausted", "RunLimits", "expand_env_vars", diff --git a/tests/lint/rules/ce071_price_turn_only.py b/tests/lint/rules/ce071_price_turn_only.py new file mode 100644 index 00000000..ecc28c45 --- /dev/null +++ b/tests/lint/rules/ce071_price_turn_only.py @@ -0,0 +1,53 @@ +"""CE071: agent adapters and the turn monitor price a turn only through ``pricing.price_turn``. + +The defect: five adapters and ``TurnMonitor`` each carried their own copy of the cost +rule on top of ``calculate_cost``. Pi and OpenCode priced a reported ``$0`` from the rate +card, the monitor took it as free, Antigravity and Codex never looked at a report, and +Claude kept a third variant for LiteLLM. So the ``max_usd`` stop and the persisted turn +cost could disagree on the same turn. ``price_turn`` is now the one rule, and an adapter +or the monitor that calls ``calculate_cost`` is re-growing a copy. + +Fires, in files under ``src/coder_eval/agents/`` and in +``src/coder_eval/orchestration/turn_monitor.py``, on any name, attribute or +``from``-import alias spelled ``calculate_cost``. Pricing outside a subject turn (the +simulator in ``models/results.py``, ``evaluation/judge_usage.py``) is out of scope. + +Blind spot: a copy of the rate arithmetic under another name, e.g. reading +``ModelPricing`` fields directly. +""" + +import ast +import re + +from tests.lint.rules._model_ctor import AGENTS_ROOT +from tests.lint.rules.base import BaseRule + + +_TURN_MONITOR = re.compile(r"(?:^|[/\\])orchestration[/\\]turn_monitor\.py$") +_BANNED = "calculate_cost" +_FIX = "price a turn with coder_eval.pricing.price_turn, the one cost rule shared with the max_usd monitor" + + +class PriceTurnOnly(BaseRule): + id = "CE071" + + def __init__(self, filepath: str) -> None: + super().__init__(filepath) + self._in_scope = bool(AGENTS_ROOT.search(filepath) or _TURN_MONITOR.search(filepath)) + + def _flag(self, node: ast.AST, name: str) -> None: + if self._in_scope and name == _BANNED: + self.violation(node, f"architectural violation: '{_BANNED}' used to price a turn — {_FIX}") + + def visit_Name(self, node: ast.Name) -> None: + self._flag(node, node.id) + self.generic_visit(node) + + def visit_Attribute(self, node: ast.Attribute) -> None: + self._flag(node, node.attr) + self.generic_visit(node) + + def visit_ImportFrom(self, node: ast.ImportFrom) -> None: + for alias in node.names: + self._flag(node, alias.name) + self.generic_visit(node) diff --git a/tests/lint/rules/ce072_emitter_sole_writer.py b/tests/lint/rules/ce072_emitter_sole_writer.py new file mode 100644 index 00000000..97aa7a40 --- /dev/null +++ b/tests/lint/rules/ce072_emitter_sole_writer.py @@ -0,0 +1,76 @@ +"""CE072: an agent adapter writes the event protocol only through ``TurnEmitter``. + +The defect class: every harness once carried its own per-turn accumulator — the open +tools, the sequence numbers, the transcript, the reported usage, the inner turn and the +end event — six copies of one state machine. Each copy shipped its own defect, and +five timing rules (now retired; see the runner note) were each written after one of +them: a window built from one clock read, an assistant message with no identity, a +window computed by hand, tool time subtracted in an adapter, a turn bracket off the +turn clock. Those rules guarded the copies. ``TurnEmitter`` removed the copies, so one rule now keeps an +adapter from growing one back: it may not construct the events, the transcript +messages or the reducer itself. + +Fires, in files under ``src/coder_eval/agents/``, on a call that constructs +``AgentStartEvent``, ``AgentEndEvent``, ``TurnStartEvent``, ``TurnEndEvent``, +``ToolStartEvent``, ``ToolEndEvent``, ``TextChunkEvent``, ``AssistantMessage`` or +``EventCollector`` — imported from ``coder_eval.streaming`` or one of its submodules, +under any import alias, through a relative import, or as a module attribute. +``CommandTelemetry`` is not banned: an adapter builds tool telemetry and hands it to +the emitter. Importing a class for a type annotation is allowed. + +Blind spots: a plugin agent outside this tree (the emitter's runtime guards and +``coder_eval.testing`` are its sensors); ``AssistantMessage.model_validate(...)`` or +``model_copy`` building a message; a re-export through an intermediate module. +""" + +import ast + +from coder_eval.models import AssistantMessage +from coder_eval.streaming import events +from coder_eval.streaming.collector import EventCollector +from tests.lint.rules._model_ctor import AGENTS_ROOT, bindings_from, constructor_name +from tests.lint.rules.base import BaseRule +from tests.lint.violation import Violation + + +# Taken from the classes, never spelled here: a rename moves the rule with it. +_BANNED: dict[str, str] = { + **{ + cls.__name__: "coder_eval.streaming" + for cls in ( + events.AgentStartEvent, + events.AgentEndEvent, + events.TurnStartEvent, + events.TurnEndEvent, + events.ToolStartEvent, + events.ToolEndEvent, + events.TextChunkEvent, + ) + }, + AssistantMessage.__name__: "coder_eval.models", + EventCollector.__name__: "coder_eval.streaming", +} +_FIX = "report it through the turn's TurnEmitter (Agent._open_emitter), the sole writer of the event protocol" + + +class EmitterSoleWriter(BaseRule): + id = "CE072" + + def __init__(self, filepath: str) -> None: + super().__init__(filepath) + self._in_scope = bool(AGENTS_ROOT.search(filepath)) + self._names: dict[str, set[str]] = {} + + def check(self, tree: ast.AST) -> list[Violation]: + if self._in_scope: + self._names = {name: bindings_from(tree, name, module) for name, module in _BANNED.items()} + return super().check(tree) + + def visit_Call(self, node: ast.Call) -> None: + if self._in_scope: + for class_name, names in self._names.items(): + spelled = constructor_name(node.func, names, class_name) + if spelled is not None: + self.violation(node, f"architectural violation: '{spelled}(...)' built in an adapter — {_FIX}") + break + self.generic_visit(node) diff --git a/tests/lint/rules/ce073_create_subprocess_explicit_stdin.py b/tests/lint/rules/ce073_create_subprocess_explicit_stdin.py new file mode 100644 index 00000000..35f2fdf9 --- /dev/null +++ b/tests/lint/rules/ce073_create_subprocess_explicit_stdin.py @@ -0,0 +1,56 @@ +"""CE073: ``asyncio.create_subprocess_exec`` / ``create_subprocess_shell`` must pass ``stdin=``. + +The defect: ``pi`` and ``opencode`` both read a non-TTY stdin TO EOF before they emit +anything. Neither adapter passed ``stdin=``, so the CLI inherited the parent's stdin, and a +``coder-eval run`` whose own stdin was a pipe that stayed open (a backgrounded or +tool-spawned batch) stalled every turn with zero events until the 300 s ``turn_timeout``. +Reproduced end to end on 2026-09-16: stdin held open → ``ERROR`` after the timeout with 0 +commands; stdin on ``/dev/null`` → ``SUCCESS`` in 10 s. The same inheritance reached the +task's ``pre_run``/``post_run`` shell commands, where an authored ``read`` hangs the task. + +Fires, anywhere under ``src/coder_eval/``, on an ``asyncio.create_subprocess_exec`` / +``create_subprocess_shell`` call (attribute form or a bare imported name) with no +``stdin=`` keyword. It requires a DECISION, not ``DEVNULL``: ``stdin=PIPE`` for a caller +that writes to the child passes. Sibling of CE015 (``limit=``); one invariant per id. + +Blind spots: synchronous ``subprocess.run`` / ``Popen`` (which includes +``Sandbox.run_command``, running task-authored shell commands), ``loop.subprocess_exec`` / +``subprocess_shell``, ``anyio.open_process`` / ``run_process``, an explicit ``stdin=None`` +(which still inherits), and a call through ``**kwargs``. +""" + +import ast +import re + +from tests.lint.rules.base import BaseRule + + +_SRC_ROOT = re.compile(r"(?:^|[/\\])src[/\\]coder_eval[/\\]") +_SPAWNERS = frozenset({"create_subprocess_exec", "create_subprocess_shell"}) + + +def _spawner_name(func: ast.expr) -> str | None: + if isinstance(func, ast.Attribute) and func.attr in _SPAWNERS: + return func.attr + if isinstance(func, ast.Name) and func.id in _SPAWNERS: + return func.id + return None + + +class CreateSubprocessExplicitStdin(BaseRule): + id = "CE073" + + def __init__(self, filepath: str) -> None: + super().__init__(filepath) + self._in_scope = bool(_SRC_ROOT.search(filepath)) + + def visit_Call(self, node: ast.Call) -> None: + name = _spawner_name(node.func) + if self._in_scope and name is not None and not any(kw.arg == "stdin" for kw in node.keywords): + self.violation( + node, + f"{name} without stdin= inherits this process's stdin; a child that reads it to EOF stalls " + + "while the parent's stdin stays open. Pass stdin= explicitly (asyncio.subprocess.DEVNULL " + + "unless you write to the child).", + ) + self.generic_visit(node) diff --git a/tests/lint/runner.py b/tests/lint/runner.py index 0b83a950..bee5599c 100644 --- a/tests/lint/runner.py +++ b/tests/lint/runner.py @@ -36,14 +36,12 @@ from tests.lint.rules.ce056_no_container_env_literal import NoContainerEnvLiteral from tests.lint.rules.ce057_sidecar_shim_stdlib_only import SidecarShimStdlibOnly from tests.lint.rules.ce058_no_timing_literal import NoTimingLiteral -from tests.lint.rules.ce059_generation_window_is_two_reads import GenerationWindowIsTwoReads -from tests.lint.rules.ce060_message_id_declared import MessageIdDeclared -from tests.lint.rules.ce061_window_via_close_window import WindowViaCloseWindow -from tests.lint.rules.ce063_no_busy_ms_in_agents import NoBusyMsInAgents -from tests.lint.rules.ce064_turn_bracket_on_the_clock import TurnBracketOnTheClock from tests.lint.rules.ce066_no_report_imports_in_core import NoReportImportsInCore from tests.lint.rules.ce068_no_kind_names_in_kernel import NoKindNamesInKernel from tests.lint.rules.ce070_no_cap_or_skill_scan_in_adapters import NoCapOrSkillScanInAdapters +from tests.lint.rules.ce071_price_turn_only import PriceTurnOnly +from tests.lint.rules.ce072_emitter_sole_writer import EmitterSoleWriter +from tests.lint.rules.ce073_create_subprocess_explicit_stdin import CreateSubprocessExplicitStdin from tests.lint.rules.no_agent_timing_access import NoAgentTimingAccess from tests.lint.rules.no_blocking_io_in_async import NoBlockingIoInAsync from tests.lint.rules.no_cli_imports_in_core import NoCliImportsInCore @@ -60,14 +58,16 @@ from tests.lint.violation import Violation -# CE062 IS DELIBERATELY UNUSED and must stay that way — the ids above jump 061 -# to 063. It was claimed during the turn-timing work and then folded into CE063 -# rather than shipped. An id is a permanent documentation anchor: a suppression -# comment carrying 062 in an older branch, review or commit message must never -# start meaning something new. +# CE059, CE060, CE061, CE062, CE063 AND CE064 ARE RETIRED and must never be reused. +# CE062 was claimed during the turn-timing work and folded into CE063 rather than +# shipped; the other five guarded the per-adapter turn accumulators that TurnEmitter +# replaced, and CE072 (EmitterSoleWriter) keeps adapters from growing one back. An id +# is a permanent documentation anchor: a suppression comment carrying one of these in +# an older branch, review or commit message must never start meaning something new. # -# Claim 071 next (069 is TestCE069HarnessParityTable, 070 is NoCapOrSkillScanInAdapters). NOTE 065 IS TAKEN and is -# not in ALL_RULES: doc-surface and +# Claim 074 next (069 is TestCE069HarnessParityTable, 070 is NoCapOrSkillScanInAdapters, 071 is +# PriceTurnOnly, 072 is EmitterSoleWriter, 073 is CreateSubprocessExplicitStdin). +# NOTE 065 IS TAKEN and is not in ALL_RULES: doc-surface and # whole-tree rules are `@pytest.mark.lint` classes in tests/test_custom_lint.py # rather than BaseRules, so the `_rule_ids` uniqueness assert below cannot see # them. Enumerating them here is how this note fell behind CE044, so grep @@ -118,14 +118,12 @@ NoRunRecordFilenameLiteral, SidecarShimStdlibOnly, NoTimingLiteral, - GenerationWindowIsTwoReads, - MessageIdDeclared, - WindowViaCloseWindow, - NoBusyMsInAgents, - TurnBracketOnTheClock, NoReportImportsInCore, NoKindNamesInKernel, NoCapOrSkillScanInAdapters, + PriceTurnOnly, + EmitterSoleWriter, + CreateSubprocessExplicitStdin, ] # Anti-shadow invariant (mirrors AgentRegistry / register_pricing): every CE rule diff --git a/tests/test_agent.py b/tests/test_agent.py index 7aa151a6..abffcfae 100644 --- a/tests/test_agent.py +++ b/tests/test_agent.py @@ -2,7 +2,6 @@ import logging import tempfile -import time from pathlib import Path from unittest.mock import MagicMock, patch @@ -11,7 +10,6 @@ from coder_eval.agent import AgentState from coder_eval.agents.claude_code_agent import ClaudeCodeAgent -from coder_eval.errors import AgentCrashError, TurnTimeoutError from coder_eval.models import AgentKind, parse_agent_config from coder_eval.streaming.events import AgentEndEvent, AgentEndStatus, StopReason @@ -31,102 +29,29 @@ def test_claude_agent_initialization(): assert agent.get_state() == AgentState.WORKING -def test_pending_turn_defaults_to_none(): - """Fresh agent has pending_turn = None (slot is empty at rest).""" - config = parse_agent_config(type=AgentKind.CLAUDE_CODE, permission_mode="acceptEdits") - agent = ClaudeCodeAgent(config) - assert agent.pending_turn is None - - -@pytest.mark.asyncio -async def test_discard_pending_turn_clears_slot_and_decrements(): - """discard_pending_turn clears the slot and rolls back _iteration once.""" - from coder_eval.models import TurnRecord - - config = parse_agent_config(type=AgentKind.CLAUDE_CODE, permission_mode="acceptEdits") - agent = ClaudeCodeAgent(config) - - partial = TurnRecord(iteration=1, user_input="p", agent_output="", crashed=True) - agent._iteration = 1 - agent.pending_turn = partial - - await agent.discard_pending_turn() - - assert agent.pending_turn is None - assert agent._iteration == 0 - - -@pytest.mark.asyncio -async def test_discard_pending_turn_idempotent(): - """discard_pending_turn is a no-op when pending_turn is already None.""" - config = parse_agent_config(type=AgentKind.CLAUDE_CODE, permission_mode="acceptEdits") - agent = ClaudeCodeAgent(config) - - assert agent.pending_turn is None - assert agent._iteration == 0 - - # First call: nothing to discard — counter must not go negative. - await agent.discard_pending_turn() - assert agent.pending_turn is None - assert agent._iteration == 0 - - # Second call after a real discard: still a no-op. - from coder_eval.models import TurnRecord - - partial = TurnRecord(iteration=2, user_input="p", agent_output="", crashed=True) - agent._iteration = 2 - agent.pending_turn = partial - await agent.discard_pending_turn() # real discard - await agent.discard_pending_turn() # idempotent second call - assert agent.pending_turn is None - assert agent._iteration == 1 # decremented once, not twice - +def test_the_deleted_turn_side_channel_is_gone_from_the_agent_base(): + """The outcome replaced the base's per-turn side channel; none of its names survive. -@pytest.mark.asyncio -async def test_discard_pending_turn_rolls_back_when_partial_build_failed(): - """If _set_pending swallowed an exception and left pending_turn=None, discard - must still roll back the iteration counter. - - Regression: previously the rollback gated on (pending_turn is not None), so - a swallowed partial-build exception caused _iteration to drift permanently - higher on every double-failure. + Built from parts so a repo grep for the deleted names does not match this test. """ - config = parse_agent_config(type=AgentKind.CLAUDE_CODE, permission_mode="acceptEdits") - agent = ClaudeCodeAgent(config) - - # Simulate communicate() incrementing the counter and then crashing before - # _set_pending could finish (partial-build exception swallowed → pending_turn None). - agent._iteration = 5 - agent._iteration_was_incremented = True - agent.pending_turn = None - - await agent.discard_pending_turn() - assert agent._iteration == 4, "rollback must fire even when pending_turn is None" - assert agent._iteration_was_incremented is False - - # Second call is idempotent — neither signal fires. - await agent.discard_pending_turn() - assert agent._iteration == 4 - - -@pytest.mark.asyncio -async def test_stop_clears_pending_turn(): - """stop() clears pending_turn so stale partials don't leak between runs.""" - import tempfile - - from coder_eval.models import TurnRecord - - config = parse_agent_config(type=AgentKind.CLAUDE_CODE, permission_mode="acceptEdits") - agent = ClaudeCodeAgent(config) - - with tempfile.TemporaryDirectory() as tmpdir: - await agent.start(tmpdir) - partial = TurnRecord(iteration=1, user_input="p", agent_output="", crashed=True) - agent.pending_turn = partial - - await agent.stop() - - assert agent.pending_turn is None + import coder_eval.agent as agent_module + from coder_eval.agent import Agent + + deleted = [ + "pending" + "_turn", + "_legacy" + "_outcome", + "discard_pending" + "_turn", + "_iteration_was" + "_incremented", + "_capture_partial" + "_turn", + "_finalize_and" + "_raise_timeout", + "_finalize_and" + "_raise_crash", + "_finalize_external" + "_cancel", + "_begin" + "_turn", + "_end_turn" + "_ok", + "_iter" + "ation", + ] + assert [name for name in deleted if hasattr(Agent, name)] == [] + assert not hasattr(agent_module, "_Finalize" + "Fn") @pytest.mark.asyncio @@ -210,7 +135,7 @@ async def mock_query(prompt, options): with tempfile.TemporaryDirectory() as tmpdir: await agent.start(tmpdir, env_path_prepend=env_path_prepend) with patch("coder_eval.agents.claude_code_agent.query", mock_query): - await agent.communicate("hello") + await agent.communicate("hello", iteration=1) return captured_options @@ -223,7 +148,7 @@ async def test_sdk_options_max_turns_reaches_claude_agent_options(): with tempfile.TemporaryDirectory() as tmpdir: await agent.start(tmpdir) - options, _transport, _model = agent._build_claude_query("hello", None, lambda _line: None) + options, _transport, _model = agent._build_claude_query("hello", 1, None, lambda _line: None) assert options.max_turns == 3 @@ -969,7 +894,9 @@ def test_format_messages_system_message_subclasses_are_filtered(): @pytest.mark.asyncio async def test_claude_agent_process_error_includes_stderr(): - """Test that ProcessError is caught and its stderr is included in RuntimeError.""" + """Test that ProcessError is caught and its stderr is included in the CRASHED outcome's error.""" + import re + config = parse_agent_config( type=AgentKind.CLAUDE_CODE, permission_mode="acceptEdits", @@ -986,18 +913,124 @@ async def mock_query(*args, **kwargs): raise ProcessError("process failed", exit_code=1, stderr="Error: invalid config") yield # makes this an async generator - with ( - patch("coder_eval.agents.claude_code_agent.query", mock_query), - pytest.raises(RuntimeError, match=r"CLI process failed \(exit code 1\): Error: invalid config"), - ): - await agent.communicate("do something") + with patch("coder_eval.agents.claude_code_agent.query", mock_query): + outcome = await agent.communicate("do something", iteration=1) + + assert outcome.status is AgentEndStatus.CRASHED + assert outcome.error is not None + assert re.search(r"CLI process failed \(exit code 1\): Error: invalid config", outcome.error) + assert agent.get_state() == AgentState.ERROR + + +class TestClaudeCommunicateOutcomes: + """A failed or interrupted turn is a ``TurnOutcome`` (or one end, then a re-raised cancel).""" + + class _Recorder: + def __init__(self) -> None: + self.events: list = [] + + def on_event(self, event) -> None: + self.events.append(event) + + async def test_a_watchdog_timeout_returns_timeout_and_leaves_the_caller_uncancelled(self): + """The real watchdog cancels the pump's child task, never the task awaiting ``communicate``.""" + import asyncio + + agent = ClaudeCodeAgent(parse_agent_config(type=AgentKind.CLAUDE_CODE, permission_mode="acceptEdits")) + + async def blocking_query(prompt, options, transport=None): + await asyncio.sleep(30) + yield None # pragma: no cover + + with tempfile.TemporaryDirectory() as tmpdir: + await agent.start(tmpdir) + with patch("coder_eval.agents.claude_code_agent.query", blocking_query): + outcome = await agent.communicate("go", iteration=1, timeout=0.2) + + assert outcome.status is AgentEndStatus.TIMEOUT + assert outcome.record.crashed is True + assert outcome.error is not None and "timed out" in outcome.error + assert agent.get_state() == AgentState.ERROR + caller = asyncio.current_task() + assert caller is not None and caller.cancelling() == 0 + await asyncio.sleep(0) # a stray cancel would land on this await + + async def test_a_process_error_returns_crashed_with_the_captured_stderr(self): + agent = ClaudeCodeAgent(parse_agent_config(type=AgentKind.CLAUDE_CODE, permission_mode="acceptEdits")) + + async def failing_query(prompt, options, transport=None): + options.stderr("fatal: invalid API key") + raise ProcessError("Command failed", exit_code=1, stderr="Check stderr output for details") + yield None # pragma: no cover + + recorder = self._Recorder() + with tempfile.TemporaryDirectory() as tmpdir: + await agent.start(tmpdir) + with patch("coder_eval.agents.claude_code_agent.query", failing_query): + outcome = await agent.communicate("go", iteration=1, stream_callback=recorder) + + assert outcome.status is AgentEndStatus.CRASHED + assert outcome.error == "CLI process failed (exit code 1): fatal: invalid API key" + assert outcome.record.crashed is True + ends = [e for e in recorder.events if isinstance(e, AgentEndEvent)] + assert [(e.status, e.crash_reason) for e in ends] == [(AgentEndStatus.CRASHED, outcome.error)] + + async def test_an_external_cancel_ends_the_turn_once_then_propagates(self): + import asyncio + + agent = ClaudeCodeAgent(parse_agent_config(type=AgentKind.CLAUDE_CODE, permission_mode="acceptEdits")) + streaming = asyncio.Event() + + class _Assistant: + content = "working" + model = "mock-model" + + async def slow_query(prompt, options, transport=None): + yield _Assistant() + streaming.set() + await asyncio.sleep(30) + + recorder = self._Recorder() + with tempfile.TemporaryDirectory() as tmpdir: + await agent.start(tmpdir) + with patch("coder_eval.agents.claude_code_agent.query", slow_query): + task = asyncio.ensure_future(agent.communicate("go", iteration=1, stream_callback=recorder)) + await streaming.wait() + task.cancel() + with pytest.raises(asyncio.CancelledError): + await task + + ends = [e for e in recorder.events if isinstance(e, AgentEndEvent)] + assert [(e.status, e.crash_reason) for e in ends] == [(AgentEndStatus.CRASHED, "turn cancelled")] + assert recorder.events[-1] is ends[0] + assert agent.get_state() == AgentState.ERROR + + async def test_a_cancel_raised_inside_the_sdk_is_a_crash_outcome(self): + import asyncio + + agent = ClaudeCodeAgent(parse_agent_config(type=AgentKind.CLAUDE_CODE, permission_mode="acceptEdits")) + + async def cancelling_query(prompt, options, transport=None): + raise asyncio.CancelledError() + yield None # pragma: no cover + + with tempfile.TemporaryDirectory() as tmpdir: + await agent.start(tmpdir) + with patch("coder_eval.agents.claude_code_agent.query", cancelling_query): + outcome = await agent.communicate("go", iteration=1) + assert outcome.status is AgentEndStatus.CRASHED + assert outcome.record.crashed is True assert agent.get_state() == AgentState.ERROR + caller = asyncio.current_task() + assert caller is not None and caller.cancelling() == 0 @pytest.mark.asyncio async def test_claude_agent_process_error_no_stderr_at_all(): """Test that ProcessError with no stderr and no stderr_lines shows sentinel message.""" + import re + config = parse_agent_config( type=AgentKind.CLAUDE_CODE, permission_mode="acceptEdits", @@ -1013,11 +1046,12 @@ async def mock_query(*args, **kwargs): raise ProcessError("process failed", exit_code=None, stderr=None) yield # makes this an async generator - with ( - patch("coder_eval.agents.claude_code_agent.query", mock_query), - pytest.raises(RuntimeError, match=r"CLI process failed \(exit code None\): No stderr captured"), - ): - await agent.communicate("do something") + with patch("coder_eval.agents.claude_code_agent.query", mock_query): + outcome = await agent.communicate("do something", iteration=1) + + assert outcome.status is AgentEndStatus.CRASHED + assert outcome.error is not None + assert re.search(r"CLI process failed \(exit code None\): No stderr captured", outcome.error) @pytest.mark.asyncio @@ -1058,12 +1092,12 @@ async def mock_query(prompt, options): with patch("coder_eval.agents.claude_code_agent.query", mock_query): # First call: no session_id yet - await agent.communicate("first prompt") + await agent.communicate("first prompt", iteration=1) assert captured_options[0].resume is None assert agent._session_id == "test-session-abc" # Second call: should pass session_id as resume - await agent.communicate("second prompt") + await agent.communicate("second prompt", iteration=2) assert captured_options[1].resume == "test-session-abc" @@ -1102,7 +1136,7 @@ async def mock_ok(prompt, options): yield ResultMessage(session_id="good-session", is_error=False) with patch("coder_eval.agents.claude_code_agent.query", mock_ok): - await agent.communicate("clean turn") + await agent.communicate("clean turn", iteration=1) assert agent._session_id == "good-session" # Second: an errored turn arriving with a NEW session_id must NOT @@ -1114,7 +1148,7 @@ async def mock_err(prompt, options): yield ResultMessage(session_id="poisoned-session", is_error=True) with patch("coder_eval.agents.claude_code_agent.query", mock_err): - await agent.communicate("errored turn") + await agent.communicate("errored turn", iteration=2) assert agent._session_id == "good-session" @@ -1154,11 +1188,11 @@ async def mock_query(prompt, options): yield ResultMessage(session_id=None) with patch("coder_eval.agents.claude_code_agent.query", mock_query): - await agent.communicate("first prompt") + await agent.communicate("first prompt", iteration=1) assert agent._session_id is None # Second call: resume should be None (fresh session) - await agent.communicate("second prompt") + await agent.communicate("second prompt", iteration=2) assert captured_options[1].resume is None @@ -1202,15 +1236,15 @@ async def mock_query(prompt, options): yield ResultMessage(session_id=f"session-{call_count}") with patch("coder_eval.agents.claude_code_agent.query", mock_query): - await agent.communicate("first prompt") + await agent.communicate("first prompt", iteration=1) assert agent._session_id == "session-1" - await agent.communicate("second prompt") + await agent.communicate("second prompt", iteration=2) assert captured_options[1].resume == "session-1" assert agent._session_id == "session-2" # Third call should use the rotated session_id - await agent.communicate("third prompt") + await agent.communicate("third prompt", iteration=3) assert captured_options[2].resume == "session-2" @@ -1248,7 +1282,7 @@ async def mock_query_ok(prompt, options): yield ResultMessage(session_id="good-session") with patch("coder_eval.agents.claude_code_agent.query", mock_query_ok): - await agent.communicate("first prompt") + await agent.communicate("first prompt", iteration=1) assert agent._session_id == "good-session" # Second call raises an error mid-stream @@ -1256,11 +1290,12 @@ async def mock_query_error(prompt, options): raise RuntimeError("SDK connection lost") yield - with ( - patch("coder_eval.agents.claude_code_agent.query", mock_query_error), - pytest.raises(RuntimeError, match="SDK connection lost"), - ): - await agent.communicate("second prompt") + with patch("coder_eval.agents.claude_code_agent.query", mock_query_error): + outcome = await agent.communicate("second prompt", iteration=2) + + assert outcome.status is AgentEndStatus.CRASHED + assert outcome.error is not None + assert "SDK connection lost" in outcome.error # session_id should still be the value from the successful call assert agent._session_id == "good-session" @@ -1385,8 +1420,10 @@ async def mock_query(prompt, options): ) with patch("coder_eval.agents.claude_code_agent.query", mock_query): - turn = await agent.communicate("hello") + outcome = await agent.communicate("hello", iteration=1) + assert outcome.status is AgentEndStatus.COMPLETED + turn = outcome.record assert turn.result_summary is not None assert turn.result_summary.is_error is False assert turn.result_summary.subtype == "success" @@ -1398,12 +1435,11 @@ async def mock_query(prompt, options): @pytest.mark.asyncio async def test_claude_agent_crash_preserves_partial_turn_record(): - """When communicate() fails mid-turn, agent.pending_turn carries a partial + """When communicate() fails mid-turn, the CRASHED outcome carries a partial TurnRecord populated with tool calls captured before the crash. - This is the whole point of the pending_turn slot + on_attempt_error - plumbing: typed criteria like skill_triggered must still be able to - observe a Skill invocation that happened before the crash. + Typed criteria like skill_triggered must still be able to observe a Skill + invocation that happened before the crash. """ config = parse_agent_config(type=AgentKind.CLAUDE_CODE, permission_mode="acceptEdits") agent = ClaudeCodeAgent(config) @@ -1427,14 +1463,11 @@ async def mock_query(prompt, options, transport=None): with tempfile.TemporaryDirectory() as tmpdir: await agent.start(tmpdir) - with ( - patch("coder_eval.agents.claude_code_agent.query", mock_query), - pytest.raises(AgentCrashError), - ): - await agent.communicate("do the thing") + with patch("coder_eval.agents.claude_code_agent.query", mock_query): + outcome = await agent.communicate("do the thing", iteration=1) - # Slot is populated before the raise; not yet cleared (caller must drain). - partial = agent.pending_turn + assert outcome.status is AgentEndStatus.CRASHED + partial = outcome.record assert partial is not None assert partial.crashed is True assert partial.tool_calls_exhausted is False @@ -1444,14 +1477,8 @@ async def mock_query(prompt, options, transport=None): assert len(partial.commands) == 1 assert partial.commands[0].tool_name == "Skill" assert partial.commands[0].parameters == {"skill": "my_skill"} - # Iteration contract: partial carries the bumped iteration number; the - # counter is NOT rolled back until discard_pending_turn() is called. + # The record carries the caller's iteration. assert partial.iteration == 1 - assert agent._iteration == 1 - - await agent.discard_pending_turn() - assert agent.pending_turn is None - assert agent._iteration == 0 @pytest.mark.asyncio @@ -1473,13 +1500,11 @@ async def mock_query(prompt, options, transport=None): with tempfile.TemporaryDirectory() as tmpdir: await agent.start(tmpdir) - with ( - patch("coder_eval.agents.claude_code_agent.query", mock_query), - pytest.raises(AgentCrashError), - ): - await agent.communicate("go") + with patch("coder_eval.agents.claude_code_agent.query", mock_query): + outcome = await agent.communicate("go", iteration=1) - partial = agent.pending_turn + assert outcome.status is AgentEndStatus.CRASHED + partial = outcome.record assert partial is not None assert partial.crash_reason is not None # The crash message is truncated at 200 chars, but a short message @@ -1504,11 +1529,11 @@ async def mock_query(prompt, options, transport=None): with ( patch("coder_eval.agents.claude_code_agent.query", mock_query), patch.object(ClaudeCodeAgent, "_timed_out", staticmethod(lambda *a, **k: True)), - pytest.raises(TurnTimeoutError), ): - await agent.communicate("go", timeout=42.0) + outcome = await agent.communicate("go", iteration=1, timeout=42.0) - partial = agent.pending_turn + assert outcome.status is AgentEndStatus.TIMEOUT + partial = outcome.record assert partial is not None # Normalised reason: integer-second formatting matches the # orchestrator's defensive fallback so report rendering is consistent. @@ -1517,13 +1542,11 @@ async def mock_query(prompt, options, transport=None): @pytest.mark.asyncio async def test_claude_agent_repeated_crashes_keep_iteration_stable(): - """Consecutive crashes in one orchestrator iteration all carry the same iteration number. + """Consecutive crashes for the same caller-supplied iteration all carry that + same iteration number. - discard_pending_turn() rolls back _iteration after each crash (simulating - what the orchestrator does), so repeated failures in a single logical - orchestrator iteration all stamp the same iteration on their partial records. - A subsequent clean call then advances the counter by one. This is what the - orchestrator's multiple-partials-per-iteration contract relies on. + Retries of one logical turn pass the same ``iteration`` on every attempt, and + the record stamps the number it was given: the agent keeps no counter. """ config = parse_agent_config(type=AgentKind.CLAUDE_CODE, permission_mode="acceptEdits") agent = ClaudeCodeAgent(config) @@ -1556,33 +1579,27 @@ async def clean_query(prompt, options, transport=None): partials: list = [] for _ in range(3): - with ( - patch("coder_eval.agents.claude_code_agent.query", crashing_query), - pytest.raises(AgentCrashError), - ): - await agent.communicate("go") - partials.append(agent.pending_turn) - # Simulate the orchestrator draining and discarding after a failed attempt. - await agent.discard_pending_turn() - assert agent._iteration == 0 + with patch("coder_eval.agents.claude_code_agent.query", crashing_query): + outcome = await agent.communicate("go", iteration=1) + assert outcome.status is AgentEndStatus.CRASHED + partials.append(outcome.record) assert all(p is not None and p.iteration == 1 and p.crashed for p in partials) - # The clean retry advances the counter and produces iteration=1 again, - # so all four records for this logical orchestrator iteration share 1. + # The clean retry passes the SAME iteration number as its failed + # predecessors — a retry of one logical turn, not a new one. with patch("coder_eval.agents.claude_code_agent.query", clean_query): - turn_record = await agent.communicate("go") + outcome = await agent.communicate("go", iteration=1) assert clean_finished + turn_record = outcome.record assert turn_record.iteration == 1 assert turn_record.crashed is False - assert agent._iteration == 1 @pytest.mark.asyncio async def test_claude_agent_timeout_preserves_partial_turn_record(): - """agent.pending_turn carries a partial TurnRecord with pre-kill tool calls - after a TurnTimeoutError. + """A TIMEOUT outcome carries a partial TurnRecord with pre-kill tool calls. Watchdog-killed turns are exactly where observational telemetry is most valuable (an agent that looped on tool calls and ran the wall @@ -1617,20 +1634,16 @@ async def mock_query(prompt, options, transport=None): with ( patch("coder_eval.agents.claude_code_agent.query", mock_query), patch.object(ClaudeCodeAgent, "_timed_out", staticmethod(lambda *a, **k: True)), - pytest.raises(TurnTimeoutError), ): - await agent.communicate("start", timeout=0.01) + outcome = await agent.communicate("start", iteration=1, timeout=0.01) - partial = agent.pending_turn + assert outcome.status is AgentEndStatus.TIMEOUT + partial = outcome.record assert partial is not None assert partial.crashed is True assert len(partial.commands) == 1 assert partial.commands[0].tool_name == "Bash" - # Slot carries the bumped iteration; counter rolls back after discard. assert partial.iteration == 1 - assert agent._iteration == 1 - await agent.discard_pending_turn() - assert agent._iteration == 0 @pytest.mark.asyncio @@ -1681,14 +1694,14 @@ async def mock_query(prompt, options, transport=None): await agent.start(tmpdir) with patch("coder_eval.agents.claude_code_agent.query", mock_query): - # Must NOT raise: error_max_turns is a clean completion path. - turn_record = await agent.communicate("solve something hard", stream_callback=recorder) + # Must NOT crash: error_max_turns is a clean completion path. + outcome = await agent.communicate("solve something hard", iteration=1, stream_callback=recorder) + assert outcome.status is AgentEndStatus.COMPLETED + turn_record = outcome.record assert turn_record.crashed is False assert turn_record.tool_calls_exhausted is False assert [e.status for e in recorder.events if isinstance(e, AgentEndEvent)] == [AgentEndStatus.COMPLETED] - # Iteration counter advances normally on a clean turn (no rollback). - assert agent._iteration == 1 # The ResultMessage details are still captured for diagnostics. assert turn_record.result_summary is not None assert turn_record.result_summary.subtype == "error_max_turns" @@ -1738,12 +1751,13 @@ async def mock_query(prompt, options, transport=None): await agent.start(tmpdir) with patch("coder_eval.agents.claude_code_agent.query", mock_query): - turn_record = await agent.communicate("solve something hard", stream_callback=recorder) + outcome = await agent.communicate("solve something hard", iteration=1, stream_callback=recorder) + assert outcome.status is AgentEndStatus.COMPLETED + turn_record = outcome.record assert turn_record.crashed is False assert turn_record.tool_calls_exhausted is False assert [e.status for e in recorder.events if isinstance(e, AgentEndEvent)] == [AgentEndStatus.COMPLETED] - assert agent._iteration == 1 assert turn_record.result_summary is not None assert turn_record.result_summary.subtype == "error_max_turns" @@ -1785,14 +1799,108 @@ async def mock_query(prompt, options, transport=None): with tempfile.TemporaryDirectory() as tmpdir: await agent.start(tmpdir) with patch("coder_eval.agents.claude_code_agent.query", mock_query): - turn_record = await agent.communicate("go", stream_callback=recorder, should_stop=lambda: reason) + outcome = await agent.communicate("go", iteration=1, stream_callback=recorder, should_stop=lambda: reason) + assert outcome.status is status + turn_record = outcome.record assert dispatched == ["first"] assert turn_record.crashed is False assert turn_record.tool_calls_exhausted is exhausted assert [e.status for e in recorder.events if isinstance(e, AgentEndEvent)] == [status] +def _cap_monitor(max_turns: int): + from coder_eval.models import FileExistsCriterion, RunLimits, SandboxConfig, TaskDefinition + from coder_eval.orchestration.turn_monitor import TurnMonitor + + task = TaskDefinition( + task_id="t", + description="d", + initial_prompt="go", + agent=parse_agent_config(type=AgentKind.CLAUDE_CODE), + sandbox=SandboxConfig(driver="tempdir"), + success_criteria=[FileExistsCriterion(description="c", path="out.txt")], + run_limits=RunLimits(max_turns=max_turns), + ) + return TurnMonitor.for_task(task, arm=False) + + +@pytest.mark.asyncio +async def test_claude_agent_stops_when_the_turn_past_the_model_cap_starts(): + """Turn N+1 starts when its first message arrives; that message is recorded and the loop breaks.""" + from tests._fixtures.golden_streams.claude_fixtures import AssistantMessage, TextBlock + + agent = ClaudeCodeAgent(parse_agent_config(type=AgentKind.CLAUDE_CODE, permission_mode="acceptEdits")) + monitor = _cap_monitor(2) + dispatched: list[str] = [] + + async def mock_query(prompt, options, transport=None): + for i in range(1, 6): + dispatched.append(f"msg-{i}") + yield AssistantMessage([TextBlock(f"reply {i}")], message_id=f"msg-{i}") + + with tempfile.TemporaryDirectory() as tmpdir: + await agent.start(tmpdir) + with patch("coder_eval.agents.claude_code_agent.query", mock_query): + outcome = await agent.communicate( + "go", iteration=1, stream_callback=monitor, should_stop=monitor.should_stop + ) + + assert outcome.status is AgentEndStatus.TOOL_CALLS_EXHAUSTED + assert monitor.stop_reason is StopReason.MODEL_TURN_CAP + assert dispatched == ["msg-1", "msg-2", "msg-3"] + assert outcome.record.tool_calls_exhausted is True + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + ("cap", "status", "dispatched_count"), + [(1, AgentEndStatus.TOOL_CALLS_EXHAUSTED, 7), (2, AgentEndStatus.COMPLETED, 8)], +) +async def test_a_sub_agent_does_not_split_the_main_turn_the_model_cap_counts(cap, status, dispatched_count): + """A sub-agent message closes and re-opens main turn A under the same id, which counts once.""" + from tests._fixtures.golden_streams.claude_fixtures import ( + AssistantMessage, + ResultMessage, + TextBlock, + ToolUseBlock, + UserMessage, + ) + + stream = [ + AssistantMessage([ToolUseBlock("t1", "Task", {"prompt": "look"})], message_id="A"), + AssistantMessage([TextBlock("sub one")], message_id="S1", parent_tool_use_id="t1"), + AssistantMessage([ToolUseBlock("t2", "Bash", {"command": "ls"})], message_id="A"), + UserMessage("t2", False, "ok"), + AssistantMessage([TextBlock("sub two")], message_id="S2", parent_tool_use_id="t1"), + UserMessage("t1", False, "sub result"), + AssistantMessage([TextBlock("done")], message_id="B"), + ResultMessage(), + ] + agent = ClaudeCodeAgent(parse_agent_config(type=AgentKind.CLAUDE_CODE, permission_mode="acceptEdits")) + monitor = _cap_monitor(cap) + dispatched: list[object] = [] + + async def mock_query(prompt, options, transport=None): + for message in stream: + dispatched.append(message) + yield message + + with tempfile.TemporaryDirectory() as tmpdir: + await agent.start(tmpdir) + with patch("coder_eval.agents.claude_code_agent.query", mock_query): + outcome = await agent.communicate( + "go", iteration=1, stream_callback=monitor, should_stop=monitor.should_stop + ) + + assert outcome.status is status + assert len(dispatched) == dispatched_count + assert monitor.model_turns == 2 + statuses = {command.tool_id: command.result_status for command in outcome.record.commands} + assert statuses["t1"] == "success" + assert statuses["t2"] == "success" + + def test_setting_sources_default_is_project(): """When config.setting_sources is None, it defaults to ['project'] at runtime.""" config = parse_agent_config( @@ -1833,18 +1941,21 @@ def test_setting_sources_custom_list(): assert agent.config.setting_sources == ["project", "user"] -class TestClaudeTurnState: - """Unit tests for the per-turn state object extracted from communicate(). +class TestClaudeDecoder: + """Unit tests for the per-turn decoder, driven over a real ``TurnEmitter``. Driving its handler methods directly gives independent coverage of the stream-dispatch logic without standing up a full communicate() turn. """ @staticmethod - def _state(agent): - from coder_eval.agents.claude_code_agent import _ClaudeTurnState - from coder_eval.streaming.callbacks import CompositeStreamCallback - from coder_eval.streaming.collector import EventCollector + def _decoder(agent): + from datetime import datetime + + from coder_eval.agents.claude_code_agent import _ClaudeDecoder + from coder_eval.models import TimingBasis + from coder_eval.streaming.emitter import TurnEmitter + from coder_eval.testing import ScriptedClock events: list = [] @@ -1852,145 +1963,67 @@ class _Collect: def on_event(self, event): events.append(event) - collector = EventCollector() - emit = CompositeStreamCallback([collector, _Collect()]) - state = _ClaudeTurnState( - agent, - emit=emit, - collector=collector, + emitter = TurnEmitter( task_id="claude_code", - user_input="hi", iteration=1, - log=agent._log, - turn_start_time=time.monotonic(), - deadline=None, + prompt="hi", + model="mock-model", + basis=TimingBasis.TURN_CLOCK, + clock=ScriptedClock(datetime(2026, 1, 1)), + sinks=[_Collect()], ) - return state, events + emitter.begin() + return _ClaudeDecoder(agent, emitter, effective_model="mock-model"), events def test_on_assistant_message_records_command_and_emits_tool_start(self): from coder_eval.streaming.events import ToolStartEvent from tests._fixtures.golden_streams.claude_fixtures import AssistantMessage, ToolUseBlock agent = ClaudeCodeAgent(parse_agent_config(type=AgentKind.CLAUDE_CODE, permission_mode="acceptEdits")) - state, events = self._state(agent) + decoder, events = self._decoder(agent) msg = AssistantMessage( [ToolUseBlock("t1", "Bash", {"command": "ls"})], usage={"input_tokens": 10, "output_tokens": 5}, message_id="m1", ) - state.on_assistant_message(msg) + decoder.on_assistant_message(msg) # One AssistantMessage telemetry record captured with its per-message tokens. - assert len(state.sdk_messages) == 1 - assert state.sdk_messages[0].input_tokens == 10 - # The tool_use block registered a pending command + emitted ToolStart. - assert "t1" in state.pending_commands - assert state.pending_commands["t1"]["telemetry"].tool_name == "Bash" + assert len(decoder.transcript) == 1 + assert decoder.transcript[0].input_tokens == 10 + # The tool_use block opened the tool and emitted ToolStart. + assert decoder.opened_tools == {"t1": "Bash"} tool_starts = [e for e in events if isinstance(e, ToolStartEvent)] assert len(tool_starts) == 1 assert tool_starts[0].tool.tool_id == "t1" + assert tool_starts[0].tool.tool_name == "Bash" - def test_on_user_message_resolves_pending_command(self): - from coder_eval.streaming.events import ToolEndEvent, ToolEndStatus + def test_on_user_message_resolves_the_open_tool(self): + from coder_eval.streaming.events import AgentEndStatus, ToolEndEvent, ToolEndStatus from tests._fixtures.golden_streams.claude_fixtures import AssistantMessage, ToolUseBlock, UserMessage agent = ClaudeCodeAgent(parse_agent_config(type=AgentKind.CLAUDE_CODE, permission_mode="acceptEdits")) - state, events = self._state(agent) + decoder, events = self._decoder(agent) - state.on_assistant_message(AssistantMessage([ToolUseBlock("t1", "Bash", {"command": "ls"})], message_id="m1")) - state.on_user_message(UserMessage("t1", False, "file1.py\nfile2.py")) + decoder.on_assistant_message(AssistantMessage([ToolUseBlock("t1", "Bash", {"command": "ls"})], message_id="m1")) + decoder.on_user_message(UserMessage("t1", False, "file1.py\nfile2.py")) - cmd = state.pending_commands["t1"]["telemetry"] - assert cmd.result_status == "success" - assert cmd.result_summary == "file1.py\nfile2.py" + assert decoder.processed_results == {"t1"} tool_ends = [e for e in events if isinstance(e, ToolEndEvent)] assert len(tool_ends) == 1 assert tool_ends[0].status == ToolEndStatus.OK - - -class TestAgentBaseHelpers: - """Unit tests for the shared mid-turn failure kernels on the Agent base.""" - - @staticmethod - def _agent(): - return ClaudeCodeAgent(parse_agent_config(type=AgentKind.CLAUDE_CODE, permission_mode="acceptEdits")) - - def test_finalize_and_raise_timeout(self): - from coder_eval.errors.agent import format_timeout_reason - from coder_eval.streaming.events import AgentEndStatus - - agent = self._agent() - agent._iteration = 3 - calls: list = [] - - def fake_finalize(status, *, crashed, crash_reason): - # Asserting here locks the ORDER: _state must already be ERROR by the - # time finalize runs (i.e. _state=ERROR precedes finalize precedes raise). - assert agent._state == AgentState.ERROR - calls.append((status, crashed, crash_reason)) - - with pytest.raises(TurnTimeoutError) as exc: - agent._finalize_and_raise_timeout(fake_finalize, 30.0) - - # _state=ERROR -> finalize(TIMEOUT, crashed=True, reason) -> raise, in order. - assert agent._state == AgentState.ERROR - assert calls == [(AgentEndStatus.TIMEOUT, True, format_timeout_reason(30.0))] - assert exc.value.timeout_seconds == 30.0 - assert exc.value.iteration == 3 - - def test_finalize_and_raise_crash_truncates_reason_but_raises_full(self): - from coder_eval.errors.agent import truncate_crash_message - from coder_eval.streaming.events import AgentEndStatus - - agent = self._agent() - calls: list = [] - - def fake_finalize(status, *, crashed, crash_reason): - assert agent._state == AgentState.ERROR # order: _state precedes finalize - calls.append((status, crashed, crash_reason)) - - long_message = "x" * 300 - with pytest.raises(AgentCrashError) as exc: - agent._finalize_and_raise_crash(fake_finalize, long_message) - - assert agent._state == AgentState.ERROR - assert calls[0][0] == AgentEndStatus.CRASHED - assert calls[0][1] is True - # crash_reason is truncated for storage... - assert calls[0][2] == truncate_crash_message(long_message) - assert len(calls[0][2]) < len(long_message) - # ...but the raised AgentCrashError carries the message as passed. - assert str(exc.value) == long_message - - def test_capture_partial_turn_sets_pending_from_collector(self): - from coder_eval.streaming.collector import EventCollector - - agent = self._agent() - agent._capture_partial_turn(EventCollector()) - assert agent.pending_turn is not None - - def test_capture_partial_turn_falls_back_to_none_on_build_error(self): - from coder_eval.models import TurnRecord - - agent = self._agent() - # Pre-seed a stale partial to prove it gets cleared on a build failure. - agent.pending_turn = TurnRecord(iteration=1, user_input="p", agent_output="stale", crashed=True) - - class _BadCollector: - def build_turn_record(self): - raise RuntimeError("cannot build") - - agent._capture_partial_turn(_BadCollector()) - assert agent.pending_turn is None + (cmd,) = decoder.end(AgentEndStatus.COMPLETED).record.commands + assert cmd.result_status == "success" + assert cmd.result_summary == "file1.py\nfile2.py" @pytest.mark.asyncio async def test_watchdog_callback_targets_its_own_turn_transport_across_calls(): - """A turn's watchdog on_timeout closure must capture its OWN transport - (``watchdog_target``), never ``self._active_transport``. + """A turn's watchdog on_timeout closure must capture its OWN transport, + never ``self._active_transport``. - Regression guard for the Phase-2 state-object extraction: a stale turn-1 + A stale turn-1 watchdog firing after turn 2 has started (and turn 1's transport reference cleared from ``self._active_transport``) must still kill turn 1's subprocess and never touch turn 2's. This cross-turn property is invisible to the @@ -2001,6 +2034,8 @@ async def test_watchdog_callback_targets_its_own_turn_transport_across_calls(): captured_callbacks: list = [] class _FakeWatchdog: + fired = False + def __init__(self, *, timeout_seconds=None, on_timeout=None, asyncio_task_to_cancel=None, label=""): captured_callbacks.append(on_timeout) @@ -2027,15 +2062,15 @@ async def mock_query(prompt, options, transport=None): with tempfile.TemporaryDirectory() as tmpdir: await agent.start(tmpdir) with ( - patch("coder_eval.agents.claude_code_agent.ThreadedWatchdog", _FakeWatchdog), + patch("coder_eval.agents.watchdog.ThreadedWatchdog", _FakeWatchdog), patch( "coder_eval.agents.claude_code_agent.SubprocessCLITransport", side_effect=[transport_a, transport_b], ), patch("coder_eval.agents.claude_code_agent.query", mock_query), ): - await agent.communicate("turn 1", timeout=30.0) - await agent.communicate("turn 2", timeout=30.0) + await agent.communicate("turn 1", iteration=1, timeout=30.0) + await agent.communicate("turn 2", iteration=2, timeout=30.0) assert len(captured_callbacks) == 2 # Both turns finished, so self._active_transport is None. Fire turn 1's diff --git a/tests/test_agent_config_registry_dispatch.py b/tests/test_agent_config_registry_dispatch.py index 316fb429..9077613d 100644 --- a/tests/test_agent_config_registry_dispatch.py +++ b/tests/test_agent_config_registry_dispatch.py @@ -8,6 +8,7 @@ import pytest from pydantic import ValidationError +from coder_eval.agents.registry import SPI_VERSION from coder_eval.models import ( AgentKind, BaseAgentConfig, @@ -152,7 +153,7 @@ class _PluginAgent: def __init__(self, config, route=None, **kwargs): self.config = config - AgentRegistry.register("plugin-kind", _PluginAgentConfig)(_PluginAgent) + AgentRegistry.register("plugin-kind", _PluginAgentConfig, spi_version=SPI_VERSION)(_PluginAgent) try: yield finally: diff --git a/tests/test_agent_golden_master.py b/tests/test_agent_golden_master.py index 216bbe80..f1d608af 100644 --- a/tests/test_agent_golden_master.py +++ b/tests/test_agent_golden_master.py @@ -3,7 +3,7 @@ This is the safety net for decomposing ``ClaudeCodeAgent.communicate`` and ``CodexAgent._run_turn_with_streaming``: each scenario replays a recorded SDK event stream through ``communicate()`` and asserts the resulting -``TurnRecord`` / ``pending_turn`` is byte-identical (post-scrub) to a committed +``TurnRecord`` (crashed or not) is byte-identical (post-scrub) to a committed JSON snapshot. The decomposition must not change any snapshot. Regenerate the snapshots after an INTENTIONAL behavior change with:: @@ -24,6 +24,7 @@ import pytest from coder_eval.models import AgentKind +from coder_eval.testing import assert_stream_balanced from tests._fixtures.golden_streams import assert_reconciliation, assert_timing_captured, scrub from tests._fixtures.golden_streams.antigravity_fixtures import ANTIGRAVITY_SCENARIOS, run_antigravity_scenario from tests._fixtures.golden_streams.claude_fixtures import CLAUDE_SCENARIOS, run_claude_scenario @@ -55,33 +56,14 @@ # deadline flips) so the deadline break is deterministic. The window # is zero by fixture construction, not by anything the harness did. "claude_i_in_loop_deadline_break", + "codex_a_agent_message_only", # the SDK stream carries no item stamps: an unmeasured generation "codex_g_items_rebuild", # rollout rebuild: Turn items carry no timestamps # Codex emissions whose ENTIRE measurable window was tool execution. # The window is subtracted down to 0 because that is the honest # answer, not because nothing was recorded — see the generation-window - # subtraction in codex_agent._flush_message. + # subtraction in codex_agent.flush. "codex_d_cross_flush_is_error", # flush lands before the tool completes: zero-width window "codex_e_orphan_tool", # the tool never completes, so the window never opens - # Same shape, reached from the opposite direction. This scenario injects - # a 5 ms CLI tool interval into a replay whose whole turn is well under - # one millisecond, so the tool spans BOTH windows entirely and the - # central subtraction takes each down to a measured 0.0. It is the tool - # interval that is fictional, not the subtraction — which is why the - # scenario is in FICTIONAL_DURATIONS too. - # - # BE HONEST ABOUT WHAT IS LEFT. With both exemptions on, this snapshot - # asserts neither the identity nor a positive window, and it does NOT - # record the tiling the scenario is named for — `SCRUB_KEYS` masks - # `started_at`, `completed_at` and `generation_duration_ms`, so nothing - # about where a window opened survives into the JSON. What it still - # pins is the STRUCTURE: two assistant messages, their content blocks, - # their token buckets, and one resolved command. OpenCode's tiling is - # asserted where it can be — `tests/test_timing_identity_contract.py` - # (scripted clock, ms-exact) and - # `tests/test_opencode_agent.py::TestGenerationWindowsTileTheTurn`. - # `pi_c_multi_turn_tiling` is the same scenario shape on a harness whose - # stamps come from its own clock, and it needs neither exemption. - "opencode_c_multi_step_tiling", } ) @@ -100,7 +82,7 @@ def _expect_window(harness: str, scenario_name: str) -> bool: # scenario, and the codex/opencode ones that inject nothing — is checked. # # The last two entries were ADDED to buy stability, and the trade is worth -# stating. They previously injected NO stamps at all, so `_flush_message` took +# stating. They previously injected NO stamps at all, so `flush` took # `_ms_to_dt(None)` for both window bounds — two adjacent `datetime.now()` # reads, which collide at microsecond resolution often enough that # `assert_timing_captured`'s `completed_at > started_at` failed roughly one run @@ -117,6 +99,13 @@ def _expect_window(harness: str, scenario_name: str) -> bool: "codex_f_collab_fallback", # 900 ms collab wait "codex_h_no_turn_completed_crash", # 200 ms of item time — see below "opencode_b_tool_call_resolved", # 17 ms tool interval + # OpenCode bounds every window and tool on the CLI's own envelope and + # `state.time` epoch stamps (timing_basis cli_epoch_ms), scripted in whole + # milliseconds, while the host bracket spans a sub-millisecond replay. + "opencode_a_single_text_turn", + "opencode_d_orphaned_tool", + "opencode_e_error_after_generation", + "opencode_f_captured_stream", # a real 2.3 s CLI timeline # 5 ms tool interval, injected as CLI epoch stamps. OpenCode takes its # tool bounds from the CLI payload rather than from its own clock, so # every scenario of this harness that resolves a tool injects them — @@ -161,7 +150,7 @@ def _compare_or_regen(name: str, actual_scrubbed: dict[str, Any]) -> None: @pytest.mark.asyncio @pytest.mark.parametrize("scenario", CLAUDE_SCENARIOS, ids=lambda s: s.name) async def test_claude_golden(scenario, tmp_path): - raw = await run_claude_scenario(scenario, str(tmp_path)) + raw, _ = await run_claude_scenario(scenario, str(tmp_path)) # Reconciliation is asserted on the UNscrubbed dump (token buckets are never # scrubbed, but cost/timestamps are — assert before masking to be explicit). assert_reconciliation(raw) @@ -177,7 +166,7 @@ async def test_claude_golden(scenario, tmp_path): @pytest.mark.asyncio @pytest.mark.parametrize("scenario", CODEX_SCENARIOS, ids=lambda s: s.name) async def test_codex_golden(scenario, tmp_path): - raw = await run_codex_scenario(scenario, str(tmp_path)) + raw, _ = await run_codex_scenario(scenario, str(tmp_path)) assert_reconciliation(raw) assert_timing_captured( raw, @@ -191,7 +180,7 @@ async def test_codex_golden(scenario, tmp_path): @pytest.mark.parametrize("scenario", CLAUDE_SCENARIOS, ids=lambda s: s.name) async def test_claude_reconciliation_invariant(scenario, tmp_path): """The per-bucket reconciliation invariant holds for every Claude snapshot.""" - raw = await run_claude_scenario(scenario, str(tmp_path)) + raw, _ = await run_claude_scenario(scenario, str(tmp_path)) assert_reconciliation(raw) @@ -200,14 +189,14 @@ async def test_claude_reconciliation_invariant(scenario, tmp_path): @pytest.mark.parametrize("scenario", CODEX_SCENARIOS, ids=lambda s: s.name) async def test_codex_reconciliation_invariant(scenario, tmp_path): """The per-bucket reconciliation invariant holds for every Codex snapshot.""" - raw = await run_codex_scenario(scenario, str(tmp_path)) + raw, _ = await run_codex_scenario(scenario, str(tmp_path)) assert_reconciliation(raw) @pytest.mark.asyncio @pytest.mark.parametrize("scenario", ANTIGRAVITY_SCENARIOS, ids=lambda s: s.name) async def test_antigravity_golden(scenario, tmp_path): - raw = await run_antigravity_scenario(scenario, str(tmp_path)) + raw, _ = await run_antigravity_scenario(scenario, str(tmp_path)) assert_reconciliation(raw) assert_timing_captured( raw, @@ -221,13 +210,13 @@ async def test_antigravity_golden(scenario, tmp_path): @pytest.mark.parametrize("scenario", ANTIGRAVITY_SCENARIOS, ids=lambda s: s.name) async def test_antigravity_reconciliation_invariant(scenario, tmp_path): """The per-bucket reconciliation invariant holds for every Antigravity snapshot.""" - assert_reconciliation(await run_antigravity_scenario(scenario, str(tmp_path))) + assert_reconciliation((await run_antigravity_scenario(scenario, str(tmp_path)))[0]) @pytest.mark.asyncio @pytest.mark.parametrize("scenario", OPENCODE_SCENARIOS, ids=lambda s: s.name) async def test_opencode_golden(scenario, tmp_path): - raw = await run_opencode_scenario(scenario, str(tmp_path)) + raw, _ = await run_opencode_scenario(scenario, str(tmp_path)) assert_reconciliation(raw) assert_timing_captured( raw, @@ -241,13 +230,13 @@ async def test_opencode_golden(scenario, tmp_path): @pytest.mark.parametrize("scenario", OPENCODE_SCENARIOS, ids=lambda s: s.name) async def test_opencode_reconciliation_invariant(scenario, tmp_path): """The per-bucket reconciliation invariant holds for every OpenCode snapshot.""" - assert_reconciliation(await run_opencode_scenario(scenario, str(tmp_path))) + assert_reconciliation((await run_opencode_scenario(scenario, str(tmp_path)))[0]) @pytest.mark.asyncio @pytest.mark.parametrize("scenario", PI_SCENARIOS, ids=lambda s: s.name) async def test_pi_golden(scenario, tmp_path): - raw = await run_pi_scenario(scenario, str(tmp_path)) + raw, _ = await run_pi_scenario(scenario, str(tmp_path)) assert_reconciliation(raw) assert_timing_captured( raw, @@ -261,7 +250,39 @@ async def test_pi_golden(scenario, tmp_path): @pytest.mark.parametrize("scenario", PI_SCENARIOS, ids=lambda s: s.name) async def test_pi_reconciliation_invariant(scenario, tmp_path): """The per-bucket reconciliation invariant holds for every Pi snapshot.""" - assert_reconciliation(await run_pi_scenario(scenario, str(tmp_path))) + assert_reconciliation((await run_pi_scenario(scenario, str(tmp_path)))[0]) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("scenario", CLAUDE_SCENARIOS, ids=lambda s: s.name) +async def test_claude_stream_balanced(scenario, tmp_path): + assert_stream_balanced((await run_claude_scenario(scenario, str(tmp_path)))[1]) + + +@pytest.mark.skipif(not _HAS_CODEX, reason="openai_codex extra not installed") +@pytest.mark.asyncio +@pytest.mark.parametrize("scenario", CODEX_SCENARIOS, ids=lambda s: s.name) +async def test_codex_stream_balanced(scenario, tmp_path): + assert run_codex_scenario is not None + assert_stream_balanced((await run_codex_scenario(scenario, str(tmp_path)))[1]) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("scenario", ANTIGRAVITY_SCENARIOS, ids=lambda s: s.name) +async def test_antigravity_stream_balanced(scenario, tmp_path): + assert_stream_balanced((await run_antigravity_scenario(scenario, str(tmp_path)))[1]) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("scenario", OPENCODE_SCENARIOS, ids=lambda s: s.name) +async def test_opencode_stream_balanced(scenario, tmp_path): + assert_stream_balanced((await run_opencode_scenario(scenario, str(tmp_path)))[1]) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("scenario", PI_SCENARIOS, ids=lambda s: s.name) +async def test_pi_stream_balanced(scenario, tmp_path): + assert_stream_balanced((await run_pi_scenario(scenario, str(tmp_path)))[1]) # The ONE place a harness is listed for golden coverage. Derived from AgentKind @@ -404,9 +425,8 @@ def test_exactly_zero_raises_too(self): def test_collapsed_bounds_raise_even_with_a_healthy_duration(self): # Two harnesses take the duration from a MONOTONIC clock and the # bounds from the wall clock, so a reducer can report a real duration - # beside two stamps that collapsed to one instant. CE059 sees that - # statically only when both bounds are the same ast.Name; this is the - # check for when they are two different names holding one value. + # beside two stamps that collapsed to one instant; this is the check + # for that. with pytest.raises(AssertionError, match="bounds that span it"): assert_timing_captured(self._record(windows=[500.0], bounds_collapse=True), expect_generation_window=True) diff --git a/tests/test_agent_judge_criterion.py b/tests/test_agent_judge_criterion.py index fa73ea6e..0b386c41 100644 --- a/tests/test_agent_judge_criterion.py +++ b/tests/test_agent_judge_criterion.py @@ -36,6 +36,8 @@ ) from coder_eval.models.routing import DirectRoute from coder_eval.sandbox import Sandbox +from coder_eval.streaming.emitter import TurnOutcome +from coder_eval.streaming.events import AgentEndStatus # ClaudeCodeAgent is now imported inside SubAgentRunner; tests patch the runner's binding. @@ -62,10 +64,14 @@ def _make_turn(agent_output: str, duration: float = 1.5) -> TurnRecord: ) +def _make_outcome(record: TurnRecord) -> TurnOutcome: + return TurnOutcome(record=record, status=AgentEndStatus.COMPLETED, error=None) + + def _make_mock_agent(agent_output: str) -> MagicMock: agent = MagicMock() agent.start = AsyncMock(return_value=None) - agent.communicate = AsyncMock(return_value=_make_turn(agent_output)) + agent.communicate = AsyncMock(return_value=_make_outcome(_make_turn(agent_output))) agent.stop = AsyncMock(return_value=None) agent.kill = AsyncMock(return_value=None) return agent @@ -215,11 +221,11 @@ def test_agent_judge_no_verdict_surfaces_untrusted_output(sandbox: Sandbox, dire def test_agent_judge_turn_timeout_maps_to_zero(sandbox: Sandbox, direct_route: DirectRoute) -> None: - from coder_eval.errors.timeout import TurnTimeoutError - criterion = AgentJudgeCriterion(description="x", prompt="grade", turn_timeout=30) mock_agent = _make_mock_agent("irrelevant") - mock_agent.communicate.side_effect = TurnTimeoutError(30.0, task_id="t", iteration=1) + mock_agent.communicate.return_value = TurnOutcome( + record=_make_turn("irrelevant"), status=AgentEndStatus.TIMEOUT, error="timed out" + ) with patch(_AGENT_PATCH_PATH, return_value=mock_agent): result = SuccessChecker(sandbox, init_registry=False, route=direct_route).check(criterion) @@ -831,7 +837,7 @@ def test_agent_judge_transcript_captures_tool_calls(sandbox: Sandbox, direct_rou mock_agent = MagicMock() mock_agent.start = AsyncMock(return_value=None) mock_agent.communicate = AsyncMock( - return_value=_make_turn_with_commands('{"score": 0.9, "rationale": "ok"}', [cmd1, cmd2]) + return_value=_make_outcome(_make_turn_with_commands('{"score": 0.9, "rationale": "ok"}', [cmd1, cmd2])) ) mock_agent.stop = AsyncMock(return_value=None) mock_agent.kill = AsyncMock(return_value=None) @@ -943,11 +949,11 @@ def test_agent_judge_round_trips_through_evaluation_result(sandbox: Sandbox, dir def test_agent_judge_timeout_uses_base_criterion_result(sandbox: Sandbox, direct_route: DirectRoute) -> None: """Timeout path returns a base CriterionResult (no transcript / verdict fields) because no turn was produced — there's nothing to capture.""" - from coder_eval.errors.timeout import TurnTimeoutError - criterion = AgentJudgeCriterion(description="x", prompt="grade", turn_timeout=30) mock_agent = _make_mock_agent("irrelevant") - mock_agent.communicate.side_effect = TurnTimeoutError(30.0, task_id="t", iteration=1) + mock_agent.communicate.return_value = TurnOutcome( + record=_make_turn("irrelevant"), status=AgentEndStatus.TIMEOUT, error="timed out" + ) with patch(_AGENT_PATCH_PATH, return_value=mock_agent): result = SuccessChecker(sandbox, init_registry=False, route=direct_route).check(criterion) diff --git a/tests/test_agent_telemetry.py b/tests/test_agent_telemetry.py index 6bc6c501..2585e478 100644 --- a/tests/test_agent_telemetry.py +++ b/tests/test_agent_telemetry.py @@ -105,7 +105,7 @@ async def mock_query(prompt, options): try: await agent.start(str(tmp_path)) - turn = await agent.communicate("List files") + turn = (await agent.communicate("List files", iteration=1)).record # Verify command telemetry assert len(turn.commands) == 1 @@ -150,7 +150,7 @@ async def mock_query(prompt, options): try: await agent.start(str(tmp_path)) - turn = await agent.communicate("Read file") + turn = (await agent.communicate("Read file", iteration=1)).record assert len(turn.commands) == 1 cmd = turn.commands[0] @@ -189,7 +189,7 @@ async def mock_query(prompt, options): try: await agent.start(str(tmp_path)) - turn = await agent.communicate("Write file") + turn = (await agent.communicate("Write file", iteration=1)).record assert len(turn.commands) == 1 cmd = turn.commands[0] @@ -239,7 +239,7 @@ async def mock_query(prompt, options): try: await agent.start(str(tmp_path)) - turn = await agent.communicate("Multiple commands") + turn = (await agent.communicate("Multiple commands", iteration=1)).record assert len(turn.commands) == 3 @@ -290,7 +290,7 @@ async def mock_query(prompt, options): try: await agent.start(str(tmp_path)) - turn = await agent.communicate("Timed command") + turn = (await agent.communicate("Timed command", iteration=1)).record assert len(turn.commands) == 1 cmd = turn.commands[0] @@ -328,7 +328,7 @@ async def mock_query(prompt, options): await agent.start(str(tmp_path)) with caplog.at_level(logging.WARNING): - turn = await agent.communicate("Orphaned result") + turn = (await agent.communicate("Orphaned result", iteration=1)).record assert len(turn.commands) == 0 assert any("Unhandled SDK message type" in record.message for record in caplog.records), ( @@ -340,7 +340,7 @@ async def mock_query(prompt, options): @pytest.mark.asyncio async def test_command_telemetry_duplicate_result_logged(self, tmp_path, caplog): - """Verify multiple results handled gracefully (last wins) and logged.""" + """A second result for the same tool id is ignored (the first stands) and logged.""" tool_use_block_cls, assistant_message_cls, user_message_cls, _, _, result_message_cls = ( create_mock_sdk_messages() ) @@ -372,13 +372,14 @@ async def mock_query(prompt, options): await agent.start(str(tmp_path)) with caplog.at_level(logging.DEBUG): - turn = await agent.communicate("Duplicate results") + turn = (await agent.communicate("Duplicate results", iteration=1)).record assert len(turn.commands) == 1 cmd = turn.commands[0] - assert cmd.result_status == "error" # From result2 - assert cmd.error_message == "second result (error)" + assert cmd.result_status == "success" + assert cmd.result_summary == "first result" + assert cmd.error_message is None # Should log debug message assert any("Multiple results" in record.message for record in caplog.records) @@ -416,17 +417,16 @@ async def mock_query(prompt, options): # Capture both INFO and WARNING levels with caplog.at_level(logging.INFO): - turn = await agent.communicate("Missing result") + turn = (await agent.communicate("Missing result", iteration=1)).record assert len(turn.commands) == 1 assert turn.commands[0].result_status == "unknown" - # Should have warning about missing result - warnings = [r for r in caplog.records if r.levelname == "WARNING"] - assert any("completed without tool result" in r.message for r in warnings) - - # Should have warning summary about unknown statuses with message types - assert any("'unknown' status" in r.message for r in warnings) + warnings = [r.message for r in caplog.records if r.levelname == "WARNING"] + unresolved = [m for m in warnings if "without a result" in m] + assert len(unresolved) == 1 + assert "toolu_unknown" in unresolved[0] + assert "AssistantMessage=1" in unresolved[0] finally: agent_module.query = original_query @@ -465,7 +465,7 @@ async def mock_query(prompt, options): try: await agent.start(str(tmp_path)) - turn = await agent.communicate("Non-dict input") + turn = (await agent.communicate("Non-dict input", iteration=1)).record # Should capture the command without crashing assert len(turn.commands) == 1 @@ -512,7 +512,7 @@ async def mock_query(prompt, options): try: await agent.start(str(tmp_path)) - await agent.communicate("Non-dict stream", stream_callback=CollectingCallback()) + await agent.communicate("Non-dict stream", iteration=1, stream_callback=CollectingCallback()) from coder_eval.streaming.events import ToolStartEvent @@ -555,7 +555,7 @@ async def mock_query(prompt, options): try: await agent.start(str(tmp_path)) - turn = await agent.communicate("Read a file") + turn = (await agent.communicate("Read a file", iteration=1)).record # Verify assistant_turns list is populated assert len(turn.messages) == 1 @@ -607,7 +607,7 @@ async def mock_query(prompt, options): try: await agent.start(str(tmp_path)) - turn = await agent.communicate("Test ordering") + turn = (await agent.communicate("Test ordering", iteration=1)).record assert len(turn.messages) == 1 aturn = turn.messages[0] @@ -654,7 +654,7 @@ async def mock_query(prompt, options): try: await agent.start(str(tmp_path)) - turn = await agent.communicate("Test thinking") + turn = (await agent.communicate("Test thinking", iteration=1)).record assert len(turn.messages) == 1 aturn = turn.messages[0] @@ -708,7 +708,7 @@ async def mock_query(prompt, options): try: await agent.start(str(tmp_path)) - turn = await agent.communicate("Multiple turns") + turn = (await agent.communicate("Multiple turns", iteration=1)).record # Verify both turns captured (a trailing ReconciliationMessage may # follow when the authoritative total exceeds the per-message sum). @@ -762,7 +762,7 @@ async def mock_query(prompt, options): try: await agent.start(str(tmp_path)) - turn = await agent.communicate("Test execution timing") + turn = (await agent.communicate("Test execution timing", iteration=1)).record assert len(turn.commands) == 1 cmd = turn.commands[0] @@ -812,7 +812,7 @@ async def mock_query(prompt, options): try: await agent.start(str(tmp_path)) - turn = await agent.communicate("Test command indexing") + turn = (await agent.communicate("Test command indexing", iteration=1)).record # Both commands should reference assistant turn index 0 assert len(turn.commands) == 2 @@ -927,7 +927,7 @@ async def mock_query(prompt, options): try: await agent.start(str(tmp_path)) - turn = await agent.communicate("hi") + turn = (await agent.communicate("hi", iteration=1)).record assert len(turn.messages) == 1 aturn = turn.messages[0] assert isinstance(aturn, AssistantMessage) @@ -999,7 +999,7 @@ async def mock_query(prompt, options): agent_module.query = mock_query try: await agent.start(str(tmp_path)) - turn = await agent.communicate("hi") + turn = (await agent.communicate("hi", iteration=1)).record assistants = [m for m in turn.messages if isinstance(m, AssistantMessage)] assert len(assistants) == 3 # A-text, A-tool, B-text a_text, a_tool, b_text = assistants @@ -1061,7 +1061,7 @@ async def mock_query(prompt, options): try: await agent.start(str(tmp_path)) - turn = await agent.communicate("hi") + turn = (await agent.communicate("hi", iteration=1)).record assert len(turn.messages) == 2 first, second = turn.messages assert isinstance(first, AssistantMessage) @@ -1123,7 +1123,7 @@ async def mock_query(prompt, options): try: await agent.start(str(tmp_path)) - turn = await agent.communicate("hi") + turn = (await agent.communicate("hi", iteration=1)).record aturn = turn.messages[0] assert isinstance(aturn, AssistantMessage) # delta wins over partial 0; ResultMessage fallback is suppressed. @@ -1166,7 +1166,7 @@ async def mock_query(prompt, options): try: await agent.start(str(tmp_path)) - turn = await agent.communicate("hi") + turn = (await agent.communicate("hi", iteration=1)).record aturn = turn.messages[0] assert isinstance(aturn, AssistantMessage) # Backfilled from ResultMessage. @@ -1216,7 +1216,7 @@ async def mock_query(prompt, options): try: await agent.start(str(tmp_path)) - turn = await agent.communicate("hi") + turn = (await agent.communicate("hi", iteration=1)).record assert len(turn.messages) == 2 a1, a2 = turn.messages assert isinstance(a1, AssistantMessage) @@ -1264,7 +1264,7 @@ async def mock_query(prompt, options): try: await agent.start(str(tmp_path)) - turn = await agent.communicate("hi") + turn = (await agent.communicate("hi", iteration=1)).record # A trailing ReconciliationMessage may follow the two assistant # emissions (here the snapshot total differs from the per-message sum). assistant_msgs = [m for m in turn.messages if isinstance(m, AssistantMessage)] @@ -1286,19 +1286,17 @@ async def mock_query(prompt, options): class TestClaudeHeadIsMeasuredAtFirstOutput: """claude-code's head is the wall clock up to the first observed model output. - `_ClaudeTurnState.__init__` stamps `last_event_wall`, and + `_ClaudeDecoder.__init__` stamps `last_event_wall`, and `_seed_first_generation_window` re-stamps it at the first `message_start`. So the first window opens where the model first spoke, and the CLI spawn, provider resolution and time to first token before it are the head. - `_build_claude_query` is NOT in the head: it runs at `communicate`'s - `:1095`, before `AgentStartEvent` is emitted at `:1106`, so it precedes the - head's own start stamp. It used to sit inside msg0's generation window - (`last_event_wall` was stamped at state construction, ahead of the build); - it now sits inside `duration_seconds` but outside all four buckets, as - unexplained residual. That is why the budget below still matters and why it - is not the same guard it was: at 0.03-0.10 ms the residual is noise, and - the two tests keep it that way. + `_build_claude_query` is NOT in the head: `communicate` runs it before it + opens the turn's emitter, so it precedes the `AgentStartEvent` stamp and the + turn's `duration_seconds` alike. It used to sit inside msg0's generation + window (`last_event_wall` was stamped ahead of the build). It is now + unmeasured wall time between turns; at 0.03-0.10 ms that is noise, and the + two tests keep it that way. It used to be `0.0`, and that was a CLAMPED NEGATIVE rather than a measurement: both marks were stamped before `AgentStartEvent` was emitted, @@ -1307,12 +1305,8 @@ class TestClaudeHeadIsMeasuredAtFirstOutput: spawns the `claude` CLI over `anyio.open_process` and `_pump_messages` calls `query()` once per `communicate()`, a fresh CLI per turn. - The two budget tests below survive the rewrite with their meaning INVERTED. - `_build_claude_query`'s cost now lands in the head rather than inside msg0's - generation, so they no longer guard "the build is cheap enough to leave - hidden by the clamp" — they guard "our own setup is a negligible part of a - head that is now published", which is what makes the head readable as the - harness's latency rather than as ours. + The two budget tests below guard that our own setup stays negligible, so the + published head reads as the harness's latency rather than as ours. """ # Measured at 0.03 ms bare and 0.10 ms with plugins. The bound is @@ -1335,7 +1329,7 @@ def _build_ms(plugin_root=None) -> float: samples = [] for _ in range(5): started = time.perf_counter() - agent._build_claude_query("hi", 60, lambda _line: None) + agent._build_claude_query("hi", 1, 60, lambda _line: None) samples.append((time.perf_counter() - started) * 1000.0) return min(samples) @@ -1343,10 +1337,9 @@ def test_the_query_build_is_a_negligible_part_of_the_published_head(self): elapsed = self._build_ms() assert elapsed < self.BUDGET_MS, ( f"_build_claude_query took {elapsed:.2f} ms, over the {self.BUDGET_MS} ms budget. It runs " - "BEFORE the AgentStartEvent, so it is inside the turn's duration_seconds but outside " - "all four buckets — unexplained residual that no bucket accounts for. At a few hundred " - "microseconds that is noise; at this size the four buckets would visibly stop summing " - "to the turn and the gap would be ours, not the harness's." + "BEFORE the AgentStartEvent, so no bucket and no turn duration accounts for it. At a few " + "hundred microseconds that is noise; at this size the unmeasured gap between turns " + "would be ours, not the harness's." ) def test_a_staged_plugin_root_does_not_change_that(self, tmp_path): @@ -1365,74 +1358,57 @@ class TestClaudeTurnTokensAreDeltas: generation would latch a budget on usage the agent never spent. """ - def test_interleaved_message_ids_never_report_the_same_tokens_twice(self, monkeypatch): - from coder_eval.streaming.callbacks import CompositeStreamCallback + def test_interleaved_message_ids_never_report_the_same_tokens_twice(self): from coder_eval.streaming.events import AgentEndStatus, TurnEndEvent - clock, state = TestClaudeFirstWindowReseed()._state(monkeypatch) - ends: list[TurnEndEvent] = [] - - class _Sink: - def on_event(self, event): - if isinstance(event, TurnEndEvent): - ends.append(event) - - state.emit = CompositeStreamCallback([state.collector, _Sink()]) + clock, decoder, events = TestClaudeFirstWindowReseed()._decoder() for mid in ("x", "y", "x", "y"): clock.at_ms += 100 - state.on_assistant_message(TestClaudeFirstWindowReseed._assistant(mid)) - state.finalize(AgentEndStatus.COMPLETED, crashed=False, crash_reason=None) + decoder.on_assistant_message(TestClaudeFirstWindowReseed._assistant(mid)) + decoder.end(AgentEndStatus.COMPLETED) + ends = [e for e in events if isinstance(e, TurnEndEvent)] reported = sum((end.tokens.output_tokens for end in ends if end.tokens is not None), 0) - recorded = sum(rec.output_tokens for records in state.emissions_by_id.values() for rec in records) + recorded = sum(rec.output_tokens for records in decoder.emissions_by_id.values() for rec in records) assert reported == recorded class TestClaudeFirstWindowReseed: """The first `message_start` moves the window mark; a later one must not. - Driven at `_ClaudeTurnState` with both clocks moved off one counter. The - window's BOUNDS come from the turn's `TurnClock`, which is INJECTED — a - derived stamp escapes a monkeypatched module `datetime` entirely, so these - tests would measure the real clock and still pass. Its DURATION side still - reads `time.monotonic()` for `turn_start_time` and the deadline, so that - global is patched off the same counter; leaving it real would straddle a - scripted clock and a live one. + Driven at `_ClaudeDecoder` over a real `TurnEmitter` on a clock the test moves + by hand. Every window bound and the turn bracket come from that one clock. """ BASE = datetime(2026, 9, 11, 9, 0, 0) class _Stepped: - """A `TurnClock` stand-in the test moves by hand, in ms from `BASE`.""" + """A clock the test moves by hand, in ms from `BASE`.""" at_ms = 0.0 def now(self): return TestClaudeFirstWindowReseed.BASE + timedelta(milliseconds=self.at_ms) - def _state(self, monkeypatch): - from coder_eval.agents import claude_code_agent as claude_module - from coder_eval.agents.claude_code_agent import ClaudeCodeAgent, _ClaudeTurnState - from coder_eval.streaming.callbacks import CompositeStreamCallback - from coder_eval.streaming.collector import EventCollector + def _decoder(self): + from coder_eval.agents.claude_code_agent import ClaudeCodeAgent, _ClaudeDecoder + from coder_eval.models import TimingBasis + from coder_eval.streaming.emitter import TurnEmitter stepped = self._Stepped() - monkeypatch.setattr(claude_module, "time", SimpleNamespace(monotonic=lambda: stepped.at_ms / 1000.0)) - - agent = ClaudeCodeAgent(parse_agent_config(type=AgentKind.CLAUDE_CODE, permission_mode="acceptEdits")) - collector = EventCollector() - return stepped, _ClaudeTurnState( - agent, - emit=CompositeStreamCallback([collector]), - collector=collector, + events: list[Any] = [] + emitter = TurnEmitter( task_id="t", - user_input="go", iteration=1, - log=agent._log, - turn_start_time=0.0, - deadline=None, + prompt="go", + model="mock-model", + basis=TimingBasis.TURN_CLOCK, clock=stepped, + sinks=[SimpleNamespace(on_event=events.append)], ) + emitter.begin() + agent = ClaudeCodeAgent(parse_agent_config(type=AgentKind.CLAUDE_CODE, permission_mode="acceptEdits")) + return stepped, _ClaudeDecoder(agent, emitter, effective_model="mock-model"), events @staticmethod def _assistant(mid: str): @@ -1440,110 +1416,93 @@ def _assistant(mid: str): return SdkAssistantMessage([], usage={"input_tokens": 10, "output_tokens": 5}, message_id=mid) - def test_cli_boot_before_the_first_message_start_is_not_msg0_generation(self, monkeypatch): + def test_cli_boot_before_the_first_message_start_is_not_msg0_generation(self): """The interval the CLI spent booting is head, not model time. Before the re-seed the window opened when the turn state was built, so this whole interval was published as msg0's `generation_duration_ms` — ~3.6 s per turn on the measured corpus. """ - clock, state = self._state(monkeypatch) + clock, decoder, _ = self._decoder() clock.at_ms = 800 # CLI spawn + provider resolution + TTFT - state.on_stream_event(_message_start("m1")) + decoder.on_stream_event(_message_start("m1")) clock.at_ms = 1000 - state.on_assistant_message(self._assistant("m1")) + decoder.on_assistant_message(self._assistant("m1")) - message = state.sdk_messages[0] + message = decoder.transcript[0] assert message.started_at == self.BASE + timedelta(milliseconds=800) assert message.generation_duration_ms == pytest.approx(200.0) - def test_only_the_first_message_start_reseeds_so_the_windows_still_tile(self, monkeypatch): + def test_only_the_first_message_start_reseeds_so_the_windows_still_tile(self): """A second re-seed would drop the gap before the next emission. That gap — a tool result landing, then the next request going out — is real model time, and falling into no bucket at all is the defect pi shipped with. """ - clock, state = self._state(monkeypatch) + clock, decoder, _ = self._decoder() clock.at_ms = 800 - state.on_stream_event(_message_start("m1")) + decoder.on_stream_event(_message_start("m1")) clock.at_ms = 1000 - state.on_assistant_message(self._assistant("m1")) + decoder.on_assistant_message(self._assistant("m1")) clock.at_ms = 1500 - state.on_stream_event(_message_start("m2")) + decoder.on_stream_event(_message_start("m2")) clock.at_ms = 2000 - state.on_assistant_message(self._assistant("m2")) + decoder.on_assistant_message(self._assistant("m2")) - first, second = state.sdk_messages[0], state.sdk_messages[1] + first, second = decoder.transcript[0], decoder.transcript[1] assert second.started_at == first.completed_at, "the second window must tile from the first" assert second.generation_duration_ms == pytest.approx(1000.0) - def test_seeding_twice_by_hand_is_a_no_op_the_second_time(self, monkeypatch): + def test_seeding_twice_by_hand_is_a_no_op_the_second_time(self): """The once-per-turn guard, stated outright rather than inferred. The sibling test above would also fail if the guard were removed, but only via the tiling it implies. This says the property directly, so a reviewer does not have to reproduce a mutation to see it. """ - clock, state = self._state(monkeypatch) + clock, decoder, _ = self._decoder() clock.at_ms = 800 - state._seed_first_generation_window() - seeded_wall = state.last_event_wall + decoder._seed_first_generation_window() + seeded_wall = decoder.last_event_wall clock.at_ms = 5000 - state._seed_first_generation_window() + decoder._seed_first_generation_window() - assert state.last_event_wall == seeded_wall + assert decoder.last_event_wall == seeded_wall - def test_a_stream_with_no_message_start_still_clamps_to_zero(self, monkeypatch): + def test_a_stream_with_no_message_start_keeps_the_turn_entry_mark(self): """Partial streaming off, a mocked query(), or a crash before the first event. - The re-seed never fires, the turn-entry mark stands, and the head - clamps exactly as it did before. That is the correct degradation, and - asserting it is what keeps it from becoming an untested branch. + The re-seed never fires and the turn-entry mark stands, so the first + window opens where the turn opened and the head is zero. """ - clock, state = self._state(monkeypatch) + from coder_eval.streaming.events import AgentEndStatus + + clock, decoder, _ = self._decoder() clock.at_ms = 1000 - state.on_assistant_message(self._assistant("m1")) + decoder.on_assistant_message(self._assistant("m1")) - assert state.first_output_seen is False, "nothing latched, so the turn-entry mark stands" - # The window still opens at turn entry, which PRECEDES the - # AgentStartEvent — so the head is a negative that decompose_turn - # clamps, exactly as it did before this phase. Asserted on the mark - # rather than by re-deriving `max(elapsed, 0.0)` from hand-built - # arguments, which would restate the implementation and could not fail. - assert state.sdk_messages[0].started_at == self.BASE + assert decoder.first_output_seen is False, "nothing latched, so the turn-entry mark stands" + assert decoder.transcript[0].started_at == self.BASE + assert decoder.end(AgentEndStatus.COMPLETED).record.harness_startup_ms == pytest.approx(0.0) - def test_the_four_buckets_account_for_a_tool_free_turn(self, monkeypatch): + def test_the_four_buckets_account_for_a_tool_free_turn(self): """head + generation + tail == the turn, with the head read DIRECTLY. The sibling tests assert the window's `started_at`, which pins the mark - but never the published `harness_startup_ms` itself — so nothing here - read the field this phase exists to change. With no tool calls the tool - bucket is empty and the other three must tile the turn exactly. + but never the published `harness_startup_ms` itself. With no tool calls + the tool bucket is empty and the other three must tile the turn exactly. """ - from coder_eval.models import TokenUsage - from coder_eval.streaming.collector import EventCollector - from coder_eval.streaming.events import AgentEndEvent, AgentEndStatus, AgentStartEvent + from coder_eval.streaming.events import AgentEndStatus - clock, state = self._state(monkeypatch) + clock, decoder, _ = self._decoder() clock.at_ms = 800 - state.on_stream_event(_message_start("m1")) + decoder.on_stream_event(_message_start("m1")) clock.at_ms = 1000 - state.on_assistant_message(self._assistant("m1")) - - collector = EventCollector() - collector.on_event(AgentStartEvent(task_id="t", prompt="go", iteration=1, timestamp=self.BASE)) - collector.on_event( - AgentEndEvent( - task_id="t", - status=AgentEndStatus.COMPLETED, - messages=list(state.sdk_messages), - usage=TokenUsage(), - timestamp=self.BASE + timedelta(milliseconds=1500), - ) - ) - record = collector.build_turn_record() + decoder.on_assistant_message(self._assistant("m1")) + clock.at_ms = 1500 + record = decoder.end(AgentEndStatus.COMPLETED).record assert record.harness_startup_ms == pytest.approx(800.0), "the CLI boot is the head, published" assert record.harness_teardown_ms == pytest.approx(500.0) @@ -1553,10 +1512,8 @@ def test_the_four_buckets_account_for_a_tool_free_turn(self, monkeypatch): class TestTheTurnBracketComesFromTheTurnClock: - """CE064's behavioural half for claude-code: the SOURCE of the two stamps. + """The SOURCE of claude-code's two bracket stamps: the turn clock. - The rule can only see that `timestamp=` is present — it cannot tell - `state.clock.now()` from a `datetime.now()` spelled out at the call site. Anchoring the stand-in a year from real time is what makes a reverted argument fail by a year instead of by the microseconds that separate the two clocks in practice. @@ -1574,13 +1531,13 @@ async def mock_query(prompt, options): yield assistant_msg yield result_message_cls() - monkeypatch.setattr(agent_module, "TurnClock", AnchoredClock) + monkeypatch.setattr("coder_eval.agent.TurnClock", AnchoredClock) monkeypatch.setattr(agent_module, "query", mock_query) agent = agent_module.ClaudeCodeAgent(parse_agent_config(type=AgentKind.CLAUDE_CODE)) await agent.start(str(tmp_path)) seen: list[Any] = [] - await agent.communicate("go", stream_callback=SimpleNamespace(on_event=seen.append)) + await agent.communicate("go", iteration=1, stream_callback=SimpleNamespace(on_event=seen.append)) assert_bracket_on_the_clock(seen) @@ -1596,11 +1553,158 @@ async def mock_query(prompt, options): yield assistant_msg yield result_message_cls() - monkeypatch.setattr(agent_module, "TurnClock", AnchoredClock) + monkeypatch.setattr("coder_eval.agent.TurnClock", AnchoredClock) monkeypatch.setattr(agent_module, "query", mock_query) agent = agent_module.ClaudeCodeAgent(parse_agent_config(type=AgentKind.CLAUDE_CODE)) await agent.start(str(tmp_path)) - record = await agent.communicate("go") + record = (await agent.communicate("go", iteration=1)).record assert_overhead_is_measured(record) + + +def _claude_replay(stream: list[Any]): + """Replay ``stream`` through a real ``_ClaudeDecoder`` on a ``ScriptedClock``, ending COMPLETED.""" + from coder_eval.agents.claude_code_agent import ClaudeCodeAgent, _ClaudeDecoder + from coder_eval.streaming.events import AgentEndStatus + from coder_eval.testing import ScriptedClock, replay + + agent = ClaudeCodeAgent(parse_agent_config(type=AgentKind.CLAUDE_CODE, model="claude-sonnet-4-5")) + return replay( + stream, + lambda emitter: _ClaudeDecoder(agent, emitter, effective_model="claude-sonnet-4-5"), + clock=ScriptedClock(datetime(2026, 9, 16, 12, 0, 0)), + model="claude-sonnet-4-5", + end=lambda decoder: decoder.end(AgentEndStatus.COMPLETED), + ) + + +class TestClaudeSubAgentScope: + """A sub-agent message (``parent_tool_use_id`` set) is nested under its spawning tool call. + + Its turn, tools and model stay off the main thread, so the tool-call cap and the + reported model see the main agent only, while ``TurnRecord.commands`` keeps every call. + """ + + MAIN_MODEL = "claude-sonnet-4-5" + SUB_MODEL = "claude-haiku-4-5" + + def _stream(self) -> list[Any]: + from coder_eval.testing import Tick + from tests._fixtures.golden_streams.claude_fixtures import ( + AssistantMessage as SdkAssistantMessage, + ) + from tests._fixtures.golden_streams.claude_fixtures import ( + ResultMessage, + ToolUseBlock, + UserMessage, + ) + + return [ + Tick(100), + SdkAssistantMessage( + [ToolUseBlock("task_1", "Task", {"prompt": "count"})], message_id="m1", model=self.MAIN_MODEL + ), + Tick(300), + SdkAssistantMessage( + [ToolUseBlock("sub_bash", "Bash", {"command": "seq 100"})], + message_id="s1", + parent_tool_use_id="task_1", + model=self.SUB_MODEL, + ), + Tick(450), + UserMessage("sub_bash", False, "1..100"), + Tick(900), + UserMessage("task_1", False, "5050", agent_id="agent-1", usage={"output_tokens": 7}), + Tick(1000), + SdkAssistantMessage( + [ToolUseBlock("write_1", "Write", {"file_path": "answer.txt"})], message_id="m2", model=self.MAIN_MODEL + ), + Tick(1200), + UserMessage("write_1", False, "ok"), + Tick(1300), + ResultMessage(num_turns=2), + ] + + def test_sub_agent_events_are_nested_and_its_tool_is_recorded(self): + from coder_eval.streaming.events import AgentEndEvent, ToolEndEvent, ToolStartEvent, TurnStartEvent + from coder_eval.testing import assert_stream_balanced + + result = _claude_replay(self._stream()) + + assert_stream_balanced(result.events) + starts = {e.turn_id: e for e in result.events if isinstance(e, TurnStartEvent)} + assert starts["s1"].parent_thread_id == "task_1" + assert starts["s1"].model == self.SUB_MODEL + assert starts["m1"].parent_thread_id is None and starts["m2"].parent_thread_id is None + for kind in (ToolStartEvent, ToolEndEvent): + by_id = {e.tool.tool_id: e for e in result.events if isinstance(e, kind)} + assert by_id["sub_bash"].parent_thread_id == "task_1" + assert by_id["task_1"].parent_thread_id is None + assert by_id["write_1"].parent_thread_id is None + (end,) = [e for e in result.events if isinstance(e, AgentEndEvent)] + assert end.model_used == self.MAIN_MODEL + assert sorted(c.tool_id for c in result.record.commands) == ["sub_bash", "task_1", "write_1"] + + def test_the_turn_monitor_counts_only_main_thread_calls(self): + from coder_eval.models import FileExistsCriterion, RunLimits, SandboxConfig, TaskDefinition + from coder_eval.orchestration.turn_monitor import TurnMonitor + + task = TaskDefinition( + task_id="subagent-cap", + description="d", + initial_prompt="go", + agent=parse_agent_config(type=AgentKind.CLAUDE_CODE), + sandbox=SandboxConfig(driver="tempdir"), + success_criteria=[FileExistsCriterion(path="answer.txt", description="answer")], + run_limits=RunLimits(max_tool_calls=3), + ) + monitor = TurnMonitor.for_task(task, arm=True) + + for event in _claude_replay(self._stream()).events: + monitor.on_event(event) + + assert monitor.tool_calls == 2, "Task + Write on the main thread; the sub-agent's Bash is not counted" + assert monitor.should_stop() is None + + def test_a_tool_duration_is_the_clock_gap_from_tool_use_to_result(self): + from coder_eval.testing import Tick + from tests._fixtures.golden_streams.claude_fixtures import AssistantMessage as SdkAssistantMessage + from tests._fixtures.golden_streams.claude_fixtures import ToolUseBlock, UserMessage + + result = _claude_replay( + [ + Tick(1000), + SdkAssistantMessage([ToolUseBlock("t1", "Bash", {"command": "sleep 1"})], message_id="m1"), + Tick(1750), + UserMessage("t1", False, "done"), + Tick(2000), + ] + ) + + (cmd,) = result.record.commands + base = datetime(2026, 9, 16, 12, 0, 0) + assert cmd.execution_started_at == base + timedelta(milliseconds=1000) + assert cmd.execution_completed_at == base + timedelta(milliseconds=1750) + assert cmd.duration_ms == pytest.approx(750.0) + + +def test_a_claude_turn_stopped_before_the_result_message_keeps_the_final_reply(): + """A stop breaks the loop before any ResultMessage: the summary is the emitter's, as on every harness.""" + from coder_eval.agents.claude_code_agent import ClaudeCodeAgent, _ClaudeDecoder + from coder_eval.models import AgentKind, parse_agent_config + from coder_eval.streaming.events import AgentEndStatus + from coder_eval.testing import ScriptedClock, replay + from tests._fixtures.golden_streams.claude_fixtures import AssistantMessage as SdkAssistantMessage + from tests._fixtures.golden_streams.claude_fixtures import TextBlock + + agent = ClaudeCodeAgent(parse_agent_config(type=AgentKind.CLAUDE_CODE, permission_mode="acceptEdits")) + result = replay( + [SdkAssistantMessage([TextBlock("partial answer")], message_id="m1")], + lambda emitter: _ClaudeDecoder(agent, emitter, effective_model=None), + clock=ScriptedClock(datetime(2026, 1, 1)), + end=lambda decoder: decoder.end(AgentEndStatus.TOOL_CALLS_EXHAUSTED), + ) + summary = result.record.result_summary + assert summary is not None + assert (summary.is_error, summary.subtype, summary.result) == (False, "tool_calls_exhausted", "partial answer") diff --git a/tests/test_agent_telemetry_advanced.py b/tests/test_agent_telemetry_advanced.py index 2345485f..81e64fe4 100644 --- a/tests/test_agent_telemetry_advanced.py +++ b/tests/test_agent_telemetry_advanced.py @@ -73,7 +73,7 @@ async def mock_query(prompt, options): try: await agent.start(str(tmp_path)) - turn_record = await agent.communicate("test prompt") + turn_record = (await agent.communicate("test prompt", iteration=1)).record # Verify command has 'unknown' status assert len(turn_record.commands) == 1 @@ -89,13 +89,11 @@ async def mock_query(prompt, options): @pytest.mark.asyncio -async def test_duplicate_result_message_last_wins(tmp_path, caplog): - """Test that duplicate ResultMessages use last-wins strategy. +async def test_duplicate_result_message_first_stands(tmp_path, caplog): + """A second result for the same tool id is ignored: the first result stands. - Hypothesis: SDK may send multiple results for same tool_id. - Expected: Last result overwrites, debug log emitted. - - Context: Lines 162-167 in claude_code_agent.py log duplicates. + The emitter closed the tool at the first result, so the duplicate gets a debug + log and no second ``ToolEndEvent``. """ tool_use_block_cls, assistant_message_cls, result_message_cls = create_mock_sdk_messages() @@ -105,7 +103,7 @@ async def test_duplicate_result_message_last_wins(tmp_path, caplog): # First result: success result_1 = result_message_cls("tool_456", False, "File read successfully") - # Second result: error (should overwrite) + # Second result: error (ignored) result_2 = result_message_cls("tool_456", True, "File not found") config = parse_agent_config(type="claude-code") @@ -125,13 +123,13 @@ async def mock_query(prompt, options): await agent.start(str(tmp_path)) with caplog.at_level(logging.DEBUG): - turn_record = await agent.communicate("test prompt") + turn_record = (await agent.communicate("test prompt", iteration=1)).record - # Verify last result wins assert len(turn_record.commands) == 1 cmd = turn_record.commands[0] - assert cmd.result_status == "error" # Last result - assert cmd.error_message == "File not found" + assert cmd.result_status == "success" + assert cmd.result_summary == "File read successfully" + assert cmd.error_message is None # Verify debug log for duplicate assert any("Multiple results" in record.message for record in caplog.records) @@ -142,13 +140,7 @@ async def mock_query(prompt, options): @pytest.mark.asyncio async def test_pending_command_without_result_finalizes_unknown(tmp_path, caplog): - """Test clean finalization of commands left pending after stream interruption. - - Hypothesis: Stream interruption leaves commands in pending state. - Expected: Commands finalized with 'unknown' status, warning logged. - - Context: Lines 187-197 handle pending command cleanup. - """ + """Tools left open when the stream ends are swept UNRESOLVED, with one warning naming them.""" tool_use_block_cls, assistant_message_cls, result_message_cls = create_mock_sdk_messages() # Create multiple tool uses, only some get results @@ -177,7 +169,7 @@ async def mock_query(prompt, options): await agent.start(str(tmp_path)) with caplog.at_level(logging.WARNING): - turn_record = await agent.communicate("test prompt") + turn_record = (await agent.communicate("test prompt", iteration=1)).record # Verify all three commands recorded assert len(turn_record.commands) == 3 @@ -193,10 +185,16 @@ async def mock_query(prompt, options): assert turn_record.commands[2].tool_name == "Read" assert turn_record.commands[2].result_status == "unknown" - # Verify warning logged for unknown status - warnings = [r for r in caplog.records if r.levelname == "WARNING"] - assert any("completed without tool result" in r.message for r in warnings) - assert any("Status set to 'unknown'" in r.message for r in warnings) + for cmd in turn_record.commands[1:]: + assert cmd.execution_started_at is not None + assert cmd.execution_completed_at is None + assert cmd.duration_ms is None + + warnings = [r.message for r in caplog.records if r.levelname == "WARNING"] + unresolved = [m for m in warnings if "without a result" in m] + assert len(unresolved) == 1 + assert "tool_002" in unresolved[0] and "tool_003" in unresolved[0] + assert "tool_001" not in unresolved[0] finally: agent_module.query = original_query @@ -231,7 +229,7 @@ async def mock_query(prompt, options): await agent.start(str(tmp_path)) with caplog.at_level(logging.WARNING): - turn_record = await agent.communicate("test prompt") + turn_record = (await agent.communicate("test prompt", iteration=1)).record # The orphaned result is handled gracefully: a single synthesized # 'unknown' command captures it (rather than being silently dropped). @@ -283,7 +281,7 @@ async def mock_query(prompt, options): try: await agent.start(str(tmp_path)) - turn_record = await agent.communicate("test prompt") + turn_record = (await agent.communicate("test prompt", iteration=1)).record assert len(turn_record.commands) == 3 diff --git a/tests/test_agent_timeout.py b/tests/test_agent_timeout.py index 5777dea0..46c935fa 100644 --- a/tests/test_agent_timeout.py +++ b/tests/test_agent_timeout.py @@ -16,6 +16,7 @@ from coder_eval.agents.claude_code_agent import ClaudeCodeAgent from coder_eval.errors.timeout import TurnTimeoutError from coder_eval.models import AgentKind, parse_agent_config +from coder_eval.streaming.events import AgentEndStatus def _make_agent() -> ClaudeCodeAgent: @@ -91,11 +92,13 @@ async def mock_query(prompt, options, transport=None): ): mock_transport_cls.return_value = MagicMock() - with pytest.raises(TurnTimeoutError) as exc: - # 100ms watchdog — bypasses AgentConfig's ge=10 validator - # because we pass it directly as a call arg, not via config. - await agent.communicate("prompt", timeout=0.1) + # 100ms watchdog — bypasses AgentConfig's ge=10 validator + # because we pass it directly as a call arg, not via config. + outcome = await agent.communicate("prompt", iteration=1, timeout=0.1) + assert outcome.status is AgentEndStatus.TIMEOUT + with pytest.raises(TurnTimeoutError) as exc: + outcome.record_or_raise(timeout_seconds=0.1) assert exc.value.timeout_seconds == 0.1 assert exc.value.layer == "turn" mock_kill_transport.assert_called() @@ -127,8 +130,9 @@ async def mock_query(prompt, options, transport=None): ): mock_transport_cls.return_value = MagicMock() - with pytest.raises(TurnTimeoutError): - await agent.communicate("prompt", timeout=0.1) + outcome = await agent.communicate("prompt", iteration=1, timeout=0.1) + + assert outcome.status is AgentEndStatus.TIMEOUT @pytest.mark.asyncio @@ -181,10 +185,11 @@ async def swap_active_transport_before_watchdog() -> None: patch("coder_eval.agents.claude_code_agent.query", mock_query), ): swapper = asyncio.create_task(swap_active_transport_before_watchdog()) - with pytest.raises(TurnTimeoutError): - await agent.communicate("A", timeout=0.1) + outcome = await agent.communicate("A", iteration=1, timeout=0.1) await swapper + assert outcome.status is AgentEndStatus.TIMEOUT + # The watchdog must have killed transport A (its captured target), # not transport B (which happened to be in self._active_transport # when the watchdog fired). @@ -230,10 +235,11 @@ def fake_monotonic() -> float: patch("coder_eval.agents.claude_code_agent.time.monotonic", fake_monotonic), ): mock_transport_cls.return_value = MagicMock() - # Must return a TurnRecord, not raise TurnTimeoutError, even - # though wall-clock is way past deadline. - result = await agent.communicate("prompt", timeout=1.0) - assert result is not None + # Must complete cleanly, not TIMEOUT, even though wall-clock is + # way past deadline. + outcome = await agent.communicate("prompt", iteration=1, timeout=1.0) + assert outcome.status is AgentEndStatus.COMPLETED + assert outcome.record is not None @pytest.mark.asyncio @@ -256,7 +262,7 @@ async def mock_query(prompt, options): patch("coder_eval.agents.claude_code_agent.SubprocessCLITransport") as mock_transport_cls, patch("coder_eval.agents.claude_code_agent.query", mock_query), ): - await agent.communicate("prompt") # no timeout + await agent.communicate("prompt", iteration=1) # no timeout mock_transport_cls.assert_not_called() @@ -277,14 +283,21 @@ def __init__(self, text: str, usage: dict[str, int]) -> None: @pytest.mark.asyncio -async def test_external_cancel_parks_the_turn_it_interrupted(): - """A turn cancelled from outside must leave its telemetry on ``pending_turn``. +async def test_external_cancel_reports_the_spend_of_the_turn_it_interrupted(): + """A turn cancelled from outside ends with one CRASHED ``AgentEndEvent`` carrying its usage. - The task-timeout kill path: the turn never returns a record and the frame that - held one unwinds, so the pending slot is the only place its spend can survive. + The task-timeout kill path: the turn never returns an outcome, so the event the + stream callback received is the only place its spend can survive. """ + from coder_eval.streaming.events import AgentEndEvent, AgentEndStatus + config = parse_agent_config(type=AgentKind.CLAUDE_CODE, permission_mode="acceptEdits", model="claude-sonnet-5") agent = ClaudeCodeAgent(config) + events: list = [] + + class _Sink: + def on_event(self, event): + events.append(event) with tempfile.TemporaryDirectory() as tmpdir: await agent.start(tmpdir) @@ -299,16 +312,19 @@ async def mock_query(prompt, options): await asyncio.sleep(30) with patch("coder_eval.agents.claude_code_agent.query", mock_query): - turn = asyncio.create_task(agent.communicate("prompt")) + turn = asyncio.create_task(agent.communicate("prompt", iteration=1, stream_callback=_Sink())) await streaming.wait() turn.cancel() with pytest.raises(asyncio.CancelledError): await asyncio.wait_for(turn, timeout=5) - partial = agent.pending_turn - assert partial is not None, "the interrupted turn's record was discarded" - assert partial.crashed is True - usage = partial.token_usage + ends = [e for e in events if isinstance(e, AgentEndEvent)] + assert len(ends) == 1, "the interrupted turn must end exactly once" + end = ends[0] + assert end.status is AgentEndStatus.CRASHED + assert end.crashed is True + assert end.crash_reason == "turn cancelled" + usage = end.usage assert usage is not None assert usage.output_tokens == 2_000 # No ResultMessage means the backend never priced this turn, so the cost is diff --git a/tests/test_agentless.py b/tests/test_agentless.py index 5db1f558..6cb3ca3c 100644 --- a/tests/test_agentless.py +++ b/tests/test_agentless.py @@ -47,6 +47,7 @@ from coder_eval.orchestration.experiment import _apply_cli_overrides, resolve_task_for_variant from coder_eval.orchestration.task_loader import resolve_initial_prompt_file from coder_eval.orchestrator import Orchestrator +from coder_eval.streaming.events import AgentEndStatus def _none_task(criteria=None, **overrides) -> TaskDefinition: @@ -67,11 +68,14 @@ def _none_task(criteria=None, **overrides) -> TaskDefinition: @pytest.mark.asyncio class TestNoOpAgent: async def test_lifecycle_returns_empty_turn(self) -> None: - """start/communicate/stop are no-ops; communicate returns an empty TurnRecord.""" + """start/communicate/stop are no-ops; communicate returns a COMPLETED outcome + wrapping an empty TurnRecord.""" agent = NoOpAgent(NoneAgentConfig(type=AgentKind.NONE)) await agent.start("/tmp/whatever") - turn = await agent.communicate("this prompt is ignored") + outcome = await agent.communicate("this prompt is ignored", iteration=1) + assert outcome.status is AgentEndStatus.COMPLETED + turn = outcome.record assert turn.agent_output == "" assert turn.iteration == 1 assert turn.commands == [] diff --git a/tests/test_antigravity_agent.py b/tests/test_antigravity_agent.py index 20d18c76..e400e925 100644 --- a/tests/test_antigravity_agent.py +++ b/tests/test_antigravity_agent.py @@ -22,20 +22,50 @@ _ANTIGRAVITY_TO_CLAUDE_TOOL_MAP, _DEFAULT_MODEL, AntigravityAgent, + _AntigravityDecoder, _enum_value, _to_token_usage, ) from coder_eval.agents.registry import AgentRegistry -from coder_eval.models import AgentKind, AntigravityAgentConfig, AssistantMessage, RunLimits, parse_agent_config +from coder_eval.models import ( + AgentKind, + AgentState, + AntigravityAgentConfig, + AssistantMessage, + RunLimits, + TimingBasis, + parse_agent_config, +) from coder_eval.orchestration.plugin_staging import stage_plugins from coder_eval.orchestration.turn_monitor import TurnMonitor from coder_eval.plugins import ensure_plugins_loaded from coder_eval.pricing import calculate_cost -from coder_eval.streaming.events import AgentEndEvent, AgentEndStatus, StopReason +from coder_eval.streaming.emitter import TurnEmitter, TurnOutcome +from coder_eval.streaming.events import ( + AgentEndEvent, + AgentEndStatus, + AgentStartEvent, + StopReason, + TextChunkEvent, + ToolEndEvent, + ToolEndStatus, + ToolStartEvent, + TurnEndEvent, + TurnStartEvent, +) +from coder_eval.testing import ( + Replay, + ScriptedClock, + Tick, + assert_identity_closes, + assert_stream_balanced, + replay, +) from tests._bracket_clock import AnchoredClock, assert_bracket_on_the_clock, assert_overhead_is_measured from tests._fixtures.golden_streams._scrub import assert_reconciliation from tests._fixtures.golden_streams.antigravity_fixtures import ( _agent_with_steps, + _FakeConversation, _no_sleep, _step, _tc, @@ -215,15 +245,20 @@ def test_gemini_models_are_priced(model: str): # --- communicate() step-stream mapping (SDK mocked via fake Step stream) --------- +_WATCHDOG = "coder_eval.agents.watchdog.ThreadedWatchdog" + + class _FiringWatchdog: """Fake ThreadedWatchdog that fires ``on_timeout`` synchronously at entry. - Sets ``state.timeout_hit = True`` before any draining happens (exactly - like the real watchdog thread firing early), so a CancelledError raised - later — whether from the first drain or from a poll loop's re-drain — is - classified via the SAME existing ``if state.timeout_hit`` branch. + Sets ``decoder.timeout_hit = True`` before any draining happens (exactly + like the real watchdog thread firing early), and reports ``fired`` so a + CancelledError the body raises later surfaces from ``run_with_watchdog`` as + ``WatchdogFired`` — the watchdog's own cancel, not the caller's. """ + fired = True + def __init__(self, *, on_timeout, **_kwargs): self._on_timeout = on_timeout @@ -278,7 +313,7 @@ async def test_communicate_maps_steps_to_turn_record(): ), ] agent = _agent_with_steps(steps) - tr = await agent.communicate("make hello.py") + tr = (await agent.communicate("make hello.py", iteration=1)).record assert tr.crashed is False assert tr.agent_output == "All done." @@ -314,7 +349,6 @@ async def test_communicate_maps_steps_to_turn_record(): # distinctness (a uuid would pass) does not. ids = [m.message_id for m in tr.messages if isinstance(m, AssistantMessage)] assert ids == ["antigravity-1-msg-0", "antigravity-1-msg-1", "antigravity-1-msg-2"] - assert agent.pending_turn is None # success path leaves no partial async def test_communicate_normalizes_arg_keys_and_strips_done_only_results(): @@ -344,7 +378,7 @@ async def test_communicate_normalizes_arg_keys_and_strips_done_only_results(): ), _step("TEXT_RESPONSE", "DONE", content="done", complete=True, usage=_usage(10, 0, 1, 0)), ] - tr = await _agent_with_steps(steps).communicate("x") + tr = (await _agent_with_steps(steps).communicate("x", iteration=1)).record ls = next(c for c in tr.commands if c.tool_name == "LS") assert ls.parameters == {"path": "/work"} # renamed, results stripped web = next(c for c in tr.commands if c.tool_name == "WebSearch") @@ -364,14 +398,13 @@ async def test_communicate_records_tool_error_from_nonzero_exit(): ), _step("TEXT_RESPONSE", "DONE", content="done", complete=True, usage=_usage(510, 0, 3, 0)), ] - tr = await _agent_with_steps(steps).communicate("run it") + tr = (await _agent_with_steps(steps).communicate("run it", iteration=1)).record bash = next(c for c in tr.commands if c.tool_name == "Bash") assert bash.result_status == "error" -async def test_communicate_crash_sets_pending_partial_turn(): - """A mid-stream SDK error raises AgentCrashError and leaves a crashed partial.""" - from coder_eval.errors import AgentCrashError +async def test_communicate_crash_returns_a_crashed_outcome(): + """A mid-stream SDK error returns a CRASHED outcome carrying a crashed partial record.""" class _Boom: last_response = "" @@ -389,28 +422,24 @@ async def receive_steps(self): agent.working_directory = Path("/tmp") agent._sdk_agent = SimpleNamespace(conversation=_Boom(), is_started=True) - with pytest.raises(AgentCrashError): - await agent.communicate("x") - assert agent.pending_turn is not None - assert agent.pending_turn.crashed is True + outcome = await agent.communicate("x", iteration=1) - await agent.discard_pending_turn() - assert agent.pending_turn is None + assert outcome.status is AgentEndStatus.CRASHED + assert outcome.error == "Antigravity turn failed: kaboom" + assert outcome.record.crashed is True + assert outcome.record.result_summary is None -async def test_communicate_timeout_sets_pending_partial_turn(monkeypatch): - """A turn timeout raises TurnTimeoutError and leaves a crashed partial turn. +async def test_communicate_timeout_returns_a_timeout_outcome(monkeypatch): + """A turn timeout returns a TIMEOUT outcome carrying a crashed partial record. Drives the timeout branch deterministically: a fake watchdog fires its - ``on_timeout`` callback synchronously on entry (setting ``state.timeout_hit``, + ``on_timeout`` callback synchronously on entry (setting ``decoder.timeout_hit``, exactly what the real watchdog thread does), and the step pump then surfaces - the cancel as ``asyncio.CancelledError`` — the :402-404 timeout branch. + the cancel as ``asyncio.CancelledError`` — which ``run_with_watchdog`` turns + into ``WatchdogFired``. """ - import asyncio - - from coder_eval.errors import TurnTimeoutError - - monkeypatch.setattr("coder_eval.agents.antigravity_agent.ThreadedWatchdog", _FiringWatchdog) + monkeypatch.setattr(_WATCHDOG, _FiringWatchdog) class _Cancelled: last_response = "" @@ -428,19 +457,89 @@ async def receive_steps(self): agent.working_directory = Path("/tmp") agent._sdk_agent = SimpleNamespace(conversation=_Cancelled(), is_started=True) - with pytest.raises(TurnTimeoutError): - await agent.communicate("x", timeout=30.0) - assert agent.pending_turn is not None - assert agent.pending_turn.crashed is True + outcome = await agent.communicate("x", iteration=1, timeout=30.0) + + assert outcome.status is AgentEndStatus.TIMEOUT + assert outcome.record.crashed is True + + +async def test_a_real_watchdog_timeout_returns_timeout_and_leaves_the_caller_uncancelled(): + """The real watchdog cancels the turn's CHILD task, never the caller. + + A harness that never yields a step past a 0.2 s budget: the outcome is + ``TIMEOUT``, and the task that awaited ``communicate`` has no pending cancel + request, so the orchestrator's next await is not torn down by a stray cancel. + """ + + class _Hangs: + last_response = "" + + async def send(self, prompt, **kwargs): + return None + + async def receive_steps(self): + await asyncio.Event().wait() + yield # pragma: no cover - makes this an async generator + + async def cancel(self): + return None + + agent = AntigravityAgent(parse_agent_config(type="antigravity")) + agent.working_directory = Path("/tmp") + agent._sdk_agent = SimpleNamespace(conversation=_Hangs(), is_started=True) + + outcome = await asyncio.wait_for(agent.communicate("x", iteration=1, timeout=0.2), timeout=10) + + assert outcome.status is AgentEndStatus.TIMEOUT + assert outcome.record.crashed is True + caller = asyncio.current_task() + assert caller is not None and caller.cancelling() == 0 + await asyncio.sleep(0) # a stray cancel would land on this await + - await agent.discard_pending_turn() - assert agent.pending_turn is None +async def test_an_exception_after_a_requested_stop_ends_with_the_stop_status(): + """Closing the step stream after a stop can raise; the stopped turn still ends clean. + + The generator's own cleanup raises when ``_drain``'s ``aclosing`` closes it + after the ``should_stop`` break — a pulled-step ``RuntimeError`` that is not + retried. The turn was already over by request, so the outcome carries the + stop's status and no error, not ``CRASHED``. + """ + + class _RaisesOnClose: + last_response = "" + + async def send(self, prompt, **kwargs): + return None + + async def receive_steps(self): + try: + yield _step( + "TEXT_RESPONSE", "DONE", content="partial", content_delta="partial", usage=_usage(5, 0, 1, 0) + ) + yield _step("TEXT_RESPONSE", "DONE", content="never pulled") + finally: + raise RuntimeError("aclose boom") + + async def cancel(self): + return None + + agent = AntigravityAgent(parse_agent_config(type="antigravity")) + agent.working_directory = Path("/tmp") + agent._sdk_agent = SimpleNamespace(conversation=_RaisesOnClose(), is_started=True) + + outcome = await agent.communicate("x", iteration=1, should_stop=lambda: StopReason.TOKEN_BUDGET) + + assert outcome.status is AgentEndStatus.TOKEN_BUDGET_EXCEEDED + assert outcome.error is None + assert outcome.record.crashed is False + assert outcome.record.agent_output == "partial" async def test_communicate_requires_started_agent(): agent = AntigravityAgent(parse_agent_config(type="antigravity")) with pytest.raises(RuntimeError, match="not started"): - await agent.communicate("x") + await agent.communicate("x", iteration=1) def _install_fake_sdk(monkeypatch, sdk_agent_cls) -> None: @@ -483,9 +582,7 @@ def test_has_orphaned_tool_call_detects_active_vs_other_statuses(): it will never become DONE on its own (Phase-2-review finding). Layered on top: a cid already in _closed_tools is never orphaned even if its last-seen status were ever left at ACTIVE by a re-emission (final-review finding).""" - from coder_eval.agents.antigravity_agent import _AntigravityTurnState - - state = _AntigravityTurnState.__new__(_AntigravityTurnState) + state = _AntigravityDecoder.__new__(_AntigravityDecoder) state._closed_tools = set() state._tool_last_status = {} assert state.has_orphaned_tool_call() is False # no tool calls at all @@ -534,7 +631,7 @@ async def _sleep_should_not_be_called(_seconds: float) -> None: _step("TEXT_RESPONSE", "DONE", content="done", complete=True, usage=_usage(10, 0, 1, 0)), ] agent = _agent_with_steps(steps) - tr = await agent.communicate("run it") + tr = (await agent.communicate("run it", iteration=1)).record conv = agent._sdk_agent.conversation assert conv.receive_steps_call_count == 1 @@ -565,7 +662,7 @@ async def _sleep_should_not_be_called(_seconds: float) -> None: _step("TEXT_RESPONSE", "DONE", content="waiting on you", complete=True, usage=_usage(10, 0, 1, 0)), ] agent = _agent_with_steps(steps) - tr = await agent.communicate("do it") + tr = (await agent.communicate("do it", iteration=1)).record conv = agent._sdk_agent.conversation assert conv.receive_steps_call_count == 1 # poll loop never entered @@ -627,7 +724,7 @@ async def _record_sleep(seconds: float) -> None: ), ] agent = _agent_with_steps([batch1, batch2]) - tr = await agent.communicate("do it") + tr = (await agent.communicate("do it", iteration=1)).record assert sleep_calls == [antigravity_agent._BACKGROUND_POLL_INTERVAL_SECONDS] bash = next(c for c in tr.commands if c.tool_name == "Bash") @@ -671,7 +768,7 @@ async def test_communicate_resolves_backgrounded_tool_call_with_no_id(monkeypatc _step("TEXT_RESPONSE", "DONE", content="All finished.", complete=True, usage=_usage(5, 0, 1, 0)), ] agent = _agent_with_steps([batch1, batch2]) - tr = await agent.communicate("do it") + tr = (await agent.communicate("do it", iteration=1)).record assert agent._sdk_agent.conversation.receive_steps_call_count == 2 # closed on the first poll, not the cap bash = next(c for c in tr.commands if c.tool_name == "Bash") @@ -707,7 +804,7 @@ async def test_id_less_tool_calls_in_different_trajectories_do_not_collide(): _step("TEXT_RESPONSE", "DONE", content="done", complete=True, usage=_usage(10, 0, 1, 0)), ] agent = _agent_with_steps(steps) - tr = await agent.communicate("do two things") + tr = (await agent.communicate("do two things", iteration=1)).record bash_calls = [c for c in tr.commands if c.tool_name == "Bash"] assert len(bash_calls) == 2 # distinct cids, not collapsed into one @@ -765,7 +862,7 @@ async def _record_sleep(seconds: float) -> None: _step("TEXT_RESPONSE", "DONE", content="all done", complete=True, usage=_usage(10, 0, 1, 0)), ] agent = _agent_with_steps([batch1, batch2, batch3]) - tr = await agent.communicate("do two things") + tr = (await agent.communicate("do two things", iteration=1)).record assert len(sleep_calls) == 2 # exactly two poll cycles, one per backgrounded job bash_calls = [c for c in tr.commands if c.tool_name == "Bash"] @@ -800,7 +897,7 @@ async def _record_sleep(seconds: float) -> None: # empty batch (see _FakeConversation's docstring, matching the real SDK) -- # the orphan is never closed, simulating a job whose state never changes. agent = _agent_with_steps([never_closing]) - tr = await agent.communicate("do it forever") + tr = (await agent.communicate("do it forever", iteration=1)).record assert len(sleep_calls) == 3 # exactly _MAX_BACKGROUND_POLLS, not infinite bash = next(c for c in tr.commands if c.tool_name == "Bash") @@ -830,9 +927,11 @@ async def test_communicate_finalizes_gracefully_under_a_realistic_turn_timeout(m # loop reads time.monotonic() at least twice per iteration (the while-head # check, then the post-sleep deadline check), so this crosses the 240s # deadline (0.8 * 300s) after exactly one poll cycle -- proving the exit is - # driven by the deadline, not by exhausting all 120 cycles. + # driven by the deadline, not by exhausting all 120 cycles. Scoped to the + # adapter module's `time`: patching the shared `time.monotonic` also moves the + # event loop's clock, and the turn body now runs in a child task on that loop. clock = iter([0.0, 130.0, 260.0]) - monkeypatch.setattr(antigravity_agent.time, "monotonic", lambda: next(clock, 1_000_000.0)) + monkeypatch.setattr(antigravity_agent, "time", SimpleNamespace(monotonic=lambda: next(clock, 1_000_000.0))) never_closing = [ _step( @@ -845,9 +944,9 @@ async def test_communicate_finalizes_gracefully_under_a_realistic_turn_timeout(m ] agent = _agent_with_steps([never_closing]) - tr = await agent.communicate("do it forever", timeout=300.0) # the real default turn_timeout + tr = (await agent.communicate("do it forever", iteration=1, timeout=300.0)).record # the real default turn_timeout - # Finalized and graded -- no TurnTimeoutError, no crash. + # Finalized and graded -- no TIMEOUT outcome, no crash. assert tr is not None assert not tr.crashed bash = next(c for c in tr.commands if c.tool_name == "Bash") @@ -859,14 +958,22 @@ async def test_communicate_finalizes_gracefully_under_a_realistic_turn_timeout(m class _WatchdogFiresLater: """Fake ThreadedWatchdog that does NOT fire on entry (unlike _FiringWatchdog - above) -- it hands its ``on_timeout`` callback to the caller so the test can - invoke it mid-poll-loop, simulating a real watchdog thread firing between - poll cycles rather than before the turn even starts.""" + above) -- the test calls ``fire()`` mid-poll-loop, simulating a real watchdog + thread firing between poll cycles rather than before the turn even starts. + ``fire()`` sets ``fired`` and runs ``on_timeout``, as the real timer thread does.""" - captured_on_timeout: Callable[[], None] | None = None + captured: "_WatchdogFiresLater | None" = None - def __init__(self, *, on_timeout, **_kwargs): - _WatchdogFiresLater.captured_on_timeout = on_timeout + def __init__(self, *, on_timeout: Callable[[], None], **_kwargs): + self._on_timeout = on_timeout + self.fired = False + _WatchdogFiresLater.captured = self + + @classmethod + def fire(cls) -> None: + assert cls.captured is not None + cls.captured.fired = True + cls.captured._on_timeout() def __enter__(self): return self @@ -876,7 +983,7 @@ def __exit__(self, *_exc): async def test_communicate_poll_loop_exits_promptly_once_watchdog_flag_lands(monkeypatch): - """A watchdog timeout landing BETWEEN poll cycles (state.timeout_hit flips + """A watchdog timeout landing BETWEEN poll cycles (decoder.timeout_hit flips to True while the loop is sleeping) must stop the loop on its next condition check, not burn through the rest of _MAX_BACKGROUND_POLLS waiting for a cancellation that may not land on this coroutine right away (final-review @@ -884,15 +991,14 @@ async def test_communicate_poll_loop_exits_promptly_once_watchdog_flag_lands(mon from coder_eval.agents import antigravity_agent monkeypatch.setattr(antigravity_agent, "_MAX_BACKGROUND_POLLS", 50) - monkeypatch.setattr("coder_eval.agents.antigravity_agent.ThreadedWatchdog", _WatchdogFiresLater) + monkeypatch.setattr(_WATCHDOG, _WatchdogFiresLater) sleep_calls: list[float] = [] async def _fire_watchdog_on_second_sleep(seconds: float) -> None: sleep_calls.append(seconds) if len(sleep_calls) == 2: - assert _WatchdogFiresLater.captured_on_timeout is not None - _WatchdogFiresLater.captured_on_timeout() + _WatchdogFiresLater.fire() monkeypatch.setattr(antigravity_agent.asyncio, "sleep", _fire_watchdog_on_second_sleep) @@ -905,20 +1011,18 @@ async def _fire_watchdog_on_second_sleep(seconds: float) -> None: ), _step("TEXT_RESPONSE", "DONE", content="waiting...", complete=True, usage=_usage(10, 0, 1, 0)), ] - from coder_eval.errors import TurnTimeoutError agent = _agent_with_steps([never_closing]) - with pytest.raises(TurnTimeoutError): - await agent.communicate("do it forever", timeout=30.0) + outcome = await agent.communicate("do it forever", iteration=1, timeout=30.0) + assert outcome.status is AgentEndStatus.TIMEOUT # Stopped right after the sleep that flipped timeout_hit -- NOT the (patched) cap of 50. assert len(sleep_calls) == 2 # 1 initial drain + 1 poll re-drain (after sleep #1) -- the mid-loop - # `if state.timeout_hit: break` skips the re-drain that would otherwise + # `if decoder.timeout_hit: break` skips the re-drain that would otherwise # follow sleep #2, so no 3rd receive_steps() call happens. assert agent._sdk_agent.conversation.receive_steps_call_count == 2 - assert agent.pending_turn is not None - bash = next(c for c in agent.pending_turn.commands if c.tool_name == "Bash") + bash = next(c for c in outcome.record.commands if c.tool_name == "Bash") assert bash.result_status == "unknown" @@ -959,7 +1063,7 @@ def should_stop() -> StopReason | None: # None for batch1's 2 steps; a reason on the post-sleep check return StopReason.EARLY_CRITERION if call_count > 2 else None - await agent.communicate("do it", should_stop=should_stop) + await agent.communicate("do it", iteration=1, should_stop=should_stop) assert conv.receive_steps_call_count == 1 # the poll's re-drain never happened assert conv.cancel_call_count == 1 @@ -1028,8 +1132,7 @@ async def test_communicate_recovers_from_transient_reentrancy_after_cooperative_ see _drain()'s docstring). The NEXT communicate() call must recover by retrying past that window (mirroring the SDK's own Conversation.send() handling of this exact RuntimeError) instead of crashing with - AgentCrashError.""" - from pathlib import Path + a CRASHED outcome.""" batch1 = [ _step( @@ -1048,13 +1151,14 @@ async def test_communicate_recovers_from_transient_reentrancy_after_cooperative_ agent.working_directory = Path("/tmp") agent._sdk_agent = SimpleNamespace(conversation=conversation, is_started=True) - await agent.communicate("do it", should_stop=lambda: StopReason.EARLY_CRITERION) # breaks after the first step + # breaks after the first step + await agent.communicate("do it", iteration=1, should_stop=lambda: StopReason.EARLY_CRITERION) - # Without the retry, this second call raises AgentCrashError wrapping the - # fake's RuntimeError (verified live before the fix landed). With it, the + # Without the retry, this second call crashes wrapping the fake's + # RuntimeError (verified live before the fix landed). With it, the # transient window clears within a couple of asyncio.sleep(0) yields and # the second turn's real content is delivered, not silently dropped. - tr = await agent.communicate("do it again") + tr = (await agent.communicate("do it again", iteration=2)).record assert tr.agent_output == "second turn" @@ -1090,8 +1194,6 @@ async def test_a_runtime_error_after_a_step_is_not_retried_as_reentrancy(monkeyp """Only an error raised before the first step is the re-entrancy window. A RuntimeError while processing a pulled step is a real failure: retrying it would re-pull the stream and emit the same steps again.""" - from coder_eval.errors import AgentCrashError - conversation_pulls = 0 class _Conversation: @@ -1111,32 +1213,33 @@ async def cancel(self): def _boom(self, step): raise RuntimeError("reducer bug") - monkeypatch.setattr(agent_module._AntigravityTurnState, "process_step", _boom) + monkeypatch.setattr(agent_module._AntigravityDecoder, "__call__", _boom) agent = AntigravityAgent(parse_agent_config(type="antigravity")) agent.working_directory = __import__("pathlib").Path("/tmp") agent._sdk_agent = SimpleNamespace(conversation=_Conversation(), is_started=True) - with pytest.raises(AgentCrashError, match="reducer bug"): - await agent.communicate("do it") + outcome = await agent.communicate("do it", iteration=1) + + assert outcome.status is AgentEndStatus.CRASHED + assert outcome.error is not None and "reducer bug" in outcome.error assert conversation_pulls == 1 async def test_communicate_poll_budget_exhausted_finalizes_via_existing_timeout_path(monkeypatch): """A watchdog timeout landing during the poll loop's re-drain (not the first - drain) must surface as TurnTimeoutError via the SAME existing exception - branch -- the poll loop must not create a second, inconsistent timeout path. + drain) must return a TIMEOUT outcome via the SAME watchdog branch -- the poll + loop must not create a second, inconsistent timeout path. Uses ``_WatchdogFiresLater`` (not ``_FiringWatchdog``, which fires at entry and would make the loop's head condition skip the poll cycle entirely, per - round-3 review) so ``state.timeout_hit`` only flips once a re-drain is + round-3 review) so ``decoder.timeout_hit`` only flips once a re-drain is genuinely in flight -- mirroring the real watchdog, whose ``on_timeout`` callback and the ``CancelledError`` it triggers are the same causal event, not two independently-timed ones.""" from coder_eval.agents import antigravity_agent - from coder_eval.errors import TurnTimeoutError monkeypatch.setattr(antigravity_agent.asyncio, "sleep", _no_sleep) - monkeypatch.setattr("coder_eval.agents.antigravity_agent.ThreadedWatchdog", _WatchdogFiresLater) + monkeypatch.setattr(_WATCHDOG, _WatchdogFiresLater) class _FiresWatchdogThenCancelsOnSecondDrain: last_response = "" @@ -1158,8 +1261,7 @@ async def receive_steps(self): ) yield _step("TEXT_RESPONSE", "DONE", content="started", complete=True, usage=_usage(10, 0, 1, 0)) else: - assert _WatchdogFiresLater.captured_on_timeout is not None - _WatchdogFiresLater.captured_on_timeout() + _WatchdogFiresLater.fire() raise asyncio.CancelledError yield # pragma: no cover - makes this an async generator @@ -1173,14 +1275,13 @@ async def cancel(self): conversation = _FiresWatchdogThenCancelsOnSecondDrain() agent._sdk_agent = SimpleNamespace(conversation=conversation, is_started=True) - with pytest.raises(TurnTimeoutError): - await agent.communicate("x", timeout=30.0) - assert conversation.call_count == 2 # the re-drain genuinely ran, not skipped - assert agent.pending_turn is not None - assert agent.pending_turn.crashed is True + outcome = await agent.communicate("x", iteration=1, timeout=30.0) - await agent.discard_pending_turn() - assert agent.pending_turn is None + assert outcome.status is AgentEndStatus.TIMEOUT + assert conversation.call_count == 2 # the re-drain genuinely ran, not skipped + assert outcome.record.crashed is True + bash = next(c for c in outcome.record.commands if c.tool_name == "Bash") + assert bash.result_status == "unknown" # --- env_path_prepend / mock-CLI PATH shadowing ----------------------------------- @@ -1507,7 +1608,7 @@ async def test_should_stop_reason_ends_the_turn_with_its_status(reason, status, agent = _agent_with_steps(_tool_steps(5)) capture = _EndCapture() - record = await agent.communicate("go", stream_callback=capture, should_stop=lambda: reason) + record = (await agent.communicate("go", iteration=1, stream_callback=capture, should_stop=lambda: reason)).record assert capture.end is not None assert capture.end.status is status @@ -1527,7 +1628,7 @@ def should_stop() -> StopReason | None: polls += 1 return StopReason.TOOL_CALL_CAP if polls >= 2 else None - record = await agent.communicate("go", should_stop=should_stop) + record = (await agent.communicate("go", iteration=1, should_stop=should_stop)).record assert len(record.commands) == 1 assert record.commands[0].result_status == "success" @@ -1537,7 +1638,7 @@ def should_stop() -> StopReason | None: async def test_no_reason_consumes_every_step(): agent = _agent_with_steps(_tool_steps(4)) - record = await agent.communicate("go", should_stop=lambda: None) + record = (await agent.communicate("go", iteration=1, should_stop=lambda: None)).record assert len(record.commands) == 4 assert record.tool_calls_exhausted is False @@ -1582,7 +1683,8 @@ async def test_cap_reached_on_a_poll_redrain_stops_polling(monkeypatch): conv = agent._sdk_agent.conversation monitor = TurnMonitor("t", [], limits=RunLimits(max_tool_calls=2)) - record = await agent.communicate("go", stream_callback=monitor, should_stop=monitor.should_stop) + outcome = await agent.communicate("go", iteration=1, stream_callback=monitor, should_stop=monitor.should_stop) + record = outcome.record assert monitor.stop_reason is StopReason.TOOL_CALL_CAP assert record.tool_calls_exhausted is True @@ -1608,59 +1710,57 @@ async def test_cap_reached_on_a_poll_redrain_stops_polling(monkeypatch): _CLOCK_BASE = datetime(2026, 1, 1, 12, 0, 0) -class _Clock: - """Controlled stand-in for the reducer's clocks — a `TurnClock` and `time`. +def _at(ms: float) -> datetime: + return _CLOCK_BASE + timedelta(milliseconds=ms) - ONE monotonically advancing counter, read by both: every read — the turn - clock's `now()` or `time.monotonic()` — costs TICK_MS. So the fixture's - timeline is driven by read ORDER, not by elapsed time, and the two are - deliberately coupled rather than independent. That is enough to pin the - arithmetic exactly. - Every WALL stamp the reducer records now derives from its per-turn - `TurnClock`, so this stands in for that object rather than for the - module's `datetime`. That distinction is load-bearing, not cosmetic: a - derived stamp does not read `datetime.now()`, so the old patch would no - longer reach it and these tests would quietly measure the real clock and - pass by accident. `time` is still patched because `duration_seconds` and - the poll deadlines read `time.monotonic()` directly, and must — a deadline - may not move when the wall clock steps. +def _replay( + stream: list[Any], + *, + status: AgentEndStatus = AgentEndStatus.COMPLETED, + reason: str | None = None, + agent_output: str | None = None, +) -> tuple[Replay, _AntigravityDecoder]: + """Drive an `_AntigravityDecoder` through `coder_eval.testing.replay` from `_CLOCK_BASE`. - What it still does NOT prove is that the reducer keeps the two in their - proper roles; with one basis for every wall stamp there is no longer a - second role to confuse it with. + Ends through `decoder.end`; returns the decoder too, for the tests that pin its + bookkeeping. """ + decoders: list[_AntigravityDecoder] = [] - TICK_MS = 100.0 + def make(emitter: TurnEmitter) -> _AntigravityDecoder: + decoder = _AntigravityDecoder(emitter) + decoders.append(decoder) + return decoder - def __init__(self) -> None: - self.ms = 0.0 + def end(decoder: _AntigravityDecoder) -> TurnOutcome: + return decoder.end(status, reason=reason, agent_output=agent_output) - def _advance(self) -> float: - self.ms += self.TICK_MS - return self.ms + result = replay(stream, make, clock=ScriptedClock(_CLOCK_BASE), model="gemini-3.5-flash", end=end) + return result, decoders[0] - def monotonic(self) -> float: - return self._advance() / 1000.0 - def now(self) -> datetime: - return _CLOCK_BASE + timedelta(milliseconds=self._advance()) +def _assistant(record): + return [m for m in record.messages if m.role == "assistant"] -def _install_clock(monkeypatch, clock: _Clock) -> None: - """Hand the reducer this clock for the turn it is about to build. +def _opening_step(): + """A MODEL step that seeds the first window's mark and adds no block.""" + return _step("THINKING", "ACTIVE", thinking="...") - `TurnClock` is replaced by a factory rather than the fake being passed - positionally, because the state — and therefore its clock — is built - inside `communicate()`, out of the caller's reach. One typed seam, and the - stand-in has to satisfy `now()`. - """ - monkeypatch.setattr(agent_module, "time", SimpleNamespace(monotonic=clock.monotonic)) - monkeypatch.setattr(agent_module, "TurnClock", lambda: clock) +def _thinking_done(text: str): + return _step("THINKING", "DONE", thinking=text, usage=_usage(100, 0, 5, 5)) + + +def _bash_active(*ids: str): + calls = [_tc("run_command", tid, {"command_line": tid}) for tid in ids] + return _step("TOOL_CALL", "ACTIVE", target="TARGET_ENVIRONMENT", tool_calls=calls) -def _assistant(record): - return [m for m in record.messages if m.role == "assistant"] + +def _bash_done(*ids: str): + calls = [_tc("run_command", tid, {"command_line": tid, "exit_code": 0}) for tid in ids] + return _step("TOOL_CALL", "DONE", target="TARGET_ENVIRONMENT", tool_calls=calls) class TestBusyMs: @@ -1716,47 +1816,43 @@ def test_a_zero_length_span_is_dropped(self): assert self._busy([(100, 100)]) == 0.0 -async def test_concurrent_tools_do_not_over_subtract(monkeypatch): +async def test_concurrent_tools_do_not_over_subtract(): """Overlapping tool calls are subtracted once, not once each. - Four calls opened by one Step and closed by the next overlap almost - entirely. Summing their durations exceeded the window and clamped + Four calls opened by one Step and closed by the next overlap entirely. + Summing their durations exceeded the window and clamped `generation_duration_ms` to 0.0 — the pre-change symptom, with a 0% breakdown on the task page and nothing failing. """ - _install_clock(monkeypatch, _Clock()) - opens = _step( - "TOOL_CALL", - "ACTIVE", - target="TARGET_ENVIRONMENT", - tool_calls=[_tc("run_command", f"t{i}", {"command_line": f"job{i}"}) for i in range(4)], - ) - closes = _step( - "TOOL_CALL", - "DONE", - target="TARGET_ENVIRONMENT", - tool_calls=[_tc("run_command", f"t{i}", {"command_line": f"job{i}", "exit_code": 0}) for i in range(4)], + ids = ("t0", "t1", "t2", "t3") + result, _ = _replay( + [ + Tick(50), + _opening_step(), + Tick(100), + _thinking_done("first"), + Tick(200), + _bash_active(*ids), + Tick(600), + _bash_done(*ids), + Tick(1000), + _thinking_done("second"), + ] ) - steps = [ - _step("THINKING", "DONE", thinking="first", usage=_usage(100, 0, 5, 5)), - opens, - closes, - _step("THINKING", "DONE", thinking="second", usage=_usage(100, 0, 5, 5)), - ] - record = await _agent_with_steps(steps).communicate("go") + record = result.record second = _assistant(record)[1] - tools = [c for c in record.commands if c.tool_id.startswith("t")] + tools = record.commands assert len(tools) == 4 span_ms = (second.completed_at - second.started_at).total_seconds() * 1000.0 summed_ms = sum(c.duration_ms or 0.0 for c in tools) + assert span_ms == pytest.approx(900.0) assert summed_ms > span_ms, "fixture must make the naive sum exceed the window" - assert second.generation_duration_ms > 0, "the naive sum clamped this to 0.0" - # Union of the four overlapping intervals, not their sum. busy_ms = ( max(c.execution_completed_at for c in tools) - min(c.execution_started_at for c in tools) ).total_seconds() * 1000.0 + assert busy_ms == pytest.approx(400.0) assert second.generation_duration_ms == pytest.approx(span_ms - busy_ms) @@ -1770,7 +1866,7 @@ async def test_generation_window_is_measured_not_zero(): _step("THINKING", "DONE", thinking="first", usage=_usage(100, 0, 5, 5)), _step("THINKING", "DONE", thinking="second", usage=_usage(120, 0, 6, 4)), ] - record = await _agent_with_steps(steps).communicate("go") + record = (await _agent_with_steps(steps).communicate("go", iteration=1)).record messages = _assistant(record) assert len(messages) == 2 @@ -1787,7 +1883,7 @@ async def test_consecutive_windows_chain_end_to_start(): _step("THINKING", "DONE", thinking="b", usage=_usage(100, 0, 5, 5)), _step("TEXT_RESPONSE", "DONE", content="c", content_delta="c", complete=True, usage=_usage(100, 0, 5, 0)), ] - record = await _agent_with_steps(steps).communicate("go") + record = (await _agent_with_steps(steps).communicate("go", iteration=1)).record messages = _assistant(record) assert len(messages) == 3 @@ -1795,7 +1891,7 @@ async def test_consecutive_windows_chain_end_to_start(): assert later.started_at == earlier.completed_at -async def test_tool_execution_is_subtracted_from_the_window(monkeypatch): +async def test_tool_execution_is_subtracted_from_the_window(): """A tool closing inside a generation is not counted as model time. This is the test that pins the design decision. Without it, "simplifying" @@ -1803,52 +1899,42 @@ async def test_tool_execution_is_subtracted_from_the_window(monkeypatch): real model time, because a harness-local tool can close 8 ms after it opens while seconds of model time separate the two flushes around it. """ - clock = _Clock() - _install_clock(monkeypatch, clock) - steps = [ - _step("THINKING", "DONE", thinking="plan", usage=_usage(100, 0, 5, 5)), - _step( - "TOOL_CALL", - "ACTIVE", - target="TARGET_ENVIRONMENT", - tool_calls=[_tc("run_command", "t1", {"command_line": "ls"})], - ), - _step( - "TOOL_CALL", - "DONE", - target="TARGET_ENVIRONMENT", - tool_calls=[_tc("run_command", "t1", {"command_line": "ls", "exit_code": 0})], - ), - _step( - "TEXT_RESPONSE", "DONE", content="done", content_delta="done", complete=True, usage=_usage(200, 0, 10, 0) - ), - ] - record = await _agent_with_steps(steps).communicate("go") + result, _ = _replay( + [ + Tick(50), + _opening_step(), + Tick(100), + _thinking_done("plan"), + Tick(300), + _bash_active("t1"), + Tick(400), + _bash_done("t1"), + Tick(1000), + _step( + "TEXT_RESPONSE", + "DONE", + content="done", + content_delta="done", + complete=True, + usage=_usage(200, 0, 10, 0), + ), + ] + ) + record = result.record messages = _assistant(record) assert len(messages) == 2 second = messages[1] bash = next(c for c in record.commands if c.tool_name == "Bash") - # The window contains a 100ms tool call, so what is left of it was the - # model generating. That relation is the assertion that matters, and it is - # independent of the fixture's tick size. span_ms = (second.completed_at - second.started_at).total_seconds() * 1000.0 assert bash.duration_ms == pytest.approx(100.0) assert second.generation_duration_ms == pytest.approx(span_ms - bash.duration_ms) - - # The absolute figures are artifacts of `_Clock`, which charges one TICK_MS - # per clock READ. They moved from 400/300 to 300/200 when the reducer - # stopped taking a monotonic reading it no longer needs: a flush now reads - # the turn clock once where it used to read two clocks, so each window is - # one tick shorter on this fixture's read-driven timeline. Nothing about - # real elapsed time changed — the 100ms tool, which is still two reads - # apart, is unmoved. - assert span_ms == pytest.approx(300.0) - assert second.generation_duration_ms == pytest.approx(200.0) + assert span_ms == pytest.approx(900.0) + assert second.generation_duration_ms == pytest.approx(800.0) -async def test_a_straddling_tool_is_charged_only_for_its_in_window_part(monkeypatch): +async def test_a_straddling_tool_is_charged_only_for_its_in_window_part(): """A tool open across a flush is clipped to the window it is subtracted from. `t1` opens before the first flush and closes after it. Only the part that @@ -1857,39 +1943,23 @@ async def test_a_straddling_tool_is_charged_only_for_its_in_window_part(monkeypa overhang, drive the result to a clamped 0.0 — the very value this change exists to stop publishing. """ - _install_clock(monkeypatch, _Clock()) - steps = [ - # t1 opens here and stays open across the first flush. - _step( - "TOOL_CALL", - "ACTIVE", - target="TARGET_ENVIRONMENT", - tool_calls=[_tc("run_command", "t1", {"command_line": "slow"})], - ), - # t2 opens and closes entirely inside the first window. - _step( - "TOOL_CALL", - "ACTIVE", - target="TARGET_ENVIRONMENT", - tool_calls=[_tc("run_command", "t2", {"command_line": "quick"})], - ), - _step( - "TOOL_CALL", - "DONE", - target="TARGET_ENVIRONMENT", - tool_calls=[_tc("run_command", "t2", {"command_line": "quick", "exit_code": 0})], - ), - _step("THINKING", "DONE", thinking="first", usage=_usage(100, 0, 5, 5)), - # t1 closes in the SECOND window, carrying the first window's overhang. - _step( - "TOOL_CALL", - "DONE", - target="TARGET_ENVIRONMENT", - tool_calls=[_tc("run_command", "t1", {"command_line": "slow", "exit_code": 0})], - ), - _step("THINKING", "DONE", thinking="second", usage=_usage(100, 0, 5, 5)), - ] - record = await _agent_with_steps(steps).communicate("go") + result, _ = _replay( + [ + Tick(100), + _bash_active("t1"), # opens here and stays open across the first flush + Tick(200), + _bash_active("t2"), # opens and closes entirely inside the first window + Tick(300), + _bash_done("t2"), + Tick(500), + _thinking_done("first"), + Tick(800), + _bash_done("t1"), # closes in the SECOND window, carrying the first window's overhang + Tick(1000), + _thinking_done("second"), + ] + ) + record = result.record second = _assistant(record)[1] slow = next(c for c in record.commands if c.tool_id == "t1") @@ -1899,10 +1969,10 @@ async def test_a_straddling_tool_is_charged_only_for_its_in_window_part(monkeypa assert slow.duration_ms > span_ms, "fixture must produce a straddling tool" assert 0 < in_window_ms < slow.duration_ms, "part of the tool ran before this window" assert second.generation_duration_ms == pytest.approx(span_ms - in_window_ms) - assert second.generation_duration_ms > 0 + assert second.generation_duration_ms == pytest.approx(200.0) -async def test_a_tool_still_open_at_the_flush_is_not_generation_time(monkeypatch): +async def test_a_tool_still_open_at_the_flush_is_not_generation_time(): """The sibling of the straddle test above, for the window the tool opened IN. Subtracting only CLOSED intervals published the part of a still-running @@ -1914,30 +1984,25 @@ async def test_a_tool_still_open_at_the_flush_is_not_generation_time(monkeypatch `duration_seconds`, on a turn whose entire headroom was 1.4 ms. Four sibling runs passed by 1.2-8.7 ms out of ~12 s, so it was a coin flip. """ - _install_clock(monkeypatch, _Clock()) - steps = [ - # t1 opens here and is STILL RUNNING when the first window is cut. - _step( - "TOOL_CALL", - "ACTIVE", - target="TARGET_ENVIRONMENT", - tool_calls=[_tc("run_command", "t1", {"command_line": "slow"})], - ), - _step("THINKING", "DONE", thinking="first", usage=_usage(100, 0, 5, 5)), - _step( - "TOOL_CALL", - "DONE", - target="TARGET_ENVIRONMENT", - tool_calls=[_tc("run_command", "t1", {"command_line": "slow", "exit_code": 0})], - ), - _step("THINKING", "DONE", thinking="second", usage=_usage(100, 0, 5, 5)), - ] - record = await _agent_with_steps(steps).communicate("go") + result, _ = _replay( + [ + Tick(100), + _opening_step(), + Tick(200), + _bash_active("t1"), # STILL RUNNING when the first window is cut + Tick(500), + _thinking_done("first"), + Tick(800), + _bash_done("t1"), + Tick(1000), + _thinking_done("second"), + ] + ) + record = result.record first = _assistant(record)[0] slow = next(c for c in record.commands if c.tool_id == "t1") span_ms = (first.completed_at - first.started_at).total_seconds() * 1000.0 - # The part of t1 that had already run when this window was cut. in_window_ms = (first.completed_at - slow.execution_started_at).total_seconds() * 1000.0 assert slow.execution_started_at < first.completed_at, "fixture must open the tool in this window" @@ -1948,7 +2013,7 @@ async def test_a_tool_still_open_at_the_flush_is_not_generation_time(monkeypatch assert first.generation_duration_ms + in_window_ms == pytest.approx(span_ms) -async def test_a_no_op_flush_does_not_move_the_mark(monkeypatch): +async def test_a_no_op_flush_does_not_move_the_mark(): """An empty generation must leave the open window alone. The early return in `_flush_generation` sits before any mark handling, so @@ -1956,29 +2021,28 @@ async def test_a_no_op_flush_does_not_move_the_mark(monkeypatch): otherwise the real generation that follows reports only the time since the empty one. """ - real = _step("THINKING", "DONE", thinking="real", usage=_usage(100, 0, 5, 5)) + real = _thinking_done("real") # Zero usage and no content: reaches the flush, appends nothing. empty = _step("THINKING", "DONE", usage=_usage(0, 0, 0, 0)) - # Two runs off identical fresh clocks. The empty flush returns before any - # clock read, so it must leave the window — and therefore the real - # generation's recorded bounds — byte-identical. - _install_clock(monkeypatch, _Clock()) - without = _assistant(await _agent_with_steps([real]).communicate("go")) - - _install_clock(monkeypatch, _Clock()) - with_empty = _assistant(await _agent_with_steps([empty, real]).communicate("go")) + without, _ = _replay([Tick(100), _opening_step(), Tick(1000), real]) + with_empty, decoder = _replay([Tick(100), _opening_step(), Tick(500), empty, Tick(1000), real]) - assert len(with_empty) == 1, "the empty generation must not produce a message" - assert with_empty[0].started_at == without[0].started_at - assert with_empty[0].generation_duration_ms == without[0].generation_duration_ms + assert decoder.generations == 1 + with_messages = _assistant(with_empty.record) + assert len(with_messages) == 1, "the empty generation must not produce a message" + assert with_messages[0].started_at == _assistant(without.record)[0].started_at == _at(100) + assert with_messages[0].generation_duration_ms == _assistant(without.record)[0].generation_duration_ms async def test_generation_and_tool_time_account_for_the_turn(): - """Σ generation + Σ tool + head + tail lands inside the turn's own duration. + """Σ generation + tool union + head + tail tiles the turn's own bracket. - Bounds, not equality: the fake conversation's own overhead sits in the - residual. Before the window existed the generation half was identically 0. + Measured against the ``AgentStartEvent`` / ``AgentEndEvent`` stamps, the span + the buckets are defined on. Not against ``duration_seconds``: the emitter reads + that on a separate monotonic call before it stamps the end event, so on this + sub-millisecond fake turn the bracket exceeds it by a few microseconds every + time. Before the window existed the generation half was identically 0. The HEAD is part of the sum, and has to be: the first window now opens at the first observed `Step` rather than at turn entry, so the dispatch before @@ -2005,32 +2069,29 @@ async def test_generation_and_tool_time_account_for_the_turn(): "TEXT_RESPONSE", "DONE", content="done", content_delta="done", complete=True, usage=_usage(200, 0, 10, 0) ), ] - record = await _agent_with_steps(steps).communicate("go") + seen: list[Any] = [] + record = ( + await _agent_with_steps(steps).communicate( + "go", iteration=1, stream_callback=SimpleNamespace(on_event=seen.append) + ) + ).record gen_ms = sum(m.generation_duration_ms or 0.0 for m in _assistant(record)) - tool_ms = sum(c.duration_ms or 0.0 for c in record.commands) head_ms = record.harness_startup_ms or 0.0 - tail_ms = record.harness_teardown_ms or 0.0 - turn_ms = record.duration_seconds * 1000.0 assert gen_ms > 0 assert head_ms > 0, "the dispatch before the first Step is now a measured bucket, not 0.0" - assert gen_ms + tool_ms + head_ms + tail_ms <= turn_ms - - # NO relative LOWER bound. This case runs on the REAL clock, and the fake - # conversation's own overhead is the residual — under parallel load the - # denominator (`duration_seconds`, the agent's monotonic span) inflates - # while the measured buckets do not, so any `>= share * turn_ms` assertion - # is a scheduler-noise detector. It was one: a `>= 0.5 *` bound survived - # here only while the sum excluded the head, and failed under `-n auto` - # once the head joined it. - # - # The share this test was reaching for IS asserted, exactly, in - # tests/test_timing_identity_contract.py — on a scripted clock, where the - # magnitudes are real and the identity closes to the millisecond. What is - # left here is what an end-to-end run can honestly claim: the buckets are - # measured, the head is no longer the clamped 0.0, and nothing overflows - # the turn. + assert_identity_closes( + record, + started_at=next(e.timestamp for e in seen if isinstance(e, AgentStartEvent)), + ended_at=next(e.timestamp for e in seen if isinstance(e, AgentEndEvent)), + ) + + # NO share-of-turn LOWER bound on generation. This case runs on the REAL + # clock, where the fake conversation's own overhead lands in head and tail, + # so any `>= share * turn_ms` assertion is a scheduler-noise detector. The + # magnitudes are asserted on a scripted clock in + # tests/test_timing_identity_contract.py. async def test_timing_change_moves_no_token_bucket(): @@ -2058,7 +2119,7 @@ async def test_timing_change_moves_no_token_bucket(): "TEXT_RESPONSE", "DONE", content="done", content_delta="done", complete=True, usage=_usage(1300, 0, 30, 0) ), ] - record = await _agent_with_steps(steps).communicate("go") + record = (await _agent_with_steps(steps).communicate("go", iteration=1)).record assert record.token_usage is not None assert record.token_usage.output_tokens == (10 + 20) + (15 + 5) + (30 + 0) @@ -2069,36 +2130,32 @@ async def test_timing_change_moves_no_token_bucket(): assert all(m.generation_duration_ms is not None for m in _assistant(record)) -async def test_the_published_window_reconciles_to_its_own_bounds(monkeypatch): +def _plan_tool_second_stream() -> list[Any]: + return [ + Tick(50), + _opening_step(), + Tick(100), + _thinking_done("plan"), + Tick(300), + _bash_active("t1"), + Tick(450), + _bash_done("t1"), + Tick(1000), + _thinking_done("second"), + ] + + +async def test_the_published_window_reconciles_to_its_own_bounds(): """The reducer subtracted exactly the spans the record carries. - The per-migrated-reducer check its three siblings gained when they moved - onto `close_window`; antigravity could not have it until its span stopped - being monotonic while these intervals were wall. `decompose_run.py` and the - evalboard's Unaccounted cell both recompute the tool UNION from the - recorded command spans and subtract it from the recorded window bounds, so - this asserts the reducer fed the window that same set. + `decompose_run.py` and the evalboard's Unaccounted cell both recompute the + tool UNION from the recorded command spans and subtract it from the + recorded window bounds, so this asserts the published window agrees with + that same set. """ from coder_eval.timing import busy_ms - _install_clock(monkeypatch, _Clock()) - steps = [ - _step("THINKING", "DONE", thinking="plan", usage=_usage(100, 0, 5, 5)), - _step( - "TOOL_CALL", - "ACTIVE", - target="TARGET_ENVIRONMENT", - tool_calls=[_tc("run_command", "t1", {"command_line": "ls"})], - ), - _step( - "TOOL_CALL", - "DONE", - target="TARGET_ENVIRONMENT", - tool_calls=[_tc("run_command", "t1", {"command_line": "ls", "exit_code": 0})], - ), - _step("THINKING", "DONE", thinking="second", usage=_usage(100, 0, 5, 5)), - ] - record = await _agent_with_steps(steps).communicate("go") + record = _replay(_plan_tool_second_stream())[0].record second = _assistant(record)[1] spans = [ @@ -2109,9 +2166,10 @@ async def test_the_published_window_reconciles_to_its_own_bounds(monkeypatch): span_ms = (second.completed_at - second.started_at).total_seconds() * 1000.0 expected = span_ms - busy_ms(spans, second.started_at, second.completed_at) assert second.generation_duration_ms == pytest.approx(expected) + assert second.generation_duration_ms == pytest.approx(750.0) -async def test_the_window_is_measured_without_relying_on_the_negative_clamp(monkeypatch): +async def test_the_window_is_measured_without_relying_on_the_negative_clamp(): """A positive window, and no clamp underneath it. The span used to be read off `time.monotonic()` while the tool intervals @@ -2121,24 +2179,7 @@ async def test_the_window_is_measured_without_relying_on_the_negative_clamp(monk that unrepresentable: `busy_ms` clips to the window and unions overlaps, so it cannot exceed a span derived from the same clock. """ - _install_clock(monkeypatch, _Clock()) - steps = [ - _step("THINKING", "DONE", thinking="plan", usage=_usage(100, 0, 5, 5)), - _step( - "TOOL_CALL", - "ACTIVE", - target="TARGET_ENVIRONMENT", - tool_calls=[_tc("run_command", "t1", {"command_line": "ls"})], - ), - _step( - "TOOL_CALL", - "DONE", - target="TARGET_ENVIRONMENT", - tool_calls=[_tc("run_command", "t1", {"command_line": "ls", "exit_code": 0})], - ), - _step("THINKING", "DONE", thinking="second", usage=_usage(100, 0, 5, 5)), - ] - record = await _agent_with_steps(steps).communicate("go") + record = _replay(_plan_tool_second_stream())[0].record second = _assistant(record)[1] assert second.generation_duration_ms > 0.0 @@ -2158,13 +2199,13 @@ async def test_each_turn_gets_a_fresh_clock(): """ step = _step("THINKING", "DONE", thinking="a", usage=_usage(100, 0, 5, 5)) agent = _agent_with_steps([step]) - first = _assistant(await agent.communicate("go")) + first = _assistant((await agent.communicate("go", iteration=1)).record) # The fake conversation yields one batch and is then spent, so borrow a # fresh one. The agent INSTANCE is deliberately the same: what is under # test is that its second turn builds its own clock rather than inheriting # the first turn's origin. agent._sdk_agent = _agent_with_steps([step])._sdk_agent - second = _assistant(await agent.communicate("again")) + second = _assistant((await agent.communicate("again", iteration=2)).record) assert first and second # Re-anchored: the later turn's window opens after the earlier one closed. @@ -2173,13 +2214,12 @@ async def test_each_turn_gets_a_fresh_clock(): class TestAntigravityFirstWindowReseed: - """The first `Step` moves `_gen_mark_wall`; a later one must not. + """The first MODEL `Step` moves `_gen_mark`; a later one must not. - Driven at `_AntigravityTurnState` with an injected clock, NOT through - `communicate()`: the fake conversation yields with no delay, so an - end-to-end run cannot pin the MAGNITUDE — the two stamps land within - microseconds of each other, so no assertion there could say the mark moved - by the right amount. + Replayed on a scripted clock, NOT through `communicate()`: the fake + conversation yields with no delay, so an end-to-end run cannot pin the + MAGNITUDE — the two stamps land within microseconds of each other, so no + assertion there could say the mark moved by the right amount. It can detect the mark moving at all, and does: `test_generation_and_tool_time_account_for_the_turn` asserts `head_ms > 0` @@ -2187,34 +2227,12 @@ class TestAntigravityFirstWindowReseed: WHERE it moved to and that it moves only once. """ - BASE = datetime(2026, 9, 11, 9, 0, 0) - class _Clock: - def __init__(self, at_ms: float = 0.0) -> None: - self.at_ms = at_ms + def __init__(self) -> None: + self.at_ms = 0.0 def now(self) -> datetime: - return TestAntigravityFirstWindowReseed.BASE + timedelta(milliseconds=self.at_ms) - - def _state(self, clock): - from coder_eval.agents.antigravity_agent import _AntigravityTurnState - from coder_eval.streaming.callbacks import CompositeStreamCallback - from coder_eval.streaming.collector import EventCollector - - agent = AntigravityAgent(parse_agent_config(type="antigravity", model="gemini-3.5-flash")) - collector = EventCollector() - return _AntigravityTurnState( - agent=agent, - emit=CompositeStreamCallback([collector]), - task_id="t", - turn_id="turn", - collector=collector, - user_input="go", - iteration=1, - model="gemini-3.5-flash", - turn_start_time=0.0, - clock=clock, - ) + return _at(self.at_ms) def test_the_first_step_moves_the_mark_off_the_turn_entry_stamp(self): """Dispatch before the first Step is head, not the first generation. @@ -2223,54 +2241,55 @@ def test_the_first_step_moves_the_mark_off_the_turn_entry_stamp(self): so this interval was published as generation — ~4.7 s per turn against a later-window median of 3.3 s. """ - clock = self._Clock() - state = self._state(clock) - assert state._gen_mark_wall == self.BASE + _, decoder = _replay([Tick(900), _opening_step()]) # dispatch + TTFT - clock.at_ms = 900 # dispatch + TTFT - state.process_step(_step("THINKING", "ACTIVE", thinking="...")) - - assert state._gen_mark_wall == self.BASE + timedelta(milliseconds=900) + assert decoder._first_output_seen is True + assert decoder._gen_mark == _at(900) def test_a_later_step_does_not_move_it(self): """Re-seeding more than once per turn is the defect, not the feature.""" - clock = self._Clock() - state = self._state(clock) - clock.at_ms = 900 - state.process_step(_step("THINKING", "ACTIVE", thinking="...")) - seeded = state._gen_mark_wall - - clock.at_ms = 5000 - state.process_step(_step("THINKING", "ACTIVE", thinking="more")) + _, decoder = _replay([Tick(900), _opening_step(), Tick(5000), _step("THINKING", "ACTIVE", thinking="more")]) - assert state._gen_mark_wall == seeded + assert decoder._gen_mark == _at(900) def test_seeding_twice_by_hand_is_a_no_op_the_second_time(self): """The once-per-turn guard, stated outright rather than inferred.""" clock = self._Clock() - state = self._state(clock) - clock.at_ms = 900 - state._seed_first_generation_window("MODEL") - seeded = state._gen_mark_wall + emitter = TurnEmitter( + task_id="t", + iteration=1, + prompt="go", + model="gemini-3.5-flash", + basis=TimingBasis.TURN_CLOCK, + clock=clock, + sinks=[], + ) + emitter.begin() + decoder = _AntigravityDecoder(emitter) + assert decoder._gen_mark == _CLOCK_BASE + clock.at_ms = 900 + decoder._seed_first_generation_window("MODEL") clock.at_ms = 5000 - state._seed_first_generation_window("MODEL") + decoder._seed_first_generation_window("MODEL") - assert state._gen_mark_wall == seeded + assert decoder._gen_mark == _at(900) def test_a_flush_still_advances_the_mark_and_opens_at_the_reseeded_one(self): """The re-seed must not break the tiling it sits in front of.""" - clock = self._Clock() - state = self._state(clock) - clock.at_ms = 900 - state.process_step(_step("THINKING", "ACTIVE", thinking="plan")) - clock.at_ms = 2000 - state.process_step(_step("THINKING", "DONE", thinking="plan", usage=_usage(100, 0, 5, 5))) + result, decoder = _replay( + [ + Tick(900), + _step("THINKING", "ACTIVE", thinking="plan"), + Tick(2000), + _thinking_done("plan"), + ] + ) - message = _assistant(state)[0] - assert message.started_at == self.BASE + timedelta(milliseconds=900), "opens at the RE-SEEDED mark" + message = _assistant(result.record)[0] + assert message.started_at == _at(900), "opens at the RE-SEEDED mark" assert message.generation_duration_ms == pytest.approx(1100.0) - assert state._gen_mark_wall == self.BASE + timedelta(milliseconds=2000), "and the flush advances it" + assert decoder._gen_mark == _at(2000), "and the flush advances it" def test_a_non_model_step_does_not_seed_the_window(self): """The field is MODEL output, and the SDK streams Steps that are not. @@ -2280,27 +2299,227 @@ def test_a_non_model_step_does_not_seed_the_window(self): one. Seeding on it would put the mark before the model spoke and hand the remainder back to msg0's generation — the defect being fixed. """ - clock = self._Clock() - state = self._state(clock) + system = _step("SYSTEM_MESSAGE", "DONE", source="SYSTEM", content="compacting") - clock.at_ms = 400 - state.process_step(_step("SYSTEM_MESSAGE", "DONE", source="SYSTEM", content="compacting")) - assert state._first_output_seen is False - assert state._gen_mark_wall == self.BASE, "a system Step must not open the generation window" + _, before = _replay([Tick(400), system]) + assert before._first_output_seen is False + assert before._gen_mark == _CLOCK_BASE, "a system Step must not open the generation window" - clock.at_ms = 900 - state.process_step(_step("THINKING", "ACTIVE", thinking="...")) - assert state._gen_mark_wall == self.BASE + timedelta(milliseconds=900), "the first MODEL Step does" + _, after = _replay([Tick(400), system, Tick(900), _opening_step()]) + assert after._gen_mark == _at(900), "the first MODEL Step does" def test_a_turn_that_streams_no_step_keeps_the_turn_entry_mark(self): - clock = self._Clock() - state = self._state(clock) - assert state._first_output_seen is False - assert state._gen_mark_wall == self.BASE + _, decoder = _replay([]) + assert decoder._first_output_seen is False + assert decoder._gen_mark == _CLOCK_BASE + + +class TestAntigravityDecoder: + """`_AntigravityDecoder` over a real emitter: ids, parameters, orphans, tokens and replies.""" + + def test_an_id_less_call_falls_back_to_a_trajectory_scoped_id(self): + """`{name}_{trajectory}:{step_index}_{call_index}`, stable across ACTIVE -> DONE.""" + + def calls(done: bool) -> list[Any]: + extra = {"exit_code": 0} if done else {} + return [ + _tc("run_command", None, {"command_line": "a", **extra}), + _tc("view_file", None, {"file_path": "x.py"}), + ] + + result, _ = _replay( + [ + _step( + "TOOL_CALL", + "ACTIVE", + target="TARGET_ENVIRONMENT", + tool_calls=calls(False), + step_index=3, + trajectory_id="traj", + ), + _step( + "TOOL_CALL", + "DONE", + target="TARGET_ENVIRONMENT", + tool_calls=calls(True), + step_index=3, + trajectory_id="traj", + ), + _step( + "TOOL_CALL", + "DONE", + target="TARGET_ENVIRONMENT", + tool_calls=[_tc("run_command", None, {"command_line": "b", "exit_code": 0})], + step_index=4, + ), + ] + ) + + commands = {c.tool_id: c for c in result.record.commands} + assert set(commands) == {"run_command_traj:3_0", "view_file_traj:3_1", "run_command_4_0"} + assert all(c.result_status == "success" for c in commands.values()) + assert_stream_balanced(result.events) + + def test_parameters_keep_only_the_input_keys_seen_at_start(self): + """A key first seen at DONE is the harness's result payload, whatever its name.""" + result, _ = _replay( + [ + _step( + "TOOL_CALL", + "ACTIVE", + target="TARGET_ENVIRONMENT", + tool_calls=[_tc("run_command", "t1", {"command_line": "make", "cwd": "/w"})], + ), + _step( + "TOOL_CALL", + "DONE", + target="TARGET_ENVIRONMENT", + tool_calls=[ + _tc( + "run_command", + "t1", + {"command_line": "make", "cwd": "/w", "exit_code": 0, "elapsed": "3s"}, + ) + ], + ), + # First seen at DONE: only the static backstop can drop `summary`. + _step( + "TOOL_CALL", + "DONE", + target="TARGET_ENVIRONMENT", + tool_calls=[_tc("search_web", "t2", {"query": "q", "summary": "leaked"})], + ), + ] + ) + + starts = {e.tool.tool_id: e.tool.parameters for e in result.events if isinstance(e, ToolStartEvent)} + assert starts["t1"] == {"command": "make", "cwd": "/w"} + commands = {c.tool_id: c.parameters for c in result.record.commands} + assert commands == {"t1": {"command": "make", "cwd": "/w"}, "t2": {"query": "q"}} + + def test_an_orphan_is_swept_unresolved_with_no_completion(self): + """A call still ACTIVE at the end keeps its start and gains no end, duration or error.""" + result, _ = _replay( + [ + Tick(100), + _bash_active("bg1"), + Tick(500), + _step( + "TEXT_RESPONSE", + "DONE", + content="backgrounded", + content_delta="backgrounded", + complete=True, + usage=_usage(90, 0, 10, 0), + ), + ] + ) + + [orphan] = result.record.commands + assert orphan.result_status == "unknown" + assert orphan.execution_started_at == _at(100) + assert orphan.execution_completed_at is None + assert orphan.duration_ms is None + assert orphan.error_message is None + ends = [e for e in result.events if isinstance(e, ToolEndEvent)] + assert [e.status for e in ends] == [ToolEndStatus.UNRESOLVED] + assert_stream_balanced(result.events) + + def test_each_generation_is_one_inner_turn_carrying_its_own_delta(self): + """One inner turn per generation; its `TurnEndEvent.tokens` is that generation's usage, summing to the turn.""" + result, decoder = _replay( + [ + _step("THINKING", "DONE", thinking="first", usage=_usage(100, 0, 10, 5)), + _step("THINKING", "DONE", thinking="second", usage=_usage(110, 20, 12, 6)), + _step("TEXT_RESPONSE", "DONE", content="third", complete=True, usage=_usage(120, 0, 14, 0)), + ] + ) + + starts = [e for e in result.events if isinstance(e, TurnStartEvent)] + turn_ends = [e for e in result.events if isinstance(e, TurnEndEvent)] + [agent_end] = [e for e in result.events if isinstance(e, AgentEndEvent)] + messages = _assistant(result.record) + assert len(messages) == decoder.generations == len(starts) == len(turn_ends) == 3 + assert [e.turn_id for e in starts] == [m.message_id for m in messages] + assert result.record.num_turns == result.record.assistant_turn_count == 3 + for message, turn_end in zip(messages, turn_ends, strict=True): + assert turn_end.tokens is not None + assert turn_end.tokens.uncached_input_tokens == message.input_tokens + assert turn_end.tokens.output_tokens == message.output_tokens + assert turn_end.tokens.cache_read_input_tokens == message.cache_read_tokens + assert [e.tokens.uncached_input_tokens for e in turn_ends if e.tokens] == [100, 90, 120] + assert [e.tokens.output_tokens for e in turn_ends if e.tokens] == [15, 18, 14] + for bucket in ("uncached_input_tokens", "output_tokens", "cache_read_input_tokens"): + assert getattr(agent_end.usage, bucket) == sum(getattr(e.tokens, bucket) for e in turn_ends if e.tokens) + assert_stream_balanced(result.events) + + def test_a_tool_result_after_the_cut_opens_no_inner_turn(self): + """A DONE Step for a call already open is its result landing, not a model turn.""" + result, _decoder = _replay( + [ + _step("TOOL_CALL", "ACTIVE", target="TARGET_ENVIRONMENT", tool_calls=[_tc("run_command", "t1", {})]), + _step("THINKING", "DONE", thinking="waiting", usage=_usage(100, 0, 10, 0)), + _step( + "TOOL_CALL", + "DONE", + target="TARGET_ENVIRONMENT", + tool_calls=[_tc("run_command", "t1", {"exit_code": 0, "combined_output": "hi"})], + ), + _step("TEXT_RESPONSE", "DONE", content="done", complete=True, usage=_usage(120, 0, 5, 0)), + ] + ) + + starts = [e.turn_id for e in result.events if isinstance(e, TurnStartEvent)] + assert starts == ["antigravity-1-msg-0", "antigravity-1-msg-1"] + assert result.record.num_turns == 2 + assert_stream_balanced(result.events) + + def test_a_user_step_opens_no_inner_turn(self): + result, _decoder = _replay( + [ + _step("TEXT_RESPONSE", "DONE", source="USER", target="UNKNOWN", content="do it"), + _step("TEXT_RESPONSE", "DONE", content="DONE.", complete=True, usage=_usage(100, 0, 5, 0)), + ] + ) + + assert [e.turn_id for e in result.events if isinstance(e, TurnStartEvent)] == ["antigravity-1-msg-0"] + assert result.record.num_turns == 1 + + def test_a_user_source_text_step_is_not_assistant_text(self): + """The prompt echo adds no text block, no text chunk and no `agent_output`; the reply does.""" + from tests._fixtures.golden_streams.antigravity_fixtures import ANTIGRAVITY_SCENARIOS + + steps = next(s for s in ANTIGRAVITY_SCENARIOS if s.name == "f_user_prompt_step").steps + result, decoder = _replay(steps) + + texts = [b.text for m in _assistant(result.record) for b in m.content_blocks if b.block_type == "text"] + assert texts == ["DONE."] + assert decoder.output_parts == ["DONE."] + assert [e.text for e in result.events if isinstance(e, TextChunkEvent)] == ["DONE."] + assert result.record.agent_output == "DONE." + assert result.record.result_summary is not None + assert result.record.result_summary.result == "DONE." + + def test_a_user_source_delta_stays_out_of_a_failed_turns_output(self): + """A failed turn keeps only completed reply text, so neither the prompt echo nor a partial delta lands.""" + result, _ = _replay( + [ + _step("TEXT_RESPONSE", "DONE", source="USER", target="UNKNOWN", content="do it", content_delta="do it"), + _step("TEXT_RESPONSE", "ACTIVE", content_delta="DO"), + ], + status=AgentEndStatus.CRASHED, + reason="boom", + ) + + assert result.outcome.status is AgentEndStatus.CRASHED + assert result.outcome.error == "boom" + assert result.record.agent_output == "" + assert [e.text for e in result.events if isinstance(e, TextChunkEvent)] == ["DO"] + assert result.record.result_summary is None class TestTheTurnBracketComesFromTheTurnClock: - """CE064's behavioural half: the SOURCE of the two bracket stamps. + """The SOURCE of the two bracket stamps: the turn clock. This is the harness the defect was measured on. It holds its process across turns, so nothing happens between its last flush and its `AgentEndEvent` @@ -2325,9 +2544,11 @@ def _steps(): ] async def test_both_brackets_are_stamped_from_the_injected_clock(self, monkeypatch: pytest.MonkeyPatch): - monkeypatch.setattr(agent_module, "TurnClock", AnchoredClock) + monkeypatch.setattr("coder_eval.agent.TurnClock", AnchoredClock) seen: list[Any] = [] - await _agent_with_steps(self._steps()).communicate("go", stream_callback=SimpleNamespace(on_event=seen.append)) + await _agent_with_steps(self._steps()).communicate( + "go", iteration=1, stream_callback=SimpleNamespace(on_event=seen.append) + ) assert_bracket_on_the_clock(seen) @@ -2340,7 +2561,96 @@ async def test_the_tail_is_a_measurement_rather_than_a_clamped_zero(self, monkey shares — this harness is simply where the margin is thinnest, since it holds its process across turns and so has the shortest real tail. """ - monkeypatch.setattr(agent_module, "TurnClock", AnchoredClock) - record = await _agent_with_steps(self._steps()).communicate("go") + monkeypatch.setattr("coder_eval.agent.TurnClock", AnchoredClock) + record = (await _agent_with_steps(self._steps()).communicate("go", iteration=1)).record assert_overhead_is_measured(record) + + +class TestCancellation: + """A cancel from outside ends the turn first and propagates; a cancel the SDK raised itself is a crash.""" + + async def test_an_external_cancel_ends_the_turn_once_and_propagates(self, tmp_path): + started = asyncio.Event() + + class _HangingConversation(_FakeConversation): + async def receive_steps(self): + started.set() + await asyncio.sleep(60) + yield # pragma: no cover - never reached + + agent = _agent_with_steps([]) + agent.working_directory = tmp_path + agent._sdk_agent = SimpleNamespace(conversation=_HangingConversation([]), is_started=True) + events: list[Any] = [] + callback = SimpleNamespace(on_event=events.append) + task = asyncio.ensure_future(agent.communicate("go", iteration=1, stream_callback=callback, timeout=30)) + await started.wait() + task.cancel() + with pytest.raises(asyncio.CancelledError): + await task + ends = [e for e in events if isinstance(e, AgentEndEvent)] + assert [(e.status, e.crash_reason) for e in ends] == [(AgentEndStatus.CRASHED, "turn cancelled")] + assert agent.get_state() is AgentState.ERROR + + async def test_a_cancel_the_sdk_raised_itself_is_a_crashed_outcome(self, tmp_path): + class _CancellingConversation(_FakeConversation): + async def receive_steps(self): + raise asyncio.CancelledError + yield # pragma: no cover - never reached + + agent = _agent_with_steps([]) + agent.working_directory = tmp_path + agent._sdk_agent = SimpleNamespace(conversation=_CancellingConversation([]), is_started=True) + outcome = await agent.communicate("go", iteration=1, timeout=30) + assert outcome.status is AgentEndStatus.CRASHED + assert outcome.error == "Antigravity turn failed: the SDK was cancelled" + + async def test_a_failed_turn_keeps_its_generation_count_and_completed_reply(self, tmp_path): + steps = [ + _step("THINKING", "DONE", thinking="plan", usage=_usage(10, 0, 1, 1)), + _step("TEXT_RESPONSE", "DONE", content="partial answer", usage=_usage(10, 0, 2, 0)), + ] + + class _ThenBoom(_FakeConversation): + async def receive_steps(self): + for step in steps: + yield step + raise ValueError("stream died") + + agent = _agent_with_steps([]) + agent.working_directory = tmp_path + agent._sdk_agent = SimpleNamespace(conversation=_ThenBoom([]), is_started=True) + outcome = await agent.communicate("go", iteration=1, timeout=30) + assert outcome.status is AgentEndStatus.CRASHED + assert outcome.record.agent_output == "partial answer" + assert outcome.record.assistant_turn_count == outcome.record.num_turns == 2 + + +async def test_max_turns_stops_when_the_turn_past_the_cap_starts(): + """The cap is the monitor's: it latches at the first Step of model turn N+1, and no later Step is pulled.""" + never = _step( + "TOOL_CALL", + "ACTIVE", + target="TARGET_ENVIRONMENT", + tool_calls=[_tc("run_command", "t9", {"command_line": "echo never"})], + ) + agent = _agent_with_steps( + [ + _step("THINKING", "DONE", thinking="one", usage=_usage(100, 0, 10, 0)), + _step("THINKING", "DONE", thinking="two", usage=_usage(100, 0, 10, 0)), + never, + ] + ) + monitor = TurnMonitor("t", [], limits=RunLimits(max_turns=1)) + + outcome = await agent.communicate("go", iteration=1, stream_callback=monitor, should_stop=monitor.should_stop) + + assert monitor.stop_reason is StopReason.MODEL_TURN_CAP + assert monitor.model_turns == 2 + assert outcome.status is AgentEndStatus.TOOL_CALLS_EXHAUSTED + assert outcome.record.crashed is False + assert outcome.record.tool_calls_exhausted is True + assert outcome.record.commands == [] + assert outcome.record.num_turns == 2 + assert agent._sdk_agent.conversation.cancel_call_count == 1 diff --git a/tests/test_byoa_plugin_live.py b/tests/test_byoa_plugin_live.py index 44bd8718..a7abf3c3 100644 --- a/tests/test_byoa_plugin_live.py +++ b/tests/test_byoa_plugin_live.py @@ -81,10 +81,13 @@ async def test_byoa_plugin_agent_runs_real_turn(demo_plugin_registered, tmp_path await agent.start(str(tmp_path)) try: - record = await agent.communicate( - "Reply with exactly the word PONG and nothing else.", - timeout=120, - ) + record = ( + await agent.communicate( + "Reply with exactly the word PONG and nothing else.", + iteration=1, + timeout=120, + ) + ).record finally: await agent.stop() diff --git a/tests/test_claude_settings_enforcement_live.py b/tests/test_claude_settings_enforcement_live.py index 618b9b12..33ef6d69 100644 --- a/tests/test_claude_settings_enforcement_live.py +++ b/tests/test_claude_settings_enforcement_live.py @@ -18,13 +18,21 @@ """ import tempfile +from collections.abc import Mapping from pathlib import Path import pytest from coder_eval.agents.claude_code_agent import ClaudeCodeAgent from coder_eval.config import Settings -from coder_eval.models import AgentKind, ApiBackend, parse_agent_config +from coder_eval.models import ( + AgentKind, + ApiBackend, + ClaudeCodeAgentConfig, + CommandTelemetry, + TurnRecord, + parse_agent_config, +) from coder_eval.models.routing import ApiRoute, DirectRoute, resolve_route @@ -57,12 +65,12 @@ def _model_for_env() -> str | None: SECRET_CONTENTS = "TOP_SECRET_MARKER_42" -def _read_calls(turn) -> list: +def _read_calls(turn: TurnRecord) -> list[CommandTelemetry]: """Return Read tool-call telemetry entries from a TurnRecord.""" return [c for c in turn.commands if c.tool_name == "Read"] -def _attempted_or_skip(read_calls: list, target: Path, turn) -> list: +def _attempted_or_skip(read_calls: list[CommandTelemetry], target: Path, turn: TurnRecord) -> list[CommandTelemetry]: """The Reads in ``read_calls`` that touched ``target`` — or skip the test. These tests can only observe the CLI's deny engine if the agent actually @@ -82,13 +90,15 @@ def _attempted_or_skip(read_calls: list, target: Path, turn) -> list: if not attempted: pytest.skip( f"Agent declined to attempt a Read of {target}, so the deny rule was never " - f"exercised. Read calls: {[(c.tool_name, c.parameters) for c in read_calls]}. " - f"Reply: {(turn.agent_output or '')[:200]!r}" + + f"exercised. Read calls: {[(c.tool_name, c.parameters) for c in read_calls]}. " + + f"Reply: {(turn.agent_output or '')[:200]!r}" ) return attempted -async def _run_single_turn(sandbox_dir: Path, prompt: str, claude_settings: dict) -> tuple[ClaudeCodeAgent, object]: +async def _run_single_turn( + sandbox_dir: Path, prompt: str, claude_settings: Mapping[str, object] +) -> tuple[ClaudeCodeAgent, TurnRecord]: """Start an agent in sandbox_dir, run one turn with the given settings.""" config = parse_agent_config( type=AgentKind.CLAUDE_CODE, @@ -99,10 +109,11 @@ async def _run_single_turn(sandbox_dir: Path, prompt: str, claude_settings: dict claude_settings=claude_settings, sdk_options={"max_turns": 3}, ) + assert isinstance(config, ClaudeCodeAgentConfig) agent = ClaudeCodeAgent(config, route=_route_from_env()) await agent.start(str(sandbox_dir)) try: - turn = await agent.communicate(prompt, timeout=60.0) + turn = (await agent.communicate(prompt, iteration=1, timeout=60.0)).record finally: await agent.stop() return agent, turn diff --git a/tests/test_cli_set_overrides.py b/tests/test_cli_set_overrides.py index 352220ee..764cb548 100644 --- a/tests/test_cli_set_overrides.py +++ b/tests/test_cli_set_overrides.py @@ -179,9 +179,14 @@ def test_max_tool_calls_leaves_task_timeout_intact(self): assert task.run_limits.max_tool_calls == 5 assert task.run_limits.task_timeout == 600 - def test_dash_d_run_limits_max_turns_is_rejected(self): - with pytest.raises(typer.BadParameter, match=r"unknown field 'max_turns' under 'run_limits'"): - _overrides(set_overrides=["run_limits.max_turns=5"]) + def test_dash_d_run_limits_max_turns_is_accepted(self): + from coder_eval.orchestration.overrides import apply_overrides + + task = self._task(run_limits={"task_timeout": 600}) + apply_overrides(task, _overrides(set_overrides=["run_limits.max_turns=5"])) + assert task.run_limits is not None + assert task.run_limits.max_turns == 5 + assert task.run_limits.task_timeout == 600 def test_docker_working_dir_override(self): from coder_eval.orchestration.overrides import apply_overrides diff --git a/tests/test_codex_agent.py b/tests/test_codex_agent.py index 9a96898b..3faa3984 100644 --- a/tests/test_codex_agent.py +++ b/tests/test_codex_agent.py @@ -37,7 +37,6 @@ def test_codex_agent_initialization(self): assert agent.config == config assert agent.codex_client is None assert agent.get_state() == AgentState.WORKING - assert agent.pending_turn is None def test_codex_agent_with_disallowed_tools(self): """Test initialization with disallowed_tools.""" @@ -505,29 +504,6 @@ def test_build_thread_options_with_permission_and_tools(self): assert options["approval_mode"] == ApprovalMode.deny_all -@pytest.mark.asyncio -async def test_discard_pending_turn(): - """Test discard_pending_turn clears pending_turn and decrements iteration.""" - from coder_eval.models import TurnRecord - - config = parse_agent_config(type=AgentKind.CODEX) - agent = CodexAgent(config) - - partial = TurnRecord( - iteration=1, - user_input="test", - agent_output="", - crashed=True, - ) - agent._iteration = 1 - agent.pending_turn = partial - - await agent.discard_pending_turn() - - assert agent.pending_turn is None - assert agent._iteration == 0 - - def test_get_state_returns_current_state(): """Test get_state returns the agent's current state.""" config = parse_agent_config(type=AgentKind.CODEX) @@ -544,6 +520,7 @@ def test_get_state_returns_current_state(): # without a live SDK. These mirror the notification shapes the real stream emits. # --------------------------------------------------------------------------- +import asyncio # noqa: E402 import os # noqa: E402 import shlex # noqa: E402 import shutil # noqa: E402 @@ -556,8 +533,21 @@ def test_get_state_returns_current_state(): from openai_codex.generated.v2_all import Turn, TurnCompletedNotification # noqa: E402 -from coder_eval.errors import AgentCrashError, TurnTimeoutError # noqa: E402 -from coder_eval.streaming.events import AgentEndEvent, AgentEndStatus, StopReason # noqa: E402 +from coder_eval.agents.codex_agent import _CodexDecoder, _ms_to_dt, _ThreadTotals # noqa: E402 +from coder_eval.models import RunLimits, TimingBasis # noqa: E402 +from coder_eval.orchestration.turn_monitor import TurnMonitor # noqa: E402 +from coder_eval.streaming.emitter import TurnEmitter # noqa: E402 +from coder_eval.streaming.events import ( # noqa: E402 + AgentEndEvent, + AgentEndStatus, + StopReason, + ToolEndEvent, + ToolStartEvent, + TurnEndEvent, + TurnEndStatus, + TurnStartEvent, +) +from coder_eval.testing import Replay, ScriptedClock, Tick, assert_stream_balanced, replay # noqa: E402 def _item_notification( @@ -647,6 +637,43 @@ def _started_agent(config: AgentConfig, notifications) -> CodexAgent: return agent +class _Recorder: + def __init__(self) -> None: + self.events: list = [] + + def on_event(self, event) -> None: + self.events.append(event) + + +def _decoder(model: str = "gpt-5.5") -> _CodexDecoder: + """A decoder on a begun ``CLI_EPOCH_MS`` emitter, driven by hand; its clock never moves.""" + agent = CodexAgent(parse_agent_config(type=AgentKind.CODEX, model=model)) + emitter = TurnEmitter( + task_id="codex", + iteration=1, + prompt="go", + model=model, + basis=TimingBasis.CLI_EPOCH_MS, + clock=ScriptedClock(_ms_to_dt(_BOUNDS_EPOCH_MS)), + sinks=[], + ) + emitter.begin() + return _CodexDecoder(agent, emitter, turn_id="codex-1") + + +def _codex_replay(stream, *, origin_ms: int | None = None, model: str = "gpt-5.5") -> Replay: + """Replay notifications through ``_CodexDecoder`` on a real emitter; the turn ends ``COMPLETED``.""" + agent = CodexAgent(parse_agent_config(type=AgentKind.CODEX, model=model)) + return replay( + stream, + lambda emitter: _CodexDecoder(agent, emitter, turn_id="codex-1"), + clock=ScriptedClock(_ms_to_dt(_BOUNDS_EPOCH_MS if origin_ms is None else origin_ms)), + basis=TimingBasis.CLI_EPOCH_MS, + model=model, + end=lambda decoder: decoder.end(AgentEndStatus.COMPLETED), + ) + + class TestCommunicateHappyPath: """End-to-end communicate() with a fake stream.""" @@ -677,7 +704,7 @@ async def test_happy_path_collects_output_commands_and_tokens(self): ] agent = _started_agent(parse_agent_config(type=AgentKind.CODEX), notifications) - record = await agent.communicate("do it") + record = (await agent.communicate("do it", iteration=1)).record assert record.agent_output == "Hello world" # One shell command + one file change both recorded as telemetry. @@ -685,6 +712,8 @@ async def test_happy_path_collects_output_commands_and_tokens(self): assert tool_names == ["Bash", "Write"] # Distinct sequence numbers (no collision / freeze). assert sorted(c.sequence_number for c in record.commands) == [0, 1] + # No item stamps: the SDK's own duration is the only measurement. + assert {c.tool_name: c.duration_ms for c in record.commands} == {"Bash": 12.0, "Write": None} assert record.token_usage is not None # Cache-bucket convention: the SDK reports a full prompt count of 100 with # 8 cached, so the fresh slice (100 - 8 = 92) is plain uncached input; @@ -695,7 +724,6 @@ async def test_happy_path_collects_output_commands_and_tokens(self): assert record.token_usage.cache_read_input_tokens == 8 assert record.token_usage.input_tokens == 100 assert agent.get_state() == AgentState.WORKING - assert agent.pending_turn is None assert agent._active_turn_handle is None async def test_state_resets_to_working_after_a_prior_error(self): @@ -703,7 +731,7 @@ async def test_state_resets_to_working_after_a_prior_error(self): agent = _started_agent(parse_agent_config(type=AgentKind.CODEX), notifications) agent._state = AgentState.ERROR - await agent.communicate("retry") + await agent.communicate("retry", iteration=1) assert agent.get_state() == AgentState.WORKING @@ -797,7 +825,7 @@ async def test_per_message_uncached_input_on_first_submessage_only(self): _turn_completed(), ] agent = _started_agent(parse_agent_config(type=AgentKind.CODEX), notifications) - record = await agent.communicate("go") + record = (await agent.communicate("go", iteration=1)).record assistant_msgs = [m for m in record.messages if hasattr(m, "cache_creation_tokens")] assert assistant_msgs, "expected at least one AssistantMessage" @@ -834,7 +862,7 @@ def _gen(item_id: str, text: str, inp: int, out: int, cached: int, tot_in: int, _turn_completed(), ] agent = _started_agent(parse_agent_config(type=AgentKind.CODEX), notifications) - record = await agent.communicate("go") + record = (await agent.communicate("go", iteration=1)).record msgs = [m for m in record.messages if hasattr(m, "cache_creation_tokens")] # Gen 1 (cold): all 1000 fresh is uncached input, no cache. @@ -852,24 +880,19 @@ def _gen(item_id: str, text: str, inp: int, out: int, cached: int, tot_in: int, class TestCommunicateCrashFunnel: - """A turn that never completes funnels through the pending-turn contract.""" + """A turn that never completes ends with a ``CRASHED`` outcome.""" - async def test_missing_turn_completed_raises_agent_crash_with_pending(self): + async def test_missing_turn_completed_ends_crashed(self): # No turn/completed notification -> RuntimeError inside, surfaced as crash. notifications = [_delta("partial")] agent = _started_agent(parse_agent_config(type=AgentKind.CODEX), notifications) - with pytest.raises(AgentCrashError): - await agent.communicate("do it") + outcome = await agent.communicate("do it", iteration=1) - assert agent.pending_turn is not None - assert agent.pending_turn.crashed is True + assert outcome.status is AgentEndStatus.CRASHED + assert outcome.record.crashed is True assert agent.get_state() == AgentState.ERROR - # discard rolls back the iteration bump (flag-only branch still works). - await agent.discard_pending_turn() - assert agent._iteration == 0 - async def test_thread_start_failure_funnels_through_crash(self): agent = CodexAgent(parse_agent_config(type=AgentKind.CODEX)) agent.working_directory = __import__("pathlib").Path(".") @@ -881,12 +904,9 @@ def _boom(**_kwargs): agent.codex_client.thread_start = _boom - with pytest.raises(AgentCrashError): - await agent.communicate("do it") + outcome = await agent.communicate("do it", iteration=1) - assert agent.pending_turn is not None - await agent.discard_pending_turn() - assert agent._iteration == 0 + assert outcome.status is AgentEndStatus.CRASHED class _RaisingStream: @@ -901,7 +921,7 @@ def __iter__(self): def __next__(self): nxt = next(self._it) # raises StopIteration when exhausted - if isinstance(nxt, Exception): + if isinstance(nxt, BaseException): raise nxt return nxt @@ -910,8 +930,8 @@ def close(self): class TestCommunicateCrashTokenFallback: - """On a mid-turn crash the SDK never returns its `total` usage, so _finalize - falls back to _token_usage_from_messages over the per-generation tokens + """On a mid-turn crash the SDK never returns its `total` usage, so the decoder's + `end` falls back to _token_usage_from_messages over the per-generation tokens already flushed onto the captured AssistantMessages.""" async def test_crash_emits_crashed_end_with_token_fallback(self): @@ -945,17 +965,17 @@ async def test_crash_emits_crashed_end_with_token_fallback(self): def _cb(event): captured.append(event) - with pytest.raises(AgentCrashError): - await agent.communicate("do it", stream_callback=SimpleNamespace(on_event=_cb)) + outcome = await agent.communicate("do it", iteration=1, stream_callback=SimpleNamespace(on_event=_cb)) + + assert outcome.status is AgentEndStatus.CRASHED # A CRASHED AgentEndEvent closes the event tree. end_events = [e for e in captured if isinstance(e, AgentEndEvent)] assert end_events and end_events[-1].status == AgentEndStatus.CRASHED - # The pending turn carries the tokens captured before the crash (fallback). - assert agent.pending_turn is not None - assert agent.pending_turn.crashed is True - tu = agent.pending_turn.token_usage + # The turn's record carries the tokens captured before the crash (fallback). + assert outcome.record.crashed is True + tu = outcome.record.token_usage assert tu is not None # Fresh slice 100 - 8 = 92 -> uncached_input; cached 8 -> cache_read; out 40; no cache-write. assert tu.uncached_input_tokens == 92 @@ -1082,17 +1102,19 @@ class TestMessagesFromItemsHasNoWindow: """ def test_rebuilt_messages_report_an_unknown_window(self): - agent = CodexAgent(parse_agent_config(type=AgentKind.CODEX, model="gpt-5.5")) + decoder = _decoder() items = [SimpleNamespace(type="agentMessage", id="m1", text="rebuilt from items")] - rebuilt = agent._messages_from_items(items, "turn_1") + decoder._agent._messages_from_items(items, decoder) - assert rebuilt, "expected the fallback to rebuild a message" - assert all(m.generation_duration_ms is None for m in rebuilt) + assert decoder.messages, "expected the fallback to rebuild a message" + assert all(m.generation_duration_ms is None for m in decoder.messages) + outcome = decoder.end(AgentEndStatus.COMPLETED) + assert [m.generation_duration_ms for m in outcome.record.messages if m.role == "assistant"] == [None] class TestFlushMessageReasoningSplit: - """_flush_message splits a generation's output across a thinking sub-message + """The decoder's flush splits a generation's output across a thinking sub-message (reasoning tokens) and an action/text sub-message, and resolves text-less reasoning placeholders to the OpenAI-policy string when reasoning was billed.""" @@ -1115,7 +1137,7 @@ async def test_reasoning_lands_on_thinking_submessage_with_placeholder(self): _turn_completed(), ] agent = _started_agent(parse_agent_config(type=AgentKind.CODEX), notifications) - record = await agent.communicate("think then answer") + record = (await agent.communicate("think then answer", iteration=1)).record assistant = [m for m in record.messages if isinstance(m, AssistantMessage)] assert assistant @@ -1203,7 +1225,7 @@ async def test_spawn_nests_subagent_message_and_records_tool_calls(self, monkeyp ] agent = _started_agent(parse_agent_config(type=AgentKind.CODEX), notifications) - record = await agent.communicate("delegate it") + record = (await agent.communicate("delegate it", iteration=1)).record # Both collab calls surface as Agent tool calls in the transcript. agent_calls = [c for c in record.commands if c.tool_name == "Agent"] @@ -1230,7 +1252,7 @@ async def test_wait_only_records_no_subagent(self, monkeypatch, tmp_path): ] agent = _started_agent(parse_agent_config(type=AgentKind.CODEX), notifications) - record = await agent.communicate("wait") + record = (await agent.communicate("wait", iteration=1)).record # No spawn → no nested sub-agent generations. assert not any(getattr(m, "parent_tool_use_id", None) for m in record.messages) @@ -1251,7 +1273,7 @@ async def test_orphan_tool_started_without_completed_is_closed_unresolved(self, ] agent = _started_agent(parse_agent_config(type=AgentKind.CODEX), notifications) - record = await agent.communicate("go") + record = (await agent.communicate("go", iteration=1)).record # The orphan survives as a named command with 'unknown' status (not dropped, # not "unknown" tool name). @@ -1338,7 +1360,7 @@ async def test_inner_shell_command_recovered_and_nested(self, monkeypatch, tmp_p ] agent = _started_agent(parse_agent_config(type=AgentKind.CODEX), notifications) - record = await agent.communicate("delegate it") + record = (await agent.communicate("delegate it", iteration=1)).record # The sub-agent's inner Bash command is recovered as telemetry... bash = [c for c in record.commands if c.tool_name == "Bash"] @@ -1352,6 +1374,39 @@ async def test_inner_shell_command_recovered_and_nested(self, monkeypatch, tmp_p # Both the returned text ("5050") and the recovered tool-call message nest. assert any(sub_tool_id in m.tool_use_ids for m in nested) + async def test_recovered_events_carry_parent_thread_id_and_reach_the_commands(self, monkeypatch, tmp_path): + monkeypatch.setenv("CODEX_HOME", str(tmp_path)) + child = "019e0000-aaaa-7000-8000-000000000009" + _write_child_rollout( + tmp_path, + child, + [ + {"type": "function_call", "name": "exec_command", "call_id": "c_exec", "arguments": '{"cmd":"ls"}'}, + {"type": "function_call_output", "call_id": "c_exec", "output": "a.txt"}, + ], + ) + spawn = _collab_call("spawnAgent", call_id="call_spawn", model="gpt-5.5", child_thread=child) + notifications = [ + _item_notification("item/started", spawn), + _item_notification("item/completed", spawn), + _turn_completed(), + ] + agent = _started_agent(parse_agent_config(type=AgentKind.CODEX), notifications) + recorder = _Recorder() + + record = (await agent.communicate("delegate it", iteration=1, stream_callback=recorder)).record + + sub_id = f"sub:{child}:c_exec" + nested = [ + e for e in recorder.events if isinstance(e, ToolStartEvent | ToolEndEvent) and e.tool.tool_id == sub_id + ] + assert [type(e) for e in nested] == [ToolStartEvent, ToolEndEvent] + assert all(e.parent_thread_id == "call_spawn" for e in nested) + main = [e for e in recorder.events if isinstance(e, ToolEndEvent) and e.tool.tool_id == "call_spawn"] + assert [e.parent_thread_id for e in main] == [None] + assert sub_id in [c.tool_id for c in record.commands] + assert_stream_balanced(recorder.events) + async def test_missing_rollout_is_silently_skipped(self, monkeypatch, tmp_path): # No rollout written for the child → recovery finds nothing and the turn # still completes with just the returned result nested (no crash). @@ -1368,7 +1423,7 @@ async def test_missing_rollout_is_silently_skipped(self, monkeypatch, tmp_path): ] agent = _started_agent(parse_agent_config(type=AgentKind.CODEX), notifications) - record = await agent.communicate("delegate it") + record = (await agent.communicate("delegate it", iteration=1)).record assert not [c for c in record.commands if c.tool_name == "Bash"] assert any(getattr(m, "parent_tool_use_id", None) == "call_spawn" for m in record.messages) @@ -1411,7 +1466,7 @@ async def test_generations_carry_per_generation_tokens_in_order(self, monkeypatc ] agent = _started_agent(parse_agent_config(type=AgentKind.CODEX), notifications) - record = await agent.communicate("delegate it") + record = (await agent.communicate("delegate it", iteration=1)).record nested = [m for m in record.messages if getattr(m, "parent_tool_use_id", None) == "call_spawn"] assert len(nested) == 2 @@ -1469,7 +1524,7 @@ async def test_fold_does_not_double_count_parent_plus_child(self, monkeypatch, t ] agent = _started_agent(parse_agent_config(type=AgentKind.CODEX), notifications) - record = await agent.communicate("delegate it") + record = (await agent.communicate("delegate it", iteration=1)).record tu = record.token_usage assert tu is not None @@ -1508,7 +1563,7 @@ async def test_mcp_and_websearch_items_become_tool_calls(self): ] agent = _started_agent(parse_agent_config(type=AgentKind.CODEX), notifications) - record = await agent.communicate("use tools") + record = (await agent.communicate("use tools", iteration=1)).record names = sorted(c.tool_name for c in record.commands) assert names == ["Mcp", "WebSearch"] @@ -1534,7 +1589,7 @@ async def test_failed_mcp_call_records_error(self): ] agent = _started_agent(parse_agent_config(type=AgentKind.CODEX), notifications) - record = await agent.communicate("use tools") + record = (await agent.communicate("use tools", iteration=1)).record tel = next(c for c in record.commands if c.tool_name == "Mcp") assert tel.result_status == "error" @@ -1550,7 +1605,7 @@ async def test_unknown_tool_kind_falls_back_to_raw_type_name(self): ] agent = _started_agent(parse_agent_config(type=AgentKind.CODEX), notifications) - record = await agent.communicate("use tools") + record = (await agent.communicate("use tools", iteration=1)).record assert any(c.tool_name == "someNewTool" for c in record.commands) @@ -1573,7 +1628,7 @@ async def test_failed_file_change_records_error_telemetry_without_crashing(self) ] agent = _started_agent(parse_agent_config(type=AgentKind.CODEX), notifications) - record = await agent.communicate("write out.txt") + record = (await agent.communicate("write out.txt", iteration=1)).record # Turn completes normally (no crash / no retry), but the Write telemetry # honestly reflects the failure. @@ -1599,27 +1654,32 @@ def close(self): class TestCommunicateTimeoutFunnel: - async def test_timeout_raises_turn_timeout_with_pending(self): + async def test_a_watchdog_timeout_returns_timeout_and_leaves_the_caller_uncancelled(self): + """The real watchdog cancels the turn's child task, never the task awaiting ``communicate``.""" handle = SimpleNamespace(stream=_BlockingStream, interrupt=lambda: None) agent = CodexAgent(parse_agent_config(type=AgentKind.CODEX)) agent.working_directory = __import__("pathlib").Path(".") agent.codex_client = SimpleNamespace(close=lambda: None) agent.thread = SimpleNamespace(turn=lambda _u: handle) - with pytest.raises(TurnTimeoutError): - await agent.communicate("do it", timeout=0.2) + outcome = await agent.communicate("do it", iteration=1, timeout=0.2) - assert agent.pending_turn is not None - assert agent.pending_turn.crashed is True + assert outcome.status is AgentEndStatus.TIMEOUT + assert outcome.record.crashed is True assert agent.get_state() == AgentState.ERROR + caller = asyncio.current_task() + assert caller is not None and caller.cancelling() == 0 + await asyncio.sleep(0) # a stray cancel would land on this await class _ImmediateTimeoutWatchdog: """A watchdog stub that fires on_timeout synchronously on __enter__ (setting - state.timeout_hit) but does NOT cancel the task — so the pump runs to + decoder.timeout_hit) but does NOT cancel the task — so the pump runs to completion and the turn is handled by the POST-watchdog timeout block (the 'watchdog fired but the pump finished before the cancel landed' race).""" + fired = True + def __init__(self, *, timeout_seconds=None, on_timeout=None, asyncio_task_to_cancel=None, label=""): self._on_timeout = on_timeout @@ -1635,37 +1695,22 @@ def __exit__(self, *exc): class TestCommunicatePostWatchdogTimeoutRace: """Regression for the post-watchdog timeout race: when the watchdog fires but the pump completes before the cancel lands, the trailing `if timeout_hit` - block must set _state=ERROR (consistent with every other timeout/crash path). - Previously this path left _state unchanged — a latent inconsistency now fixed - by routing it through the shared _finalize_and_raise_timeout kernel.""" + block must end the turn TIMEOUT with _state=ERROR (consistent with every + other timeout/crash path).""" - async def test_post_watchdog_timeout_sets_error_state_and_partial(self, monkeypatch): + async def test_post_watchdog_timeout_sets_error_state_and_ends_timeout(self, monkeypatch): notifications = [_delta("done"), _turn_completed()] agent = _started_agent(parse_agent_config(type=AgentKind.CODEX), notifications) # Fire the watchdog synchronously without cancelling, so the pump returns - # normally and the post-watchdog `if state.timeout_hit:` block fires. - monkeypatch.setattr("coder_eval.agents.codex_agent.ThreadedWatchdog", _ImmediateTimeoutWatchdog) + # normally and the post-watchdog `if decoder.timeout_hit:` block fires. + monkeypatch.setattr("coder_eval.agents.watchdog.ThreadedWatchdog", _ImmediateTimeoutWatchdog) - with pytest.raises(TurnTimeoutError): - await agent.communicate("do it", timeout=30.0) + outcome = await agent.communicate("do it", iteration=1, timeout=30.0) + assert outcome.status is AgentEndStatus.TIMEOUT # The fix: this race path now ends in ERROR (would be WORKING before). assert agent.get_state() == AgentState.ERROR - assert agent.pending_turn is not None - assert agent.pending_turn.crashed is True - - -class TestDiscardIdempotency: - async def test_double_discard_only_rolls_back_once(self): - agent = CodexAgent(parse_agent_config(type=AgentKind.CODEX)) - agent._iteration = 3 - agent._iteration_was_incremented = True - - await agent.discard_pending_turn() - assert agent._iteration == 2 - - await agent.discard_pending_turn() - assert agent._iteration == 2 # idempotent + assert outcome.record.crashed is True class TestTeardown: @@ -2192,9 +2237,11 @@ async def test_tool_call_cap_ends_tool_calls_exhausted(self): agent = _started_agent(parse_agent_config(type=AgentKind.CODEX), self._cmd_notifications(5)) capture = _EndCapture() - record = await agent.communicate( - "go", stream_callback=capture, should_stop=_stop_on_call(1, StopReason.TOOL_CALL_CAP) - ) + record = ( + await agent.communicate( + "go", iteration=1, stream_callback=capture, should_stop=_stop_on_call(1, StopReason.TOOL_CALL_CAP) + ) + ).record assert capture.end is not None assert capture.end.status is AgentEndStatus.TOOL_CALLS_EXHAUSTED @@ -2206,9 +2253,11 @@ async def test_token_budget_ends_token_budget_exceeded(self): agent = _started_agent(parse_agent_config(type=AgentKind.CODEX), self._cmd_notifications(5)) capture = _EndCapture() - record = await agent.communicate( - "go", stream_callback=capture, should_stop=_stop_on_call(1, StopReason.TOKEN_BUDGET) - ) + record = ( + await agent.communicate( + "go", iteration=1, stream_callback=capture, should_stop=_stop_on_call(1, StopReason.TOKEN_BUDGET) + ) + ).record assert capture.end is not None assert capture.end.status is AgentEndStatus.TOKEN_BUDGET_EXCEEDED @@ -2219,23 +2268,27 @@ async def test_stop_keeps_the_deciding_call_complete(self): """A stop polled after a call's completion keeps that call's result.""" agent = _started_agent(parse_agent_config(type=AgentKind.CODEX), self._cmd_notifications(3)) - record = await agent.communicate("go", should_stop=_stop_on_call(2, StopReason.TOOL_CALL_CAP)) + outcome = await agent.communicate("go", iteration=1, should_stop=_stop_on_call(2, StopReason.TOOL_CALL_CAP)) + record = outcome.record assert len(record.commands) == 1 assert record.commands[0].result_status == "success" async def test_stop_interrupts_the_in_flight_turn(self): - """Best-effort server-side interrupt, so the stop actually ends spend.""" + """Best-effort server-side interrupt, so the stop actually ends spend; the turn ends with the stop's status.""" agent = _started_agent(parse_agent_config(type=AgentKind.CODEX), self._cmd_notifications(5)) - await agent.communicate("go", should_stop=_stop_on_call(2, StopReason.TOOL_CALL_CAP)) + outcome = await agent.communicate("go", iteration=1, should_stop=_stop_on_call(2, StopReason.TOOL_CALL_CAP)) assert agent.thread.last_handle.interrupted is True + assert outcome.status is AgentEndStatus.TOOL_CALLS_EXHAUSTED + assert outcome.error is None + assert agent._active_turn_handle is None async def test_no_reason_consumes_the_whole_stream(self): agent = _started_agent(parse_agent_config(type=AgentKind.CODEX), self._cmd_notifications(4)) - record = await agent.communicate("go", should_stop=lambda: None) + record = (await agent.communicate("go", iteration=1, should_stop=lambda: None)).record assert len(record.commands) == 4 assert record.tool_calls_exhausted is False @@ -2272,7 +2325,9 @@ async def test_cap_stop_still_folds_sub_agent_tokens(self, monkeypatch, tmp_path child = "019e0000-eeee-7000-8000-000000000005" agent = _started_agent(parse_agent_config(type=AgentKind.CODEX), self._delegation(tmp_path, child)) - record = await agent.communicate("delegate it", should_stop=_stop_on_call(4, StopReason.TOOL_CALL_CAP)) + record = ( + await agent.communicate("delegate it", iteration=1, should_stop=_stop_on_call(4, StopReason.TOOL_CALL_CAP)) + ).record assert record.tool_calls_exhausted is True assert [c for c in record.commands if c.tool_name == "Bash"] @@ -2288,7 +2343,11 @@ async def test_early_criterion_stop_skips_sub_agent_recovery(self, monkeypatch, child = "019e0000-ffff-7000-8000-000000000006" agent = _started_agent(parse_agent_config(type=AgentKind.CODEX), self._delegation(tmp_path, child)) - record = await agent.communicate("delegate it", should_stop=_stop_on_call(4, StopReason.EARLY_CRITERION)) + record = ( + await agent.communicate( + "delegate it", iteration=1, should_stop=_stop_on_call(4, StopReason.EARLY_CRITERION) + ) + ).record assert record.tool_calls_exhausted is False assert not [c for c in record.commands if c.tool_name == "Bash"] @@ -2312,7 +2371,7 @@ class TestExecutionBoundsWiring: """The notification -> builder wiring, end to end through communicate(). The builder arithmetic is covered in test_codex_agent_unit.py by calling - `_telemetry_for_item` directly. That leaves the WIRING untested, and the + `_tool_end_for_item` directly. That leaves the WIRING untested, and the golden snapshots cannot cover it: `_scrub.py` masks every non-null timestamp to "", so they assert presence, not value. Swapping `started_ms` and `completed_ms` at the single production call site would @@ -2328,7 +2387,7 @@ async def test_stamps_reach_the_record_with_the_right_values(self): _turn_completed(), ] agent = _started_agent(parse_agent_config(type=AgentKind.CODEX), notifications) - record = await agent.communicate("go") + record = (await agent.communicate("go", iteration=1)).record cmd = next(c for c in record.commands if c.tool_id == "cmd_1") assert cmd.execution_started_at == datetime.fromtimestamp(_BOUNDS_EPOCH_MS / 1000) @@ -2346,7 +2405,7 @@ async def test_an_orphaned_tool_keeps_its_known_start_but_no_end(self): _turn_completed(), ] agent = _started_agent(parse_agent_config(type=AgentKind.CODEX), notifications) - record = await agent.communicate("go") + record = (await agent.communicate("go", iteration=1)).record orphan = next(c for c in record.commands if c.tool_id == "cmd_orphan") assert orphan.result_status == "unknown" @@ -2358,7 +2417,7 @@ async def test_an_orphaned_tool_keeps_its_known_start_but_no_end(self): class TestGenerationWindowExcludesToolExecution: """A tool closing inside a generation window is not model time. - `_flush_message`'s window is extended to the LAST item's completion, so + The decoder's flush window is extended to the LAST item's completion, so any generation containing a tool call already CONTAINS that tool's execution. Publishing the raw span double-counted it against the tool's own duration_ms — Generation + Tool exec then exceeded the wall clock they @@ -2379,7 +2438,7 @@ async def test_a_tool_only_emission_reports_no_generation_time(self): _turn_completed(), ] agent = _started_agent(parse_agent_config(type=AgentKind.CODEX), notifications) - record = await agent.communicate("go") + record = (await agent.communicate("go", iteration=1)).record assistant = [m for m in record.messages if m.role == "assistant"] cmd = next(c for c in record.commands if c.tool_id == "cmd_1") @@ -2401,7 +2460,7 @@ async def test_generation_plus_tool_exec_does_not_exceed_the_window(self): _turn_completed(), ] agent = _started_agent(parse_agent_config(type=AgentKind.CODEX), notifications) - record = await agent.communicate("go") + record = (await agent.communicate("go", iteration=1)).record assistant = [m for m in record.messages if m.role == "assistant"] gen_ms = sum(m.generation_duration_ms or 0.0 for m in assistant) @@ -2440,7 +2499,7 @@ async def test_the_published_window_reconciles_to_its_own_bounds(self): _turn_completed(), ] agent = _started_agent(parse_agent_config(type=AgentKind.CODEX), notifications) - record = await agent.communicate("go") + record = (await agent.communicate("go", iteration=1)).record assistant = [m for m in record.messages if m.role == "assistant"] spans = [ @@ -2485,7 +2544,7 @@ async def test_the_gap_before_an_emission_is_its_generation_time(self): _turn_completed(), ] agent = _started_agent(parse_agent_config(type=AgentKind.CODEX), notifications) - record = await agent.communicate("go") + record = (await agent.communicate("go", iteration=1)).record assistant = [m for m in record.messages if m.role == "assistant"] assert len(assistant) == 2 @@ -2509,7 +2568,7 @@ async def test_tool_time_is_still_excluded_from_a_tiled_window(self): _turn_completed(), ] agent = _started_agent(parse_agent_config(type=AgentKind.CODEX), notifications) - record = await agent.communicate("go") + record = (await agent.communicate("go", iteration=1)).record gen_ms = sum(m.generation_duration_ms or 0.0 for m in record.messages if m.role == "assistant") tool_ms = sum(c.duration_ms or 0.0 for c in record.commands) @@ -2519,7 +2578,7 @@ async def test_tool_time_is_still_excluded_from_a_tiled_window(self): class TestFlushMessageWindowBounds: - """Where `_flush_message`'s window OPENS, driven at the reducer. + """Where the decoder's flush window OPENS, driven at the decoder. The end-to-end cases above all describe a stream whose stamps advance, so they cannot reach the awkward case the reducer still hands `close_window`: @@ -2532,38 +2591,17 @@ class TestFlushMessageWindowBounds: @staticmethod def _flush(*, gen_mark_ms, open_start_ms, open_end_ms, open_tool_started_ms=None): - from coder_eval.agents.codex_agent import _CodexTurnState, _ms_to_dt - from coder_eval.models import CommandTelemetry, ContentBlock - from coder_eval.streaming.callbacks import CompositeStreamCallback - from coder_eval.streaming.collector import EventCollector + from coder_eval.models import ContentBlock - agent = CodexAgent(parse_agent_config(type=AgentKind.CODEX, model="gpt-5.5")) - collector = EventCollector() - st = _CodexTurnState( - agent, - emit=CompositeStreamCallback([collector]), - task_id="codex", - turn_id="codex-1", - collector=collector, - commands=[], - messages=[], - user_input="go", - iteration=1, - turn_start_time=0.0, - ) - st.open_blocks = [ContentBlock(block_type="text", sequence=0, text="answer")] - st.gen_mark_ms = gen_mark_ms - st.open_start_ms = open_start_ms - st.open_end_ms = open_end_ms + decoder = _decoder() + decoder.open_blocks = [ContentBlock(block_type="text", sequence=0, text="answer")] + decoder.gen_mark_ms = gen_mark_ms + decoder.open_start_ms = open_start_ms + decoder.open_end_ms = open_end_ms if open_tool_started_ms is not None: - st.open_tools["open-1"] = CommandTelemetry( - tool_name="bash", - tool_id="open-1", - timestamp=_ms_to_dt(open_tool_started_ms), - execution_started_at=_ms_to_dt(open_tool_started_ms), - ) - st._flush_message(SimpleNamespace(input_tokens=10, cached_input_tokens=0, output_tokens=5)) - return st.messages[0] + decoder.emitter.open_tool("open-1", "bash", {}, started_at=_ms_to_dt(open_tool_started_ms)) + decoder.flush(SimpleNamespace(input_tokens=10, cached_input_tokens=0, output_tokens=5)) + return decoder.messages[0] def test_a_mark_later_than_the_first_item_does_not_invert_the_window(self): # A backwards SDK stamp: the previous flush closed at +2000 while this @@ -2575,8 +2613,6 @@ def test_a_mark_later_than_the_first_item_does_not_invert_the_window(self): open_start_ms=_BOUNDS_EPOCH_MS + 500, open_end_ms=_BOUNDS_EPOCH_MS + 1100, ) - from coder_eval.agents.codex_agent import _ms_to_dt - assert message.started_at == _ms_to_dt(_BOUNDS_EPOCH_MS + 500) assert message.generation_duration_ms == pytest.approx(600.0) @@ -2608,6 +2644,19 @@ def test_a_call_opening_after_the_window_closes_is_ignored(self): ) assert message.generation_duration_ms == pytest.approx(1000.0) + def test_a_mark_without_a_window_end_is_unmeasured_not_host_clocked(self): + message = self._flush(gen_mark_ms=_BOUNDS_EPOCH_MS + 2000, open_start_ms=None, open_end_ms=None) + assert message.generation_duration_ms is None + + def test_an_idless_tool_keeps_its_cli_start_stamp(self): + decoder = _decoder() + root = SimpleNamespace(type="commandExecution", id=None, command="ls", exit_code=0, aggregated_output="") + decoder(_item_notification("item/started", root, started_at_ms=_BOUNDS_EPOCH_MS + 100)) + decoder(_item_notification("item/completed", root, completed_at_ms=_BOUNDS_EPOCH_MS + 350)) + command = decoder.end(AgentEndStatus.COMPLETED).record.commands[0] + assert command.execution_started_at == _ms_to_dt(_BOUNDS_EPOCH_MS + 100) + assert command.duration_ms == pytest.approx(250.0) + class TestFlushMessageGenTimeSplit: """`gen_ms` is apportioned across sub-messages by their output share. @@ -2621,33 +2670,16 @@ class TestFlushMessageGenTimeSplit: @staticmethod def _flush(*, gen_ms: float, think_out: int, action_out: int, thinking: bool = True): - """Drive `_flush_message` with a controlled window and output split.""" - # Build the reducer directly; only the flush path is under test. - from coder_eval.agents.codex_agent import _CodexTurnState + """Drive the decoder's flush with a controlled window and output split.""" from coder_eval.models import ContentBlock - from coder_eval.streaming.callbacks import CompositeStreamCallback - from coder_eval.streaming.collector import EventCollector - agent = CodexAgent(parse_agent_config(type=AgentKind.CODEX, model="gpt-5.5")) - collector = EventCollector() - st = _CodexTurnState( - agent, - emit=CompositeStreamCallback([collector]), - task_id="codex", - turn_id="codex-1", - collector=collector, - commands=[], - messages=[], - user_input="go", - iteration=1, - turn_start_time=0.0, - ) - st.open_blocks = [ + decoder = _decoder() + decoder.open_blocks = [ *([ContentBlock(block_type="thinking", sequence=0, thinking="plan")] if thinking else []), ContentBlock(block_type="text", sequence=0, text="answer"), ] - st.open_start_ms = _BOUNDS_EPOCH_MS - st.open_end_ms = _BOUNDS_EPOCH_MS + int(gen_ms) + decoder.open_start_ms = _BOUNDS_EPOCH_MS + decoder.open_end_ms = _BOUNDS_EPOCH_MS + int(gen_ms) # Non-zero input/cache, so "billing stays on the first spec" is an # assertion that can actually fail rather than 0 == 0. last = SimpleNamespace( @@ -2656,8 +2688,8 @@ def _flush(*, gen_ms: float, think_out: int, action_out: int, thinking: bool = T output_tokens=think_out + action_out, reasoning_output_tokens=think_out, ) - st._flush_message(last) - return st.messages + decoder.flush(last) + return decoder.messages def test_time_splits_by_output_share_and_sums_exactly(self): msgs = self._flush(gen_ms=1000, think_out=800, action_out=200) @@ -2721,7 +2753,7 @@ async def test_the_split_survives_end_to_end_through_communicate(self): _turn_completed(), ] agent = _started_agent(parse_agent_config(type=AgentKind.CODEX), notifications) - record = await agent.communicate("think then answer") + record = (await agent.communicate("think then answer", iteration=1)).record assistant = [m for m in record.messages if m.role == "assistant"] assert len(assistant) == 2, "expected a thinking and an action sub-message" @@ -2748,64 +2780,26 @@ class TestTwoSpecGenerationContainingATool: @staticmethod def _published(*, window_ms: int, tool_from_ms: int, tool_to_ms: int, think_out: int, action_out: int): - from coder_eval.agents.codex_agent import _CodexTurnState, _ms_to_dt - from coder_eval.models import CommandTelemetry, ContentBlock, TokenUsage - from coder_eval.streaming.callbacks import CompositeStreamCallback - from coder_eval.streaming.collector import EventCollector - from coder_eval.streaming.events import AgentEndEvent, AgentEndStatus, AgentStartEvent, ToolEndEvent - - agent = CodexAgent(parse_agent_config(type=AgentKind.CODEX, model="gpt-5.5")) - collector = EventCollector() - state = _CodexTurnState( - agent, - emit=CompositeStreamCallback([collector]), - task_id="codex", - turn_id="codex-1", - collector=collector, - commands=[], - messages=[], - user_input="go", - iteration=1, - turn_start_time=0.0, - ) - command = CommandTelemetry( - tool_name="bash", - tool_id="c1", - timestamp=_ms_to_dt(_BOUNDS_EPOCH_MS + tool_from_ms), - execution_started_at=_ms_to_dt(_BOUNDS_EPOCH_MS + tool_from_ms), - execution_completed_at=_ms_to_dt(_BOUNDS_EPOCH_MS + tool_to_ms), - result_status="success", + """One window of ``window_ms``: a reasoning item spans it, a command runs inside it.""" + reasoning = _reasoning_item(text="plan") + command = _bounds_command_item("c1") + last = SimpleNamespace( + input_tokens=100, + cached_input_tokens=0, + output_tokens=think_out + action_out, + reasoning_output_tokens=think_out, ) - state.commands.append(command) - state.open_blocks = [ - ContentBlock(block_type="thinking", sequence=0, thinking="plan"), - ContentBlock(block_type="tool_use", sequence=0, tool_use_id="c1"), - ] - state.open_start_ms = _BOUNDS_EPOCH_MS - state.open_end_ms = _BOUNDS_EPOCH_MS + window_ms - state._flush_message( + stream = [ + _item_notification("item/started", reasoning, started_at_ms=_BOUNDS_EPOCH_MS), + _item_notification("item/started", command, started_at_ms=_BOUNDS_EPOCH_MS + tool_from_ms), + _item_notification("item/completed", command, completed_at_ms=_BOUNDS_EPOCH_MS + tool_to_ms), + _item_notification("item/completed", reasoning, completed_at_ms=_BOUNDS_EPOCH_MS + window_ms), SimpleNamespace( - input_tokens=100, - cached_input_tokens=0, - output_tokens=think_out + action_out, - reasoning_output_tokens=think_out, - ) - ) - - collector.on_event( - AgentStartEvent(task_id="codex", prompt="go", iteration=1, timestamp=_ms_to_dt(_BOUNDS_EPOCH_MS)) - ) - collector.on_event(ToolEndEvent(task_id="codex", turn_id="codex-1", tool=command)) - collector.on_event( - AgentEndEvent( - task_id="codex", - status=AgentEndStatus.COMPLETED, - messages=list(state.messages), - usage=TokenUsage(), - timestamp=_ms_to_dt(_BOUNDS_EPOCH_MS + window_ms), - ) - ) - record = collector.build_turn_record() + method="thread/tokenUsage/updated", payload=SimpleNamespace(token_usage=SimpleNamespace(last=last)) + ), + Tick(window_ms), + ] + record = _codex_replay(stream).record return [m for m in record.messages if m.role == "assistant"] def test_the_group_is_subtracted_once_and_the_parts_still_sum(self): @@ -2826,3 +2820,360 @@ def test_both_sub_messages_still_share_one_window(self): def test_a_window_entirely_covered_by_its_tool_splits_zero_two_ways(self): published = self._published(window_ms=1000, tool_from_ms=0, tool_to_ms=1000, think_out=800, action_out=200) assert [m.generation_duration_ms for m in published] == [0.0, 0.0] + + +class TestCodexDecoder: + """``_CodexDecoder`` over a real ``TurnEmitter`` on ``CLI_EPOCH_MS``.""" + + @staticmethod + def _assistant(result: Replay) -> list: + return [m for m in result.record.messages if m.role == "assistant"] + + def test_the_codex_b_window_matches_the_pre_port_reducer(self): + """``codex_b_command_execution``: a 400 ms window holding a 250 ms command. + + 150.0 is what the pre-port reducer published for this stream. + """ + from tests._fixtures.golden_streams.codex_fixtures import _T0_MS, CODEX_SCENARIOS + + scenario = next(s for s in CODEX_SCENARIOS if s.name == "b_command_execution") + result = _codex_replay([*scenario.notifications, Tick(500)], origin_ms=_T0_MS) + + assert [m.generation_duration_ms for m in self._assistant(result)] == [150.0] + + def test_a_two_part_split_of_the_codex_b_stream_matches_the_pre_port_reducer(self): + """``codex_b`` with a reasoning item ahead of the command: a thinking and an action part. + + The window is 400 ms, the command 250 ms, and 12 of 30 output tokens are + reasoning, so the 150 ms apportions 60/90. Both literals are what the + pre-port reducer published for this stream. + """ + from tests._fixtures.golden_streams.codex_fixtures import ( + _T0_MS, + _agent_message, + _command, + _delta, + _item, + _reasoning, + _token_usage, + _turn_completed, + ) + + command = _command("cmd_b") + reasoning = _reasoning("plan") + stream = [ + _item("item/started", reasoning, started_at_ms=_T0_MS), + _item("item/completed", reasoning, completed_at_ms=_T0_MS + 50), + _item("item/started", command, started_at_ms=_T0_MS + 60), + _item("item/completed", command, completed_at_ms=_T0_MS + 310), + _delta("done"), + _item("item/completed", _agent_message("done"), completed_at_ms=_T0_MS + 400), + _token_usage(inp=120, out=30, cached=0, reasoning=12), + _turn_completed(), + Tick(500), + ] + + assistant = self._assistant(_codex_replay(stream, origin_ms=_T0_MS)) + + assert [[b.block_type for b in m.content_blocks] for m in assistant] == [["thinking"], ["tool_use", "text"]] + assert [m.generation_duration_ms for m in assistant] == [60.0, 90.0] + assert [m.output_tokens for m in assistant] == [12, 18] + assert [m.input_tokens for m in assistant] == [120, 0] + + def test_a_tool_with_no_item_id_keeps_one_id_from_start_to_completion(self): + first = SimpleNamespace(type="webSearch", query="first") + second = SimpleNamespace(type="webSearch", query="second") + stream = [ + _item_notification("item/started", first), + _item_notification("item/started", second), + _item_notification("item/completed", first), + _item_notification("item/completed", second), + ] + + result = _codex_replay(stream) + + starts = [e.tool.tool_id for e in result.events if isinstance(e, ToolStartEvent)] + ends = [e.tool.tool_id for e in result.events if isinstance(e, ToolEndEvent)] + assert len(set(starts)) == 2 + assert ends == starts + commands = {c.tool_id: c for c in result.record.commands} + assert set(commands) == set(starts) + assert [commands[i].parameters["query"] for i in starts] == ["first", "second"] + assert all(c.result_status == "success" for c in commands.values()) + assert [c.sequence_number for c in result.record.commands] == [0, 1] + assert_stream_balanced(result.events) + + def test_a_completion_with_no_start_opens_then_closes_the_call(self): + result = _codex_replay([_item_notification("item/completed", _bounds_command_item("late"))]) + + tool_events = [type(e) for e in result.events if isinstance(e, ToolStartEvent | ToolEndEvent)] + assert tool_events == [ToolStartEvent, ToolEndEvent] + assert [(c.tool_id, c.tool_name, c.result_status) for c in result.record.commands] == [ + ("late", "Bash", "success") + ] + assert_stream_balanced(result.events) + + def test_a_window_with_no_stamp_is_an_unmeasured_generation(self): + """Neither bound is known, so the message carries no duration rather than a zero-width window.""" + stream = [ + _delta("Hello"), + _item_notification("item/completed", SimpleNamespace(type="agentMessage", id="m1", text="Hello")), + SimpleNamespace( + method="thread/tokenUsage/updated", + payload=SimpleNamespace( + token_usage=SimpleNamespace( + last=SimpleNamespace( + input_tokens=10, output_tokens=5, cached_input_tokens=0, reasoning_output_tokens=0 + ) + ) + ), + ), + ] + + assistant = self._assistant(_codex_replay(stream)) + + assert len(assistant) == 1 + assert assistant[0].generation_duration_ms is None + assert assistant[0].started_at == assistant[0].completed_at + assert assistant[0].output_tokens == 5 + + +class _SequencedThread: + """A fake thread that serves one notification list per ``turn()`` call, in order.""" + + def __init__(self, *turns) -> None: + self._turns = list(turns) + + def turn(self, _user_input): + stream = _RaisingStream(self._turns.pop(0)) + return SimpleNamespace(stream=lambda: stream, interrupt=lambda: None) + + +def _usage_notification(*, last: tuple[int, int, int], total: tuple[int, int, int]) -> SimpleNamespace: + """A tokenUsage notification; each tuple is ``(input, output, cached)``.""" + return SimpleNamespace( + method="thread/tokenUsage/updated", + payload=SimpleNamespace( + token_usage=SimpleNamespace( + last=SimpleNamespace( + input_tokens=last[0], output_tokens=last[1], cached_input_tokens=last[2], reasoning_output_tokens=0 + ), + total=SimpleNamespace(input_tokens=total[0], output_tokens=total[1], cached_input_tokens=total[2]), + ) + ), + ) + + +def _reply(item_id: str, text: str) -> SimpleNamespace: + return _item_notification("item/completed", SimpleNamespace(type="agentMessage", id=item_id, text=text)) + + +class TestThreadUsageBaselinePerTurn: + """The thread-cumulative baseline advances exactly once per turn, clean or crashed.""" + + @staticmethod + def _agent(*turns) -> CodexAgent: + agent = CodexAgent(parse_agent_config(type=AgentKind.CODEX, model="gpt-5.5")) + agent.working_directory = __import__("pathlib").Path(".") + agent.codex_client = SimpleNamespace(close=lambda: None) + agent.thread = _SequencedThread(*turns) + return agent + + async def test_a_clean_turn_advances_the_baseline_to_the_sdk_total(self): + agent = self._agent( + [_reply("m1", "one"), _usage_notification(last=(100, 40, 8), total=(100, 40, 8)), _turn_completed()], + [_reply("m2", "two"), _usage_notification(last=(150, 20, 12), total=(250, 60, 20)), _turn_completed()], + ) + + first = await agent.communicate("one", iteration=1) + assert agent._thread_usage_baseline == _ThreadTotals(input=100, output=40, cached=8) + second = await agent.communicate("two", iteration=2) + + assert first.status is second.status is AgentEndStatus.COMPLETED + assert agent._thread_usage_baseline == _ThreadTotals(input=250, output=60, cached=20) + usage = second.record.token_usage + assert usage is not None + assert (usage.uncached_input_tokens, usage.output_tokens, usage.cache_read_input_tokens) == (138, 20, 12) + + async def test_a_crashed_turn_advances_the_baseline_once_past_its_flushed_tokens(self): + agent = self._agent( + [_reply("m1", "one"), _usage_notification(last=(100, 40, 8), total=(100, 40, 8)), RuntimeError("boom")], + [_reply("m2", "two"), _usage_notification(last=(150, 20, 12), total=(250, 60, 20)), _turn_completed()], + ) + + crashed = await agent.communicate("one", iteration=1) + assert crashed.status is AgentEndStatus.CRASHED + assert agent._thread_usage_baseline == _ThreadTotals(input=100, output=40, cached=8) + second = await agent.communicate("two", iteration=2) + + assert second.status is AgentEndStatus.COMPLETED + assert agent._thread_usage_baseline == _ThreadTotals(input=250, output=60, cached=20) + usage = second.record.token_usage + assert usage is not None + assert (usage.uncached_input_tokens, usage.output_tokens, usage.cache_read_input_tokens) == (138, 20, 12) + + async def test_a_timeout_race_keeps_the_committed_sdk_total_and_reply(self, monkeypatch): + agent = self._agent( + [_delta("one"), _usage_notification(last=(100, 40, 8), total=(100, 40, 8)), _turn_completed()], + ) + monkeypatch.setattr("coder_eval.agents.watchdog.ThreadedWatchdog", _ImmediateTimeoutWatchdog) + + outcome = await agent.communicate("one", iteration=1, timeout=30.0) + + assert outcome.status is AgentEndStatus.TIMEOUT + assert outcome.record.agent_output == "one" + assert agent._thread_usage_baseline == _ThreadTotals(input=100, output=40, cached=8) + usage = outcome.record.token_usage + assert usage is not None + assert (usage.uncached_input_tokens, usage.output_tokens, usage.cache_read_input_tokens) == (92, 40, 8) + + +class _SlowStream: + """Yields its notifications, then blocks the reader thread until ``release`` is set.""" + + def __init__(self, notifications) -> None: + self._it = iter(notifications) + self.release = __import__("threading").Event() + + def __iter__(self): + return self + + def __next__(self): + for nxt in self._it: + return nxt + self.release.wait(5) + raise StopIteration + + def close(self): + pass + + +class TestCommunicateCancellation: + async def test_an_external_cancel_ends_the_turn_once_then_propagates(self): + stream = _SlowStream([_reply("m1", "working")]) + agent = CodexAgent(parse_agent_config(type=AgentKind.CODEX, model="gpt-5.5")) + agent.working_directory = __import__("pathlib").Path(".") + agent.codex_client = SimpleNamespace(close=lambda: None) + agent.thread = SimpleNamespace(turn=lambda _u: SimpleNamespace(stream=lambda: stream, interrupt=lambda: None)) + recorder = _Recorder() + + task = asyncio.ensure_future(agent.communicate("go", iteration=1, stream_callback=recorder)) + for _ in range(200): + if any(isinstance(e, ToolStartEvent) or getattr(e, "text", None) for e in recorder.events): + break + await asyncio.sleep(0.01) + await asyncio.sleep(0.05) + task.cancel() + try: + with pytest.raises(asyncio.CancelledError): + await task + finally: + stream.release.set() + + ends = [e for e in recorder.events if isinstance(e, AgentEndEvent)] + assert [(e.status, e.crash_reason) for e in ends] == [(AgentEndStatus.CRASHED, "turn cancelled")] + assert agent.get_state() == AgentState.ERROR + + async def test_a_cancel_raised_inside_the_sdk_is_a_crash_outcome(self): + agent = TestThreadUsageBaselinePerTurn._agent([_reply("m1", "one"), asyncio.CancelledError()]) + + outcome = await agent.communicate("one", iteration=1) + + assert outcome.status is AgentEndStatus.CRASHED + assert outcome.record.crashed is True + assert agent.get_state() == AgentState.ERROR + caller = asyncio.current_task() + assert caller is not None and caller.cancelling() == 0 + + +class TestInnerTurnsPerGeneration: + """One inner turn per model generation, so `max_turns` counts Codex the way it counts Claude Code.""" + + @staticmethod + def _cmd(item_id: str) -> SimpleNamespace: + return SimpleNamespace( + type="commandExecution", id=item_id, command="echo", exit_code=0, aggregated_output="", duration_ms=5 + ) + + def test_each_generation_is_one_inner_turn_carrying_its_own_delta(self): + """A tool result landing after its generation's cut opens no turn; the next model output does.""" + result = _codex_replay( + [ + _reply("m1", "one"), + _usage_notification(last=(100, 40, 8), total=(100, 40, 8)), + _item_notification("item/started", self._cmd("c1")), + _usage_notification(last=(150, 20, 12), total=(250, 60, 20)), + _item_notification("item/completed", self._cmd("c1")), + _reply("m3", "three"), + _usage_notification(last=(50, 10, 0), total=(300, 70, 20)), + _turn_completed(), + ] + ) + + starts = [e.turn_id for e in result.events if isinstance(e, TurnStartEvent)] + assert starts == ["codex-1-msg-0", "codex-1-msg-1", "codex-1-msg-2"] + ends = [e for e in result.events if isinstance(e, TurnEndEvent)] + assert [ + (e.tokens.uncached_input_tokens, e.tokens.output_tokens, e.tokens.cache_read_input_tokens) + for e in ends + if e.tokens + ] == [(92, 40, 8), (138, 20, 12), (50, 10, 0)] + assert [m.message_id for m in result.record.messages if m.role == "assistant"] == starts + assert result.record.num_turns == result.record.assistant_turn_count == 3 + assert_stream_balanced(result.events) + + def test_a_billed_cut_with_no_content_counts_and_the_next_turn_gets_a_fresh_id(self): + agent = CodexAgent(parse_agent_config(type=AgentKind.CODEX, model="gpt-5.5")) + final = _usage_notification(last=(50, 10, 0), total=(150, 15, 0)) + result = replay( + [_usage_notification(last=(100, 5, 0), total=(100, 5, 0)), _reply("m1", "one"), final, _turn_completed()], + lambda emitter: _CodexDecoder(agent, emitter, turn_id="codex-1"), + clock=ScriptedClock(_ms_to_dt(_BOUNDS_EPOCH_MS)), + basis=TimingBasis.CLI_EPOCH_MS, + model="gpt-5.5", + end=lambda decoder: decoder.end(AgentEndStatus.COMPLETED, sdk_token_usage=final.payload.token_usage), + ) + + starts = [e.turn_id for e in result.events if isinstance(e, TurnStartEvent)] + assert starts == ["codex-1-msg-0", "codex-1-msg-1"] + assert [m.message_id for m in result.record.messages if m.role == "assistant"] == ["codex-1-msg-1"] + assert result.record.num_turns == 2 + assert_stream_balanced(result.events) + + def test_a_stop_mid_generation_closes_that_turn_with_the_stop_status(self): + result = _codex_replay( + [ + _reply("m1", "one"), + _usage_notification(last=(100, 40, 8), total=(100, 40, 8)), + _item_notification("item/started", self._cmd("c1")), + ] + ) + + ends = [(e.turn_id, e.status) for e in result.events if isinstance(e, TurnEndEvent)] + assert ends == [("codex-1-msg-0", TurnEndStatus.COMPLETED), ("codex-1-msg-1", TurnEndStatus.COMPLETED)] + assert_stream_balanced(result.events) + + async def test_max_turns_stops_when_the_turn_past_the_cap_starts(self): + """The cap is the monitor's: it latches at the first item of model turn N+1, and the pump pulls no more.""" + agent = _started_agent( + parse_agent_config(type=AgentKind.CODEX, model="gpt-5.5"), + [ + _reply("m1", "one"), + _usage_notification(last=(100, 40, 8), total=(100, 40, 8)), + _item_notification("item/started", self._cmd("c1")), + _item_notification("item/completed", self._cmd("c1")), + _turn_completed(), + ], + ) + monitor = TurnMonitor("t", [], limits=RunLimits(max_turns=1)) + + outcome = await agent.communicate("go", iteration=1, stream_callback=monitor, should_stop=monitor.should_stop) + + assert monitor.stop_reason is StopReason.MODEL_TURN_CAP + assert monitor.model_turns == 2 + assert outcome.status is AgentEndStatus.TOOL_CALLS_EXHAUSTED + assert outcome.record.crashed is False + assert outcome.record.tool_calls_exhausted is True + assert [(c.tool_id, c.result_status) for c in outcome.record.commands] == [("c1", "unknown")] + assert outcome.record.num_turns == 2 + assert agent.thread.last_handle.interrupted is True diff --git a/tests/test_codex_agent_live.py b/tests/test_codex_agent_live.py index 8e7f2c34..a22b1bbb 100644 --- a/tests/test_codex_agent_live.py +++ b/tests/test_codex_agent_live.py @@ -22,7 +22,7 @@ pytest.importorskip("openai_codex") from coder_eval.agents.codex_agent import CodexAgent -from coder_eval.models import AgentKind, parse_agent_config +from coder_eval.models import AgentKind, CodexAgentConfig, parse_agent_config _live = pytest.mark.live @@ -42,6 +42,7 @@ def _make_agent() -> CodexAgent: permission_mode="bypassPermissions", # full access so it can run shell + write files model=os.getenv("CODEX_MODEL"), ) + assert isinstance(config, CodexAgentConfig) return CodexAgent(config, instance_name="codex-live") @@ -51,10 +52,13 @@ async def test_codex_live_produces_text(tmp_path): agent = _make_agent() await agent.start(str(tmp_path)) try: - record = await agent.communicate( - "Reply with exactly the word PONG and nothing else.", - timeout=120, - ) + record = ( + await agent.communicate( + "Reply with exactly the word PONG and nothing else.", + iteration=1, + timeout=120, + ) + ).record finally: await agent.stop() @@ -69,10 +73,13 @@ async def test_codex_live_runs_shell_command_captured_as_telemetry(tmp_path): agent = _make_agent() await agent.start(str(tmp_path)) try: - record = await agent.communicate( - "Run the shell command `echo coder-eval-live` and report its output.", - timeout=120, - ) + record = ( + await agent.communicate( + "Run the shell command `echo coder-eval-live` and report its output.", + iteration=1, + timeout=120, + ) + ).record finally: await agent.stop() @@ -93,10 +100,13 @@ async def test_codex_live_edits_file_and_records_telemetry(tmp_path): agent = _make_agent() await agent.start(str(tmp_path)) try: - record = await agent.communicate( - "Create a file named hello.txt in the current directory containing the text 'hi'.", - timeout=120, - ) + record = ( + await agent.communicate( + "Create a file named hello.txt in the current directory containing the text 'hi'.", + iteration=1, + timeout=120, + ) + ).record finally: await agent.stop() @@ -127,19 +137,21 @@ def on_event(self, event): agent = _make_agent() await agent.start(str(tmp_path)) try: - record = await agent.communicate( - "Run `echo one`, then `echo two`, then `echo three`, each as a separate shell command, " - "then create three files a.txt, b.txt and c.txt.", - timeout=180, - stream_callback=sink, - should_stop=lambda: StopReason.EARLY_CRITERION if sink.tool_started else None, - ) + record = ( + await agent.communicate( + "Run `echo one`, then `echo two`, then `echo three`, each as a separate shell command, " + + "then create three files a.txt, b.txt and c.txt.", + iteration=1, + timeout=180, + stream_callback=sink, + should_stop=lambda: StopReason.EARLY_CRITERION if sink.tool_started else None, + ) + ).record finally: await agent.stop() # Clean cooperative stop: no crash, no pending partial, STOPPED_EARLY status. assert record.crashed is False - assert agent.pending_turn is None assert sink.ends, "expected an AgentEndEvent" assert sink.ends[-1].status == AgentEndStatus.STOPPED_EARLY @@ -150,7 +162,7 @@ async def test_codex_live_token_usage_populated(tmp_path): agent = _make_agent() await agent.start(str(tmp_path)) try: - record = await agent.communicate("Say hello.", timeout=120) + record = (await agent.communicate("Say hello.", iteration=1, timeout=120)).record finally: await agent.stop() diff --git a/tests/test_codex_agent_unit.py b/tests/test_codex_agent_unit.py index 1fcec84c..df8cb998 100644 --- a/tests/test_codex_agent_unit.py +++ b/tests/test_codex_agent_unit.py @@ -1,10 +1,11 @@ """SDK-independent unit tests for CodexAgent. -These tests exercise pure-logic seams of ``codex_agent.py`` — the per-turn -``_CodexTurnState`` list-mutation contract — that need NO Codex SDK. ``codex_agent`` imports ``openai_codex`` -only lazily (inside ``start`` / ``_build_thread_options`` / the turn-completed -handler), so the module imports cleanly without the extra and these tests run -in the base Quality Gate. +These tests exercise pure-logic seams of ``codex_agent.py`` — ``_CodexDecoder`` +driven over a real ``TurnEmitter`` and the telemetry builders — that need NO +Codex SDK. ``codex_agent`` imports ``openai_codex`` only lazily (inside +``start`` / ``_build_thread_options`` / the turn-completed handler), so the +module imports cleanly without the extra and these tests run in the base +Quality Gate. The SDK-dependent tests (anything constructing real SDK notification/Turn types or driving ``communicate``) stay in ``test_codex_agent.py`` behind that @@ -13,79 +14,81 @@ from __future__ import annotations -import time from datetime import datetime from types import SimpleNamespace import pytest -from coder_eval.agents.codex_agent import CodexAgent -from coder_eval.models import AgentKind, parse_agent_config +from coder_eval.agents.codex_agent import CodexAgent, _CodexDecoder +from coder_eval.models import AgentKind, TimingBasis, parse_agent_config +from coder_eval.streaming.emitter import TurnEmitter +from coder_eval.streaming.events import AgentEndStatus +from coder_eval.testing import ScriptedClock def _item_notification(method: str, root: SimpleNamespace) -> SimpleNamespace: return SimpleNamespace(method=method, payload=SimpleNamespace(item=SimpleNamespace(root=root))) -class TestCodexTurnState: - """Unit tests for the per-turn state object extracted from _run_turn_with_streaming.""" +class TestCodexDecoder: + """Unit tests for the per-turn decoder that ``_run_turn_with_streaming`` feeds.""" @staticmethod - def _state(agent): - from coder_eval.agents.codex_agent import _CodexTurnState - from coder_eval.streaming.callbacks import CompositeStreamCallback - from coder_eval.streaming.collector import EventCollector - - commands: list = [] - messages: list = [] - collector = EventCollector() - state = _CodexTurnState( - agent, - emit=CompositeStreamCallback([collector]), + def _decoder() -> _CodexDecoder: + agent = CodexAgent(parse_agent_config(type=AgentKind.CODEX, model="gpt-5-codex")) + emitter = TurnEmitter( task_id="codex", - turn_id="codex-1", - collector=collector, - commands=commands, - messages=messages, - user_input="go", iteration=1, - turn_start_time=time.monotonic(), + prompt="go", + model="gpt-5-codex", + basis=TimingBasis.CLI_EPOCH_MS, + clock=ScriptedClock(datetime(2026, 1, 1)), + sinks=[], ) - return state, commands, messages + emitter.begin() + return _CodexDecoder(agent, emitter, turn_id="codex-1") - def test_holds_commands_and_messages_by_identity(self): - agent = CodexAgent(parse_agent_config(type=AgentKind.CODEX, model="gpt-5-codex")) - state, commands, messages = self._state(agent) - # The state must hold the caller's SAME list objects (no copy) so a - # mid-turn crash keeps the partial transcript. - assert state.commands is commands - assert state.messages is messages + def test_a_crash_keeps_the_partial_transcript(self): + # A mid-turn crash must keep what the turn already produced: the flushed + # message and the completed command both reach the crashed record. + decoder = self._decoder() + cmd_root = SimpleNamespace( + type="commandExecution", id="c1", command="echo hi", exit_code=0, aggregated_output="hi\n", duration_ms=5 + ) + decoder(_item_notification("item/started", cmd_root)) + decoder(_item_notification("item/completed", cmd_root)) + decoder.flush(SimpleNamespace(input_tokens=10, cached_input_tokens=0, output_tokens=5)) - def test_command_dispatch_mutates_lists_in_place(self): - agent = CodexAgent(parse_agent_config(type=AgentKind.CODEX, model="gpt-5-codex")) - state, commands, _messages = self._state(agent) + outcome = decoder.end(AgentEndStatus.CRASHED, reason="stream blew up") + + assert outcome.record.crashed is True + assert [m.message_id for m in outcome.record.messages if m.role == "assistant"] == ["codex-1-msg-0"] + assert [m.message_id for m in decoder.messages] == ["codex-1-msg-0"] + assert [c.tool_id for c in outcome.record.commands] == ["c1"] + + def test_command_dispatch_reaches_the_record(self): + decoder = self._decoder() cmd_root = SimpleNamespace( type="commandExecution", id="c1", command="echo hi", exit_code=0, aggregated_output="hi\n", duration_ms=5 ) - state.on_item_started(_item_notification("item/started", cmd_root)) - state.on_item_completed(_item_notification("item/completed", cmd_root)) + decoder(_item_notification("item/started", cmd_root)) + decoder(_item_notification("item/completed", cmd_root)) - # Telemetry recorded into the SAME commands list, by identity. - assert commands is state.commands + assert decoder.opened_tools == {"c1"} + # A tool_use block was recorded into the open buffer (cut at the next + # tokenUsage flush, not here), joinable to the command by tool_id. + assert any(b.block_type == "tool_use" and b.tool_use_id == "c1" for b in decoder.open_blocks) + commands = decoder.end(AgentEndStatus.COMPLETED).record.commands assert len(commands) == 1 assert commands[0].tool_name == "Bash" assert commands[0].result_status == "success" - # A tool_use block was recorded into the open buffer (cut at the next - # tokenUsage flush, not here), joinable to the command by tool_id. - assert any(b.block_type == "tool_use" and b.tool_use_id == "c1" for b in state.open_blocks) def test_command_output_recorded_whole_not_truncated(self): # Regression for the Codex `output[:100]` bug (CE043): result_summary must # carry the FULL command output so result_tokens reflects real tool-output # size instead of being pinned at a ~31-token, 100-char cap. - agent = CodexAgent(parse_agent_config(type=AgentKind.CODEX, model="gpt-5-codex")) - state, commands, _messages = self._state(agent) + decoder = self._decoder() big_output = "X" * 4000 # far beyond the old 100-char clip cmd_root = SimpleNamespace( @@ -96,10 +99,10 @@ def test_command_output_recorded_whole_not_truncated(self): aggregated_output=big_output, duration_ms=5, ) - state.on_item_started(_item_notification("item/started", cmd_root)) - state.on_item_completed(_item_notification("item/completed", cmd_root)) + decoder(_item_notification("item/started", cmd_root)) + decoder(_item_notification("item/completed", cmd_root)) - cmd = commands[0] + cmd = decoder.end(AgentEndStatus.COMPLETED).record.commands[0] assert big_output in (cmd.result_summary or ""), "full output must be recorded, not truncated" # result_tokens (ceil(len/4)) must scale with the real output, not ~31. assert cmd.result_tokens >= len(big_output) // 4 @@ -176,17 +179,39 @@ def _file_change_item(item_id: str = "fc_1", *, duration_ms: object = None) -> S _BUILDERS_WITH_SDK_DURATION = [_BUILDERS[0], _BUILDERS[2]] +class TestResultStatus: + """A completed item is a resolved call: its status is the tool end status, never ``unknown``.""" + + @pytest.mark.parametrize( + ("root", "expected"), + [ + pytest.param( + SimpleNamespace(type="commandExecution", id="c1", command="rm x", exit_code=None, status="declined"), + "error", + id="declined-command", + ), + pytest.param(SimpleNamespace(type="webSearch", id="w1", query="q"), "success", id="no-status-field"), + ], + ) + def test_a_completed_item_records_a_resolved_status(self, root, expected): + assert TestExecutionBoundsFromSdkStamps._build(root, root.type).result_status == expected + + class TestExecutionBoundsFromSdkStamps: - """All three builders derive bounds and duration from the SDK stamps.""" + """For all three item kinds, the recorded command's bounds and duration come from the SDK stamps.""" @staticmethod def _build(root, root_type, *, started_ms=None, completed_ms=None): - agent = CodexAgent(parse_agent_config(type=AgentKind.CODEX, model="gpt-5.5")) - telemetry, _ = agent._telemetry_for_item( - root, root_type, getattr(root, "id", "x"), 0, started_ms=started_ms, completed_ms=completed_ms - ) - assert telemetry is not None - return telemetry + """The command the turn record keeps after the decoder saw the item start and complete.""" + del root_type + decoder = TestCodexDecoder._decoder() + for method, stamps in ( + ("item/started", {"started_at_ms": started_ms}), + ("item/completed", {"completed_at_ms": completed_ms}), + ): + decoder(SimpleNamespace(method=method, payload=SimpleNamespace(item=SimpleNamespace(root=root), **stamps))) + (command,) = decoder.emitter.finalize(AgentEndStatus.COMPLETED).record.commands + return command @pytest.mark.parametrize(("factory", "root_type"), _BUILDERS) def test_both_stamps_give_bounds_and_an_exact_duration(self, factory, root_type): @@ -207,8 +232,9 @@ def test_only_one_stamp_never_fabricates_an_interval(self, factory, root_type): # _ms_to_dt(None) is datetime.now(), so pairing a real stamp with a # missing one would invent an interval running to the present moment. tel = self._build(factory(), root_type, started_ms=_EPOCH_MS, completed_ms=None) - assert tel.execution_started_at is None + assert tel.execution_started_at == datetime.fromtimestamp(_EPOCH_MS / 1000) assert tel.execution_completed_at is None + assert tel.duration_ms is None @pytest.mark.parametrize(("factory", "root_type"), _BUILDERS) def test_a_backwards_pair_clamps_but_keeps_both_bounds(self, factory, root_type): diff --git a/tests/test_command_telemetry_result_data.py b/tests/test_command_telemetry_result_data.py index 7fa214a9..b1788774 100644 --- a/tests/test_command_telemetry_result_data.py +++ b/tests/test_command_telemetry_result_data.py @@ -2,31 +2,35 @@ from __future__ import annotations -import time from datetime import datetime from typing import Any -from coder_eval.agents.claude_code_agent import ClaudeCodeAgent -from coder_eval.models import AgentKind, CommandTelemetry, parse_agent_config - - -def _agent() -> ClaudeCodeAgent: - """Build a minimal ClaudeCodeAgent instance just to reach _resolve_pending_command.""" - return ClaudeCodeAgent(parse_agent_config(type=AgentKind.CLAUDE_CODE)) - - -def _make_pending(tool_id: str, tool_name: str = "Bash") -> dict[str, dict[str, Any]]: - """Build a pending_commands dict matching the agent's internal shape.""" - telemetry = CommandTelemetry( - tool_name=tool_name, - tool_id=tool_id, - timestamp=datetime.now(), - parameters={}, - sequence_number=0, - result_status=None, - duration_ms=None, +from coder_eval.agents.claude_code_agent import ClaudeCodeAgent, _ClaudeDecoder +from coder_eval.models import AgentKind, CommandTelemetry, TimingBasis, parse_agent_config +from coder_eval.streaming.emitter import TurnEmitter +from coder_eval.streaming.events import AgentEndStatus +from coder_eval.testing import ScriptedClock +from tests._fixtures.golden_streams.claude_fixtures import AssistantMessage, ToolUseBlock, UserMessage + + +def _resolve(tool_id: str, content: Any, tool_name: str = "Bash") -> CommandTelemetry: + """Open one tool through the Claude decoder, resolve it with ``content``, and return its record.""" + emitter = TurnEmitter( + task_id="t", + iteration=1, + prompt="go", + model="m", + basis=TimingBasis.TURN_CLOCK, + clock=ScriptedClock(datetime(2026, 1, 1)), + sinks=[], ) - return {tool_id: {"telemetry": telemetry, "command_start_time": time.monotonic()}} + emitter.begin() + agent = ClaudeCodeAgent(parse_agent_config(type=AgentKind.CLAUDE_CODE)) + decoder = _ClaudeDecoder(agent, emitter, effective_model="m") + decoder(AssistantMessage([ToolUseBlock(tool_id, tool_name, {})], message_id="m1")) + decoder(UserMessage(tool_id, False, content)) + (cmd,) = decoder.end(AgentEndStatus.COMPLETED).record.commands + return cmd def test_command_telemetry_result_data_defaults_to_none() -> None: @@ -83,179 +87,90 @@ def test_try_parse_json_value_tolerates_leading_whitespace_for_array() -> None: assert ClaudeCodeAgent._try_parse_json_value(" [1,2]") == [1, 2] -def test_resolve_pending_command_populates_result_data_for_json_object() -> None: +def test_tool_result_populates_result_data_for_json_object() -> None: tool_id = "toolu_json_obj" - pending = _make_pending(tool_id) content = '{"a":1,"b":"x"}' - _agent()._resolve_pending_command( - tool_id, - False, - content, - pending, - set(), - now=datetime.now(), - ) - - cmd = pending[tool_id]["telemetry"] + cmd = _resolve(tool_id, content) assert cmd.result_data == {"a": 1, "b": "x"} assert cmd.result_summary == content -def test_resolve_pending_command_does_not_truncate_long_result_summary() -> None: +def test_tool_result_does_not_truncate_long_result_summary() -> None: tool_id = "toolu_long" - pending = _make_pending(tool_id) content = "x" * 5000 # well past the old 200-char cap - _agent()._resolve_pending_command( - tool_id, - False, - content, - pending, - set(), - now=datetime.now(), - ) - - cmd = pending[tool_id]["telemetry"] + cmd = _resolve(tool_id, content) assert cmd.result_summary == content assert len(cmd.result_summary) == 5000 -def test_resolve_pending_command_populates_result_data_for_json_array() -> None: +def test_tool_result_populates_result_data_for_json_array() -> None: tool_id = "toolu_json_arr" - pending = _make_pending(tool_id) content = '[{"a":1}]' - _agent()._resolve_pending_command( - tool_id, - False, - content, - pending, - set(), - now=datetime.now(), - ) - - cmd = pending[tool_id]["telemetry"] + cmd = _resolve(tool_id, content) assert cmd.result_data == [{"a": 1}] -def test_resolve_pending_command_leaves_result_data_none_for_plain_text() -> None: +def test_tool_result_leaves_result_data_none_for_plain_text() -> None: tool_id = "toolu_plain" - pending = _make_pending(tool_id) content = "hello world" - _agent()._resolve_pending_command( - tool_id, - False, - content, - pending, - set(), - now=datetime.now(), - ) - - cmd = pending[tool_id]["telemetry"] + cmd = _resolve(tool_id, content) assert cmd.result_data is None assert cmd.result_summary == "hello world" -def test_resolve_pending_command_populates_result_data_for_flow_debug_fixture() -> None: +def test_tool_result_populates_result_data_for_flow_debug_fixture() -> None: tool_id = "toolu_flow_debug" - pending = _make_pending(tool_id, tool_name="mcp__maestro__run_flow") content = ( '{"Code":"FlowDebug","Data":{"finalStatus":"Completed",' '"elementExecutions":[{"elementId":"e1","status":"Completed"}]}}' ) - _agent()._resolve_pending_command( - tool_id, - False, - content, - pending, - set(), - now=datetime.now(), - ) - - cmd = pending[tool_id]["telemetry"] + cmd = _resolve(tool_id, content, tool_name="mcp__maestro__run_flow") assert cmd.result_data is not None assert isinstance(cmd.result_data, dict) assert cmd.result_data["Code"] == "FlowDebug" assert cmd.result_data["Data"]["elementExecutions"][0]["elementId"] == "e1" -def test_resolve_pending_command_handles_sdk_list_content_shape() -> None: +def test_tool_result_handles_sdk_list_content_shape() -> None: """MCP tool results arrive as list[{'type': 'text', 'text': '...'}]; extract and parse.""" tool_id = "toolu_mcp_flow_debug" - pending = _make_pending(tool_id, tool_name="mcp__maestro__run_flow") content = [ {"type": "text", "text": '{"Code":"FlowDebug","Data":{"finalStatus":"Completed"}}'}, ] - _agent()._resolve_pending_command( - tool_id, - False, - content, - pending, - set(), - now=datetime.now(), - ) - - cmd = pending[tool_id]["telemetry"] + cmd = _resolve(tool_id, content, tool_name="mcp__maestro__run_flow") assert cmd.result_data == {"Code": "FlowDebug", "Data": {"finalStatus": "Completed"}} -def test_resolve_pending_command_concatenates_multiple_text_blocks() -> None: +def test_tool_result_concatenates_multiple_text_blocks() -> None: tool_id = "toolu_mcp_multi_text" - pending = _make_pending(tool_id) content = [ {"type": "text", "text": '{"a":'}, {"type": "text", "text": '1,"b":2}'}, ] - _agent()._resolve_pending_command( - tool_id, - False, - content, - pending, - set(), - now=datetime.now(), - ) - - cmd = pending[tool_id]["telemetry"] + cmd = _resolve(tool_id, content) assert cmd.result_data == {"a": 1, "b": 2} -def test_resolve_pending_command_list_without_text_blocks_yields_none() -> None: +def test_tool_result_list_without_text_blocks_yields_none() -> None: """An SDK list of only non-text blocks (e.g., images) produces no JSON.""" tool_id = "toolu_mcp_image" - pending = _make_pending(tool_id) content = [{"type": "image", "source": {"data": "..."}}] - _agent()._resolve_pending_command( - tool_id, - False, - content, - pending, - set(), - now=datetime.now(), - ) - - assert pending[tool_id]["telemetry"].result_data is None + cmd = _resolve(tool_id, content) + assert cmd.result_data is None -def test_resolve_pending_command_none_content_yields_none() -> None: +def test_tool_result_none_content_yields_none() -> None: tool_id = "toolu_none" - pending = _make_pending(tool_id) - - _agent()._resolve_pending_command( - tool_id, - False, - None, - pending, - set(), - now=datetime.now(), - ) - cmd = pending[tool_id]["telemetry"] + cmd = _resolve(tool_id, None) assert cmd.result_data is None assert cmd.result_summary is None diff --git a/tests/test_custom_lint.py b/tests/test_custom_lint.py index 84cd2067..fcc078fd 100644 --- a/tests/test_custom_lint.py +++ b/tests/test_custom_lint.py @@ -2198,6 +2198,16 @@ def test_run_limits_render_pins_the_header_and_the_noop_cap_cell(self): cap = next(line for line in lines if line.startswith("| `max_tool_calls` |")) assert cap.endswith("| not polled (never fires) |") + def test_run_limits_render_pins_the_model_turn_cells(self): + from tests.lint.harness_parity import render_run_limits_table + + rows = render_run_limits_table(["max_turns", "expected_turns"]).splitlines()[2:] + assert len(rows) == 2 + for row in rows: + cells = [cell.strip() for cell in row.strip("|").split("|")] + assert "TurnMonitor" in cells[1] + assert cells[-1] == "rejected at resolution" + def test_a_run_limits_field_without_a_cell_rule_fails_the_render(self): from tests.lint.harness_parity import render_run_limits_table @@ -2219,7 +2229,7 @@ def test_render_pins_the_header_and_a_known_cell(self): system_prompt = next(line for line in lines if line.startswith("| `system_prompt` |")) assert system_prompt.split(" | ")[5] == "enforced" granularity = next(line for line in lines if line.startswith("| `usage_granularity` |")) - assert granularity == "| `usage_granularity` | generation | turn | turn | step | step | turn |" + assert granularity == "| `usage_granularity` | generation | generation | generation | step | step | turn |" @pytest.mark.lint @@ -2294,6 +2304,7 @@ def _violations(source: str, filepath: str) -> list: [ "n = self.max_tool_calls", "state.max_turns_hit = True", + "x = limits.expected_turns", "limits: RunLimits | None = None", "from coder_eval.utils import expand_env_vars", "def communicate(self, max_turns=None): ...", @@ -4054,6 +4065,159 @@ def _write_pair(root: Path, entry_extra: dict | None = None) -> None: (market_dir / "marketplace.json").write_text(json.dumps({"name": "demo", "plugins": [entry]}), encoding="utf-8") +class TestCE071PriceTurnOnly: + """CE071 — adapters and the turn monitor price a turn only through ``price_turn``.""" + + @staticmethod + def _violations(source: str, filepath: str) -> list: + import ast + + from tests.lint.rules.ce071_price_turn_only import PriceTurnOnly + + return list(PriceTurnOnly(filepath).check(ast.parse(source))) + + @pytest.mark.parametrize( + "source", + [ + "from coder_eval.pricing import calculate_cost", + "cost = calculate_cost(model, 1, 2)", + "cost = pricing.calculate_cost(model, 1, 2)", + ], + ) + @pytest.mark.parametrize( + "filepath", + ["/repo/src/coder_eval/agents/x_agent.py", "/repo/src/coder_eval/orchestration/turn_monitor.py"], + ) + def test_calculate_cost_in_an_adapter_or_the_monitor_violates(self, source: str, filepath: str): + found = self._violations(source, filepath) + assert found + assert "price_turn" in found[0].message + + @pytest.mark.parametrize( + "filepath", + [ + "/repo/src/coder_eval/pricing.py", + "/repo/src/coder_eval/evaluation/judge_usage.py", + "/repo/src/coder_eval/orchestration/early_stop.py", + ], + ) + def test_the_same_code_elsewhere_is_allowed(self, filepath: str): + source = "from coder_eval.pricing import calculate_cost\ncost = calculate_cost(model, 1, 2)" + assert not self._violations(source, filepath) + + def test_price_turn_is_allowed_in_an_adapter(self): + source = "from coder_eval.pricing import price_turn\ncost = price_turn(usage, (model,))" + assert not self._violations(source, "/repo/src/coder_eval/agents/x_agent.py") + + +class TestCE072EmitterSoleWriter: + """CE072 — an adapter writes the event protocol only through ``TurnEmitter``.""" + + AGENT = "/repo/src/coder_eval/agents/x_agent.py" + + @staticmethod + def _violations(source: str, filepath: str) -> list: + import ast + + from tests.lint.rules.ce072_emitter_sole_writer import EmitterSoleWriter + + return list(EmitterSoleWriter(filepath).check(ast.parse(source))) + + @pytest.mark.parametrize( + ("imported", "module"), + [ + ("AgentStartEvent", "coder_eval.streaming.events"), + ("AgentEndEvent", "coder_eval.streaming.events"), + ("TurnStartEvent", "coder_eval.streaming.events"), + ("TurnEndEvent", "coder_eval.streaming.events"), + ("ToolStartEvent", "coder_eval.streaming.events"), + ("ToolEndEvent", "coder_eval.streaming.events"), + ("TextChunkEvent", "coder_eval.streaming.events"), + ("AssistantMessage", "coder_eval.models"), + ("EventCollector", "coder_eval.streaming.collector"), + ], + ) + def test_each_banned_constructor_in_an_adapter_violates(self, imported: str, module: str): + found = self._violations(f"from {module} import {imported}\nx = {imported}()\n", self.AGENT) + assert len(found) == 1 + assert "TurnEmitter" in found[0].message + + @pytest.mark.parametrize( + "source", + [ + "from coder_eval.models import AssistantMessage as Msg\nx = Msg(content_blocks=[])\n", + "from ..models import AssistantMessage as Msg\nx = Msg()\n", + "from ..streaming.events import ToolEndEvent as End\nx = End()\n", + "import coder_eval.models as models\nx = models.AssistantMessage()\n", + "from coder_eval.streaming import EventCollector as Collector\nx = Collector()\n", + "from ..streaming import TurnEndEvent\nx = TurnEndEvent()\n", + "from coder_eval.streaming import events\nx = events.ToolEndEvent()\n", + "from ...streaming.collector import EventCollector as C\nx = C()\n", + ], + ) + def test_an_alias_a_relative_import_or_an_attribute_violates(self, source: str): + assert len(self._violations(source, self.AGENT)) == 1 + + def test_a_type_only_import_and_command_telemetry_are_allowed(self): + source = ( + "from coder_eval.models import AssistantMessage, CommandTelemetry\n" + "def f(m: AssistantMessage) -> None:\n" + " CommandTelemetry(tool_name='Bash', tool_id='t', timestamp=now)\n" + ) + assert not self._violations(source, self.AGENT) + + def test_the_same_calls_in_the_emitter_pass(self): + source = "from coder_eval.streaming.events import AgentEndEvent\nx = AgentEndEvent()\n" + assert not self._violations(source, "/repo/src/coder_eval/streaming/emitter.py") + + def test_the_live_tree_has_no_violations(self): + from tests.lint.runner import check_file + + root = Path(__file__).parent.parent / "src" / "coder_eval" + assert [str(v) for path in sorted(root.rglob("*.py")) for v in check_file(path) if v.rule_id == "CE072"] == [] + + +class TestCE073CreateSubprocessExplicitStdin: + """CE073 — every asyncio subprocess spawn under src/coder_eval decides its stdin.""" + + SRC = "/repo/src/coder_eval/agents/x_agent.py" + + @staticmethod + def _violations(source: str, filepath: str) -> list: + import ast + + from tests.lint.rules.ce073_create_subprocess_explicit_stdin import CreateSubprocessExplicitStdin + + return list(CreateSubprocessExplicitStdin(filepath).check(ast.parse(source))) + + @pytest.mark.parametrize( + "source", + [ + "p = await asyncio.create_subprocess_exec('pi', stdout=PIPE, limit=1)", + "p = await asyncio.create_subprocess_shell('ls', limit=1)", + "p = await create_subprocess_exec('pi', limit=1)", + "p = await create_subprocess_shell('ls', limit=1)", + ], + ) + def test_a_spawn_without_stdin_violates(self, source: str): + found = self._violations(source, self.SRC) + assert found + assert "stdin=" in found[0].message + + @pytest.mark.parametrize( + "source", + [ + "p = await asyncio.create_subprocess_exec('pi', stdin=asyncio.subprocess.DEVNULL, limit=1)", + "p = await asyncio.create_subprocess_shell('ls', stdin=asyncio.subprocess.PIPE, limit=1)", + ], + ) + def test_an_explicit_stdin_passes(self, source: str): + assert not self._violations(source, self.SRC) + + def test_outside_src_is_out_of_scope(self): + assert not self._violations("p = await asyncio.create_subprocess_exec('x')", "/repo/tests/test_x.py") + + class TestCE054EnvInfoKeyRoundTrip: """CE054 fires when an environment_info key is read with no writer anywhere. @@ -4272,130 +4436,6 @@ def test_the_real_module_is_clean(self): assert "_os._exit(137)" in source, "the guarded call must still exist" -class TestCE061WindowViaCloseWindow: - """CE061 flags a reducer that computes a generation window of its own. - - Every source string carries its own import line: the rule derives its - constructor set from the module's own `coder_eval.models` imports (shared - with CE060 via `_model_ctor`), so a bare `AssistantMessage(...)` with no - import is correctly invisible to it. - """ - - _IMPORT = "from coder_eval.models import AssistantMessage\n" - _HELPER = "from coder_eval.timing import close_window\n" - - @staticmethod - def _run(src: str, filepath: str = "src/coder_eval/agents/pi_agent.py"): - import ast - - from tests.lint.rules.ce061_window_via_close_window import WindowViaCloseWindow - - return WindowViaCloseWindow(filepath).check(ast.parse(src)) - - def test_flags_a_measured_window_without_the_helper(self): - assert len(self._run(self._IMPORT + "m = AssistantMessage(generation_duration_ms=x)")) == 1 - - def test_allows_a_measured_window_when_the_helper_is_imported(self): - assert not self._run(self._IMPORT + self._HELPER + "m = AssistantMessage(generation_duration_ms=x)") - - def test_allows_an_explicit_none(self): - # "Never measured" is an honest claim and needs no window arithmetic — - # codex's rollout rebuild and claude-code's sub-agent synthesis. - assert not self._run(self._IMPORT + "m = AssistantMessage(generation_duration_ms=None)") - - def test_allows_the_kwarg_absent(self): - # Defaults to None, which is the same honest claim. - assert not self._run(self._IMPORT + "m = AssistantMessage(model=model)") - - def test_flags_an_arbitrary_alias(self): - # The gap CE058 concedes: a name list guards the in-tree spelling by - # coincidence and misses `as Msg` outright. - assert ( - len(self._run("from coder_eval.models import AssistantMessage as Msg\nm = Msg(generation_duration_ms=x)")) - == 1 - ) - - def test_flags_the_module_attribute_spelling(self): - assert ( - len(self._run("import coder_eval.models as models\nm = models.AssistantMessage(generation_duration_ms=x)")) - == 1 - ) - - def test_accepts_a_relative_helper_import(self): - # `agents/` uses relative imports; matching only the absolute path - # would leave the rule blind for a whole file. - assert not self._run( - self._IMPORT + "from ..timing import close_window\nm = AssistantMessage(generation_duration_ms=x)" - ) - - def test_accepts_the_module_import_spelling_of_the_helper(self): - # `timing.close_window(...)` is a working call site; a rule that saw - # only the from-import would tell its author to change it. - assert not self._run( - self._IMPORT + "from coder_eval import timing\nm = AssistantMessage(generation_duration_ms=x)" - ) - - def test_an_unrelated_timing_import_does_not_disarm_the_rule(self): - # `from somewhere.else import timing` is not this module; accepting any - # name spelled `timing` would switch the rule off for a whole file. - assert ( - len( - self._run( - self._IMPORT + "from vendor.sdk import timing\nm = AssistantMessage(generation_duration_ms=x)" - ) - ) - == 1 - ) - - def test_ignores_a_file_outside_agents(self): - assert not self._run( - self._IMPORT + "m = AssistantMessage(generation_duration_ms=x)", - filepath="src/coder_eval/streaming/collector.py", - ) - - def test_keys_on_the_helper_name_rather_than_a_literal(self): - from coder_eval.timing import close_window as _helper - from tests.lint.rules import ce061_window_via_close_window as rule_mod - - assert _helper.__name__ == rule_mod._HELPER - - def test_the_real_agents_tree_is_clean(self): - # After claude-code's single permanent suppression. Antigravity - # carried a temporary one until it moved onto `close_window`. - import pathlib - - from tests.lint.rules.ce061_window_via_close_window import WindowViaCloseWindow - from tests.lint.runner import check_file - - root = pathlib.Path(__file__).resolve().parent.parent / "src" / "coder_eval" / "agents" - found = [v for path in sorted(root.glob("*.py")) for v in check_file(path, [WindowViaCloseWindow])] - assert not found, found - - def test_the_rule_is_now_exemption_free(self): - """No reducer needs a `# noqa: CE061` any more, and the set is PINNED empty. - - A noqa nobody needs is a noqa that outlives its reason, so this asserts - the exact set rather than merely that it shrank. It has earned that - twice: antigravity carried a TEMPORARY suppression until it moved onto - `close_window`, and claude-code carried a permanent one until the tool - subtraction moved to `timing.subtract_tool_time` — at which - point it could call the same shrunken helper as the other four. This - test is what failed each time the reason expired. - """ - import ast - import pathlib - - from tests.lint.rules.ce061_window_via_close_window import WindowViaCloseWindow - - root = pathlib.Path(__file__).resolve().parent.parent / "src" / "coder_eval" / "agents" - suppressed = { - path.name - for path in sorted(root.glob("*.py")) - if WindowViaCloseWindow(str(path)).check(ast.parse(path.read_text(encoding="utf-8"))) - } - assert suppressed == set() - - @pytest.mark.lint class TestRuffExternalCoversEveryRule: """Every CE rule's documented `# noqa` must be accepted by ruff. @@ -4901,8 +4941,8 @@ def test_ignores_a_bare_union_ms(self): assert not self._run("cfg = TurnRecord(tool_union_ms_limit=0)") # Form 6 — the PLAIN assignment. Form 4 without the `is None` guard, or - # under a guard that tests something else. The live `_finalize_commands` - # defect was caught by form 4 only because it happened to spell its guard + # under a guard that tests something else. The live defect in the + # Claude adapter's former command finalizer was caught by form 4 only because it happened to spell its guard # `if cmd.duration_ms is None:`; written under the enclosing # `if cmd.result_status is None:` instead — which reads just as naturally # and books the identical lie — it was invisible to forms 1-5. @@ -4966,330 +5006,14 @@ def test_noqa_suppresses(self): assert not [v for v in check_file(path) if v.rule_id == "CE058"] -class TestCE059GenerationWindowIsTwoReads: - """CE059 flags a generation window built from one clock read.""" - - @staticmethod - def _run(src: str, filepath: str = "src/coder_eval/agents/antigravity_agent.py"): - import ast - - from tests.lint.rules.ce059_generation_window_is_two_reads import GenerationWindowIsTwoReads - - return GenerationWindowIsTwoReads(filepath).check(ast.parse(src)) - - def test_flags_the_same_name_for_both_bounds(self): - assert self._run("m = AssistantMessage(started_at=now, completed_at=now, generation_duration_ms=0.0)") - - def test_flags_an_omitted_duration_too(self): - # An omitted duration is not a disclaimer; the field defaults to None - # but the call has not said so, and the bounds still read as a window. - assert self._run("m = AssistantMessage(started_at=now, completed_at=now)") - - def test_flags_the_alias_spelling_too(self): - assert self._run("m = AssistantMessageTelemetry(started_at=now, completed_at=now, generation_duration_ms=g)") - - def test_allows_collapsed_bounds_when_the_call_says_no_window_exists(self): - # Passing None in the field built to say "unmeasurable" is the honest - # record, not a claim the two stamps have to support. - assert not self._run("m = AssistantMessage(started_at=now, completed_at=now, generation_duration_ms=None)") - - def test_allows_two_different_names(self): - assert not self._run("m = AssistantMessage(started_at=started, completed_at=completed)") - - def test_ignores_attribute_expressions(self): - # `self.a` vs `self.b` cannot be compared without guessing. - assert not self._run("m = AssistantMessage(started_at=self.mark, completed_at=self.mark)") - - def test_ignores_a_call_expression(self): - assert not self._run("m = AssistantMessage(started_at=datetime.now(), completed_at=datetime.now())") - - def test_ignores_an_unrelated_constructor(self): - assert not self._run("s = Span(started_at=now, completed_at=now)") - - def test_is_out_of_scope_outside_agents(self): - assert not self._run( - "m = AssistantMessage(started_at=now, completed_at=now)", - filepath="src/coder_eval/orchestrator.py", - ) - - def test_noqa_suppresses(self): - from tests.lint.runner import check_file - - # The Antigravity flush carries the only live instance, noqa'd until - # Phase 3 replaces it with a real measured window. - path = SRC / "coder_eval/agents/antigravity_agent.py" - assert path.is_file(), "the noqa fixture file must exist or this test passes vacuously" - assert not [v for v in check_file(path) if v.rule_id == "CE059"] - - -class TestCE060MessageIdDeclared: - """CE060 flags an assistant message built without an identity. - - Every source string carries its own import line: the rule derives its - constructor set from the module's own `coder_eval.models` imports, so a - bare `AssistantMessage(...)` with no import is correctly invisible to it. - """ - - _IMPORT = "from coder_eval.models import AssistantMessage\n" - - @staticmethod - def _run(src: str, filepath: str = "src/coder_eval/agents/antigravity_agent.py"): - import ast - - from tests.lint.rules.ce060_message_id_declared import MessageIdDeclared - - return MessageIdDeclared(filepath).check(ast.parse(src)) - - def test_flags_an_omitted_message_id(self): - assert self._run(self._IMPORT + "m = AssistantMessage(model=model, output_tokens=3)") - - def test_flags_an_explicit_none(self): - # Passing None is a claim that no id exists, which is never true for a - # harness that can synthesize one. - assert self._run(self._IMPORT + "m = AssistantMessage(model=model, message_id=None)") - - def test_flags_the_in_tree_alias_spelling(self): - assert self._run( - "from coder_eval.models import AssistantMessage as AssistantMessageTelemetry\n" - "m = AssistantMessageTelemetry(model=model)" - ) - - def test_flags_an_arbitrary_alias(self): - # The case a hardcoded name list misses entirely — the whole reason - # CE060 resolves aliases instead. - assert self._run("from coder_eval.models import AssistantMessage as Msg\nm = Msg(model=model)") - - def test_flags_the_module_alias_spelling(self): - # The realistic way to write `models.AssistantMessage(...)`: the class - # itself is never bound, so only the attribute is left to match on. - assert self._run("import coder_eval.models as models\nm = models.AssistantMessage(model=model)") - - def test_flags_the_attribute_spelling_beside_a_direct_import(self): - assert self._run(self._IMPORT + "m = models.AssistantMessage(model=model)") - - def test_flags_a_relative_import(self): - # `agents/` does use relative imports, and the absolute path test alone - # left the rule silently blind for a whole file. - assert self._run("from ..models import AssistantMessage\nm = AssistantMessage(model=model)") - - def test_keys_on_the_model_name_rather_than_a_literal(self): - # The constant moved into the shared resolver when CE061 was added; it - # is still derived from the model, which is the property under test. - from coder_eval.models import AssistantMessage as _Model - from tests.lint.rules import _model_ctor - - assert _Model.__name__ == _model_ctor.ASSISTANT_MESSAGE - - def test_flags_a_star_expanded_call(self): - # `**fields` has not declared the field at the site. - assert self._run(self._IMPORT + "m = AssistantMessage(**fields)") - - def test_allows_a_literal_id(self): - assert not self._run(self._IMPORT + 'm = AssistantMessage(message_id="x")') - - def test_allows_an_fstring_id(self): - assert not self._run(self._IMPORT + 'm = AssistantMessage(message_id=f"{turn_id}-msg-{i}")') - - def test_allows_a_fallback_expression(self): - # The runtime-None blind spot, exempted deliberately: passing a - # fallback expression IS deciding what the id is. - assert not self._run(self._IMPORT + "m = AssistantMessage(message_id=str(x) or None)") - - def test_allows_a_star_expanded_call_that_also_passes_the_field(self): - assert not self._run(self._IMPORT + "m = AssistantMessage(**fields, message_id=mid)") - - def test_ignores_an_unrelated_constructor(self): - assert not self._run(self._IMPORT + "s = Span(model=model)") - - def test_ignores_a_module_with_no_matching_import(self): - # Nothing is bound, so the rule claims nothing here. A construction - # site has to import the class to reach it. - assert not self._run("m = AssistantMessage(model=model)") - - def test_is_out_of_scope_outside_agents(self): - assert not self._run( - self._IMPORT + "m = AssistantMessage(model=model)", - filepath="src/coder_eval/orchestrator.py", - ) - - def test_the_real_antigravity_flush_declares_its_id(self): - from tests.lint.runner import check_file - - path = SRC / "coder_eval/agents/antigravity_agent.py" - assert path.is_file(), "the fixture file must exist or this test passes vacuously" - assert not [v for v in check_file(path) if v.rule_id == "CE060"] - - -class TestCE063NoBusyMsInAgents: - """CE063 flags a reducer that would subtract tool time itself. - - The subtraction lives once, in - `coder_eval.timing.subtract_tool_time`. A reducer that also - does it has its tool time taken out TWICE — once by itself, once by the - collector — which under-reports generation on that harness alone. - """ - - @staticmethod - def _run(src: str, filepath: str = "src/coder_eval/agents/pi_agent.py"): - import ast - - from tests.lint.rules.ce063_no_busy_ms_in_agents import NoBusyMsInAgents - - return NoBusyMsInAgents(filepath).check(ast.parse(src)) - - def test_flags_the_bare_name_import(self): - assert len(self._run("from coder_eval.timing import busy_ms")) == 1 - - def test_flags_it_under_an_alias(self): - # The import is what is banned, whatever it is bound to. - assert len(self._run("from coder_eval.timing import busy_ms as union")) == 1 - - def test_flags_it_alongside_an_allowed_import(self): - assert len(self._run("from coder_eval.timing import busy_ms, close_window")) == 1 - - def test_flags_a_relative_import(self): - # `agents/` uses relative imports; matching only the absolute path - # would leave the rule blind for a whole file. - assert len(self._run("from ..timing import busy_ms")) == 1 - - def test_flags_the_module_attribute_spelling(self): - assert len(self._run("from coder_eval import timing\nx = timing.busy_ms(s, lo, hi)")) == 1 - - def test_does_not_fire_on_close_window_through_the_module(self): - """The exact false positive a naive inversion of CE061's resolver gives. - - `_imports_the_helper` returns True for a bare module import so that - `timing.close_window(...)` counts as reaching the helper. Inverted into - a ban, that branch flags every reducer importing the module — which - after the migration is four of the five. - """ - assert not self._run("from coder_eval import timing\nx = timing.close_window(mark=m, now=n)") - - def test_does_not_fire_on_close_window_by_name(self): - assert not self._run("from coder_eval.timing import close_window\nx = close_window(mark=m, now=n)") - - def test_does_not_fire_on_an_unrelated_attribute_named_busy_ms(self): - # `self.busy_ms` is not `timing.busy_ms`; only the module spelling counts. - assert not self._run("x = self.busy_ms") - - def test_does_not_fire_outside_agents(self): - # The collector is where the subtraction belongs, so it must import it. - assert not self._run("from coder_eval.timing import busy_ms", filepath="src/coder_eval/streaming/collector.py") - - def test_is_suppressible(self, tmp_path): - from tests.lint.rules.ce063_no_busy_ms_in_agents import NoBusyMsInAgents - from tests.lint.runner import check_file - - agents = tmp_path / "src" / "coder_eval" / "agents" - agents.mkdir(parents=True) - target = agents / "pi_agent.py" - target.write_text("from coder_eval.timing import busy_ms # noqa: CE063\n", encoding="utf-8") - assert not check_file(target, [NoBusyMsInAgents]) - - -class TestCE064TurnBracketOnTheClock: - """CE064 flags a clocked reducer that lets its turn BRACKET default. - - `decompose_turn` subtracts a generation-window bound from an - AgentStart/AgentEnd timestamp. A harness that derives the first from a - `TurnClock` and lets the second fall back to `StreamEvent.timestamp`'s - `default_factory=datetime.now` puts two bases inside one subtraction. - Measured on antigravity: a tail of -0.017 ms, an agent end stamped 17 us - before its own last message finished, clamped to the `0.0` that means - "measured, and instant". - """ - - CLOCKED = "from coder_eval.timing import TurnClock\n" - START = "from coder_eval.streaming.events import AgentStartEvent\n" - END = "from coder_eval.streaming.events import AgentEndEvent\n" - - @staticmethod - def _run(src: str, filepath: str = "src/coder_eval/agents/pi_agent.py"): - import ast - - from tests.lint.rules.ce064_turn_bracket_on_the_clock import TurnBracketOnTheClock - - return TurnBracketOnTheClock(filepath).check(ast.parse(src)) - - def test_flags_a_defaulted_agent_start(self): - assert len(self._run(self.CLOCKED + self.START + "e = AgentStartEvent(task_id='t', prompt='p')")) == 1 - - def test_flags_a_defaulted_agent_end(self): - assert len(self._run(self.CLOCKED + self.END + "e = AgentEndEvent(task_id='t', status=s)")) == 1 - - def test_flags_both_brackets_in_one_module(self): - src = ( - self.CLOCKED + self.START + self.END + "a = AgentStartEvent(task_id='t')\nb = AgentEndEvent(task_id='t')\n" - ) - assert len(self._run(src)) == 2 - - def test_accepts_an_explicit_timestamp(self): - src = self.CLOCKED + self.START + "e = AgentStartEvent(task_id='t', timestamp=state.clock.now())" - assert not self._run(src) - - def test_accepts_it_through_any_clock_expression(self): - """Presence, not spelling — see the rule's BLIND SPOT note. - - Three harnesses reach their clock three ways (a `communicate` local, - `state.clock`, `self.clock`); pinning a spelling would make the rule a - syntax check on their internal structure. - """ - for expr in ("clock.now()", "self.clock.now()", "state.clock.now()"): - src = self.CLOCKED + self.END + f"e = AgentEndEvent(task_id='t', timestamp={expr})" - assert not self._run(src), expr - - def test_does_not_fire_on_an_unclocked_harness(self): - """Codex and OpenCode take their spans from the CLI's epoch stamps. - - They deliberately have no `TurnClock`, so a raw `datetime.now()` - bracket is CONSISTENT with their bounds. Firing here would push them - toward the mixed basis the rule exists to prevent. - """ - assert not self._run(self.START + "e = AgentStartEvent(task_id='t', prompt='p')") - - def test_starts_applying_the_day_an_unclocked_harness_adopts_one(self): - # Scope is derived from the import, never a hardcoded harness list. - src = self.START + "e = AgentStartEvent(task_id='t')" - assert not self._run(src, filepath="src/coder_eval/agents/codex_agent.py") - assert len(self._run(self.CLOCKED + src, filepath="src/coder_eval/agents/codex_agent.py")) == 1 - - def test_resolves_an_aliased_import(self): - src = self.CLOCKED + "from coder_eval.streaming.events import AgentEndEvent as Done\n" + "e = Done(task_id='t')" - assert len(self._run(src)) == 1 - - def test_resolves_an_aliased_clock_import(self): - """The scope side of the same question: `TurnClock as Clock` still clocks the module. - - A harness reaching its clock through an alias is still a clocked - harness; missing the binding would put it silently out of scope, which - is the half a hardcoded harness list would also get wrong. - """ - src = "from coder_eval.timing import TurnClock as Clock\n" + self.START + "e = AgentStartEvent(task_id='t')" - assert len(self._run(src)) == 1 - - def test_resolves_a_relative_import(self): - src = ( - "from ..timing import TurnClock\nfrom ..streaming.events import AgentStartEvent\ne = AgentStartEvent(t='t')" - ) - assert len(self._run(src)) == 1 - - def test_does_not_fire_outside_agents(self): - src = self.CLOCKED + self.START + "e = AgentStartEvent(task_id='t')" - assert not self._run(src, filepath="src/coder_eval/streaming/collector.py") - - def test_does_not_fire_on_an_unrelated_event(self): - src = self.CLOCKED + "from coder_eval.streaming.events import ToolEndEvent\ne = ToolEndEvent(task_id='t')" - assert not self._run(src) - - def test_is_suppressible(self, tmp_path): - from tests.lint.rules.ce064_turn_bracket_on_the_clock import TurnBracketOnTheClock - from tests.lint.runner import check_file - - agents = tmp_path / "src" / "coder_eval" / "agents" - agents.mkdir(parents=True) - target = agents / "pi_agent.py" - target.write_text( - self.CLOCKED + self.START + "e = AgentStartEvent(task_id='t') # noqa: CE064\n", - encoding="utf-8", - ) - assert not check_file(target, [TurnBracketOnTheClock]) +@pytest.mark.lint +class TestPyrightConfigCoversLiveTests: + def test_include_names_every_live_test_and_the_byoa_demo(self): + from tests.lint.pyright_config import REPO_ROOT, build_config + + include = build_config()["include"] + assert isinstance(include, list) + live = sorted(p.relative_to(REPO_ROOT).as_posix() for p in (REPO_ROOT / "tests").glob("*_live.py")) + assert live, "no tests/*_live.py found, so this test would pass vacuously" + missing = [p for p in [*live, "tests/fixtures/byoa_demo_plugin/byoa_demo.py"] if p not in include] + assert not missing, f"the second pyright pass skips {missing}" diff --git a/tests/test_detached_grading_guards.py b/tests/test_detached_grading_guards.py index f3939774..95e40e48 100644 --- a/tests/test_detached_grading_guards.py +++ b/tests/test_detached_grading_guards.py @@ -7,7 +7,6 @@ from __future__ import annotations -import os from datetime import datetime from pathlib import Path from unittest.mock import AsyncMock, MagicMock, patch @@ -208,90 +207,6 @@ def test_run_evaluation_has_real_defaults_not_typer_sentinels(tmp_path: Path) -> assert sig.parameters["preserve"].default is True -# -------------------------------------------------------------------------- -# The PATH round trip -# -------------------------------------------------------------------------- - - -def test_the_agents_path_is_persisted_so_a_later_grade_can_restore_it(tmp_path: Path) -> None: - """Without the persisted value a detached grade resolves `run_command` - binaries against ambient PATH and can disagree with the run it grades.""" - task = TaskDefinition( - task_id="t", - description="d", - initial_prompt="p", - agent=parse_agent_config(type=AgentKind.CLAUDE_CODE), - success_criteria=[FileExistsCriterion(path="x.txt", description="x")], - ) - orch = Orchestrator(task=task, run_dir=tmp_path, variant_id="v") - orch.result = EvaluationResult( - task_id="t", - task_description="d", - variant_id="v", - agent_type=AgentKind.CLAUDE_CODE, - started_at=datetime(2026, 1, 1), - final_status=FinalStatus.FAILURE, - iteration_count=0, - ) - orch.sandbox = MagicMock() - orch.agent = MagicMock() - orch.agent.get_sdk_options.return_value = {"env": {"PATH": f"{tmp_path}:/usr/bin"}} - - orch._sync_sandbox_command_path_with_agent() - - assert "command_base_path" in orch.result.environment_info - - -def test_a_restored_path_drops_entries_inside_the_graded_run(tmp_path: Path) -> None: - """The restored value is PREPENDED ahead of the host PATH and comes out of the - run's own task.json — a shareable artifact. Every entry an attacker could - have placed there must be dropped; only the run's real toolchain survives. - - The run-directory SIBLING case is the one this test used to pin the wrong way - round: it asserted such an entry was kept. The workspace is only part of the - run dir, and ``artifacts/`` and the run root travel in the same archive. - """ - run_dir = tmp_path / "run" - workspace = run_dir / "ws" - (workspace / "bin").mkdir(parents=True) - sibling = run_dir / "artifacts-shim" # inside the run dir, outside the workspace - sibling.mkdir() - toolchain = tmp_path / "toolchain" # a genuine location outside the run entirely - toolchain.mkdir() - relative = Path("evilbin") - - task = TaskDefinition( - task_id="t", - description="d", - initial_prompt="p", - agent=parse_agent_config(type=AgentKind.CLAUDE_CODE), - success_criteria=[FileExistsCriterion(path="x.txt", description="x")], - ) - orch = Orchestrator(task=task, run_dir=run_dir, variant_id="v") - orch.sandbox = MagicMock() - orch.sandbox.sandbox_dir = workspace - - # os.pathsep, not a hardcoded ":" — the separator is ";" on Windows, where a - # colon-joined value parses as one (non-existent) entry and every assertion - # below passes vacuously against an empty result. - recorded = os.pathsep.join( - [ - str(workspace / "bin"), - str(sibling), - str(relative), - str(toolchain), - str(tmp_path / "gone"), - ] - ) - kept = orch._sanitize_restored_path(recorded) - - assert str(toolchain.resolve()) in kept, "a real out-of-run toolchain entry is the point of the restore" - assert str(workspace) not in kept, "an entry inside the graded workspace must be dropped" - assert str(sibling) not in kept, "an entry elsewhere in the run directory must be dropped too" - assert "evilbin" not in kept, "a relative entry would resolve against the grader's cwd" - assert "gone" not in kept, "a non-existent entry buys no parity" - - # -------------------------------------------------------------------------- # The LiteLLM cost join # -------------------------------------------------------------------------- diff --git a/tests/test_early_stop.py b/tests/test_early_stop.py index 869c3585..1673b3a9 100644 --- a/tests/test_early_stop.py +++ b/tests/test_early_stop.py @@ -36,14 +36,13 @@ from coder_eval.agents.antigravity_agent import AntigravityAgent from coder_eval.agents.claude_code_agent import ClaudeCodeAgent -from coder_eval.agents.codex_agent import CodexAgent, _CodexTurnState -from coder_eval.agents.registry import AgentRegistry +from coder_eval.agents.codex_agent import CodexAgent, _CodexDecoder +from coder_eval.agents.registry import SPI_VERSION, AgentRegistry from coder_eval.cli.plan_command import run_plan from coder_eval.config import settings from coder_eval.criteria import CriterionRegistry, init_criteria from coder_eval.criteria.command_executed import CommandExecutedChecker from coder_eval.criteria.skill_triggered import SkillTriggeredChecker, _engaged_skill_names -from coder_eval.errors import AgentCrashError, TurnTimeoutError from coder_eval.models import ( AgentKind, ApiBackend, @@ -81,6 +80,7 @@ from coder_eval.reports import ReportGenerator from coder_eval.reports.html import _render_criteria, _render_header from coder_eval.run_record import eval_result_to_task_dict +from coder_eval.streaming.emitter import TurnOutcome from coder_eval.streaming.events import ( AgentEndEvent, AgentEndStatus, @@ -91,6 +91,7 @@ ToolStartEvent, TurnEndStatus, TurnStartEvent, + end_status_for, ) from tests._fixtures.live_criteria import FROZEN_TS, make_command, make_turn from tests.fixtures.harness_stubs import config_for_kind, stub_contract @@ -161,7 +162,7 @@ class _DummyNoStopAgent: def dummy_no_stop_kind() -> Iterator[str]: """Register a non-supporting agent kind for guardrail-1 tests, then clean up.""" kind = "dummy-no-stop" - AgentRegistry.register(kind, config_for_kind(kind))(_DummyNoStopAgent) + AgentRegistry.register(kind, config_for_kind(kind), spi_version=SPI_VERSION)(_DummyNoStopAgent) try: yield kind finally: @@ -810,7 +811,7 @@ def test_guardrail3_unregistered_agent_type_rejected(self) -> None: # An armed task whose agent type vanished from the registry (plugin not # installed/loaded) must fail with the plugin-pointing diagnosis. kind = "vanishing-agent" - AgentRegistry.register(kind, config_for_kind(kind))(_DummyNoStopAgent) + AgentRegistry.register(kind, config_for_kind(kind), spi_version=SPI_VERSION)(_DummyNoStopAgent) try: task = _task(criteria=[_skill_crit("s", "s", stop_on_pass=True)], agent_type=kind) finally: @@ -1132,14 +1133,14 @@ def on_event(self, event: Any) -> None: async def _run_claude_communicate( *, stop_after: int | None = None, never: bool = False, n_messages: int = 3 -) -> tuple[ClaudeCodeAgent, TurnRecord, _EventSink, int]: +) -> tuple[ClaudeCodeAgent, TurnOutcome, _EventSink, int]: """Drive ``ClaudeCodeAgent.communicate`` over a mocked ``query`` yielding ``n_messages`` dummy messages. ``stop_after``: build a should_stop that returns ``EARLY_CRITERION`` once that many messages have been pulled (checked after each dispatch). ``never``: pass an explicit always-None should_stop. Neither: pass ``should_stop=None``. Returns - ``(agent, record, sink, pulled_count)``. + ``(agent, outcome, sink, pulled_count)``. """ config = parse_agent_config(type=AgentKind.CLAUDE_CODE, permission_mode="acceptEdits") agent = ClaudeCodeAgent(config) @@ -1169,8 +1170,8 @@ async def mock_query(prompt: Any, options: Any, transport: Any = None) -> Any: sink = _EventSink() with patch("coder_eval.agents.claude_code_agent.query", mock_query): - record = await agent.communicate("prompt", stream_callback=sink, should_stop=should_stop) - return agent, record, sink, pulled["n"] + outcome = await agent.communicate("prompt", iteration=1, stream_callback=sink, should_stop=should_stop) + return agent, outcome, sink, pulled["n"] def _agent_end_events(sink: _EventSink) -> list[AgentEndEvent]: @@ -1180,6 +1181,8 @@ def _agent_end_events(sink: _EventSink) -> list[AgentEndEvent]: class _NoopWatchdog: """No-op stand-in for ThreadedWatchdog so only the in-loop deadline guard fires.""" + fired = False + def __init__(self, *args: Any, **kwargs: Any) -> None: pass @@ -1190,14 +1193,13 @@ def __exit__(self, *args: Any) -> bool: return False -async def _run_claude_communicate_timeout() -> tuple[ClaudeCodeAgent, _EventSink, BaseException | None]: +async def _run_claude_communicate_timeout() -> tuple[ClaudeCodeAgent, _EventSink, TurnOutcome]: """Drive ``communicate`` with a slow query (50ms) against a 10ms deadline AND a should_stop returning ``EARLY_CRITERION`` — the deadline guard must win. Returns - ``(agent, sink, raised_exception)``.""" + ``(agent, sink, outcome)``.""" config = parse_agent_config(type=AgentKind.CLAUDE_CODE, permission_mode="acceptEdits") agent = ClaudeCodeAgent(config) sink = _EventSink() - raised: BaseException | None = None with tempfile.TemporaryDirectory() as tmpdir: await agent.start(tmpdir) @@ -1207,15 +1209,12 @@ async def slow_query(prompt: Any, options: Any, transport: Any = None) -> Any: with ( patch("coder_eval.agents.claude_code_agent.query", slow_query), - patch("coder_eval.agents.claude_code_agent.ThreadedWatchdog", _NoopWatchdog), + patch("coder_eval.agents.watchdog.ThreadedWatchdog", _NoopWatchdog), ): - try: - await agent.communicate( - "p", stream_callback=sink, timeout=0.01, should_stop=lambda: StopReason.EARLY_CRITERION - ) - except TurnTimeoutError as exc: - raised = exc - return agent, sink, raised + outcome = await agent.communicate( + "p", iteration=1, stream_callback=sink, timeout=0.01, should_stop=lambda: StopReason.EARLY_CRITERION + ) + return agent, sink, outcome class TestNewFixtureTasksResolve: @@ -1252,31 +1251,31 @@ def test_turnendstatus_conversion_from_agentendstatus(self) -> None: assert TurnEndStatus(AgentEndStatus.STOPPED_EARLY.value) == TurnEndStatus.STOPPED_EARLY async def test_stop_after_first_dispatched_message(self) -> None: - _agent, record, sink, pulled = await _run_claude_communicate(stop_after=1, n_messages=3) + _agent, outcome, sink, pulled = await _run_claude_communicate(stop_after=1, n_messages=3) # The deciding message is kept; the next is never pulled. assert pulled == 1 - assert record.crashed is False + assert outcome.record.crashed is False ends = _agent_end_events(sink) assert len(ends) == 1 assert ends[0].status == AgentEndStatus.STOPPED_EARLY assert ends[0].crashed is False async def test_early_stop_is_clean_not_crashed(self) -> None: - agent, record, _sink, _pulled = await _run_claude_communicate(stop_after=1) - # A clean stop: no partial pending_turn, no ERROR state, no raise (we got here). - assert agent.pending_turn is None + agent, outcome, _sink, _pulled = await _run_claude_communicate(stop_after=1) + # A clean stop: no CRASHED/TIMEOUT status, no ERROR state. + assert outcome.status is AgentEndStatus.STOPPED_EARLY assert agent.get_state().value != "error" - assert record.crashed is False + assert outcome.record.crashed is False async def test_should_stop_none_consumes_full_stream(self) -> None: - _agent, _record, sink, pulled = await _run_claude_communicate(stop_after=None, n_messages=3) + _agent, _outcome, sink, pulled = await _run_claude_communicate(stop_after=None, n_messages=3) assert pulled == 3 ends = _agent_end_events(sink) assert len(ends) == 1 assert ends[0].status == AgentEndStatus.COMPLETED async def test_should_stop_returning_none_consumes_full_stream(self) -> None: - _agent, _record, sink, pulled = await _run_claude_communicate(never=True, n_messages=3) + _agent, _outcome, sink, pulled = await _run_claude_communicate(never=True, n_messages=3) assert pulled == 3 assert _agent_end_events(sink)[0].status == AgentEndStatus.COMPLETED @@ -1284,14 +1283,14 @@ async def test_timeout_beats_stop_precedence(self) -> None: # Both signals live in one turn: a deadline breach AND should_stop=True. # The top-of-loop deadline guard returns BEFORE dispatch, so the stop # check is never reached — TIMEOUT wins over the pending stop. - agent, sink, raised = await _run_claude_communicate_timeout() - assert isinstance(raised, TurnTimeoutError) + _agent, sink, outcome = await _run_claude_communicate_timeout() + assert outcome.status is AgentEndStatus.TIMEOUT ends = _agent_end_events(sink) assert len(ends) == 1 assert ends[0].status == AgentEndStatus.TIMEOUT assert ends[0].crashed is True # The crashed partial is preserved for the orchestrator to drain. - assert agent.pending_turn is not None and agent.pending_turn.crashed is True + assert outcome.record.crashed is True # STOPPED_EARLY must NOT appear — the stop lost the race. assert AgentEndStatus.STOPPED_EARLY not in {e.status for e in ends} @@ -2276,12 +2275,12 @@ def test_decision_budget_accumulates_across_retry_attempts(self) -> None: class _ScriptedAgent: """Duck-typed agent: replays scripted events through the callback, polling ``should_stop`` after each and breaking on a reason (mirrors the real - message-boundary cut). Returns a fixed ``TurnRecord``.""" + message-boundary cut). Returns a fixed ``TurnRecord`` wrapped in a + ``TurnOutcome`` whose status reflects whether ``should_stop`` fired.""" def __init__(self, events: list[Any], turn: TurnRecord) -> None: self._events = events self._turn = turn - self.pending_turn: TurnRecord | None = None self.delivered = 0 def get_sdk_options(self) -> dict[str, Any] | None: @@ -2291,17 +2290,22 @@ async def communicate( self, prompt: str, *, + iteration: int, stream_callback: Any = None, timeout: float | None = None, should_stop: Callable[[], StopReason | None] | None = None, - ) -> TurnRecord: + ) -> TurnOutcome: + reason: StopReason | None = None for event in self._events: if stream_callback is not None: stream_callback.on_event(event) self.delivered += 1 - if should_stop is not None and should_stop(): - break - return self._turn + if should_stop is not None: + reason = should_stop() + if reason is not None: + break + status = end_status_for(reason) if reason is not None else AgentEndStatus.COMPLETED + return TurnOutcome(record=self._turn, status=status, error=None) async def _run_wiring( @@ -2893,7 +2897,7 @@ def _codex_completed() -> SimpleNamespace: def _stub_on_turn_completed(self: Any, notification: Any) -> bool: - """Stand-in for ``_CodexTurnState.on_turn_completed`` (the real one isinstance- + """Stand-in for ``_CodexDecoder.on_turn_completed`` (the real one isinstance- checks an openai_codex type). Sets the terminal turn and breaks the pump.""" self.turn_result = notification.payload return True @@ -2912,7 +2916,7 @@ async def _run_codex_communicate( stop_after: int | None = None, never: bool = False, timeout: float | None = None, -) -> tuple[CodexAgent, TurnRecord, _EventSink, _FakeCodexStream, _FakeCodexTurnHandle]: +) -> tuple[CodexAgent, TurnOutcome, _EventSink, _FakeCodexStream, _FakeCodexTurnHandle]: """Drive ``CodexAgent.communicate`` over a fake notification stream. ``stop_after``: should_stop returns ``EARLY_CRITERION`` once that many @@ -2933,39 +2937,41 @@ async def _run_codex_communicate( should_stop = None sink = _EventSink() - with patch.object(_CodexTurnState, "on_turn_completed", _stub_on_turn_completed): - record = await agent.communicate("prompt", stream_callback=sink, timeout=timeout, should_stop=should_stop) - return agent, record, sink, stream, handle + with patch.object(_CodexDecoder, "on_turn_completed", _stub_on_turn_completed): + outcome = await agent.communicate( + "prompt", iteration=1, stream_callback=sink, timeout=timeout, should_stop=should_stop + ) + return agent, outcome, sink, stream, handle class TestCodexCooperativeStopSeam: async def test_stop_after_first_dispatched_notification(self) -> None: notifications = [_codex_delta(0), _codex_delta(1), _codex_delta(2), _codex_completed()] - agent, record, sink, stream, handle = await _run_codex_communicate(notifications=notifications, stop_after=1) + agent, outcome, sink, stream, handle = await _run_codex_communicate(notifications=notifications, stop_after=1) # The deciding notification is kept; the next is never pulled. assert stream.iter.pulled == 1 # The in-flight turn was interrupted exactly once (server-side spend cut). assert handle.interrupts == 1 - assert record.crashed is False + assert outcome.record.crashed is False ends = _agent_end_events(sink) assert len(ends) == 1 assert ends[0].status == AgentEndStatus.STOPPED_EARLY assert ends[0].crashed is False - # A clean stop: no partial pending_turn, no ERROR state, no raise. - assert agent.pending_turn is None + # A clean stop: no CRASHED/TIMEOUT status, no ERROR state. + assert outcome.status is AgentEndStatus.STOPPED_EARLY assert agent.get_state().value != "error" async def test_should_stop_none_consumes_full_stream(self) -> None: notifications = [_codex_delta(0), _codex_delta(1), _codex_completed()] - _agent, record, sink, stream, handle = await _run_codex_communicate(notifications=notifications) + _agent, outcome, sink, stream, handle = await _run_codex_communicate(notifications=notifications) assert stream.iter.pulled == 3 assert handle.interrupts == 0 - assert record.crashed is False + assert outcome.record.crashed is False assert _agent_end_events(sink)[0].status == AgentEndStatus.COMPLETED async def test_should_stop_returning_none_consumes_full_stream(self) -> None: notifications = [_codex_delta(0), _codex_delta(1), _codex_completed()] - _agent, _record, sink, stream, _handle = await _run_codex_communicate(notifications=notifications, never=True) + _agent, _outcome, sink, stream, _handle = await _run_codex_communicate(notifications=notifications, never=True) assert stream.iter.pulled == 3 assert _agent_end_events(sink)[0].status == AgentEndStatus.COMPLETED @@ -2973,88 +2979,92 @@ async def test_stop_before_turn_completed_does_not_raise(self) -> None: # The stream is cut before any turn/completed: turn_result is None, but the # stop makes the "turn never completed" raise conditional — no crash. notifications = [_codex_delta(0), _codex_delta(1), _codex_delta(2)] - _agent, record, sink, _stream, _handle = await _run_codex_communicate(notifications=notifications, stop_after=1) - assert record.crashed is False + _agent, outcome, sink, _stream, _handle = await _run_codex_communicate( + notifications=notifications, stop_after=1 + ) + assert outcome.record.crashed is False assert _agent_end_events(sink)[0].status == AgentEndStatus.STOPPED_EARLY - async def test_stream_dying_without_stop_still_raises(self) -> None: + async def test_stream_dying_without_stop_still_crashes(self) -> None: # Regression guard: a stream that ends with NO turn/completed and NO stop # is still a crash (the RuntimeError survives for genuine stream deaths). agent = _codex_agent() stream = _FakeCodexStream([_codex_delta(0)]) agent.thread = SimpleNamespace(turn=lambda _prompt: _FakeCodexTurnHandle(stream)) - with pytest.raises(AgentCrashError, match="did not complete"): - await agent.communicate("prompt", stream_callback=_EventSink(), should_stop=None) - assert agent.pending_turn is not None - assert agent.pending_turn.crashed is True - await agent.discard_pending_turn() + outcome = await agent.communicate("prompt", iteration=1, stream_callback=_EventSink(), should_stop=None) + assert outcome.status is AgentEndStatus.CRASHED + assert "did not complete" in (outcome.error or "") + assert outcome.record.crashed is True async def test_timeout_beats_stop_precedence(self, monkeypatch: pytest.MonkeyPatch) -> None: # Both signals in one turn: the watchdog fires (timeout_hit) AND should_stop # returns a reason. The post-pump timeout check must win — TIMEOUT, crashed=True. class _FiringWatchdog: + fired = True + def __init__(self, *, on_timeout: Callable[[], None], **_kwargs: Any) -> None: self._on_timeout = on_timeout def __enter__(self) -> _FiringWatchdog: - self._on_timeout() # watchdog fired: state.timeout_hit = True + self._on_timeout() # watchdog fired: decoder.timeout_hit = True return self def __exit__(self, *_exc: Any) -> bool: return False - monkeypatch.setattr("coder_eval.agents.codex_agent.ThreadedWatchdog", _FiringWatchdog) + monkeypatch.setattr("coder_eval.agents.watchdog.ThreadedWatchdog", _FiringWatchdog) agent = _codex_agent() stream = _FakeCodexStream([_codex_delta(0), _codex_delta(1)]) agent.thread = SimpleNamespace(turn=lambda _prompt: _FakeCodexTurnHandle(stream)) sink = _EventSink() - with pytest.raises(TurnTimeoutError): - await agent.communicate( - "prompt", stream_callback=sink, timeout=30.0, should_stop=lambda: StopReason.EARLY_CRITERION - ) + outcome = await agent.communicate( + "prompt", iteration=1, stream_callback=sink, timeout=30.0, should_stop=lambda: StopReason.EARLY_CRITERION + ) + assert outcome.status is AgentEndStatus.TIMEOUT ends = _agent_end_events(sink) assert len(ends) == 1 assert ends[0].status == AgentEndStatus.TIMEOUT assert ends[0].crashed is True - assert agent.pending_turn is not None and agent.pending_turn.crashed is True + assert outcome.record.crashed is True # STOPPED_EARLY must NOT appear — the stop lost the race. assert AgentEndStatus.STOPPED_EARLY not in {e.status for e in ends} - await agent.discard_pending_turn() async def test_post_stop_exception_stays_clean(self, monkeypatch: pytest.MonkeyPatch) -> None: # The retry-poisoning gap: an exception AFTER the cooperative break (here: - # the pump's finally-side cleanup) must NOT crash-finalize the turn — a + # the pump's finally-side trailing flush) must NOT crash the turn — a # crash would trigger the orchestrator retry with the monitor's decision # still latched, stopping the retry at turn 0. - def _boom(self: Any) -> None: + def _boom(self: Any, _last: Any) -> None: raise RuntimeError("post-stop cleanup boom") - monkeypatch.setattr(_CodexTurnState, "close_open_tools", _boom) + monkeypatch.setattr(_CodexDecoder, "flush", _boom) notifications = [_codex_delta(0), _codex_delta(1)] - agent, record, sink, _stream, _handle = await _run_codex_communicate(notifications=notifications, stop_after=1) - # No AgentCrashError raised (we got a record back), clean STOPPED_EARLY. - assert record.crashed is False + _agent, outcome, sink, _stream, _handle = await _run_codex_communicate( + notifications=notifications, stop_after=1 + ) + # No crash outcome (we got a clean record back), clean STOPPED_EARLY. + assert outcome.record.crashed is False ends = _agent_end_events(sink) assert len(ends) == 1 assert ends[0].status == AgentEndStatus.STOPPED_EARLY assert ends[0].crashed is False - assert agent.pending_turn is None + assert outcome.status is AgentEndStatus.STOPPED_EARLY async def test_post_stop_cleanup_exception_without_stop_still_crashes( self, monkeypatch: pytest.MonkeyPatch ) -> None: # The guard is scoped to stopped turns only: the same cleanup exception on # a NON-stopped turn keeps crashing (no behavior change for real failures). - def _boom(self: Any) -> None: + def _boom(self: Any, _last: Any) -> None: raise RuntimeError("cleanup boom") - monkeypatch.setattr(_CodexTurnState, "close_open_tools", _boom) + monkeypatch.setattr(_CodexDecoder, "flush", _boom) agent = _codex_agent() stream = _FakeCodexStream([_codex_delta(0)]) agent.thread = SimpleNamespace(turn=lambda _prompt: _FakeCodexTurnHandle(stream)) - with pytest.raises(AgentCrashError): - await agent.communicate("prompt", stream_callback=_EventSink(), should_stop=None) - await agent.discard_pending_turn() + outcome = await agent.communicate("prompt", iteration=1, stream_callback=_EventSink(), should_stop=None) + assert outcome.status is AgentEndStatus.CRASHED + assert outcome.record.crashed is True async def test_stopped_turn_skips_subagent_recovery(self) -> None: # A stopped turn must not attempt rollout recovery: children may have no @@ -3065,23 +3075,24 @@ async def test_stopped_turn_skips_subagent_recovery(self) -> None: recover = AsyncMock() captured: dict[str, Any] = {} - original_init = _CodexTurnState.__init__ + original_init = _CodexDecoder.__init__ def _capturing_init(self: Any, *args: Any, **kwargs: Any) -> None: original_init(self, *args, **kwargs) self.spawned_children = [("child-thread", "tool-1", None)] - captured["state"] = self + captured["decoder"] = self with ( - patch.object(_CodexTurnState, "__init__", _capturing_init), + patch.object(_CodexDecoder, "__init__", _capturing_init), patch.object(CodexAgent, "_recover_subagent_tool_calls", recover), ): await agent.communicate( "prompt", + iteration=1, stream_callback=_EventSink(), should_stop=lambda: StopReason.EARLY_CRITERION if stream.iter.pulled >= 1 else None, ) - assert captured["state"].stop_reason is StopReason.EARLY_CRITERION + assert captured["decoder"].stop_reason is StopReason.EARLY_CRITERION recover.assert_not_awaited() @@ -3145,7 +3156,7 @@ async def _run_antigravity_communicate( stop_after: int | None = None, never: bool = False, cancel_raises: bool = False, -) -> tuple[AntigravityAgent, TurnRecord, _EventSink, _CountingConversation]: +) -> tuple[AntigravityAgent, TurnOutcome, _EventSink, _CountingConversation]: """Drive ``AntigravityAgent.communicate`` over a fake step stream (same stop_after / never / None semantics as the Claude and Codex drivers).""" conversation = _CountingConversation([_ag_step(i) for i in range(n_steps)], cancel_raises=cancel_raises) @@ -3160,45 +3171,45 @@ async def _run_antigravity_communicate( should_stop = None sink = _EventSink() - record = await agent.communicate("prompt", stream_callback=sink, should_stop=should_stop) - return agent, record, sink, conversation + outcome = await agent.communicate("prompt", iteration=1, stream_callback=sink, should_stop=should_stop) + return agent, outcome, sink, conversation class TestAntigravityCooperativeStopSeam: async def test_stop_after_first_processed_step(self) -> None: - agent, record, sink, conversation = await _run_antigravity_communicate(stop_after=1, n_steps=3) + agent, outcome, sink, conversation = await _run_antigravity_communicate(stop_after=1, n_steps=3) # The deciding step is kept; the next is never pulled. assert conversation.yielded == 1 # The conversation was cancelled once (best-effort server-side cut). assert conversation.cancels == 1 - assert record.crashed is False + assert outcome.record.crashed is False ends = _agent_end_events(sink) assert len(ends) == 1 assert ends[0].status == AgentEndStatus.STOPPED_EARLY assert ends[0].crashed is False - assert agent.pending_turn is None + assert outcome.status is AgentEndStatus.STOPPED_EARLY assert agent.get_state().value != "error" async def test_should_stop_none_consumes_full_stream(self) -> None: - _agent, record, sink, conversation = await _run_antigravity_communicate(n_steps=3) + _agent, outcome, sink, conversation = await _run_antigravity_communicate(n_steps=3) assert conversation.yielded == 3 assert conversation.cancels == 0 - assert record.crashed is False + assert outcome.record.crashed is False assert _agent_end_events(sink)[0].status == AgentEndStatus.COMPLETED async def test_should_stop_returning_none_consumes_full_stream(self) -> None: - _agent, _record, sink, conversation = await _run_antigravity_communicate(never=True, n_steps=3) + _agent, _outcome, sink, conversation = await _run_antigravity_communicate(never=True, n_steps=3) assert conversation.yielded == 3 assert _agent_end_events(sink)[0].status == AgentEndStatus.COMPLETED async def test_raising_cancel_still_stops_clean(self) -> None: # conversation.cancel() is best-effort: a raising cancel must not escalate # a stopped turn to a crash. - agent, record, sink, conversation = await _run_antigravity_communicate(stop_after=1, cancel_raises=True) + _agent, outcome, sink, conversation = await _run_antigravity_communicate(stop_after=1, cancel_raises=True) assert conversation.cancels == 1 - assert record.crashed is False + assert outcome.record.crashed is False assert _agent_end_events(sink)[0].status == AgentEndStatus.STOPPED_EARLY - assert agent.pending_turn is None + assert outcome.status is AgentEndStatus.STOPPED_EARLY async def test_timeout_beats_stop_precedence(self, monkeypatch: pytest.MonkeyPatch) -> None: class _FiringWatchdog: @@ -3212,21 +3223,20 @@ def __enter__(self) -> _FiringWatchdog: def __exit__(self, *_exc: Any) -> bool: return False - monkeypatch.setattr("coder_eval.agents.antigravity_agent.ThreadedWatchdog", _FiringWatchdog) + monkeypatch.setattr("coder_eval.agents.watchdog.ThreadedWatchdog", _FiringWatchdog) conversation = _CountingConversation([_ag_step(0), _ag_step(1)]) agent = _antigravity_agent(conversation) sink = _EventSink() - with pytest.raises(TurnTimeoutError): - await agent.communicate( - "prompt", stream_callback=sink, timeout=30.0, should_stop=lambda: StopReason.EARLY_CRITERION - ) + outcome = await agent.communicate( + "prompt", iteration=1, stream_callback=sink, timeout=30.0, should_stop=lambda: StopReason.EARLY_CRITERION + ) + assert outcome.status is AgentEndStatus.TIMEOUT ends = _agent_end_events(sink) assert len(ends) == 1 assert ends[0].status == AgentEndStatus.TIMEOUT assert ends[0].crashed is True - assert agent.pending_turn is not None and agent.pending_turn.crashed is True + assert outcome.record.crashed is True assert AgentEndStatus.STOPPED_EARLY not in {e.status for e in ends} - await agent.discard_pending_turn() async def test_post_stop_exception_stays_clean(self, monkeypatch: pytest.MonkeyPatch) -> None: # The retry-poisoning gap, antigravity flavor: an exception raised by @@ -3245,14 +3255,14 @@ def __exit__(self, exc_type: Any, *_exc: Any) -> bool: raise RuntimeError("post-stop cleanup boom") return False - monkeypatch.setattr("coder_eval.agents.antigravity_agent.ThreadedWatchdog", _ExplodingExitWatchdog) - agent, record, sink, _conversation = await _run_antigravity_communicate(stop_after=1) - assert record.crashed is False + monkeypatch.setattr("coder_eval.agents.watchdog.ThreadedWatchdog", _ExplodingExitWatchdog) + _agent, outcome, sink, _conversation = await _run_antigravity_communicate(stop_after=1) + assert outcome.record.crashed is False ends = _agent_end_events(sink) assert len(ends) == 1 assert ends[0].status == AgentEndStatus.STOPPED_EARLY assert ends[0].crashed is False - assert agent.pending_turn is None + assert outcome.status is AgentEndStatus.STOPPED_EARLY async def test_post_stop_cleanup_exception_without_stop_still_crashes( self, monkeypatch: pytest.MonkeyPatch @@ -3271,12 +3281,12 @@ def __exit__(self, exc_type: Any, *_exc: Any) -> bool: raise RuntimeError("cleanup boom") return False - monkeypatch.setattr("coder_eval.agents.antigravity_agent.ThreadedWatchdog", _ExplodingExitWatchdog) + monkeypatch.setattr("coder_eval.agents.watchdog.ThreadedWatchdog", _ExplodingExitWatchdog) conversation = _CountingConversation([_ag_step(0)]) agent = _antigravity_agent(conversation) - with pytest.raises(AgentCrashError): - await agent.communicate("prompt", stream_callback=_EventSink(), should_stop=None) - await agent.discard_pending_turn() + outcome = await agent.communicate("prompt", iteration=1, stream_callback=_EventSink(), should_stop=None) + assert outcome.status is AgentEndStatus.CRASHED + assert outcome.record.crashed is True # --------------------------------------------------------------------------- # diff --git a/tests/test_event_collector.py b/tests/test_event_collector.py index b51609e4..b0693667 100644 --- a/tests/test_event_collector.py +++ b/tests/test_event_collector.py @@ -102,47 +102,101 @@ def test_zero_tokens_with_cost_is_kept(self): assert record.token_usage.total_cost_usd == 0.0 -class TestSubAgentEventFiltering: - """Events with parent_thread_id set are ignored (collector.py ~61).""" +class TestSubAgentEvents: + """A nested event (parent_thread_id set) records its tool call and shapes nothing else.""" - def test_sub_agent_events_do_not_affect_record(self): + def test_nested_start_and_end_do_not_affect_record_but_the_nested_tool_is_a_command(self): collector = EventCollector() _feed( collector, [ AgentStartEvent(task_id=TASK_ID, prompt="main prompt", iteration=2), - # A nested sub-agent's events (parent_thread_id set) must be skipped. AgentStartEvent( task_id=TASK_ID, prompt="child prompt", iteration=99, thread_id="tool_x", - parent_thread_id="main", + parent_thread_id="tool_x", ), ToolEndEvent( task_id=TASK_ID, tool=_tool("child_tool", 0), thread_id="tool_x", - parent_thread_id="main", + parent_thread_id="tool_x", ), + AgentEndEvent(task_id=TASK_ID, iteration=99, thread_id="tool_x", parent_thread_id="tool_x"), AgentEndEvent( task_id=TASK_ID, iteration=2, user_input="main prompt", agent_output="main out", usage=TokenUsage(output_tokens=10), - parent_thread_id=None, ), ], ) record = collector.build_turn_record() - # The child AgentStart did not overwrite iteration/user_input. assert record.iteration == 2 assert record.user_input == "main prompt" assert record.agent_output == "main out" - # The child ToolEnd contributed no command. - assert record.commands == [] + assert [c.tool_id for c in record.commands] == ["child_tool"] + + def test_a_nested_turn_start_sets_no_model_and_counts_no_turn(self): + collector = EventCollector() + _feed( + collector, + [ + AgentStartEvent(task_id=TASK_ID, model="main-model"), + TurnStartEvent(task_id=TASK_ID, model="sub-model", thread_id="t1", parent_thread_id="t1"), + ], + ) + record = collector.build_turn_record() + assert record.model_used == "main-model" + assert record.assistant_turn_count == 0 + + def test_ended_tracks_the_current_attempt(self): + collector = EventCollector() + assert not collector.ended + _feed(collector, [AgentStartEvent(task_id=TASK_ID), AgentEndEvent(task_id=TASK_ID, parent_thread_id="x")]) + assert not collector.ended + collector.on_event(AgentEndEvent(task_id=TASK_ID)) + assert collector.ended + collector.on_event(AgentStartEvent(task_id=TASK_ID)) + assert not collector.ended + + +class TestAssistantTurnIndexIsDerived: + """``assistant_turn_index`` is the owning AssistantMessage's position, computed by the collector.""" + + @staticmethod + def _message(*tool_ids: str) -> AssistantMessage: + now = datetime(2026, 1, 1) + return AssistantMessage(started_at=now, completed_at=now, tool_use_ids=list(tool_ids)) + + def test_the_index_counts_assistant_messages_only(self): + from coder_eval.models import UserMessage + + collector = EventCollector() + tool_a, tool_b, orphan = _tool("a", 0), _tool("b", 1), _tool("orphan", 2) + tool_a.assistant_turn_index = 7 + messages = [ + self._message("a"), + UserMessage(text="hi"), + ReconciliationMessage(), + self._message(), + self._message("b"), + ] + _feed( + collector, + [ + AgentStartEvent(task_id=TASK_ID), + *(ToolEndEvent(task_id=TASK_ID, tool=t) for t in (tool_a, tool_b, orphan)), + AgentEndEvent(task_id=TASK_ID, messages=messages), + ], + ) + record = collector.build_turn_record() + assert [(c.tool_id, c.assistant_turn_index) for c in record.commands] == [("a", 0), ("b", 2), ("orphan", None)] + assert tool_a.assistant_turn_index == 7, "the event's own telemetry is never mutated" class TestToolReduction: @@ -900,10 +954,9 @@ class TestAPublishedWindowMustMatchItsOwnBounds: That equality is what lets `generation_duration_ms` stay a PUBLISHED field instead of one the collector derives from the bounds — the migration that was considered and cut, on the grounds that this check makes deferring it - safe. It is largely true by construction (CE061 forces every reducer - through `timing.close_window`); what it catches is a reducer that bypasses - the helper, and a third-party agent registered through the - `coder_eval.plugins` SPI, which no lint rule scoped to `agents/` can see. + safe. It is largely true by construction (`TurnEmitter.add_generation` takes a + `Window`); what it catches is an in-tree writer that bypasses the emitter, and + a third-party agent registered through the `coder_eval.plugins` SPI, which no lint rule scoped to `agents/` can see. """ BASE: ClassVar[datetime] = datetime(2026, 9, 11, 9, 0, 0) @@ -932,7 +985,7 @@ def test_a_window_that_matches_its_bounds_passes(self): assert out[0].generation_duration_ms == pytest.approx(1000.0) def test_codexs_split_passes_when_the_parts_sum_to_the_window(self): - """Built with `_flush_message`'s own idiom, not a hand-picked pair. + """Built with `flush`'s own idiom, not a hand-picked pair. Codex divides one window across two sub-messages by output-token share, rounding every share but the last to 6 places and giving the last the diff --git a/tests/test_execute_evaluate_loop.py b/tests/test_execute_evaluate_loop.py index 27135bf0..9d87a7a1 100644 --- a/tests/test_execute_evaluate_loop.py +++ b/tests/test_execute_evaluate_loop.py @@ -473,22 +473,20 @@ def _run(command: str, run_dir: Path) -> Any: assert regraded["tool_calls_exhausted"] is True, "the fact must survive the grade too" -def test_a_recorded_run_limits_max_turns_re_grades_from_the_source_yaml_loudly(tmp_path: Path) -> None: - """A run recorded before `run_limits.max_turns` became `max_tool_calls` no longer - validates. The grade must fall back to the source YAML and say so, not refuse.""" +def test_a_recorded_main_era_max_turns_re_grades_from_the_recorded_config(tmp_path: Path) -> None: + """A `main`-era record carries `run_limits.max_turns`; it validates, so the grade uses the recorded config. + Detached grading skips the harness gate, so the agentless kind that rejects the field still re-grades.""" run_dir = tmp_path / "r" _invoke(["execute", str(AGENTLESS_TASK), "--run-dir", str(run_dir)]) task_dir = _task_dir(run_dir) row = _row(task_dir) - run_limits = row["task_config"]["resolved"]["run_limits"] - run_limits["max_turns"] = run_limits.pop("max_tool_calls") + row["task_config"]["resolved"]["run_limits"]["max_turns"] = 100 (task_dir / "task.json").write_text(json.dumps(row), encoding="utf-8") output = _invoke(["evaluate", str(task_dir)]).output - assert "falling back to" in output - assert "NOT reapplied" in output + assert "falling back to" not in output assert _row(task_dir)["final_status"] == FinalStatus.SUCCESS.value diff --git a/tests/test_harness_conformance.py b/tests/test_harness_conformance.py index a513c95a..074fdc58 100644 --- a/tests/test_harness_conformance.py +++ b/tests/test_harness_conformance.py @@ -27,25 +27,27 @@ from coder_eval.agents.registry import AgentRegistry from coder_eval.models import ( AgentKind, - Enforcement, - FileExistsCriterion, HarnessContract, PermissionMode, - SandboxConfig, - TaskDefinition, parse_agent_config, ) -from coder_eval.orchestration.harness_contract import HarnessContractError, validate_harness_contract from coder_eval.orchestration.plugin_staging import stage_plugins from coder_eval.plugins import ensure_plugins_loaded -from coder_eval.streaming.events import AgentEndEvent, StopReason, ToolEndEvent, end_status_for +from coder_eval.streaming.events import StopReason +from coder_eval.testing import ( + FIRST_TOOL_ID, + SECOND_TOOL_ID, + StopAfterFirstTool, + conformance, + enforced_cells, + rejections, + stop_conformance, +) from tests.test_antigravity_agent import _install_fake_sdk MARKER = "CONFORMANCE-MARKER-7f3a" USER_TURN = "do the task" -_FIELDS = ("system_prompt", "plugin_skills", "permission_mode", "allowed_tools", "disallowed_tools") -_CONFIG_FIELD = {"plugin_skills": "plugins"} _KINDS = [kind for kind in AgentKind if kind is not AgentKind.UNKNOWN] type Probe = Callable[[Path, pytest.MonkeyPatch], Awaitable[None]] @@ -58,17 +60,6 @@ def _contract(kind: AgentKind) -> HarnessContract: return registration.agent_class.contract -def _task(kind: AgentKind, **agent: Any) -> TaskDefinition: - return TaskDefinition( - task_id="t", - description="d", - initial_prompt=None if kind is AgentKind.NONE else USER_TURN, - agent=parse_agent_config(type=kind, **agent), - sandbox=SandboxConfig(driver="tempdir"), - success_criteria=[FileExistsCriterion(description="c", path="out.txt")], - ) - - def _plugin_root(tmp_path: Path) -> Path: """A root staged by ``stage_plugins`` over one authored ``probe-skill``.""" skill = tmp_path / "plugin" / "skills" / "probe-skill" @@ -77,49 +68,26 @@ def _plugin_root(tmp_path: Path) -> Path: return stage_plugins([{"type": "local", "path": str(tmp_path / "plugin")}], tmp_path / "plugin_root").root -_GATED_VALUES: dict[str, Any] = { - "system_prompt": MARKER, - "plugins": [{"type": "local", "path": "/plugins/p"}], - "permission_mode": "plan", - "allowed_tools": ["Bash"], - "disallowed_tools": ["Bash"], -} - - # --- rejections, derived from the contracts ---------------------------------------- @pytest.mark.parametrize( - ("kind", "field"), - [(k, f) for k in _KINDS for f in _FIELDS if getattr(_contract(k), f) is Enforcement.UNSUPPORTED], + ("kind", "check"), + [pytest.param(k, check, id=f"{k.value}-{name}") for k in _KINDS for name, check in rejections(k.value)], ) -def test_unsupported_field_is_rejected(kind: AgentKind, field: str) -> None: - config_field = _CONFIG_FIELD.get(field, field) - with pytest.raises(HarnessContractError, match=rf"agent\.{config_field}.*{kind.value!r}"): - validate_harness_contract(_task(kind, **{config_field: _GATED_VALUES[config_field]})) +def test_contract_rejection(kind: AgentKind, check: Callable[[], None]) -> None: + check() -@pytest.mark.parametrize( - ("kind", "mode"), - [ - (k, m) - for k in _KINDS - if _contract(k).permission_mode is Enforcement.ENFORCED - for m in PermissionMode - if m not in (_contract(k).permission_modes or frozenset()) - ], -) -def test_undeclared_permission_value_is_rejected(kind: AgentKind, mode: PermissionMode) -> None: - with pytest.raises(HarnessContractError, match="has no documented meaning"): - validate_harness_contract(_task(kind, permission_mode=mode)) - - -@pytest.mark.parametrize( - "kind", [k for k in _KINDS if AgentRegistry.get(k) and AgentRegistry.get(k).agent_class.tool_names] -) -def test_unknown_tool_name_is_rejected(kind: AgentKind) -> None: - with pytest.raises(HarnessContractError, match="did you mean 'Bash'"): - validate_harness_contract(_task(kind, allowed_tools=["Bassh"])) +def test_every_kind_rejects_what_its_contract_does_not_honor() -> None: + names = {name for k in _KINDS for name, _ in rejections(k.value)} + assert { + "unsupported system_prompt", + "undeclared permission_mode=default", + "misspelled tool name", + "unsupported run_limits.max_turns", + "unsupported run_limits.expected_turns", + } <= names # --- probes: the value reaches the native call -------------------------------------- @@ -139,7 +107,7 @@ async def fake_query(prompt: str, options: Any): claude = ClaudeCodeAgent(parse_agent_config(type=AgentKind.CLAUDE_CODE, **agent)) await claude.start(str(tmp_path), plugin_root=plugin_root) with patch("coder_eval.agents.claude_code_agent.query", fake_query): - await claude.communicate(USER_TURN) + await claude.communicate(USER_TURN, iteration=1) return captured["options"], captured["prompt"] @@ -152,7 +120,7 @@ async def _probe_claude_system_prompt(tmp_path: Path, _mp: pytest.MonkeyPatch) - async def _probe_claude_plugins(tmp_path: Path, _mp: pytest.MonkeyPatch) -> None: root = _plugin_root(tmp_path) options, _ = await _claude(tmp_path, plugin_root=root) - assert options.plugins == [{"type": "local", "path": str(root)}] + assert options.plugins == [{"type": "local", "path": str(root / "plugins" / "plugin")}] async def test_claude_loads_no_plugin_without_a_plugin_root(tmp_path: Path) -> None: @@ -191,7 +159,7 @@ def turn(self, user_input: str): # type: ignore[override] codex = _started_agent(parse_agent_config(type=AgentKind.CODEX, system_prompt=MARKER), [_turn_completed()]) options = codex._build_thread_options() codex.thread = _RecordingThread([_turn_completed()]) - await codex.communicate(USER_TURN) + await codex.communicate(USER_TURN, iteration=1) assert options["developer_instructions"] == MARKER assert turn_inputs == [USER_TURN] @@ -251,7 +219,7 @@ async def send(self, prompt: str, **kwargs: Any) -> None: agent = AntigravityAgent(parse_agent_config(type=AgentKind.ANTIGRAVITY, system_prompt=MARKER)) agent.working_directory = tmp_path agent._sdk_agent = SimpleNamespace(conversation=_RecordingConversation([done]), is_started=True) - await agent.communicate(USER_TURN) + await agent.communicate(USER_TURN, iteration=1) assert sent == [USER_TURN] @@ -301,10 +269,10 @@ async def _opencode( monkeypatch.delenv("OPENCODE_CONFIG_CONTENT", raising=False) opencode = await _cli_agent(OpenCodeAgent, AgentKind.OPENCODE, tmp_path, monkeypatch, plugin_root, **agent) try: - raw = opencode._build_env().get("OPENCODE_CONFIG_CONTENT") + raw = opencode.env().get("OPENCODE_CONFIG_CONTENT") config = json.loads(raw) if raw else {} config["instructions_text"] = [Path(p).read_text(encoding="utf-8") for p in config.get("instructions", [])] - return config, opencode._build_argv(USER_TURN) + return config, opencode.argv(USER_TURN) finally: await opencode.stop() @@ -348,7 +316,7 @@ async def _pi_argv( ) -> list[str]: pi = await _cli_agent(PiAgent, AgentKind.PI, tmp_path, monkeypatch, plugin_root, **agent) try: - return pi._build_argv(USER_TURN) + return pi.argv(USER_TURN) finally: await pi.stop() @@ -418,17 +386,7 @@ async def _probe_pi_disallowed(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) def _enforced_cells() -> set[tuple[str, str]]: - cells: set[tuple[str, str]] = set() - for kind in _KINDS: - contract = _contract(kind) - for field in _FIELDS: - if getattr(contract, field) is not Enforcement.ENFORCED: - continue - if field == "permission_mode": - cells |= {(kind.value, f"permission_mode={m.value}") for m in contract.permission_modes or ()} - else: - cells.add((kind.value, field)) - return cells + return {cell for kind in _KINDS for cell in enforced_cells(_contract(kind), kind.value)} def test_every_enforced_cell_has_exactly_one_probe() -> None: @@ -440,29 +398,22 @@ async def test_probe(cell: tuple[str, str], tmp_path: Path, monkeypatch: pytest. await _PROBES[cell](tmp_path, monkeypatch) -# --- cooperative stop: every StopReason ends the turn at the boundary --------------- - +@pytest.mark.parametrize("kind", _KINDS, ids=lambda k: k.value) +async def test_conformance_per_kind(kind: AgentKind, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + def bound(cell: tuple[str, str], probe: Probe) -> Callable[[], Awaitable[None]]: + directory = tmp_path / cell[1].replace("=", "-") + directory.mkdir() + return lambda: probe(directory, monkeypatch) -class _StopAfterFirstTool: - """``should_stop`` stub: ``reason`` once one tool call has resolved; also keeps the end event.""" - - def __init__(self, reason: StopReason) -> None: - self.reason = reason - self.tool_ends = 0 - self.end: AgentEndEvent | None = None + await conformance( + kind.value, {cell: bound(cell, probe) for cell, probe in _PROBES.items() if cell[0] == kind.value} + ) - def on_event(self, event: object) -> None: - if isinstance(event, ToolEndEvent): - self.tool_ends += 1 - elif isinstance(event, AgentEndEvent): - self.end = event - def __call__(self) -> StopReason | None: - return self.reason if self.tool_ends else None +# --- cooperative stop: every StopReason ends the turn at the boundary --------------- -type StopProbe = Callable[[Path, pytest.MonkeyPatch, _StopAfterFirstTool], Awaitable[list[Any]]] -_SECOND = "second-call" +type StopProbe = Callable[[Path, pytest.MonkeyPatch, StopAfterFirstTool], Awaitable[list[Any]]] def _recording(items: list[Any], pulled: list[Any]) -> Iterator[Any]: @@ -471,7 +422,7 @@ def _recording(items: list[Any], pulled: list[Any]) -> Iterator[Any]: yield item -async def _stop_claude(tmp_path: Path, _mp: pytest.MonkeyPatch, stop: _StopAfterFirstTool) -> list[Any]: +async def _stop_claude(tmp_path: Path, _mp: pytest.MonkeyPatch, stop: StopAfterFirstTool) -> list[Any]: from tests._fixtures.golden_streams.claude_fixtures import ( AssistantMessage, ResultMessage, @@ -481,10 +432,10 @@ async def _stop_claude(tmp_path: Path, _mp: pytest.MonkeyPatch, stop: _StopAfter pulled: list[Any] = [] events = [ - AssistantMessage([ToolUseBlock("first", "Bash", {"command": "ls"})], message_id="m1"), - UserMessage("first", False, "ok"), - AssistantMessage([ToolUseBlock(_SECOND, "Bash", {"command": "ls"})], message_id="m2"), - UserMessage(_SECOND, False, "ok"), + AssistantMessage([ToolUseBlock(FIRST_TOOL_ID, "Bash", {"command": "ls"})], message_id="m1"), + UserMessage(FIRST_TOOL_ID, False, "ok"), + AssistantMessage([ToolUseBlock(SECOND_TOOL_ID, "Bash", {"command": "ls"})], message_id="m2"), + UserMessage(SECOND_TOOL_ID, False, "ok"), ResultMessage(), ] @@ -495,11 +446,11 @@ async def fake_query(prompt: Any, options: Any, transport: Any = None): claude = ClaudeCodeAgent(parse_agent_config(type=AgentKind.CLAUDE_CODE)) await claude.start(str(tmp_path)) with patch("coder_eval.agents.claude_code_agent.query", fake_query): - await claude.communicate(USER_TURN, stream_callback=stop, should_stop=stop) + await claude.communicate(USER_TURN, iteration=1, stream_callback=stop, should_stop=stop) return [getattr(e.content[0], "id", None) for e in pulled if hasattr(e, "content")] -async def _stop_codex(_tmp: Path, _mp: pytest.MonkeyPatch, stop: _StopAfterFirstTool) -> list[Any]: +async def _stop_codex(_tmp: Path, _mp: pytest.MonkeyPatch, stop: StopAfterFirstTool) -> list[Any]: from tests.test_codex_agent import _FakeThread, _FakeTurnHandle, _item_notification, _started_agent pulled: list[Any] = [] @@ -511,7 +462,7 @@ def command(item_id: str) -> SimpleNamespace: notifications = [ _item_notification(method, command(item_id)) - for item_id in ("first", _SECOND) + for item_id in (FIRST_TOOL_ID, SECOND_TOOL_ID) for method in ("item/started", "item/completed") ] @@ -526,11 +477,11 @@ def turn(self, _user_input: str): # type: ignore[override] codex = _started_agent(parse_agent_config(type=AgentKind.CODEX), notifications) codex.thread = _RecordingThread(notifications) - await codex.communicate(USER_TURN, stream_callback=stop, should_stop=stop) + await codex.communicate(USER_TURN, iteration=1, stream_callback=stop, should_stop=stop) return [n.payload.item.root.id for n in pulled] -async def _stop_antigravity(tmp_path: Path, _mp: pytest.MonkeyPatch, stop: _StopAfterFirstTool) -> list[Any]: +async def _stop_antigravity(tmp_path: Path, _mp: pytest.MonkeyPatch, stop: StopAfterFirstTool) -> list[Any]: from tests._fixtures.golden_streams.antigravity_fixtures import _FakeConversation, _step, _tc pulled: list[Any] = [] @@ -547,7 +498,7 @@ def call(tool_id: str) -> list[Any]: ), ] - steps = [*call("first"), *call(_SECOND)] + steps = [*call(FIRST_TOOL_ID), *call(SECOND_TOOL_ID)] class _RecordingConversation(_FakeConversation): async def receive_steps(self): @@ -558,7 +509,7 @@ async def receive_steps(self): agent = AntigravityAgent(parse_agent_config(type=AgentKind.ANTIGRAVITY)) agent.working_directory = tmp_path agent._sdk_agent = SimpleNamespace(conversation=_RecordingConversation([]), is_started=True) - await agent.communicate(USER_TURN, stream_callback=stop, should_stop=stop) + await agent.communicate(USER_TURN, iteration=1, stream_callback=stop, should_stop=stop) return [s.tool_calls[0].id for s in pulled] @@ -586,23 +537,23 @@ async def fake_exec(*_argv: str, **_kwargs: Any) -> _RecordingProcess: monkeypatch.setattr("os.killpg", lambda _pgid, _sig: None, raising=False) cli = await _cli_agent(cls, kind, tmp_path, monkeypatch) try: - await cli.communicate(USER_TURN, stream_callback=stop, should_stop=stop) + await cli.communicate(USER_TURN, iteration=1, stream_callback=stop, should_stop=stop) finally: await cli.stop() - return [tool_id for tool_id in ("first", _SECOND) if any(tool_id in json.dumps(p) for p in pulled)] + return [tool_id for tool_id in (FIRST_TOOL_ID, SECOND_TOOL_ID) if any(tool_id in json.dumps(p) for p in pulled)] -async def _stop_pi(tmp_path: Path, monkeypatch: pytest.MonkeyPatch, stop: _StopAfterFirstTool) -> list[Any]: +async def _stop_pi(tmp_path: Path, monkeypatch: pytest.MonkeyPatch, stop: StopAfterFirstTool) -> list[Any]: from tests._fixtures.golden_streams.pi_fixtures import _tool_end, _tool_start, _turn_end, _turn_start lines = [_turn_start()] - for tool_id in ("first", _SECOND): + for tool_id in (FIRST_TOOL_ID, SECOND_TOOL_ID): lines += [_tool_start(tool_id, "bash", {"command": "ls"}), _tool_end(tool_id, "bash", "ok")] lines.append(_turn_end(inp=1, out=1)) return await _stop_cli(PiAgent, AgentKind.PI, lines, tmp_path, monkeypatch, stop) -async def _stop_opencode(tmp_path: Path, monkeypatch: pytest.MonkeyPatch, stop: _StopAfterFirstTool) -> list[Any]: +async def _stop_opencode(tmp_path: Path, monkeypatch: pytest.MonkeyPatch, stop: StopAfterFirstTool) -> list[Any]: from tests._fixtures.golden_streams.opencode_fixtures import _evt def tool_use(tool_id: str) -> str: @@ -622,8 +573,8 @@ def tool_use(tool_id: str) -> str: monkeypatch.delenv("OPENCODE_CONFIG_CONTENT", raising=False) lines = [ _evt("step_start", {"id": "prt_0", "messageID": "msg_1", "type": "step-start"}), - tool_use("first"), - tool_use(_SECOND), + tool_use(FIRST_TOOL_ID), + tool_use(SECOND_TOOL_ID), ] return await _stop_cli(OpenCodeAgent, AgentKind.OPENCODE, lines, tmp_path, monkeypatch, stop) @@ -641,17 +592,11 @@ def test_every_cooperative_kind_has_a_stop_probe() -> None: assert set(_STOP_PROBES) == {k.value for k in _KINDS if _contract(k).cooperative_stop} -@pytest.mark.parametrize("reason", list(StopReason)) @pytest.mark.parametrize("kind", sorted(_STOP_PROBES)) -async def test_stop_reason_ends_the_turn_before_the_next_call( - kind: str, reason: StopReason, tmp_path: Path, monkeypatch: pytest.MonkeyPatch -) -> None: - stop = _StopAfterFirstTool(reason) - - pulled = await _STOP_PROBES[kind](tmp_path, monkeypatch, stop) - - assert stop.end is not None - assert stop.end.status is end_status_for(reason) - assert stop.end.crashed is False - assert "first" in pulled - assert _SECOND not in pulled +async def test_stop_conformance(kind: str, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + async def probe(stop: StopAfterFirstTool, reason: StopReason) -> list[Any]: + directory = tmp_path / reason.value + directory.mkdir() + return await _STOP_PROBES[kind](directory, monkeypatch, stop) + + await stop_conformance(kind, probe) diff --git a/tests/test_harness_contract.py b/tests/test_harness_contract.py index be36a089..3983c7e8 100644 --- a/tests/test_harness_contract.py +++ b/tests/test_harness_contract.py @@ -11,7 +11,7 @@ from pydantic import BaseModel, ConfigDict, ValidationError from coder_eval.agents.pi_agent import PiAgent -from coder_eval.agents.registry import AgentRegistry +from coder_eval.agents.registry import SPI_VERSION, AgentRegistry from coder_eval.models import ( CANONICAL_TOOL_NAMES, READ_ONLY_DENIED_TOOLS, @@ -25,9 +25,11 @@ FileExistsCriterion, HarnessContract, PermissionMode, + RunLimits, SandboxConfig, TaskDefinition, ToolNameMap, + UsageGranularity, parse_agent_config, ) from coder_eval.orchestration.config import BatchRunConfig @@ -39,6 +41,7 @@ resolve_task_for_variant, ) from coder_eval.orchestration.harness_contract import ( + MODEL_TURN_LIMITS, HarnessContractError, TaskResolutionError, validate_harness_contract, @@ -86,10 +89,20 @@ def test_usage_granularity_is_required(self) -> None: with pytest.raises(ValidationError, match="usage_granularity"): HarnessContract(**fields) - def test_unknown_field_rejected(self) -> None: + def test_timing_basis_is_required(self) -> None: + fields = stub_contract().model_dump() + del fields["timing_basis"] + with pytest.raises(ValidationError, match="timing_basis"): + HarnessContract(**fields) + + def test_unknown_timing_basis_rejected(self) -> None: with pytest.raises(ValidationError, match="timing_basis"): HarnessContract(**{**stub_contract().model_dump(), "timing_basis": "wall"}) + def test_unknown_field_rejected(self) -> None: + with pytest.raises(ValidationError, match="clock_basis"): + HarnessContract(**{**stub_contract().model_dump(), "clock_basis": "wall"}) + class TestPermissionModes: def _contract(self, **fields: Any) -> HarnessContract: @@ -184,7 +197,7 @@ class NoContractAgent: pass with pytest.raises(TypeError, match=rf"{KIND}.*NoContractAgent.*HarnessContract"): - AgentRegistry.register(KIND, config_for_kind(KIND))(NoContractAgent) + AgentRegistry.register(KIND, config_for_kind(KIND), spi_version=SPI_VERSION)(NoContractAgent) assert AgentRegistry.get(KIND) is None def test_dict_contract_rejected(self, restored_registry: None) -> None: @@ -192,7 +205,7 @@ class DictContractAgent: contract = stub_contract().model_dump() with pytest.raises(TypeError, match=rf"{KIND}.*DictContractAgent"): - AgentRegistry.register(KIND, config_for_kind(KIND))(DictContractAgent) + AgentRegistry.register(KIND, config_for_kind(KIND), spi_version=SPI_VERSION)(DictContractAgent) def test_config_not_a_base_agent_config_rejected(self, restored_registry: None) -> None: class ForeignConfig(BaseModel): @@ -200,7 +213,7 @@ class ForeignConfig(BaseModel): type: Literal["contract-test-kind"] with pytest.raises(TypeError, match=rf"{KIND}.*ForeignConfig.*BaseAgentConfig"): - AgentRegistry.register(KIND, ForeignConfig)(_ContractAgent) # type: ignore[type-var] + AgentRegistry.register(KIND, ForeignConfig, spi_version=SPI_VERSION)(_ContractAgent) # type: ignore[type-var] def test_config_without_extra_forbid_rejected(self, restored_registry: None) -> None: class LaxConfig(BaseAgentConfig): @@ -208,18 +221,18 @@ class LaxConfig(BaseAgentConfig): type: Literal["contract-test-kind"] # type: ignore[assignment] with pytest.raises(TypeError, match=rf"{KIND}.*LaxConfig.*extra='forbid'"): - AgentRegistry.register(KIND, LaxConfig)(_ContractAgent) + AgentRegistry.register(KIND, LaxConfig, spi_version=SPI_VERSION)(_ContractAgent) def test_type_literal_not_naming_the_kind_rejected(self, restored_registry: None) -> None: with pytest.raises(TypeError, match=rf"{KIND}.*ClaudeCodeAgentConfig.*Literal"): - AgentRegistry.register(KIND, ClaudeCodeAgentConfig)(_ContractAgent) + AgentRegistry.register(KIND, ClaudeCodeAgentConfig, spi_version=SPI_VERSION)(_ContractAgent) def test_type_literal_covering_several_kinds_accepted(self, restored_registry: None) -> None: class TwoKindConfig(BaseAgentConfig): type: Literal["contract-test-kind", "other-kind"] # type: ignore[assignment] - AgentRegistry.register(KIND, TwoKindConfig)(_ContractAgent) - AgentRegistry.register("other-kind", TwoKindConfig)(_ContractAgent) + AgentRegistry.register(KIND, TwoKindConfig, spi_version=SPI_VERSION)(_ContractAgent) + AgentRegistry.register("other-kind", TwoKindConfig, spi_version=SPI_VERSION)(_ContractAgent) assert AgentRegistry.get("other-kind") is not None def test_enforced_tool_lists_require_tool_names(self, restored_registry: None) -> None: @@ -227,7 +240,7 @@ class NoMapAgent: contract = HarnessContract(**{**stub_contract().model_dump(), "allowed_tools": "enforced"}) with pytest.raises(TypeError, match=rf"{KIND}.*NoMapAgent.*tool_names"): - AgentRegistry.register(KIND, config_for_kind(KIND))(NoMapAgent) + AgentRegistry.register(KIND, config_for_kind(KIND), spi_version=SPI_VERSION)(NoMapAgent) def test_unsupported_tool_lists_reject_tool_names(self, restored_registry: None) -> None: class StrayMapAgent: @@ -235,12 +248,12 @@ class StrayMapAgent: tool_names = ToolNameMap(names=_identity_names()) with pytest.raises(TypeError, match=rf"{KIND}.*StrayMapAgent.*tool_names"): - AgentRegistry.register(KIND, config_for_kind(KIND))(StrayMapAgent) + AgentRegistry.register(KIND, config_for_kind(KIND), spi_version=SPI_VERSION)(StrayMapAgent) def test_valid_pair_registers_idempotently(self, restored_registry: None) -> None: config = config_for_kind(KIND) - AgentRegistry.register(KIND, config)(_ContractAgent) - AgentRegistry.register(KIND, config)(_ContractAgent) + AgentRegistry.register(KIND, config, spi_version=SPI_VERSION)(_ContractAgent) + AgentRegistry.register(KIND, config, spi_version=SPI_VERSION)(_ContractAgent) registration = AgentRegistry.get(KIND) assert registration is not None and registration.agent_class is _ContractAgent @@ -263,7 +276,7 @@ def test_every_builtin_accepts_cost_log_tags(kind: AgentKind) -> None: assert agent.cost_log_tags == tags -def _task(kind: str, **agent_fields: Any) -> TaskDefinition: +def _task(kind: str, *, run_limits: RunLimits | None = None, **agent_fields: Any) -> TaskDefinition: prompt = None if kind == AgentKind.NONE else "do it" return TaskDefinition( task_id="t", @@ -272,6 +285,7 @@ def _task(kind: str, **agent_fields: Any) -> TaskDefinition: agent=parse_agent_config(type=kind, **agent_fields), sandbox=SandboxConfig(driver="tempdir"), success_criteria=[FileExistsCriterion(description="c", path="out.txt")], + run_limits=run_limits, ) @@ -329,7 +343,7 @@ def test_task_without_agent_type_is_left_to_the_type_guard(self) -> None: validate_harness_contract(_task(AgentKind.CODEX).model_copy(update={"agent": None})) def test_unregistered_kind_is_rejected(self, restored_registry: None) -> None: - AgentRegistry.register(KIND, config_for_kind(KIND))(_ContractAgent) + AgentRegistry.register(KIND, config_for_kind(KIND), spi_version=SPI_VERSION)(_ContractAgent) task = TaskDefinition( task_id="t", description="d", @@ -342,6 +356,84 @@ def test_unregistered_kind_is_rejected(self, restored_registry: None) -> None: validate_harness_contract(task) +class TestModelTurnLimits: + @pytest.mark.parametrize("cooperative_stop", [True, False]) + @pytest.mark.parametrize("granularity", list(UsageGranularity)) + def test_counts_model_turns_follows_usage_granularity_and_cooperative_stop( + self, cooperative_stop: bool, granularity: UsageGranularity + ) -> None: + contract = stub_contract(cooperative_stop=cooperative_stop).model_copy( + update={"usage_granularity": granularity} + ) + assert contract.counts_model_turns is (cooperative_stop and granularity is not UsageGranularity.TURN) + + @pytest.mark.parametrize( + ("kind", "accepted"), + [ + (AgentKind.CLAUDE_CODE, True), + (AgentKind.OPENCODE, True), + (AgentKind.PI, True), + (AgentKind.CODEX, True), + (AgentKind.ANTIGRAVITY, True), + (AgentKind.NONE, False), + ], + ) + @pytest.mark.parametrize("field", MODEL_TURN_LIMITS) + def test_model_turn_limit_gate(self, kind: AgentKind, accepted: bool, field: str) -> None: + task = _task(kind, run_limits=RunLimits.model_validate({field: 3})) + if accepted: + validate_harness_contract(task) + return + with pytest.raises(HarnessContractError) as exc: + validate_harness_contract(task) + message = str(exc.value) + assert f"run_limits.{field}" in message + assert f"{kind.value!r}" in message + assert "docs/agents/HARNESS_PARITY.md" in message + assert "claude-code" in message.split("counts model turns", 1)[1] + + def test_unset_model_turn_limits_pass_everywhere(self) -> None: + validate_harness_contract(_task(AgentKind.CODEX, run_limits=RunLimits())) + + +class TestMaxUsdPriceable: + @pytest.mark.parametrize( + ("kind", "reports_cost"), + [ + (AgentKind.CLAUDE_CODE, True), + (AgentKind.NONE, True), + (AgentKind.CODEX, False), + (AgentKind.PI, False), + (AgentKind.OPENCODE, False), + (AgentKind.ANTIGRAVITY, False), + ], + ) + def test_reports_cost_per_builtin(self, kind: AgentKind, reports_cost: bool) -> None: + registration = AgentRegistry.get(kind.value) + assert registration is not None + assert registration.agent_class.contract.reports_cost is reports_cost + + @pytest.mark.parametrize("model", [None, "provider/not-on-the-card"]) + @pytest.mark.parametrize("kind", [AgentKind.CODEX, AgentKind.PI, AgentKind.OPENCODE, AgentKind.ANTIGRAVITY]) + def test_an_unpriced_model_is_rejected_where_the_harness_reports_no_cost( + self, kind: AgentKind, model: str | None + ) -> None: + task = _task(kind, run_limits=RunLimits(max_usd=1.0), model=model) + with pytest.raises(HarnessContractError, match=r"run_limits\.max_usd is set") as exc: + validate_harness_contract(task) + assert "claude-code" in str(exc.value) + + def test_a_priced_model_is_accepted_where_the_harness_reports_no_cost(self) -> None: + validate_harness_contract(_task(AgentKind.CODEX, run_limits=RunLimits(max_usd=1.0), model="gpt-5-codex")) + + @pytest.mark.parametrize("model", [None, "provider/not-on-the-card"]) + def test_a_harness_that_reports_cost_needs_no_rate(self, model: str | None) -> None: + validate_harness_contract(_task(AgentKind.CLAUDE_CODE, run_limits=RunLimits(max_usd=1.0), model=model)) + + def test_no_max_usd_needs_no_rate(self) -> None: + validate_harness_contract(_task(AgentKind.CODEX, run_limits=RunLimits(max_output_tokens=10))) + + _NON_CLAUDE_ENFORCING = [AgentKind.PI, AgentKind.OPENCODE, AgentKind.ANTIGRAVITY] @@ -536,6 +628,12 @@ def test_the_shipped_default_experiment_resolves_every_builtin_kind(self) -> Non _PARITY_FIXTURE_DIRS = ("tasks/run_limits", "tasks/skills") +def _contract_of(kind: AgentKind) -> HarnessContract: + registration = AgentRegistry.get(kind) + assert registration is not None + return registration.agent_class.contract + + def _cooperative_kinds() -> list[AgentKind]: ensure_plugins_loaded() kinds = [kind for kind in AgentKind if kind is not AgentKind.UNKNOWN] @@ -552,11 +650,21 @@ def _cooperative_kinds() -> list[AgentKind]: ids=lambda p: Path(p).name, ) def test_multi_harness_fixtures_run_on_every_cooperative_harness(fixture: str) -> None: - """A fixture documented to run with `--type ` must not carry a field one harness rejects.""" + """A fixture documented to run with `--type ` must not carry a field one harness rejects. + + A fixture that sets a model-turn limit runs on the harnesses that count model turns, and the rest reject it. + """ from coder_eval.orchestration.task_loader import load_task task, _source = load_task(Path(fixture)) authored = task.agent.model_dump(exclude_unset=True, exclude={"type"}) if task.agent is not None else {} + counts_turns_only = task.run_limits is not None and any( + getattr(task.run_limits, field) is not None for field in MODEL_TURN_LIMITS + ) for kind in _cooperative_kinds(): retyped = task.model_copy(update={"agent": parse_agent_config(type=kind, **authored)}) - validate_harness_contract(retyped) + if counts_turns_only and not _contract_of(kind).counts_model_turns: + with pytest.raises(HarnessContractError, match=r"run_limits\."): + validate_harness_contract(retyped) + else: + validate_harness_contract(retyped) diff --git a/tests/test_harness_live.py b/tests/test_harness_live.py new file mode 100644 index 00000000..60838e26 --- /dev/null +++ b/tests/test_harness_live.py @@ -0,0 +1,134 @@ +"""One tiny real turn per installed harness, through the ``communicate`` contract. + +Skipped by default; runs only with ``pytest -m live``. Each case skips when its +harness is not installed or its credential is absent. +""" + +import importlib.util +import os +import shutil +from dataclasses import dataclass, field +from pathlib import Path + +import pytest + +from coder_eval.agents.registry import create_agent +from coder_eval.config import Settings +from coder_eval.models import ( + AgentKind, + ApiBackend, + ApiRoute, + AssistantMessage, + DirectRoute, + TurnRecord, + parse_agent_config, + resolve_route, +) +from coder_eval.plugins import ensure_plugins_loaded +from coder_eval.streaming.events import AgentEndStatus, StreamEvent +from coder_eval.testing import assert_stream_balanced + + +pytestmark = pytest.mark.live + +PROMPT = "Create a file named ok.txt containing ok, then reply DONE." +OPENROUTER_HAIKU = "openrouter/anthropic/claude-haiku-4.5" + + +@dataclass(frozen=True) +class Harness: + kind: AgentKind + model: str | None + cli: str | None = None + module: str | None = None + credential: str | None = None + # A mode the harness contract documents; None leaves an unsupported field unset. + permission_mode: str | None = "bypassPermissions" + + +HARNESSES = [ + Harness(AgentKind.CLAUDE_CODE, "claude-haiku-4-5-20251001", cli="claude"), + Harness( + AgentKind.CODEX, + os.getenv("CODEX_MODEL"), + module="openai_codex", + credential="CODEX_API_KEY", + permission_mode=None, + ), + Harness(AgentKind.PI, OPENROUTER_HAIKU, cli="pi", credential="OPENROUTER_API_KEY"), + Harness(AgentKind.OPENCODE, OPENROUTER_HAIKU, cli="opencode", credential="OPENROUTER_API_KEY"), + Harness(AgentKind.ANTIGRAVITY, "gemini-3.5-flash-lite", module="google.antigravity", credential="GEMINI_API_KEY"), +] + + +@dataclass +class Recorder: + events: list[StreamEvent] = field(default_factory=list) + + def on_event(self, event: StreamEvent) -> None: + self.events.append(event) + + +def _importable(module: str) -> bool: + try: + return importlib.util.find_spec(module) is not None + except ModuleNotFoundError: + return False + + +def _skip_unless_installed(harness: Harness) -> None: + if harness.cli is not None and shutil.which(harness.cli) is None: + pytest.skip(f"the '{harness.cli}' CLI is not on PATH") + if harness.module is not None and not _importable(harness.module): + pytest.skip(f"'{harness.module}' is not importable") + if harness.credential is not None and not os.getenv(harness.credential): + pytest.skip(f"{harness.credential} is not set") + + +def _claude_route(model: str | None) -> tuple[ApiRoute, str | None]: + """The configured backend's route, as the orchestrator builds it; a Bedrock route keeps its own model.""" + settings = Settings() + if settings.api_backend == ApiBackend.DIRECT: + return DirectRoute(), model + return resolve_route(settings), None if settings.api_backend == ApiBackend.BEDROCK else model + + +def _bucket_sum_ms(record: TurnRecord) -> float: + generation_ms = sum( + m.generation_duration_ms or 0.0 + for m in record.messages + if isinstance(m, AssistantMessage) and m.parent_tool_use_id is None + ) + return ( + (record.harness_startup_ms or 0.0) + + generation_ms + + (record.tool_union_ms or 0.0) + + (record.harness_teardown_ms or 0.0) + ) + + +@pytest.mark.parametrize("harness", HARNESSES, ids=[str(h.kind) for h in HARNESSES]) +async def test_harness_completes_a_tiny_turn(harness: Harness, tmp_path: Path) -> None: + _skip_unless_installed(harness) + ensure_plugins_loaded() + route: ApiRoute | None = None + model = harness.model + if harness.kind is AgentKind.CLAUDE_CODE: + route, model = _claude_route(model) + fields = {"model": model} | ({"permission_mode": harness.permission_mode} if harness.permission_mode else {}) + config = parse_agent_config(type=harness.kind, **fields) + agent = create_agent(harness.kind, config, route) + recorder = Recorder() + try: + await agent.start(str(tmp_path)) + outcome = await agent.communicate(PROMPT, iteration=1, timeout=180, stream_callback=recorder) + finally: + await agent.stop() + + record = outcome.record + assert outcome.status is AgentEndStatus.COMPLETED, outcome.error + assert (tmp_path / "ok.txt").exists(), "the agent did not create ok.txt" + assert record.commands, "expected at least one command" + assert PROMPT not in record.agent_output + assert_stream_balanced(recorder.events) + assert _bucket_sum_ms(record) <= record.duration_seconds * 1000 + 1000 diff --git a/tests/test_harness_version.py b/tests/test_harness_version.py new file mode 100644 index 00000000..6ea79e71 --- /dev/null +++ b/tests/test_harness_version.py @@ -0,0 +1,77 @@ +"""``Agent.harness_version``: every built-in adapter names the CLI or SDK version it drives.""" + +from __future__ import annotations + +import os +import sys +from importlib.metadata import version +from pathlib import Path +from types import SimpleNamespace + +import pytest + +from coder_eval.agent import command_version +from coder_eval.agents.antigravity_agent import AntigravityAgent +from coder_eval.agents.claude_code_agent import ClaudeCodeAgent +from coder_eval.agents.codex_agent import CodexAgent +from coder_eval.agents.noop_agent import NoOpAgent +from coder_eval.agents.opencode_agent import OpenCodeAgent +from coder_eval.agents.pi_agent import PiAgent +from coder_eval.models import AgentKind, parse_agent_config + + +posix_only = pytest.mark.skipif(os.name != "posix", reason="the fake CLI is a shell script") + + +class TestCommandVersion: + async def test_the_first_non_empty_stdout_line(self) -> None: + argv = [sys.executable, "-c", "print(); print(' tool 1.2.3 '); print('extra')"] + assert await command_version(argv) == "tool 1.2.3" + + async def test_a_non_zero_exit_is_unknown(self) -> None: + assert await command_version([sys.executable, "-c", "print('1.0'); raise SystemExit(2)"]) is None + + async def test_a_missing_binary_is_unknown(self) -> None: + assert await command_version(["coder-eval-no-such-binary-7f3a", "--version"]) is None + + +@posix_only +@pytest.mark.parametrize(("cls", "kind"), [(PiAgent, AgentKind.PI), (OpenCodeAgent, AgentKind.OPENCODE)]) +async def test_a_cli_harness_runs_its_executable_on_the_turn_path( + cls: type[PiAgent] | type[OpenCodeAgent], kind: AgentKind, tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + bin_dir = tmp_path / "bin" + bin_dir.mkdir() + fake = bin_dir / cls.executable + fake.write_text(f"#!/bin/sh\n[ \"$1\" = --version ] && echo '{cls.executable} 9.9.9'\n") + fake.chmod(0o755) + monkeypatch.delenv("OPENCODE_CONFIG_CONTENT", raising=False) + monkeypatch.setattr("shutil.which", lambda name: str(bin_dir / name)) + agent = cls(parse_agent_config(type=kind, model="provider/model"), task_id="t1") # type: ignore[arg-type] + await agent.start(str(tmp_path / "work"), env_path_prepend=[str(bin_dir)]) + try: + assert await agent.harness_version() == f"{cls.executable} 9.9.9" + finally: + await agent.stop() + + +async def test_claude_code_names_the_sdk_and_its_cli() -> None: + agent = ClaudeCodeAgent(parse_agent_config(type=AgentKind.CLAUDE_CODE)) + reported = await agent.harness_version() + assert reported is not None + assert reported.startswith(f"claude-agent-sdk {version('claude-agent-sdk')}; Claude Code ") + + +async def test_codex_names_the_sdk_and_the_app_server_from_the_handshake() -> None: + agent = CodexAgent(parse_agent_config(type=AgentKind.CODEX)) + agent.codex_client = SimpleNamespace(metadata=SimpleNamespace(serverInfo=SimpleNamespace(version="0.200.0"))) + assert await agent.harness_version() == f"openai-codex {version('openai-codex')}; codex app-server 0.200.0" + + +async def test_antigravity_names_the_sdk_package() -> None: + agent = AntigravityAgent(parse_agent_config(type=AgentKind.ANTIGRAVITY)) + assert await agent.harness_version() == f"google-antigravity {version('google-antigravity')}" + + +async def test_the_base_default_is_unknown() -> None: + assert await NoOpAgent(parse_agent_config(type=AgentKind.NONE)).harness_version() is None diff --git a/tests/test_judge_burn_in_live.py b/tests/test_judge_burn_in_live.py index bf780fac..1c3fe6ad 100644 --- a/tests/test_judge_burn_in_live.py +++ b/tests/test_judge_burn_in_live.py @@ -30,6 +30,7 @@ from coder_eval.evaluation.checker import SuccessChecker from coder_eval.models import ( AgentJudgeCriterion, + ClaudeCodeAgentConfig, LLMJudgeCriterion, SandboxConfig, parse_agent_config, @@ -123,16 +124,18 @@ def test_agent_judge_sdk_tool_channel(hello_sandbox: Sandbox) -> None: if not api_key or api_key.startswith("sk-ant-test-"): pytest.skip("ANTHROPIC_API_KEY not set (or placeholder)") + agent = parse_agent_config( + type="claude-code", + model="claude-haiku-4-5-20251001", + allowed_tools=["Read", "Bash"], + ) + assert isinstance(agent, ClaudeCodeAgentConfig) criterion = AgentJudgeCriterion( description="burn-in / SDK", prompt=_JUDGE_PROMPT, max_turns=6, turn_timeout=120, - agent=parse_agent_config( - type="claude-code", - model="claude-haiku-4-5-20251001", - allowed_tools=["Read", "Bash"], - ), + agent=agent, ) result = SuccessChecker(hello_sandbox, init_registry=False, route=DirectRoute()).check(criterion) diff --git a/tests/test_litellm_route.py b/tests/test_litellm_route.py index 8d1bd195..f70b89fd 100644 --- a/tests/test_litellm_route.py +++ b/tests/test_litellm_route.py @@ -584,6 +584,21 @@ def test_cost_log_tags_reject_header_injection(self): with pytest.raises(ValueError, match="single-line ASCII"): ClaudeCodeAgent._build_sdk_env(route, cost_log_tags={"x-ce-task-id": "ok\nAuthorization: Bearer forged"}) + def test_the_query_stamps_the_caller_iteration_header(self, tmp_path): + """The ``x-ce-iteration`` header is the ``iteration`` passed to the turn, not agent state.""" + from pathlib import Path + + agent = ClaudeCodeAgent( + parse_agent_config(type=AgentKind.CLAUDE_CODE, model="m"), + route=LiteLLMRoute(model="m"), + cost_log_tags={"x-ce-run-id": "r"}, + ) + agent.working_directory = Path(tmp_path) + + options, _transport, _model = agent._build_claude_query("hi", 7, None, lambda _line: None) + + assert options.env["ANTHROPIC_CUSTOM_HEADERS"] == "x-ce-run-id: r\nx-ce-iteration: 7" + class TestResolveEffectiveModelCustom: """_resolve_effective_model() on the LiteLLM route — no prefixing.""" @@ -672,25 +687,31 @@ class TestRepriceWiring: and disable the max_usd gate — the static-only tests above wouldn't catch it.""" def _usage_after_finalize(self, effective_model: str | None) -> TokenUsage: - from types import SimpleNamespace + from datetime import datetime - from coder_eval.agents.claude_code_agent import _ClaudeTurnState + from coder_eval.agents.claude_code_agent import _ClaudeDecoder + from coder_eval.models import TimingBasis + from coder_eval.streaming.emitter import TurnEmitter + from coder_eval.testing import ScriptedClock agent = _make_agent( LiteLLMRoute(model="zai.glm-5"), config_model="zai.glm-5", ) - stub = SimpleNamespace( - _agent=agent, - sdk_messages=[], - sdk_result_usage=None, - sdk_result_cost=None, - # model_usage carries the SDK's Claude-priced estimate (3.68); the - # reprice must override it from the litellm rate table. - sdk_result_model_usage={"m": {"inputTokens": 1_000_000, "outputTokens": 1_000_000, "costUSD": 3.68}}, - effective_model=effective_model, + emitter = TurnEmitter( + task_id="t", + iteration=1, + prompt="go", + model=effective_model, + basis=TimingBasis.TURN_CLOCK, + clock=ScriptedClock(datetime(2026, 1, 1)), + sinks=[], ) - return _ClaudeTurnState._finalize_token_usage(stub) # type: ignore[arg-type] + decoder = _ClaudeDecoder(agent, emitter, effective_model=effective_model) + # model_usage carries the SDK's Claude-priced estimate (3.68); the + # reprice must override it from the litellm rate table. + decoder.sdk_result_model_usage = {"m": {"inputTokens": 1_000_000, "outputTokens": 1_000_000, "costUSD": 3.68}} + return decoder._finalize_token_usage() def test_finalize_reprices_priced_litellm_model(self): usage = self._usage_after_finalize("zai.glm-5") diff --git a/tests/test_opencode_agent.py b/tests/test_opencode_agent.py index 5483bf6a..975ca2e9 100644 --- a/tests/test_opencode_agent.py +++ b/tests/test_opencode_agent.py @@ -3,7 +3,9 @@ The CLI is never invoked: ``asyncio.create_subprocess_exec`` is patched with a fake process that replays a newline-delimited JSON event stream, so the whole reduction path (nd-JSON -> standardized events -> ``TurnRecord``) is exercised -offline and without credentials. +offline and without credentials. Timing cases drive ``_OpenCodeDecoder`` +directly through ``coder_eval.testing.replay`` under ``cli_epoch_ms``: the +windows come from the scripted envelope stamps, never from the clock. The fixtures below mirror event lines CAPTURED FROM A LIVE ``opencode run --format json`` — the CLI's own compact vocabulary (``step_start`` / @@ -28,19 +30,32 @@ from coder_eval.agents import opencode_agent as agent_module from coder_eval.agents.opencode_agent import ( OpenCodeAgent, - _OpenCodeTurnState, + _OpenCodeDecoder, _unwrap, ) -from coder_eval.errors import AgentCrashError, TurnTimeoutError -from coder_eval.models import AssistantMessage, CommandTelemetry, OpenCodeAgentConfig, PermissionMode, TokenUsage +from coder_eval.errors.agent import format_timeout_reason +from coder_eval.models import ( + AssistantMessage, + FileExistsCriterion, + OpenCodeAgentConfig, + PermissionMode, + RunLimits, + SandboxConfig, + TaskDefinition, + TimingBasis, + TurnRecord, + parse_agent_config, +) from coder_eval.orchestration.plugin_staging import stage_plugins +from coder_eval.orchestration.turn_monitor import TurnMonitor from coder_eval.pricing import calculate_cost -from coder_eval.streaming.collector import EventCollector +from coder_eval.streaming.emitter import TurnOutcome from coder_eval.streaming.events import ( AgentEndEvent, AgentEndStatus, AgentStartEvent, StopReason, + StreamEvent, ToolEndEvent, ToolEndStatus, ToolStartEvent, @@ -48,7 +63,10 @@ TurnEndStatus, TurnStartEvent, ) +from coder_eval.testing import Replay, ScriptedClock, Tick, assert_identity_closes, assert_stream_balanced, replay from tests._fixtures.golden_streams.opencode_fixtures import ( + _T0_MS, + CAPTURED_STREAM, HAPPY_STREAM, SESSION, _evt, @@ -87,43 +105,125 @@ async def fake_exec(*argv: str, **kwargs: Any) -> _FakeProcess: async def _run( agent: OpenCodeAgent, tmp_path: Any, prompt: str = "do the thing", *, plugin_root: Path | None = None, **kwargs: Any -): +) -> TurnOutcome: await agent.start(str(tmp_path), plugin_root=plugin_root) + kwargs.setdefault("iteration", 1) return await agent.communicate(prompt, **kwargs) +async def _record(agent: OpenCodeAgent, tmp_path: Any, prompt: str = "do the thing", **kwargs: Any) -> TurnRecord: + """The record of a turn that must not end CRASHED or TIMEOUT.""" + outcome = await _run(agent, tmp_path, prompt, **kwargs) + assert outcome.error is None, f"{outcome.status}: {outcome.error}" + return outcome.record + + def _agent(**overrides: Any) -> OpenCodeAgent: config = OpenCodeAgentConfig(type="opencode", **{"model": "deepseek/deepseek-v4-pro", **overrides}) return OpenCodeAgent(config, task_id="t1") +_BASE = datetime.fromtimestamp(_T0_MS / 1000) + + +def _at(ms: float) -> datetime: + """The naive-local instant of the CLI stamp ``_T0_MS + ms``, as the decoder converts it.""" + return datetime.fromtimestamp((_T0_MS + ms) / 1000) + + +def _event(event_type: str, part: dict[str, Any] | None = None, *, at_ms: int | None = 0) -> dict[str, Any]: + """One decoded CLI event; ``at_ms=None`` drops the envelope ``timestamp``.""" + event = json.loads(_evt(event_type, part or {}, at_ms=at_ms or 0)) + if at_ms is None: + del event["timestamp"] + return event + + +def _tool(call_id: str, status: str, *, at_ms: int, start: int | None = None, end: int | None = None) -> dict[str, Any]: + """A ``bash`` ``tool_use`` event whose ``state.time`` carries the given bounds, in ms after ``_T0_MS``.""" + times = {key: _T0_MS + ms for key, ms in (("start", start), ("end", end)) if ms is not None} + state: dict[str, Any] = {"status": status, "input": {"command": "ls"}} + if times: + state["time"] = times + return _event("tool_use", {"callID": call_id, "tool": "bash", "state": state}, at_ms=at_ms) + + +def _finish(at_ms: int | None) -> dict[str, Any]: + return _event("step_finish", {"reason": "stop", "tokens": {"input": 10, "output": 5}}, at_ms=at_ms) + + +def _replay( + stream: list[Any], *, status: AgentEndStatus = AgentEndStatus.COMPLETED, reason: str | None = None +) -> tuple[Replay, _OpenCodeDecoder]: + """Drive an `_OpenCodeDecoder` under `cli_epoch_ms` on a clock at `_BASE`; return the decoder too.""" + decoders: list[_OpenCodeDecoder] = [] + + def end(decoder: _OpenCodeDecoder) -> TurnOutcome: + decoders.append(decoder) + return decoder.end(status, reason=reason) + + result = replay(stream, _OpenCodeDecoder, clock=ScriptedClock(_BASE), basis=TimingBasis.CLI_EPOCH_MS, end=end) + return result, decoders[0] + + +def _assistants(result: Replay) -> list[AssistantMessage]: + return [m for m in result.record.messages if isinstance(m, AssistantMessage)] + + class TestEnvelopeNormalization: def test_part_envelope(self): """Normal events carry their payload under `part`.""" - t, part = _unwrap({"type": "step_finish", "sessionID": "s", "part": {"reason": "stop"}}) + t, part, stamp = _unwrap({"type": "step_finish", "sessionID": "s", "part": {"reason": "stop"}}) assert t == "step_finish" assert part["reason"] == "stop" + assert stamp is None def test_flat_envelope(self): """The CLI's own error path emits a flat object with no `part`.""" - t, props = _unwrap({"type": "error", "sessionID": "s", "error": {"name": "UnknownError"}}) + t, props, _ = _unwrap({"type": "error", "sessionID": "s", "error": {"name": "UnknownError"}}) assert t == "error" assert props["error"]["name"] == "UnknownError" + def test_the_envelope_stamp_is_returned(self): + """The envelope `timestamp` (epoch ms) is the CLI's own stamp for the event.""" + _, _, stamp = _unwrap({"type": "text", "timestamp": _T0_MS + 250, "part": {"text": "hi"}}) + assert stamp == _at(250) + class TestHappyPath: async def test_builds_turn_record(self, patch_exec, tmp_path): patch_exec(_FakeProcess(HAPPY_STREAM)) - record = await _run(_agent(), tmp_path) + outcome = await _run(_agent(), tmp_path) + record = outcome.record + assert outcome.status is AgentEndStatus.COMPLETED assert record.crashed is False assert record.agent_output == "Created the file." assert record.assistant_turn_count == 2 assert record.model_used == "deepseek/deepseek-v4-pro" + async def test_the_result_summary_carries_the_final_reply(self, patch_exec, tmp_path): + patch_exec(_FakeProcess(HAPPY_STREAM)) + record = await _record(_agent(), tmp_path) + + assert record.result_summary is not None + assert record.result_summary.is_error is False + assert record.result_summary.stop_reason == "stop" + assert record.result_summary.result == "Created the file." + + async def test_events_carry_no_session_thread_id(self, patch_exec, tmp_path): + """The session id is replayed via `--session`; it is not a sub-agent thread.""" + patch_exec(_FakeProcess(HAPPY_STREAM)) + recorder = _EventRecorder() + await _run(_agent(), tmp_path, stream_callback=recorder) + + assert recorder.events + assert all(e.thread_id is None for e in recorder.events) + assert_stream_balanced(recorder.events) + async def test_token_buckets_accumulate_across_steps(self, patch_exec, tmp_path): patch_exec(_FakeProcess(HAPPY_STREAM)) - record = await _run(_agent(), tmp_path) + record = await _record(_agent(), tmp_path) usage = record.token_usage assert usage is not None @@ -140,7 +240,7 @@ async def test_token_buckets_accumulate_across_steps(self, patch_exec, tmp_path) async def test_reconciliation_invariant(self, patch_exec, tmp_path): """Summing the four buckets across messages must equal token_usage exactly.""" patch_exec(_FakeProcess(HAPPY_STREAM)) - record = await _run(_agent(), tmp_path) + record = await _record(_agent(), tmp_path) usage = record.token_usage assert usage is not None @@ -151,13 +251,14 @@ async def test_reconciliation_invariant(self, patch_exec, tmp_path): async def test_tool_call_captured(self, patch_exec, tmp_path): patch_exec(_FakeProcess(HAPPY_STREAM)) - record = await _run(_agent(), tmp_path) + record = await _record(_agent(), tmp_path) assert len(record.commands) == 1 cmd = record.commands[0] # Normalized to the canonical vocabulary criteria are written against. assert cmd.tool_name == "Read" assert cmd.tool_id == "call_1" + assert cmd.sequence_number == 0 assert cmd.result_status == "success" # ...including the ARGUMENT keys: the fixture's native `filePath` is # recorded under Claude's `file_path` (see TestCrossHarnessNormalization). @@ -168,7 +269,7 @@ async def test_tool_call_captured(self, patch_exec, tmp_path): async def test_messages_attributed_to_steps(self, patch_exec, tmp_path): patch_exec(_FakeProcess(HAPPY_STREAM)) - record = await _run(_agent(), tmp_path) + record = await _record(_agent(), tmp_path) assistants = [m for m in record.messages if isinstance(m, AssistantMessage)] assert len(assistants) == 2 @@ -197,7 +298,7 @@ async def test_flat_convention_keeps_input_verbatim(self, patch_exec, tmp_path, ) patch_exec(_FakeProcess([step])) with caplog.at_level("WARNING"): - record = await _run(_agent(), tmp_path) + record = await _record(_agent(), tmp_path) usage = record.token_usage assert usage is not None @@ -211,7 +312,7 @@ async def test_nested_convention_subtracts_the_cache_buckets(self, patch_exec, t fresh slice must come back out or the cached portion is billed twice.""" patch_exec(_FakeProcess(HAPPY_STREAM)) # _tokens() builds nested totals with caplog.at_level("WARNING"): - record = await _run(_agent(), tmp_path) + record = await _record(_agent(), tmp_path) usage = record.token_usage assert usage is not None @@ -222,7 +323,7 @@ async def test_total_matching_neither_convention_warns(self, patch_exec, tmp_pat """nested=350, flat=8030, reported 8000 — the schema moved; keep `input`.""" patch_exec(_FakeProcess([self._step({"total": 8000, "input": 300, "output": 50, "cache": {"read": 7680}})])) with caplog.at_level("WARNING"): - record = await _run(_agent(), tmp_path) + record = await _record(_agent(), tmp_path) usage = record.token_usage assert usage is not None @@ -243,7 +344,7 @@ async def test_missing_total_with_cache_traffic_defaults_flat_but_warns(self, pa silent — but `input` is still taken verbatim, the live-verified convention.""" patch_exec(_FakeProcess([self._step({"input": 500, "output": 20, "cache": {"read": 200}})])) with caplog.at_level("WARNING"): - record = await _run(_agent(), tmp_path) + record = await _record(_agent(), tmp_path) usage = record.token_usage assert usage is not None @@ -255,7 +356,7 @@ async def test_missing_total_without_cache_traffic_is_silent(self, patch_exec, t """No arbiter but no cache either ⇒ the conventions agree; nothing to verify.""" patch_exec(_FakeProcess([self._step({"input": 500, "output": 20})])) with caplog.at_level("WARNING"): - record = await _run(_agent(), tmp_path) + record = await _record(_agent(), tmp_path) usage = record.token_usage assert usage is not None @@ -266,7 +367,7 @@ async def test_nested_total_contradicted_by_small_input_warns(self, patch_exec, """`total` says nested but input < cache: self-contradictory; keep `input`.""" patch_exec(_FakeProcess([self._step({"total": 350, "input": 300, "output": 50, "cache": {"read": 7680}})])) with caplog.at_level("WARNING"): - record = await _run(_agent(), tmp_path) + record = await _record(_agent(), tmp_path) usage = record.token_usage assert usage is not None @@ -278,7 +379,7 @@ class TestCostFallsBackToTheRateCard: async def test_stream_cost_wins_when_reported(self, patch_exec, tmp_path): """The provider's own accounting beats a static headline rate.""" patch_exec(_FakeProcess(HAPPY_STREAM)) - record = await _run(_agent(), tmp_path) + record = await _record(_agent(), tmp_path) assert record.token_usage is not None assert record.token_usage.total_cost_usd == pytest.approx(0.003) # 0.001 + 0.002 @@ -290,7 +391,7 @@ async def test_missing_cost_is_priced_from_the_rate_card(self, patch_exec, tmp_p _evt("step_finish", {"id": "prt_2", "messageID": "msg_1", "reason": "stop", "tokens": _tokens(1000, 500)}), ] patch_exec(_FakeProcess(stream)) - record = await _run(_agent(), tmp_path) + record = await _record(_agent(), tmp_path) assert record.token_usage is not None expected = calculate_cost("deepseek/deepseek-v4-pro", uncached_input_tokens=1000, output_tokens=500) @@ -300,7 +401,7 @@ async def test_missing_cost_is_priced_from_the_rate_card(self, patch_exec, tmp_p async def test_unpriced_model_reports_no_cost(self, patch_exec, tmp_path): """`None` (not 0.0) so "unpriceable" stays distinct from "ran for free".""" patch_exec(_FakeProcess([_evt("step_finish", {"id": "p", "reason": "stop", "tokens": _tokens(10, 5)})])) - record = await _run(_agent(model="nowhere/not-a-real-model"), tmp_path) + record = await _record(_agent(model="nowhere/not-a-real-model"), tmp_path) assert record.token_usage is not None assert record.token_usage.total_cost_usd is None @@ -316,14 +417,14 @@ async def test_zero_reported_cost_on_a_priced_model_uses_the_rate_card(self, pat ), ] patch_exec(_FakeProcess(stream)) - with caplog.at_level("WARNING"): - record = await _run(_agent(), tmp_path) + with caplog.at_level("DEBUG", logger="coder_eval.pricing"): + record = await _record(_agent(), tmp_path) expected = calculate_cost("deepseek/deepseek-v4-pro", uncached_input_tokens=1000, output_tokens=500) assert expected is not None and expected > 0 assert record.token_usage is not None assert record.token_usage.total_cost_usd == pytest.approx(expected) - assert "not understated" in caplog.text + assert "using the rate card" in caplog.text async def test_zero_reported_cost_on_an_unpriced_model_stays_zero(self, patch_exec, tmp_path): """With no rate to fall back to, the stream's 0 is the best information we have.""" @@ -335,7 +436,7 @@ async def test_zero_reported_cost_on_an_unpriced_model_stays_zero(self, patch_ex ), ] patch_exec(_FakeProcess(stream)) - record = await _run(_agent(model="nowhere/not-a-real-model"), tmp_path) + record = await _record(_agent(model="nowhere/not-a-real-model"), tmp_path) assert record.token_usage is not None assert record.token_usage.total_cost_usd == 0.0 @@ -362,7 +463,7 @@ async def test_native_names_map_to_canonical(self, patch_exec, tmp_path): `parameters["command"]` only for that name — OpenCode's `bash` would match nothing and fall back to raw-JSON matching.""" patch_exec(_FakeProcess([self._tool_event("bash"), self._tool_event("write")])) - record = await _run(_agent(), tmp_path) + record = await _record(_agent(), tmp_path) assert [c.tool_name for c in record.commands] == ["Bash", "Write"] async def test_gpt_family_apply_patch_maps_to_write(self, patch_exec, tmp_path): @@ -377,20 +478,20 @@ async def test_gpt_family_apply_patch_maps_to_write(self, patch_exec, tmp_path): `_TOOL_ITEM_NAMES["apply_patch"] = "Write"`. """ patch_exec(_FakeProcess([self._tool_with_input("apply_patch", {"patchText": "*** Begin Patch\n"})])) - record = await _run(_agent(), tmp_path) + record = await _record(_agent(), tmp_path) assert [c.tool_name for c in record.commands] == ["Write"] async def test_unknown_tool_passes_through(self, patch_exec, tmp_path): """An unmapped tool still surfaces under its own name rather than vanishing.""" patch_exec(_FakeProcess([self._tool_event("some_new_tool")])) - record = await _run(_agent(), tmp_path) + record = await _record(_agent(), tmp_path) assert [c.tool_name for c in record.commands] == ["some_new_tool"] async def test_native_skill_tool_maps_to_canonical_skill(self, patch_exec, tmp_path): """`skill_triggered` keys on the canonical `Skill`; OpenCode emits lowercase `skill`, so without the mapping a real engagement scores as a miss.""" patch_exec(_FakeProcess([self._tool_event("skill")])) - record = await _run(_agent(), tmp_path) + record = await _record(_agent(), tmp_path) assert [c.tool_name for c in record.commands] == ["Skill"] @staticmethod @@ -433,7 +534,7 @@ async def test_argument_keys_map_to_canonical(self, patch_exec, tmp_path, tool, and scores 0 on OpenCode for identical agent behaviour. """ patch_exec(_FakeProcess([self._tool_with_input(tool, native)])) - record = await _run(_agent(), tmp_path) + record = await _record(_agent(), tmp_path) assert record.commands[0].parameters == expected @pytest.mark.parametrize( @@ -449,7 +550,7 @@ async def test_already_canonical_keys_are_left_alone(self, patch_exec, tmp_path, """The rename is per-tool: `path` means `file_path` on Read/Write/Edit and stays `path` on the search tools, which is exactly Claude's split.""" patch_exec(_FakeProcess([self._tool_with_input(tool, native)])) - record = await _run(_agent(), tmp_path) + record = await _record(_agent(), tmp_path) assert record.commands[0].parameters == native @@ -493,14 +594,16 @@ async def test_the_completion_supplies_the_parameters(self, patch_exec, tmp_path ] ) ) - record = await _run(_agent(), tmp_path) + record = await _record(_agent(), tmp_path) assert len(record.commands) == 1 # one tool, not two cmd = record.commands[0] assert cmd.tool_name == "Bash" assert cmd.parameters == {"command": "pytest -q"} assert cmd.result_status == "success" - assert cmd.execution_started_at is not None + assert cmd.execution_completed_at == datetime.fromtimestamp(1786663018231 / 1000.0) + assert cmd.execution_started_at is not None, "a start that arrives only with the result still times the call" + assert cmd.duration_ms is not None async def test_one_tool_start_end_pair_is_emitted(self, patch_exec, tmp_path): patch_exec( @@ -543,10 +646,12 @@ async def test_the_completion_time_end_becomes_execution_completed_at(self, patc ] ) ) - record = await _run(_agent(), tmp_path) + record = await _record(_agent(), tmp_path) cmd = record.commands[0] assert cmd.execution_completed_at == datetime.fromtimestamp(1786663018231 / 1000.0) + assert cmd.execution_started_at is not None, "a start that arrives only with the result still times the call" + assert cmd.duration_ms is not None assert cmd.execution_started_at == datetime.fromtimestamp(1786663018214 / 1000.0) assert cmd.duration_ms == pytest.approx(17.0) @@ -561,7 +666,7 @@ async def test_a_later_event_without_input_never_clears_what_we_have(self, patch ] ) ) - record = await _run(_agent(), tmp_path) + record = await _record(_agent(), tmp_path) assert record.commands[0].parameters == {"command": "ls"} @@ -607,7 +712,7 @@ async def test_prepends_mock_dirs_ahead_of_the_inherited_path(self, patch_exec, captured = patch_exec(_FakeProcess(HAPPY_STREAM)) agent = _agent() await agent.start(str(tmp_path), env_path_prepend=["/sandbox/mocks", "/sandbox/bins"]) - await agent.communicate("do the thing") + await agent.communicate("do the thing", iteration=1) expected = os.pathsep.join(["/sandbox/mocks", "/sandbox/bins", "/parent/bin"]) assert captured["kwargs"]["env"]["PATH"] == expected @@ -617,7 +722,7 @@ async def test_plugin_tools_dir_is_exported(self, patch_exec, tmp_path, monkeypa captured = patch_exec(_FakeProcess(HAPPY_STREAM)) agent = _agent() await agent.start(str(tmp_path), plugin_tools_dir="/sandbox/tools") - await agent.communicate("do the thing") + await agent.communicate("do the thing", iteration=1) assert captured["kwargs"]["env"]["PLUGIN_TOOLS_DIR"] == "/sandbox/tools" @@ -627,7 +732,7 @@ async def test_inherited_plugin_tools_dir_wins(self, patch_exec, tmp_path, monke captured = patch_exec(_FakeProcess(HAPPY_STREAM)) agent = _agent() await agent.start(str(tmp_path), plugin_tools_dir="/sandbox/tools") - await agent.communicate("do the thing") + await agent.communicate("do the thing", iteration=1) assert captured["kwargs"]["env"]["PLUGIN_TOOLS_DIR"] == "/host/tools" @@ -805,14 +910,17 @@ async def test_inherited_wildcard_allow_cannot_outrank_our_allowlist(self, patch assert rules[0] == ("webfetch", "allow") assert rules[1] == ("*", "deny") - async def test_an_allowlist_keeps_a_host_rule_for_a_non_tool_permission(self, patch_exec, tmp_path, monkeypatch): - monkeypatch.setenv("OPENCODE_CONFIG_CONTENT", json.dumps({"permission": {"external_directory": "deny"}})) + @pytest.mark.parametrize("host_rule", ["allow", "deny"]) + async def test_an_allowlist_keeps_a_host_rule_for_a_non_tool_permission( + self, patch_exec, tmp_path, monkeypatch, host_rule + ): + monkeypatch.setenv("OPENCODE_CONFIG_CONTENT", json.dumps({"permission": {"external_directory": host_rule}})) captured = patch_exec(_FakeProcess(HAPPY_STREAM)) await _run(_agent(allowed_tools=["Read"]), tmp_path) - rules = json.loads(captured["kwargs"]["env"]["OPENCODE_CONFIG_CONTENT"])["permission"] - assert rules["external_directory"] == "deny" - assert rules["doom_loop"] == "allow" - assert next(iter(rules)) == "external_directory" + rules = list(json.loads(captured["kwargs"]["env"]["OPENCODE_CONFIG_CONTENT"])["permission"].items()) + assert rules[0] == ("*", "deny") + assert rules[-1] == ("external_directory", host_rule) + assert ("doom_loop", "allow") in rules async def test_no_prompt_writes_no_file(self, patch_exec, tmp_path): captured = patch_exec(_FakeProcess(HAPPY_STREAM)) @@ -858,6 +966,12 @@ async def test_explicit_line_limit_is_passed(self, patch_exec, tmp_path): await _run(_agent(), tmp_path) assert captured["kwargs"]["limit"] > 64 * 1024 + async def test_the_cli_never_inherits_stdin(self, patch_exec, tmp_path): + """OpenCode reads a non-TTY stdin to EOF before it emits; an inherited open stdin stalls the turn.""" + captured = patch_exec(_FakeProcess(HAPPY_STREAM)) + await _run(_agent(), tmp_path) + assert captured["kwargs"]["stdin"] is asyncio.subprocess.DEVNULL + class TestSessionContinuity: async def test_first_turn_omits_session(self, patch_exec, tmp_path): @@ -874,7 +988,7 @@ async def test_second_turn_resumes_session(self, patch_exec, tmp_path): assert agent._session_id == SESSION captured2 = patch_exec(_FakeProcess(HAPPY_STREAM)) - await agent.communicate("follow up") + await agent.communicate("follow up", iteration=2) argv = captured2["argv"] assert argv[argv.index("--session") + 1] == SESSION @@ -907,10 +1021,6 @@ async def test_session_id_recorded_after_a_turn(self, patch_exec, tmp_path): class TestErrorEventShapes: """`error` is the CLI's own flat envelope, and its payload shape varies.""" - @staticmethod - def _state() -> _OpenCodeTurnState: - return _OpenCodeTurnState(task_id="t1", iteration=1, user_input="p", model="m") - @pytest.mark.parametrize( ("payload", "expected"), [ @@ -924,9 +1034,8 @@ def _state() -> _OpenCodeTurnState: ], ) def test_message_extraction(self, payload, expected): - state = self._state() - state.on_error(payload) - assert state.error_message == expected + _, decoder = _replay([{"type": "error", "sessionID": SESSION, **payload}]) + assert decoder.error == expected class TestTokenCastsNeverRaise: @@ -958,7 +1067,7 @@ def _stream(tokens: dict[str, Any]) -> list[str]: async def test_a_non_numeric_bucket_warns_instead_of_crashing(self, patch_exec, tmp_path, tokens, caplog): patch_exec(_FakeProcess(self._stream(tokens))) with caplog.at_level("WARNING"): - record = await _run(_agent(), tmp_path) + record = await _record(_agent(), tmp_path) assert record.crashed is False assert "unexpected token accounting" in caplog.text @@ -969,7 +1078,7 @@ async def test_numeric_strings_are_still_accepted(self, patch_exec, tmp_path, ca """A stringly-typed but numeric count is a serialization detail, not drift.""" patch_exec(_FakeProcess(self._stream({"input": "100", "output": "20", "total": 120}))) with caplog.at_level("WARNING"): - record = await _run(_agent(), tmp_path) + record = await _record(_agent(), tmp_path) assert record.token_usage is not None assert record.token_usage.uncached_input_tokens == 100 @@ -980,7 +1089,7 @@ async def test_a_float_count_truncates(self, patch_exec, tmp_path, caplog): """JSON has one number type, so a provider may serialize a count as 100.0.""" patch_exec(_FakeProcess(self._stream({"input": 100.0, "output": 20.7, "total": 120}))) with caplog.at_level("WARNING"): - record = await _run(_agent(), tmp_path) + record = await _record(_agent(), tmp_path) assert record.token_usage is not None assert record.token_usage.uncached_input_tokens == 100 @@ -991,15 +1100,24 @@ async def test_a_bool_is_not_a_token_count(self, patch_exec, tmp_path, caplog): """`int(True) == 1` would book a phantom token.""" patch_exec(_FakeProcess(self._stream({"input": True, "output": 20}))) with caplog.at_level("WARNING"): - record = await _run(_agent(), tmp_path) + record = await _record(_agent(), tmp_path) assert record.token_usage is not None assert record.token_usage.uncached_input_tokens == 0 assert "unexpected token accounting" in caplog.text +_ERROR_LINE = json.dumps( + { + "type": "error", + "sessionID": SESSION, + "error": {"name": "UnknownError", "data": {"message": "provider exploded"}}, + } +) + + class TestFailurePaths: - async def test_error_event_raises_and_parks_partial(self, patch_exec, tmp_path): + async def test_error_event_then_clean_exit_crashes_with_the_clis_message(self, patch_exec, tmp_path): stream = [ _evt("step_start", {"id": "prt_1", "messageID": "msg_1", "type": "step-start"}), _evt( @@ -1013,36 +1131,40 @@ async def test_error_event_raises_and_parks_partial(self, patch_exec, tmp_path): "state": {"status": "running", "input": {"command": "ls"}}, }, ), - json.dumps( - { - "type": "error", - "sessionID": SESSION, - "error": {"name": "UnknownError", "data": {"message": "provider exploded"}}, - } - ), + _ERROR_LINE, ] - patch_exec(_FakeProcess(stream)) + patch_exec(_FakeProcess(stream, returncode=0)) agent = _agent() - with pytest.raises(AgentCrashError, match="provider exploded"): - await _run(agent, tmp_path) + outcome = await _run(agent, tmp_path) - partial = agent.pending_turn - assert partial is not None + assert outcome.status is AgentEndStatus.CRASHED + assert outcome.error == "OpenCode error: provider exploded" + partial = outcome.record assert partial.crashed is True + assert partial.result_summary is None # The in-flight tool was force-closed rather than dropped. assert [c.result_status for c in partial.commands] == ["unknown"] + async def test_a_stream_error_crashes_even_when_a_stop_was_requested(self, patch_exec, tmp_path): + """OpenCode's `error` event is final: an error read on the line a stop fires on is still a crash.""" + patch_exec(_RunningProcess([_ERROR_LINE])) + outcome = await _run(_agent(), tmp_path, should_stop=lambda: StopReason.TOOL_CALL_CAP) + + assert outcome.status is AgentEndStatus.CRASHED + assert outcome.error is not None and outcome.error.startswith("OpenCode error:") + async def test_nonzero_exit_without_error_event_crashes(self, patch_exec, tmp_path): patch_exec(_FakeProcess([], returncode=1, stderr=b"boom: bad model")) - with pytest.raises(AgentCrashError, match="boom: bad model"): - await _run(_agent(), tmp_path) + outcome = await _run(_agent(), tmp_path) + assert outcome.status is AgentEndStatus.CRASHED + assert outcome.error == "OpenCode exited non-zero: boom: bad model" async def test_malformed_line_is_skipped(self, patch_exec, tmp_path): """Non-JSON noise on stdout must not kill the turn.""" stream = ["warn: CPU lacks AVX support", *HAPPY_STREAM] patch_exec(_FakeProcess(stream)) - record = await _run(_agent(), tmp_path) + record = await _record(_agent(), tmp_path) assert record.crashed is False assert record.assistant_turn_count == 2 @@ -1080,20 +1202,24 @@ async def test_unrecognized_vocabulary_crashes_and_names_the_types(self, patch_e patch_exec(_FakeProcess(stream)) agent = _agent() - with pytest.raises(AgentCrashError, match="no recognized events") as exc: - await _run(agent, tmp_path) + outcome = await _run(agent, tmp_path) + + assert outcome.status is AgentEndStatus.CRASHED + assert outcome.error is not None and "no recognized events" in outcome.error # The crash names what it DID see, for diagnosis. - assert "session.next.step.ended" in str(exc.value) + assert "session.next.step.ended" in outcome.error - partial = agent.pending_turn + partial = outcome.record assert partial is not None assert partial.crashed is True async def test_empty_stdout_with_clean_exit_crashes(self, patch_exec, tmp_path): """Zero events at all is the same zero-telemetry hole as wrong vocabulary.""" patch_exec(_FakeProcess([], returncode=0)) - with pytest.raises(AgentCrashError, match="no recognized events"): - await _run(_agent(), tmp_path) + outcome = await _run(_agent(), tmp_path) + assert outcome.status is AgentEndStatus.CRASHED + assert outcome.error is not None + assert outcome.error.startswith("OpenCode exited cleanly but the turn captured no recognized events.") async def test_intentional_cuts_are_exempt(self, patch_exec, tmp_path): """A cooperative stop can land before the first recognized event; that is @@ -1101,7 +1227,7 @@ async def test_intentional_cuts_are_exempt(self, patch_exec, tmp_path): stream = [json.dumps({"id": "evt_1", "type": "session.next.idle", "properties": {"sessionID": SESSION}})] proc = _RunningProcess(stream) patch_exec(proc) - record = await _run(_agent(), tmp_path, should_stop=lambda: StopReason.EARLY_CRITERION) + record = await _record(_agent(), tmp_path, should_stop=lambda: StopReason.EARLY_CRITERION) assert record.crashed is False @staticmethod @@ -1126,24 +1252,29 @@ async def test_finished_step_without_tokens_crashes(self, patch_exec, tmp_path): patch_exec(_FakeProcess(self._stream_without_tokens())) agent = _agent() - with pytest.raises(AgentCrashError, match="zero token telemetry") as exc: - await _run(agent, tmp_path) - assert "1 finished step(s)" in str(exc.value) - assert agent.pending_turn is not None # telemetry captured so far still parked + outcome = await _run(agent, tmp_path) + + assert outcome.status is AgentEndStatus.CRASHED + assert outcome.error is not None + assert "zero token telemetry" in outcome.error + assert "1 finished step(s)" in outcome.error + assert outcome.record.crashed is True # telemetry captured so far is still on the record + assert len(outcome.record.messages) >= 1 async def test_cost_without_tokens_still_crashes(self, patch_exec, tmp_path): """Reported cost does not excuse missing tokens: the USD gate might trip, but every token gate and aggregate is still silently blind.""" patch_exec(_FakeProcess(self._stream_without_tokens(cost=0.004))) - with pytest.raises(AgentCrashError, match="cost reported: yes"): - await _run(_agent(), tmp_path) + outcome = await _run(_agent(), tmp_path) + assert outcome.status is AgentEndStatus.CRASHED + assert outcome.error is not None and "cost reported: yes" in outcome.error async def test_require_token_telemetry_false_warns_and_scores(self, patch_exec, tmp_path, caplog): """The escape hatch, for a provider that genuinely reports no usage: crashing every turn there would make the harness unusable, not merely imprecise.""" patch_exec(_FakeProcess(self._stream_without_tokens())) with caplog.at_level("WARNING"): - record = await _run(_agent(require_token_telemetry=False), tmp_path) + record = await _record(_agent(require_token_telemetry=False), tmp_path) assert record.crashed is False assert "require_token_telemetry is off" in caplog.text @@ -1152,21 +1283,22 @@ async def test_the_hatch_never_relaxes_the_vocabulary_check(self, patch_exec, tm """Vocabulary drift has silently zeroed a whole run before, and no provider quirk explains it — so this arm stays fatal even with the hatch open.""" patch_exec(_FakeProcess([json.dumps({"type": "session.next.idle", "properties": {"sessionID": SESSION}})])) - with pytest.raises(AgentCrashError, match="no recognized events"): - await _run(_agent(require_token_telemetry=False), tmp_path) + outcome = await _run(_agent(require_token_telemetry=False), tmp_path) + assert outcome.status is AgentEndStatus.CRASHED + assert outcome.error is not None and "no recognized events" in outcome.error async def test_a_cut_before_any_step_finished_is_exempt(self, patch_exec, tmp_path): """The arm keys on a step the CLI reported FINISHED. A stop landing between a step's start and its `step_finish` is an intentional cut, not drift.""" proc = _RunningProcess([_evt("step_start", {"id": "prt_1", "messageID": "msg_1", "type": "step-start"})]) patch_exec(proc) - record = await _run(_agent(), tmp_path, should_stop=lambda: StopReason.EARLY_CRITERION) + record = await _record(_agent(), tmp_path, should_stop=lambda: StopReason.EARLY_CRITERION) assert record.crashed is False async def test_real_tokens_are_never_condemned(self, patch_exec, tmp_path): """The guard must not fire on the ordinary path it lives beside.""" patch_exec(_FakeProcess(HAPPY_STREAM)) - record = await _run(_agent(), tmp_path) + record = await _record(_agent(), tmp_path) assert record.crashed is False assert record.token_usage is not None @@ -1195,15 +1327,14 @@ def on_event(self, event: Any) -> None: class TestUnexpectedErrorContract: - """An unanticipated exception must still honor the pending-turn contract. + """An unanticipated exception must still end the turn as a crashed outcome. - Escaping raw would break it three ways: no terminal ``AgentEndEvent`` (an - unbalanced event tree for every renderer), captured telemetry dropped instead - of parked on ``pending_turn``, and ``_iteration`` left incremented because the - orchestrator never reaches ``discard_pending_turn``. + Escaping raw would break it two ways: no terminal ``AgentEndEvent`` (an + unbalanced event tree for every renderer), and captured telemetry dropped + instead of kept on the crashed record. """ - async def test_stream_error_becomes_a_crash_with_partial_parked(self, patch_exec, tmp_path): + async def test_stream_error_becomes_a_crash_with_the_crashed_partial(self, patch_exec, tmp_path): stream = [ _evt("step_start", {"id": "prt_1", "messageID": "msg_1", "type": "step-start"}), _evt( @@ -1221,11 +1352,11 @@ async def test_stream_error_becomes_a_crash_with_partial_parked(self, patch_exec patch_exec(_ExplodingProcess(stream)) agent = _agent() - with pytest.raises(AgentCrashError, match="OpenCode turn failed"): - await _run(agent, tmp_path) + outcome = await _run(agent, tmp_path) - partial = agent.pending_turn - assert partial is not None + assert outcome.status is AgentEndStatus.CRASHED + assert outcome.error is not None and outcome.error.startswith("OpenCode turn failed: ") + partial = outcome.record assert partial.crashed is True # Telemetry captured before the failure survives, orphan tool force-closed. assert [c.result_status for c in partial.commands] == ["unknown"] @@ -1239,16 +1370,17 @@ async def boom(*_argv: str, **_kwargs: Any): monkeypatch.setattr(asyncio, "create_subprocess_exec", boom) monkeypatch.setattr("shutil.which", lambda _name: "/usr/local/bin/opencode") - with pytest.raises(AgentCrashError, match="no fork for you"): - await _run(_agent(), tmp_path) + outcome = await _run(_agent(), tmp_path) + assert outcome.status is AgentEndStatus.CRASHED + assert outcome.error == "OpenCode turn failed: no fork for you" async def test_terminal_event_is_emitted_exactly_once(self, patch_exec, tmp_path): """The protocol allows exactly one AgentEnd per communicate(), crash included.""" patch_exec(_ExplodingProcess([_evt("step_start", {"id": "prt_1", "messageID": "msg_1"})])) recorder = _EventRecorder() - with pytest.raises(AgentCrashError): - await _run(_agent(), tmp_path, stream_callback=recorder) + outcome = await _run(_agent(), tmp_path, stream_callback=recorder) + assert outcome.status is AgentEndStatus.CRASHED seen = recorder.events assert len([e for e in seen if isinstance(e, AgentStartEvent)]) == 1 @@ -1257,18 +1389,6 @@ async def test_terminal_event_is_emitted_exactly_once(self, patch_exec, tmp_path assert ends[0].crashed is True assert ends[0].status is AgentEndStatus.CRASHED - async def test_iteration_rolls_back_after_the_crash(self, patch_exec, tmp_path): - """`discard_pending_turn` must find the bump it needs to undo.""" - patch_exec(_ExplodingProcess([])) - agent = _agent() - - with pytest.raises(AgentCrashError): - await _run(agent, tmp_path) - assert agent._iteration == 1 - await agent.discard_pending_turn() - assert agent._iteration == 0 - assert agent.pending_turn is None - class _LeakyPipeProcess(_FakeProcess): """Replays events, then never signals EOF — the real CLI's behavior. @@ -1295,7 +1415,7 @@ class TestLeakedPipeDrain: async def test_completes_without_eof(self, patch_exec, tmp_path): """A stdout pipe that never closes must not stall the turn.""" patch_exec(_LeakyPipeProcess(HAPPY_STREAM)) - record = await asyncio.wait_for(_run(_agent(), tmp_path, timeout=300), timeout=30) + record = await asyncio.wait_for(_record(_agent(), tmp_path, timeout=300), timeout=30) assert record.crashed is False assert record.assistant_turn_count == 2 @@ -1327,7 +1447,7 @@ class TestStderrIsDrainedConcurrently: async def test_turn_completes_under_stderr_backpressure(self, patch_exec, tmp_path): patch_exec(_StderrBackpressureProcess(HAPPY_STREAM, stderr=b"noisy")) # Bounded so a regression fails here instead of hanging the suite. - record = await asyncio.wait_for(_run(_agent(), tmp_path, timeout=300), timeout=10) + record = await asyncio.wait_for(_record(_agent(), tmp_path, timeout=300), timeout=10) assert record.assistant_turn_count == 2 assert record.crashed is False @@ -1353,10 +1473,12 @@ async def test_early_criterion_ends_turn_stopped_early(self, patch_exec, tmp_pat proc = _RunningProcess(HAPPY_STREAM) patch_exec(proc) recorder = _EventRecorder() - record = await _run( + outcome = await _run( _agent(), tmp_path, should_stop=lambda: StopReason.EARLY_CRITERION, stream_callback=recorder ) + record = outcome.record + assert outcome.status is AgentEndStatus.STOPPED_EARLY assert record.crashed is False assert proc.terminated is True # Stopped at the first event boundary rather than draining the stream. @@ -1368,7 +1490,9 @@ async def test_tool_call_cap_ends_turn_tool_calls_exhausted(self, patch_exec, tm proc = _RunningProcess(HAPPY_STREAM) patch_exec(proc) recorder = _EventRecorder() - record = await _run(_agent(), tmp_path, should_stop=lambda: StopReason.TOOL_CALL_CAP, stream_callback=recorder) + record = await _record( + _agent(), tmp_path, should_stop=lambda: StopReason.TOOL_CALL_CAP, stream_callback=recorder + ) assert proc.terminated is True assert record.crashed is False @@ -1380,7 +1504,9 @@ async def test_token_budget_ends_turn_token_budget_exceeded(self, patch_exec, tm proc = _RunningProcess(HAPPY_STREAM) patch_exec(proc) recorder = _EventRecorder() - record = await _run(_agent(), tmp_path, should_stop=lambda: StopReason.TOKEN_BUDGET, stream_callback=recorder) + record = await _record( + _agent(), tmp_path, should_stop=lambda: StopReason.TOKEN_BUDGET, stream_callback=recorder + ) assert proc.terminated is True assert record.crashed is False @@ -1390,7 +1516,7 @@ async def test_token_budget_ends_turn_token_budget_exceeded(self, patch_exec, tm async def test_no_stop_is_uncapped(self, patch_exec, tmp_path): patch_exec(_FakeProcess(HAPPY_STREAM)) - record = await _run(_agent(), tmp_path, should_stop=lambda: None) + record = await _record(_agent(), tmp_path, should_stop=lambda: None) assert record.tool_calls_exhausted is False assert record.assistant_turn_count == 2 @@ -1403,7 +1529,7 @@ async def test_the_deciding_step_is_kept_whole(self, patch_exec, tmp_path): """ patch_exec(_RunningProcess(HAPPY_STREAM)) recorder = _EventRecorder() - record = await _run( + record = await _record( _agent(), tmp_path, should_stop=_stop_after(3, StopReason.TOOL_CALL_CAP), stream_callback=recorder ) @@ -1422,7 +1548,7 @@ async def test_the_deciding_step_is_kept_whole(self, patch_exec, tmp_path): async def test_an_intentional_stop_is_exempt_from_a_non_zero_exit(self, patch_exec, tmp_path): """Killing the CLI makes it exit non-zero; that must not crash an intentional stop.""" patch_exec(_RunningProcess(HAPPY_STREAM, returncode=-15, stderr=b"terminated")) - record = await _run(_agent(), tmp_path, should_stop=_stop_after(3, StopReason.TOOL_CALL_CAP)) + record = await _record(_agent(), tmp_path, should_stop=_stop_after(3, StopReason.TOOL_CALL_CAP)) assert record.crashed is False assert record.tool_calls_exhausted is True @@ -1480,8 +1606,8 @@ class _EofNoExitProcess(_HangingProcess): """Replays its lines, signals EOF — but never exits until killed. Models a CLI that closed its stream during shutdown and then wedged: the one - window where the read loop is already done, so only a bounded reap in - ``_settle_turn`` stands between the turn and an unbounded hang. + window where the read loop is already done, so only the bounded post-EOF + exit wait in the transport's settle stands between the turn and an unbounded hang. """ async def readline(self) -> bytes: @@ -1491,28 +1617,26 @@ async def readline(self) -> bytes: class TestTimeoutContract: - async def test_deadline_raises_turn_timeout_with_partial_parked(self, patch_exec, tmp_path): - """A wedged CLI must yield TurnTimeoutError + a crashed partial record, + async def test_deadline_returns_a_timeout_outcome_with_the_partial(self, patch_exec, tmp_path): + """A wedged CLI must yield a TIMEOUT outcome with a crashed partial record, with exactly one terminal AgentEndEvent (status TIMEOUT) emitted.""" proc = _HangingProcess([_evt("step_start", {"id": "prt_1", "messageID": "msg_1"})]) patch_exec(proc) agent = _agent() recorder = _EventRecorder() - with pytest.raises(TurnTimeoutError): - await _run(agent, tmp_path, timeout=0.2, stream_callback=recorder) + outcome = await _run(agent, tmp_path, timeout=0.2, stream_callback=recorder) - partial = agent.pending_turn - assert partial is not None + assert outcome.status is AgentEndStatus.TIMEOUT + assert outcome.error == format_timeout_reason(0.2) + partial = outcome.record assert partial.crashed is True + assert partial.result_summary is None assert proc.terminated is True # the CLI was torn down, not abandoned ends = [e for e in recorder.events if isinstance(e, AgentEndEvent)] assert len(ends) == 1 assert ends[0].status is AgentEndStatus.TIMEOUT - await agent.discard_pending_turn() - assert agent._iteration == 0 # the failed turn's bump was rolled back - async def test_eof_without_exit_hits_the_deadline(self, patch_exec, tmp_path): """Stream closed, process wedged: the post-EOF reap must be bounded by the turn deadline instead of waiting for an exit that never comes.""" @@ -1520,12 +1644,11 @@ async def test_eof_without_exit_hits_the_deadline(self, patch_exec, tmp_path): patch_exec(proc) agent = _agent() - with pytest.raises(TurnTimeoutError): - await asyncio.wait_for(_run(agent, tmp_path, timeout=0.3), timeout=10) + outcome = await asyncio.wait_for(_run(agent, tmp_path, timeout=0.3), timeout=10) + assert outcome.status is AgentEndStatus.TIMEOUT # Everything parsed before the wedge survives on the partial record. - partial = agent.pending_turn - assert partial is not None + partial = outcome.record assert partial.crashed is True assert partial.token_usage is not None assert partial.token_usage.output_tokens > 0 @@ -1533,37 +1656,39 @@ async def test_eof_without_exit_hits_the_deadline(self, patch_exec, tmp_path): async def test_eof_without_exit_and_no_deadline_crashes(self, patch_exec, monkeypatch, tmp_path): """With no turn deadline configured, the reap still gets a fixed grace — a stream-closed-but-wedged CLI is a crash, not an indefinite hang.""" - monkeypatch.setattr("coder_eval.agents.opencode_agent._TERM_GRACE_SECONDS", 0.1) + monkeypatch.setattr("coder_eval.agents._transport.subprocess_jsonl._EXIT_GRACE_SECONDS", 0.1) proc = _EofNoExitProcess(HAPPY_STREAM) patch_exec(proc) - with pytest.raises(AgentCrashError, match="did not exit"): - await asyncio.wait_for(_run(_agent(), tmp_path), timeout=10) + outcome = await asyncio.wait_for(_run(_agent(), tmp_path), timeout=10) + + assert outcome.status is AgentEndStatus.CRASHED + assert outcome.error is not None + assert outcome.error.startswith("OpenCode closed its event stream but did not exit within") + assert proc.terminated is True class TestExternalCancel: - async def test_cancel_parks_partial_and_reraises(self, patch_exec, tmp_path): + async def test_cancel_ends_the_turn_and_reraises(self, patch_exec, tmp_path): """The watchdog's CancelledError must not swallow captured telemetry: the - partial record is parked, the terminal event says CRASHED, and the - cancellation still propagates.""" + turn is ended, the terminal event says CRASHED, and the cancellation + still propagates.""" proc = _HangingProcess([_evt("step_start", {"id": "prt_1", "messageID": "msg_1"})]) patch_exec(proc) agent = _agent() await agent.start(str(tmp_path)) recorder = _EventRecorder() - task = asyncio.ensure_future(agent.communicate("do the thing", stream_callback=recorder)) + task = asyncio.ensure_future(agent.communicate("do the thing", iteration=1, stream_callback=recorder)) await asyncio.sleep(0.05) # let it spawn and read the first event task.cancel() with pytest.raises(asyncio.CancelledError): _ = await task # the await re-raises the cancellation; no value ever exists - partial = agent.pending_turn - assert partial is not None - assert partial.crashed is True ends = [e for e in recorder.events if isinstance(e, AgentEndEvent)] assert len(ends) == 1 assert ends[0].status is AgentEndStatus.CRASHED + assert ends[0].crashed is True assert ends[0].crash_reason == "turn cancelled" assert proc.killed is True # not abandoned mid-stream — see TestTurnAlwaysReapsTheCli @@ -1572,9 +1697,9 @@ class TestTurnEventsAreBalanced: """`Agent.communicate`'s contract is one TurnStart/TurnEnd pair per inner turn. `on_step_start` opens one per CLI step and `on_step_finish` closes it, but a - turn that dies (or is cut) between the two left the last TurnStartEvent open - forever — a task.log with `>>> Turn start` and no matching `--- Turn end`. - All three sibling agents close it from `finalize`. + turn that dies (or is cut) between the two must still close the last + TurnStartEvent — a task.log with `>>> Turn start` and no matching + `--- Turn end` is the defect. The emitter closes it at the end of the turn. """ @staticmethod @@ -1594,8 +1719,8 @@ async def test_a_timeout_closes_the_open_step(self, patch_exec, tmp_path): patch_exec(proc) recorder = _EventRecorder() - with pytest.raises(TurnTimeoutError): - await _run(_agent(), tmp_path, timeout=0.2, stream_callback=recorder) + outcome = await _run(_agent(), tmp_path, timeout=0.2, stream_callback=recorder) + assert outcome.status is AgentEndStatus.TIMEOUT assert self._pairs(recorder) == (1, 1) end = next(e for e in recorder.events if isinstance(e, TurnEndEvent)) @@ -1609,7 +1734,7 @@ async def test_a_cancel_closes_the_open_step(self, patch_exec, tmp_path): await agent.start(str(tmp_path)) recorder = _EventRecorder() - task = asyncio.ensure_future(agent.communicate("do the thing", stream_callback=recorder)) + task = asyncio.ensure_future(agent.communicate("do the thing", iteration=1, stream_callback=recorder)) await asyncio.sleep(0.05) task.cancel() with pytest.raises(asyncio.CancelledError): @@ -1623,8 +1748,8 @@ async def test_a_crash_closes_the_open_step(self, patch_exec, tmp_path): patch_exec(_ExplodingProcess([_evt("step_start", {"id": "prt_1", "messageID": "msg_1"})])) recorder = _EventRecorder() - with pytest.raises(AgentCrashError): - await _run(_agent(), tmp_path, stream_callback=recorder) + outcome = await _run(_agent(), tmp_path, stream_callback=recorder) + assert outcome.status is AgentEndStatus.CRASHED assert self._pairs(recorder) == (1, 1) @@ -1641,24 +1766,40 @@ async def test_a_clean_cut_closes_the_open_step(self, patch_exec, tmp_path): assert end.status is TurnEndStatus.STOPPED_EARLY async def test_a_completed_step_is_never_closed_twice(self, patch_exec, tmp_path): - """Unlike the siblings, completed steps close themselves in `on_step_finish`, - so `finalize` must fire ONLY for a straggler.""" + """Completed steps close themselves in `on_step_finish`, so the end of the + turn must close ONLY a straggler.""" patch_exec(_FakeProcess(HAPPY_STREAM)) recorder = _EventRecorder() - record = await _run(_agent(), tmp_path, stream_callback=recorder) + record = await _record(_agent(), tmp_path, stream_callback=recorder) assert record.crashed is False starts, ends = self._pairs(recorder) assert starts == ends == 2 assert all(e.status is TurnEndStatus.COMPLETED for e in recorder.events if isinstance(e, TurnEndEvent)) + def test_a_dangling_step_is_closed_crashed_at_the_next_step_start(self): + result, _ = _replay( + [ + _event("step_start", {"messageID": "m1"}, at_ms=0), + _event("step_start", {"messageID": "m2"}, at_ms=100), + _finish(200), + ] + ) + + turn_ends = [e for e in result.events if isinstance(e, TurnEndEvent)] + assert [(e.turn_id, e.status) for e in turn_ends] == [ + ("m1", TurnEndStatus.CRASHED), + ("m2", TurnEndStatus.COMPLETED), + ] + assert_stream_balanced(result.events) + class TestTurnAlwaysReapsTheCli: """No exit from `communicate()` may leave the CLI running. - `AgentCrashError` is categorized AGENT_CRASH (max_retries=2) and the - orchestrator's attempt-failure hook only drains `pending_turn` — it never - kills the agent. An abandoned CLI therefore means attempt 2 spawns a SECOND + A crash is categorized AGENT_CRASH (max_retries=2), and the orchestrator + only appends the crashed record — it never kills the agent. An abandoned CLI + therefore means attempt 2 spawns a SECOND `opencode --dir --session ` while attempt 1 is still editing the very files the criteria are about to score, and whichever writer wins decides the task's result. @@ -1668,13 +1809,14 @@ class TestTurnAlwaysReapsTheCli: """ async def test_read_loop_crash_kills_the_cli(self, patch_exec, tmp_path): - """`_crash_turn` is synchronous and raises — nothing below it reaps.""" + """A read-loop crash ends the turn as an outcome, and `finally` still reaps the live CLI.""" proc = _ExplodingRunningProcess([_evt("step_start", {"id": "prt_1", "messageID": "msg_1"})]) patch_exec(proc) - with pytest.raises(AgentCrashError, match="OpenCode turn failed"): - await _run(_agent(), tmp_path) + outcome = await _run(_agent(), tmp_path) + assert outcome.status is AgentEndStatus.CRASHED + assert outcome.error is not None and outcome.error.startswith("OpenCode turn failed: ") assert proc.killed is True async def test_external_cancel_kills_the_cli(self, patch_exec, tmp_path): @@ -1685,7 +1827,7 @@ async def test_external_cancel_kills_the_cli(self, patch_exec, tmp_path): agent = _agent() await agent.start(str(tmp_path)) - task = asyncio.ensure_future(agent.communicate("do the thing")) + task = asyncio.ensure_future(agent.communicate("do the thing", iteration=1)) await asyncio.sleep(0.05) # let it spawn and read the first event task.cancel() with pytest.raises(asyncio.CancelledError): @@ -1693,15 +1835,14 @@ async def test_external_cancel_kills_the_cli(self, patch_exec, tmp_path): assert proc.killed is True - async def test_a_clean_turn_kills_nothing(self, patch_exec, tmp_path): - """The happy path is unchanged: the CLI exited, so the guard is a no-op - and the server child survives for the next turn's `--session` resume.""" + async def test_a_clean_turn_sweeps_only_the_group(self, patch_exec, tmp_path): + """The CLI exited, so it is not killed; the server child it left is swept with the turn.""" proc = _FakeProcess(HAPPY_STREAM) captured = patch_exec(proc) await _run(_agent(), tmp_path) assert proc.killed is False - assert captured["killpg"] == [] + assert captured["killpg"] == [(4242, signal.SIGKILL)] async def test_a_spawn_failure_has_no_process_to_reap(self, monkeypatch, tmp_path): """`proc` is unbound on this path; the guard must not raise NameError over it.""" @@ -1712,8 +1853,9 @@ async def boom(*_argv: str, **_kwargs: Any): monkeypatch.setattr(asyncio, "create_subprocess_exec", boom) monkeypatch.setattr("shutil.which", lambda _name: "/usr/local/bin/opencode") - with pytest.raises(AgentCrashError, match="no fork for you"): - await _run(_agent(), tmp_path) + outcome = await _run(_agent(), tmp_path) + assert outcome.status is AgentEndStatus.CRASHED + assert outcome.error is not None and "no fork for you" in outcome.error @pytest.mark.skipif(os.name != "posix", reason="process-group teardown (killpg/SIGKILL) is POSIX-only by design") @@ -1725,24 +1867,24 @@ async def test_spawn_uses_its_own_session(self, patch_exec, tmp_path): await _run(_agent(), tmp_path) assert captured["kwargs"]["start_new_session"] is (os.name == "posix") - async def test_stop_sweeps_the_spawned_group(self, patch_exec, tmp_path): - """`opencode run` leaves a server child holding the pipes; stop() must - SIGKILL the whole group or every task in a batch leaks one.""" + async def test_a_clean_turn_sweeps_the_spawned_group_once(self, patch_exec, tmp_path): + """`opencode run` leaves a server child holding the pipes; the turn must + SIGKILL the whole group, and stop() must not signal a pgid that may be reused.""" captured = patch_exec(_FakeProcess(HAPPY_STREAM)) agent = _agent() await _run(agent, tmp_path) - assert captured["killpg"] == [] # a clean turn does not kill mid-run state + assert captured["killpg"] == [(4242, signal.SIGKILL)] await agent.stop() - assert (4242, signal.SIGKILL) in captured["killpg"] + assert captured["killpg"] == [(4242, signal.SIGKILL)] async def test_a_crashed_turn_sweeps_the_group_too(self, patch_exec, tmp_path): """Killing the CLI pid alone would orphan the server child it left holding the pipes — across a retried batch, that is the leak that compounds.""" captured = patch_exec(_ExplodingRunningProcess([_evt("step_start", {"id": "prt_1", "messageID": "msg_1"})])) - with pytest.raises(AgentCrashError): - await _run(_agent(), tmp_path) + outcome = await _run(_agent(), tmp_path) + assert outcome.status is AgentEndStatus.CRASHED assert (4242, signal.SIGKILL) in captured["killpg"] @@ -1787,7 +1929,7 @@ def _failing_tool(error: str) -> str: async def test_tool_error_is_captured_not_dropped(self, patch_exec, tmp_path): recorder = _EventRecorder() patch_exec(_FakeProcess([self._failing_tool("boom: command exploded")])) - record = await _run(_agent(), tmp_path, stream_callback=recorder) + record = await _record(_agent(), tmp_path, stream_callback=recorder) [cmd] = record.commands assert cmd.result_status == "error" @@ -1798,180 +1940,239 @@ async def test_tool_error_is_captured_not_dropped(self, patch_exec, tmp_path): async def test_permission_denial_gets_its_own_status(self, patch_exec, tmp_path): recorder = _EventRecorder() patch_exec(_FakeProcess([self._failing_tool("Permission denied by policy")])) - record = await _run(_agent(), tmp_path, stream_callback=recorder) + record = await _record(_agent(), tmp_path, stream_callback=recorder) [cmd] = record.commands assert cmd.result_status == "error" # the persisted tri-state folds both [end] = [e for e in recorder.events if isinstance(e, ToolEndEvent)] assert end.status is ToolEndStatus.PERMISSION_DENIED - def test_orphan_result_is_never_dropped(self): - """A result with no matching call still surfaces as an `unknown` tool.""" - state = _OpenCodeTurnState(task_id="t", iteration=1, user_input="x", model=None) - events: list[Any] = [] - state.bind(events.append) + def test_an_unresolved_call_is_swept_never_dropped(self): + """A call the CLI never resolved still surfaces, as an `unknown` tool with no invented error.""" + result, decoder = _replay([_event("step_start", {"messageID": "m1"}), _tool("ghost", "pending", at_ms=1)]) - state._close_tool("ghost", status=ToolEndStatus.UNRESOLVED, summary=None, error="no result observed") - - [event] = events - assert isinstance(event, ToolEndEvent) - assert event.tool.tool_name == "unknown" + [event] = [e for e in result.events if isinstance(e, ToolEndEvent)] + assert event.status is ToolEndStatus.UNRESOLVED + assert event.tool.tool_id == "ghost" + assert event.tool.tool_name == "Bash" assert event.tool.result_status == "unknown" - assert event.tool.error_message == "no result observed" + assert event.tool.error_message is None + assert decoder.open_tools == {"ghost": {"command": "ls"}} + assert [c.tool_id for c in result.record.commands] == ["ghost"] + + +class TestTimingIsTheClisOwn: + """Under `cli_epoch_ms` every recorded bound is a CLI stamp, never a read of the host clock. + + Each case moves the scripted clock far away from the stamps, so a bound taken + from the clock instead of the stream lands seconds off and fails. + """ + + def test_window_bounds_are_the_envelope_stamps(self): + result, _ = _replay( + [ + Tick(7_000), + _event("step_start", {"messageID": "m1"}, at_ms=100), + Tick(50_000), + _finish(900), + ] + ) + + [message] = _assistants(result) + assert message.started_at == _at(100) + assert message.completed_at == _at(900) + assert message.generation_duration_ms == pytest.approx(800.0) + + def test_tool_span_is_state_time(self): + result, _ = _replay( + [ + _event("step_start", {"messageID": "m1"}, at_ms=0), + Tick(9_000), + _tool("c1", "completed", at_ms=600, start=200, end=450), + _finish(1000), + ] + ) + + [command] = result.record.commands + assert command.execution_started_at == _at(200) + assert command.execution_completed_at == _at(450) + assert command.duration_ms == pytest.approx(250.0) + + def test_a_resolved_tool_without_an_end_stamp_gets_no_completion_or_duration(self): + result, _ = _replay( + [ + _event("step_start", {"messageID": "m1"}, at_ms=0), + Tick(600), + _tool("c1", "completed", at_ms=600, start=200), + _finish(1000), + ] + ) + + [command] = result.record.commands + assert command.result_status == "success" + assert command.execution_started_at == _at(200) + assert command.execution_completed_at is None + assert command.duration_ms is None + + def test_an_orphan_has_no_completion_stamp(self): + """Force-closing is not observing a completion; the CLI's start stamp is kept.""" + result, _ = _replay( + [ + _event("step_start", {"messageID": "m1"}, at_ms=0), + _tool("c1", "running", at_ms=200, start=200), + _finish(1000), + Tick(4_000), + ] + ) + + [command] = result.record.commands + assert command.result_status == "unknown" + assert command.error_message is None + assert command.execution_started_at == _at(200) + assert command.execution_completed_at is None + assert command.duration_ms is None + + def test_a_missing_envelope_stamp_bounds_the_window_on_the_host_clock_and_warns_once(self, caplog): + with caplog.at_level("WARNING", logger="coder_eval.agents.opencode_agent"): + result, decoder = _replay( + [ + _event("step_start", {"messageID": "m1"}, at_ms=0), + Tick(700), + _finish(None), + Tick(1_200), + _event("step_start", {"messageID": "m2"}, at_ms=None), + Tick(1_500), + _finish(None), + ] + ) + + first, second = _assistants(result) + assert first.started_at == _at(0) + assert first.completed_at == _BASE + timedelta(milliseconds=700) + assert second.started_at == first.completed_at + assert second.completed_at == _BASE + timedelta(milliseconds=1_500) + assert decoder.warned_missing_stamp is True + warnings = [r for r in caplog.records if "no envelope timestamp" in r.getMessage()] + assert len(warnings) == 1 + + def test_the_captured_stream_tiles_on_its_own_stamps(self): + """A real `opencode run` stream: every window and tool bound is a CLI stamp, and the identity closes.""" + lines = [json.loads(line) for line in CAPTURED_STREAM] + first_ms = lines[0]["timestamp"] + last_ms = max(event.get("timestamp", first_ms) for event in lines) + origin = datetime.fromtimestamp(first_ms / 1000) + stream: list[Any] = [*lines, Tick(last_ms - first_ms)] + + decoders: list[_OpenCodeDecoder] = [] + + def end(decoder: _OpenCodeDecoder) -> TurnOutcome: + decoders.append(decoder) + return decoder.end(AgentEndStatus.COMPLETED) + + result = replay(stream, _OpenCodeDecoder, clock=ScriptedClock(origin), basis=TimingBasis.CLI_EPOCH_MS, end=end) + + assert result.outcome.status is AgentEndStatus.COMPLETED + assert decoders[0].warned_missing_stamp is False + assert [c.tool_name for c in result.record.commands] == ["Write", "Bash"] + assert all(c.duration_ms is not None for c in result.record.commands) + assert_stream_balanced(result.events) + assert_identity_closes(result.record, started_at=result.started_at, ended_at=result.ended_at) class TestGenerationWindowExcludesToolExecution: """A tool running inside a step is not model time — asserted where it is now DECIDED. - The reducer no longer subtracts anything. It publishes the RAW window, and + The decoder does not subtract anything. It publishes the RAW window, and `timing.subtract_tool_time` takes the tool union back out of it - once, for all five harnesses. So these cases drive the reducer and then a - real collector, and assert the PUBLISHED number — the one that reaches - `task.json` — rather than an intermediate the reducer used to own. + once, for all five harnesses. So these cases replay the decoder through a + real emitter and assert the PUBLISHED number — the one that reaches + `task.json` — rather than an intermediate the decoder used to own. They are not duplicates of `tests/test_event_collector.py::TestSubtractToolTime`: those pin the - arithmetic, these pin that THIS reducer hands the collector a window and a + arithmetic, these pin that THIS decoder hands the collector a window and a span set the arithmetic can be right about. """ - WINDOW_START = datetime(2026, 1, 1, 12, 0, 0) - WINDOW_END = datetime(2026, 1, 1, 12, 0, 1) # a 1000ms step - - def _finish_step(self, monkeypatch, spans, open_starts=()): - """Drive the reducer, then publish through a real collector. + def _finish_step(self, spans: list[tuple[int, int]], open_starts: tuple[int, ...] = ()) -> AssistantMessage: + """Replay one step stamped 0 -> 1000 ms with tool calls whose `state.time` is at the given offsets. `spans` are RESOLVED calls (both bounds); `open_starts` are calls that - never returned. An unresolved call now contributes NO span — it has no + never returned. An unresolved call contributes NO span — it has no `execution_completed_at`, and inventing one is what `None` exists to - prevent — where the reducer used to bound it at the window's end. That - is a real change and a better one: the collector sees every span at - once, so a call straddling a boundary is clipped to each window it - actually overlapped instead of approximated at the boundary. + prevent. The collector sees every span at once, so a call straddling a + boundary is clipped to each window it actually overlapped. """ - - class _Clock(datetime): - @staticmethod - def now(tz=None): - return TestGenerationWindowExcludesToolExecution.WINDOW_END - - state = _OpenCodeTurnState(task_id="t1", iteration=1, user_input="do it", model="deepseek/deepseek-v4-pro") - state.step_started_at = self.WINDOW_START - commands = [ - CommandTelemetry( - tool_name="bash", - tool_id=f"closed-{i}", - timestamp=started, - execution_started_at=started, - execution_completed_at=completed, - result_status="success", - ) + stream: list[Any] = [_event("step_start", {"messageID": "m1"}, at_ms=0)] + stream += [ + _tool(f"closed-{i}", "completed", at_ms=completed, start=started, end=completed) for i, (started, completed) in enumerate(spans) ] - commands += [ - CommandTelemetry(tool_name="bash", tool_id=f"open-{i}", timestamp=st, execution_started_at=st) - for i, st in enumerate(open_starts) - ] - monkeypatch.setattr(agent_module, "datetime", _Clock) - state.on_step_finish({"reason": "stop", "tokens": {"input": 100, "output": 20}}) - - collector = EventCollector() - collector.on_event(AgentStartEvent(task_id="t1", prompt="do it", iteration=1, timestamp=self.WINDOW_START)) - for command in commands: - collector.on_event(ToolEndEvent(task_id="t1", turn_id="s1", tool=command)) - collector.on_event( - AgentEndEvent( - task_id="t1", - status=AgentEndStatus.COMPLETED, - messages=list(state.messages), - usage=TokenUsage(), - timestamp=self.WINDOW_END, - ) - ) - published = [m for m in collector.build_turn_record().messages if m.role == "assistant"] + stream += [_tool(f"open-{i}", "running", at_ms=started, start=started) for i, started in enumerate(open_starts)] + stream.append(_finish(1000)) + + result, _ = _replay(stream) + published = _assistants(result) assert len(published) == 1 return published[0] - def test_tool_time_inside_the_step_is_subtracted(self, monkeypatch): - message = self._finish_step( - monkeypatch, - [(self.WINDOW_START + timedelta(milliseconds=200), self.WINDOW_START + timedelta(milliseconds=700))], - ) + def test_tool_time_inside_the_step_is_subtracted(self): + message = self._finish_step([(200, 700)]) span_ms = (message.completed_at - message.started_at).total_seconds() * 1000.0 - assert span_ms == pytest.approx(1000.0), "the reducer still publishes the whole window as its bounds" + assert span_ms == pytest.approx(1000.0), "the decoder still publishes the whole window as its bounds" assert message.generation_duration_ms == pytest.approx(500.0) - def test_a_step_with_no_tools_keeps_its_whole_window(self, monkeypatch): - assert self._finish_step(monkeypatch, []).generation_duration_ms == pytest.approx(1000.0) + def test_a_step_with_no_tools_keeps_its_whole_window(self): + assert self._finish_step([]).generation_duration_ms == pytest.approx(1000.0) - def test_concurrent_tools_are_subtracted_once(self, monkeypatch): + def test_concurrent_tools_are_subtracted_once(self): # Two overlapping 500ms tools occupy 600ms, not 1000ms. Summing them # would leave 0 generation for a step that generated 400. - message = self._finish_step( - monkeypatch, - [ - (self.WINDOW_START + timedelta(milliseconds=100), self.WINDOW_START + timedelta(milliseconds=600)), - (self.WINDOW_START + timedelta(milliseconds=200), self.WINDOW_START + timedelta(milliseconds=700)), - ], - ) + message = self._finish_step([(100, 600), (200, 700)]) assert message.generation_duration_ms == pytest.approx(400.0) - def test_the_window_never_goes_negative(self, monkeypatch): - message = self._finish_step( - monkeypatch, - [(self.WINDOW_START - timedelta(seconds=30), self.WINDOW_END + timedelta(seconds=30))], - ) + def test_the_window_never_goes_negative(self): + message = self._finish_step([(-30_000, 31_000)]) assert message.generation_duration_ms == 0.0 - def test_a_tool_still_open_at_the_boundary_contributes_no_span(self, monkeypatch): - """The behaviour that CHANGED with the move, stated rather than implied. + def test_a_tool_still_open_at_the_boundary_contributes_no_span(self): + """A call with no `execution_completed_at` was never timed. - The reducer used to bound a still-open call at the window's end and - subtract that slice. The collector cannot: a call with no - `execution_completed_at` was never timed. Its time is subtracted when it - RESOLVES, from whichever windows its real interval overlaps. + Its time is subtracted when it RESOLVES, from whichever windows its real + interval overlaps — never bounded at the window's end. """ - message = self._finish_step(monkeypatch, [], open_starts=[self.WINDOW_START + timedelta(milliseconds=600)]) + message = self._finish_step([], open_starts=(600,)) assert message.generation_duration_ms == pytest.approx(1000.0) - def test_a_resolved_tool_overlapping_an_unresolved_one_counts_only_the_resolved(self, monkeypatch): - message = self._finish_step( - monkeypatch, - [(self.WINDOW_START + timedelta(milliseconds=200), self.WINDOW_START + timedelta(milliseconds=700))], - open_starts=[self.WINDOW_START + timedelta(milliseconds=500)], - ) + def test_a_resolved_tool_overlapping_an_unresolved_one_counts_only_the_resolved(self): + message = self._finish_step([(200, 700)], open_starts=(500,)) assert message.generation_duration_ms == pytest.approx(500.0) - def test_a_mark_later_than_the_step_start_does_not_invert_the_window(self, monkeypatch): - """The backwards-clock defence, pinned at the reducer, not in isolation. + def test_a_mark_later_than_the_step_start_does_not_invert_the_window(self): + """The backwards-stamp defence, pinned at the decoder, not in isolation. - `close_window`'s `min()` only fires if the reducer actually passes the + `close_window`'s `min()` only fires if the decoder actually passes the step's own start as `item_start`. Drop that argument and the window opens at the (later) mark instead, so the span shrinks — or inverts and - clamps to 0.0, publishing a fabricated instant generation. Nothing else - in this file fails when it is dropped, which is the whole reason it is - here: the mark is what the reducer still owns after the tool - subtraction moved to the collector. + clamps to 0.0, publishing a fabricated instant generation. The CLI's + stamps are not monotonic by contract: here the first step's + `step_finish` is stamped 400 ms AFTER the second step's `step_start`. """ - state = _OpenCodeTurnState(task_id="t1", iteration=1, user_input="do it", model="m") - state.step_started_at = self.WINDOW_START - # A mark 400ms AFTER this step began: the CLI's step_finish for the - # previous step landed late, or the clock stepped. - state.gen_mark = self.WINDOW_START + timedelta(milliseconds=400) - - class _Clock(datetime): - @staticmethod - def now(tz=None): - return TestGenerationWindowExcludesToolExecution.WINDOW_END - - monkeypatch.setattr(agent_module, "datetime", _Clock) - state.on_step_finish({"reason": "stop", "tokens": {"input": 100, "output": 20}}) - - message = next(m for m in state.messages if m.role == "assistant") - assert message.started_at == self.WINDOW_START + result, decoder = _replay( + [ + _event("step_start", {"messageID": "m0"}, at_ms=-500), + _finish(400), + _event("step_start", {"messageID": "m1"}, at_ms=0), + _finish(1000), + ] + ) + + _, message = _assistants(result) + assert decoder.gen_mark == _at(1000) + assert message.started_at == _at(0) assert message.generation_duration_ms == pytest.approx(1000.0) - def test_the_published_window_reconciles_to_its_own_bounds(self, monkeypatch): + def test_the_published_window_reconciles_to_its_own_bounds(self): """The collector subtracted exactly the spans the record carries. `scripts/timing/decompose_run.py` and the evalboard's Unaccounted cell @@ -1982,10 +2183,9 @@ def test_the_published_window_reconciles_to_its_own_bounds(self, monkeypatch): """ from coder_eval.timing import busy_ms - closed = [(self.WINDOW_START + timedelta(milliseconds=200), self.WINDOW_START + timedelta(milliseconds=700))] - message = self._finish_step(monkeypatch, closed) + message = self._finish_step([(200, 700)]) span_ms = (message.completed_at - message.started_at).total_seconds() * 1000.0 - expected = span_ms - busy_ms(closed, message.started_at, message.completed_at) + expected = span_ms - busy_ms([(_at(200), _at(700))], message.started_at, message.completed_at) assert message.generation_duration_ms == pytest.approx(expected) @@ -1998,235 +2198,178 @@ class TestGenerationWindowsTileTheTurn: carrying no tool at all (the Write inside them took 7 ms), attributed to nothing — 24% of the turn, on its own enough to hold OpenCode above the evalboard's 25% "Unaccounted" red threshold. - - Driven at the reducer for the same reason as the sibling class above: the - window is two `datetime.now()` reads, so only setting them explicitly - makes the arithmetic deterministic. """ - T0 = datetime(2026, 1, 1, 12, 0, 0) - - @staticmethod - def _finish_at(monkeypatch, state, *, step_start, now): - class _Clock(datetime): - @staticmethod - def now(tz=None): - return now - - state.step_started_at = step_start - monkeypatch.setattr(agent_module, "datetime", _Clock) - state.on_step_finish({"reason": "stop", "tokens": {"input": 100, "output": 20}}) - - def _two_steps(self, monkeypatch): - state = _OpenCodeTurnState(task_id="t1", iteration=1, user_input="do it", model="deepseek/deepseek-v4-pro") - # Step 1 runs T0 -> T0+1000. - self._finish_at(monkeypatch, state, step_start=self.T0, now=self.T0 + timedelta(milliseconds=1000)) - # 800ms of model time, then a step the CLI only announces at T0+1800. - self._finish_at( - monkeypatch, - state, - step_start=self.T0 + timedelta(milliseconds=1800), - now=self.T0 + timedelta(milliseconds=2000), + def _two_steps(self) -> list[AssistantMessage]: + # Step 1 runs 0 -> 1000; then 800ms of model time, then a step the CLI only announces at 1800. + result, _ = _replay( + [ + _event("step_start", {"messageID": "m1"}, at_ms=0), + _finish(1000), + _event("step_start", {"messageID": "m2"}, at_ms=1800), + _finish(2000), + ] ) - return [m for m in state.messages if m.role == "assistant"] + return _assistants(result) - def test_the_gap_before_a_step_is_its_generation_time(self, monkeypatch): - first, second = self._two_steps(monkeypatch) + def test_the_gap_before_a_step_is_its_generation_time(self): + first, second = self._two_steps() # Bounded by its own step_start, this window was 200ms and the 800ms # that produced it was attributed to nothing. assert second.generation_duration_ms == pytest.approx(1000.0) assert second.started_at == first.completed_at - def test_the_first_step_keeps_its_own_start(self, monkeypatch): + def test_the_first_step_keeps_its_own_start(self): """Everything before the first `step_start` is CLI spawn, not model time. Tiling the first window back to the turn's start would report Node's boot — 3.1 s of OpenCode's measured head — as generation. """ - first, _ = self._two_steps(monkeypatch) - assert first.started_at == self.T0 + first, _ = self._two_steps() + assert first.started_at == _at(0) assert first.generation_duration_ms == pytest.approx(1000.0) - def test_the_steps_leave_no_gap_between_them(self, monkeypatch): - first, second = self._two_steps(monkeypatch) + def test_the_steps_leave_no_gap_between_them(self): + first, second = self._two_steps() covered = (second.completed_at - first.started_at).total_seconds() * 1000.0 gen = sum(m.generation_duration_ms or 0.0 for m in (first, second)) assert gen == pytest.approx(covered) -_SPAN_EPOCH_MS = 1_800_000_000_000 -_SPAN_BASE = datetime.fromtimestamp(_SPAN_EPOCH_MS / 1000) - - -class _SteppedClock(datetime): - """A clock the test moves by hand, in ms from `_SPAN_BASE`. - - Subclasses `datetime` rather than stubbing it, because `_epoch_ms_to_dt` - calls `datetime.fromtimestamp` through the same module global and must keep - resolving to the real implementation — the CLI's epoch stamps and the - reducer's own `now()` reads have to land on ONE timeline for the span - arithmetic under test to mean anything. - """ - - at_ms = 0.0 - - @staticmethod - def now(tz=None): - return _SPAN_BASE + timedelta(milliseconds=_SteppedClock.at_ms) - - class TestToolSpansSurviveTheStepBoundary: """A tool that closes BETWEEN two steps still belongs to the next window. - This used to be a bookkeeping problem: a per-step span list, cleared at - `step_start` — after the window it feeds had already opened at `gen_mark` — - so a call closing in the gap had its span wiped before the next - `step_finish` could subtract it. That list is gone. - `timing.subtract_tool_time` sees every span at once and clips each - to the windows it overlaps, so the property now holds by construction - rather than by a reset rule. Kept, and re-pointed at the collector, because - the property is what matters: a future reducer change could still break it - by moving a mark or failing to emit the ToolEnd the collector reduces. + `timing.subtract_tool_time` sees every span at once and clips each to the + windows it overlaps, so the property holds by construction rather than by a + reset rule. Kept because the property is what matters: a future decoder + change could still break it by moving a mark or failing to close the tool + the collector reduces. It needs the NON-TERMINAL tool path to reach: the CLI normally emits one already-`completed` event per call, which closes inside the step that opened it. That is why the measured corpus reads 0.00% and a reproduction - has to drive the state object. + has to script the stream. """ - def _run(self, monkeypatch): - monkeypatch.setattr(agent_module, "datetime", _SteppedClock) - state = _OpenCodeTurnState(task_id="t1", iteration=1, user_input="go", model="m") - # The resolved telemetry leaves the state via ToolEnd; the identity - # case below reconciles against what was RECORDED, not against the - # clock the test scripted. - resolved: list[Any] = [] - state.bind(lambda e: resolved.append(e.tool) if isinstance(e, ToolEndEvent) else None) - - def tool(status, *, end_ms=None): - times = {"start": _SPAN_EPOCH_MS + 100} - if end_ms is not None: - times["end"] = _SPAN_EPOCH_MS + end_ms - state.on_tool_use({"callID": "c1", "tool": "bash", "state": {"status": status, "time": times}}) - - _SteppedClock.at_ms = 0 - state.on_step_start({"messageID": "m1"}) - _SteppedClock.at_ms = 100 - tool("running") # non-terminal: stays open across the boundary - _SteppedClock.at_ms = 1000 - state.on_step_finish({"reason": "stop", "tokens": {"input": 10, "output": 5}}) - _SteppedClock.at_ms = 1500 - tool("completed", end_ms=1500) # closes in the GAP between the steps - _SteppedClock.at_ms = 1600 - state.on_step_start({"messageID": "m2"}) - _SteppedClock.at_ms = 2000 - state.on_step_finish({"reason": "stop", "tokens": {"input": 10, "output": 5}}) - - # Published through the real collector: the reducer hands over raw - # windows, and the tool subtraction happens once, there. - collector = EventCollector() - collector.on_event(AgentStartEvent(task_id="t1", prompt="go", iteration=1, timestamp=_SPAN_BASE)) - for command in resolved: - collector.on_event(ToolEndEvent(task_id="t1", turn_id="s1", tool=command)) - collector.on_event( - AgentEndEvent( - task_id="t1", - status=AgentEndStatus.COMPLETED, - messages=list(state.messages), - usage=TokenUsage(), - timestamp=_SPAN_BASE + timedelta(milliseconds=2000), - ) - ) - published = [m for m in collector.build_turn_record().messages if m.role == "assistant"] - return resolved, published - - def test_the_gap_slice_of_a_straddling_call_is_not_published_as_generation(self, monkeypatch): - _, messages = self._run(monkeypatch) + def _run(self) -> Replay: + return _replay( + [ + _event("step_start", {"messageID": "m1"}, at_ms=0), + _tool("c1", "running", at_ms=100, start=100), # non-terminal: stays open across the boundary + _finish(1000), + _tool("c1", "completed", at_ms=1500, start=100, end=1500), # closes in the GAP between the steps + _event("step_start", {"messageID": "m2"}, at_ms=1600), + _finish(2000), + Tick(2000), + ] + )[0] + + def test_the_gap_slice_of_a_straddling_call_is_not_published_as_generation(self): + messages = _assistants(self._run()) assert len(messages) == 2 # Window 2 tiles 1000 -> 2000. c1 ran for 1000 -> 1500 of it, so 500ms # is model time. Before the reset moved, this published 1000.0 — a 100% # overstatement, with c1's own duration_ms counting the same 500ms. assert messages[1].generation_duration_ms == pytest.approx(500.0) - def test_the_call_is_subtracted_from_exactly_one_window(self, monkeypatch): - # Window 1 owns c1's 100 -> 1000 slice (it was open at that boundary - # and bounded there); window 2 owns 1000 -> 1500. Neither owns both. - _, messages = self._run(monkeypatch) + def test_the_call_is_subtracted_from_exactly_one_window(self): + # Window 1 owns c1's 100 -> 1000 slice; window 2 owns 1000 -> 1500. Neither owns both. + messages = _assistants(self._run()) assert messages[0].generation_duration_ms == pytest.approx(100.0) assert messages[1].generation_duration_ms == pytest.approx(500.0) - def test_the_four_bucket_identity_closes_exactly_across_the_boundary(self, monkeypatch): + def test_the_four_bucket_identity_closes_exactly_across_the_boundary(self): """generation + UNION(tool) accounts for the whole span, to the ms. The assertion the golden corpus CANNOT make: `_scrub.py` masks `generation_duration_ms` and both bounds to a placeholder, so a snapshot records that a window was measured and never what it - measured, and its identity check is an upper bound besides — so - under-accounting, the defect this phase fixes, passes it silently. + measured. """ from coder_eval.timing import busy_ms - resolved, messages = self._run(monkeypatch) + result = self._run() + messages = _assistants(result) lo, hi = messages[0].started_at, messages[1].completed_at generation_ms = sum(m.generation_duration_ms or 0.0 for m in messages) - command = next(c for c in resolved if c.tool_id == "c1") + command = next(c for c in result.record.commands if c.tool_id == "c1") + assert command.execution_started_at is not None and command.execution_completed_at is not None tool_ms = busy_ms([(command.execution_started_at, command.execution_completed_at)], lo, hi) assert generation_ms + tool_ms == pytest.approx((hi - lo).total_seconds() * 1000.0) + assert_identity_closes(result.record, started_at=result.started_at, ended_at=result.ended_at) - def test_a_duplicate_step_finish_does_not_republish_the_previous_window(self, monkeypatch): + def test_a_duplicate_step_finish_does_not_republish_the_previous_window(self): """A spent `step_started_at` must not seed the next window. `close_window`'s `min(mark, item_start)` pulls the window open to cover - the item's own start. That is the backwards-clock defence — which this - reducer genuinely needs, since its stamps are raw `datetime.now()` and - not on a `TurnClock`. But a start stamp left in place after its step was - published is not a backwards clock: it is a stale value BEFORE the mark, - so the guard reopens the next window at the previous step's start and - publishes that whole span again. Reproduced on Pi's identical twin - before the fix: 3000 ms of generation for a 2000 ms turn. + the item's own start. A start stamp left in place after its step was + published is a stale value BEFORE the mark, so the guard would reopen the + next window at the previous step's start and publish that whole span + again. Reproduced on Pi's identical twin before the fix: 3000 ms of + generation for a 2000 ms turn. """ - monkeypatch.setattr(agent_module, "datetime", _SteppedClock) - state = _OpenCodeTurnState(task_id="t1", iteration=1, user_input="go", model="m") - _SteppedClock.at_ms = 0 - state.on_step_start({"messageID": "m1"}) - _SteppedClock.at_ms = 1000 - state.on_step_finish({"reason": "stop", "tokens": {"input": 10, "output": 5}}) - _SteppedClock.at_ms = 2000 - state.on_step_finish({"reason": "stop", "tokens": {"input": 10, "output": 5}}) - - messages = [m for m in state.messages if m.role == "assistant"] + result, _ = _replay( + [_event("step_start", {"messageID": "m1"}, at_ms=0), _finish(1000), _finish(2000)] + ) # no intervening `step_start` + + messages = _assistants(result) assert len(messages) == 2 assert messages[1].started_at == messages[0].completed_at assert sum(m.generation_duration_ms or 0.0 for m in messages) == pytest.approx(2000.0) - def test_a_step_that_never_finishes_does_not_advance_the_mark(self, monkeypatch): - """The half of this that is still the reducer's job. + def test_a_step_that_never_finishes_does_not_advance_the_mark(self): + """The half of this that is still the decoder's job. - There is no span list to preserve any more — the collector reduces the - ToolEnd stream itself. What the reducer still owns is the MARK: a step - that published nothing must not advance it, or its time is handed to + There is no span list to preserve — the collector reduces the ToolEnd + stream itself. What the decoder still owns is the MARK: a step that + published nothing must not advance it, or its time is handed to whichever step finishes next. """ - monkeypatch.setattr(agent_module, "datetime", _SteppedClock) - state = _OpenCodeTurnState(task_id="t1", iteration=1, user_input="go", model="m") - _SteppedClock.at_ms = 0 - state.on_step_start({"messageID": "m1"}) - _SteppedClock.at_ms = 1000 - state.on_step_finish({"reason": "stop", "tokens": {"input": 10, "output": 5}}) - mark_after_flush = state.gen_mark - - _SteppedClock.at_ms = 1600 - state.on_step_start({"messageID": "m2"}) - _SteppedClock.at_ms = 1700 - state.on_tool_use( - { - "callID": "c2", - "tool": "bash", - "state": {"status": "running", "time": {"start": _SPAN_EPOCH_MS + 1700}}, - } + result, decoder = _replay( + [ + _event("step_start", {"messageID": "m1"}, at_ms=0), + _finish(1000), + _event("step_start", {"messageID": "m2"}, at_ms=1600), + _tool("c2", "running", at_ms=1700, start=1700), + Tick(1900), + ], + status=AgentEndStatus.CRASHED, + reason="turn cancelled", ) - _SteppedClock.at_ms = 1900 - state.close_open_tools() # crash/timeout orphan sweep — no message appended - assert state.gen_mark == mark_after_flush + assert decoder.gen_mark == _at(1000) + assert len(_assistants(result)) == 1 + assert result.outcome.status is AgentEndStatus.CRASHED + + +class TestModelTurnCap: + def test_the_model_turn_cap_stops_at_the_next_turn_start_with_the_last_turn_resolved(self) -> None: + lines = Path("tests/fixtures/opencode_happy_stream.jsonl").read_text(encoding="utf-8").splitlines() + result, _ = _replay([json.loads(line) for line in lines if line.strip()]) + task = TaskDefinition( + task_id="t", + description="d", + initial_prompt="go", + agent=parse_agent_config(type="opencode"), + sandbox=SandboxConfig(driver="tempdir"), + success_criteria=[FileExistsCriterion(description="c", path="out.txt")], + run_limits=RunLimits(max_turns=1), + ) + monitor = TurnMonitor.for_task(task, arm=False) + ends: list[ToolEndEvent] = [] + latch: tuple[StreamEvent, list[ToolEndEvent]] | None = None + for event in result.events: + monitor.on_event(event) + if latch is None and monitor.stop_reason is not None: + latch = (event, list(ends)) + if isinstance(event, ToolEndEvent) and event.parent_thread_id is None: + ends.append(event) + + assert monitor.model_turns == 2 + assert monitor.stop_reason is StopReason.MODEL_TURN_CAP + assert latch is not None + latched_on, ends_at_latch = latch + assert isinstance(latched_on, TurnStartEvent) + unresolved = sum(end.status is ToolEndStatus.UNRESOLVED for end in ends_at_latch) + assert (len(ends_at_latch) - unresolved, unresolved) == (2, 0) diff --git a/tests/test_orchestrator.py b/tests/test_orchestrator.py index 416bd4ad..b7bde1ea 100644 --- a/tests/test_orchestrator.py +++ b/tests/test_orchestrator.py @@ -7,6 +7,7 @@ import pytest from coder_eval import orchestrator as orchestrator_module +from coder_eval.errors import AgentCrashError from coder_eval.models import ( AgentKind, BedrockRoute, @@ -329,142 +330,6 @@ def test_finalize_result_score_mismatch_marks_error_and_writes_task_json(tmp_pat assert orchestrator.report_path.exists() -class _SdkOptionsAgent(MockAgent): - """``MockAgent`` subclass that returns a configurable ``get_sdk_options``. - - Used by the PATH-sync tests so the dummy agent satisfies the full - ``Agent`` ABC (``start`` / ``communicate`` / ``stop`` / ``get_state``) - rather than only the one method ``_sync_…`` happens to call today — - keeps the test surface aligned with the production contract. - """ - - def __init__(self, task, sdk_options): - super().__init__(task) - self._sdk_options = sdk_options - - def get_sdk_options(self): - return self._sdk_options - - -class _AsyncSdkOptionsAgent(MockAgent): - """Returns a fresh coroutine every call — mimics ``AsyncMock`` leakage.""" - - def get_sdk_options(self): - async def _coro(): - return {"env": {"PATH": "/agent/bin"}} - - return _coro() - - -@pytest.fixture -def path_sync_orchestrator(tmp_path): - """Yield ``(orchestrator, task)`` ready for PATH-sync helper tests. - - Removes the boilerplate (load task, build orchestrator, setup sandbox, - cleanup in finally) that the per-test body would otherwise repeat. - """ - task_file = Path("tasks/hello_date.yaml") - task, _ = load_task(task_file) - task.sandbox = SandboxConfig(driver="tempdir", python=None) - orchestrator = Orchestrator(task=task, run_dir=tmp_path / "run", variant_id="t") - orchestrator.sandbox = Sandbox(task.sandbox, task_id=task.task_id) - orchestrator.sandbox.setup() - try: - yield orchestrator, task - finally: - orchestrator.sandbox.cleanup() - - -def test_sync_sandbox_command_path_from_agent_sdk_options(path_sync_orchestrator, monkeypatch, tmp_path): - """Happy path: agent SDK PATH wins for criteria ``run_command`` resolution.""" - from tests._path_helpers import write_uip_shim - - orchestrator, task = path_sync_orchestrator - stale_bin = tmp_path / "stale" - agent_bin = tmp_path / "agent" - stale_bin.mkdir() - agent_bin.mkdir() - write_uip_shim(stale_bin, "stale") - write_uip_shim(agent_bin, "agent") - monkeypatch.setenv("PATH", str(stale_bin)) - - orchestrator.agent = _SdkOptionsAgent(task, {"env": {"PATH": f"{agent_bin}{os.pathsep}{stale_bin}"}}) - orchestrator._sync_sandbox_command_path_with_agent() - - exit_code, stdout, _stderr = orchestrator.sandbox.run_command("uip") - assert exit_code == 0 - assert stdout.strip() == "agent" - - -def test_sync_sandbox_command_path_preserves_host_path_for_system_bins(path_sync_orchestrator, monkeypatch, tmp_path): - """The agent's narrow PATH must not clobber the host PATH for system bins. - - Locks in the prepend (not replace) semantics flagged HIGH in the - multi-model review (PR #249 thread). If a future refactor accidentally - re-introduces ``env['PATH'] = base_path`` (replace), this fails because - the host's ``/usr/bin``-style binary becomes unreachable. - """ - from tests._path_helpers import write_uip_shim - - orchestrator, task = path_sync_orchestrator - # Host PATH carries a `uip` named "host". Agent PATH carries no `uip`. - host_bin = tmp_path / "host" - agent_bin = tmp_path / "agent" # intentionally empty - host_bin.mkdir() - agent_bin.mkdir() - write_uip_shim(host_bin, "host") - monkeypatch.setenv("PATH", str(host_bin)) - - orchestrator.agent = _SdkOptionsAgent(task, {"env": {"PATH": str(agent_bin)}}) - orchestrator._sync_sandbox_command_path_with_agent() - - # Agent PATH wins for binaries it provides; falls through to host PATH - # for binaries it does not. A replace-style implementation would return - # exit_code == 1 with the "not found" message. - exit_code, stdout, _stderr = orchestrator.sandbox.run_command("uip") - assert exit_code == 0 - assert stdout.strip() == "host" - - -def test_sync_sandbox_command_path_awaitable_sdk_options_is_closed_and_noops(path_sync_orchestrator, caplog): - """AsyncMock-style coroutine returns: close, do not leak warnings.""" - orchestrator, task = path_sync_orchestrator - orchestrator.agent = _AsyncSdkOptionsAgent(task) - with caplog.at_level("DEBUG", logger="coder_eval.orchestrator"): - orchestrator._sync_sandbox_command_path_with_agent() - assert orchestrator.sandbox.command_base_path is None - # Logged at DEBUG (test-fixture concern), not WARNING. - debug_records = [r for r in caplog.records if r.levelname == "DEBUG"] - assert any("awaitable" in r.message for r in debug_records) - assert not any(r.levelname == "WARNING" for r in caplog.records) - - -@pytest.mark.parametrize( - "sdk_options, warn_substring", - [ - pytest.param(None, None, id="none-sdk-options"), - pytest.param(["unexpected"], "non-dict", id="non-dict-sdk-options"), - pytest.param({"env": {"HOME": "/tmp"}}, None, id="missing-path-key"), - pytest.param({"env": "not-a-dict"}, None, id="env-not-a-dict"), - ], -) -def test_sync_sandbox_command_path_contract_edge_cases_are_noops( - path_sync_orchestrator, caplog, sdk_options, warn_substring -): - """Edge inputs leave ``command_base_path`` unset, with the right log level.""" - orchestrator, task = path_sync_orchestrator - orchestrator.agent = _SdkOptionsAgent(task, sdk_options) - with caplog.at_level("WARNING", logger="coder_eval.orchestrator"): - orchestrator._sync_sandbox_command_path_with_agent() - assert orchestrator.sandbox.command_base_path is None - if warn_substring is None: - # Silent no-op — these are valid pre-communicate / sparse-env states. - assert not any(r.levelname == "WARNING" for r in caplog.records) - else: - # Contract violation — must be visible at WARNING. - assert any(r.levelname == "WARNING" and warn_substring in r.message for r in caplog.records) - - def test_orchestrator_load_task(): """Test loading a task from YAML.""" task_file = Path("tasks/hello_date.yaml") @@ -768,6 +633,9 @@ def get_environment_info(self): # carries agent markers like system_prompt_semantics into run.json. return {"system_prompt_semantics": "append"} + async def harness_version(self): + return "dummy 1.2.3" + async def create_dummy_agent(_self): return DummyAgent() @@ -798,9 +666,12 @@ async def create_dummy_agent(_self): await orchestrator._setup() + assert orchestrator._counts_model_turns is True + # An agent-supplied environment_info key survives the merge into the # run record (the cross-repo contract seam external consumers read). assert orchestrator.result.environment_info["system_prompt_semantics"] == "append" + assert orchestrator.result.environment_info["harness_version"] == "dummy 1.2.3" assert isinstance(orchestrator.sandbox, Sandbox) assert orchestrator.sandbox.sandbox_dir is not None @@ -952,6 +823,60 @@ async def create_dummy_agent(_self): await orchestrator._cleanup() +@pytest.mark.asyncio +async def test_criterion_path_gets_the_mock_dirs_the_agent_gets(tmp_path, monkeypatch): + """The agent's ``env_path_prepend`` and the criterion PATH come from one list, before any turn. + + Harness-independent on purpose: each harness's own tests prove it puts + ``env_path_prepend`` first on its PATH, so this is the one remaining link. + """ + from datetime import datetime + + from coder_eval.models import ApiBackend, DirectRoute, EvaluationResult + + captured: dict[str, list[str] | None] = {} + + class DummyAgent: + async def start(self, working_directory, *, env_path_prepend=None, plugin_tools_dir=None, plugin_root=None): + captured["env_path_prepend"] = env_path_prepend + + def get_environment_info(self): + return {} + + async def create_dummy_agent(_self): + return DummyAgent() + + task, _ = load_task(Path("tasks/mock_path_dirs_smoke.yaml")) + orchestrator = Orchestrator(task=task, run_dir=tmp_path / "run", variant_id="v") + orchestrator.result = EvaluationResult( + task_id=task.task_id, + task_description=task.description, + variant_id="v", + agent_type=AgentKind.CLAUDE_CODE, + started_at=datetime.now(), + final_status="FAILURE", + iteration_count=0, + environment_info={}, + ) + monkeypatch.setattr(orchestrator_module.settings, "api_backend", ApiBackend.DIRECT) + monkeypatch.setattr(type(orchestrator_module.settings), "validate_api_keys", lambda _self, _agent_type: None) + monkeypatch.setattr(orchestrator_module, "resolve_route", lambda _settings: DirectRoute(judge_transport=None)) + monkeypatch.setattr(Orchestrator, "_create_agent", create_dummy_agent) + + await orchestrator._setup() + try: + assert orchestrator.sandbox is not None + assert orchestrator.sandbox.sandbox_dir is not None + mocks = str((orchestrator.sandbox.sandbox_dir / "mocks").resolve()) + assert captured["env_path_prepend"] == [mocks] + assert orchestrator.sandbox.command_base_path == mocks + exit_code, stdout, _stderr = orchestrator.sandbox.run_command("say_hello criterion") + assert exit_code == 0 + assert "MOCK_PATH_OK" in stdout + finally: + orchestrator.sandbox.cleanup() + + @pytest.mark.asyncio async def test_orchestrator_cleanup_persistent_sandbox(tmp_path): """DIRECT_WRITE: sandbox already lives in artifacts; _cleanup keeps it in place (no move).""" @@ -1776,13 +1701,14 @@ async def test_overrides_apply_max_tool_calls_field_merge(tmp_path): assert task.run_limits.expected_tool_calls == baseline_expected_tool_calls -def test_overrides_reject_removed_run_limits_max_turns(): - """run_limits.max_turns no longer exists; the schema-validated override rejects it.""" - from coder_eval.orchestration.overrides import OverrideError, apply_overrides +def test_overrides_set_run_limits_max_turns(): + from coder_eval.orchestration.overrides import apply_overrides task, _ = load_task(Path("tasks/hello_date.yaml")) - with pytest.raises(OverrideError, match="max_turns"): - apply_overrides(task, {"run_limits.max_turns": 42}) + apply_overrides(task, {"run_limits.max_turns": 42}) + + assert task.run_limits is not None + assert task.run_limits.max_turns == 42 # ==================== Duplicate Task ID Validation Tests ==================== @@ -1838,7 +1764,13 @@ def test_resolve_all_tasks_rejects_duplicate_task_ids(tmp_path): # --- Evaluation loop: tool-call cap via the TurnMonitor --- -def _cap_task(task_id: str, max_tool_calls: int) -> TaskDefinition: +def _cap_task( + task_id: str, + max_tool_calls: int | None = None, + *, + max_turns: int | None = None, + expected_turns: int | None = None, +) -> TaskDefinition: from coder_eval.models import RunLimits agent_cfg = ClaudeCodeAgentConfig.model_construct( @@ -1857,7 +1789,7 @@ def _cap_task(task_id: str, max_tool_calls: int) -> TaskDefinition: agent=agent_cfg, sandbox=SandboxConfig(driver="tempdir"), success_criteria=[FileExistsCriterion(type="file_exists", path="test.py", description="test.py must exist")], - run_limits=RunLimits(max_tool_calls=max_tool_calls), + run_limits=RunLimits(max_tool_calls=max_tool_calls, max_turns=max_turns, expected_turns=expected_turns), task_timeout=None, reference=None, ) @@ -1899,7 +1831,8 @@ class _CooperativeToolAgent: """Fake agent that emits resolved tool calls and polls ``should_stop`` at each boundary. ``plan`` holds one entry per ``communicate`` attempt: the number of tool calls the - attempt intends to make, and whether it then crashes with a partial turn. ``host`` + attempt intends to make, and whether it then crashes with a partial turn. The crash + is a rate limit, which retries after tool calls. ``host`` is the ``AsyncMock`` the orchestrator talks to; its ``communicate`` is this fake's. """ @@ -1913,19 +1846,21 @@ def __init__(self, plan: list[tuple[int, bool]]) -> None: self._tool_seq = 0 self.host = AsyncMock() self.host.communicate = self.communicate - self.host.pending_turn = None - async def communicate(self, user_input, *, stream_callback=None, timeout=None, should_stop=None): + async def communicate(self, user_input, *, iteration=1, stream_callback=None, timeout=None, should_stop=None): from datetime import datetime - from coder_eval.errors import AgentCrashError from coder_eval.models import CommandTelemetry, TurnRecord + from coder_eval.streaming.emitter import TurnOutcome from coder_eval.streaming.events import ( AgentEndEvent, + AgentEndStatus, AgentStartEvent, StopReason, ToolEndEvent, ToolStartEvent, + TurnEndEvent, + TurnStartEvent, end_status_for, ) @@ -1934,35 +1869,43 @@ async def communicate(self, user_input, *, stream_callback=None, timeout=None, s intended, crash = self._plan[self.attempt] self.attempt += 1 self.should_stop_callables.append(should_stop) - stream_callback.on_event(AgentStartEvent(task_id="t", prompt=user_input, iteration=1)) + stream_callback.on_event(AgentStartEvent(task_id="t", prompt=user_input, iteration=iteration)) commands: list[CommandTelemetry] = [] reason: StopReason | None = should_stop() while reason is None and len(commands) < intended: self._tool_seq += 1 + stream_callback.on_event(TurnStartEvent(task_id="t", turn_id=f"turn-{self._tool_seq}")) + reason = should_stop() + if reason is not None: + break tool = CommandTelemetry(tool_name="Bash", tool_id=f"tool-{self._tool_seq}", timestamp=datetime.now()) stream_callback.on_event(ToolStartEvent(task_id="t", tool=tool)) stream_callback.on_event(ToolEndEvent(task_id="t", tool=tool)) + stream_callback.on_event(TurnEndEvent(task_id="t", turn_id=f"turn-{self._tool_seq}")) commands.append(tool) reason = should_stop() self.emitted_per_attempt.append(len(commands)) if crash: - self.host.pending_turn = TurnRecord( - iteration=1, user_input=user_input, agent_output="", commands=commands, crashed=True + partial = TurnRecord( + iteration=iteration, user_input=user_input, agent_output="", commands=commands, crashed=True ) - raise AgentCrashError("mid-turn failure") + return TurnOutcome(record=partial, status=AgentEndStatus.CRASHED, error="429 rate limit") status = end_status_for(reason) if reason is not None else None if status is not None: - stream_callback.on_event(AgentEndEvent(task_id="t", status=status, iteration=1, user_input=user_input)) - return TurnRecord( - iteration=1, + stream_callback.on_event( + AgentEndEvent(task_id="t", status=status, iteration=iteration, user_input=user_input) + ) + record = TurnRecord( + iteration=iteration, user_input=user_input, agent_output="stopped", commands=commands, - tool_calls_exhausted=reason is StopReason.TOOL_CALL_CAP, + tool_calls_exhausted=status is AgentEndStatus.TOOL_CALLS_EXHAUSTED, ) + return TurnOutcome(record=record, status=status or AgentEndStatus.COMPLETED, error=None) @pytest.mark.asyncio @@ -1996,14 +1939,16 @@ async def test_a_latched_cap_the_agent_did_not_stop_on_is_not_labelled_exhausted from unittest.mock import AsyncMock, patch from coder_eval.models import CommandTelemetry, TurnRecord - from coder_eval.streaming.events import StopReason, ToolEndEvent + from coder_eval.streaming.emitter import TurnOutcome + from coder_eval.streaming.events import AgentEndStatus, StopReason, ToolEndEvent orchestrator = _cap_orchestrator(_cap_task("late_latch_test", max_tool_calls=1), tmp_path) - async def _communicate(user_input, *, stream_callback=None, timeout=None, should_stop=None): + async def _communicate(user_input, *, iteration=1, stream_callback=None, timeout=None, should_stop=None): tool = CommandTelemetry(tool_name="Bash", tool_id="late", timestamp=datetime.now()) stream_callback.on_event(ToolEndEvent(task_id="t", tool=tool)) - return TurnRecord(iteration=1, user_input=user_input, agent_output="done", commands=[tool]) + record = TurnRecord(iteration=iteration, user_input=user_input, agent_output="done", commands=[tool]) + return TurnOutcome(record=record, status=AgentEndStatus.COMPLETED, error=None) mock_agent = AsyncMock() mock_agent.communicate = _communicate @@ -2066,18 +2011,19 @@ async def test_tool_call_cap_counts_a_crashed_attempts_calls_toward_the_retry(tm @pytest.mark.asyncio -async def test_evaluation_loop_preserves_partial_on_crash_retry(tmp_path): - """First agent.communicate raises AgentCrashError with a partial; retry succeeds. - - Locks the orchestrator wiring between `execute_with_retry` and the - `_preserve_partial_on_failure` callback: the partial record reaches - `result.iterations` before the successful retry's record, and both share the - same iteration number (per the agent-side rollback contract). +@pytest.mark.parametrize("tool_calls", [0, 1]) +async def test_evaluation_loop_preserves_partial_on_crash_retry(tmp_path, tool_calls): + """First agent.communicate outcome is CRASHED with a partial. + + Locks the orchestrator wiring in `_communicate_with_retry`: a CRASHED + outcome's record reaches `result.iterations` before it is raised. With no + tool call it is retried, the partial lands before the successful retry's + record, and both share the same iteration number. With a tool call it is + not retried. """ from datetime import datetime from unittest.mock import AsyncMock, MagicMock, patch - from coder_eval.errors import AgentCrashError from coder_eval.models import ( CommandTelemetry, CriterionResult, @@ -2085,6 +2031,8 @@ async def test_evaluation_loop_preserves_partial_on_crash_retry(tmp_path): SandboxConfig, TurnRecord, ) + from coder_eval.streaming.emitter import TurnOutcome + from coder_eval.streaming.events import AgentEndStatus agent_cfg = ClaudeCodeAgentConfig.model_construct( type=AgentKind.CLAUDE_CODE, @@ -2132,7 +2080,7 @@ async def test_evaluation_loop_preserves_partial_on_crash_retry(tmp_path): iteration=1, user_input="p", agent_output="", - commands=[partial_cmd], + commands=[partial_cmd] * tool_calls, duration_seconds=0.1, crashed=True, crash_reason="mid-turn failure", @@ -2151,9 +2099,8 @@ async def test_evaluation_loop_preserves_partial_on_crash_retry(tmp_path): async def crash_then_succeed_impl(_prompt, **kwargs): call_index[0] += 1 if call_index[0] == 1: - mock_agent.pending_turn = partial_record - raise AgentCrashError("mid-turn failure") - return success_record + return TurnOutcome(record=partial_record, status=AgentEndStatus.CRASHED, error="mid-turn failure") + return TurnOutcome(record=success_record, status=AgentEndStatus.COMPLETED, error=None) mock_agent.communicate.side_effect = crash_then_succeed_impl orchestrator.agent = mock_agent @@ -2173,16 +2120,24 @@ async def crash_then_succeed_impl(_prompt, **kwargs): patch("coder_eval.orchestrator.resolve_reference_dir", return_value=None), patch("asyncio.sleep", new_callable=AsyncMock), ): - success = await orchestrator._evaluation_loop() - - assert success is True + if tool_calls: + with pytest.raises(AgentCrashError): + await orchestrator._evaluation_loop() + else: + assert await orchestrator._evaluation_loop() is True + + if tool_calls: + assert mock_agent.communicate.call_count == 1 + [preserved] = orchestrator.result.iterations + assert preserved.crashed is True + assert preserved.commands[0].tool_name == "Skill" + return # communicate called twice: once crashing, once clean. assert mock_agent.communicate.call_count == 2 # Both records reach result.iterations: partial first (via callback), then successful (via main flow). assert len(orchestrator.result.iterations) == 2 preserved, clean = orchestrator.result.iterations assert preserved.crashed is True - assert preserved.commands[0].tool_name == "Skill" assert clean.crashed is False # Orchestrator-visible iteration number matches across both records. assert preserved.iteration == clean.iteration == 1 @@ -2207,6 +2162,8 @@ async def test_evaluation_loop_stamps_timeout_reason_on_partial(tmp_path): SandboxConfig, TurnRecord, ) + from coder_eval.streaming.emitter import TurnOutcome + from coder_eval.streaming.events import AgentEndStatus agent_cfg = ClaudeCodeAgentConfig.model_construct( type=AgentKind.CLAUDE_CODE, @@ -2255,8 +2212,9 @@ async def test_evaluation_loop_stamps_timeout_reason_on_partial(tmp_path): mock_agent = AsyncMock() async def timeout_impl(_prompt, **kwargs): - mock_agent.pending_turn = partial_record - raise TurnTimeoutError(600.0, iteration=1) + return TurnOutcome( + record=partial_record, status=AgentEndStatus.TIMEOUT, error="Agent turn timed out after 600s" + ) mock_agent.communicate.side_effect = timeout_impl orchestrator.agent = mock_agent @@ -2275,9 +2233,9 @@ async def timeout_impl(_prompt, **kwargs): with ( patch("coder_eval.orchestrator.resolve_reference_dir", return_value=None), patch("asyncio.sleep", new_callable=AsyncMock), - # TurnTimeoutError is non-retryable, so the loop re-raises after the - # on_attempt_error callback has already stamped + appended the partial. - # We only care about the side-effect, so suppress the re-raise. + # TurnTimeoutError is non-retryable, so the loop re-raises after + # `_communicate_with_retry` has already appended the TIMEOUT outcome's + # partial. We only care about the side-effect, so suppress the re-raise. pytest.raises(TurnTimeoutError), ): await orchestrator._evaluation_loop() @@ -2691,12 +2649,15 @@ async def communicate(self, user_input: str, **kwargs): from datetime import datetime from coder_eval.models import CommandTelemetry, TurnRecord + from coder_eval.streaming.emitter import TurnOutcome + from coder_eval.streaming.events import AgentEndStatus self._iteration += 1 skill = CommandTelemetry( tool_name="Skill", tool_id="s1", timestamp=datetime.now(), parameters={"skill": "probe-skill"} ) - return TurnRecord(iteration=self._iteration, user_input=user_input, agent_output="done", commands=[skill]) + record = TurnRecord(iteration=self._iteration, user_input=user_input, agent_output="done", commands=[skill]) + return TurnOutcome(record=record, status=AgentEndStatus.COMPLETED, error=None) def _patch_routes(monkeypatch) -> None: @@ -2735,7 +2696,7 @@ async def _spy(self, criteria, **kwargs): @pytest.mark.asyncio async def test_setup_stages_plugins_and_records_skills_offered(tmp_path, monkeypatch): - """Staging writes the canonical root, records the offer, and hands the root to the agent and the checker.""" + """Staging writes the plugin root, records the offer, and hands the root to the agent and the checker.""" from coder_eval.models import FinalStatus from coder_eval.path_utils import PLUGIN_ROOT_DIRNAME @@ -2831,3 +2792,67 @@ async def test_evaluate_only_reads_skills_offered_from_the_prior_result(tmp_path assert seen and all(offered == ("probe-skill",) for offered in seen) assert result.final_status == FinalStatus.ERROR assert "absent-skill" in (result.error_message or "") + + +@pytest.mark.asyncio +async def test_evaluation_loop_breaks_on_model_turn_cap(tmp_path): + """Turn N+1 starting latches the model-turn cap; the run records the cap fact like the tool-call cap.""" + from unittest.mock import patch + + from coder_eval.streaming.events import StopReason + + orchestrator = _cap_orchestrator(_cap_task("model_turn_cap_test", max_turns=3), tmp_path) + agent = _CooperativeToolAgent([(10, False)]) + orchestrator.agent = agent.host + + with patch("coder_eval.orchestrator.resolve_reference_dir", return_value=None): + success = await orchestrator._evaluation_loop() + + assert success is False + assert agent.emitted_per_attempt == [3] + assert orchestrator._monitor is not None + assert orchestrator._monitor.stop_reason is StopReason.MODEL_TURN_CAP + assert orchestrator._monitor.model_turns == 4 + assert orchestrator.result is not None + assert orchestrator.result.tool_calls_exhausted is True + + +@pytest.mark.asyncio +async def test_model_turns_is_recorded_and_expected_turns_warns(tmp_path, caplog): + import logging + from unittest.mock import patch + + orchestrator = _cap_orchestrator(_cap_task("expected_turns_test", expected_turns=2), tmp_path) + orchestrator._counts_model_turns = True + orchestrator.agent = _CooperativeToolAgent([(3, False)]).host + + with ( + caplog.at_level(logging.WARNING), + patch("coder_eval.orchestrator.resolve_reference_dir", return_value=None), + ): + await orchestrator._evaluation_loop() + + assert orchestrator.result is not None + assert orchestrator.result.model_turns == 3 + assert caplog.text.count("exceeded expected_turns") == 1 + assert orchestrator.result.tool_calls_exhausted is False + + +@pytest.mark.asyncio +async def test_model_turns_is_none_where_the_harness_does_not_count_them(tmp_path, caplog): + import logging + from unittest.mock import patch + + orchestrator = _cap_orchestrator(_cap_task("expected_turns_uncounted", expected_turns=2), tmp_path) + orchestrator._counts_model_turns = False + orchestrator.agent = _CooperativeToolAgent([(3, False)]).host + + with ( + caplog.at_level(logging.WARNING), + patch("coder_eval.orchestrator.resolve_reference_dir", return_value=None), + ): + await orchestrator._evaluation_loop() + + assert orchestrator.result is not None + assert orchestrator.result.model_turns is None + assert "exceeded expected_turns" not in caplog.text diff --git a/tests/test_pi_agent.py b/tests/test_pi_agent.py index 004c349f..c804b1ae 100644 --- a/tests/test_pi_agent.py +++ b/tests/test_pi_agent.py @@ -23,17 +23,28 @@ import pytest -from coder_eval.agents.pi_agent import PiAgent, _PiTurnState, _result_text -from coder_eval.errors import AgentCrashError, TurnTimeoutError -from coder_eval.models import AgentKind, AssistantMessage, CommandTelemetry, PiAgentConfig, TokenUsage +from coder_eval.agents.pi_agent import PiAgent, _PiDecoder, _result_text +from coder_eval.errors.agent import format_timeout_reason +from coder_eval.models import ( + AgentKind, + AssistantMessage, + FileExistsCriterion, + PiAgentConfig, + RunLimits, + SandboxConfig, + TaskDefinition, + parse_agent_config, +) from coder_eval.orchestration.plugin_staging import stage_plugins +from coder_eval.orchestration.turn_monitor import TurnMonitor from coder_eval.pricing import calculate_cost -from coder_eval.streaming.collector import EventCollector +from coder_eval.streaming.emitter import TurnEmitter, TurnOutcome from coder_eval.streaming.events import ( AgentEndEvent, AgentEndStatus, AgentStartEvent, StopReason, + StreamEvent, ToolEndEvent, ToolEndStatus, ToolStartEvent, @@ -41,6 +52,7 @@ TurnEndStatus, TurnStartEvent, ) +from coder_eval.testing import Replay, ScriptedClock, Tick, assert_identity_closes, assert_stream_balanced, replay from coder_eval.timing import TurnClock from tests._bracket_clock import AnchoredClock, assert_bracket_on_the_clock, assert_overhead_is_measured from tests._fixtures.golden_streams.pi_fixtures import ( @@ -79,9 +91,9 @@ async def fake_exec(*argv: str, **kwargs: Any) -> _FakeProcess: return _install -async def _run(agent: PiAgent, tmp_path: Any, prompt: str = "do the thing", **kwargs: Any): +async def _run(agent: PiAgent, tmp_path: Any, prompt: str = "do the thing", *, iteration: int = 1, **kwargs: Any): await agent.start(str(tmp_path)) - return await agent.communicate(prompt, **kwargs) + return await agent.communicate(prompt, iteration=iteration, **kwargs) def _agent(**overrides: Any) -> PiAgent: @@ -97,10 +109,58 @@ def on_event(self, event: Any) -> None: self.events.append(event) +_SPAN_BASE = datetime(2026, 3, 1, 9, 0, 0) + + +def _ms(at_ms: float) -> datetime: + return _SPAN_BASE + timedelta(milliseconds=at_ms) + + +def _event(line: str) -> dict[str, Any]: + return json.loads(line) + + +def _start() -> dict[str, Any]: + return {"type": "turn_start"} + + +def _end(*, inp: int = 10, out: int = 5) -> dict[str, Any]: + return { + "type": "turn_end", + "message": {"role": "assistant", "usage": {"input": inp, "output": out}, "stopReason": "stop"}, + } + + +def _open(call_id: str) -> dict[str, Any]: + return {"type": "tool_execution_start", "toolCallId": call_id, "toolName": "bash", "args": {}} + + +def _close(call_id: str) -> dict[str, Any]: + return {"type": "tool_execution_end", "toolCallId": call_id, "result": "ok"} + + +def _replay( + stream: list[Any], *, status: AgentEndStatus = AgentEndStatus.COMPLETED, reason: str | None = None +) -> tuple[Replay, _PiDecoder]: + """Drive a `_PiDecoder` through `coder_eval.testing.replay` from `_SPAN_BASE`; return the decoder too.""" + decoders: list[_PiDecoder] = [] + + def end(decoder: _PiDecoder) -> TurnOutcome: + decoders.append(decoder) + return decoder.end(status, reason=reason) + + return replay(stream, _PiDecoder, clock=ScriptedClock(_SPAN_BASE), end=end), decoders[0] + + +def _assistants(result: Replay) -> list[AssistantMessage]: + return [m for m in result.record.messages if isinstance(m, AssistantMessage)] + + class TestHappyPath: async def test_builds_turn_record(self, patch_exec, tmp_path): patch_exec(_FakeProcess(HAPPY_STREAM)) - record = await _run(_agent(), tmp_path) + outcome = await _run(_agent(), tmp_path) + record = outcome.record assert record.crashed is False # 3 turn_start steps in the fixture (write, read, summarize). @@ -110,7 +170,8 @@ async def test_builds_turn_record(self, patch_exec, tmp_path): async def test_token_buckets_sum_across_turns(self, patch_exec, tmp_path): patch_exec(_FakeProcess(HAPPY_STREAM)) - record = await _run(_agent(), tmp_path) + outcome = await _run(_agent(), tmp_path) + record = outcome.record usage = record.token_usage assert usage is not None @@ -123,7 +184,8 @@ async def test_token_buckets_sum_across_turns(self, patch_exec, tmp_path): async def test_reconciliation_invariant(self, patch_exec, tmp_path): """Summing the four buckets across messages must equal token_usage exactly.""" patch_exec(_FakeProcess(HAPPY_STREAM)) - record = await _run(_agent(), tmp_path) + outcome = await _run(_agent(), tmp_path) + record = outcome.record usage = record.token_usage assert usage is not None @@ -133,7 +195,8 @@ async def test_reconciliation_invariant(self, patch_exec, tmp_path): async def test_tool_calls_captured(self, patch_exec, tmp_path): patch_exec(_FakeProcess(HAPPY_STREAM)) - record = await _run(_agent(), tmp_path) + outcome = await _run(_agent(), tmp_path) + record = outcome.record # write + read, normalized to the canonical vocabulary. assert [c.tool_name for c in record.commands] == ["Write", "Read"] @@ -146,7 +209,8 @@ async def test_tool_calls_captured(self, patch_exec, tmp_path): async def test_messages_attributed_to_turns(self, patch_exec, tmp_path): patch_exec(_FakeProcess(HAPPY_STREAM)) - record = await _run(_agent(), tmp_path) + outcome = await _run(_agent(), tmp_path) + record = outcome.record assistants = [m for m in record.messages if isinstance(m, AssistantMessage)] assert len(assistants) == 3 @@ -191,7 +255,8 @@ async def test_bash_maps_to_canonical(self, patch_exec, tmp_path): json.dumps({"type": "agent_settled"}), ] patch_exec(_FakeProcess(stream)) - record = await _run(_agent(), tmp_path) + outcome = await _run(_agent(), tmp_path) + record = outcome.record assert record.commands[0].tool_name == "Bash" assert record.commands[0].parameters == {"command": "pytest -q"} @@ -207,7 +272,8 @@ async def test_find_maps_to_glob(self, patch_exec, tmp_path): json.dumps({"type": "agent_settled"}), ] patch_exec(_FakeProcess(stream)) - record = await _run(_agent(), tmp_path) + outcome = await _run(_agent(), tmp_path) + record = outcome.record assert record.commands[0].tool_name == "Glob" async def test_unknown_tool_passes_through(self, patch_exec, tmp_path): @@ -218,7 +284,8 @@ async def test_unknown_tool_passes_through(self, patch_exec, tmp_path): _turn_end(inp=10, out=5), ] patch_exec(_FakeProcess(stream)) - record = await _run(_agent(), tmp_path) + outcome = await _run(_agent(), tmp_path) + record = outcome.record assert record.commands[0].tool_name == "some_new_tool" assert record.commands[0].parameters == {"whatever": 1} @@ -230,7 +297,8 @@ async def test_edit_arg_keys_map_to_canonical(self, patch_exec, tmp_path): _turn_end(inp=10, out=5), ] patch_exec(_FakeProcess(stream)) - record = await _run(_agent(), tmp_path) + outcome = await _run(_agent(), tmp_path) + record = outcome.record assert record.commands[0].parameters == { "file_path": "a.py", "old_string": "a", @@ -274,6 +342,12 @@ async def test_explicit_line_limit_is_passed(self, patch_exec, tmp_path): await _run(_agent(), tmp_path) assert captured["kwargs"]["limit"] > 64 * 1024 + async def test_the_cli_never_inherits_stdin(self, patch_exec, tmp_path): + """Pi reads a non-TTY stdin to EOF before it emits; an inherited open stdin stalls the turn.""" + captured = patch_exec(_FakeProcess(HAPPY_STREAM)) + await _run(_agent(), tmp_path) + assert captured["kwargs"]["stdin"] is asyncio.subprocess.DEVNULL + class TestToolFlags: def test_allowlist_maps_claude_names_to_pi_tools(self): @@ -323,7 +397,7 @@ async def test_successive_calls_reuse_the_same_session(self, patch_exec, tmp_pat sdir1 = argv1[argv1.index("--session-dir") + 1] captured2 = patch_exec(_FakeProcess(HAPPY_STREAM)) - await agent.communicate("follow up") + await agent.communicate("follow up", iteration=2) argv2 = captured2["argv"] assert argv2[argv2.index("--session-id") + 1] == sid1 assert argv2[argv2.index("--session-dir") + 1] == sdir1 @@ -388,7 +462,7 @@ async def test_prepends_mock_dirs_ahead_of_the_inherited_path(self, patch_exec, captured = patch_exec(_FakeProcess(HAPPY_STREAM)) agent = _agent() await agent.start(str(tmp_path), env_path_prepend=["/sandbox/mocks", "/sandbox/bins"]) - await agent.communicate("do the thing") + await agent.communicate("do the thing", iteration=1) assert captured["kwargs"]["env"]["PATH"] == os.pathsep.join(["/sandbox/mocks", "/sandbox/bins", "/parent/bin"]) @@ -418,7 +492,8 @@ async def test_two_agent_cycles_reduce_to_one_agent_end(self, patch_exec, tmp_pa ] patch_exec(_FakeProcess(stream)) recorder = _EventRecorder() - record = await _run(_agent(), tmp_path, stream_callback=recorder) + outcome = await _run(_agent(), tmp_path, stream_callback=recorder) + record = outcome.record assert record.crashed is False assert len([e for e in recorder.events if isinstance(e, AgentEndEvent)]) == 1 @@ -449,9 +524,10 @@ async def test_early_criterion_ends_turn_stopped_early(self, patch_exec, tmp_pat proc = _RunningProcess(HAPPY_STREAM) patch_exec(proc) recorder = _EventRecorder() - record = await _run( + outcome = await _run( _agent(), tmp_path, should_stop=lambda: StopReason.EARLY_CRITERION, stream_callback=recorder ) + record = outcome.record assert record.crashed is False assert proc.terminated is True @@ -465,7 +541,8 @@ async def test_tool_call_cap_ends_turn_tool_calls_exhausted(self, patch_exec, tm proc = _RunningProcess(HAPPY_STREAM) patch_exec(proc) recorder = _EventRecorder() - record = await _run(_agent(), tmp_path, should_stop=lambda: StopReason.TOOL_CALL_CAP, stream_callback=recorder) + outcome = await _run(_agent(), tmp_path, should_stop=lambda: StopReason.TOOL_CALL_CAP, stream_callback=recorder) + record = outcome.record assert proc.terminated is True assert record.crashed is False @@ -477,7 +554,8 @@ async def test_token_budget_ends_turn_token_budget_exceeded(self, patch_exec, tm proc = _RunningProcess(HAPPY_STREAM) patch_exec(proc) recorder = _EventRecorder() - record = await _run(_agent(), tmp_path, should_stop=lambda: StopReason.TOKEN_BUDGET, stream_callback=recorder) + outcome = await _run(_agent(), tmp_path, should_stop=lambda: StopReason.TOKEN_BUDGET, stream_callback=recorder) + record = outcome.record assert proc.terminated is True assert record.crashed is False @@ -489,9 +567,10 @@ async def test_the_deciding_turn_is_kept_whole(self, patch_exec, tmp_path): """A stop that lands on turn 2's `turn_start` keeps turn 1 complete.""" second_turn_start = [i for i, line in enumerate(HAPPY_STREAM) if json.loads(line)["type"] == "turn_start"][1] patch_exec(_RunningProcess(HAPPY_STREAM)) - record = await _run( + outcome = await _run( _agent(), tmp_path, should_stop=_stop_after(second_turn_start + 1, StopReason.TOOL_CALL_CAP) ) + record = outcome.record assert record.tool_calls_exhausted is True assert len(record.commands) == 1 # turn 1's write @@ -503,19 +582,22 @@ async def test_the_deciding_turn_is_kept_whole(self, patch_exec, tmp_path): async def test_an_intentional_stop_is_exempt_from_a_non_zero_exit(self, patch_exec, tmp_path): """Killing the CLI makes it exit non-zero; that must not crash an intentional stop.""" patch_exec(_RunningProcess(HAPPY_STREAM, returncode=-15, stderr=b"terminated")) - record = await _run(_agent(), tmp_path, should_stop=lambda: StopReason.TOOL_CALL_CAP) + outcome = await _run(_agent(), tmp_path, should_stop=lambda: StopReason.TOOL_CALL_CAP) + record = outcome.record assert record.crashed is False assert record.tool_calls_exhausted is True async def test_an_intentional_stop_is_exempt_from_no_recognized_events(self, patch_exec, tmp_path): """A stop can land before the first recognized event; that is not vocabulary drift.""" patch_exec(_RunningProcess([json.dumps({"type": "not_a_pi_event"}), *HAPPY_STREAM])) - record = await _run(_agent(), tmp_path, should_stop=lambda: StopReason.TOKEN_BUDGET) + outcome = await _run(_agent(), tmp_path, should_stop=lambda: StopReason.TOKEN_BUDGET) + record = outcome.record assert record.crashed is False async def test_no_stop_is_uncapped(self, patch_exec, tmp_path): patch_exec(_FakeProcess(HAPPY_STREAM)) - record = await _run(_agent(), tmp_path, should_stop=lambda: None) + outcome = await _run(_agent(), tmp_path, should_stop=lambda: None) + record = outcome.record assert record.tool_calls_exhausted is False assert record.assistant_turn_count == 3 @@ -550,25 +632,65 @@ def kill(self) -> None: class TestTimeoutContract: - async def test_deadline_raises_turn_timeout_with_partial_parked(self, patch_exec, tmp_path): + async def test_deadline_returns_a_timeout_outcome_with_the_partial(self, patch_exec, tmp_path): proc = _HangingProcess([_turn_start()]) patch_exec(proc) agent = _agent() recorder = _EventRecorder() + ends_seen_at_kill: list[int] = [] + real_kill = agent.kill - with pytest.raises(TurnTimeoutError): - await _run(agent, tmp_path, timeout=0.2, stream_callback=recorder) + async def spy_kill() -> None: + ends_seen_at_kill.append(len([e for e in recorder.events if isinstance(e, AgentEndEvent)])) + await real_kill() - partial = agent.pending_turn + agent.kill = spy_kill # type: ignore[method-assign] + + outcome = await _run(agent, tmp_path, timeout=0.2, stream_callback=recorder) + + assert outcome.status is AgentEndStatus.TIMEOUT + assert outcome.error == format_timeout_reason(0.2) + partial = outcome.record assert partial is not None assert partial.crashed is True assert proc.terminated is True + assert ends_seen_at_kill[:1] == [0], "the CLI is killed BEFORE the turn ends" ends = [e for e in recorder.events if isinstance(e, AgentEndEvent)] assert len(ends) == 1 assert ends[0].status is AgentEndStatus.TIMEOUT - await agent.discard_pending_turn() - assert agent._iteration == 0 + +class _EofButAliveProcess(_HangingProcess): + """Stdout reaches EOF, but the process never exits until it is signalled.""" + + async def readline(self) -> bytes: + if self._lines: + return self._lines.pop(0) + return b"" + + +class TestSettleWaitsForTheExit: + async def test_no_exit_before_the_deadline_is_a_timeout(self, patch_exec, tmp_path): + proc = _EofButAliveProcess([_turn_start()]) + patch_exec(proc) + recorder = _EventRecorder() + outcome = await _run(_agent(), tmp_path, timeout=0.3, stream_callback=recorder) + assert outcome.status is AgentEndStatus.TIMEOUT + assert proc.terminated is True + assert [e.status for e in recorder.events if isinstance(e, AgentEndEvent)] == [AgentEndStatus.TIMEOUT] + + async def test_no_exit_without_a_deadline_is_a_crash(self, patch_exec, tmp_path, monkeypatch): + from coder_eval.agents._transport import subprocess_jsonl + + monkeypatch.setattr(subprocess_jsonl, "_EXIT_GRACE_SECONDS", 0.1) + proc = _EofButAliveProcess([_turn_start()]) + patch_exec(proc) + recorder = _EventRecorder() + outcome = await _run(_agent(), tmp_path, stream_callback=recorder) + assert outcome.status is AgentEndStatus.CRASHED + assert outcome.error is not None and "did not exit within" in outcome.error + assert proc.terminated is True + assert [e.status for e in recorder.events if isinstance(e, AgentEndEvent)] == [AgentEndStatus.CRASHED] class _ExplodingProcess(_FakeProcess): @@ -581,13 +703,15 @@ async def readline(self) -> bytes: class TestFailurePaths: async def test_nonzero_exit_crashes(self, patch_exec, tmp_path): patch_exec(_FakeProcess([], returncode=1, stderr=b"boom: bad model")) - with pytest.raises(AgentCrashError, match="boom: bad model"): - await _run(_agent(), tmp_path) + outcome = await _run(_agent(), tmp_path) + assert outcome.status is AgentEndStatus.CRASHED + assert outcome.error is not None and "boom: bad model" in outcome.error async def test_empty_clean_exit_crashes_on_no_recognized_events(self, patch_exec, tmp_path): patch_exec(_FakeProcess([], returncode=0)) - with pytest.raises(AgentCrashError, match="no recognized events"): - await _run(_agent(), tmp_path) + outcome = await _run(_agent(), tmp_path) + assert outcome.status is AgentEndStatus.CRASHED + assert outcome.error is not None and "no recognized events" in outcome.error async def test_drift_crash_names_the_unrecognized_types(self, patch_exec, tmp_path): """A clean exit whose events are all unrecognized (schema drift) crashes and @@ -597,19 +721,22 @@ async def test_drift_crash_names_the_unrecognized_types(self, patch_exec, tmp_pa json.dumps({"type": "another.unknown", "bar": 2}), ] patch_exec(_FakeProcess(stream)) - with pytest.raises(AgentCrashError, match=r"another\.unknown, some\.new\.event") as exc: - await _run(_agent(), tmp_path) - assert "no recognized events" in str(exc.value) + outcome = await _run(_agent(), tmp_path) + assert outcome.status is AgentEndStatus.CRASHED + assert outcome.error is not None + assert "another.unknown, some.new.event" in outcome.error + assert "no recognized events" in outcome.error - async def test_stream_error_becomes_a_crash_with_partial_parked(self, patch_exec, tmp_path): + async def test_stream_error_becomes_a_crash_with_the_crashed_partial(self, patch_exec, tmp_path): stream = [_turn_start(), _tool_start("w:0", "write", {"path": "a.txt", "content": "x"})] patch_exec(_ExplodingProcess(stream)) agent = _agent() - with pytest.raises(AgentCrashError, match="Pi turn failed"): - await _run(agent, tmp_path) + outcome = await _run(agent, tmp_path) - partial = agent.pending_turn + assert outcome.status is AgentEndStatus.CRASHED + assert outcome.error is not None and "Pi turn failed" in outcome.error + partial = outcome.record assert partial is not None assert partial.crashed is True # The in-flight tool was force-closed rather than dropped. @@ -622,13 +749,15 @@ async def boom(*_argv: str, **_kwargs: Any): monkeypatch.setattr(asyncio, "create_subprocess_exec", boom) monkeypatch.setattr("shutil.which", lambda _name: "/usr/local/bin/pi") - with pytest.raises(AgentCrashError, match="no fork for you"): - await _run(_agent(), tmp_path) + outcome = await _run(_agent(), tmp_path) + assert outcome.status is AgentEndStatus.CRASHED + assert outcome.error is not None and "no fork for you" in outcome.error async def test_malformed_line_is_skipped(self, patch_exec, tmp_path): stream = ["not json at all", *HAPPY_STREAM] patch_exec(_FakeProcess(stream)) - record = await _run(_agent(), tmp_path) + outcome = await _run(_agent(), tmp_path) + record = outcome.record assert record.crashed is False assert record.assistant_turn_count == 3 @@ -643,25 +772,14 @@ async def test_terminal_event_emitted_exactly_once_on_crash(self, patch_exec, tm patch_exec(_ExplodingProcess([_turn_start()])) recorder = _EventRecorder() - with pytest.raises(AgentCrashError): - await _run(_agent(), tmp_path, stream_callback=recorder) + outcome = await _run(_agent(), tmp_path, stream_callback=recorder) + assert outcome.status is AgentEndStatus.CRASHED ends = [e for e in recorder.events if isinstance(e, AgentEndEvent)] assert len(ends) == 1 assert ends[0].crashed is True assert ends[0].status is AgentEndStatus.CRASHED - async def test_iteration_rolls_back_after_the_crash(self, patch_exec, tmp_path): - patch_exec(_ExplodingProcess([])) - agent = _agent() - - with pytest.raises(AgentCrashError): - await _run(agent, tmp_path) - assert agent._iteration == 1 - await agent.discard_pending_turn() - assert agent._iteration == 0 - assert agent.pending_turn is None - class TestTurnEventsAreBalanced: @staticmethod @@ -681,8 +799,8 @@ async def test_a_timeout_closes_the_open_turn(self, patch_exec, tmp_path): patch_exec(proc) recorder = _EventRecorder() - with pytest.raises(TurnTimeoutError): - await _run(_agent(), tmp_path, timeout=0.2, stream_callback=recorder) + outcome = await _run(_agent(), tmp_path, timeout=0.2, stream_callback=recorder) + assert outcome.status is AgentEndStatus.TIMEOUT assert self._pairs(recorder) == (1, 1) end = next(e for e in recorder.events if isinstance(e, TurnEndEvent)) @@ -763,7 +881,8 @@ async def test_tool_error_is_captured(self, patch_exec, tmp_path): ] patch_exec(_FakeProcess(stream)) recorder = _EventRecorder() - record = await _run(_agent(), tmp_path, stream_callback=recorder) + outcome = await _run(_agent(), tmp_path, stream_callback=recorder) + record = outcome.record [cmd] = record.commands assert cmd.result_status == "error" @@ -785,15 +904,12 @@ async def test_permission_denial_gets_its_own_status(self, patch_exec, tmp_path) assert end.status is ToolEndStatus.PERMISSION_DENIED def test_orphan_result_is_never_dropped(self): - state = _PiTurnState(task_id="t", iteration=1, user_input="x", model=None) - events: list[Any] = [] - state.bind(events.append) - state._close_tool("ghost", status=ToolEndStatus.UNRESOLVED, summary=None, error="no result observed") + result, _ = _replay([_close("ghost")]) - [event] = events - assert isinstance(event, ToolEndEvent) + [event] = [e for e in result.events if isinstance(e, ToolEndEvent)] + assert event.tool.tool_id == "ghost" assert event.tool.tool_name == "unknown" - assert event.tool.result_status == "unknown" + assert [c.tool_id for c in result.record.commands] == ["ghost"] class TestZeroUsageTurn: @@ -807,7 +923,8 @@ async def test_zero_usage_turn_is_scored_not_crashed(self, patch_exec, tmp_path) json.dumps({"type": "agent_settled"}), ] patch_exec(_FakeProcess(stream)) - record = await _run(_agent(), tmp_path) + outcome = await _run(_agent(), tmp_path) + record = outcome.record assert record.crashed is False @@ -857,9 +974,10 @@ async def test_terminal_error_crashes_the_turn(self, patch_exec, tmp_path): stream = [_turn_start(), _turn_end_error("404: blocked by guardrail"), json.dumps({"type": "agent_settled"})] patch_exec(_FakeProcess(stream)) agent = _agent() - with pytest.raises(AgentCrashError, match="blocked by guardrail"): - await _run(agent, tmp_path) - partial = agent.pending_turn + outcome = await _run(agent, tmp_path) + assert outcome.status is AgentEndStatus.CRASHED + assert outcome.error is not None and "blocked by guardrail" in outcome.error + partial = outcome.record assert partial is not None assert partial.crashed is True @@ -869,17 +987,63 @@ async def test_a_stop_after_an_error_turn_finalizes_cleanly(self, patch_exec, tm status — NOT crash on the stale error.""" stream = [_turn_start(), _turn_end_error("transient 429"), _turn_start(), _turn_end(inp=1, out=1)] patch_exec(_RunningProcess(stream)) - record = await _run(_agent(), tmp_path, should_stop=_stop_after(3, StopReason.TOOL_CALL_CAP)) + outcome = await _run(_agent(), tmp_path, should_stop=_stop_after(3, StopReason.TOOL_CALL_CAP)) + record = outcome.record assert record.tool_calls_exhausted is True assert record.crashed is False def test_error_message_resets_on_a_recovered_turn(self): """#3: an intermediate error a later cycle recovers from must not leak into the result.""" - state = _PiTurnState(task_id="t", iteration=1, user_input="x", model="m") - state.on_turn_end(json.loads(_turn_end_error("transient"))) - assert state.error_message == "transient" - state.on_turn_end(json.loads(_turn_end(inp=5, out=2))) # a later, successful turn - assert state.error_message is None + _, errored = _replay([_start(), _event(_turn_end_error("transient"))]) + assert errored.error == "transient" + _, recovered = _replay([_start(), _event(_turn_end_error("transient")), _start(), _end()]) + assert recovered.error is None + + def test_dangling_turn_is_closed_crashed_at_the_next_turn_start(self): + result, _ = _replay([_start(), _start(), _end()]) + + turn_ends = [e for e in result.events if isinstance(e, TurnEndEvent)] + assert [(e.turn_id, e.status) for e in turn_ends] == [ + ("turn_1", TurnEndStatus.CRASHED), + ("turn_2", TurnEndStatus.COMPLETED), + ] + assert_stream_balanced(result.events) + + def test_a_duplicate_turn_end_publishes_no_second_turn_end_but_books_its_tokens(self): + result, _ = _replay([_start(), _end(inp=10, out=5), _end(inp=7, out=3)]) + + assert len([e for e in result.events if isinstance(e, TurnEndEvent)]) == 1 + assert len(_assistants(result)) == 2 + [agent_end] = [e for e in result.events if isinstance(e, AgentEndEvent)] + assert agent_end.usage.uncached_input_tokens == 17 + assert agent_end.usage.output_tokens == 8 + assert_stream_balanced(result.events) + + def test_token_shape_warns_once_per_turn(self, caplog): + no_usage = {"type": "turn_end", "message": {"role": "assistant", "stopReason": "stop"}} + bad_total = { + "type": "turn_end", + "message": {"role": "assistant", "usage": {"input": 1, "output": 1, "totalTokens": 99}}, + } + + def warnings() -> int: + return len([r for r in caplog.records if "unexpected token accounting" in r.getMessage()]) + + with caplog.at_level("WARNING"): + _replay([_start(), no_usage, _start(), bad_total]) + assert warnings() == 1 + _replay([_start(), bad_total]) + assert warnings() == 2 # the flag is per turn, not per agent + + def test_total_tokens_mismatch_warns_at_the_decoder(self, caplog): + bad_total = { + "type": "turn_end", + "message": {"role": "assistant", "usage": {"input": 10, "output": 5, "totalTokens": 999}}, + } + with caplog.at_level("WARNING"): + _, decoder = _replay([_start(), bad_total]) + assert decoder.warned_token_shape is True + assert any("does not reconcile" in r.getMessage() for r in caplog.records) async def test_bad_token_bucket_warns_once(self, patch_exec, tmp_path, caplog): """#1: a bucket whose type drifted (here a dict) coerces to 0 but warns, once.""" @@ -930,7 +1094,8 @@ async def test_all_zero_usage_object_warns(self, patch_exec, tmp_path, caplog): stream = [_turn_start(), zero, json.dumps({"type": "agent_settled"})] patch_exec(_FakeProcess(stream)) with caplog.at_level("WARNING"): - record = await _run(_agent(), tmp_path) + outcome = await _run(_agent(), tmp_path) + record = outcome.record assert record.crashed is False # score, don't crash (documented Pi policy) assert any("all-zero token buckets" in r.getMessage() for r in caplog.records) @@ -953,7 +1118,8 @@ async def test_total_tokens_mismatch_warns(self, patch_exec, tmp_path, caplog): stream = [_turn_start(), bad, json.dumps({"type": "agent_settled"})] patch_exec(_FakeProcess(stream)) with caplog.at_level("WARNING"): - record = await _run(_agent(), tmp_path) + outcome = await _run(_agent(), tmp_path) + record = outcome.record assert record.crashed is False assert any("does not reconcile" in r.getMessage() for r in caplog.records) @@ -972,7 +1138,7 @@ async def test_staged_root_emits_skill_arg(self, patch_exec, tmp_path): captured = patch_exec(_FakeProcess(HAPPY_STREAM)) agent = _agent() await agent.start(str(tmp_path), plugin_root=root) - await agent.communicate("do the thing") + await agent.communicate("do the thing", iteration=1) argv = captured["argv"] assert argv[argv.index("--skill") + 1] == str(root / "skills") assert "pi_skill_paths" not in agent.get_environment_info() @@ -984,20 +1150,21 @@ async def test_no_plugin_root_means_no_skill_arg(self, patch_exec, tmp_path): class TestTurnAlwaysReapsTheCli: - """No exit from ``communicate()`` may leave the CLI running. ``AgentCrashError`` - is categorized AGENT_CRASH (max_retries=2) and the orchestrator's attempt-failure - hook only drains ``pending_turn`` — it never kills the agent. An abandoned CLI + """No exit from ``communicate()`` may leave the CLI running. A crash is + categorized AGENT_CRASH (max_retries=2), and the orchestrator only appends the + crashed record — it never kills the agent. An abandoned CLI therefore means attempt 2 spawns a SECOND ``pi`` editing the very files the criteria are about to score. The graceful ``kill()`` covers the intentional cuts and the timeout; these pin the two paths that reach ``finally`` with a live child. """ async def test_read_loop_crash_kills_the_cli(self, patch_exec, tmp_path): - """``_crash_turn`` is synchronous and raises — nothing below it reaps.""" + """A read-loop crash ends the turn as an outcome, and ``finally`` still reaps the live CLI.""" proc = _ExplodingRunningProcess([_turn_start()]) patch_exec(proc) - with pytest.raises(AgentCrashError, match="Pi turn failed"): - await _run(_agent(), tmp_path) + outcome = await _run(_agent(), tmp_path) + assert outcome.status is AgentEndStatus.CRASHED + assert outcome.error is not None and "Pi turn failed" in outcome.error assert proc.killed is True async def test_external_cancel_kills_the_cli(self, patch_exec, tmp_path): @@ -1007,50 +1174,48 @@ async def test_external_cancel_kills_the_cli(self, patch_exec, tmp_path): patch_exec(proc) agent = _agent() await agent.start(str(tmp_path)) - task = asyncio.ensure_future(agent.communicate("do the thing")) + task = asyncio.ensure_future(agent.communicate("do the thing", iteration=1)) await asyncio.sleep(0.05) # let it spawn and read the first event task.cancel() with pytest.raises(asyncio.CancelledError): _ = await task # the await re-raises the cancellation; no value ever exists assert proc.killed is True - async def test_a_clean_turn_kills_nothing(self, patch_exec, tmp_path): - """The happy path is unchanged: the CLI exited, so the reaper is a no-op.""" + async def test_a_clean_turn_sweeps_only_the_group(self, patch_exec, tmp_path): + """The CLI exited, so it is not killed; any child it left is swept with the turn.""" proc = _FakeProcess(HAPPY_STREAM) captured = patch_exec(proc) await _run(_agent(), tmp_path) assert proc.killed is False - assert captured["killpg"] == [] + assert captured["killpg"] == [(4242, signal.SIGKILL)] class TestExternalCancel: - async def test_cancel_parks_partial_and_reraises(self, patch_exec, tmp_path): + async def test_cancel_ends_the_turn_and_reraises(self, patch_exec, tmp_path): """The watchdog's CancelledError must not swallow captured telemetry: the - partial is parked, the terminal event says CRASHED, and the cancellation + turn is ended, the terminal event says CRASHED, and the cancellation still propagates.""" proc = _HangingProcess([_turn_start()]) patch_exec(proc) agent = _agent() await agent.start(str(tmp_path)) recorder = _EventRecorder() - task = asyncio.ensure_future(agent.communicate("do the thing", stream_callback=recorder)) + task = asyncio.ensure_future(agent.communicate("do the thing", iteration=1, stream_callback=recorder)) await asyncio.sleep(0.05) task.cancel() with pytest.raises(asyncio.CancelledError): _ = await task # the await re-raises the cancellation; no value ever exists - partial = agent.pending_turn - assert partial is not None - assert partial.crashed is True ends = [e for e in recorder.events if isinstance(e, AgentEndEvent)] assert len(ends) == 1 assert ends[0].status is AgentEndStatus.CRASHED + assert ends[0].crashed is True assert ends[0].crash_reason == "turn cancelled" assert proc.killed is True # not abandoned mid-stream — see TestTurnAlwaysReapsTheCli def _turn_end_no_cost(*, inp: int, out: int) -> str: """A `turn_end` whose usage object omits the `cost` key (provider/auth mode that - reports no cost) — so `_resolve_cost` must fall back to the rate card.""" + reports no cost) — so `price_turn` must fall back to the rate card.""" return json.dumps( { "type": "turn_end", @@ -1071,7 +1236,8 @@ async def test_stream_cost_wins_when_reported(self, patch_exec, tmp_path): """The provider's own accounting beats a static headline rate.""" stream = [_turn_start(), _turn_end(inp=1000, out=500, cost=0.5), self._SETTLED] patch_exec(_FakeProcess(stream)) - record = await _run(_agent(), tmp_path) + outcome = await _run(_agent(), tmp_path) + record = outcome.record assert record.token_usage is not None assert record.token_usage.total_cost_usd == pytest.approx(0.5) @@ -1079,7 +1245,8 @@ async def test_missing_cost_is_priced_from_the_rate_card(self, patch_exec, tmp_p """No `cost` key at all — without the fallback the turn books tokens with no money.""" stream = [_turn_start(), _turn_end_no_cost(inp=1000, out=500), self._SETTLED] patch_exec(_FakeProcess(stream)) - record = await _run(_agent(), tmp_path) + outcome = await _run(_agent(), tmp_path) + record = outcome.record expected = calculate_cost("openrouter/moonshotai/kimi-k3", uncached_input_tokens=1000, output_tokens=500) assert expected is not None and expected > 0 assert record.token_usage is not None @@ -1089,7 +1256,8 @@ async def test_unpriced_model_reports_no_cost(self, patch_exec, tmp_path): """`None` (not 0.0) so "unpriceable" stays distinct from "ran for free".""" stream = [_turn_start(), _turn_end_no_cost(inp=10, out=5), self._SETTLED] patch_exec(_FakeProcess(stream)) - record = await _run(_agent(model="nowhere/not-a-real-model"), tmp_path) + outcome = await _run(_agent(model="nowhere/not-a-real-model"), tmp_path) + record = outcome.record assert record.token_usage is not None assert record.token_usage.total_cost_usd is None @@ -1099,105 +1267,65 @@ async def test_zero_reported_cost_on_a_priced_model_uses_the_rate_card(self, pat stream = [_turn_start(), _turn_end(inp=1000, out=500, cost=0.0), self._SETTLED] patch_exec(_FakeProcess(stream)) with caplog.at_level("DEBUG"): - record = await _run(_agent(), tmp_path) + outcome = await _run(_agent(), tmp_path) + record = outcome.record expected = calculate_cost("openrouter/moonshotai/kimi-k3", uncached_input_tokens=1000, output_tokens=500) assert expected is not None and expected > 0 assert record.token_usage is not None assert record.token_usage.total_cost_usd == pytest.approx(expected) - assert "not understated" in caplog.text + assert "using the rate card" in caplog.text async def test_zero_reported_cost_on_an_unpriced_model_stays_zero(self, patch_exec, tmp_path): """With no rate to fall back to, the stream's 0 is the best information we have.""" stream = [_turn_start(), _turn_end(inp=10, out=5, cost=0.0), self._SETTLED] patch_exec(_FakeProcess(stream)) - record = await _run(_agent(model="nowhere/not-a-real-model"), tmp_path) + outcome = await _run(_agent(model="nowhere/not-a-real-model"), tmp_path) + record = outcome.record assert record.token_usage is not None assert record.token_usage.total_cost_usd == 0.0 -class _FixedClock: - """A `TurnClock` stand-in frozen at one instant, injected into the state.""" - - def __init__(self, at: datetime) -> None: - self.at = at - - def now(self) -> datetime: - return self.at - - class TestGenerationWindowExcludesToolExecution: """A tool running inside a turn is not model time — asserted where it is now DECIDED. - The reducer no longer subtracts anything. It publishes the RAW window, and + The decoder no longer subtracts anything. It publishes the RAW window, and `timing.subtract_tool_time` takes the tool union back out of it - once, for all five harnesses. So these cases drive the reducer and then a - real collector, and assert the PUBLISHED number — the one that reaches - `task.json` — rather than an intermediate the reducer used to own. + once, for all five harnesses. So these cases replay the decoder through a + real emitter and assert the PUBLISHED number — the one that reaches + `task.json` — rather than an intermediate the decoder used to own. They are not duplicates of `tests/test_event_collector.py::TestSubtractToolTime`: those pin the - arithmetic, these pin that THIS reducer hands the collector a window and a + arithmetic, these pin that THIS decoder hands the collector a window and a span set the arithmetic can be right about. """ - WINDOW_START = datetime(2026, 1, 1, 12, 0, 0) - WINDOW_END = datetime(2026, 1, 1, 12, 0, 1) # a 1000ms turn - - def _finish_turn(self, spans, open_starts=()): - """Drive the reducer, then publish through a real collector. + def _finish_turn(self, spans: list[tuple[float, float]], open_starts: tuple[float, ...] = ()) -> AssistantMessage: + """Replay one 0 -> 1000 ms turn with tool calls at the given ms offsets. `spans` are RESOLVED calls (both bounds); `open_starts` are calls that - never returned. An unresolved call now contributes NO span — it has no + never returned. An unresolved call contributes NO span — it has no `execution_completed_at`, and inventing one is what `None` exists to - prevent — where the reducer used to bound it at the window's end. That - is a real change and a better one: the collector sees every span at - once, so a call straddling a boundary is clipped to each window it - actually overlapped instead of approximated at the boundary. + prevent. The collector sees every span at once, so a call straddling a + boundary is clipped to each window it actually overlapped. """ - state = _PiTurnState(task_id="t", iteration=1, user_input="x", model="m", clock=_FixedClock(self.WINDOW_END)) - state.turn_started_at = self.WINDOW_START - commands = [ - CommandTelemetry( - tool_name="bash", - tool_id=f"closed-{i}", - timestamp=started, - execution_started_at=started, - execution_completed_at=completed, - result_status="success", - ) - for i, (started, completed) in enumerate(spans) - ] - commands += [ - CommandTelemetry(tool_name="bash", tool_id=f"open-{i}", timestamp=s, execution_started_at=s) - for i, s in enumerate(open_starts) - ] - state.on_turn_end( - {"message": {"role": "assistant", "usage": {"input": 100, "output": 20}, "stopReason": "stop"}} - ) - - collector = EventCollector() - collector.on_event(AgentStartEvent(task_id="t", prompt="x", iteration=1, timestamp=self.WINDOW_START)) - for command in commands: - collector.on_event(ToolEndEvent(task_id="t", turn_id="t1", tool=command)) - collector.on_event( - AgentEndEvent( - task_id="t", - status=AgentEndStatus.COMPLETED, - messages=list(state.messages), - usage=TokenUsage(), - timestamp=self.WINDOW_END, - ) - ) - published = [m for m in collector.build_turn_record().messages if m.role == "assistant"] + timeline: list[tuple[float, dict[str, Any]]] = [(0.0, _start()), (1000.0, _end(inp=100, out=20))] + for i, (started, completed) in enumerate(spans): + timeline += [(started, _open(f"closed-{i}")), (completed, _close(f"closed-{i}"))] + timeline += [(started, _open(f"open-{i}")) for i, started in enumerate(open_starts)] + stream: list[Any] = [] + for at_ms, event in sorted(timeline, key=lambda item: item[0]): + stream += [Tick(at_ms), event] + + result, _ = _replay(stream) + published = _assistants(result) assert len(published) == 1 return published[0] def test_tool_time_inside_the_turn_is_subtracted(self): - message = self._finish_turn( - [(self.WINDOW_START + timedelta(milliseconds=200), self.WINDOW_START + timedelta(milliseconds=700))], - ) + message = self._finish_turn([(200, 700)]) span_ms = (message.completed_at - message.started_at).total_seconds() * 1000.0 - assert span_ms == pytest.approx(1000.0), "the reducer still publishes the whole window as its bounds" + assert span_ms == pytest.approx(1000.0), "the decoder still publishes the whole window as its bounds" assert message.generation_duration_ms == pytest.approx(500.0) def test_a_turn_with_no_tools_keeps_its_whole_window(self): @@ -1206,36 +1334,24 @@ def test_a_turn_with_no_tools_keeps_its_whole_window(self): def test_concurrent_tools_are_subtracted_once(self): # Two overlapping 500ms tools occupy 600ms, not 1000ms. Summing them # would leave 0 generation for a turn that generated 400. - message = self._finish_turn( - [ - (self.WINDOW_START + timedelta(milliseconds=100), self.WINDOW_START + timedelta(milliseconds=600)), - (self.WINDOW_START + timedelta(milliseconds=200), self.WINDOW_START + timedelta(milliseconds=700)), - ], - ) + message = self._finish_turn([(100, 600), (200, 700)]) assert message.generation_duration_ms == pytest.approx(400.0) def test_the_window_never_goes_negative(self): - message = self._finish_turn( - [(self.WINDOW_START - timedelta(seconds=30), self.WINDOW_END + timedelta(seconds=30))], - ) + message = self._finish_turn([(-30_000, 31_000)]) assert message.generation_duration_ms == 0.0 def test_a_tool_still_open_at_the_boundary_contributes_no_span(self): - """The behaviour that CHANGED with the move, stated rather than implied. + """A call with no `execution_completed_at` was never timed. - The reducer used to bound a still-open call at the window's end and - subtract that slice. The collector cannot: a call with no - `execution_completed_at` was never timed. Its time is subtracted when it - RESOLVES, from whichever windows its real interval overlaps. + Its time is subtracted when it RESOLVES, from whichever windows its real + interval overlaps — never bounded at the window's end. """ - message = self._finish_turn([], open_starts=[self.WINDOW_START + timedelta(milliseconds=600)]) + message = self._finish_turn([], open_starts=(600,)) assert message.generation_duration_ms == pytest.approx(1000.0) def test_a_resolved_tool_overlapping_an_unresolved_one_counts_only_the_resolved(self): - message = self._finish_turn( - [(self.WINDOW_START + timedelta(milliseconds=200), self.WINDOW_START + timedelta(milliseconds=700))], - open_starts=[self.WINDOW_START + timedelta(milliseconds=500)], - ) + message = self._finish_turn([(200, 700)], open_starts=(500,)) assert message.generation_duration_ms == pytest.approx(500.0) def test_the_published_window_reconciles_to_its_own_bounds(self): @@ -1249,44 +1365,18 @@ def test_the_published_window_reconciles_to_its_own_bounds(self): """ from coder_eval.timing import busy_ms - closed = [(self.WINDOW_START + timedelta(milliseconds=200), self.WINDOW_START + timedelta(milliseconds=700))] - message = self._finish_turn(closed) + message = self._finish_turn([(200, 700)]) span_ms = (message.completed_at - message.started_at).total_seconds() * 1000.0 - expected = span_ms - busy_ms(closed, message.started_at, message.completed_at) + expected = span_ms - busy_ms([(_ms(200), _ms(700))], message.started_at, message.completed_at) assert message.generation_duration_ms == pytest.approx(expected) -_SPAN_BASE = datetime(2026, 3, 1, 9, 0, 0) - - -class _SteppedClock: - """A `TurnClock` stand-in the test moves by hand, in ms from `_SPAN_BASE`. - - INJECTED, never monkeypatched onto the module. Pi derives every wall stamp - from its turn clock now, so patching `agent_module.datetime` would no - longer reach it: the tests would quietly start measuring the real clock and - pass by accident instead of failing. Injection also puts the "one clock per - turn" lifetime in the constructor signature where it can be read. - """ - - def __init__(self, at_ms: float = 0.0) -> None: - self.at_ms = at_ms - - def now(self) -> datetime: - return _SPAN_BASE + timedelta(milliseconds=self.at_ms) - - -def _turn_end_payload(): - return {"message": {"role": "assistant", "usage": {"input": 10, "output": 5}, "stopReason": "stop"}} - - class TestGenerationWindowsTileTheTurn: """Each window runs from the PREVIOUS `turn_end`, not from its own `turn_start`. - Pi was the only harness measuring from its own turn start, so the wall - clock between one `turn_end` and the next `turn_start` — the model time - that PRODUCED the next turn — fell into no bucket at all. The four-bucket - identity is asserted only as an upper bound, so nothing failed. + Measured from its own turn start, the wall clock between one `turn_end` and + the next `turn_start` — the model time that PRODUCED the next turn — fell + into no bucket at all. The gap is small in practice (measured across 25 real window pairs: median 0.25 ms, max 0.75 ms). The value here is that it closes, and that the tool @@ -1294,17 +1384,9 @@ class TestGenerationWindowsTileTheTurn: which is the half that carries the weight. """ - def _two_turns(self): - clock = _SteppedClock() - state = _PiTurnState(task_id="t", iteration=1, user_input="go", model="m", clock=clock) - state.on_turn_start() - clock.at_ms = 1000 - state.on_turn_end(_turn_end_payload()) - clock.at_ms = 1600 - state.on_turn_start() - clock.at_ms = 2000 - state.on_turn_end(_turn_end_payload()) - return [m for m in state.messages if m.role == "assistant"] + def _two_turns(self) -> list[AssistantMessage]: + result, _ = _replay([_start(), Tick(1000), _end(), Tick(1600), _start(), Tick(2000), _end()]) + return _assistants(result) def test_the_second_window_abuts_the_first(self): messages = self._two_turns() @@ -1321,64 +1403,38 @@ def test_the_inter_turn_gap_is_inside_a_window_rather_than_unaccounted(self): class TestToolSpansSurviveTheTurnBoundary: """A tool that closes BETWEEN two turns still belongs to the next window. - This used to be a bookkeeping problem: a per-turn span list, cleared at - `turn_start` — after the window it feeds had already opened at the mark — - so a call closing in the gap had its span wiped before the flush could - subtract it. That list is gone. `timing.subtract_tool_time` sees - every span at once and clips each to the windows it overlaps, so the - property now holds by construction rather than by a reset rule. - - Kept, and re-pointed at the collector, because the property itself is what - matters and a future reducer change could still break it — by moving a - mark, or by failing to emit the ToolEnd the collector reduces. + `timing.subtract_tool_time` sees every span at once and clips each to the + windows it overlaps, so the property holds by construction rather than by a + reset rule. Kept because the property itself is what matters and a future + decoder change could still break it — by moving a mark, or by failing to + close the tool the collector reduces. """ - def _run(self): - clock = _SteppedClock() - state = _PiTurnState(task_id="t", iteration=1, user_input="go", model="m", clock=clock) - # The resolved telemetry leaves the state via ToolEnd; the identity - # case below reconciles against what was RECORDED, not against the - # clock the test scripted. - resolved: list[Any] = [] - state.bind(lambda e: resolved.append(e.tool) if isinstance(e, ToolEndEvent) else None) - state.on_turn_start() - clock.at_ms = 100 - state.on_tool_execution_start({"toolCallId": "c1", "toolName": "bash", "args": {}}) - clock.at_ms = 1000 - state.on_turn_end(_turn_end_payload()) - clock.at_ms = 1500 - state.on_tool_execution_end({"toolCallId": "c1", "result": "ok"}) # closes in the GAP - clock.at_ms = 1600 - state.on_turn_start() - clock.at_ms = 2000 - state.on_turn_end(_turn_end_payload()) - - # Published through the real collector: the reducer hands over raw - # windows, and the tool subtraction happens once, there. - collector = EventCollector() - collector.on_event(AgentStartEvent(task_id="t", prompt="go", iteration=1, timestamp=_SPAN_BASE)) - for command in resolved: - collector.on_event(ToolEndEvent(task_id="t", turn_id="t1", tool=command)) - collector.on_event( - AgentEndEvent( - task_id="t", - status=AgentEndStatus.COMPLETED, - messages=list(state.messages), - usage=TokenUsage(), - timestamp=_SPAN_BASE + timedelta(milliseconds=2000), - ) - ) - published = [m for m in collector.build_turn_record().messages if m.role == "assistant"] - return resolved, published + def _run(self) -> Replay: + return _replay( + [ + _start(), + Tick(100), + _open("c1"), + Tick(1000), + _end(), + Tick(1500), + _close("c1"), # closes in the GAP + Tick(1600), + _start(), + Tick(2000), + _end(), + ] + )[0] def test_the_gap_slice_of_a_straddling_call_is_not_published_as_generation(self): - _, messages = self._run() + messages = _assistants(self._run()) # Window 2 tiles 1000 -> 2000. c1 ran for 1000 -> 1500 of it, so 500ms # is model time. With the reset left at `turn_start` this reads 1000.0. assert messages[1].generation_duration_ms == pytest.approx(500.0) def test_the_call_is_subtracted_from_exactly_one_window(self): - _, messages = self._run() + messages = _assistants(self._run()) # Window 1 bounded c1 at its own close (100 -> 1000); window 2 takes # only the remainder. assert messages[0].generation_duration_ms == pytest.approx(100.0) @@ -1390,71 +1446,49 @@ def test_the_four_bucket_identity_closes_exactly_across_the_boundary(self): This is the assertion the golden corpus CANNOT make: `_scrub.py` masks `generation_duration_ms` and both bounds to a placeholder, so a snapshot records that a window was measured and never what it measured. - Its identity check (`_scrub.py`) is an upper bound besides, so - under-accounting — the defect this phase fixes — passes it silently. - `scripts/timing/decompose_run.py --max-residual-pct` is the two-sided - check on live runs; this is the committed one. """ from coder_eval.timing import busy_ms - resolved, messages = self._run() + result = self._run() + messages = _assistants(result) lo, hi = messages[0].started_at, messages[1].completed_at generation_ms = sum(m.generation_duration_ms or 0.0 for m in messages) - command = next(c for c in resolved if c.tool_id == "c1") + command = next(c for c in result.record.commands if c.tool_id == "c1") + assert command.execution_started_at is not None and command.execution_completed_at is not None tool_ms = busy_ms([(command.execution_started_at, command.execution_completed_at)], lo, hi) assert generation_ms + tool_ms == pytest.approx((hi - lo).total_seconds() * 1000.0) + assert_identity_closes(result.record, started_at=result.started_at, ended_at=result.ended_at) def test_a_duplicate_turn_end_does_not_republish_the_previous_window(self): """A spent `turn_started_at` must not seed the next window. `close_window`'s `min(mark, item_start)` pulls the window open to cover - the item's own start. That is the backwards-clock defence, but a start - stamp left in place after its turn was published is not a backwards - clock — it is a stale value BEFORE the mark, so the guard reopens the - next window at the previous turn's start and publishes that whole span - again. Reproduced before the fix: 3000 ms of generation for a 2000 ms - turn. This reducer promises to survive a malformed stream, and Pi's CLI - retries internally, so a duplicate or replayed `turn_end` is a transport - hiccup rather than a hypothetical. + the item's own start. A start stamp left in place after its turn was + published is a stale value BEFORE the mark, so the guard would reopen the + next window at the previous turn's start and publish that whole span + again (3000 ms of generation for a 2000 ms turn). Pi's CLI retries + internally, so a duplicate `turn_end` is a transport hiccup rather than a + hypothetical. """ - clock = _SteppedClock() - state = _PiTurnState(task_id="t", iteration=1, user_input="go", model="m", clock=clock) - state.on_turn_start() - clock.at_ms = 1000 - state.on_turn_end(_turn_end_payload()) - clock.at_ms = 2000 - state.on_turn_end(_turn_end_payload()) # no intervening `turn_start` - - messages = [m for m in state.messages if m.role == "assistant"] + result, _ = _replay([_start(), Tick(1000), _end(), Tick(2000), _end()]) # no intervening `turn_start` + + messages = _assistants(result) assert len(messages) == 2 assert messages[1].started_at == messages[0].completed_at assert sum(m.generation_duration_ms or 0.0 for m in messages) == pytest.approx(2000.0) def test_a_duplicate_turn_end_does_not_republish_the_previous_content(self): - """The CONTENT half of the same reset, and the same argument. - - `turn_text_parts` / `turn_tool_ids` were cleared in `on_turn_start` - only, so the replayed line re-emitted the first turn's text as its own - assistant message and re-listed the same `tool_use_ids` — one tool call - appearing to belong to two generations, and the text counted twice by - anything that reads the transcript. The sibling above pinned the timing - half while this one silently stayed broken, which is why it is asserted - separately rather than folded in. + """The CONTENT half of the same reset. + + Without it the replayed line re-emits the first turn's text as its own + assistant message and re-lists the same `tool_use_ids` — one tool call + appearing to belong to two generations. """ - clock = _SteppedClock() - state = _PiTurnState(task_id="t", iteration=1, user_input="go", model="m", clock=clock) - state.on_turn_start() - state.on_message_update( - {"assistantMessageEvent": {"type": "text_delta", "delta": "First."}}, - ) - state.on_tool_execution_start({"toolCallId": "c1", "toolName": "bash", "args": {}}) - clock.at_ms = 1000 - state.on_turn_end(_turn_end_payload()) - clock.at_ms = 2000 - state.on_turn_end(_turn_end_payload()) # no intervening `turn_start` + text = {"type": "message_update", "assistantMessageEvent": {"type": "text_delta", "delta": "First."}} + result, _ = _replay([_start(), text, _open("c1"), Tick(1000), _end(), Tick(2000), _end()]) - messages = [m for m in state.messages if m.role == "assistant"] + messages = _assistants(result) assert len(messages) == 2 assert [b.text for b in messages[0].content_blocks if b.block_type == "text"] == ["First."] assert messages[0].tool_use_ids == ["c1"] @@ -1464,76 +1498,31 @@ def test_a_duplicate_turn_end_does_not_republish_the_previous_content(self): def test_an_unresolved_orphan_is_not_given_a_completion_or_a_duration(self): """Force-closing is not observing a completion. - The orphan sweep runs at finalization; stamping its instant as - `execution_completed_at` manufactures a bound, and the `duration_ms` - derived from it is the distance to whenever the sweep happened to run. - The pair then reads as a measured span that - `timing.subtract_tool_time` takes back out of a generation - window the tool never occupied. `execution_started_at` IS kept: the CLI - really did emit that start, and one bound alone forms no span. Same - rule as claude-code's `_finalize_commands` — unknown status and unknown - duration are one fact (CE058). + Stamping the sweep's instant as `execution_completed_at` manufactures a + bound that `timing.subtract_tool_time` then takes back out of a + generation window the tool never occupied. `execution_started_at` IS + kept: the CLI really did emit that start, and one bound alone forms no + span (CE058). """ - clock = _SteppedClock() - state = _PiTurnState(task_id="t", iteration=1, user_input="go", model="m", clock=clock) - state.on_turn_start() - clock.at_ms = 500 - state.on_tool_execution_start({"toolCallId": "c1", "toolName": "bash", "args": {}}) - clock.at_ms = 4000 - closed: list[CommandTelemetry] = [] - state.bind(lambda e: closed.append(e.tool) if isinstance(e, ToolEndEvent) else None) - state.close_open_tools() + result, _ = _replay([_start(), Tick(500), _open("c1"), Tick(4000)]) + closed = [e.tool for e in result.events if isinstance(e, ToolEndEvent)] assert len(closed) == 1 assert closed[0].result_status == "unknown" - assert closed[0].execution_started_at == _SPAN_BASE + timedelta(milliseconds=500) + assert closed[0].error_message is None + assert closed[0].execution_started_at == _ms(500) assert closed[0].execution_completed_at is None assert closed[0].duration_ms is None def test_a_resolved_tool_still_gets_both_bounds_and_a_duration(self): - """The guard narrows the UNRESOLVED case only. - - Without this, deleting the whole stamping block would leave the sibling - above green while every real tool call lost its timing. - """ - clock = _SteppedClock() - state = _PiTurnState(task_id="t", iteration=1, user_input="go", model="m", clock=clock) - state.on_turn_start() - clock.at_ms = 500 - state.on_tool_execution_start({"toolCallId": "c1", "toolName": "bash", "args": {}}) - clock.at_ms = 1200 - closed: list[CommandTelemetry] = [] - state.bind(lambda e: closed.append(e.tool) if isinstance(e, ToolEndEvent) else None) - state.on_tool_execution_end({"toolCallId": "c1", "result": "ok"}) + """The guard narrows the UNRESOLVED case only.""" + result, _ = _replay([_start(), Tick(500), _open("c1"), Tick(1200), _close("c1")]) + closed = [e.tool for e in result.events if isinstance(e, ToolEndEvent)] assert len(closed) == 1 - assert closed[0].execution_completed_at == _SPAN_BASE + timedelta(milliseconds=1200) + assert closed[0].execution_completed_at == _ms(1200) assert closed[0].duration_ms == pytest.approx(700.0) - def test_a_turn_that_never_finishes_does_not_advance_the_mark(self): - """The half of this that is still the reducer's job. - - There is no span list to preserve any more — the collector reduces the - ToolEnd stream itself. What the reducer still owns is the MARK: a turn - that published nothing must not advance it, or its time is handed to - whichever turn finishes next. - """ - clock = _SteppedClock() - state = _PiTurnState(task_id="t", iteration=1, user_input="go", model="m", clock=clock) - state.on_turn_start() - clock.at_ms = 1000 - state.on_turn_end(_turn_end_payload()) - mark_after_flush = state.gen_mark - - clock.at_ms = 1600 - state.on_turn_start() - clock.at_ms = 1700 - state.on_tool_execution_start({"toolCallId": "c2", "toolName": "bash", "args": {}}) - clock.at_ms = 1900 - state.close_open_tools() # crash/timeout orphan sweep — no message appended - - assert state.gen_mark == mark_after_flush - class TestClockIsFreshPerTurn: """A retried turn must not inherit the crashed turn's clock. @@ -1541,20 +1530,19 @@ class TestClockIsFreshPerTurn: `TurnClock` anchors once and derives every later stamp from that anchor, so one surviving a retry would stamp the new turn against the old turn's wall origin — and over a long run accumulate drift against real wall time. The - lifetime is structural (the clock is built with the turn state, and the - state is built per `communicate()`), which is exactly the kind of property + lifetime is structural (the clock is built with the turn's emitter, and the + emitter is built per `communicate()`), which is exactly the kind of property that stays true only while someone is checking. """ async def test_a_turn_after_a_crash_is_anchored_to_a_fresh_clock(self, patch_exec, tmp_path): agent = _agent() patch_exec(_FakeProcess([], returncode=1, stderr=b"boom: bad model")) - with pytest.raises(AgentCrashError): - await _run(agent, tmp_path) - crashed_clock = agent # the state is gone; only the agent survives a crash + outcome = await _run(agent, tmp_path) + assert outcome.status is AgentEndStatus.CRASHED patch_exec(_FakeProcess(HAPPY_STREAM)) - record = await crashed_clock.communicate("try again") + record = (await agent.communicate("try again", iteration=2)).record # The recovered turn measured a real window of its own, rather than one # anchored before the crash — which a stale clock would have produced @@ -1568,8 +1556,8 @@ async def test_a_turn_after_a_crash_is_anchored_to_a_fresh_clock(self, patch_exe async def test_the_agent_retains_no_clock_between_turns(self, patch_exec, tmp_path): """Nothing to reset, because nothing survives — the structural half. - The clock is reachable only through the turn state, and the turn state - is a local of `communicate()`. If either were ever hoisted onto the + The clock is reachable only through the turn's emitter and decoder, which + are locals of `communicate()`. If either were ever hoisted onto the agent (a plausible refactor — several other fields are), the next turn would silently inherit the previous turn's anchor and no assertion about a single turn's numbers would notice. @@ -1578,15 +1566,16 @@ async def test_the_agent_retains_no_clock_between_turns(self, patch_exec, tmp_pa patch_exec(_FakeProcess(HAPPY_STREAM)) await _run(agent, tmp_path) - leaked = [name for name, value in vars(agent).items() if isinstance(value, _PiTurnState | TurnClock)] + leaked = [ + name for name, value in vars(agent).items() if isinstance(value, _PiDecoder | TurnEmitter | TurnClock) + ] assert not leaked, f"a turn's clock outlived its turn via {leaked}" class TestTheTurnBracketComesFromTheTurnClock: - """CE064's behavioural half: the SOURCE of the two bracket stamps. + """The SOURCE of the two bracket stamps: the turn clock. - The rule can only see that `timestamp=` is present. Reverting it to - `StreamEvent.timestamp`'s `default_factory=datetime.now` would leave the + A bracket stamped from `StreamEvent.timestamp`'s `default_factory=datetime.now` would leave the stamp within microseconds of the clock-derived one, which is precisely why the stand-in is anchored a year out — the revert then fails by a year. """ @@ -1594,7 +1583,7 @@ class TestTheTurnBracketComesFromTheTurnClock: async def test_both_brackets_are_stamped_from_the_injected_clock( self, patch_exec, tmp_path, monkeypatch: pytest.MonkeyPatch ): - from coder_eval.agents import pi_agent as agent_module + import coder_eval.agent as agent_module monkeypatch.setattr(agent_module, "TurnClock", AnchoredClock) patch_exec(_FakeProcess(HAPPY_STREAM)) @@ -1611,10 +1600,43 @@ async def test_the_head_and_tail_are_measured_within_one_basis( A mixed pair is off by the anchor offset, not by a millisecond, so the bound here is what the assertion rests on rather than the sign. """ - from coder_eval.agents import pi_agent as agent_module + import coder_eval.agent as agent_module monkeypatch.setattr(agent_module, "TurnClock", AnchoredClock) patch_exec(_FakeProcess(HAPPY_STREAM)) - record = await _run(_agent(), tmp_path) + outcome = await _run(_agent(), tmp_path) + record = outcome.record assert_overhead_is_measured(record) + + +class TestModelTurnCap: + def test_the_model_turn_cap_stops_at_the_next_turn_start_with_the_last_turn_resolved(self) -> None: + lines = Path("tests/fixtures/pi_happy_stream.jsonl").read_text(encoding="utf-8").splitlines() + result, _ = _replay([json.loads(line) for line in lines if line.strip()]) + task = TaskDefinition( + task_id="t", + description="d", + initial_prompt="go", + agent=parse_agent_config(type="pi"), + sandbox=SandboxConfig(driver="tempdir"), + success_criteria=[FileExistsCriterion(description="c", path="out.txt")], + run_limits=RunLimits(max_turns=1), + ) + monitor = TurnMonitor.for_task(task, arm=False) + ends: list[ToolEndEvent] = [] + latch: tuple[StreamEvent, list[ToolEndEvent]] | None = None + for event in result.events: + monitor.on_event(event) + if latch is None and monitor.stop_reason is not None: + latch = (event, list(ends)) + if isinstance(event, ToolEndEvent) and event.parent_thread_id is None: + ends.append(event) + + assert monitor.model_turns == 3 + assert monitor.stop_reason is StopReason.MODEL_TURN_CAP + assert latch is not None + latched_on, ends_at_latch = latch + assert isinstance(latched_on, TurnStartEvent) + unresolved = sum(end.status is ToolEndStatus.UNRESOLVED for end in ends_at_latch) + assert (len(ends_at_latch) - unresolved, unresolved) == (1, 0) diff --git a/tests/test_plan_command.py b/tests/test_plan_command.py index 3ee2517f..7be62b11 100644 --- a/tests/test_plan_command.py +++ b/tests/test_plan_command.py @@ -363,3 +363,29 @@ def test_plugin_path_with_no_skill_flips_the_exit_code(self, tmp_path: Path) -> assert exit_code == 1 assert "config error" in printed assert "offers no skill" in printed + + +class TestPlanResolvesLikeRun: + def test_a_variant_system_prompt_file_on_a_kind_without_system_prompt_fails_plan(self, tmp_path: Path) -> None: + (tmp_path / "prompt.txt").write_text("be terse") + task_file = tmp_path / "task.yaml" + task_file.write_text( + "task_id: t\ndescription: d\nagent:\n type: none\n" + + "success_criteria:\n - type: file_exists\n path: x\n description: x\n" + ) + experiment_file = tmp_path / "experiment.yaml" + experiment_file.write_text( + "experiment_id: e\nvariants:\n - variant_id: v\n agent:\n system_prompt_file: prompt.txt\n" + ) + + with ( + patch("coder_eval.cli.plan_command.check_tools"), + patch("coder_eval.cli.plan_command.check_api_keys"), + patch("coder_eval.cli.plan_command.console") as mock_console, + pytest.raises(typer.Exit) as exc_info, + ): + run_plan(task_files=[task_file], experiment=experiment_file) + + assert exc_info.value.exit_code == 1 + printed = " ".join(str(call) for call in mock_console.print.call_args_list) + assert "system_prompt" in printed diff --git a/tests/test_plugin_staging.py b/tests/test_plugin_staging.py index c240d890..37b0b23c 100644 --- a/tests/test_plugin_staging.py +++ b/tests/test_plugin_staging.py @@ -11,8 +11,10 @@ from coder_eval.orchestration.plugin_staging import ( PluginStagingError, link_or_copy, + scan_plugin_roots, scan_plugin_skills, stage_plugins, + staged_plugin_dirs, validate_plugins, ) @@ -34,6 +36,19 @@ def _local(path: Path | str) -> dict[str, str]: return {"type": "local", "path": str(path)} +def _named(root: Path, name: str) -> None: + (root / ".claude-plugin").mkdir(parents=True, exist_ok=True) + (root / ".claude-plugin" / "plugin.json").write_text(json.dumps({"name": name}), encoding="utf-8") + + +def _refuse_symlinks(self: Path, *args: object, **kwargs: object) -> None: + raise OSError("symlinks unavailable") + + +def _stage(tmp_path: Path, *roots: Path) -> Path: + return stage_plugins([_local(root) for root in roots], tmp_path / "run" / "plugin_root").root + + class TestLayouts: def test_a_plugin_root_with_the_default_skills_dir(self, tmp_path: Path) -> None: _skill(tmp_path / "plugin" / "skills", "alpha") @@ -115,6 +130,71 @@ def test_zero_skills_is_refused(self, tmp_path: Path) -> None: with pytest.raises(PluginStagingError, match="offers no skill"): scan_plugin_skills([_local(tmp_path / "empty")]) + def test_two_plugins_with_one_name_are_ambiguous(self, tmp_path: Path) -> None: + _skill(tmp_path / "a" / ".claude" / "skills", "alpha") + _skill(tmp_path / "b" / ".claude" / "skills", "beta") + one, two = (tmp_path / "a" / ".claude").resolve(), (tmp_path / "b" / ".claude").resolve() + with pytest.raises(PluginStagingError, match=r"ambiguous plugin name '\.claude'") as exc: + scan_plugin_roots([_local(one), _local(two)]) + assert str(one) in str(exc.value) + assert str(two) in str(exc.value) + assert ".claude-plugin/plugin.json" in str(exc.value) + + def test_the_same_plugin_root_listed_twice_is_not_ambiguous(self, tmp_path: Path) -> None: + _skill(tmp_path / "plugin" / "skills", "alpha") + assert list(scan_plugin_roots([_local(tmp_path / "plugin"), _local(tmp_path / "plugin")])) == ["plugin"] + + def test_two_manifests_with_one_name_are_ambiguous(self, tmp_path: Path) -> None: + for directory, skill in (("one", "alpha"), ("two", "beta")): + _skill(tmp_path / directory / "skills", skill) + _named(tmp_path / directory, "shared") + with pytest.raises(PluginStagingError, match="ambiguous plugin name 'shared'"): + scan_plugin_roots([_local(tmp_path / "one"), _local(tmp_path / "two")]) + + def test_a_declared_skills_path_outside_the_root_is_refused(self, tmp_path: Path) -> None: + _skill(tmp_path / "shared", "alpha") + (tmp_path / "plugin").mkdir() + _manifest(tmp_path / "plugin", ["../shared"]) + with pytest.raises(PluginStagingError, match=r"manifest skills path '\.\./shared' leaves the plugin root"): + scan_plugin_skills([_local(tmp_path / "plugin")]) + + @pytest.mark.parametrize("name", ["a/b", "..", ".", "a\\b"]) + def test_a_manifest_name_that_is_not_one_path_segment_is_refused(self, tmp_path: Path, name: str) -> None: + _skill(tmp_path / "plugin" / "skills", "alpha") + _named(tmp_path / "plugin", name) + with pytest.raises(PluginStagingError, match="is not one path segment"): + scan_plugin_roots([_local(tmp_path / "plugin")]) + + def test_plugin_names_that_differ_only_in_case_are_ambiguous(self, tmp_path: Path) -> None: + """On a case-folding filesystem the second link would land inside the first plugin's source.""" + _skill(tmp_path / "Foo" / "skills", "alpha") + _skill(tmp_path / "other" / "foo" / "skills", "beta") + with pytest.raises(PluginStagingError, match="ambiguous plugin name 'foo'"): + scan_plugin_roots([_local(tmp_path / "Foo"), _local(tmp_path / "other" / "foo")]) + + def test_skill_names_that_differ_only_in_case_are_ambiguous(self, tmp_path: Path) -> None: + _skill(tmp_path / "one" / "skills", "Alpha") + _skill(tmp_path / "two" / "skills", "alpha") + with pytest.raises(PluginStagingError, match="ambiguous skill name 'alpha'"): + scan_plugin_skills([_local(tmp_path / "one"), _local(tmp_path / "two")]) + + @pytest.mark.parametrize("name", ["../../escaped", "/abs/path", "a/b", ".."]) + def test_a_skill_name_that_is_not_one_path_segment_is_refused(self, tmp_path: Path, name: str) -> None: + _skill(tmp_path / "plugin" / "skills", "alpha", frontmatter_name=f'"{name}"') + with pytest.raises(PluginStagingError, match=r"skill name .* is not one path segment"): + scan_plugin_skills([_local(tmp_path / "plugin")]) + + def test_a_manifest_name_holding_a_nul_is_refused(self, tmp_path: Path) -> None: + _skill(tmp_path / "plugin" / "skills", "alpha") + _named(tmp_path / "plugin", "a\x00b") + with pytest.raises(PluginStagingError, match="is not one path segment"): + scan_plugin_roots([_local(tmp_path / "plugin")]) + + def test_an_empty_manifest_name_falls_back_to_the_directory_name(self, tmp_path: Path) -> None: + _skill(tmp_path / "plugin" / "skills", "alpha") + _named(tmp_path / "plugin", "") + assert list(scan_plugin_roots([_local(tmp_path / "plugin")])) == ["plugin"] + def test_a_missing_directory_is_refused_with_the_hint(self, tmp_path: Path) -> None: with pytest.raises(PluginStagingError, match="path does not exist"): scan_plugin_skills([_local(tmp_path / "nowhere")]) @@ -167,6 +247,12 @@ def _skill_task(plugins: list[dict[str, str]], skill_name: str) -> TaskDefinitio ], ) + def test_an_ambiguous_plugin_name_fails_resolution(self, tmp_path: Path) -> None: + _skill(tmp_path / "a" / "plugin" / "skills", "alpha") + _skill(tmp_path / "b" / "plugin" / "skills", "beta") + with pytest.raises(PluginStagingError, match="ambiguous plugin name 'plugin'"): + validate_plugins(self._task([_local(tmp_path / "a" / "plugin"), _local(tmp_path / "b" / "plugin")])) + def test_a_skill_triggered_target_the_plugins_offer_passes(self, tmp_path: Path) -> None: _skill(tmp_path / "plugin" / "skills", "alpha") validate_plugins(self._skill_task([_local(tmp_path / "plugin")], "alpha")) @@ -194,12 +280,124 @@ def test_without_plugins_the_target_is_not_checked(self) -> None: class TestStaging: - def test_staged_manifest_names_the_plugin_and_declares_no_skills_key(self, tmp_path: Path) -> None: + def test_the_staged_root_has_no_merged_manifest(self, tmp_path: Path) -> None: + _skill(tmp_path / "plugin" / "skills", "alpha") + assert not (_stage(tmp_path, tmp_path / "plugin") / ".claude-plugin").exists() + + def test_a_plugin_root_is_linked_whole(self, tmp_path: Path) -> None: + root = tmp_path / "plugin" + _skill(root / "skills", "alpha") + _named(root, "p") + files = ["agents/a.md", "commands/c.md", "hooks/hooks.json", ".mcp.json", "scripts/x.sh"] + for relative in files: + (root / relative).parent.mkdir(parents=True, exist_ok=True) + (root / relative).write_text("x", encoding="utf-8") + link = _stage(tmp_path, root) / "plugins" / "p" + assert link.is_symlink() + assert link.resolve() == root.resolve() + for relative in [*files, "skills/alpha/SKILL.md", ".claude-plugin/plugin.json"]: + assert (link / relative).is_file(), relative + + def test_the_plugin_name_is_the_manifest_name_else_the_directory_name(self, tmp_path: Path) -> None: + _skill(tmp_path / "renamed-dir" / "skills", "baz") + _named(tmp_path / "renamed-dir", "realname") + _skill(tmp_path / "nomanifest" / "skills", "bar") + staged = _stage(tmp_path, tmp_path / "renamed-dir", tmp_path / "nomanifest") + assert sorted(p.name for p in (staged / "plugins").iterdir()) == ["nomanifest", "realname"] + + def test_a_single_skill_root_is_linked_whole(self, tmp_path: Path) -> None: + root = tmp_path / "solo" + root.mkdir() + (root / "SKILL.md").write_text("---\nname: solo\ndescription: d\n---\n", encoding="utf-8") + link = _stage(tmp_path, root) / "plugins" / "solo" + assert link.is_symlink() + assert link.resolve() == root.resolve() + + def test_a_manifest_plugin_with_skills_only_under_the_root_is_wrapped_under_its_manifest_name( + self, tmp_path: Path + ) -> None: + root = tmp_path / "mroot" + _skill(root, "zed") + _named(root, "manifroot") + wrapper = _stage(tmp_path, root) / "plugins" / "manifroot" + assert not wrapper.is_symlink() + assert (wrapper / ".claude-plugin" / "plugin.json").read_text(encoding="utf-8") == '{"name": "manifroot"}' + assert (wrapper / "skills").resolve() == root.resolve() + + def test_a_bare_skills_directory_is_wrapped_with_a_name_only_manifest(self, tmp_path: Path) -> None: """A manifest `skills` key made the Claude CLI load no skill at all (spike, 2026-09-17).""" + bare = tmp_path / "bare" + _skill(bare, "foo") + wrapper = _stage(tmp_path, bare) / "plugins" / "bare" + assert (wrapper / ".claude-plugin" / "plugin.json").read_text(encoding="utf-8") == '{"name": "bare"}' + assert (wrapper / "skills").is_symlink() + assert (wrapper / "skills").resolve() == bare.resolve() + + def test_a_root_with_no_skills_is_linked_whole_not_wrapped(self, tmp_path: Path) -> None: + agents_only = tmp_path / "agentsonly" + (agents_only / "agents").mkdir(parents=True) + (agents_only / "agents" / "a.md").write_text("x", encoding="utf-8") _skill(tmp_path / "plugin" / "skills", "alpha") - staged = stage_plugins([_local(tmp_path / "plugin")], tmp_path / "run" / "plugin_root") - manifest = staged.root / ".claude-plugin" / "plugin.json" - assert manifest.read_text(encoding="utf-8") == '{"name": "coder-eval-plugins"}' + link = _stage(tmp_path, agents_only, tmp_path / "plugin") / "plugins" / "agentsonly" + assert link.is_symlink() + assert link.resolve() == agents_only.resolve() + + def test_an_unreadable_manifest_names_the_plugin_by_its_directory(self, tmp_path: Path) -> None: + root = tmp_path / "broken" + _skill(root / "skills", "alpha") + (root / ".claude-plugin").mkdir() + (root / ".claude-plugin" / "plugin.json").write_text("{", encoding="utf-8") + assert [p.name for p in (_stage(tmp_path, root) / "plugins").iterdir()] == ["broken"] + + def test_the_copy_fallback_does_not_copy_the_run_dir_into_itself(self, tmp_path: Path, monkeypatch) -> None: + root = tmp_path / "project" + _skill(root / "skills", "alpha") + monkeypatch.setattr(Path, "symlink_to", _refuse_symlinks) + staged = stage_plugins([_local(root)], root / "run" / "plugin_root") + copied = staged.root / "plugins" / "project" + assert (copied / "skills" / "alpha" / "SKILL.md").is_file() + assert not (copied / "run").exists() + + def test_the_copy_fallback_follows_links_and_skips_a_link_loop(self, tmp_path: Path, monkeypatch) -> None: + """The fallback runs where no symlink can be made, so the copy must not make one either.""" + import os + + root = tmp_path / "plugin" + _skill(root / "skills", "alpha") + _skill(tmp_path / "outside", "shared") + (root / "loop").symlink_to(root, target_is_directory=True) + (root / "linked").symlink_to(tmp_path / "outside", target_is_directory=True) + monkeypatch.setattr(Path, "symlink_to", _refuse_symlinks) + monkeypatch.setattr(os, "symlink", _refuse_symlinks) + copied = _stage(tmp_path, root) / "plugins" / "plugin" + assert not (copied / "loop").exists() + assert not (copied / "linked").is_symlink() + assert (copied / "linked" / "shared" / "SKILL.md").is_file() + assert (copied / "skills" / "alpha" / "SKILL.md").is_file() + + def test_the_copy_fallback_wraps_a_bare_skills_directory(self, tmp_path: Path, monkeypatch) -> None: + _skill(tmp_path / "bare", "foo") + monkeypatch.setattr(Path, "symlink_to", _refuse_symlinks) + wrapper = _stage(tmp_path, tmp_path / "bare") / "plugins" / "bare" + assert (wrapper / ".claude-plugin" / "plugin.json").is_file() + assert (wrapper / "skills" / "foo" / "SKILL.md").is_file() + + def test_link_or_copy_does_not_copy_into_an_existing_target(self, tmp_path: Path) -> None: + source = _skill(tmp_path / "skills", "alpha") + target = tmp_path / "target" + target.mkdir() + with pytest.raises(FileExistsError): + link_or_copy(source, target) + assert list(target.iterdir()) == [] + + def test_staged_plugin_dirs_lists_every_plugin_in_name_order(self, tmp_path: Path) -> None: + _skill(tmp_path / "zeta" / "skills", "alpha") + _skill(tmp_path / "bare", "beta") + staged = _stage(tmp_path, tmp_path / "zeta", tmp_path / "bare") + assert staged_plugin_dirs(staged) == [staged / "plugins" / "bare", staged / "plugins" / "zeta"] + + def test_staged_plugin_dirs_is_empty_without_a_plugins_dir(self, tmp_path: Path) -> None: + assert staged_plugin_dirs(tmp_path) == [] def test_each_skill_is_a_symlink_to_its_source(self, tmp_path: Path) -> None: source = _skill(tmp_path / "plugin" / "skills", "alpha") diff --git a/tests/test_plugins.py b/tests/test_plugins.py index 54a3f907..3a87db6d 100644 --- a/tests/test_plugins.py +++ b/tests/test_plugins.py @@ -5,7 +5,7 @@ import pytest import coder_eval.plugins as plugins -from coder_eval.agents.registry import AgentRegistry +from coder_eval.agents.registry import SPI_VERSION, AgentRegistry from coder_eval.models import AgentKind, ClaudeCodeAgentConfig from tests.fixtures.harness_stubs import config_for_kind, stub_contract @@ -69,39 +69,40 @@ def test_load_plugins_idempotent(monkeypatch): assert len(calls) == 2 -def test_load_plugins_skips_failing_plugin(monkeypatch): - succeeded = [] +@pytest.mark.parametrize("name", ["coder_eval", "third-party"]) +def test_a_failing_plugin_stops_the_load_with_a_named_error(monkeypatch, name): + """A broken plugin, built-in or third-party, is a named error, never skipped.""" + later = [] def boom(reg): raise RuntimeError("plugin exploded") _patch_entry_points( monkeypatch, - [_FakeEntryPoint("bad", boom), _FakeEntryPoint("good", lambda reg: succeeded.append(reg))], + [_FakeEntryPoint(name, boom), _FakeEntryPoint("later", lambda reg: later.append(reg))], ) - # Must not propagate: a broken plugin never aborts startup. - load_plugins() + with pytest.raises(plugins.PluginLoadError, match=f"{name!r}.*plugin exploded") as exc: + load_plugins() - assert succeeded == [AgentRegistry] + assert isinstance(exc.value.__cause__, RuntimeError) + assert later == [] + # Cleared, so a caller that catches and retries re-runs the scan. + assert plugins._loaded is False -def test_load_plugins_reraises_builtin_failure(monkeypatch): - """A failure registering the built-in 'coder_eval' entry point is FATAL — not - swallowed — so a broken built-in import fails loudly instead of as a later - misleading 'No agent registered for claude-code'.""" +def test_a_plugin_built_for_another_spi_version_stops_the_load(monkeypatch): + class _Agent: + contract = stub_contract() - def boom(reg): - raise RuntimeError("builtin broke") + def register(reg): + reg.register("old-spi-kind", config_for_kind("old-spi-kind"), spi_version=SPI_VERSION + 1)(_Agent) - _patch_entry_points(monkeypatch, [_FakeEntryPoint("coder_eval", boom)]) + _patch_entry_points(monkeypatch, [_FakeEntryPoint("old", register)]) - with pytest.raises(RuntimeError, match="builtin broke"): + with pytest.raises(plugins.PluginLoadError, match=f"SPI {SPI_VERSION + 1}.*provides SPI {SPI_VERSION}"): load_plugins() - - # Flag is cleared so a caller that catches and retries re-runs the scan - # instead of getting a no-op against an empty registry. - assert plugins._loaded is False + assert AgentRegistry.get("old-spi-kind") is None def test_register_builtins_raises_if_a_builtin_missing(): @@ -146,7 +147,7 @@ def __init__(self, config, route=None, **kwargs): self.config = config try: - AgentRegistry.register("totally-custom-kind", cfg)(_Agent) + AgentRegistry.register("totally-custom-kind", cfg, spi_version=SPI_VERSION)(_Agent) reg = AgentRegistry.get("totally-custom-kind") assert reg is not None assert reg.agent_class is _Agent @@ -176,9 +177,9 @@ def __init__(self, config, route=None, **kwargs): self.config = config try: - AgentRegistry.register("collide-kind", cfg_a)(_AgentA) + AgentRegistry.register("collide-kind", cfg_a, spi_version=SPI_VERSION)(_AgentA) with pytest.raises(ValueError, match="already registered"): - AgentRegistry.register("collide-kind", cfg_b)(_AgentB) + AgentRegistry.register("collide-kind", cfg_b, spi_version=SPI_VERSION)(_AgentB) # The incumbent is untouched — the conflict did not overwrite it. reg = AgentRegistry.get("collide-kind") assert reg is not None and reg.agent_class is _AgentA @@ -199,8 +200,8 @@ def __init__(self, config, route=None, **kwargs): self.config = config try: - AgentRegistry.register("idem-kind", cfg)(_Agent) - AgentRegistry.register("idem-kind", cfg)(_Agent) # no raise + AgentRegistry.register("idem-kind", cfg, spi_version=SPI_VERSION)(_Agent) + AgentRegistry.register("idem-kind", cfg, spi_version=SPI_VERSION)(_Agent) # no raise reg = AgentRegistry.get("idem-kind") assert reg is not None and reg.agent_class is _Agent finally: @@ -222,9 +223,10 @@ class _PluginNoneConfig(NoneAgentConfig): agent = NoOpAgent(_PluginNoneConfig()) await agent.start(str(tmp_path)) # Would raise AttributeError if communicate used `.type.value` instead of str(...). - record = await agent.communicate("hello") + outcome = await agent.communicate("hello", iteration=1) await agent.stop() + record = outcome.record assert record.crashed is False assert record.iteration == 1 diff --git a/tests/test_price_turn.py b/tests/test_price_turn.py new file mode 100644 index 00000000..fa9b48ad --- /dev/null +++ b/tests/test_price_turn.py @@ -0,0 +1,204 @@ +"""``pricing.price_turn``: the one rule for a turn's cost, shared by every adapter and the monitor.""" + +from __future__ import annotations + +import asyncio +import json +import os +from types import SimpleNamespace +from typing import Any +from unittest.mock import patch + +import pytest + +from coder_eval.models import RunLimits, TokenUsage +from coder_eval.orchestration.turn_monitor import TurnMonitor +from coder_eval.pricing import calculate_cost, price_turn +from coder_eval.streaming.events import AgentEndEvent, AgentStartEvent, StreamEvent + + +_HAIKU = "claude-haiku-4-5" +_USAGE = TokenUsage( + uncached_input_tokens=1000, output_tokens=500, cache_creation_input_tokens=10, cache_read_input_tokens=20 +) + + +def _rate(model: str, usage: TokenUsage = _USAGE) -> float: + cost = calculate_cost( + model, + usage.uncached_input_tokens, + usage.output_tokens, + usage.cache_creation_input_tokens, + usage.cache_read_input_tokens, + ) + assert cost is not None + return cost + + +def _reported(cost: float | None, usage: TokenUsage = _USAGE) -> TokenUsage: + return usage.model_copy(update={"total_cost_usd": cost}) + + +class TestTheRule: + def test_a_finite_non_zero_reported_cost_wins(self) -> None: + assert price_turn(_reported(0.42), (_HAIKU,)) == 0.42 + + @pytest.mark.parametrize("reported", [None, 0.0, 1.5]) + def test_empty_usage_returns_the_reported_cost_unchanged(self, reported: float | None) -> None: + assert price_turn(TokenUsage(total_cost_usd=reported), (_HAIKU,)) == reported + + def test_the_rate_card_prices_an_unreported_cost(self) -> None: + assert price_turn(_USAGE, (_HAIKU,)) == pytest.approx(_rate(_HAIKU)) + + def test_a_reported_zero_on_a_priced_model_uses_the_rate_card(self, caplog: pytest.LogCaptureFixture) -> None: + with caplog.at_level("DEBUG", logger="coder_eval.pricing"): + assert price_turn(_reported(0.0), (_HAIKU,)) == pytest.approx(_rate(_HAIKU)) + assert "using the rate card" in caplog.text + + def test_a_reported_zero_that_no_model_prices_stays_zero(self) -> None: + assert price_turn(_reported(0.0), ("nowhere/not-a-model", None)) == 0.0 + + def test_nothing_reported_and_nothing_priced_is_none(self) -> None: + assert price_turn(_USAGE, ("nowhere/not-a-model", None)) is None + assert price_turn(_USAGE, ()) is None + + @pytest.mark.parametrize("reported", [float("nan"), float("inf"), float("-inf")]) + def test_a_non_finite_reported_cost_counts_as_unreported(self, reported: float) -> None: + assert price_turn(_reported(reported), (_HAIKU,)) == pytest.approx(_rate(_HAIKU)) + assert price_turn(_reported(reported), ("nowhere/not-a-model",)) is None + assert price_turn(TokenUsage(total_cost_usd=reported), (_HAIKU,)) is None + + def test_the_first_priced_model_wins_and_none_is_skipped(self) -> None: + models = (None, "", "nowhere/not-a-model", "claude-sonnet-4-6", _HAIKU) + assert price_turn(_USAGE, models) == pytest.approx(_rate("claude-sonnet-4-6")) + + def test_a_bedrock_prefixed_id_prices_like_the_bare_id(self) -> None: + assert price_turn(_USAGE, ("eu.anthropic.claude-haiku-4-5",)) == pytest.approx(_rate(_HAIKU)) + + +class _Recorder: + def __init__(self) -> None: + self.events: list[StreamEvent] = [] + + def on_event(self, event: StreamEvent) -> None: + self.events.append(event) + + +def _monitor_cost(events: list[StreamEvent], model: str, reported: float | None) -> float | None: + """The monitor's price for the same turn, fed the harness's RAW report instead of the adapter's price.""" + monitor = TurnMonitor("t", [], limits=RunLimits(max_usd=1000.0), model=model) + for event in events: + if isinstance(event, AgentEndEvent): + event = event.model_copy(update={"usage": _reported(reported, event.usage)}) + monitor.on_event(event) + return monitor.cost_usd() + + +def _adapter_cost(events: list[StreamEvent]) -> float | None: + ends = [e for e in events if isinstance(e, AgentEndEvent)] + assert len(ends) == 1 + return ends[0].usage.total_cost_usd + + +async def _run_cli(agent: Any, cli: str, lines: list[str], working_dir: str) -> list[StreamEvent]: + from tests._fixtures.golden_streams.pi_fixtures import _FakeProcess + + proc = _FakeProcess(lines) + + async def fake_exec(*_argv: str, **_kwargs: Any) -> _FakeProcess: + proc.stderr = proc # type: ignore[assignment] + return proc + + recorder = _Recorder() + with ( + patch.object(asyncio, "create_subprocess_exec", fake_exec), + patch("shutil.which", lambda _name: f"/usr/local/bin/{cli}"), + patch.object(os, "killpg", lambda _pgid, _sig: None, create=True), + ): + await agent.start(working_dir) + await agent.communicate("do it", iteration=1, stream_callback=recorder) + return recorder.events + + +def _pi_lines(cost: float) -> list[str]: + from tests._fixtures.golden_streams.pi_fixtures import _turn_end, _turn_start + + return [_turn_start(), _turn_end(inp=1000, out=500, cost=cost)] + + +def _opencode_lines(cost: float) -> list[str]: + tokens = {"total": 1500, "input": 1000, "output": 500, "reasoning": 0, "cache": {"write": 0, "read": 0}} + part = {"sessionID": "ses_1", "id": "prt_2", "messageID": "msg_1", "reason": "stop", "cost": cost, "tokens": tokens} + return [ + json.dumps({"type": "step_start", "sessionID": "ses_1", "part": {"sessionID": "ses_1", "id": "prt_1"}}), + json.dumps({"type": "step_finish", "sessionID": "ses_1", "part": part}), + ] + + +class TestAdapterAndMonitorAgree: + """Each harness's published turn cost equals what the ``max_usd`` monitor sums for the same turn.""" + + @pytest.mark.parametrize("cost", [0.25, 0.0]) + async def test_pi(self, cost: float, tmp_path: Any) -> None: + from coder_eval.agents.pi_agent import PiAgent + from coder_eval.models import PiAgentConfig + + model = "openrouter/moonshotai/kimi-k3" + agent = PiAgent(PiAgentConfig(type="pi", model=model), task_id="t") + events = await _run_cli(agent, "pi", _pi_lines(cost), str(tmp_path)) + expected = cost or _rate(model, TokenUsage(uncached_input_tokens=1000, output_tokens=500)) + assert _adapter_cost(events) == pytest.approx(expected) + assert _monitor_cost(events, model, cost) == pytest.approx(expected) + + @pytest.mark.parametrize("cost", [0.25, 0.0]) + async def test_opencode(self, cost: float, tmp_path: Any) -> None: + from coder_eval.agents.opencode_agent import OpenCodeAgent + from coder_eval.models import OpenCodeAgentConfig + + model = "deepseek/deepseek-v4-pro" + agent = OpenCodeAgent(OpenCodeAgentConfig(type="opencode", model=model), task_id="t") + events = await _run_cli(agent, "opencode", _opencode_lines(cost), str(tmp_path)) + expected = cost or _rate(model, TokenUsage(uncached_input_tokens=1000, output_tokens=500)) + assert _adapter_cost(events) == pytest.approx(expected) + assert _monitor_cost(events, model, cost) == pytest.approx(expected) + + async def test_antigravity(self, tmp_path: Any) -> None: + from coder_eval.agents import antigravity_agent + from tests._fixtures.golden_streams.antigravity_fixtures import _agent_with_steps, _no_sleep, _step, _usage + + steps = [_step("TEXT_RESPONSE", "DONE", content="ok", complete=True, usage=_usage(1000, 200, 300, 50))] + agent = _agent_with_steps(steps) + agent.working_directory = tmp_path + recorder = _Recorder() + with patch.object(antigravity_agent.asyncio, "sleep", _no_sleep): + await agent.communicate("do it", iteration=1, stream_callback=recorder) + expected = _rate( + "gemini-3.5-flash", TokenUsage(uncached_input_tokens=800, output_tokens=350, cache_read_input_tokens=200) + ) + assert _adapter_cost(recorder.events) == pytest.approx(expected) + assert _monitor_cost(recorder.events, "gemini-3.5-flash", None) == pytest.approx(expected) + + def test_codex(self) -> None: + from coder_eval.agents.codex_agent import CodexAgent + from coder_eval.models import CodexAgentConfig + + model = "gpt-5.6-terra" + agent = CodexAgent(CodexAgentConfig(type="codex", model=model)) + sdk = SimpleNamespace(total=SimpleNamespace(input_tokens=1000, output_tokens=500, cached_input_tokens=400)) + usage = agent._token_usage_from_sdk(sdk) + expected = _rate(model, TokenUsage(uncached_input_tokens=600, output_tokens=500, cache_read_input_tokens=400)) + assert usage is not None and usage.total_cost_usd == pytest.approx(expected) + events: list[StreamEvent] = [AgentStartEvent(task_id="t"), AgentEndEvent(task_id="t", usage=usage)] + assert _monitor_cost(events, model, None) == pytest.approx(expected) + + @pytest.mark.parametrize("sdk_cost", [None, 0.0, 0.33]) + def test_claude(self, sdk_cost: float | None) -> None: + from coder_eval.agents.claude_code_agent import ClaudeCodeAgent + + sdk_usage = {"input_tokens": 1000, "output_tokens": 500, "cache_read_input_tokens": 20} + usage = ClaudeCodeAgent._build_token_usage([], sdk_usage, sdk_cost, None, _HAIKU) + assert usage is not None and usage.total_cost_usd is not None + expected = sdk_cost or _rate(_HAIKU, usage) + assert usage.total_cost_usd == pytest.approx(expected) + events: list[StreamEvent] = [AgentStartEvent(task_id="t"), AgentEndEvent(task_id="t", usage=usage)] + assert _monitor_cost(events, _HAIKU, sdk_cost) == pytest.approx(expected) diff --git a/tests/test_reference_permissions.py b/tests/test_reference_permissions.py index e9165768..ceb0e9d8 100644 --- a/tests/test_reference_permissions.py +++ b/tests/test_reference_permissions.py @@ -561,6 +561,8 @@ async def test_agent_cannot_read_reference_mid_turn(self, tmp_path, monkeypatch) from coder_eval.models import EvaluationResult, FinalStatus, TurnRecord from coder_eval.orchestrator import Orchestrator + from coder_eval.streaming.emitter import TurnOutcome + from coder_eval.streaming.events import AgentEndStatus task_dir = tmp_path / "task" reference = task_dir / "reference" @@ -598,11 +600,11 @@ async def test_agent_cannot_read_reference_mid_turn(self, tmp_path, monkeypatch) async def _cheating_communicate(prompt, **kwargs): observed["reference"] = _try_read(staged / "solution.py") observed["task_dir"] = _try_read(task_file) - return TurnRecord(iteration=1, prompt=prompt, user_input=prompt, agent_output="done") + record = TurnRecord(iteration=1, prompt=prompt, user_input=prompt, agent_output="done") + return TurnOutcome(record=record, status=AgentEndStatus.COMPLETED, error=None) agent = MagicMock() agent.communicate = _cheating_communicate - agent.pending_turn = None orchestrator.agent = agent await orchestrator._communicate_with_retry(prompt="go", iteration=1, operation_label="test") @@ -633,6 +635,8 @@ async def test_reference_is_unshielded_by_the_time_criteria_run(self, tmp_path, from coder_eval.models import CriterionResult, EvaluationResult, FinalStatus, TurnRecord from coder_eval.orchestrator import Orchestrator + from coder_eval.streaming.emitter import TurnOutcome + from coder_eval.streaming.events import AgentEndStatus task_dir = tmp_path / "task" reference = task_dir / "reference" @@ -664,11 +668,11 @@ async def test_reference_is_unshielded_by_the_time_criteria_run(self, tmp_path, async def _communicate(prompt, **kwargs): seen["during_turn"] = _mode(staged) - return TurnRecord(iteration=1, prompt=prompt, user_input=prompt, agent_output="done") + record = TurnRecord(iteration=1, prompt=prompt, user_input=prompt, agent_output="done") + return TurnOutcome(record=record, status=AgentEndStatus.COMPLETED, error=None) agent = MagicMock() agent.communicate = _communicate - agent.pending_turn = None orchestrator.agent = agent async def _check_all_async(*args, **kwargs): diff --git a/tests/test_registry.py b/tests/test_registry.py index d54d2143..d95f1b8b 100644 --- a/tests/test_registry.py +++ b/tests/test_registry.py @@ -2,7 +2,7 @@ import pytest -from coder_eval.agents.registry import AgentRegistry, create_agent +from coder_eval.agents.registry import SPI_VERSION, AgentRegistry, create_agent from coder_eval.models import AgentKind, ClaudeCodeAgentConfig, parse_agent_config from tests.fixtures.harness_stubs import config_for_kind, stub_contract @@ -23,7 +23,9 @@ def __init__(self, config, route=None, **kwargs): # guard, then roll it back so the process-global registry doesn't leak. try: registration = AgentRegistry.register( - "fake-identity-kind", config_for_kind("fake-identity-kind", ClaudeCodeAgentConfig) + "fake-identity-kind", + config_for_kind("fake-identity-kind", ClaudeCodeAgentConfig), + spi_version=SPI_VERSION, )(FakeAgent) assert registration is FakeAgent finally: diff --git a/tests/test_reports.py b/tests/test_reports.py index 74d5decf..da17f4ba 100644 --- a/tests/test_reports.py +++ b/tests/test_reports.py @@ -1247,10 +1247,12 @@ def _summary_with_notes( *, tool_calls_exhausted: bool = False, expected_tool_calls_overage: list[int] | None = None, + expected_turns_overage: list[int] | None = None, ) -> RunSummary: task = _make_task_result("t1", "SUCCESS", 1.0, 10.0) task["tool_calls_exhausted"] = tool_calls_exhausted task["expected_tool_calls_overage"] = expected_tool_calls_overage + task["expected_turns_overage"] = expected_turns_overage return RunSummary( run_id="r", start_time=datetime(2026, 5, 21, 12, 0, 0), @@ -1274,6 +1276,13 @@ def test_generate_markdown_renders_expected_tool_calls_marker_when_exceeded(): assert "7/5" in report_md +def test_generate_markdown_renders_expected_turns_marker_when_exceeded(): + summary = _summary_with_notes(expected_turns_overage=[5, 3]) + report_md = ReportGenerator.generate_markdown(summary) + assert "## Run-time Notes" in report_md + assert "expected_turns exceeded: 5/3 (cumulative model turns)" in report_md + + def test_generate_markdown_no_expected_tool_calls_marker_when_under(): # Under-budget: the dict carries no overage field — emit nothing. summary = _summary_with_notes(expected_tool_calls_overage=None) diff --git a/tests/test_reports_html.py b/tests/test_reports_html.py index d6f576ec..71dce9cd 100644 --- a/tests/test_reports_html.py +++ b/tests/test_reports_html.py @@ -898,8 +898,9 @@ def _result_with_expected_tool_calls( commands_per_turn: list[int] | None = None, final_reply: str | None = None, task_config: bool = True, + model_turns: int | None = None, ) -> EvaluationResult: - """Build an EvaluationResult that exercises expected_tool_calls_overage. + """Build an EvaluationResult that exercises expected_tool_calls_overage and expected_turns_overage. Visible turns = sum(commands_per_turn) + (1 if final_reply else 0). """ @@ -930,6 +931,7 @@ def _result_with_expected_tool_calls( ) ) result = _make_result(iterations=turns) + result.model_turns = model_turns if task_config: resolved: dict = {} if resolved_run_limits is not None: @@ -938,6 +940,18 @@ def _result_with_expected_tool_calls( return result +def test_task_html_renders_expected_turns_badge_when_exceeded(): + result = _result_with_expected_tool_calls({"expected_turns": 3}, model_turns=5) + html = HTMLReportGenerator.generate_task_html(result) + assert "expected_turns exceeded (5/3)" in html + + +def test_task_html_no_expected_turns_badge_when_under(): + result = _result_with_expected_tool_calls({"expected_turns": 5}, model_turns=5) + html = HTMLReportGenerator.generate_task_html(result) + assert "expected_turns exceeded" not in html + + def test_task_html_renders_expected_tool_calls_badge_when_exceeded(): # 6 tools + reply = 7 visible turns; budget 5 → 7/5 overage. result = _result_with_expected_tool_calls( diff --git a/tests/test_result_metrics.py b/tests/test_result_metrics.py index 8b07a990..55a816d2 100644 --- a/tests/test_result_metrics.py +++ b/tests/test_result_metrics.py @@ -22,6 +22,7 @@ from coder_eval.result_metrics import ( TurnTimeBuckets, expected_tool_calls_overage, + expected_turns_overage, has_final_reply, turn_time_buckets, visible_turn_count, @@ -135,7 +136,7 @@ def _turn_with_commands(commands: int = 0, reply: str | None = None) -> TurnReco ) -class TestExpectedTurnsOverage: +class TestExpectedToolCallsOverage: def test_strict_greater_than(self): # 5 tools + reply = 6 visible turns. Budget 6 → no overage (equal). result = _make_result( @@ -172,7 +173,7 @@ def test_expected_tool_calls_unset(self): result = _make_result(resolved={"run_limits": {"max_turns": 10}}, turns=[_turn_with_commands(commands=20)]) assert expected_tool_calls_overage(result) is None - def test_the_historical_expected_turns_key_is_not_read(self): + def test_expected_turns_does_not_feed_the_tool_call_overage(self): result = _make_result(resolved={"run_limits": {"expected_turns": 5}}, turns=[_turn_with_commands(commands=20)]) assert expected_tool_calls_overage(result) is None @@ -223,3 +224,27 @@ def test_tools_without_final_reply(self): result = _make_result(turns=[_turn_with_commands(commands=4)]) stats = calculate_command_statistics(result.iterations) assert visible_turn_count(result) == stats.total_commands + (1 if has_final_reply(result) else 0) + + +class TestExpectedTurnsOverage: + @staticmethod + def _result(expected: object, model_turns: int | None) -> EvaluationResult: + run_limits = {} if expected is None else {"expected_turns": expected} + result = _make_result(resolved={"run_limits": run_limits}) + result.model_turns = model_turns + return result + + def test_over(self): + assert expected_turns_overage(self._result(3, 5)) == (5, 3) + + def test_equal_is_not_over(self): + assert expected_turns_overage(self._result(3, 3)) is None + + def test_no_model_turn_count(self): + assert expected_turns_overage(self._result(3, None)) is None + + def test_key_absent(self): + assert expected_turns_overage(self._result(None, 5)) is None + + def test_invalid_expected_type(self): + assert expected_turns_overage(self._result("ten", 5)) is None diff --git a/tests/test_retry_logic_comprehensive.py b/tests/test_retry_logic_comprehensive.py index 0bf0dfb3..dc70dd6b 100644 --- a/tests/test_retry_logic_comprehensive.py +++ b/tests/test_retry_logic_comprehensive.py @@ -9,7 +9,7 @@ import pytest -from coder_eval.errors.categories import RETRY_CONFIG, ErrorCategory +from coder_eval.errors.categories import ErrorCategory from coder_eval.errors.categorization import categorize_error from coder_eval.errors.executor import execute_with_retry from coder_eval.errors.retry import get_retry_delay, should_retry @@ -229,111 +229,6 @@ async def flaky_operation(): assert attempts == 3 # Failed twice, succeeded third time -@pytest.mark.asyncio -async def test_on_attempt_error_fires_for_every_failure(): - """Callback fires on every failed attempt, including retried and terminal ones. - - The orchestrator uses this hook to drain partial telemetry from - AgentCrashError before the retry decision is made. - """ - # Lock the assumption: AGENT_RATE_LIMIT must be retryable for this test to - # exercise the "fires across multiple retries" path. If the policy ever - # flips to non-retryable, the test would silently degenerate into the - # terminal-failure case. - assert RETRY_CONFIG[ErrorCategory.AGENT_RATE_LIMIT].max_retries >= 2 - - attempts = 0 - calls: list[tuple[str, int]] = [] - - async def flaky_operation(): - nonlocal attempts - attempts += 1 - if attempts < 3: - raise Exception("Rate limit exceeded") # Retryable (AGENT_RATE_LIMIT) - return "success" - - async def callback(err: Exception, attempt: int) -> None: - calls.append((str(err), attempt)) - - context = {"task_id": "test-task", "component": "agent"} - - with patch("asyncio.sleep", new_callable=AsyncMock): - result = await execute_with_retry( - flaky_operation, "test_op", context, max_attempts=5, on_attempt_error=callback - ) - - assert result == "success" - # Two failures before success → callback fires twice, with the - # zero-indexed attempt number. - assert calls == [("Rate limit exceeded", 0), ("Rate limit exceeded", 1)] - - -@pytest.mark.asyncio -async def test_on_attempt_error_fires_on_terminal_failure(): - """Callback also fires on the final, non-retried attempt. - - This is the hook's whole point on terminal failures: preserve - telemetry before the exception propagates up and the run is abandoned. - """ - # Lock the assumption this test is built on: AGENT_AUTH_ERROR is - # non-retryable. If that policy ever flips, the "calls == [0]" - # assertion below would still pass for the wrong reason, so fail - # loudly here instead. The default RetryConfig has max_retries=0, - # and AGENT_AUTH_ERROR is not overridden in RETRY_CONFIG — so either - # a missing entry (treated as default) or an explicit max_retries=0 - # passes. - from coder_eval.errors.categories import RetryConfig - - assert RETRY_CONFIG.get(ErrorCategory.AGENT_AUTH_ERROR, RetryConfig()).max_retries == 0 - - calls: list[int] = [] - - async def always_fails(): - raise Exception("Invalid API Key") # Non-retryable - - async def callback(err: Exception, attempt: int) -> None: - calls.append(attempt) - - context = {"task_id": "test-task", "component": "agent"} - - with pytest.raises(Exception, match="Invalid API Key"): - await execute_with_retry(always_fails, "test_op", context, max_attempts=5, on_attempt_error=callback) - - assert calls == [0] # Fired once, before the error propagates. - - -@pytest.mark.asyncio -async def test_on_attempt_error_exceptions_are_swallowed(): - """A raising callback must not mask the original error. - - The comment in executor.py explicitly calls this out as a contract: - telemetry-draining code should never take down the retry loop. - """ - # Pin the categorization + retry policy this test relies on so a - # change to the categorizer's string patterns can't silently turn - # this into a no-retry scenario that passes for the wrong reason. - sentinel = Exception("Connection failed") - assert categorize_error(sentinel, {"component": "agent"}) == ErrorCategory.AGENT_API_ERROR - assert RETRY_CONFIG[ErrorCategory.AGENT_API_ERROR].max_retries >= 1 - - callback_called = False - - async def flaky_operation(): - raise Exception("Connection failed") # Retryable (AGENT_API_ERROR) - - async def bad_callback(err: Exception, attempt: int) -> None: - nonlocal callback_called - callback_called = True - raise RuntimeError("callback blew up") - - context = {"task_id": "test-task", "component": "agent"} - - with patch("asyncio.sleep", new_callable=AsyncMock), pytest.raises(Exception, match="Connection failed"): - await execute_with_retry(flaky_operation, "test_op", context, max_attempts=2, on_attempt_error=bad_callback) - - assert callback_called # Callback was invoked despite raising. - - def test_should_retry_respects_config(): """Test that should_retry correctly uses RetryConfig. @@ -370,3 +265,40 @@ def test_get_retry_delay_exponential_backoff(): assert abs(delay_0 - 5.0) < 0.01 # 5.0 * 2^0 = 5.0 assert abs(delay_1 - 10.0) < 0.01 # 5.0 * 2^1 = 10.0 assert abs(delay_2 - 20.0) < 0.01 # 5.0 * 2^2 = 20.0 + + +@pytest.mark.asyncio +@pytest.mark.parametrize(("tool_calls", "expected_attempts"), [(0, 3), (1, 1)]) +async def test_agent_crash_retries_only_without_tool_calls(tool_calls: int, expected_attempts: int): + """A crashed attempt that made a tool call changed the sandbox, so it is not retried.""" + from coder_eval.errors import AgentCrashError + + attempts = 0 + + async def crashes(): + nonlocal attempts + attempts += 1 + raise AgentCrashError("mid-turn failure", tool_calls) + + with patch("asyncio.sleep", new_callable=AsyncMock), pytest.raises(AgentCrashError): + await execute_with_retry(crashes, "test_op", {"task_id": "t", "component": "agent"}) + + assert attempts == expected_attempts + + +@pytest.mark.asyncio +async def test_rate_limit_crash_after_tool_calls_keeps_its_retry_policy(): + from coder_eval.errors import AgentCrashError + + attempts = 0 + + async def rate_limited(): + nonlocal attempts + attempts += 1 + if attempts == 1: + raise AgentCrashError("429 too many requests", 3) + return "ok" + + with patch("asyncio.sleep", new_callable=AsyncMock): + assert await execute_with_retry(rate_limited, "test_op", {"task_id": "t", "component": "agent"}) == "ok" + assert attempts == 2 diff --git a/tests/test_run_limits_models.py b/tests/test_run_limits_models.py index ffe1b51a..4bc98e15 100644 --- a/tests/test_run_limits_models.py +++ b/tests/test_run_limits_models.py @@ -65,13 +65,18 @@ def test_max_tool_calls_validation(self): RunLimits(max_tool_calls=0) RunLimits(max_tool_calls=1) - def test_max_turns_under_run_limits_is_rejected(self): - with pytest.raises(ValidationError, match=r"max_turns\n\s+Extra inputs are not permitted"): - RunLimits.model_validate({"max_turns": 5}) + def test_max_turns_defaults_to_none(self): + assert RunLimits().max_turns is None - def test_max_turns_under_task_run_limits_is_rejected(self): - with pytest.raises(ValidationError, match="Extra inputs are not permitted"): - _minimal_task(run_limits={"max_turns": 5}) + def test_max_turns_validation(self): + with pytest.raises(ValidationError, match="greater than 0"): + RunLimits(max_turns=0) + assert RunLimits(max_turns=1).max_turns == 1 + + def test_max_turns_under_task_run_limits_loads(self): + task = _minimal_task(run_limits={"max_turns": 5}) + assert task.run_limits is not None + assert task.run_limits.max_turns == 5 def test_task_timeout_validation(self): with pytest.raises(ValidationError, match="greater than or equal to 30"): @@ -125,6 +130,14 @@ def test_expected_tool_calls_lower_bound(self): RunLimits(expected_tool_calls=0) assert RunLimits(expected_tool_calls=1).expected_tool_calls == 1 + def test_expected_turns_default_none(self): + assert RunLimits().expected_turns is None + + def test_expected_turns_lower_bound(self): + with pytest.raises(ValidationError, match="greater than or equal to 1"): + RunLimits(expected_turns=0) + assert RunLimits(expected_turns=1).expected_turns == 1 + def test_expected_tool_calls_yaml_coercion(self): assert RunLimits.model_validate({"expected_tool_calls": "10"}).expected_tool_calls == 10 diff --git a/tests/test_run_limits_orchestrator.py b/tests/test_run_limits_orchestrator.py index 18ad333d..d0fa9f10 100644 --- a/tests/test_run_limits_orchestrator.py +++ b/tests/test_run_limits_orchestrator.py @@ -26,7 +26,8 @@ TurnRecord, ) from coder_eval.orchestrator import Orchestrator -from coder_eval.streaming.events import AgentEndEvent, AgentStartEvent +from coder_eval.streaming.emitter import TurnOutcome +from coder_eval.streaming.events import AgentEndEvent, AgentEndStatus, AgentStartEvent def _make_task(*, run_limits: RunLimits | None = None) -> TaskDefinition: @@ -90,12 +91,12 @@ def _reporting_agent(*turns: TurnRecord) -> AsyncMock: """A fake agent whose each ``communicate`` reports its turn's usage on the stream, as real agents do.""" remaining = list(turns) - async def communicate(user_input, *, stream_callback=None, timeout=None, should_stop=None): + async def communicate(user_input, *, iteration, stream_callback=None, timeout=None, should_stop=None): turn = remaining.pop(0) if len(remaining) > 1 else remaining[0] assert stream_callback is not None stream_callback.on_event(AgentStartEvent(task_id="budget_test", prompt=user_input)) stream_callback.on_event(AgentEndEvent(task_id="budget_test", usage=turn.token_usage or TokenUsage())) - return turn + return TurnOutcome(record=turn, status=AgentEndStatus.COMPLETED, error=None) agent = AsyncMock() agent.communicate = communicate @@ -325,6 +326,7 @@ async def test_dialog_aborts_with_run_limit_stop_reason(self, tmp_path): task = task.model_copy(update={"simulation": sim, "initial_prompt": "first message"}) orch = _make_orchestrator(task, tmp_path) + orch._counts_model_turns = True # The agent's first turn reports tokens above the budget. orch.agent = _reporting_agent(_make_turn(input_tokens=200, output_tokens=10)) @@ -357,6 +359,8 @@ async def test_dialog_aborts_with_run_limit_stop_reason(self, tmp_path): assert orch.result.simulation is not None assert orch.result.simulation.stop_reason == "run_limit_exceeded" assert orch.result.simulation.total_turns == 1 + # The model-turn count is recorded before the budget gate raises. + assert orch.result.model_turns == 0 # Simulator must not have been asked for another message after the budget trip. mock_simulator.next_user_message.assert_not_called() @@ -389,13 +393,13 @@ async def test_an_unpriceable_max_usd_ends_the_dialog_after_its_first_turn(self, class TestCheckExpectedTurnsUnit: - """Direct unit tests of Orchestrator._check_expected_tool_calls.""" + """Direct unit tests of Orchestrator._check_expected_targets.""" def test_noop_when_run_limits_is_none(self, tmp_path, caplog): orch = _make_orchestrator(_make_task(), tmp_path) orch.result.iterations.append(_make_turn(commands=100)) with caplog.at_level(logging.WARNING): - orch._check_expected_tool_calls(iteration=1) + orch._check_expected_targets(iteration=1) assert "expected_tool_calls" not in caplog.text.lower() assert orch._expected_tool_calls_warning_emitted is False @@ -403,7 +407,7 @@ def test_noop_when_expected_tool_calls_unset(self, tmp_path, caplog): orch = _make_orchestrator(_make_task(run_limits=RunLimits(max_tool_calls=10)), tmp_path) orch.result.iterations.append(_make_turn(commands=20)) with caplog.at_level(logging.WARNING): - orch._check_expected_tool_calls(iteration=1) + orch._check_expected_targets(iteration=1) assert "expected_tool_calls" not in caplog.text.lower() assert orch._expected_tool_calls_warning_emitted is False @@ -413,7 +417,7 @@ def test_no_warning_at_exact_equal(self, tmp_path, caplog): orch.result.iterations.append(_make_turn(iteration=1, commands=3)) orch.result.iterations.append(_make_turn(iteration=2, commands=2, reply="done")) with caplog.at_level(logging.WARNING): - orch._check_expected_tool_calls(iteration=2) + orch._check_expected_targets(iteration=2) assert "Visible turns" not in caplog.text assert orch._expected_tool_calls_warning_emitted is False @@ -423,13 +427,13 @@ def test_warning_fires_once_when_exceeded(self, tmp_path, caplog): orch.result.iterations.append(_make_turn(iteration=1, commands=2)) orch.result.iterations.append(_make_turn(iteration=2, commands=2)) with caplog.at_level(logging.WARNING): - orch._check_expected_tool_calls(iteration=2) + orch._check_expected_targets(iteration=2) assert "Visible turns" not in caplog.text # +3 tools = 7 visible turns, over 5 → fires. orch.result.iterations.append(_make_turn(iteration=3, commands=3)) with caplog.at_level(logging.WARNING): - orch._check_expected_tool_calls(iteration=3) + orch._check_expected_targets(iteration=3) assert "Visible tool calls (7) exceeded expected_tool_calls (5)" in caplog.text assert orch._expected_tool_calls_warning_emitted is True @@ -437,7 +441,7 @@ def test_warning_fires_once_when_exceeded(self, tmp_path, caplog): caplog.clear() orch.result.iterations.append(_make_turn(iteration=4, commands=5)) with caplog.at_level(logging.WARNING): - orch._check_expected_tool_calls(iteration=4) + orch._check_expected_targets(iteration=4) assert "Visible turns" not in caplog.text def test_warning_counts_reply_as_one(self, tmp_path, caplog): @@ -448,14 +452,41 @@ def test_warning_counts_reply_as_one(self, tmp_path, caplog): orch.result.iterations.append(_make_turn(iteration=1, commands=2)) orch.result.iterations.append(_make_turn(iteration=2, commands=2, reply="ok")) with caplog.at_level(logging.WARNING): - orch._check_expected_tool_calls(iteration=2) + orch._check_expected_targets(iteration=2) assert "Visible tool calls (5) exceeded expected_tool_calls (3)" in caplog.text + def test_expected_turns_warns_once_when_model_turns_exceed_it(self, tmp_path, caplog): + orch = _make_orchestrator(_make_task(run_limits=RunLimits(expected_turns=2)), tmp_path) + orch.result.model_turns = 2 + with caplog.at_level(logging.WARNING): + orch._check_expected_targets(iteration=1) + assert "expected_turns" not in caplog.text + + orch.result.model_turns = 3 + with caplog.at_level(logging.WARNING): + orch._check_expected_targets(iteration=2) + assert "Model turns (3) exceeded expected_turns (2)" in caplog.text + + caplog.clear() + orch.result.model_turns = 5 + with caplog.at_level(logging.WARNING): + orch._check_expected_targets(iteration=3) + assert "expected_turns" not in caplog.text + + def test_expected_turns_is_silent_without_a_model_turn_count(self, tmp_path, caplog): + orch = _make_orchestrator(_make_task(run_limits=RunLimits(expected_tool_calls=1, expected_turns=1)), tmp_path) + orch.result.iterations.append(_make_turn(commands=3)) + with caplog.at_level(logging.WARNING): + orch._check_expected_targets(iteration=1) + assert "exceeded expected_tool_calls" in caplog.text + assert "expected_turns" not in caplog.text + assert orch._expected_turns_warning_emitted is False + def test_noop_when_result_is_none(self, tmp_path, caplog): orch = _make_orchestrator(_make_task(run_limits=RunLimits(expected_tool_calls=1)), tmp_path) orch.result = None with caplog.at_level(logging.WARNING): - orch._check_expected_tool_calls(iteration=1) + orch._check_expected_targets(iteration=1) assert "Visible turns" not in caplog.text @@ -470,7 +501,9 @@ async def test_warning_does_not_abort_run(self, tmp_path, caplog): orch = _make_orchestrator(task, tmp_path) mock_agent = AsyncMock() - mock_agent.communicate = AsyncMock(return_value=turn) + mock_agent.communicate = AsyncMock( + return_value=TurnOutcome(record=turn, status=AgentEndStatus.COMPLETED, error=None) + ) orch.agent = mock_agent mock_checker = MagicMock() mock_checker.check_all_async = AsyncMock( @@ -515,7 +548,9 @@ async def test_warning_fires_in_simulation_and_does_not_abort(self, tmp_path, ca # Each agent turn = 1 tool call → cumulative still under 3 after one turn. turn = _make_turn(commands=1) mock_agent = AsyncMock() - mock_agent.communicate = AsyncMock(return_value=turn) + mock_agent.communicate = AsyncMock( + return_value=TurnOutcome(record=turn, status=AgentEndStatus.COMPLETED, error=None) + ) orch.agent = mock_agent mock_checker = MagicMock() @@ -571,7 +606,9 @@ async def test_warning_fires_when_single_simulation_turn_exceeds(self, tmp_path, # 4 tools + reply = 5 visible turns, exceeds 2. turn = _make_turn(commands=4, reply="done") mock_agent = AsyncMock() - mock_agent.communicate = AsyncMock(return_value=turn) + mock_agent.communicate = AsyncMock( + return_value=TurnOutcome(record=turn, status=AgentEndStatus.COMPLETED, error=None) + ) orch.agent = mock_agent mock_checker = MagicMock() diff --git a/tests/test_run_limits_resolver.py b/tests/test_run_limits_resolver.py index 46625748..49da3adb 100644 --- a/tests/test_run_limits_resolver.py +++ b/tests/test_run_limits_resolver.py @@ -86,9 +86,35 @@ def resolve(*, exp_default=None, task_value=None, variant_value=None, cli_value= assert resolve(exp_default=50, task_value=20, variant_value=10) == (10, "variant") assert resolve(exp_default=50, task_value=20, variant_value=10, cli_value=3) == (3, "cli") - def test_max_turns_under_variant_run_limits_is_rejected(self): - with pytest.raises(ValueError, match="Extra inputs are not permitted"): - ExperimentVariant.model_validate({"variant_id": "v", "run_limits": {"max_turns": 5}}) + def test_max_turns_merges_through_all_five_layers(self): + from coder_eval.orchestration.config import BatchRunConfig + from coder_eval.orchestration.experiment import _apply_cli_overrides + + def resolve(*, exp_default=None, task_value=None, variant_value=None, cli_value=None): + default_exp = _default_exp(RunLimits(max_turns=100)) + task = _make_task(run_limits={"max_turns": task_value}) if task_value else _make_task() + exp = ExperimentDefinition( + experiment_id="e", + defaults=ExperimentDefaults(run_limits=RunLimits(max_turns=exp_default)) if exp_default else None, + variants=[ + ExperimentVariant( + variant_id="v", + run_limits=RunLimits(max_turns=variant_value) if variant_value else None, + ) + ], + ) + resolved, lineage, _ = resolve_task_for_variant(default_exp, task, exp, exp.variants[0]) + if cli_value: + config = BatchRunConfig(run_dir=Path("runs/test"), overrides={"run_limits.max_turns": cli_value}) + _apply_cli_overrides(resolved, config, lineage=lineage) + assert resolved.run_limits is not None + return resolved.run_limits.max_turns, lineage["run_limits.max_turns"].source + + assert resolve() == (100, "default") + assert resolve(exp_default=50) == (50, "experiment-defaults") + assert resolve(exp_default=50, task_value=20) == (20, "task") + assert resolve(exp_default=50, task_value=20, variant_value=10) == (10, "variant") + assert resolve(exp_default=50, task_value=20, variant_value=10, cli_value=3) == (3, "cli") def test_experiment_defaults_overrides_default(self): default_exp = _default_exp(RunLimits(max_usd=1.0)) diff --git a/tests/test_run_record.py b/tests/test_run_record.py index f522d379..26d4458f 100644 --- a/tests/test_run_record.py +++ b/tests/test_run_record.py @@ -44,6 +44,8 @@ "expected_commands": None, "expected_tool_calls": None, "expected_tool_calls_overage": None, + "expected_turns": None, + "expected_turns_overage": None, "gate_threshold": None, "generation_ms": None, "has_final_reply": False, @@ -62,6 +64,7 @@ ], "judge_cost_usd": None, "tool_calls_exhausted": False, + "model_turns": None, "model_used": "claude-haiku-4-5", "output_tokens": 200, "reference_similarity": None, @@ -250,7 +253,7 @@ def test_none_when_task_config_none(self): d = eval_result_to_task_dict(result) assert d["expected_tool_calls"] is None - def test_row_carries_the_tool_call_keys_and_none_of_the_historical_ones(self): + def test_row_carries_the_tool_call_and_model_turn_keys_and_none_of_the_historical_ones(self): result = _make_result(resolved={"run_limits": {"expected_turns": 12}}, turns=[_turn_with_expected(5)]) d = eval_result_to_task_dict(result) assert { @@ -258,18 +261,28 @@ def test_row_carries_the_tool_call_keys_and_none_of_the_historical_ones(self): "tool_calls_remaining_at_stop", "expected_tool_calls", "expected_tool_calls_overage", + "model_turns", + "expected_turns", + "expected_turns_overage", } <= d.keys() - assert ( - not { - "max_turns_exhausted", - "turns_remaining_at_stop", - "expected_turns", - "expected_turns_overage", - } - & d.keys() - ) + assert not {"max_turns_exhausted", "turns_remaining_at_stop"} & d.keys() assert d["expected_tool_calls"] is None + def test_row_carries_the_model_turn_target_and_its_overage(self): + result = _make_result(resolved={"run_limits": {"expected_turns": 3}}) + result.model_turns = 5 + d = eval_result_to_task_dict(result) + assert d["model_turns"] == 5 + assert d["expected_turns"] == 3 + assert d["expected_turns_overage"] == [5, 3] + + def test_a_record_without_a_model_turn_count_carries_no_expected_turns(self): + result = _make_result(resolved={"run_limits": {"expected_turns": 3}}) + d = eval_result_to_task_dict(result) + assert d["model_turns"] is None + assert d["expected_turns"] is None + assert d["expected_turns_overage"] is None + def test_row_carries_tool_calls_remaining_at_stop_from_early_stop(self): result = _make_result() result.early_stop = EarlyStopInfo( diff --git a/tests/test_runtime_tool_versions.py b/tests/test_runtime_tool_versions.py index 69084c2d..c7fe12a2 100644 --- a/tests/test_runtime_tool_versions.py +++ b/tests/test_runtime_tool_versions.py @@ -303,18 +303,19 @@ def test_override_filters_nonversion_junk_cli(): def test_sandbox_refresh_plugin_tools_dir_real_plumbing(tmp_path: Path): - """uip_search_path + refresh_plugin_tools_dir re-derive from the agent-aligned PATH.""" + """uip_search_path + refresh_plugin_tools_dir re-derive from the mock-dir PATH prefix.""" from coder_eval.models import SandboxConfig from coder_eval.sandbox import Sandbox - dist = tmp_path / "node_modules" / "@uipath" / "cli" / "dist" + workspace = tmp_path.resolve() + dist = workspace / "node_modules" / "@uipath" / "cli" / "dist" _make_fake_uip(dist, "9.9.9") - sandbox = Sandbox(config=SandboxConfig(), task_id="t") + sandbox = Sandbox(config=SandboxConfig(mock_path_dirs=["node_modules/@uipath/cli/dist"]), task_id="t") - sandbox.set_command_base_path(str(dist)) + sandbox.adopt(workspace) assert sandbox.uip_search_path.startswith(f"{dist}{os.pathsep}") - assert sandbox.plugin_tools_dir == str(tmp_path / "node_modules" / "@uipath") + assert sandbox.plugin_tools_dir == str(workspace / "node_modules" / "@uipath") # ---------- version-string validation (junk-envelope guard) ---------- diff --git a/tests/test_sandbox.py b/tests/test_sandbox.py index f1f2175f..c77b73b0 100644 --- a/tests/test_sandbox.py +++ b/tests/test_sandbox.py @@ -157,30 +157,41 @@ def test_sandbox_run_command_task_dir_absent(): sandbox.cleanup() -def test_sandbox_run_command_uses_agent_command_base_path(monkeypatch, tmp_path): - """Criteria commands can be pinned to the PATH seen by the agent.""" +@pytest.mark.parametrize("entry", ["setup", "adopt"]) +@pytest.mark.parametrize(("mock_label", "expected"), [("mock", "mock"), (None, "host")], ids=["mock-wins", "host-kept"]) +def test_run_command_resolves_mock_path_dirs_ahead_of_host_path(monkeypatch, tmp_path, entry, mock_label, expected): + """A criterion resolves the mock binaries the agent got, and still reaches the host PATH.""" from tests._path_helpers import write_uip_shim - stale_bin = tmp_path / "stale" - agent_bin = tmp_path / "agent" - stale_bin.mkdir() - agent_bin.mkdir() - write_uip_shim(stale_bin, "stale") - write_uip_shim(agent_bin, "agent") - monkeypatch.setenv("PATH", str(stale_bin)) - - config = SandboxConfig(driver="tempdir", python=None) - sandbox = Sandbox(config, task_id="test_agent_path") + host_bin = tmp_path / "host" + host_bin.mkdir() + write_uip_shim(host_bin, "host") + monkeypatch.setenv("PATH", str(host_bin)) + workspace = tmp_path / "ws" + (workspace / "mocks").mkdir(parents=True) + if mock_label is not None: + write_uip_shim(workspace / "mocks", mock_label) + sandbox = Sandbox(SandboxConfig(driver="tempdir", python=None, mock_path_dirs=["mocks"]), task_id="t") try: - sandbox.setup() - sandbox.set_command_base_path(f"{agent_bin}{os.pathsep}{stale_bin}") + if entry == "setup": + sandbox.setup(target_dir=workspace) + else: + sandbox.adopt(workspace) + assert sandbox.command_base_path == str((workspace / "mocks").resolve()) exit_code, stdout, _stderr = sandbox.run_command("uip") assert exit_code == 0 - assert stdout.strip() == "agent" - # Read-only view exposes the same value `set_…` accepts. - assert sandbox.command_base_path == f"{agent_bin}{os.pathsep}{stale_bin}" + assert stdout.strip() == expected + finally: + sandbox.cleanup() + + +def test_command_base_path_is_none_without_mock_path_dirs(tmp_path): + sandbox = Sandbox(SandboxConfig(driver="tempdir", python=None), task_id="t") + try: + sandbox.setup() + assert sandbox.command_base_path is None finally: sandbox.cleanup() @@ -272,31 +283,21 @@ def test_refresh_plugin_tools_dir_none_when_uip_outside_uipath_tree(monkeypatch, sys.platform == "win32", reason="Fixture uses POSIX symlink + extensionless `uip`; `shutil.which` on Windows needs PATHEXT match.", ) -def test_set_command_base_path_refreshes_plugin_tools_dir(monkeypatch, tmp_path): - """PATH alignment from the agent should re-derive the canonical tools dir. - - Without this, criterion subprocesses might pin to a tools dir derived from - a `uip` that the agent never resolved to. - """ +def test_mock_path_dirs_drive_the_plugin_tools_dir_pin(monkeypatch, tmp_path): + """A `uip` in a mock dir wins the pin over the host PATH, as it wins the agent's lookup.""" pre_root = tmp_path / "pre" pre_root.mkdir() - agent_root = tmp_path / "agent" - agent_root.mkdir() + workspace = tmp_path / "ws" + (workspace / "tools").mkdir(parents=True) pre_bin = _install_fake_uip_tree(pre_root) - agent_bin = _install_fake_uip_tree(agent_root) - # Initial PATH points at the "pre" tools tree; agent PATH override (later) - # points at a different "agent" tree. `set_command_base_path` must - # re-resolve and update. + _install_fake_uip_tree(workspace / "tools") monkeypatch.setenv("PATH", str(pre_bin)) - config = SandboxConfig(driver="tempdir", python=None) - sandbox = Sandbox(config, task_id="test_plugin_tools_dir_refresh") + config = SandboxConfig(driver="tempdir", python=None, mock_path_dirs=["tools/bin"]) + sandbox = Sandbox(config, task_id="test_plugin_tools_dir_mock") try: - sandbox.setup() - pre_expected = str((pre_root / "node_modules" / "@uipath").resolve(strict=True)) - assert sandbox.plugin_tools_dir == pre_expected - sandbox.set_command_base_path(str(agent_bin)) - agent_expected = str((agent_root / "node_modules" / "@uipath").resolve(strict=True)) - assert sandbox.plugin_tools_dir == agent_expected + sandbox.setup(target_dir=workspace) + expected = str((workspace / "tools" / "node_modules" / "@uipath").resolve(strict=True)) + assert sandbox.plugin_tools_dir == expected finally: sandbox.cleanup() diff --git a/tests/test_sandbox_record_cli.py b/tests/test_sandbox_record_cli.py index c466b1f5..04c5e436 100644 --- a/tests/test_sandbox_record_cli.py +++ b/tests/test_sandbox_record_cli.py @@ -131,7 +131,6 @@ def test_recorder_dir_is_excluded_from_plugin_discovery(self): try: sandbox_dir = sandbox.setup() recorder = str((sandbox_dir / RECORD_CLI_DIR).resolve()) - sandbox.set_command_base_path(f"{recorder}{os.pathsep}{os.environ.get('PATH', '')}") assert recorder in sandbox.uip_search_path.split(os.pathsep) assert recorder not in sandbox._plugin_discovery_path().split(os.pathsep) finally: diff --git a/tests/test_seed_from_prior_result.py b/tests/test_seed_from_prior_result.py index fd0f4843..38a9df95 100644 --- a/tests/test_seed_from_prior_result.py +++ b/tests/test_seed_from_prior_result.py @@ -46,6 +46,7 @@ "iteration_count", "early_stop", "tool_calls_exhausted", + "model_turns", "error_message", "error_details", "error_log_tail", @@ -118,6 +119,7 @@ def _prior() -> EvaluationResult: iterations=[TurnRecord(iteration=1, user_input="prior prompt", agent_output="prior reply")], simulation=SimulationTelemetry(n_trials=3, replicate_index=2, stop_reason="stop_token", total_turns=4), tool_calls_exhausted=True, + model_turns=4, error_message="prior message", error_details={"where": "prior"}, error_log_tail="prior tail", diff --git a/tests/test_simulation_integration.py b/tests/test_simulation_integration.py index 1935c199..3d8cac84 100644 --- a/tests/test_simulation_integration.py +++ b/tests/test_simulation_integration.py @@ -27,6 +27,7 @@ ) from coder_eval.orchestrator import Orchestrator from coder_eval.simulation.user_simulator import UserSimulator +from coder_eval.streaming.emitter import TurnOutcome from tests.fixtures.harness_stubs import stub_contract from tests.fixtures.mock_agent import MockAgent from tests.fixtures.text_stub_agent import TextStubAgent @@ -77,11 +78,11 @@ def __init__(self, task: TaskDefinition, calls_per_turn: int) -> None: self._tool_seq = 0 self.emitted_per_turn: list[int] = [] - async def communicate(self, user_input: str, **kwargs: Any) -> TurnRecord: + async def communicate(self, user_input: str, **kwargs: Any) -> TurnOutcome: from datetime import datetime from coder_eval.models import CommandTelemetry - from coder_eval.streaming.events import StopReason, ToolEndEvent, ToolStartEvent + from coder_eval.streaming.events import AgentEndStatus, StopReason, ToolEndEvent, ToolStartEvent self._iteration += 1 stream_callback = kwargs["stream_callback"] @@ -94,13 +95,14 @@ async def communicate(self, user_input: str, **kwargs: Any) -> TurnRecord: stream_callback.on_event(ToolEndEvent(task_id=self.task.task_id, tool=tool)) commands.append(tool) self.emitted_per_turn.append(len(commands)) - return TurnRecord( + record = TurnRecord( iteration=self._iteration, user_input=user_input, agent_output="working", commands=commands, tool_calls_exhausted=should_stop() is StopReason.TOOL_CALL_CAP, ) + return TurnOutcome(record=record, status=AgentEndStatus.COMPLETED, error=None) def _install_fake_simulator( @@ -131,15 +133,16 @@ def __init__(self, responses: list[str], *, tokens: tuple[int, int] = (5, 7)) -> super().__init__(responses) self._tokens = tokens - async def communicate(self, user_input: str, **kwargs: object) -> TurnRecord: - turn = await super().communicate(user_input, **kwargs) + async def communicate(self, user_input: str, **kwargs: object) -> TurnOutcome: + outcome = await super().communicate(user_input, **kwargs) in_tok, out_tok = self._tokens - return TurnRecord( - iteration=turn.iteration, - user_input=turn.user_input, - agent_output=turn.agent_output, + record = TurnRecord( + iteration=outcome.record.iteration, + user_input=outcome.record.user_input, + agent_output=outcome.record.agent_output, token_usage=TokenUsage(uncached_input_tokens=in_tok, output_tokens=out_tok), ) + return TurnOutcome(record=record, status=outcome.status, error=outcome.error) class _ExplodingAgent(Agent): diff --git a/tests/test_spi.py b/tests/test_spi.py index 714683dd..d1cc0f91 100644 --- a/tests/test_spi.py +++ b/tests/test_spi.py @@ -7,19 +7,80 @@ _ORIGINS = ( "coder_eval.agent", + "coder_eval.agents._transport", "coder_eval.agents.registry", + "coder_eval.agents.watchdog", "coder_eval.errors", "coder_eval.models", "coder_eval.pricing", "coder_eval.streaming.callbacks", - "coder_eval.streaming.collector", + "coder_eval.streaming.emitter", "coder_eval.streaming.events", "coder_eval.timing", ) -def test_spi_version_is_two() -> None: - assert spi.SPI_VERSION == 2 +_FINAL_EXPORTS = [ + "Agent", + "AgentConfigError", + "AgentCrashError", + "AgentEndStatus", + "AgentRegistry", + "AgentState", + "ApiRoute", + "BaseAgentConfig", + "CANONICAL_TOOL_NAMES", + "CommandTelemetry", + "ContentBlock", + "Enforcement", + "Generation", + "HarnessContract", + "JsonlDecoder", + "LocalPluginConfig", + "ModelPricing", + "PermissionMode", + "READ_ONLY_DENIED_TOOLS", + "ResultSummary", + "SPI_VERSION", + "StopReason", + "StreamCallback", + "SubprocessJsonlAgent", + "SystemPromptMode", + "TimingBasis", + "TokenUsage", + "ToolEndStatus", + "ToolNameMap", + "TranscriptMessage", + "TurnClock", + "TurnEmitter", + "TurnEndStatus", + "TurnOutcome", + "TurnRecord", + "TurnTimeoutError", + "UsageGranularity", + "WatchdogFired", + "Window", + "close_window", + "end_status_for", + "format_timeout_reason", + "price_turn", + "register_pricing", + "run_with_watchdog", +] + + +def test_the_export_list_is_pinned() -> None: + """A plugin writes the event protocol through ``TurnEmitter``: no event class, collector or composite callback.""" + assert spi.__all__ == _FINAL_EXPORTS + + +def test_spi_version_is_one() -> None: + assert spi.SPI_VERSION == 1 + + +def test_the_emitter_surface_is_exported() -> None: + assert {"TurnEmitter", "TurnOutcome", "Generation", "Window", "TimingBasis"} <= set(spi.__all__) + assert {"run_with_watchdog", "WatchdogFired", "SubprocessJsonlAgent", "JsonlDecoder"} <= set(spi.__all__) def test_the_stop_channel_is_exported() -> None: diff --git a/tests/test_streaming_agent_integration.py b/tests/test_streaming_agent_integration.py index 4e05a753..6df400ef 100644 --- a/tests/test_streaming_agent_integration.py +++ b/tests/test_streaming_agent_integration.py @@ -156,7 +156,7 @@ async def fake_query(**kwargs): yield msg with patch("coder_eval.agents.claude_code_agent.query", side_effect=fake_query): - await agent.communicate("test prompt", stream_callback=callback) + await agent.communicate("test prompt", iteration=1, stream_callback=callback) _assert_well_formed_tree(callback.events, expected_agent_status=AgentEndStatus.COMPLETED) @@ -191,7 +191,7 @@ async def fake_query(**kwargs): yield msg with patch("coder_eval.agents.claude_code_agent.query", side_effect=fake_query): - await agent.communicate("test prompt", stream_callback=callback) + await agent.communicate("test prompt", iteration=1, stream_callback=callback) _assert_well_formed_tree(callback.events, expected_agent_status=AgentEndStatus.COMPLETED) @@ -226,7 +226,7 @@ async def fake_query(**kwargs): yield msg with patch("coder_eval.agents.claude_code_agent.query", side_effect=fake_query): - await agent.communicate("test prompt", stream_callback=callback) + await agent.communicate("test prompt", iteration=1, stream_callback=callback) end_events = [e for e in callback.events if isinstance(e, ToolEndEvent)] assert len(end_events) == 1 @@ -260,7 +260,7 @@ async def fake_query(**kwargs): yield msg with patch("coder_eval.agents.claude_code_agent.query", side_effect=fake_query): - await agent.communicate("test prompt", stream_callback=callback) + await agent.communicate("test prompt", iteration=1, stream_callback=callback) _assert_well_formed_tree(callback.events, expected_agent_status=AgentEndStatus.COMPLETED) @@ -292,7 +292,7 @@ async def fake_query(**kwargs): yield msg with patch("coder_eval.agents.claude_code_agent.query", side_effect=fake_query): - await agent.communicate("hello world", stream_callback=callback) + await agent.communicate("hello world", iteration=1, stream_callback=callback) start = callback.events[0] assert isinstance(start, AgentStartEvent) @@ -323,7 +323,7 @@ async def fake_query(**kwargs): yield msg with patch("coder_eval.agents.claude_code_agent.query", side_effect=fake_query): - returned = await agent.communicate("test prompt", stream_callback=callback) + outcome = await agent.communicate("test prompt", iteration=1, stream_callback=callback) # Replay the captured stream through a fresh collector and confirm it # reduces to a record consistent with what communicate() returned. @@ -335,7 +335,7 @@ async def fake_query(**kwargs): assert rebuilt.user_input == "test prompt" assert len(rebuilt.commands) == 1 assert rebuilt.commands[0].tool_id == "tc" - assert len(returned.commands) == len(rebuilt.commands) + assert len(outcome.record.commands) == len(rebuilt.commands) @pytest.mark.asyncio @@ -359,10 +359,10 @@ async def fake_query(**kwargs): yield msg with patch("coder_eval.agents.claude_code_agent.query", side_effect=fake_query): - turn = await agent.communicate("test prompt") # No callback + outcome = await agent.communicate("test prompt", iteration=1) # No callback - assert turn is not None - assert len(turn.commands) == 1 + assert outcome.record is not None + assert len(outcome.record.commands) == 1 @pytest.mark.asyncio @@ -389,7 +389,7 @@ async def fake_query(**kwargs): yield msg with patch("coder_eval.agents.claude_code_agent.query", side_effect=fake_query): - await agent.communicate("test prompt", stream_callback=scoped_callback) + await agent.communicate("test prompt", iteration=1, stream_callback=scoped_callback) # All events (AgentStart -> ... -> AgentEnd) should now carry the real task ID. assert len(inner_callback.events) > 0 diff --git a/tests/test_sub_agent_runner.py b/tests/test_sub_agent_runner.py index 6e58843f..4d5f439c 100644 --- a/tests/test_sub_agent_runner.py +++ b/tests/test_sub_agent_runner.py @@ -9,6 +9,7 @@ import pytest +from coder_eval.errors.agent import AgentCrashError from coder_eval.errors.timeout import TurnTimeoutError from coder_eval.evaluation.sub_agent import ( SubAgentRunner, @@ -17,6 +18,8 @@ from coder_eval.models import AgentKind, ClaudeCodeAgentConfig, TurnRecord, parse_agent_config from coder_eval.models.routing import DirectRoute from coder_eval.sandbox import Sandbox +from coder_eval.streaming.emitter import TurnOutcome +from coder_eval.streaming.events import AgentEndStatus # Symlink creation on Windows requires either admin privileges or Developer @@ -56,10 +59,14 @@ def _make_turn() -> TurnRecord: return TurnRecord(iteration=1, user_input="x", agent_output='{"score": 1.0, "rationale": "ok"}') +def _make_outcome(record: TurnRecord | None = None) -> TurnOutcome: + return TurnOutcome(record=record or _make_turn(), status=AgentEndStatus.COMPLETED, error=None) + + def _make_mock_agent() -> MagicMock: agent = MagicMock() agent.start = AsyncMock(return_value=None) - agent.communicate = AsyncMock(return_value=_make_turn()) + agent.communicate = AsyncMock(return_value=_make_outcome()) agent.stop = AsyncMock(return_value=None) agent.kill = AsyncMock(return_value=None) return agent @@ -83,7 +90,7 @@ async def test_runner_happy_path(sandbox: Sandbox, tmp_path: Path) -> None: assert turn.agent_output == '{"score": 1.0, "rationale": "ok"}' mock_agent.start.assert_awaited_once() - mock_agent.communicate.assert_awaited_once_with("grade this", timeout=30.0) + mock_agent.communicate.assert_awaited_once_with("grade this", iteration=1, timeout=30.0) mock_agent.stop.assert_awaited() @@ -115,13 +122,13 @@ async def capture_start(workdir: str, **_kwargs: object) -> None: mock_agent.start.side_effect = capture_start with patch("coder_eval.evaluation.sub_agent.ClaudeCodeAgent", return_value=mock_agent): # Capture the workdir before it's torn down by the finally block. - async def capture_files(_msg: str, **_kw: object) -> TurnRecord: + async def capture_files(_msg: str, **_kw: object) -> TurnOutcome: workdir = Path(captured["workdir"]) captured["has_reference_dir"] = str((workdir / "_reference").is_dir()) captured["has_main"] = str((workdir / "_reference" / "Main.xaml").is_file()) captured["main_content"] = (workdir / "_reference" / "Main.xaml").read_text() captured["has_subdir"] = str((workdir / "_reference" / "subdir" / "Helper.xaml").is_file()) - return _make_turn() + return _make_outcome() mock_agent.communicate.side_effect = capture_files await runner.run_async("grade", turn_timeout=30.0) @@ -166,12 +173,12 @@ async def capture_start(workdir: str, **_kwargs: object) -> None: mock_agent.start.side_effect = capture_start with patch("coder_eval.evaluation.sub_agent.ClaudeCodeAgent", return_value=mock_agent): - async def capture_state(_msg: str, **_kw: object) -> TurnRecord: + async def capture_state(_msg: str, **_kw: object) -> TurnOutcome: workdir = Path(captured["workdir"]) ref_main = workdir / "_reference" / "Main.xaml" captured["ref_main_content"] = ref_main.read_text() captured["agent_planted_present"] = str((workdir / "_reference" / "agent_planted.txt").exists()) - return _make_turn() + return _make_outcome() mock_agent.communicate.side_effect = capture_state # Must not raise — this is the regression assertion. @@ -202,10 +209,10 @@ async def capture_start(workdir: str, **_kwargs: object) -> None: mock_agent.start.side_effect = capture_start with patch("coder_eval.evaluation.sub_agent.ClaudeCodeAgent", return_value=mock_agent): - async def capture_no_ref(_msg: str, **_kw: object) -> TurnRecord: + async def capture_no_ref(_msg: str, **_kw: object) -> TurnOutcome: workdir = Path(captured["workdir"]) captured["has_reference_dir"] = str((workdir / "_reference").exists()) - return _make_turn() + return _make_outcome() mock_agent.communicate.side_effect = capture_no_ref await runner.run_async("grade", turn_timeout=30.0) @@ -274,7 +281,7 @@ async def test_runner_cleans_up_when_cancelled_mid_communicate(sandbox: Sandbox) async def capture_start(path: str, **_kwargs: object) -> None: captured["path"] = path - async def hang_forever(*_args: object, **_kwargs: object) -> TurnRecord: + async def hang_forever(*_args: object, **_kwargs: object) -> TurnOutcome: started.set() await asyncio.sleep(3600) raise AssertionError("should have been cancelled before waking up") @@ -399,7 +406,9 @@ async def test_runner_propagates_turn_timeout(sandbox: Sandbox) -> None: route=DirectRoute(), ) mock_agent = _make_mock_agent() - mock_agent.communicate.side_effect = TurnTimeoutError(30.0, task_id="t", iteration=1) + mock_agent.communicate.return_value = TurnOutcome( + record=_make_turn(), status=AgentEndStatus.TIMEOUT, error="timed out" + ) captured: dict[str, str] = {} async def capture_start(path: str, **_kwargs: object) -> None: @@ -416,6 +425,34 @@ async def capture_start(path: str, **_kwargs: object) -> None: assert not Path(captured["path"]).exists() +async def test_runner_propagates_agent_crash_error(sandbox: Sandbox) -> None: + runner = SubAgentRunner( + sandbox=sandbox, + agent_config=_make_agent_config(), + ignore_patterns=[], + route=DirectRoute(), + ) + mock_agent = _make_mock_agent() + crashed_record = TurnRecord(iteration=1, user_input="x", agent_output="", crashed=True, crash_reason="boom") + mock_agent.communicate.return_value = TurnOutcome( + record=crashed_record, status=AgentEndStatus.CRASHED, error="boom" + ) + captured: dict[str, str] = {} + + async def capture_start(path: str, **_kwargs: object) -> None: + captured["path"] = path + + mock_agent.start.side_effect = capture_start + + with ( + patch("coder_eval.evaluation.sub_agent.ClaudeCodeAgent", return_value=mock_agent), + pytest.raises(AgentCrashError, match="boom"), + ): + await runner.run_async("grade", turn_timeout=30.0) + + assert not Path(captured["path"]).exists() + + # --- security contract --- @@ -648,12 +685,12 @@ async def capture_start(workdir: str, **_kwargs: object) -> None: mock_agent.start.side_effect = capture_start with patch("coder_eval.evaluation.sub_agent.ClaudeCodeAgent", return_value=mock_agent): - async def capture_state(_msg: str, **_kw: object) -> TurnRecord: + async def capture_state(_msg: str, **_kw: object) -> TurnOutcome: workdir = Path(captured["workdir"]) inner = workdir / "_reference" / "nested" / "_reference" / "inner.txt" captured["inner_present"] = str(inner.exists()) captured["inner_content"] = inner.read_text(encoding="utf-8") if inner.exists() else "" - return _make_turn() + return _make_outcome() mock_agent.communicate.side_effect = capture_state await runner.run_async("grade", turn_timeout=30.0) @@ -688,11 +725,11 @@ async def capture_start(workdir: str, **_kwargs: object) -> None: mock_agent.start.side_effect = capture_start with patch("coder_eval.evaluation.sub_agent.ClaudeCodeAgent", return_value=mock_agent): - async def capture_state(_msg: str, **_kw: object) -> TurnRecord: + async def capture_state(_msg: str, **_kw: object) -> TurnOutcome: workdir = Path(captured["workdir"]) captured["keep_present"] = str((workdir / "_reference" / "keep.txt").exists()) captured["log_present"] = str((workdir / "_reference" / "drop.log").exists()) - return _make_turn() + return _make_outcome() mock_agent.communicate.side_effect = capture_state await runner.run_async("grade", turn_timeout=30.0) diff --git a/tests/test_subprocess_jsonl_agent.py b/tests/test_subprocess_jsonl_agent.py new file mode 100644 index 00000000..2cda5cab --- /dev/null +++ b/tests/test_subprocess_jsonl_agent.py @@ -0,0 +1,247 @@ +"""``SubprocessJsonlAgent``: the shared nd-JSON CLI transport, against real child processes.""" + +from __future__ import annotations + +import asyncio +import json +import os +import sys +from typing import Any + +import pytest + +from coder_eval.agents._transport import JsonlDecoder, SubprocessJsonlAgent, subprocess_jsonl +from coder_eval.models import Enforcement, HarnessContract, PiAgentConfig, TimingBasis, UsageGranularity +from coder_eval.streaming.emitter import TurnOutcome +from coder_eval.streaming.events import AgentEndEvent, AgentEndStatus, StopReason, StreamEvent, end_status_for + + +pytestmark = pytest.mark.skipif(os.name != "posix", reason="process-group teardown is POSIX-only") + + +class _Decoder(JsonlDecoder): + def __call__(self, event: dict[str, Any]) -> None: + if event.get("type") == "say": + self.emitter.text(str(event.get("text", ""))) + elif event.get("type") == "err": + self.error = str(event.get("message")) + + def end(self, status: AgentEndStatus, *, reason: str | None = None) -> TurnOutcome: + if status is AgentEndStatus.CRASHED or status is AgentEndStatus.TIMEOUT: + return self.emitter.fail(status, reason or status.value) + return self.emitter.finalize(status) + + +class _ScriptAgent(SubprocessJsonlAgent[PiAgentConfig]): + """Runs ``python -c