feat(evaluations): preserve the tool trajectory for judges

[donei003]

Now carries two commits: the trajectory capture, and #98 (merged in here) which made every judge path share one message_history.

Problem

A judge can only grade what it is shown. Handler packages record tool traffic onto OpenTelemetry spans and return only {output, usage}, so by the time a judge ran, the calls made on the way to that output were gone. "Did the agent call the right tool, in the right order, with the right arguments?" was an unaskable question of an SDK that had just run the agent that answered it.

Underneath that, the three judge paths had already drifted. Each built message_history with its own inline join:

Path	message_history was
Offline evals	row input + output + format block
Online inline (run_judges)	user input + output + format block
Online deferred (run_judge)	output + format block — no input at all

A deferred judge was grading a response with no request beside it. That is a pre-existing bug, fixed here as a consequence of unifying.

Approach

Capture by wrapping the tool map, which both paths already own — so every handler package is covered without changing any of them, and a custom caller-supplied handler is covered too. A handler still resolves a tool by the key the model named and calls it.

• Offline: once per row, in evaluations/runner.py.
• Online: once per invocation, in tracking.execute_and_track / execute_and_stream, which now return the rendered trajectory alongside response and track_data.

judge_scoring.build_message_history is the only place a history is built — in the module that already owns the {score, reasoning} contract, for exactly the same reason. It orders the conversation the way it happened: input → trajectory → output → format block, skipping empty parts. All three paths call it.

What a judge now sees:

Where is order A1?

Tools available: lookup_order, issue_refund
Tool calls made while producing the response, in order:
1. lookup_order
   arguments: {"id":"A1"}
   result: order A1 shipped 2026-08-02
2. issue_refund
   arguments: {"id":"A1","amount":19.99}
   error: refund window closed

That order shipped on Aug 2 and is outside the refund window.

Your response MUST be in valid JSON format with the following structure:
...

Verified live against a real LaunchDarkly judge: the judge's reasoning cited the call, its arguments, and its result — all three only visible via the trajectory.

Properties pinned by tests

• The recorder observes; it never intervenes. A wrapped tool returns exactly what the original returned and raises exactly what the original raised. Calls past the 50-call cap still execute and are only counted — truncation drops the record, never the work, because a harness that changed the agent's behaviour would no longer be evaluating the agent.
• A recorder belongs to one invocation. Rows generate concurrently against one shared tool map, so a shared recorder would splice one row's calls into another's and hand the judge a conversation that never happened. Tested with two rows whose calls interleave.
• Order is call-start order, not completion order. A judge asked whether the agent searched before it refunded is reading a sequence.
• Inline and deferred produce byte-identical history for the same row. This is the property that makes a rubric portable between a production sample and a dataset replay.
• A tool result stays literal in the judge prompt. A tool result is a new injection surface alongside generated output, closed by the existing rule: the judge config is passed unrendered and the handler makes exactly one template pass. Tested with a tool returning the literal text {{expected_output}}.
• Back-compatible. A run with no observable tools adds no block, so a judge authored before this reads exactly the message_history it read before.
• Nothing is added to any event payload. The trajectory reaches LaunchDarkly only inside the prompt a judge was shown, never as a wire field the backend has not specified.

Design notes

• No standalone trajectory variable. An earlier revision exposed {{tool_trajectory}} too; it overlapped message_history and bought nothing, while inviting a rubric to interpolate both and pay for the trajectory twice. Confirmed live — a real judge config's own scaffolding already interpolates {{message_history}}. A test pins its absence. Since message_history is what judges cloned from the AI Library's default templates read, an existing judge becomes a trajectory judge by editing its rubric text alone.
• Native provider tools are skipped in both paths, and left out of "Tools available". Online, wrap_tool_handlers substitutes a callable tracking stub, so a native call is locally observable — but the stub returns nothing, so recording it would show a judge a call with an empty result while the provider's real result stayed invisible. Recording is therefore composed inside that wrapper, on the original map. Tests assert both the exclusion and that $ld:ai:tool_call still fires underneath.
• JudgeTask gains user_input and trajectory as plain strings — every field on it has to survive pickling to a worker thread; a test pins that.
• Graph-level judges get no trajectory, deliberately: graph_judge grades an answer produced across several nodes, and splicing their trajectories would describe a conversation that never happened. Per-node judges get their own node's.
• Bounds: 50 recorded calls per invocation, 2000 characters per rendered argument bag or result. A trajectory goes into a judge prompt, so an agent looping over a large result set would otherwise spend the judge's context window — and budget — on a tail no judge reads.
• trajectory.py sits at the package root, not under evaluations/, since it is no longer evaluations-specific.

Deliberately out of scope

Scorers cannot see the trajectory. Scorer.fn(row, output) is the contract, and the trajectory is not dataset-owned so it does not belong on DatasetRow. A deterministic check like "called lookup_order exactly once" is a natural follow-up but needs a contract change, not a quiet signature widening.

Validation

• uv run pytest -q — 1282 passed, 11 skipped
• uv run mypy packages/client/src/launchdarkly_ai_server — clean; ruff check / format --check — clean
• Exercised end-to-end in ai-sdk-evaluations-example (launchdarkly-labs/ai-sdk-evaluations-example#5), including a live run against a real judge

Language-agnostic spec: launchdarkly/ai-sdks-monorepo#13 — being restructured to describe the shared flow rather than the offline phase alone, now that this shape is settled. Submodule pointer: launchdarkly/ai-sdks-monorepo#14.

🤖 Generated with Claude Code