feat(gooddata-eval): make the K verdict a gate, pass@K or pass^K

packages/gooddata-eval/README.md

@@ -126,7 +126,8 @@ gd-eval run \
 
 | Flag | Default | Description |
 |---|---|---|
-| `--runs K` | `2` | Independent runs per item (pass@K). An item passes if any run passes. |
+| `--runs K` | `2` | Independent runs per item. |
+| `--gate` | `any` | Which verdict decides an item: `any` = pass@K (a run passing is enough), `power` = pass^K (every run must pass, so the verdict measures stability). Identical at `--runs 1`. Kinds that repeat K runs only — `power` is refused when the dataset also has non-agentic items or `agentic_conversation`, which are always decided on pass@K. |
 | `--concurrency K` | `1` | Number of items evaluated concurrently. `1` = sequential (default). Increase to load-test the agent under simultaneous requests — see *Concurrency and workspace safety* below. |
 | `--judge-model MODEL` | `gpt-4o` | Model used for LLM-as-judge scoring — `agentic_general_question`, `agentic_guardrail`, `general_question`, `guardrail` and `dashboard_summary`. Also settable via `GD_EVAL_JUDGE_MODEL`. Two things to weigh before changing it: the gpt-5 family rejects `temperature=0`, so verdicts stop being reproducible (the run warns when this happens); and choosing the same model the agent runs means the judge grades its own family's output. |
 | `--reasoning-effort LEVEL` | server default | `LOW`, `MEDIUM` or `HIGH`, sent as `options.reasoningEffort` on every chat message. Requires the `enableGenAiReasoningEffort` feature flag on the target organization — without it the server ignores the value. Applies to chat items only; `dashboard_summary` items go through the summary endpoint, which has no such option. |
@@ -231,10 +232,12 @@ Winner is selected by **pass rate → quality score → latency** (lower latency
 Each item reports **how many of its runs passed**, not only whether one did:
 
 ```json
-"runs": 5, "runs_passed": 4, "pass_at_k": true, "pass_power_k": false
+"runs": 5, "runs_passed": 4, "pass_at_k": true, "pass_power_k": false, "gate_passed": true
 ```
 
-`pass_at_k` is "did any run pass" and is what `passed` counts. `runs_passed` is the fact that separates a
+`pass_at_k` is "did any run pass" — always literal, whatever the gate. `gate_passed` is the verdict the item
+was decided on and is what `passed` counts; the two differ only under `--gate power`, where an item that
+passed 4 of 5 runs is `"pass_at_k": true, "gate_passed": false`. `runs_passed` is the fact that separates a
 reliable item from a coin-flip — without it a 5/5 item and a 1/5 item are identical in every field, because
 `quality_score` is derived from the best run alone. `pass_power_k` is true only when every run passed, and the
 run summary carries `passed_all_runs` beside `passed`; a large gap between the two means the model is
@@ -244,6 +247,14 @@ stays quiet for a unanimous one, and its summary line reads `3/4 passed, 1 on ev
 `runs` is what the item actually ran, which is not always the requested `--runs`: `agentic_conversation` takes
 no K and drives its fixture exactly once.
 
+Which of the two decides pass/fail is `--gate`: `any` (default) gates on `pass_at_k`, `power` gates on
+`pass_power_k`. The run records it as a top-level `gate`, and a failure under `power` says so —
+`Gate pass^3 failed: 2/3 runs passed — unstable, not a clean failure` — because the message body describes
+the best run, which under pass^K can be a run that passed, and the console repeats the count in `Notes` for
+the same reason. When some runs were ungraded the note says so instead of calling the remainder unstable.
+`agentic_conversation` has no K gate, so `--gate power` is refused for a dataset containing one rather than
+labelling a report `power` that only part of the dataset was decided under.
+
 Each item additionally carries a per-phase breakdown:
 
 ```json
@@ -410,8 +421,13 @@ the item's own root span. On the agentic path each score is mirrored onto the ag
 
 | Score | Description |
 |---|---|
-| `pass_at_k` | 1 if any of the K runs passed strict checks, else 0. |
+| `pass_at_k` | 1 if **any** of the K runs passed strict checks, else 0. |
+| `pass_power_k` | 1 only if **every** one of the K runs passed. Agentic kinds only. |
+| `gate_passed` | The verdict that decided the item: `pass_at_k` under `--gate any`, `pass_power_k` under `--gate power`. Agentic kinds only. |
 | `quality_score` | Fraction of strict check flags that are `True` (0.0–1.0). Shown in CLI as a percentage. |
 | `value_score` | Weighted blend: 0.6 × quality + 0.2 × speed (speed = max(0, 1 − latency/60s)). |
 | `latency_s` | Average per-run latency in seconds. |
 | `provider_type` | Model vendor + gateway label (e.g. `ANTHROPIC`, `BEDROCK/ANTHROPIC`, `AZURE/OPENAI`). Stored in Langfuse trace metadata and tags. |
+
+Score names carry no K; K and the gate are on the dataset-run metadata as `eval_k` and `eval_gate`.
+`agentic_visualization` also still writes its historic `pass_at_{K}` / `pass_power_{K}` pair.

packages/gooddata-eval/src/gooddata_eval/cli/agentic_runner.py

@@ -8,6 +8,7 @@
 from concurrent.futures import ThreadPoolExecutor, as_completed
 from typing import Any, TypedDict
 
+from gooddata_eval.core.agentic._gate import DEFAULT_GATE, EvalGate, normalize_gate
 from gooddata_eval.core.agentic._langfuse import make_langfuse_client
 from gooddata_eval.core.agentic._trace_linker import BackgroundTraceLinker, SubmitTraceLink, run_trace_link_inline
 from gooddata_eval.core.agentic.alert_skill import evaluate_agentic_alert_skill
@@ -48,6 +49,13 @@ class _LfKw(TypedDict, total=False):
 )
 
 
+# Agentic kinds that no gate applies to: they drive their fixture exactly once, so there is
+# no K to take pass@K or pass^K over. Named here rather than inline in _dispatch_agentic so
+# the CLI can refuse --gate power for a dataset containing one instead of labelling the whole
+# report `power` when part of it was never gated.
+UNGATED_AGENTIC_TEST_KINDS = frozenset({"agentic_conversation"})
+
+
 # Kinds cleared to run several at a time. An EXPLICIT allowlist, not a subtraction: nothing
 # in this package can prove a kind is read-only, because the mutation happens server-side in
 # whichever tools the agent decides to call. So each entry here is a reviewed judgement, and
@@ -128,9 +136,13 @@ def _dispatch_agentic(
     reasoning_effort: ReasoningEffort | None = None,
     agent_id: str | None = None,
     submit_trace_link: SubmitTraceLink = run_trace_link_inline,
+    gate: EvalGate = DEFAULT_GATE,
 ) -> AgenticEvalOutcome:
     """Call the appropriate evaluate_agentic_* function for the item's test_kind.
 
+    `gate` reaches every kind except those in UNGATED_AGENTIC_TEST_KINDS, which have no K
+    to gate over; the CLI refuses --gate power for a dataset containing one.
+
     Every evaluate_agentic_* function returns an AgenticEvalOutcome (reasoning_steps,
     conversation_id, response_id, detail) on success and attaches the same four attributes
     to its raised *AssertionError on failure -- no kind is exempt.
@@ -155,6 +167,7 @@ def _dispatch_agentic(
             question=item.question,
             expected_outputs=_parse_visualization_expected(eo),
             k=k,
+            gate=gate,
             agent_id=agent_id,
             **lf_kw,
         )
@@ -166,6 +179,7 @@ def _dispatch_agentic(
             question=item.question,
             expected_output=eo if isinstance(eo, (dict, list)) else {},
             k=k,
+            gate=gate,
             agent_id=agent_id,
             **lf_kw,
         )
@@ -177,6 +191,7 @@ def _dispatch_agentic(
             question=item.question,
             expected_output=eo if isinstance(eo, dict) else {},
             k=k,
+            gate=gate,
             agent_id=agent_id,
             **lf_kw,
         )
@@ -191,6 +206,7 @@ def _dispatch_agentic(
             question=item.question,
             expected_tool_call=expected_args,
             k=k,
+            gate=gate,
             agent_id=agent_id,
             **lf_kw,
         )
@@ -202,6 +218,7 @@ def _dispatch_agentic(
             question=item.question,
             expected_output=eo if isinstance(eo, str) else str(eo),
             k=k,
+            gate=gate,
             agent_id=agent_id,
             user_context=item.user_context,
             **lf_kw,
@@ -214,6 +231,7 @@ def _dispatch_agentic(
             question=item.question,
             expected_output=eo if isinstance(eo, str) else str(eo),
             k=k,
+            gate=gate,
             agent_id=agent_id,
             **lf_kw,
         )
@@ -225,6 +243,7 @@ def _dispatch_agentic(
             question=item.question,
             expected_output=eo if isinstance(eo, dict) else {},
             k=k,
+            gate=gate,
             agent_id=agent_id,
             **lf_kw,
         )
@@ -291,6 +310,7 @@ def run_agentic_items(
     on_item_done: Any = None,
     agent_id: str | None = None,
     concurrency: int = 1,
+    gate: EvalGate = DEFAULT_GATE,
 ) -> EvalReport:
     """Run agentic items through evaluate_agentic_* and return an EvalReport.
 
@@ -303,7 +323,7 @@ def run_agentic_items(
     """
     langfuse = make_langfuse_client() if use_langfuse else None
 
-    report = EvalReport(model=model_version)
+    report = EvalReport(model=model_version, gate=normalize_gate(gate))
     total = len(items)
     # Trace linking runs here rather than inside each evaluate_agentic_*, so an item's
     # Langfuse poll overlaps the NEXT item's agent call instead of extending its own
@@ -325,6 +345,9 @@ def _process_item(index: int, item: DatasetItem) -> ItemReport:
             test_kind=item.test_kind,
             question=item.question,
         )
+        # None, not False, for the kinds _dispatch_agentic passes no gate to: ItemReport.passed
+        # then falls back to pass_at_k and gate_passed keeps meaning "a gate ran".
+        gated = item.test_kind not in UNGATED_AGENTIC_TEST_KINDS
         t0 = time.perf_counter()
         try:
             outcome = _dispatch_agentic(
@@ -339,6 +362,7 @@ def _process_item(index: int, item: DatasetItem) -> ItemReport:
                 reasoning_effort,
                 agent_id,
                 submit_trace_link=linker.submit,
+                gate=gate,
             )
             if isinstance(outcome, AgenticEvalOutcome):
                 reasoning_steps = outcome.reasoning_steps
@@ -347,6 +371,8 @@ def _process_item(index: int, item: DatasetItem) -> ItemReport:
                 detail = outcome.detail
             else:
                 reasoning_steps, conversation_id, response_id, detail = outcome, None, None, {}
+            item_report.gate_passed = True if gated else None
+            # Whichever gate decided the item, clearing it means at least one run passed.
             item_report.pass_at_k = True
             item_report.runs = k
             item_report.reasoning_steps = reasoning_steps or []
@@ -356,14 +382,18 @@ def _process_item(index: int, item: DatasetItem) -> ItemReport:
             _apply_timings(item_report, getattr(outcome, "timings", None))
             _apply_run_counts(item_report, outcome)
         except AssertionError as exc:
-            item_report.pass_at_k = False
+            item_report.gate_passed = False if gated else None
             item_report.runs = k
             item_report.reasoning_steps = getattr(exc, "reasoning_steps", None) or []
             item_report.conversation_id = getattr(exc, "conversation_id", None)
             item_report.response_id = getattr(exc, "response_id", None)
             item_report.best_detail = getattr(exc, "detail", None) or {}
             _apply_timings(item_report, getattr(exc, "timings", None))
             _apply_run_counts(item_report, exc)
+            # Read off the counts, not off the gate: pass^K fails items where runs did pass,
+            # and reporting those as pass_at_k False would contradict the Langfuse score of
+            # the same name. Kinds that report no count read as 0, i.e. a clean failure.
+            item_report.pass_at_k = item_report.runs_passed > 0
             print(f"[agentic] {item.id} FAIL: {exc}", flush=True)
         except Exception as exc:
             item_report.error = f"{type(exc).__name__}: {exc}"

packages/gooddata-eval/src/gooddata_eval/cli/main.py

@@ -14,11 +14,20 @@
 from rich.console import Console
 from rich.table import Table
 
-from gooddata_eval.cli.agentic_runner import AGENTIC_TEST_KINDS, run_agentic_items
+from gooddata_eval.cli.agentic_runner import AGENTIC_TEST_KINDS, UNGATED_AGENTIC_TEST_KINDS, run_agentic_items
 from gooddata_eval.core.chat.sse_client import ChatClient
-from gooddata_eval.core.config import DEFAULT_JUDGE_MODEL, JUDGE_MODEL_ENV_VAR, ReasoningEffort, RunConfig
+from gooddata_eval.core.config import (
+    DEFAULT_GATE,
+    DEFAULT_JUDGE_MODEL,
+    JUDGE_MODEL_ENV_VAR,
+    EvalGate,
+    ReasoningEffort,
+    RunConfig,
+    normalize_gate,
+)
 from gooddata_eval.core.connection import ConnectionError_, resolve_connection
 from gooddata_eval.core.dataset.local import load_local_dataset
+from gooddata_eval.core.evaluators import supported_test_kinds
 from gooddata_eval.core.langfuse.sink import LangfuseSink
 from gooddata_eval.core.models import ChatResult, DatasetItem
 from gooddata_eval.core.reporting.console import render_comparison, render_console
@@ -91,7 +100,15 @@ def _build_parser() -> argparse.ArgumentParser:
             "Default: workspace's current active model."
         ),
     )
-    run.add_argument("--runs", type=int, default=2, help="Independent runs per item (pass@K). Default 2.")
+    run.add_argument("--runs", type=int, default=2, help="Independent runs per item. Default 2.")
+    run.add_argument(
+        "--gate",
+        choices=get_args(EvalGate),
+        default=DEFAULT_GATE,
+        help="Which verdict decides an item: 'any' = pass@K (a run passing is enough, the "
+        "default and historic behaviour), 'power' = pass^K (every run must pass, so the verdict "
+        "measures stability). Identical at --runs 1. Agentic kinds only.",
+    )
     run.add_argument(
         "--concurrency",
         type=int,
@@ -180,6 +197,37 @@ def _apply_timer_flag(enabled: bool) -> None:
         os.environ[TIMERS_ENV_VAR] = "1"
 
 
+def _reject_power_gate_on_ungated_items(config: RunConfig, items: list) -> None:
+    """Refuse a pass^K request the run cannot honour for every item.
+
+    Two kinds of item are never gated: everything on the non-agentic path, because
+    `run_items` has no gate and always decides on pass@K, and agentic_conversation, which
+    drives its fixture once whatever --runs says and so has no K to gate over. Running a
+    mixed dataset anyway would decide part of it under each rule and label the whole report
+    `power`. test_kind is resolved per item, so a dataset does not have to be homogeneous.
+
+    Kinds no evaluator supports are not counted: those items are skipped rather than
+    decided, so refusing on them would make --gate power fail where --gate any runs.
+    """
+    if normalize_gate(config.gate) != "power":
+        return
+    supported = supported_test_kinds()
+    ungated = [
+        i
+        for i in items
+        if i.test_kind in UNGATED_AGENTIC_TEST_KINDS
+        or (i.test_kind not in AGENTIC_TEST_KINDS and i.test_kind in supported)
+    ]
+    if not ungated:
+        return
+    kinds = sorted({i.test_kind for i in ungated})
+    raise ValueError(
+        f"--gate power applies to kinds that repeat K runs, but this dataset has {len(ungated)} "
+        f"item(s) of kind {kinds}, which are always decided on pass@K. Run them separately, or "
+        f"use --gate any."
+    )
+
+
 def _warn_if_local_dataset_cannot_link(config: RunConfig, agentic_items: list) -> None:
     """Say up front that experiment assembly will fail, rather than after the run.
 
@@ -252,7 +300,7 @@ def on_item_done(index: int, total: int, report: ItemReport) -> None:
             tag = "[yellow]SKIP[/yellow]"
         elif report.error:
             tag = "[red]ERR [/red]"
-        elif report.pass_at_k:
+        elif report.passed:
             tag = "[green]PASS[/green]"
         else:
             tag = "[red]FAIL[/red]"
@@ -342,6 +390,7 @@ def _run(config: RunConfig) -> int:
     items = _load_dataset(config)
     agentic_items = [i for i in items if i.test_kind in AGENTIC_TEST_KINDS]
     non_agentic_items = [i for i in items if i.test_kind not in AGENTIC_TEST_KINDS]
+    _reject_power_gate_on_ungated_items(config, items)
     _warn_if_local_dataset_cannot_link(config, agentic_items)
     models = config.models or []
     run_ts = datetime.now(timezone.utc).strftime("%Y-%m-%d-%H-%M")
@@ -416,6 +465,7 @@ def on_langfuse_item_done(
                     token=config.token,
                     workspace_id=config.workspace_id,
                     k=config.runs,
+                    gate=config.gate,
                     model_version=resolved.model_id,
                     reasoning_effort=config.reasoning_effort,
                     use_langfuse=config.log_to_langfuse,
@@ -465,6 +515,7 @@ def on_langfuse_item_done(
                 provider_name=resolved.provider_name or resolved.provider_id,
                 provider_type=resolved.provider_type,
                 workspace_id=config.workspace_id,
+                gate=config.gate,
             )
             if agentic_report is not None:
                 report.items.extend(agentic_report.items)
@@ -529,6 +580,7 @@ def main(argv: list[str] | None = None) -> int:
             kind=args.kind,
             preserve_failed=args.preserve_failed,
             reasoning_effort=args.reasoning_effort,
+            gate=normalize_gate(args.gate),
             agent_id=args.agent_id or os.environ.get("GD_EVAL_AGENT_ID"),
         )
         return _run(config)