diff --git a/src/assets/factories/benchmark-run-launch.png b/src/assets/factories/benchmark-run-launch.png new file mode 100644 index 000000000..939bad260 Binary files /dev/null and b/src/assets/factories/benchmark-run-launch.png differ diff --git a/src/assets/factories/benchmark-run-results-detail.png b/src/assets/factories/benchmark-run-results-detail.png new file mode 100644 index 000000000..b512fe67b Binary files /dev/null and b/src/assets/factories/benchmark-run-results-detail.png differ diff --git a/src/assets/factories/benchmark-run-results-overview.png b/src/assets/factories/benchmark-run-results-overview.png new file mode 100644 index 000000000..80772de2c Binary files /dev/null and b/src/assets/factories/benchmark-run-results-overview.png differ diff --git a/src/assets/factories/benchmark-suite-editor.png b/src/assets/factories/benchmark-suite-editor.png new file mode 100644 index 000000000..ceef1cd4d Binary files /dev/null and b/src/assets/factories/benchmark-suite-editor.png differ diff --git a/src/assets/factories/benchmark-suite-runs.png b/src/assets/factories/benchmark-suite-runs.png new file mode 100644 index 000000000..ea965e2e9 Binary files /dev/null and b/src/assets/factories/benchmark-suite-runs.png differ diff --git a/src/assets/factories/benchmark-task-editor.png b/src/assets/factories/benchmark-task-editor.png new file mode 100644 index 000000000..af4a892e9 Binary files /dev/null and b/src/assets/factories/benchmark-task-editor.png differ diff --git a/src/assets/factories/benchmark-task-source-run.png b/src/assets/factories/benchmark-task-source-run.png new file mode 100644 index 000000000..8f0833742 Binary files /dev/null and b/src/assets/factories/benchmark-task-source-run.png differ diff --git a/src/assets/factories/factory-benchmarks-list.png b/src/assets/factories/factory-benchmarks-list.png new file mode 100644 index 000000000..5221264cd Binary files /dev/null and b/src/assets/factories/factory-benchmarks-list.png differ diff --git a/src/content/docs/factories/benchmarks.mdx b/src/content/docs/factories/benchmarks.mdx new file mode 100644 index 000000000..865e77094 --- /dev/null +++ b/src/content/docs/factories/benchmarks.mdx @@ -0,0 +1,109 @@ +--- +title: Benchmarking factory agent configurations +description: >- + Benchmarks compare a factory agent's model and runner configurations on fixed + tasks. Use the results to choose a production configuration. +sidebar: + label: "Benchmarks" +--- +import { VARS } from '@data/vars'; +import VideoEmbed from '@components/VideoEmbed.astro'; + +:::note +Warp Factories is in **Early Access** and available to a limited set of teams. [Request access](https://www.warp.dev/factories/request-access) to use it with your team. +::: + +Benchmarks compare model and runner configurations for one factory agent on the same fixed tasks. Use a benchmark to test a change on representative work before you apply it to your factory. +This video shows how to compare benchmark results for quality and cost before changing a coding agent's configuration. + + +## How benchmarks work + +A benchmark helps you choose an agent configuration based on repeatable evidence. Run representative tasks with different configurations, then compare their results before you change a live factory. + +A benchmark suite is a reusable collection of tasks that evaluates one factory agent. Each task has a prompt and "Correctness criteria," which tell the built-in Correctness Scorer what a successful trial must do. A trial is a single run of a task under one configuration. Repetitions create additional trials. + +When you launch a suite, choose the configurations, Scorers, and repetitions to compare. Warp runs the trials, then scores the completed ones. The run keeps those inputs, so later changes to the suite do not change its past results. + +
+![The Runs tab for a benchmark suite, with task, configuration, run time, and cost columns.](../../../assets/factories/benchmark-suite-runs.png) +
The Runs tab for a benchmark suite.
+
+ +Use separate suites for focused questions: + +* **Factory default** - Compare models and runners to choose the default configuration for a coding agent. +* **Frontend changes** - Compare configurations on representative frontend tasks before applying one to that workflow. + +## Create and run a benchmark + +To use Benchmarks, you need a factory with an agent to evaluate. Use a completed run from that agent when you want a task to reproduce real work, or write a task yourself. + +This video shows how to turn your team's coding tasks into a reusable benchmark suite. + +1. In the {VARS.FACTORY_WEB_APP}, open your factory, click **Benchmarks**, then click **New**. + +
+ ![The Benchmarks page showing saved benchmark suites, a search field, and the New button.](../../../assets/factories/factory-benchmarks-list.png) +
The Benchmarks page for a factory.
+
+2. Enter a name and optional description, then choose the agent to evaluate. The suite runs every task as that agent. + +
+ ![The benchmark editor showing a documentation-link review suite, a selected triage agent, and the Add task action.](../../../assets/factories/benchmark-suite-editor.png) +
The benchmark editor with a selected agent.
+
+ +3. Click **Add task**. Warp saves the benchmark, then opens task setup. +4. Select a completed run, then click **Add task**. To write a task instead, click **Start from scratch instead**. + +
+ ![The Add task pane with a run search field, the list of completed runs, and the Start from scratch instead option.](../../../assets/factories/benchmark-task-source-run.png) +
The task source picker.
+
+5. Review the task prompt and enter "Correctness criteria" for the task. + +
+ ![The task editor with fields for a title, prompt, correctness criteria, and pinned repositories.](../../../assets/factories/benchmark-task-editor.png) +
The task editor for a benchmark suite.
+
+6. Click **Run**. In the launch dialog, choose the model and runner for each configuration. Optionally mark one configuration as the baseline. Third-party harness comparisons are not available yet. +7. Add configurations, select Scorers, and set "Repetitions." The dialog shows the number of trials created. More trials and Scorers increase the run's cost. + +
+ ![The New benchmark run dialog with model and runner configurations, a baseline option, selected Scorers, a repetition count, and projected trial count.](../../../assets/factories/benchmark-run-launch.png) +
The launch configuration for a benchmark run.
+
+8. Click **Run benchmark**. The benchmark page shows its status and scored trials. You can cancel a running or scoring benchmark. + +## Review benchmark results + +After the run completes, review the result as a comparison, not as a universal model ranking: +
+![A completed benchmark result showing the overall recommendation, additional recommendations, total cost, and a correctness versus average cost chart.](../../../assets/factories/benchmark-run-results-overview.png) +
A completed benchmark result and comparison chart.
+
+ +* **Overall recommendation** - Identifies the highest-quality configuration when at least two configurations have comparable results. +* **Additional recommendations** - Highlight the most efficient and lowest-cost configurations when the result supports those comparisons. +* **Comparison chart** - Compare the selected result dimensions across configurations. +* **Overall table** - Compare each Scorer's average and the combined Overall value. Expand a configuration, task, and repetition to inspect its individual trials. +* **Scorer grids** - Show each task's results across configurations for a selected Scorer. +
+![Expanded benchmark results showing trial run duration, cost, and scores for a configuration.](../../../assets/factories/benchmark-run-results-detail.png) +
Expanded trial results for a configuration.
+
+ +The run's "Total cost" includes model usage for trials and Scorer usage. It estimates those costs from credits at your team's current rate, so it is not a billed amount. A failed or cancelled benchmark shows only results that finished scoring before the run stopped. + +## Apply a result + +Change one configuration at a time. If the evidence supports a candidate, update the agent's model or runner in the factory dashboard, or submit the change through your [factory definition](/factories/factory-as-code/). Keep the relevant Scorers active, then compare later production runs with the baseline you recorded before the change. + +For version-controlled factories, define reusable suites in `benchmarks//suite.yaml` and their tasks in `benchmarks//tasks/.yaml`. See [benchmark suite files](/factories/factory-as-code/#benchmarkssuite-slugsuiteyaml). + +## Related pages + +* [Measure and improve a factory](/factories/measure-and-improve/) - Configure Scorers and use benchmark evidence in an improvement loop. +* [Factory dashboard](/factories/factory-dashboard/) - Track factory work, runs, and benchmark suites. +* [Factory definition syntax](/factories/factory-as-code/) - Define factories and benchmark suites as code. diff --git a/src/content/docs/factories/factory-as-code.mdx b/src/content/docs/factories/factory-as-code.mdx index ad088d5fc..9340a0cae 100644 --- a/src/content/docs/factories/factory-as-code.mdx +++ b/src/content/docs/factories/factory-as-code.mdx @@ -2,13 +2,13 @@ title: Factory definition syntax description: >- Look up every file and key in a factory definition: factory.yaml, agents, - automations, runners, scorers, skills, and webhooks. + automations, runners, benchmarks, scorers, skills, and webhooks. sidebar: label: "Definitions as code" --- import { VARS } from '@data/vars'; -Every factory is defined by files: a `factory.yaml` plus directories of agents, automations, runners, scorers, skills, and webhooks, versioned in a Git repository. The files are the source of truth — when they change, Warp updates the factory to match. This page describes every file and key in a definition. +Every factory is defined by files: a `factory.yaml` plus directories of agents, automations, runners, benchmarks, scorers, skills, and webhooks, versioned in a Git repository. The files are the source of truth — when they change, Warp updates the factory to match. Definition files are YAML and Markdown. Keys are case-sensitive. @@ -46,6 +46,11 @@ automations/ automation.md runners/ linux-build.yaml +benchmarks/ + pull-request-review/ + suite.yaml + tasks/ + broken-doc-link.yaml scorers/ tests-run/ scorer.md @@ -337,6 +342,66 @@ Optional. The compute size, as `vcpus` and `memoryGb`. Omit it to take the works The operating system and architecture. `os` is `linux` (the default) or `macos`, and `arch` is `x86_64` (the default on Linux) or `aarch64` (the only option on macOS). Linux runners require `linux.dockerImage`, the container image the sandbox boots, so every Linux runner declares a `platform` section. macOS runners accept an optional `mac.version` (`"14"`, `"15"`, `"26"`, or `"27"`; quote it, and it defaults to `"26"`). +## `benchmarks//suite.yaml` + +Optional. This file defines a [benchmark suite](/factories/benchmarks/): one agent and an ordered set of reusable tasks. The suite slug is its stable file location, not its display name, so renaming `name` does not move the file or change the suite's identity. + +```yaml title="benchmarks/pull-request-review/suite.yaml" +name: Pull request review +description: Compare configurations for the review agent. +agent: reviewer +tasks: + - broken-doc-link +``` + +### `name` + +Required. The suite display name. It must be unique in the factory. + +### `description` + +Optional. A short summary of what the suite measures. + +### `agent` + +Required. The name of an agent declared by the factory. Every task in the suite runs as this agent. + +### `tasks` + +Optional. An ordered list of unique task slugs. Each slug must match exactly one file under `benchmarks//tasks/`, and every task file must appear once in this list. A suite without tasks saves successfully but cannot run. + +## `benchmarks//tasks/.yaml` + +Optional. Each file defines one task in its parent suite. The task slug is the file name without `.yaml` and stays stable when you change the task's title. + +```yaml title="benchmarks/pull-request-review/tasks/broken-doc-link.yaml" +title: Fix a broken documentation link +prompt: Find the broken internal documentation link and update it. +successCriteria: The destination resolves and the link text names the destination. +startingRepoRefs: + - github.com:acme/payments-docs@0123456789abcdef0123456789abcdef01234567 +``` + +### `title` + +Required. The task display name. It does not need to match the task slug. + +### `prompt` + +Required. The instructions the agent receives for the task. + +### `successCriteria` + +Required. The requirements the Correctness Scorer uses to evaluate each trial. + +### `sourceRunId` + +Optional. The ID of the prior run the task came from. This records provenance only and can refer to a deleted run. + +### `startingRepoRefs` + +Optional. The GitHub or GitLab repositories and exact commits the task starts from. Use either `github.com:OWNER/REPO@COMMIT_SHA` or `gitlab.com:OWNER/REPO@COMMIT_SHA`; `COMMIT_SHA` must be a full 40-character commit SHA, not a branch or tag. Omit this key to use the agent's normal checkout defaults. + ## `scorers//scorer.md` Optional. Each file defines a scorer: an LLM judge that classifies a sample of an agent's finished runs against a rubric. The directory segment is only a stable filesystem slug — the required `name` field is the scorer's identity. The YAML frontmatter declares the classification contract, and the Markdown body after the closing `---` fence is the rubric. See [Configure Scorers](/factories/measure-and-improve/#configure-scorers) for how scores are used. diff --git a/src/content/docs/factories/factory-dashboard.mdx b/src/content/docs/factories/factory-dashboard.mdx index f7b68adff..649198ae9 100644 --- a/src/content/docs/factories/factory-dashboard.mdx +++ b/src/content/docs/factories/factory-dashboard.mdx @@ -88,7 +88,7 @@ When an agent proposes a change to a Warp-managed definition, its work item on * ## Score and benchmark -**Scorers** is where you create Scorers and read their results. **Self-improvement** lists the pull requests the self-improvement flow opens after analyzing runs your Scorers mark as failing, and **Benchmarks** compares harness, model, and runner configurations against a fixed set of tasks. See [Configure Scorers](/factories/measure-and-improve/#configure-scorers), [Configure and review Self-improvement](/factories/measure-and-improve/#configure-and-review-self-improvement), and [Compare configurations with benchmarks](/factories/measure-and-improve/#compare-configurations-with-benchmarks) for what each one is and how to set it up. +**Scorers** is where you create Scorers and read their results. **Self-improvement** lists the pull requests the self-improvement flow opens after analyzing runs your Scorers mark as failing, and **Benchmarks** compares model and runner configurations against a fixed set of tasks. See [Configure Scorers](/factories/measure-and-improve/#configure-scorers), [Configure and review Self-improvement](/factories/measure-and-improve/#configure-and-review-self-improvement), and [benchmarking factory agent configurations](/factories/benchmarks/) for what each one is and how to use it. ## Change factory settings @@ -109,5 +109,6 @@ For a file-managed factory, `runners/*.yaml` in the repository is the source of * [How Warp Factories work](/factories/how-factories-work/) - The lifecycle behind Activity's stages and where humans stay in the loop. * [Definitions as code](/factories/factory-as-code/) - Define agents, automations, runners, and source ownership in code. * [Factory agents](/factories/factory-agents/) - What each default agent does and how to configure it. -* [Measure and improve a factory](/factories/measure-and-improve/) - Configure the Scorers and benchmarks behind the **Dashboard** page. +* [Measure and improve a factory](/factories/measure-and-improve/) - Configure Scorers and use them in an improvement loop. +* [Benchmarking factory agent configurations](/factories/benchmarks/) - Create a benchmark suite and compare configurations on fixed tasks. * [Troubleshooting Warp Factories](/factories/troubleshooting/) - Fixes for setup problems, work that doesn't start, and stuck runs. diff --git a/src/content/docs/factories/index.mdx b/src/content/docs/factories/index.mdx index 40cc08918..67f74eaaa 100644 --- a/src/content/docs/factories/index.mdx +++ b/src/content/docs/factories/index.mdx @@ -52,7 +52,7 @@ Warp Factories is designed for engineering teams with repeatable work that exten * **Definitions as code** - [Version-controlled definition files](/factories/factory-as-code/) describe your repositories, agents, automations, runners, [skills](/factories/factory-skills/), and MCP servers, so factory changes get the same review, history, and rollback as code changes. * **Integrations and the Factory MCP** - Work flows in from [Slack](/factories/integrations/slack/), [GitHub](/factories/integrations/github/), [GitLab](/factories/integrations/gitlab/), [Linear](/factories/integrations/linear/), and [Jira](/factories/integrations/jira/), plus [custom webhooks](/factories/webhooks/), direct runs, and schedules. The [Factory MCP](/factories/factory-mcp/) connects coding agents and other MCP clients. * **Model and harness choice** - Each agent can use a different model and [supported harness](/platform/harnesses/), including the Warp Agent, Claude Code, and Codex. -* **Measurement and self-improvement** - The [factory dashboard](/factories/factory-dashboard/) shows work-item status, runs, automations, costs, and benchmarks. [Scorers](/factories/measure-and-improve/) classify completed runs, and [Self-improvement](/factories/measure-and-improve/#configure-and-review-self-improvement) turns repeated failures into follow-up work the factory proposes for review. +* **Measurement and self-improvement** - The [factory dashboard](/factories/factory-dashboard/) shows work-item status, runs, automations, costs, and benchmarks. [Scorers](/factories/measure-and-improve/) classify completed runs, [Benchmarks](/factories/benchmarks/) compare fixed tasks across configurations, and [Self-improvement](/factories/measure-and-improve/#configure-and-review-self-improvement) turns repeated failures into follow-up work the factory proposes for review. * **Infrastructure control** - Run on Warp-hosted infrastructure, or self-host execution on an eligible Enterprise plan. Teams can also connect supported inference providers, scope secrets, and (if eligible) store transcripts, artifacts, and run attachments in their own S3 or GCS buckets. See [infrastructure and security](/factories/infrastructure-and-security/) for the available controls. ## How Warp Factories relates to other Warp products diff --git a/src/content/docs/factories/measure-and-improve.mdx b/src/content/docs/factories/measure-and-improve.mdx index aec5a5416..ea32b2f10 100644 --- a/src/content/docs/factories/measure-and-improve.mdx +++ b/src/content/docs/factories/measure-and-improve.mdx @@ -63,17 +63,7 @@ Changing **Pass threshold** updates how past scores display as pass or fail; the ## Compare configurations with benchmarks -A benchmark compares configurations of a single agent on the same fixed tasks, so you can test a model, harness, or runner change before adopting it. A benchmark suite includes: - -* **Agent** - The agent whose configurations you compare. -* **Tasks** - Fixed prompts with success criteria. -* **Configurations** - The harness, model, and runner combinations to test. -* **Scorers** - Your classification Scorers, applied to every trial. -* **Repetitions** - The number of trials per task and configuration. - -You can create a benchmark task from a completed run's detail pane, and Warp copies the run's input into the task. Add success criteria before you launch. - -Every benchmark also runs **Correctness**, a built-in Scorer that marks each trial as pass or fail against the task's success criteria. Results show pass rates, cost, and quality for each configuration, with per-task detail. Warp doesn't combine these signals into one score or pick a winner; you weigh the results and decide. Benchmark credit totals don't include model usage, so the true cost is higher. +A benchmark compares model and runner configurations for a single agent on the same fixed tasks. Use it to test a configuration change before you apply it to production. See [benchmarking factory agent configurations](/factories/benchmarks/) for the workflow. ## Configure and review Self-improvement diff --git a/src/sidebar.ts b/src/sidebar.ts index 5ab6218b4..6b45e1ad3 100644 --- a/src/sidebar.ts +++ b/src/sidebar.ts @@ -457,6 +457,7 @@ export const sidebarTopics: StarlightSidebarTopicsUserConfig = [ items: [ { slug: 'factories/factory-dashboard', label: 'Factory dashboard' }, { slug: 'factories/measure-and-improve', label: 'Measure and improve' }, + { slug: 'factories/benchmarks', label: 'Benchmarks' }, ], }, // Troubleshooting sits outside the groups, last in the tab. It was in