Repository navigation
Conversation
The eval ran on every PR commit touching docs or the TS SDK, then again on merge. At roughly $3 per run that was ~$165/month over the last 90 days and pacing ~$540/month in October. - Drop the pull_request trigger (and the now-unneeded fork guard) - Stop docs/agent-evaluation/** from triggering on main; dispatch manually instead - Document manual dispatch for pre-merge checks; scheduled drift monitoring stays in hookdeck/evals Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Copilot review overview
🟢 Approval recommended
The workflow and documentation changes are consistent and correctly implement the stated trigger policy.
Review effort: Balanced
Findings: None
What changed in this PR
Overview
Restricts automatic docs agent evaluations to relevant pushes on main, reducing duplicate paid runs while retaining manual branch evaluation.
Changes:
- Removes pull-request and harness-change triggers.
- Documents post-merge evaluation and manual dispatch.
| File | Description |
|---|---|
.github/workflows/docs-agent-eval-ci.yml |
Limits automatic evaluation triggers. |
docs/agent-evaluation/README.md |
Documents the revised CI behavior. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Post-merge runs have no PR to go red on, so failures would go unnoticed. On failure, open or comment on a single agent-eval-failure issue, assigned to a reviewer for triage. Mirrors the hookdeck/evals regression alert. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| # newer merge queues behind it, which is expected and not worth an alert. | ||
| eval-alert: | ||
| needs: [eval-ci] | ||
| if: ${{ failure() && github.event_name == 'push' }} |
There was a problem hiding this comment.
nit: nothing closes the agent-eval-failure issue when main goes green again, so it stays open until someone closes it by hand, and the next failure gets added as a comment on the old issue. A small step that closes the open issue when eval-ci succeeds on a push would handle it.
Why
The docs agent eval (
docs-agent-eval-ci.yml) ran on every PR commit that touched docs, OpenAPI or the TS SDK, then ran again on the merge tomain. Each run bills Anthropic for the agent and the LLM judge.total_cost_usdacross 11 recent runs, plus ~$0.25 for the judge)$500 ($165/month)Costs are the SDK's list-price figures, not taken from billing.
What changes
pull_requesttrigger, and with it the fork-PR guard, which can no longer fire.maintriggers: runs on pushes tomainthat changedocs/content/**,docs/apis/**,docs/README.md,docs/AGENTS.md,sdks/outpost-typescript/**or the workflow file. Based on the last 90 days that's ~14 runs/month, ~$44.docs/agent-evaluation/**no longer triggers a run. Check harness changes on their branch before merge withgh workflow run docs-agent-eval-ci.yml --ref <branch>.eval-alertjob runs only when the eval fails on a push tomain. It opens anagent-eval-failureissue assigned to @leggetter for triage, or comments on the one already open, so a failure that persists stays as one issue. It doesn't fire on cancellations, because the concurrency group cancels pending runs whenever a newer merge queues behind them. The commit message is passed in throughenvso it can't inject shell commands. This follows thehookdeck/evalsregression alert.Trade-off
This check now runs after merge. The website deploys on the same push, so a failure means the docs are already live and you fix forward.
Merging this PR triggers one run on
main, because it changes the workflow file.🤖 Generated with Claude Code