Add --agent flag to test runners for low-noise output - #14900
Merged
Merged
Conversation
A green run of the deno pretty reporter scales with test count and becomes expensive to drive from an AI coding session - 165 passing tests cost roughly 52KB and 14000 input tokens for a run whose only signal is the final tally line. --reporter=dot collapses that to roughly two bytes per test while preserving what a failure needs: assertion message, source frame, stack, and the harness-assembled rerun command all survive, since they're folded into the thrown failure message rather than captured stdout. Captured console output (progress prints outside the assertion message) is dropped for every test; the few places that was the only diagnostic are tracked separately and don't block this. The flag is explicit opt-in only - no environment sniffing, no TTY detection - so a human and an agent running the same command see the same output unless --agent is passed.
Covers the flag itself, the failure-recovery fallback chain (harness- printed rerun command first, smoke-all document-name caveat, direct- registration files, then original-command-minus-flag as the universal fallback), and the bash-only reporter-collision caveat with QUARTO_DENO_EXTRA_OPTIONS.
Two clauses pointed readers at things they cannot reach: a prior log-marker suppression incident that has no artifact anywhere in this repository, and "tracked as follow-up work" naming a tracker other contributors cannot read. A rationale nobody can check is worse than none, because it reads as load-bearing while being unfollowable, so it tends to get preserved rather than questioned. The dropped-output bullet now names a concrete consequence instead - a snapshot mismatch loses its printed unified diff, while the .diff file is still written to disk - which a reader can confirm in verify.ts. That bullet was also carrying four sentences, so it is split to match the surrounding style.
The duplicate-reporter check printed a literal version string in its PASS line while asserting nothing at all about the version, so the next bundled deno bump would have left it confidently reporting a version that was never exercised. Read it from the binary under test instead, and fall back to a placeholder rather than an empty string if the probe fails.
Collaborator
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
typescript-tests.md and smoke-all-tests.md predate the --agent flag and still showed plain run-tests examples; agents following those docs would miss the low-noise option documented in overview.md.
Rule doc is Claude's every-session lookup; --agent is the default invocation now, plain form and full flag list pushed to tests/README.md.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Driving the test suite from an AI coding session is expensive when a green run's only signal is the final tally line, because the deno pretty reporter's output scales with test count.
--agentswitches deno to its dot reporter, collapsing a green run to about two bytes per test plus the tally line. On the three-file, eight-test unit fixture used to measure this, console output drops from roughly 2KB to just over 200 bytes.Decisions
The flag is explicit opt-in, with no environment sniffing and no TTY detection, so a human and an agent running the same command see the same output unless
--agentis passed.deno's own
-q/--quietis never forwarded. It is a measured no-op under--reporter=dot, so carrying it would add a second meaning to the flag for no output benefit.The reporter's stdout is not post-filtered. Filtering would remove the only liveness signal on a long run and complicate exit-code plumbing, for no gain over what the dot reporter already gives.
The single
--reporter=dottoken goes after the import-map argument and before the first target. The position is not cosmetic: appending it to the end of argv would place it after the--separator that smoke-all document routing depends on, which breaks that routing silently.Passing
--agenttogether with a reporter already set inQUARTO_DENO_EXTRA_OPTIONSis unsupported. deno rejects duplicate--reporterarguments and exits non-zero; the wrapper does not detect the collision, it surfaces as a deno error. This can only arise on the bash runner, since the PowerShell one does not currently honour that variable at all.What a failure keeps
Assertion message, source frame, stack, exit code and the harness-assembled rerun command all survive, because they are part of the thrown failure message rather than captured stdout. What is dropped is captured console output, for every test, passing or failing. A snapshot mismatch is the notable loss: its unified diff is printed rather than thrown, though the
.difffile is still written to disk.tests/README.mddocuments the fallback chain for recovering from a failure under the flag, including the cases that do not get an assembled rerun command and the reason theFAILURESsummary location is not usable as a rerun target.Test plan
tests/run-tests-agent-flag.test.shis the automated contract for the bash runner, manual-only alongside its two existing siblings rather than wired into CI. It asserts the token appears exactly once and in the correct position across flag positions, composes with--keep-outputs, survives smoke-all document routing, never reaches the file-type classifier, and that real deno rejects the duplicate-reporter case end to end through the wrapper.ERRORSsection, assertion message, source frame, stack andFAILURESrestatement, with non-zero exit--keep-outputsartifact gating identical with and without--agent, checked on both runners against a real render/usr/bin/time, which is absent on the Windows dev host used here. The script declares the skip explicitly rather than passing vacuously, and runs them where it is available.No changelog entry: this is developer-facing test tooling with no user-visible behaviour change.
Checklist
I have (if applicable):
AI-assisted PR