Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 5 additions & 10 deletions .claude/rules/testing/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,24 +9,19 @@ Quarto's test suite lives in `tests/`. For comprehensive documentation, see `tes

## Running Tests

Use `--agent` — collapses a green run to a dot per test plus a tally line; failures keep assertion message, source frame, stack, exit code.

```bash
cd tests

# Linux/macOS
./run-tests.sh # All tests
./run-tests.sh smoke/render/render.test.ts # Specific test
./run-tests.sh docs/smoke-all/path/test.qmd # Smoke-all document
QUARTO_TESTS_NO_CONFIG="true" ./run-tests.sh --agent unit/my-test.test.ts

# Windows (PowerShell 7+)
.\run-tests.ps1
.\run-tests.ps1 smoke/render/render.test.ts
$env:QUARTO_TESTS_NO_CONFIG="true"; .\run-tests.ps1 --agent unit/my-test.test.ts
```

**Skip dependency configuration:**
```bash
QUARTO_TESTS_NO_CONFIG="true" ./run-tests.sh test.ts # Linux/macOS
$env:QUARTO_TESTS_NO_CONFIG=$true; .\run-tests.ps1 # Windows
```
Plain form (no `--agent`), full flag list, rerun-on-failure workflow, bash-only reporter-collision caveat: `tests/README.md`.

## Test Types

Expand Down
2 changes: 2 additions & 0 deletions .claude/rules/testing/smoke-all-tests.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,8 @@ Document-based tests using YAML metadata for verification. Tests live in `tests/
.\run-tests.ps1 docs/smoke-all/path/to/test.qmd
```

Add `--agent` for low-noise output — see `.claude/rules/testing/overview.md` § Running Tests.

## Test Structure

Tests are defined in `_quarto.tests` YAML metadata:
Expand Down
2 changes: 2 additions & 0 deletions .claude/rules/testing/typescript-tests.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@ TypeScript-based tests using Deno. Smoke tests render documents; unit tests veri
.\run-tests.ps1 smoke/render/render.test.ts
```

Add `--agent` for low-noise output — see `.claude/rules/testing/overview.md` § Running Tests.

## Core Infrastructure

Core test files (`test.ts`, `verify.ts`, `utils.ts`) are described in `.claude/rules/testing/overview.md` § Core Files.
Expand Down
23 changes: 23 additions & 0 deletions tests/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -224,6 +224,29 @@ $env:QUARTO_TEST_KEEP_OUTPUTS="true"
./run-tests.ps1
```

**--agent flag**
- Switches deno's reporter to `--reporter=dot`, collapsing a green run's output to roughly two bytes per test plus deno's tally line
- Failures keep what identifies them: assertion message, source frame, stack, exit code, and the harness-assembled rerun command (see below) all survive
- What's dropped is captured console output (progress prints outside the assertion message), for every test, passing or failing - a few places rely on that as their only diagnostic, notably a snapshot mismatch, whose unified diff is printed rather than thrown (the `.diff` file is still written to disk)
- Explicit opt-in only - no environment or TTY detection, so a human and an agent running the same command see the same output unless this flag is passed
- Not a general "quiet" flag: deno's own `-q`/`--quiet` is never forwarded (it's a no-op under `--reporter=dot`)

```bash
./run-tests.sh --agent unit/my-test.test.ts
```

```powershell
./run-tests.ps1 --agent unit/my-test.test.ts
```

*Recovering from a failure under `--agent`:* the failure message the harness prints already contains a ready-to-run rerun command for most test failures (tests registered through `unitTest`/`testQuartoCmd`, failing inside the render/verify step). A few cases don't get that assembled command and need a fallback:
- A **smoke-all** document failure: the printed command reruns the whole `smoke-all.test.ts` corpus rather than just the failing document - use the document path shown in the test name instead.
- A few files register directly with `Deno.test` rather than through the harness (`smoke/create/create.test.ts`, `smoke/logging/log-level-and-formats.test.ts`, `integration/playwright-tests.test.ts`) - for these the reported source location is the actual test file, but deno prints it as `path/to/file.test.ts:line:column`; strip the `:line:column` suffix before passing it back to the runner, since the runner's file-type check only accepts a path ending in `.ts`/`.qmd`/`.md`/`.ipynb` and rejects the location as printed.
- For anything else (a setup/teardown failure, or any failure that isn't Error-shaped), rerun the original command with `--agent` removed - this always works and needs no output parsing.
- Do **not** use the location in the `FAILURES` summary section as a rerun target for harness-registered tests - it resolves to `test.ts`'s own `Deno.test` call site, which registers no tests of its own and would rerun nothing.

*Reporter collision (bash only):* passing `--agent` together with a reporter already set via `QUARTO_DENO_EXTRA_OPTIONS` is unsupported - deno rejects duplicate `--reporter` arguments and exits non-zero. This combination is not detected or blocked by the wrapper; it surfaces as a deno error. (On PowerShell this combination cannot occur: `QUARTO_DENO_EXTRA_OPTIONS` isn't currently honored there at all.)

**Other environment variables**
- `QUARTO_TEST_VERBOSE` - Enable verbose test output
- `QUARTO_TESTS_NO_CHECK` - Not currently used (legacy variable)
Expand Down
Loading
Loading