[SDTEST-3895] Generate HTML testdrive reports

[anmarchenko]

What

Only flag slow tests when their duration is at least five times the median and strictly greater than five seconds. Generate a static HTML testdrive report with findings, runtime facts with a “Datadog library” label, missing-coverage status, and links to captured traffic and test output. Show a clickable report link with a relative path. Keep the preview concise, use relative output paths, and name only dependency files found for the selected language. Missing events and tracer errors remain visible separately from application failures. Failed commands show captured output and the relative path to the complete log; CLI errors print once.

Part 7/14 of the replacement stack for #128. Previous: #139 (merged). Next: #141.

Why

A persistent report makes local validation reviewable after the command exits.

E2E testing

Use this PR’s checkout or its prerelease. When building from source, use Go 1.27.1 (as pinned in go.mod) and run go build -o /tmp/ddtest-qa . from the DDTest root. Commands below use /tmp/ddtest-qa.

Prerequisites: a disposable Git/Jest project with installed dependencies, Node.js 22+, and npm/network access. No Datadog credentials are required.

1. Run /tmp/ddtest-qa testdrive --yes with passing tests. Open the printed report link in a browser. Expect an HTML report showing the same run outcome and test/coverage counts as the terminal.
2. Follow links to decoded traffic and complete test output. Expect existing files belonging to this exact session; repeat from a project path containing spaces. On Windows, verify the relative report link opens the correct local file from both a drive path and a UNC share. The underlying hyperlink target uses file:///C:/... or file://server/share/..., respectively. On Unix, a literal backslash in the report directory must be encoded as %5C, not converted to a directory separator.
3. Break an assertion and rerun. Open the new report: expect the test failure and successful instrumentation to be shown separately. The previous session’s report must remain intact.
4. Run /tmp/ddtest-qa testdrive --yes --command 'node -e "console.log(123)"'. Open the generated report: expect “No test events received”, “No test results received”, and missing coverage. Neither “Passed” nor “No findings” may appear.
5. Disconnect networking after report generation and reopen the local report. Expect the report to render without remote assets. Restore the project and remove temporary sessions after review.

Additional reporting check: in the disposable Jest project add this test, then run testdrive. Expect “Test events received.” and a separate “Tracer configuration errors: skippable_tests” diagnostic. Add a failing assertion in another test and rerun: the configuration error must remain visible alongside the test failure. Receiving events must not produce a “Test Optimization is ready” claim. Remove this fixture afterward.

const { createRequire } = require('node:module');
const preload = process.env.NODE_OPTIONS.match(/-r "([^"]+)"/)[1];
const tracer = createRequire(preload)('dd-trace');
test('diagnostic fixture', () => {
  tracer.scope().active().setTag('_dd.ci.library_configuration_error.skippable_tests', 'true');
});

Check the same diagnostic in the HTML report, alongside any test-failure findings.

Additional tracer-error scenario: in the disposable Jest project above, retain sum.js and add tracer-error.test.js with this content. Use Node.js 22+ so fetch, FormData, and Blob are available.

const sum = require('./sum');
test('exposes empty coverage as a tracer error', async () => {
  expect(sum(1, 2)).toBe(3);
  const payload = Buffer.from('81a9636f766572616765739183af746573745f73657373696f6e5f69640aad746573745f73756974655f696414a566696c657390', 'hex');
  const form = new FormData();
  form.append('coverage1', new Blob([payload], {type: 'application/x-msgpack'}), 'coverage.msgpack');
  const response = await fetch(`${process.env.DD_CIVISIBILITY_AGENTLESS_URL}/api/v2/citestcov`, {method:'POST', body:form});
  expect(response.status).toBe(200);
});

Build this PR’s binary and run /tmp/ddtest-qa testdrive --yes from the project. Expect the test to pass and the output to contain Tracer error: received with a positive count of coverage entries with empty files. The count can exceed one if the tracer retries the test. The output must explain that affected payloads were excluded and point to captured traffic; it must not say No findings.. Inspect the intake JSON artifacts and confirm the empty files list was preserved.
Open the generated report and expect a card titled Tracer error: empty coverage entries, with the count and exclusion explanation. The tracer error must remain visible even if the real tracer also supplied valid coverage.
Remove the injected test and disposable project/session artifacts after inspection.

Datadog-library label check: run once without a project tracer, then install dd-trace in the disposable project and rerun. Expect the run-details label Datadog library in both terminal and HTML, with one isolated suffix for fallback and one reused suffix for the installed library. The reused tracer must not also be labelled isolated. Restore the disposable project afterward.

Finding-count check: run two failing tests and expect 2 findings. in both terminal and report, then leave only one failing test and expect 1 finding.. Combine an application failure with the configuration diagnostic and injected empty-coverage test above: expect both summaries to count the individual failures, configuration errors, and empty entries. A diagnostic must remain visible even if no ordinary test events arrive.

Output-preview check: run ddtest testdrive from the disposable project before confirming. Expect only the output-folder, tracer action, and run-command lines; the old “save a clickable report” and “save captured traffic” lines must be absent. Folder paths and the installation prefix must start with .testoptimization/, not the full project path. After running, the visible “Open report” path must also be relative and its clickable target must still open the saved report. Repeat from a project directory containing spaces.

Dependency-file preview check: in the disposable Jest project, run ddtest testdrive --platform javascript --framework jest before confirming. With only package.json, expect It will not change package.json.. Add an empty pnpm-lock.yaml and expect that filename in the notice alongside the manifest and any other existing JavaScript lockfiles. Add unrelated Gemfile and pyproject.toml files and repeat with the same explicit selection: those names must not appear in the Jest notice. Remove the temporary files afterward.

Failure-output check: break a Jest assertion and run /tmp/ddtest-qa testdrive --yes. Expect assertion details and a source location under Jest command output, a relative Full test output path, and one final Error: line without a duplicate FAILURE log. Read the printed file and verify it contains the complete stdout/stderr. For output over 80 lines, expect the first and last 40 lines with an omission count; the saved file must remain complete. Restore the assertion, then add require('./missing-setup') at the top of a test and repeat: expect the missing-module diagnostic even if no test events arrive. Remove that statement and run with --command 'node -e "process.exit(7)"': expect an explicit no-output message, one error line, and shell exit status 7. Restore the normal command and passing tests; expect no command-output excerpt. Clean up the disposable project and sessions.

Slow-finding check: in the disposable Jest project, create three tests that each wait 100 ms, one that waits 4 seconds, and one that waits 6 seconds (give the latter an explicit 15-second Jest timeout). Run testdrive and inspect both the terminal and HTML slow-test findings: only the 6-second test should appear. Then increase the three baseline waits to 2 seconds and repeat: the 6-second test must no longer appear because it is below five times the median. Restore the fixtures and remove their session artifacts afterward.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-22T11:18:41.797302Z	9ac695d	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9ac695ddc6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[datadog-prod-us1-6]

On Windows, report.go line 151 assigns the raw OS path directly to url.URL.Path, producing an invalid file URL with encoded backslashes (e.g. file://C:%5Crepo%5Creport.html) instead of the required file:///C:/repo/report.html form, breaking the clickable report link for all Windows users.

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit 9ac695d · @DataDog review to ask questions}

[datadog-prod-us1-6]

 

✅ All CI checks and tests passed.

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
• Patch Coverage: 95.14%
• Overall Coverage: 91.98% (+0.05%)

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 93dced6 | Docs | View more details | Give us feedback!}

[datadog-prod-us1-6]

The HTML report misrepresents no-event runs and tracer installation mode, and its summary counts categories rather than the underlying findings.

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit 1b10c13 · @DataDog review to ask questions}

[anmarchenko]

Addressed all five current review threads in signed commits 68a4235 and 925c772. GitHub verifies both signatures. The report now preserves no-event status, renders the supplied tracer installation mode, counts individual findings consistently with the terminal, and preserves literal Unix backslashes while retaining Windows drive/UNC URL handling.

Validation on 925c77210e9dbf470bca4f5bbf3f5cf2817e950c:

• make test and make lint passed using the module-pinned Go 1.27.1. Focused regressions cover the reviewed behavior, including no-event diagnostics and mixed finding categories.
• Real Node 22.14/Jest 30.2/dd-trace 6.17 runs: passing exit 0; failing assertion exit 1; combined configuration/empty-coverage/application errors exit 1; no-event command exit 1. Generated HTML and terminal agree: one application failure is 1 finding.; one application failure + two empty entries + one configuration error is 4 findings.. Valid coverage remains visible alongside those errors.
• Inspected generated HTML contents, escaped test names, encoded report URLs and existing artifact targets from a path containing spaces. Both fallback and reused tracer labels were checked.
• Browser visual QA remains unverified because Computer Use was not approved to access Chrome. Native Windows browser behavior was not exercised. These runtime/content checks are not a substitute for that manual browser step.

Propagated to #141–147 with explicit leases after test/lint on every final tip, preserving their logical changes. All 37 new or rewritten commits are GitHub Verified. #152 remains untouched; its owner needs to rebase onto #147 at 55e119b9d2f0c0b87072a45eac5330dcad9a71f0.

Final remote validation: all exact-head checks are green: #140 has 45/45; each of #141–147 has 44/44. #144 required one retry after pip reported no available ddtrace distribution before tests ran; that same-head retry passed. Human approval and browser visual QA remain outstanding.

[anmarchenko]

@autotest review

[datadog-prod-us1-6]

More details

Report generation, artifact links, platform-specific file URLs, and testdrive outcome integration had no reportable static defects.

Was this helpful? React 👍 or 👎

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit 925c772 · @DataDog review to ask questions}

[anmarchenko]

Simplified the testdrive output in signed commit bf83efa48c7d806b2136d76b0ff14bef7a36c2eb:

• Removed the preview lines for saving the report and captured traffic.
• Made the output-folder and installation-command session paths relative to the current folder.
• Made the final report-link label relative while retaining its valid absolute file-URL target.

Existing regressions now cover the simpler preview, a project path containing spaces, unchanged installation arguments, and the relative report-link label. make test and make lint passed on #140 and all propagated #141–147 heads. Real CLI preview/run checks passed. GitHub verifies all 36 new or rewritten signatures, and #140 has 45/45 successful checks on this exact head. All downstream exact-head checks have now passed as well: 44/44 on each of #141–147. #152 was not changed.

[anmarchenko]

Validation for c875aed:

• Preview now names only dependency files discovered for the selected language. Real Jest/pnpm preview: It will not change package.json or pnpm-lock.yaml. Unrelated Gemfile and pyproject.toml files were omitted.
• Regression coverage checks manifests, npm/pnpm/yarn/bun locks, Python manifests and requirements files, Ruby reuse, mixed-language directories, no discovered files, directory entries, and repository paths containing glob metacharacters.
• make test and make lint passed. All 45 exact-head GitHub checks passed.
• Changes propagated through [SDTEST-3895] Browse tests, retries, coverages in testdirve reports #141–147 with signed commits and explicit leases; each final tip passed make test and make lint. Downstream CI is running. GitHub verified all 36 published commits (verified=true, reason=valid). Prototype Jest onboarding validation and agent evaluations #152 was unchanged.
• Real downstream CLI previews confirmed It will not change pyproject.toml or uv.lock. for pytest/uv and Bundler updates Gemfile and Gemfile.lock. for Ruby installation. These are terminal preview checks only.

[anmarchenko]

Failure diagnostics are published in signed commit 1615d584d28b99285b885a38da5f72cb5884c4ef:

• Failed test commands now show captured stdout/stderr and a relative path to the complete test-output.txt. Logs longer than 80 lines show the first and last 40 lines with an explicit omission count. The saved log remains byte-for-byte complete. Empty output is stated explicitly.
• Removed the duplicate slog failure message already printed by Cobra. The original process exit code and signal handling remain intact. This is the same one-line removal proposed in Make CLI error output concise and consistent #133; that branch was not changed.
• make test and make lint passed. Regression coverage includes assertions, setup errors without failed-test events, no-event startup failures, empty output, the 80-line boundary, truncated display with complete saved logs, relative paths containing spaces, and single CLI error printing.
• Real CLI runs in a disposable Jest project showed assertion expected/received values and source location, missing-module diagnostics during suite setup, startup error text with exit 7, explicit empty output with exit 7, and quiet successful output with exit 0. Every failed run printed only one final Error: line. These are terminal checks.

Propagated through #141–147 with explicit leases; all eight final tips passed make test and make lint. GitHub verifies all 36 new/rewritten commits (verified=true, reason=valid). Logical downstream diffs are unchanged and #152 was untouched. #140 now has all 45 exact-head checks passing. Each of #141–147 has 42 successful checks and its final test job pending, with no failures observed.

[anmarchenko]

Changed slow-test findings in the shared intake analyzer in signed commit 93dced666e6a3d7903ec9958f8df422d702c98c1. A test must now satisfy both duration > 5 seconds and duration >= 5 × median. Terminal and HTML consume the same filtered findings.

make test and make lint passed. Boundary regressions cover sub-second outliers, exactly five seconds, one nanosecond above five seconds, one nanosecond below five times the median, exactly five times the median, above the ratio, and uniformly slow suites. Existing failure/retry and coverage-preservation fixtures were adjusted to retain a qualifying slow test. The PR’s manual QA plan includes both an absolute-threshold and a relative-threshold scenario.

Propagated through #141–147 with all logical downstream diffs preserved. All eight final tips passed make test and make lint; GitHub verified all 36 new/rewritten commits. #152 was untouched. All exact-head CI checks passed: #140 has 45/45 and each of #141–147 has 44/44.

[anmarchenko]

E2E Test Report: PASS with environment limits

Tested by: Shepherd Agent (autonomous QA for Datadog Test Optimization)
Environment: macOS arm64, Go 1.27.1, Node.js 22.14.0, Jest 30.2.0, dd-trace 6.15.0, disposable local Git/Jest project
Revision tested: 93dced666e6a3d7903ec9958f8df422d702c98c1 (PR #140 head, rechecked after testing)

Results

Check	Result
Passing Jest run and HTML report	Pass: 2 test events, 1/1 tests with coverage, Jest: Passed, No findings. in terminal and HTML.
Failing assertions and output	Pass: one failed test produced 1 finding.; two failed tests produced 2 findings.. Jest assertion/source details appeared under Jest command output, the complete output was saved, and the CLI printed one final Error: line.
Missing test events	Pass: a successful node -e command produced No test events received, No test results received, and missing coverage in the HTML report, with no Passed or No findings claim.
Configuration diagnostic	Pass: skippable_tests remained visible with passing tests and alongside failures, with terminal/HTML finding counts. No readiness claim appeared.
Empty coverage entries	Pass: injected files: [] payloads remained in captured intake JSON; terminal and HTML reported two empty entries as a tracer error, excluded them from coverage counts, and kept the error visible alongside valid coverage.
Combined findings	Pass: two failed tests, one configuration error, and two empty coverage entries produced 5 findings. in terminal and HTML.
Slow-test threshold	Pass: only the six-second test was flagged when the median was about 102 ms; it was not flagged when baseline tests took two seconds. The four-second test was never flagged.
Tracer label	Pass: the fallback tracer was labelled Datadog library: dd-trace@6.15.0 · isolated; the project-installed tracer was labelled reused once, in terminal and HTML.
Preview and dependency files	Pass: the preview showed relative output/install paths and only the three intended action lines. It named existing JavaScript files, including a temporary pnpm-lock.yaml, without naming unrelated Gemfile or pyproject.toml files.
No-output and setup failures	Pass: a silent command preserved shell exit status 7, showed an explicit no-output message and one error line. A missing-module failure showed the Jest diagnostic even with zero test events.
Long command output	Pass: the terminal showed first/last 40 lines with an omission count; the saved 1,047-line output retained both copies of lines 001–100 from tracer retries.
Report/artifact URLs	Pass: all 15 generated report URLs resolved to their session's existing HTML, intake/, and test-output.txt. Reports had inline styling and no remote asset references. Paths with spaces were encoded; a Unix literal backslash was encoded as %5C. Earlier session reports remained intact.
Source checks	Pass: make test and make lint (including fmt, vet, golangci-lint: 0 issues).

Limits and issue observed

• A full Jest execution from a Unix directory containing a literal backslash failed in Node.js/dd-trace before tests started: Node rejected the encoded %5C path to dc-polyfill.js with ERR_INVALID_MODULE_SPECIFIER. DDTest still generated a valid report and correctly encoded %5C in its file URL. This is a dependency/runtime path limitation encountered during the PR's backslash scenario; it did not reproduce in the directory containing spaces.
• Windows drive and UNC URLs were covered by the PR's Go tests, but were not manually opened on a Windows host. The connected browser list was empty and computer-use access to Chrome was not approved, so visual rendering, link clicks, and reopening after disconnecting network were not exercised. HTML links and absence of remote assets were checked on disk.
• This PR changes local testdrive reporting and intake analysis, not the real backend API client. Testdrive uses a local intake, so there was no Datadog backend submission or Datadog UI verification for this change.

Method: Built the PR head from source, ran the described Jest scenarios against a disposable project, compared terminal summaries with each generated report, inspected captured intake JSON and output files, and ran the Go test and lint gates. The fixture was restored; generated session artifacts and logs were validated before cleanup.

---

This E2E test was performed by Shepherd, autonomous QA for Datadog Test Optimization.