[SDTEST-3895] Run a complete local Jest testdrive

[anmarchenko]

What

Run a complete local Jest testdrive with preview and confirmation, project-tracer reuse or isolated fallback, console findings, and saved traffic and test output. Report tracer errors, preserve test-command exit statuses, and omit CLI usage for runtime failures.

Part 6/14 of the replacement stack for #128. Previous: #138 (merged). Next: #140.

Why

This completes the first executable local journey using the independently tested detection and intake components. The selected Platform prepares the tracer; testdrive passes the chosen version and test command and applies its installation environment.

E2E testing

Use this PR’s checkout or its prerelease. When building from source, use Go 1.27.1 (as pinned in go.mod) and run go build -o /tmp/ddtest-qa . from the DDTest root. Commands below use /tmp/ddtest-qa.

Prerequisites: Node.js 22+, npm/network access, and a disposable Git repository with Jest installed, initially no dd-trace, and a passing test. No Datadog credentials or Agent are needed. Record manifest/lockfile checksums.

1. Run /tmp/ddtest-qa testdrive interactively. Review the detected Jest framework and planned installation and test commands, then decline. Expect no tracer installation, test execution, or new session artifacts.
2. Run it again and accept; also try /tmp/ddtest-qa testdrive --yes. Expect a unique session, a passing Jest result, nonzero received test events, honest coverage counts, decoded intake/*.json, and test-output.txt. Manifest/lockfile checksums must remain unchanged. HTML reports are not part of this PR yet.
3. Break one assertion and repeat with --yes. Check echo $?: expect nonzero. The output must distinguish failed tests from successfully received instrumentation and retain the failed test output.
4. Run /tmp/ddtest-qa testdrive --yes --command 'node -e "console.log(123)"'. Expect a no-test-events diagnostic and nonzero exit status, even though the supplied command itself succeeds.
5. Run two testdrives concurrently. Expect separate session directories and no mixing of saved traffic/output. Restore the passing assertion, then remove the disposable repository and its sessions.

Additional reporting check: in the disposable Jest project add this test, then run testdrive. Expect “Test events received.” and a separate “Tracer configuration errors: skippable_tests” diagnostic. Add a failing assertion in another test and rerun: the configuration error must remain visible alongside the test failure. Receiving events must not produce a “Test Optimization is ready” claim. Remove this fixture afterward.

const { createRequire } = require('node:module');
const preload = process.env.NODE_OPTIONS.match(/-r "([^"]+)"/)[1];
const tracer = createRequire(preload)('dd-trace');
test('diagnostic fixture', () => {
  tracer.scope().active().setTag('_dd.ci.library_configuration_error.skippable_tests', 'true');
});

Additional tracer-error scenario: in the disposable Jest project above, retain sum.js and add tracer-error.test.js with this content. Use Node.js 22+ so fetch, FormData, and Blob are available.

const sum = require('./sum');
test('exposes empty coverage as a tracer error', async () => {
  expect(sum(1, 2)).toBe(3);
  const payload = Buffer.from('81a9636f766572616765739183af746573745f73657373696f6e5f69640aad746573745f73756974655f696414a566696c657390', 'hex');
  const form = new FormData();
  form.append('coverage1', new Blob([payload], {type: 'application/x-msgpack'}), 'coverage.msgpack');
  const response = await fetch(`${process.env.DD_CIVISIBILITY_AGENTLESS_URL}/api/v2/citestcov`, {method:'POST', body:form});
  expect(response.status).toBe(200);
});

Build this PR’s binary and run /tmp/ddtest-qa testdrive --yes from the project. Expect the test to pass and the output to contain Tracer error: received with a positive count of coverage entries with empty files. The count can exceed one if the tracer retries the test. The output must explain that affected payloads were excluded and point to captured traffic; it must not say No findings.. Inspect the intake JSON artifacts and confirm the empty files list was preserved.
Remove the injected test and disposable project/session artifacts after inspection.

Version selection: repeat the successful journey with --tracer-version 6.15.0, then with --tracer-version git:COMMIT_SHA using a full SHA returned by git ls-remote https://github.com/DataDog/dd-trace-js.git HEAD. The preview and run details must show the selection, tests must execute with the session tracer, and project manifests/lockfiles must stay unchanged. --tracer-version git: must fail before creating a session.

Project precedence: install dd-trace 6.15.0 in that project, then repeat with another release and git:does-not-exist. Expect project instrumentation and dd-trace@6.15.0 · reused in the run details, no tracer installation under the session, and unchanged project dependencies.

Exit-status and error-output checks (use the disposable Jest project above):

1. With a failing assertion, run /tmp/ddtest-qa testdrive --yes --command 'npx jest --runInBand --testFailureExitCode=7', then echo $?. Expect status 7, received test events, failure findings, and a saved test-output.txt; stderr must not include Usage:.
2. On macOS/Linux, run /tmp/ddtest-qa testdrive --yes --command 'sh -c "kill -TERM $$"', then echo $?. Expect status 143 and a runtime error without usage.
3. Run /tmp/ddtest-qa testdrive --yes --command 'node -e "console.log(123)"', then echo $?. Expect status 1, a no-test-events diagnostic, and no usage.
4. Run /tmp/ddtest-qa testdrive --unknown. Expect status 1, an unknown-flag error, and command usage. Repeat with /tmp/ddtest-qa testdrive unexpected; expect an argument error and usage.
5. Restore the passing assertion and run /tmp/ddtest-qa testdrive --yes --command 'npx jest --runInBand'. Expect status 0 and received test events. Confirm manifest/lockfile checksums are unchanged, then remove the disposable project and session artifacts.

Preview accuracy checks:

1. In a disposable Jest project without dd-trace, run /tmp/ddtest-qa testdrive --tracer-version 6.15.0 and decline. Expect a concrete output-folder path and the complete npm install --prefix ... --global=false --no-save --package-lock=false --no-audit --no-fund dd-trace@6.15.0 command. Neither the output folder nor project dependency changes should be created by previewing or declining.
2. Repeat and accept. Expect the tracer to be installed in the folder printed by this run's preview, received test events, and unchanged project manifest/lockfile checksums.
3. Install dd-trace in the project and run with --tracer-version git:does-not-exist. The preview must identify the installed dd-trace library and its version, and say it will reuse it and show no npm install command. Accept; expect received test events and project-tracer reuse.
4. Run the binary from a directory outside any Git repository. Expect a prerequisite error without usage text. Invalid flags and positional arguments must still show usage.
5. Remove the disposable project and generated output folders.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-22T11:21:37.833258Z	7ffc34b	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7ffc34b078

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[datadog-official]

An existing dd-trace/ci/init entry in NODE_OPTIONS remains active with the isolated tracer. This can load two tracer versions and break the promised isolation.

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit 7ffc34b · @DataDog review to ask questions}

[datadog-official]

 

✅ All CI checks and tests passed.

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
• Patch Coverage: 94.56%
• Overall Coverage: 91.93% (+0.12%)

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 068dad6 | Docs | View more details | Give us feedback!}

[anmarchenko]

Validated the exit-status and runtime-error changes on the rebased PR head:

• make test and make lint pass.
• Focused race checks pass for the main, command, testdrive, and intake packages; Windows cross-compilation also passes.
• Manual CLI checks with Node 22, Jest 30.2.0, and dd-trace 6.15.0: passing tests return 0; a failing test with --testFailureExitCode=7 returns 7 and still reports received events; SIGTERM returns 143; a successful non-test command returns 1 with the no-events diagnostic. Runtime failures omit usage; invalid flags retain it. Project manifest and lockfile checksums remain unchanged.

The output-buffering review thread remains open intentionally. No streaming change or new E2E workflow was added.

All nine published PR heads (#139–147) now have green remote CI. GitHub reports verified=true and reason=valid for all 53 published commits. Downstream range-diff checks preserve every original logical patch.

[anmarchenko]

@autotest review

[datadog-official]

Testdrive converts external cancellation into SIGKILL, buffers complete suite output in memory, and prints usage for valid invocations that fail during persistent pre-run.

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit 6217d96 · @DataDog review to ask questions}

[anmarchenko]

Encapsulation review addressed in 3c5a9cb. Framework.Command() reuses the existing command builders; testdrive-specific policy is in internal/testdrive and framework/testdrive.go is removed. Downstream testdrive command tests now live in the testdrive package.

Validation: make test and make lint passed at each rebased head #139–147. Focused framework/testdrive race tests passed. Manual Node 22 + Jest + dd-trace checks verified the default command, an explicit command override, received events, malformed-command errors, and unchanged project manifests/lockfiles. All 54 stack commits are GitHub Verified. Remote CI is now green on all nine exact published heads (#139–147).

[anmarchenko]

Addressed the preview and environment review: tracer detection happens before confirmation, the preview states reuse or installation and shows a concrete output folder and the install command used by the platform, and no output folder is created until confirmation. Node-specific environment handling lives in testdrive/javascript.go; downstream Python and Ruby settings are separate from shared intake settings. Runtime prerequisite errors no longer print usage.

Manual checks: project-tracer preview created no output folder and showed no installation; confirmed reuse produced test events. Fallback npm installation used the exact previewed folder, Jest passed with events, and project manifest/lockfile checksums stayed unchanged. Outside-Git prerequisite errors omitted usage. Automated validation and exact-head CI are tracked separately.

[anmarchenko]

E2E Test Report: fixes verified; Jest EFD issue remains

Tested by: Shepherd Agent (autonomous QA for Datadog Test Optimization)
Test environment: Local macOS arm64; Node 22.14.0; open source test projects and a disposable Jest fixture
Revision tested: 068dad6eed67f7dab27ecc777aed63023c374666 (PR #139 head, verified after testing)

Results

Check	Result
Published summary fix: tracer configuration error	Pass. skippable_tests appears as 1 finding.; no contradictory No findings.
Published summary fix: successful command with no test events	Pass. Testdrive exits 1, says Jest: No test results received, and does not claim tests passed.
Jest testdrive on jest-community/jest-junit	Instrumentation works: 102 test events and coverage for 51/51 tests. Bare ddtest testdrive --yes still exits 1 in worker mode; see issue below.
Command overrides	Pass. Nine frameworks each ran the default command plus two plausible overrides: 27/27 executions exited 0 and sent test events. Frameworks: Jest, Vitest, Mocha, Cypress, Playwright, Cucumber, pytest, RSpec, Minitest.
Repository checks	make test passed; make lint passed with 0 issues.

Open issue

On jest-community/jest-junit (5fc51895f764d119d66ca0e3056fb87a7dc04571), the bare testdrive command exits 1 despite reporting 51/51 tests with coverage and three flaky tests. With dd-trace 6.17.0, the same suite exits 0 with --runInBand and 1 with --maxWorkers=2. Both modes emit 102 test events; in the worker run all 102 events have test.final_status=pass (99 passing attempts, 3 failed EFD retries). The EFD contract requires a passing attempt to prevent retry failures alone from failing the command. The earlier direct-Node investigation isolated the aggregate-verdict problem to the dd-trace/Jest worker integration. The DDTest summary fix in this PR does not address it.

Methodology and verification

Built the PR head from an isolated checkout. Exercised the two summary scenarios with local Jest testdrive, reproduced EFD behavior on the real Jest repository, and ran the nine-framework command matrix through Shepherd Crook and mockdog (standalone fixture for Playwright). Verified the captured intake events and mockdog test-event counts. Datadog UI verification was not performed; these local runs used mockdog/local intake and no live backend target. Logs and artifacts were retained locally for follow-up.

E2E testing performed by Shepherd Agent.