Skip to content

README.md

Latest commit

 

History

History
345 lines (265 loc) · 15.9 KB

README.md

File metadata and controls

345 lines (265 loc) · 15.9 KB
Raw

E2E Tests

Warning

These test applications are not intended to be used as example apps or templates. They may contain:

  • Outdated or deprecated dependency versions used for backwards compatibility testing
  • Known security vulnerabilities in dependencies that we intentionally test against
  • Non-production-ready configurations and code patterns

For official examples and best practices, please refer to our official documentation.

E2E tests enable us to verify the behavior of the packages in this repository as if they were to be published in their current state.

How to run

  • Copy .env.example to .env
  • OPTIONAL: Fill in auth information in .env for an example Sentry project - you only need this to run E2E tests that send data to Sentry.
  • OPTIONAL: Fill in the Cloudflare credentials in .env - you only need this to run E2E tests that deploy a real Cloudflare Worker (e.g. cloudflare-workers-send-to-sentry). A local run deploys a throwaway worker and deletes it again afterwards; set E2E_KEEP_WORKER=1 to keep it for debugging. CI keeps one worker per branch or PR instead, and PR workers are deleted by the cleanup-e2e-workers workflow when the PR closes.
  • Run yarn build:tarball in the root of the repository (needs to be rerun after every update in /packages for the changes to have effect on the tests).

To finally run all of the tests:

yarn test:e2e

Or run only a single E2E test app:

yarn test:run <app-name>

Or you can run a single E2E test app with a specific variant:

yarn test:run <app-name> --variant <variant-name>

Variant name matching is case-insensitive and partial. For example, --variant 13 will match nextjs-pages-dir (next@13) if a matching variant is present in the test app's package.json.

Using the Makefile

Alternatively, you can use the provided Makefile for an interactive test selection experience:

Prerequisites: Install fzf with Homebrew:

brew install fzf

Run tests interactively:

make run

This will display a fuzzy-finder menu of all available test applications. Select one to run it automatically.

List all test applications:

make list

For example, if you have the following variants in your test app's package.json:

"sentryTest": {
  "variants": [
    {
      "build-command": "pnpm test:build-13",
      "label": "nextjs-pages-dir (next@13)"
    },
    {
      "build-command": "pnpm test:build-13-canary",
      "label": "nextjs-pages-dir (next@13-canary)"
    },
    {
      "build-command": "pnpm test:build-15",
      "label": "nextjs-pages-dir (next@15)"
    }
  ]
}

If you run yarn test:run nextjs-pages-dir --variant 13, it will match against the very first matching variant, which is nextjs-pages-dir (next@13). If you need to target the second variant in the example, you need to be more specific and use --variant 13-canary.

A variant runs its build-command and assert-command. Without them, it runs the app's test:build and test:assert scripts, like the app's default job.

How they work

We build our packages, pack them into tarballs (yarn build:tarball), and create symlinks in the packed/ directory that point to the versioned tarballs. When a test application is run, pnpm overrides are injected into its package.json to pin all @sentry/* and @sentry-internal/* packages to those local tarballs. This means test apps install the packages as if they were published, but from the local build output instead of a registry.

The E2E test script looks for test applications in the test-applications folder. These are standalone apps that use our SDKs and can be used to verify their behavior.

How to set up a new test

Test applications are completely standalone applications that can be used to verify our SDKs. To set one up:

cd dev-packages/e2e-tests
mkdir test-applications/my-new-test-application

Make sure to add a test:build and test:assert command to the new app's package.json file.

Sentry packages are automatically resolved to the local build via pnpm overrides injected at test time, so no manual registry configuration is needed.

Test apps that deploy a real Cloudflare Worker

Apps like cloudflare-workers-send-to-sentry deploy the built app as a real Worker and check what arrives in Sentry:

  • Use createWorkerGlobalSetup and workerGlobalTeardown from @sentry-internal/test-utils/cloudflare as the Playwright global setup and teardown. Tests read the Worker URL from E2E_TEST_WORKER_URL, send requests with fetchFromWorker, and poll Sentry with the helpers from @sentry-internal/test-utils/cli (the app needs sentry as a dev dependency).
  • The Worker gets E2E_TEST_DSN, E2E_TEST_WORKER_TOKEN and the values of the secrets option as secrets.
  • CI keeps the Worker after the run, and anyone can reach its workers.dev URL. If the Worker can spend money (for example, it holds an LLM API key), it must reject requests without Authorization: Bearer <E2E_TEST_WORKER_TOKEN>. The token is new for every deploy, and fetchFromWorker sends it. See cloudflare-flue-send-to-sentry/src/auth.ts.
  • Add the workerPrefix to .github/workflows/cleanup-e2e-workers.yml, which deletes the Worker of a PR when the PR closes.
  • Set "sentryTest": { "optional": true } in package.json, because only the optional CI job has the Cloudflare secrets.

Runtime variants (Bun, Deno, Cloudflare)

To test a framework on a runtime other than Node.js, add an optionalVariants entry to the existing test app instead of creating a new app. react-router-8-framework is the reference setup.

  • runtime: set "runtime": "bun", "deno" or "cloudflare" on the variant. For such a variant, CI installs Bun or Deno, and the runner and CI set the RUNTIME env var for the build and the assert command. Without runtime, RUNTIME is empty or not set, which means node. So a runtime variant can use the default test:build and test:assert scripts, with no test:assert:<runtime> script, and a new variant needs no change to .github/workflows/build.yml.
  • RUNTIME env var: read it with getRuntime() from @sentry-internal/test-utils, which throws on any value other than node, bun, deno or cloudflare. Tests branch on it where the runtimes are expected to differ (for example platform or sdk.name).
  • SDK per runtime: each runtime inits its own SDK, the way a user of that runtime would: Node the framework SDK (instrument.mjs), Bun @sentry/bun (instrument.bun.mjs), Deno @sentry/deno (instrument.deno.mjs) and Cloudflare @sentry/cloudflare. The framework SDK only provides the framework wrappers on the other runtimes, so values that come from its init() (sdk.name, default integrations) are Node-only.
  • Start commands: playwright.config.mjs selects the start command with getRuntime(). Bun and Deno use the same build as Node and only change the start command, for example bun --bun --preload ./instrument.bun.mjs ./node_modules/@react-router/serve/bin.cjs ./build/server/index.js and deno run -A --preload=@sentry/deno/import --preload=./instrument.deno.mjs ./node_modules/@react-router/serve/bin.cjs ./build/server/index.js. The app's deno.json maps @sentry/deno/import to node_modules, because the e2e dependencies are file: tarballs that npm:@sentry/deno/import does not resolve.
  • Cloudflare: the app has the Cloudflare dependencies installed all the time. The Cloudflare build has its own vite.cloudflare.config.ts with @cloudflare/vite-plugin and sentryCloudflareVitePlugin from @sentry/cloudflare/vite, which the runner uses in place of vite.config.ts (see runtime-specific files). List the Sentry plugins after all other plugins: they set no enforce, so their position decides when their hooks run, and a sourcemap upload has to see the final output. Node, Bun and Deno share vite.config.ts. The Worker entry lives in a separate file (for example workers/app.ts) and exports a plain handler: the Sentry plugin wraps it with withSentry and reads the init options from instrument.server.ts next to the entry. The start command runs wrangler dev on the build output. Code at module scope must not do I/O (for example open a database connection), because workerd does not allow it.
  • Runtime-specific files: a file that has the runtime as a part of its name replaces the existing file without that part, for example app/entry.server.cloudflare.tsx replaces app/entry.server.tsx and vite.cloudflare.config.ts replaces vite.config.ts. For a variant with runtime, the runner copies these files over the others in the temporary copy of the app, before the build. This needs no framework or bundler config. A build in the app folder itself does not get these files, so run a variant with yarn test:run <app> --variant <runtime>.
  • Deno version: a variant can pin the Deno version, for example "deno-version": "v2.9.0".
  • Bun: under bun run the SDK cannot inject diagnostics channels into packages that stay outside the build (for example Express behind react-router-serve), so those produce no spans on Bun. Where a test depends on them, branch on RUNTIME and say why in a comment.
"sentryTest": {
  "optionalVariants": [
    { "runtime": "bun", "label": "my-app (bun)" },
    { "runtime": "deno", "label": "my-app (deno)" },
    { "runtime": "cloudflare", "label": "my-app (cloudflare)" }
  ]
}

Troubleshooting

Common Issues

Tests fail with "Cannot find module '@sentry/...'" or use wrong package version

  1. Rebuild tarballs: yarn build && yarn build:tarball
  2. Re-run yarn test:prepare to refresh symlinks
  3. Delete node_modules in the test application and re-run the test

Tests pass locally but fail in CI (or vice versa)

  • Verify all @sentry/* dependencies use latest || * version specifier
  • Check if the test relies on environment-specific behavior

Debugging Tips

  1. Enable Sentry debug mode: Add debug: true to the Sentry init config to see detailed SDK logs
  2. Check browser console: Look for SDK initialization errors or warnings
  3. Inspect network requests: Verify events are being sent to the expected endpoint
  4. Check installed versions: cat node_modules/@sentry/browser/package.json | grep version

Bundler-Specific Behavior

Different bundlers handle environment variables and code replacement differently. This is important when writing tests or SDK code that relies on build-time constants.

Webpack

  • DefinePlugin replaces variables in your application code
  • Does NOT replace values inside node_modules
  • Environment variables must be explicitly defined

Vite

  • define option replaces variables in your application code
  • Does NOT replace values inside node_modules
  • import.meta.env.VITE_* variables are replaced at build time
  • For replacing values in dependencies, use @rollup/plugin-replace

Next.js

  • Automatically injects process.env via webpack/turbopack
  • Handles environment variables more seamlessly than raw webpack/Vite
  • Server and client bundles may have different environment variable access

import.meta.env Considerations

  • Only available in Vite and ES modules
  • Webpack and Turbopack do not have import.meta.env
  • SDK code accessing import.meta.env must use try-catch to handle environments where it doesn't exist
// Safe pattern for SDK code
let envValue: string | undefined;
try {
  envValue = import.meta.env.VITE_SOME_VAR;
} catch {
  // import.meta.env not available in this bundler
}

Test apps in the folder test-applications will be automatically picked up by CI in the job job_e2e_tests (in .github/workflows/build.yml). The test matrix for CI is generated in dev-packages/e2e-tests/lib/getTestMatrix.mjs.

For each test app, CI checks its dependencies (and devDependencies) to see if any of them have changed in the current PR (based on nx affected projects). For example, if something is changed in the browser package, only E2E test apps that depend on browser will run, while others will be skipped.

You can add additional information about the test (e.g. canary versions, optional in CI) by adding sentryTest in the package.json of a test application.

If you add Sentry dependencies to your test application, set the dependency versions to latest || * in order for it to work with both regular and prerelease versions:

// package.json
{
  "name": "my-new-test-application",
  "version": "1.0.0",
  "private": true,
  "scripts": {
    "test": "echo \"Hello world!\"",
    "test:build": "pnpm install",
    "test:assert": "pnpm test",
  },
  "dependencies": {
    "@sentry/node": "latest || *",
  },
}

All that is left for you to do now is to create a test app and run yarn test:e2e.

Standardized Test Apps

For some of our E2E tests we define a standard for test applications as to how they should look and behave. Standardized test apps enables us to reuse the same test suite over a number of different frameworks/SDKs.

Standardized Frontend Test Apps

TODO: This is not up to date.

A standardized frontend test application has the following features:

  • Just for the sake of consistency we prefix the standardized frontend tests with standard-frontend-. For example standard-frontend-nextjs.

  • A page at path /

    • Having a <input type="button" id="exception-button"> that captures an Exception when clicked. The returned eventId from the Sentry.captureException() call must be written to window.capturedExceptionId. It does not matter what the captured error looks like.
    • Having an link with id="navigation" that navigates to /user/5. It doesn't have to be an <a> tag, for example if a framework has another way of doing routing, the important part is that the element to click for navigation has the correct id. Text of the link doesn't matter.

  • An empty page at /user/5

  • Apps should write all pageload and navigation transaction IDs into an array at window.recordedTransactions. This can be done with an event processor:

    Sentry.addEventProcessor(event => {
  if (
    event.type === 'transaction' &&
    (event.contexts?.trace?.op === 'pageload' || event.contexts?.trace?.op === 'navigation')
  ) {
    const eventId = event.event_id;
    window.recordedTransactions = window.recordedTransactions || [];
    window.recordedTransactions.push(eventId);
  }

  return event;
});

Standardized Backend Test Apps

TBD

Standardized Frontend-to-Backend Test Apps

A standardized Meta-Framework test application has the following features:

  • Has a parameterized backend API route /user/:id that returns a JSON object with the user ID.
  • Has a parameterized frontend page (can be SSR) /user/:id that fetches the user data on the client-side from the API route and displays it.

This setup creates the scenario where the frontend page loads, and then immediately makes an API request to the backend API.

The following test cases for connected tracing should be implemented in the test app:

  • Capturing a distributed page load trace when a page is loaded
    • The HTML meta-tag should include the Sentry trace data and baggage
    • The server root span should be the parent of the client pageload span
    • All routes (server and client) should be parameterized, e.g. /user/5 should be captured as /user/:id route
  • Capturing a distributed trace when requesting the API from the client-side
    • There should be three transactions involved: the client pageload, the server "pageload", and the server API request
    • The client pageload should include an http.client span that is the parent of the server API request span
    • All three transactions and the http.client span should share the same trace_id
    • All transaction names and the span description should be parameterized, e.g. /user/5 should be captured as /user/:id route