Fix CI: re-enable and fix `test-vscode-e2e`

[cmgoffena13]

Why this exists

The VSCode extension has Playwright end-to-end tests that drive a real editor (via code-server, which is VS Code in the browser). They cover completions, diagnostics, lineage, go-to-definition, format, etc.

That CI job is test-vscode-e2e in .github/workflows/pr.yaml. It has been turned off since 2026-01-12 (if: false) because it was failing on every PR and blocking merges. The unit/lint job (test-vscode) is already back on; this issue is only the e2e job.

Called out as follow-up on #6004.

This is not the same as test-vscode. That one runs lint + TypeScript compile + a handful of Vitest unit tests. E2E is the suite that actually opens the editor.

Goal

1. Get the e2e tests passing locally.
2. Turn the GitHub Actions job back on (with a path filter, so Python-only PRs are not blocked).
3. Keep it green.

---

How to turn the job on

In .github/workflows/pr.yaml, find test-vscode-e2e. It currently looks like this:

  test-vscode-e2e:
    runs-on:
      labels: [ubuntu-2204-8]
    # As at 2026-01-12 this job flakes 100% of the time. It needs investigation
    if: false

Change it to match test-vscode (path filter + only run when VSCode or CI config changed):

  test-vscode-e2e:
    needs: changes
    if:
      needs.changes.outputs.vscode == 'true' || needs.changes.outputs.ci ==
      'true' || github.ref == 'refs/heads/main'
    runs-on:
      labels: [ubuntu-2204-8]

Leave the rest of the steps as they are until you know they need changing. The vscode path filter already exists in the changes job (vscode/**).

---

How to run it locally

You need Node 22+, pnpm 10+, a Python 3.12 venv at the repo root named .venv, and code-server on your PATH. The tests look for .venv/bin/python and spawn code-server.

From the repo root:

# 1. Python env the extension will use
python3.12 -m venv .venv
source .venv/bin/activate
make install-dev

# 2. JS deps
pnpm install

# 3. code-server (VS Code in the browser). Pin a version if you can;
#    CI currently installs whatever "latest" is.
# macOS:
brew install code-server
# or:
# curl -fsSL https://code-server.dev/install.sh | sh

# 4. Playwright's Chromium browser
pnpm --prefix vscode/extension exec playwright install

# 5. Package the extension and run the e2e suite
cd vscode/extension
pnpm run test:e2e

That last command builds the VSCode webview, packs a .vsix, then runs Playwright. It takes a while.

Useful variants:

cd vscode/extension

# Watch the browser (easier to see what is stuck)
pnpm run test:e2e:headed

# Playwright UI
pnpm run test:e2e:ui

# After a vsix already exists, run one file without re-packing
pnpm exec playwright test tests/completions.spec.ts

When tests fail, open the HTML report (screenshots + video of the failure):

open vscode/extension/playwright-report/index.html

On CI the same report is uploaded as the playwright-report artifact (kept 30 days).

---

What these tests actually do

Very short version:

1. pnpm run vscode:package builds the React webview and packs sqlmesh-<version>.vsix.
2. Setup installs that vsix into a temp code-server.
3. Each test copies examples/sushi into a temp folder, opens it in code-server, and clicks the UI like a user.
4. Almost every test waits for the status text Loaded SQLMesh Context before doing anything else (waitForLoadedSQLMesh in vscode/extension/tests/utils.ts).

If that text never appears, most of the suite will time out. Start there.

---

Things to look into (most likely first)

Work these in order. You do not need to be a JS expert; the report will name the failing spec.

1. Unpinned code-server (best first guess)

CI does this every run:

- name: Install code-server
  run: curl -fsSL https://code-server.dev/install.sh | sh

That is latest, not a version. The tests click VS Code UI by visible text (models, Explorer, Loaded SQLMesh Context). If a code-server/VS Code update changes the UI, every test can fail the same way.

Try: pin a version that the tests were written against, install that same version locally, and re-run.

2. Selector / timeout failures

Playwright config (vscode/extension/playwright.config.ts):

• 60 second timeout per test
• 2 workers in CI, 4 locally
• retries already set to 2 in CI

If the report is full of waitForSelector timeouts, it is usually:

• SQLMesh never finished loading (Python/LSP problem → no Loaded SQLMesh Context)
• or the VS Code UI changed (code-server problem)

The tests hardcode the interpreter as <repo>/.venv/bin/python. If you did not create that venv, loading will fail.

3. Known-flaky specs (skip vs fix)

Some files already document CI-only failures. Do not treat those as a mystery:

• tests/lineage.spec.ts — comment: works locally when debugging, not on CI
• tests/bad_setup.spec.ts — typing a filename is flaky
• tests/completions.spec.ts — macro completions skipped as flaky
• tests/tcloud.spec.ts — one case skipped; sign-in window not usable by Playwright

Getting the common path green (open sushi → loaded context → completions/diagnostics/go-to-definition) is more important than unskipping these.

4. Shared code-server + 2 CI workers

Each Playwright worker starts its own code-server on a random port and they share one extensions directory. That can cause races. If pinning code-server is not enough, try workers: 1 in CI (there is history of doing that in this repo).

5. Self-hosted runner

The job uses runs-on: labels: [ubuntu-2204-8] (bigger machine). GitHub ubuntu-latest may be too small; that is why it was moved. If the job never starts, the runner label is the problem, not the tests.

---

Background (so you do not have to rediscover it)

• Disabled in Chore: update databricks and snowflake auth in integration tests #5652 (2026-01-12) with the comment that it failed 100% of the time.
• Last known green CI run found: 2025-11-05. From ~2025-11-20 through disable, sampled runs were all red.
• Failed jobs ran ~15 minutes and uploaded a ~32 MB Playwright report, so the suite was executing, not dying at install.
• Actions logs and that report have expired (HTTP 410). We do not have the failing spec names. You have to reproduce.

Done when

• pnpm run test:e2e passes locally (or remaining failures are documented test.skips with a reason)
• test-vscode-e2e is re-enabled in pr.yaml with the vscode / ci path filter (not if: false)
• The job is green on that PR
• code-server is pinned (or another root cause is written down so the next bump does not silently break CI again)