Skip to content

feat(test): treat model file paths as test selectors - #6061

Open
tripleaceme wants to merge 4 commits into
SQLMesh:mainfrom
tripleaceme:test-path-selectors
Open

tripleaceme wants to merge 4 commits into
SQLMesh:mainfrom
tripleaceme:test-path-selectors

Conversation

@tripleaceme

Copy link
Copy Markdown
Contributor

Description

Closes #6023.

sqlmesh test already took multiple positional arguments, but they were only ever looked up in the test-file index. A model file matched nothing and the command exited 0, so a pre-commit hook passing staged model files silently ran no tests.

Each positional argument now resolves to one of:

  • a test file → every test in it
  • file.yaml::test_name → that one test
  • a model file → the tests targeting that model

The results are unioned. Per your note on the issue, they are also deduplicated on fully_qualified_test_name, so overlapping selectors don't cause duplicate runs — sqlmesh test models/a.sql tests/test_a.yaml, where the YAML holds a's tests, runs each of them once, and tests/a.yaml::test_x tests/a.yaml collapses to one run too.

An argument resolving to neither a known test nor a known model file is now an error rather than a silent skip. That part is opt-in through raise_on_unknown_paths and only the CLI turns it on, because LSPContext.get_document_tests calls select_tests with whatever document happens to be open and depends on getting an empty list back — raising unconditionally would make the extension throw on every non-test file. A known model that simply has no tests still selects nothing and is not an error, since failing a hook because a model lacks a unit test isn't the intent here.

--select-model is unchanged: it still narrows the selection rather than adding to it.

Two notes on cost, since path matching is on the hot path for the pre-commit use case:

  • Selectors are matched exactly as given before any normalization, so the existing absolute-path callers pay nothing new, and normalization is os.path.abspath rather than resolve() — no stat or symlink syscalls per model.
  • The model-path index is built at most once per call, and only if some selector turns out not to be a test file. Passing 50 staged model files doesn't rebuild it 50 times.

Test Plan

Nine tests in tests/core/test_test.py:

  • test_model_path_selects_its_tests / test_model_path_without_tests_selects_nothing
  • test_model_and_test_paths_are_unioned_without_duplicates — the dedup case from your comment
  • test_overlapping_yaml_and_named_test_are_deduplicated
  • test_relative_paths_select_tests — pre-commit passes repo-relative paths
  • test_unknown_path_is_ignored_by_default — guards the LSP's behavior
  • test_unknown_path_errors_when_requested / test_unknown_test_name_errors_when_requested
  • test_select_model_still_filters_path_selection

Two in tests/cli/test_cli.py covering the CLI wiring and the non-zero exit on an unknown path.

pytest tests/core/test_test.py tests/cli/test_cli.py tests/lsp
222 passed, 1 skipped

The 4 remaining failures in those files (test_pyspark_python_model, three test_dlt_*) are pre-existing on main in my environment — missing pyspark and dlt.

Checklist

  • I have run make style and fixed any issues
  • I have added tests for my changes (if applicable)
  • All existing tests pass (make fast-test)
  • My commits are signed off (git commit -s) per the DCO

`sqlmesh test` already accepted multiple positional arguments, but they
were only ever looked up in the test-file index. Passing a model file
matched nothing and exited 0, so a commit hook handing over staged model
files silently ran no tests at all.

Each positional argument now resolves to a test file, a
`file.yaml::test_name`, or a model file, in which case the tests
targeting that model are selected. The results are unioned and
deduplicated on the fully qualified test name, so a model file and the
test file holding its tests select the same test once rather than
running it twice.

An argument that resolves to neither a known test nor a known model file
is now an error instead of a silent skip. That is opt-in via
raise_on_unknown_paths and only the CLI turns it on, because
LSPContext.get_document_tests selects against arbitrary open documents
and relies on an empty result. A known model that simply has no tests
still selects nothing, which is not an error.

Selectors are matched as given before being normalized, so relative
paths work from the project root without any resolve() calls per model.
The model path index is built at most once per call, and only when a
selector is not a test file.

Signed-off-by: Adegbite Ayoade <tripleaceme@gmail.com>
@tripleaceme

Copy link
Copy Markdown
Contributor Author

@cmgoffena13 — closes #6023, and implements the dedupe you described on the issue.

Selectors are unioned and then deduplicated on fully_qualified_test_name, so passing a model file and the test file holding its tests runs each test once rather than twice.

One decision worth your eye, described more fully in the PR body: making an unresolvable path a hard error is opt-in, because LSPContext.get_document_tests calls select_tests with whatever document is open and relies on an empty result. Only the CLI turns it on.

Zero checks here too, pending workflow approval.

@cmgoffena13

Copy link
Copy Markdown
Collaborator

Hey @tripleaceme -- would you please resolve the file conflicts. Just merged the --local flag PR, will tackle this one next.

Resolves the conflicts with SQLMesh#6060, which added `--local` to `sqlmesh
test`. Both conflicts were additive rather than contradictory:

- tests/cli/test_cli.py: each branch appended its own tests to the end
  of the file, so both blocks are kept
- docs/concepts/tests.md: path selection and `--local` document separate
  things, so both sections are kept, selection first

sqlmesh/cli/main.py and docs/reference/cli.md merged cleanly, and both
changes are present afterwards: the `--local` flag and its group-callback
gating from SQLMesh#6060, and `raise_on_unknown_paths=True` plus the TESTS
docstring from this branch.

Adds test_test_local_with_model_paths to cover the combination, which
neither branch could test on its own. `sqlmesh test --local
models/a.sql` runs that model's tests without reaching state, and an
unresolvable path still fails loudly. That is the pre-commit hook shape
from SQLMesh#6020.

Signed-off-by: Adegbite Ayoade <tripleaceme@gmail.com>
@tripleaceme

Copy link
Copy Markdown
Contributor Author

Done in ab4a951 — rebuilt on top of main now that #6060 is in, and it shows MERGEABLE again.

Both conflicts were additive rather than contradictory, so both sides are kept:

  • tests/cli/test_cli.py — each branch appended its own tests to the end of the file.
  • docs/concepts/tests.md — path selection and --local document different things. Selection comes first, then the --local section.

sqlmesh/cli/main.py and docs/reference/cli.md merged cleanly, but I checked them rather than trusting the auto-merge: the --local flag and its group-callback gating from #6060 are both still there, alongside raise_on_unknown_paths=True and the TESTS docstring from this branch. I also diffed the test section of the CLI reference page against real sqlmesh test --help output to confirm the two entries still mirror it.

One thing I added while resolving, since neither branch could cover it alone: test_test_local_with_model_paths pins that the two features compose.

$ sqlmesh test --local models/full_model.sql
.**Successfully Ran `1` Tests Against `duckdb`**

$ sqlmesh test --local models/nope.sql
Error: '.../models/nope.sql' is not a known model or test file.

That's the pre-commit shape from #6020--local for no state, a model path for selection, and a stale path still failing loudly rather than silently running nothing. Happy to drop that test from the merge commit if you'd rather keep the resolution purely mechanical.

pytest tests/cli/test_cli.py tests/core/test_test.py tests/lsp
230 passed

The test_dlt_* and test_pyspark_python_model failures are pre-existing on main in my environment (ModuleNotFoundError).

Unrelated and not touched here, but while diffing that page I noticed two entries in the test block still drift from the real --help: -v, --verbose is missing "Use -vv for very verbose output", and --select-model TEXT carries a "Can be specified multiple times" line the command doesn't print. Both predate these PRs. I left them out to keep this commit to the conflict resolution — happy to fix them separately if useful.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

sqlmesh test should treat model file paths as test selectors

2 participants