Just: test the tooling here, whose failures are all silent ones

Nothing under this directory had tests, and the ways it can break mostly exit zero: an
exclusion that stops excluding, a verb that stops being found, an argument that arrives
split in two. None of those is visible until someone notices work not being done, which
is why a paired repository was carrying a redundant path pattern purely as a hedge.

Each such test comes with a positive control, asserting the same setup can produce the
loud outcome, so that a zero means the rule held rather than that nothing ran. Writing
them turned up one crash: an argument list of nothing but blanks was counted before the
blanks were dropped, leaving no root to take and an IndexError instead of the usage.

Plain unittest and no new dependency, this being the layer that runs other tooling and
so a poor place to need a package manager. A `test` recipe reaches them by the usual
verb, and CI runs them on any change here.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Author: redsun82

.github/workflows/python-tooling.yml

@@ -5,6 +5,7 @@ on:
     paths:
       - "misc/bazel/**"
       - "misc/codegen/**"
+      - "misc/just/**"
       - "misc/scripts/models-as-data/*.py"
       - "*.bazel*"
       - .github/workflows/codegen.yml
@@ -33,3 +34,7 @@ jobs:
         shell: bash
         run: |
           bazel test //misc/codegen/...
+      - name: Run just tooling tests
+        shell: bash
+        run: |
+          bazel test //misc/just/...

misc/just/BUILD.bazel

@@ -0,0 +1,27 @@
+load("@rules_python//python:defs.bzl", "py_library", "py_test")
+
+py_library(
+    name = "tooling",
+    srcs = [
+        "codeql_test_run.py",
+        "forward_command.py",
+        "language_tests.py",
+        "run_on_files.py",
+    ],
+    imports = ["."],
+    visibility = ["//visibility:public"],
+)
+
+[
+    py_test(
+        name = src[:-len(".py")],
+        size = "small",
+        srcs = [src],
+        deps = [":tooling"],
+    )
+    for src in glob(["test_*.py"])
+]
+
+test_suite(
+    name = "test",
+)

misc/just/README.md

@@ -186,3 +186,17 @@ With no terminal to ask — a pipe, a log, a shell without `stty` — it falls b
 fixed 57 columns, so logs and CI output are the same width every time. That is one
 branch rather than a platform test: `just` runs `sh` everywhere, so Windows takes
 whichever arm fits rather than a path of its own.
+
+# Tests
+
+The scripts here have tests, run by `just test misc/just`, by `bazel test
+//misc/just/...`, or by CI on any change under this directory. They are plain
+`unittest`: this is the layer that runs other tooling, so it should not need a package
+manager to check itself.
+
+They exist for the class of bug that leaves no trace. An exclusion that stops excluding,
+a verb that stops being found, an argument that arrives split in two — each of those
+still exits zero, and the only symptom is work quietly not done. So the tests come in
+pairs: one asserting the quiet outcome, and one positive control asserting the same
+setup can produce the loud one. Without the second, a passing first test is also what a
+test that runs nothing at all looks like.

misc/just/justfile

@@ -1,4 +1,15 @@
+import "defs.just"
+
 [no-cd]
 [positional-arguments]
 format *ARGS=".":
     npx prettier --write "$@"
+
+# Test the tooling in this directory.
+# Arguments are `unittest` ones, so a test file or a dotted test name rather than a
+# directory. A path given stays valid as this does not change directory: `PYTHONPATH` is
+# what lets the tests import the modules they are about, wherever they are run from.
+[no-cd]
+[positional-arguments]
+@test *ARGS=['discover', '-s', source_dir()]:
+    PYTHONPATH="{{ source_dir() }}" {{ py }} -m unittest "$@"

misc/just/language_tests.py

@@ -15,7 +15,10 @@
 
 
 def main():
-    argv = sys.argv[1:]
+    # Blank arguments are dropped before the count is taken: one comes of a caller
+    # interpolating a variable that was never set, and a list of nothing but those is no
+    # arguments at all rather than a root to find a justfile above.
+    argv = [arg for arg in sys.argv[1:] if arg]
     if not argv:
         print("Usage: language_tests.py ROOT [ARG...]", file=sys.stderr)
         return 1
@@ -25,9 +28,7 @@ def main():
     # the internal checkout, so relativize them there to keep command lines readable.
     # Anything else (flags, environment assignments, relative paths) is passed verbatim.
     args = [
-        os.path.relpath(arg, semmle_code) if os.path.isabs(arg) else arg
-        for arg in argv
-        if arg
+        os.path.relpath(arg, semmle_code) if os.path.isabs(arg) else arg for arg in argv
     ]
 
     just = os.environ.get("JUST_EXECUTABLE", "just")

misc/just/test_codeql_test_run.py

@@ -0,0 +1,119 @@
+#!/usr/bin/env python3
+"""Tests for `codeql_test_run.py`.
+
+Sorting arguments is the whole of what this file decides, and it decides it by looking
+at each one: a word is a test, a `-` is a flag, `NAME=value` is an environment
+assignment. Several of these pin down that an argument arrives whole, spaces and all,
+which is what taking them as a list rather than re-splitting a string bought.
+"""
+
+import os
+import unittest
+from unittest import mock
+
+import codeql_test_run
+
+
+def empty_args():
+    """What `main` builds before sorting, limited to what `parse_args` fills in."""
+    return {
+        "tests": [],
+        "flags": [],
+        "env": [],
+        "all_checks": [],
+        "codeql": "host",
+        "all": False,
+    }
+
+
+def sorted_args(*argv):
+    args = empty_args()
+    codeql_test_run.parse_args(args, list(argv))
+    return args
+
+
+class TestParseArgs(unittest.TestCase):
+    def test_a_plain_word_is_a_test(self):
+        self.assertEqual(sorted_args("ql/test/Foo")["tests"], ["ql/test/Foo"])
+
+    def test_a_dash_is_a_flag(self):
+        self.assertEqual(sorted_args("--fail-on-trap-errors")["flags"], ["--fail-on-trap-errors"])
+
+    def test_an_uppercase_assignment_is_an_environment_variable(self):
+        self.assertEqual(sorted_args("CPUS=4")["env"], ["CPUS=4"])
+
+    def test_a_lowercase_assignment_is_a_test(self):
+        # Only shouting counts, so a path that happens to contain `=` stays a path.
+        self.assertEqual(sorted_args("dir/a=b")["tests"], ["dir/a=b"])
+
+    def test_codeql_selects_the_executable(self):
+        self.assertEqual(sorted_args("--codeql=built")["codeql"], "built")
+
+    def test_the_last_codeql_wins(self):
+        self.assertEqual(sorted_args("--codeql=host", "--codeql=built")["codeql"], "built")
+
+    def test_all_checks_is_asked_for_by_either_spelling(self):
+        self.assertTrue(sorted_args("--all-checks")["all"])
+        self.assertTrue(sorted_args("+")["all"])
+
+    def test_an_extra_check_is_held_back_until_it_is_asked_for(self):
+        held = sorted_args("--all-checks=--check-databases")
+        self.assertEqual(held["all_checks"], ["--check-databases"])
+        # Held back means held back: it is not a flag until `--all-checks` arrives.
+        self.assertEqual(held["flags"], [])
+        self.assertFalse(held["all"])
+
+    def test_an_empty_argument_is_ignored(self):
+        # One of these comes of a caller interpolating a variable that was never set.
+        self.assertEqual(sorted_args("", "test")["tests"], ["test"])
+
+    def test_a_test_path_containing_a_space_stays_one_argument(self):
+        self.assertEqual(sorted_args("some dir/test")["tests"], ["some dir/test"])
+
+    def test_an_assignment_whose_value_contains_a_space_stays_whole(self):
+        """The value is the point, and a split one used to arrive as three characters.
+
+        An argument list carries this; a whitespace-separated string cannot, as there
+        is nothing left in it to tell a separator from part of a value.
+        """
+        self.assertEqual(sorted_args("EXTRA=a b")["env"], ["EXTRA=a b"])
+
+    def test_sorts_a_whole_command_line_at_once(self):
+        args = sorted_args("-j2", "CPUS=4", "ql/test", "+", "--all-checks=--check-diff")
+        self.assertEqual(args["flags"], ["-j2"])
+        self.assertEqual(args["env"], ["CPUS=4"])
+        self.assertEqual(args["tests"], ["ql/test"])
+        self.assertEqual(args["all_checks"], ["--check-diff"])
+        self.assertTrue(args["all"])
+
+
+class TestEnvValue(unittest.TestCase):
+    def test_prefers_a_test_argument(self):
+        args = sorted_args("CPUS=4")
+        with mock.patch.dict(os.environ, {"CPUS": "8"}):
+            self.assertEqual(codeql_test_run.env_value(args, "CPUS", "1"), "4")
+
+    def test_falls_back_to_the_environment(self):
+        with mock.patch.dict(os.environ, {"CPUS": "8"}):
+            self.assertEqual(codeql_test_run.env_value(empty_args(), "CPUS", "1"), "8")
+
+    def test_falls_back_to_the_default(self):
+        with mock.patch.dict(os.environ, {}, clear=True):
+            self.assertEqual(codeql_test_run.env_value(empty_args(), "CPUS", "1"), "1")
+
+    def test_the_last_assignment_wins(self):
+        args = sorted_args("CPUS=4", "CPUS=2")
+        self.assertEqual(codeql_test_run.env_value(args, "CPUS", "1"), "2")
+
+    def test_an_empty_value_does_not_count_as_a_setting(self):
+        args = sorted_args("CPUS=")
+        with mock.patch.dict(os.environ, {"CPUS": "8"}):
+            self.assertEqual(codeql_test_run.env_value(args, "CPUS", "1"), "8")
+
+    def test_a_value_containing_a_space_survives(self):
+        args = sorted_args("EXTRA=a b")
+        self.assertEqual(codeql_test_run.env_value(args, "EXTRA", "none"), "a b")
+
+
+if __name__ == "__main__":
+    unittest.main()