diff --git a/HISTORY.asc b/HISTORY.asc index 0b02dcd..5ea70f3 100644 --- a/HISTORY.asc +++ b/HISTORY.asc @@ -1,5 +1,12 @@ STABLE ------ +== Fix `make results` refusing to create a new test's first expected output +`make verify-results` used to block `make results` anytime +`regression.diffs` existed, which would be the case when there were no +`test/expected/` files at all. Switch to actually examining +`regression.diffs` to determine if tests are broken. Note this is only in +effect when `PGXNTOOL_VERIFY_RESULTS_MODE=pgtap`. + == Add `PGXNTOOL_ENABLE_FS_INSTALL` and `PGXNTOOL_ENABLE_PGXN_INSTALL` `test`/`verify-results` unconditionally filesystem-installed the extension (via PGXS's `install`), and `installcheck` unconditionally auto-installed @@ -61,7 +68,7 @@ silently swallowed too, making the file "pass" regardless of what happened. `test/install/*.out` was never really compared against anything, and is rewritten by every run, it's now gitignored -- stop committing it. -Issues fixed in this release: #87, #21, #55, #90, #97, #108 +Issues fixed in this release: #87, #21, #55, #90, #97, #108, #119 2.3.0 ----- diff --git a/README.asc b/README.asc index 3c56ac7..d194c92 100644 --- a/README.asc +++ b/README.asc @@ -263,11 +263,13 @@ Never run `make results` without first verifying the test changes are correct. T ==== verify-results safeguard By default, `make results` will refuse to run if your tests are failing, so you can't accidentally promote incorrect output into the new expected results. Which failures it looks for is controlled by `PGXNTOOL_VERIFY_RESULTS_MODE`: -`pgtap` (the default):: Scans `test/results/*.out` for pgTAP `not ok` lines (ignoring `# TODO` items) and plan-count mismatches, then also checks `test/results/regression.diffs` for any output mismatch pgTAP itself wouldn't catch. -`diffs`:: Only checks whether `test/results/regression.diffs` exists. +`pgtap` (the default):: Scans `test/results/*.out` for pgTAP `not ok` lines (ignoring `# TODO` items) and plan-count mismatches, then also checks `test/regression.diffs` for any output mismatch pgTAP itself wouldn't catch. A test that has no expected output file yet is the one exception: it always shows up in `regression.diffs` (pgxntool creates an empty expected file for it, because `pg_regress` aborts on a missing one), so `make results` is allowed to create its first expected output — provided its own pgTAP output is clean and free of SQL errors. If a new test legitimately needs output containing a SQL error blessed, use `PGXNTOOL_ENABLE_VERIFY_RESULTS=no` for that one run. +`diffs`:: Only checks whether `test/regression.diffs` exists, so it can't create a new test's first expected output; use `pgtap` mode, or `PGXNTOOL_ENABLE_VERIFY_RESULTS=no` for that one run. If tests are failing, `make results` prints an explanation of what failed and exits without touching `test/expected/`; fix the failures, then re-run `make results`. +It also refuses when `pg_regress` didn't finish at all — an unreachable server, an interrupt — since `test/results/` then still holds an earlier run's output and blessing it would make that stale output the new expected results. + To select a mode explicitly: ---- # In your Makefile @@ -755,7 +757,7 @@ Default: `pgtap`. Controls how the <<_verify_results_safeguard,verify-results sa ==== pgtap -Scans `test/results/*.out` for pgTap `not ok` lines and plan mismatches, falling back to checking for `regression.diffs` too. Use this mode when your test suite uses pgTap. +Scans `test/results/*.out` for pgTap `not ok` lines and plan mismatches, falling back to checking `regression.diffs` too — except for a test that has no expected output yet, whose first expected file `make results` is allowed to create. Use this mode when your test suite uses pgTap. ==== diffs diff --git a/README.html b/README.html index 191cd0a..e7bc83c 100644 --- a/README.html +++ b/README.html @@ -1060,11 +1060,11 @@

pgtap (the default)
-

Scans test/results/*.out for pgTAP not ok lines (ignoring # TODO items) and plan-count mismatches, then also checks test/results/regression.diffs for any output mismatch pgTAP itself wouldn’t catch.

+

Scans test/results/*.out for pgTAP not ok lines (ignoring # TODO items) and plan-count mismatches, then also checks test/regression.diffs for any output mismatch pgTAP itself wouldn’t catch. A test that has no expected output file yet is the one exception: it always shows up in regression.diffs (pgxntool creates an empty expected file for it, because pg_regress aborts on a missing one), so make results is allowed to create its first expected output — provided its own pgTAP output is clean and free of SQL errors. If a new test legitimately needs output containing a SQL error blessed, use PGXNTOOL_ENABLE_VERIFY_RESULTS=no for that one run.

diffs
-

Only checks whether test/results/regression.diffs exists.

+

Only checks whether test/regression.diffs exists, so it can’t create a new test’s first expected output; use pgtap mode, or PGXNTOOL_ENABLE_VERIFY_RESULTS=no for that one run.

@@ -1072,6 +1072,9 @@

+

It also refuses when pg_regress didn’t finish at all — an unreachable server, an interrupt — since test/results/ then still holds an earlier run’s output and blessing it would make that stale output the new expected results.

+ +

To select a mode explicitly:

@@ -2040,7 +2043,7 @@

8.8.1. pgtap

-

Scans test/results/*.out for pgTap not ok lines and plan mismatches, falling back to checking for regression.diffs too. Use this mode when your test suite uses pgTap.

+

Scans test/results/*.out for pgTap not ok lines and plan mismatches, falling back to checking regression.diffs too — except for a test that has no expected output yet, whose first expected file make results is allowed to create. Use this mode when your test suite uses pgTap.

@@ -2134,7 +2137,7 @@

diff --git a/verify-results-pgtap.sh b/verify-results-pgtap.sh index dd8d7e9..afd24a0 100755 --- a/verify-results-pgtap.sh +++ b/verify-results-pgtap.sh @@ -2,7 +2,8 @@ # pgxntool/verify-results-pgtap.sh - Check pgtap results before 'make results' # # Scans pgtap output files for failures and plan mismatches, then checks -# regression.diffs as a fallback. Exits non-zero if any problems are found. +# regression.diffs as a fallback, ignoring entries for tests that have no +# expected output yet. Exits non-zero if any problems are found. # # Usage: verify-results-pgtap.sh TESTOUT # @@ -36,12 +37,111 @@ if [ $failed -ne 0 ]; then exit 1 fi -# Also check regression.diffs (output mismatch even if pgtap all passed) -if [ -r "$TESTOUT/regression.diffs" ]; then - echo "ERROR: Tests are failing. Cannot run 'make results'." - echo "Fix test failures first, then run 'make results'." - echo - echo "See $TESTOUT/regression.diffs for details:" - cat "$TESTOUT/regression.diffs" +# Also check regression.diffs (output mismatch even if pgtap all passed). +# +# A test whose output matched ANY of its expected files - including a +# pg_regress _N.out alternate - is left out of regression.diffs entirely, so +# every block in here is a test that matched none of them. The one exception +# is a test that has no expected output yet: pg_regress aborts outright on a +# missing expected file, so base.mk touches an empty test/expected/.out +# for every test/sql/*.sql lacking one, and diffing actual output against that +# placeholder always reports a difference. Blocking on that is unwinnable - +# seeding that first expected file is precisely what `make results` is for - +# so those blocks are skipped here. +diffs="$TESTOUT/regression.diffs" + +# pg_regress truncates regression.diffs at startup and deletes it again on a +# clean finish with nothing to report, so an empty one means it died before +# comparing anything - an unreachable server, an interrupt. Whatever is in +# results/ is then left over from some earlier run, and base.mk's `.IGNORE: +# installcheck` means `make results` gets here anyway, so blessing it would +# bless stale output. +if [ -e "$diffs" ] && [ ! -s "$diffs" ]; then + echo "ERROR: pg_regress did not complete. Cannot run 'make results'." + echo "$diffs is empty, so nothing was actually compared;" + echo "anything in $TESTOUT/results/ is left over from an earlier run." exit 1 fi + +if [ -s "$diffs" ]; then + # pg_regress heads each block with "diff " (see + # results_differ() in pg_regress.c), then appends the diff itself. Only + # those headers and diff hunk headers can start with "diff " or "@@ " + # unprefixed; every line carrying file content is prefixed with ' ', '+', + # '-' or '\'. A block whose hunks all read "@@ -0,0 +N,M @@" was diffed + # against an empty expected file, i.e. that placeholder. Anything that + # fits none of those shapes is something this has no business + # interpreting, so it blocks the whole file. + # + # Hunk headers only exist in unified diffs. PostgreSQL 12 was the first + # to write them (11 and older use -C3 context diffs), so on those every + # block lands in the blocked pile - the same refusal `make results` gave + # before any of this existed. + classified=$(awk ' + function flush( status) { + if (results == "") return + status = (hunks > 0 && unblessed) ? "unblessed" : "regression" + print status "\t" results + } + /^diff / { flush(); results = $NF; unblessed = 1; hunks = 0; next } + /^@@ / { hunks++; if ($0 !~ /^@@ -0,0 /) unblessed = 0; next } + /^[ +-]/ { next } + /^\\/ { next } + /^$/ { next } + { unrecognized = 1 } + END { flush(); if (unrecognized) print "unrecognized\t" FILENAME } + ' "$diffs") + + blocked= + blessable= + while IFS="$(printf '\t')" read -r status file; do + case $status in + unblessed) + # An unblessed file holding a hard SQL error is not a passing test + # the pgtap scan above was merely quiet about: ON_ERROR_STOP aborts + # the script, so pgtap emits neither 'not ok' nor its plan-mismatch + # line. Blessing that would make the error the baseline. + # + # pg_regress feeds psql on stdin, so a failing statement in the + # test file itself reports as "ERROR: ..." at column 0, while one + # inside an \i'd file (setup.sql, finish.sql) reports as + # "psql:test/pgxntool/setup.sql:3: ERROR: ...". Matching either + # position, and psql's own two spaces after the severity, keeps a + # passing test whose description merely mentions "ERROR:" out of it. + if [ -r "$file" ] && ! grep -qE '(^|: )ERROR: ' "$file"; then + blessable="$blessable $file +" + continue + fi + ;; + unrecognized) + blocked="$blocked (unrecognized content in $file) +" + continue + ;; + esac + blocked="$blocked $file +" + done <