Skip to content

Cache deterministic Z3 errors - #1840

Merged
arcuri82 merged 3 commits into
masterfrom
fix/z3-cache-deterministic-errors
Oct 8, 2026
Merged

arcuri82 merged 3 commits into
masterfrom
fix/z3-cache-deterministic-errors

Conversation

@agusaldasoro

@agusaldasoro agusaldasoro commented Oct 7, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

SMTLibZ3DbConstraintSolver never cached an ERROR result, on the grounds that it might be a transient Docker failure. Many errors are not transient, though:

  • Z3 rejects the formula, e.g. because a generated literal is malformed. It prints (error "...") and exits with code 1.
  • Z3's output cannot be parsed by SMTResultParser.

The same query produces the same formula and the same output every time. The search keeps producing such queries, so each reappearance paid for SMT-LIB generation, a temporary file and a full round-trip to the Z3 container, only to fail again. Each repeat was also counted as a new Z3 error in the statistics.

Fix

  • Z3Result gets an isDeterministicError() flag and a deterministicError(message) factory. error(message) keeps its meaning: an error that may be transient.

  • Z3DockerExecutor reports two cases as deterministic:

    • a non-zero exit whose stdout starts with (error, i.e. Z3 rejected the formula;
    • an exception thrown while parsing a sat model.

    Everything else stays a plain error: I/O or interruption, other exit codes, empty output, and other runtime exceptions while running the container. Their message no longer claims a parsing failure.

  • SMTLibZ3DbConstraintSolver caches an ERROR only when it is deterministic. A cached error is served like a cached UNSAT: a cache hit that returns no actions. UNKNOWN and transient errors are still never cached.

Tests

Z3DockerExecutorTest, against the Z3 image:

  • whenSolvingInvalidFileItReturnsError now also checks that the error is deterministic.
  • New whenFormulaIsRejectedTheErrorIsDeterministic: a formula with an unescaped quote in a string literal, which Z3 rejects partway through the file.
  • whenSolvingEmptyFileItReturnsError checks that empty output is not treated as deterministic.

All tests in core-extra/solver and under org.evomaster.core.database.sql.solver pass. The comments that said errors are never cached have been updated.

The caching itself is a single guarded line in the solver. It has no end-to-end test, because every query that currently makes Z3 fail is a bug fixed in another open PR (#1836, #1838), and such a test would break once those are merged.

  An ERROR result was never cached, so a query whose formula Z3 rejects,
  or whose model cannot be parsed, paid a full Z3 round-trip again on
  every reappearance. The executor now marks these errors as
  deterministic and the solver caches them. Errors that may be transient,
  such as container or I/O failures, are still never cached.
@agusaldasoro agusaldasoro changed the title Cache deterministic Z3 errors Cache deterministic Z3 errors Oct 7, 2026
@agusaldasoro
agusaldasoro marked this pull request as ready for review October 7, 2026 12:50
@agusaldasoro
agusaldasoro requested a review from jgaleotti October 7, 2026 12:50
@jgaleotti
jgaleotti requested a review from arcuri82 October 7, 2026 13:39
@arcuri82
arcuri82 merged commit 6f58eea into master Oct 8, 2026
33 checks passed
@arcuri82
arcuri82 deleted the fix/z3-cache-deterministic-errors branch October 8, 2026 07:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants