Fix DBBench SELECT grading for multi-column results

[appleweiping]

DBBench evaluates agent-controlled result strings, drops multi-column MySQL rows, and maps missing or non-finite answers to zero. On the training SELECT split, this can reward an empty answer for a non-empty result.

This change parses result literals without executing arbitrary calls. It permits Decimal, retains every column, and keeps genuine empty results matchable. Missing and non-finite answers do not match zero. Invalid or truncated literals do not become empty results. Quote-wrapped items keep their existing matching behavior.

The eight parser regressions cover all 1,290 shipped training SELECT labels, all 100 non-mutation standard rows, all 20 dev rows, and an attempted code-execution payload. They previously passed on Windows Python 3.14 and WSL Python 3.12 at a89fba7.

The current head bcec10d adds four opt-in live MySQL tests. All 12 tests pass on Windows Python 3.12 with MySQL 8.4.11 and Connector 26.7.0. The live checks use the actual database class methods with a real localhost connection, then the grader. They cover multiple columns, Decimal, NULL, empty results, zero, invalid answers, and quote compatibility. Using the upstream grader reproduces one multi-column failure and three false zero matches.

DBBENCH_MYSQL_PORT=33060 python -m unittest discover -s tests -v

Without that environment variable, the four live tests are skipped. The test loader isolates repository class definitions to avoid optional AgentRL imports; it bypasses container allocation. A full AgentRL session, Docker allocation, and root-level pytest have not been validated here. Ruff and diff checks pass.

Addresses only the DBBench parsing and reward parts of #234. The other issues remain open.

[shaurya416]

Ran the base (d1e4a10) and head (9fadd06) versions of DBResultProcessor.compare_results side by side on the cases from #234 and its follow-up comment, and over the shipped data/dbbench files.

Fixed at this head:

• None, ["None"], ["null"], ["undefined"], ["nan"], ["NaN"], ["inf"], ["-infinity"], [""] and a bare "" against gold "[(0,)]" change from True to False; ["0"] still matches and ["5"] still fails. In standard.jsonl, the two rows with gold ['0.0'] (lines 39 and 43) accepted ["null"] on base and reject it on head.
• An empty answer no longer matches multi-column gold. Across the 1,290 SELECT rows of db_out_new.jsonl, [] matches 1,001 rows on base and 21 on head (the 21 whose gold is "[]"). Echoing the gold result string as the answer now matches all 980 multi-column golds (0 on base).
• A call inside a string answer is no longer evaluated: "[(len('x'),)]" matched gold "[(1,)]" on base because the call ran, and does not match on head. The literal "[(1,)]" matches on both.

One behaviour change to confirm as intended: the list branch of _clean_answer no longer strips surrounding quotes from each item (result_processor.py:243; base applied str(item).strip().strip("'\"") before normalising). With each gold item wrapped in single quotes (for example ["'Women +60kg Bronze'"] for gold ['Women +60kg Bronze']), all 100 non-mutation rows of standard.jsonl and all 20 of dev.jsonl match on base and none match on head; the same answers without quotes match on both. The string branch (:222) still strips quotes, so a quoted bare string and a quoted list item are now normalised differently.

Still open, as the description says:

• src/assigner.py:321 computes the overall result only once every index has completed; errored samples are re-queued (:345-349) and written to error.jsonl instead of being recorded (:368-376), and src/analysis.py:86 reads only directories that contain overall.json.
• extra/docker-compose.yml:5-10 runs the controller on the host network with no authentication settings.
• OS check scripts run through execute_independent in the same container the agent used (src/server/tasks/os_interaction/task.py:371, :726).

[appleweiping]

Thanks for catching the quoted-list regression. I reproduced it with standard.jsonl:1; the new test failed on the previous PR head. In a89fba7 I restored the old quote stripping for ordinary list items after the tuple parsing path. The regression test now covers all 100 non-mutation rows in standard.jsonl and all 20 in dev.jsonl.

The full db_out_new.jsonl SELECT-label test still passes. pytest tests -q passes 8 tests on Windows Python 3.14 and WSL Python 3.12, and Ruff passes. I have not run the AgentRL or MySQL integration test.

[shaurya416]

Confirmed at a89fba7 with the same harness: quote-wrapped list answers match again on all 100 non-mutation rows of standard.jsonl and all 20 of dev.jsonl, and the unquoted answers still match. The earlier fixes are unchanged at this head. None/null/NaN/inf/"" still fail against "[(0,)]" while ["0"] passes. [] matches only the 21 empty golds out of 1,290 SELECT rows in db_out_new.jsonl. Echoed gold strings match all 980 multi-column golds. A call inside a string answer is still not evaluated.

[appleweiping]

I added four opt-in live MySQL tests. They run actual SELECT queries through the existing MySQLDatabase methods and then the grader, covering multiple columns, Decimal, NULL, empty results, zero, and quoted single-column answers.

With MySQL 8.4.11, Connector 26.7.0, and Python 3.12 on Windows, all 12 tests pass, including the eight existing regressions. Running the four live checks against the upstream grader reproduces the multi-column mismatch and three false zero matches for missing or non-finite answers.

The test loader isolates the existing database class bodies to avoid the optional AgentRL imports. It bypasses container allocation and supplies a real localhost connection. This is database-method and grading coverage, not a full AgentRL session or container test.

To rerun against a disposable local server:

DBBENCH_MYSQL_PORT=33060 python -m unittest discover -s tests -v