Skip to content

fix(dialect): convert JSON test values to NULL and valid JSON - #6113

Open
manan28025 wants to merge 1 commit into
SQLMesh:mainfrom
manan28025:test-json-null
Open

manan28025 wants to merge 1 commit into
SQLMesh:mainfrom
manan28025:test-json-null

Conversation

@manan28025

Copy link
Copy Markdown
Contributor

Description

Fixes #5387.

transform_values() wrapped every JSON value as PARSE_JSON('{value}'). When a JSON column is left out of a unit test's input rows, its value is None, so the fixture became PARSE_JSON('None') and DuckDB failed with Malformed JSON at byte 0 of input: unexpected character. Input: "None". The same formatting also broke two other cases: a dict given for a JSON column was rendered with Python's repr ({'a': 1}, not valid JSON), and a single quote in the value ended the SQL string early.

Now None becomes NULL, dicts, lists and other Python scalars are serialized with json.dumps, and the value is passed to PARSE_JSON as an escaped string literal. JSON strings without quotes produce the same SQL as before.

Test Plan

  • Added test_select_from_values_for_batch_range_json_values (NULL, a dict, and a string with a single quote). It fails on main.
  • Added test_json_column_missing_from_input_rows, a unit test that leaves a JSON column out of its input rows. It fails on main with the error from the issue and passes now.
  • Ran the fast and slow tests in tests/core. The only failures are MSSQL/Fabric connection tests that need pyodbc / mssql-python, which I don't have installed. They fail the same way on main.

Checklist

  • I have run make style and fixed any issues
  • I have added tests for my changes (if applicable)
  • All existing tests pass (make fast-test) (ran the tests/core fast and slow suites, see above)
  • My commits are signed off (git commit -s) per the DCO

transform_values wrapped every JSON value in PARSE_JSON('{value}'), so a
column left out of a unit test's input rows became PARSE_JSON('None') and
DuckDB failed with "Malformed JSON". Dicts were rendered with Python's repr,
and quotes in the value broke the SQL string. None now becomes NULL, dicts
and lists are serialized with json.dumps, and the value is passed as an
escaped string literal.

Signed-off-by: neatninja <manan81140@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Tests (DuckDB error) - Malformed JSON at byte 0 of input

2 participants