analyze_column(): handle null values in the common-value analysis

[feiiiiii5]

Follow-up to #887, which has the longer write-up. analyze_column() mis-handles columns that contain nulls, in three separate ways. All three are in sqlite_utils/db.py.

1. The "single distinct value" shortcut reports a count that is too high

count(distinct col) skips nulls in SQL. So a column holding nulls plus one other value reports num_distinct == 1 and takes the shortcut that skips the group by:

if num_distinct == 1:
    value = db.execute(f"select {column_quoted} from {table_quoted} limit 1").fetchone()[0]
    most_common_results = [(truncate(value), total_rows)]

>>> db["t"].insert_all([{"c": None}, {"c": None}, {"c": "a"}, {"c": "a"}])
>>> db["t"].analyze_column("c").most_common
[(None, 4)]
>>> db.execute("select c, count(*) from t group by c").fetchall()
[(None, 2), ('a', 2)]

One value, counted four times, with the two nulls not mentioned at all. The shortcut is only valid when every row holds that one value, so it now also requires the column to have no nulls.

2. A null tied on count with a string raises TypeError

Both sorts break ties on the value itself, and None cannot be ordered against a string:

>>> db["t"].insert_all([{"c": None}, {"c": "a"}] + [{"c": c} for c in "bcdefg" for _ in range(2)])
>>> db["t"].analyze_column("c", common_limit=5).least_common
TypeError: '<' not supported between instances of 'NoneType' and 'str'

sort_key() keeps nulls out of the comparison and sorts them last. This preserves the existing ordering of non-null ties — test_analyze_tables.py asserts "Terryterryterry" comes before "Kumar" at equal counts, and that still passes.

3. A null renders as the string "None"

truncate() did str(value) on anything that was not a float or int, so a null became "None" — indistinguishable from a column that really holds "None". This is the path analyze-tables takes, since it passes value_truncate=80:

>>> db["t"].insert_all([{"c": None}, {"c": None}, {"c": "None"}])
>>> db["t"].analyze_column("c", total_rows=3, value_truncate=80).most_common
[('None', 2), ('None', 1)]      # two entries, both spelled "None"

With --save that string is also persisted into _analyze_tables_.most_common.

What I did not change, and why

What I did not change, and why

num_distinct itself still follows SQL, so an all-null column reports 0 — test_analyze_table_column_all_nulls pins that output and I did not want to change a tested value as a side effect of a crash fix.

That leaves one consequence I can see but did not fix: num_distinct can be lower than len(most_common) when a column holds nulls alongside otherwise all-distinct values.

>>> db["t"].insert_all([{"c": None}, {"c": "a"}, {"c": "b"}])
>>> d = db["t"].analyze_column("c")
>>> d.num_distinct, len(d.most_common)
(2, 3)

The fix for that is one line — add 1 to num_distinct when num_null — but it changes what Distinct values: means for every column containing nulls, and whether a null should count as a distinct value is a question about the number's meaning rather than a bug. Happy to make that change too if you would rather it did.

Testing

Six new cases in tests/test_analyze.py, all failing on 6bc1d33:

• most_common_includes_nulls — the shortcut case, with and without value_truncate
• most_common_ties_with_nulls and least_common_ties_with_nulls — the two sorts
• does_not_render_null_as_the_string_none — the truncate() case
• num_distinct_ignores_nulls — pins the current SQL semantics, passes both before and after, so the decision above is documented either way

Suite: 1506 passed, 16 skipped. mypy sqlite_utils tests, pyright sqlite_utils tests, flake8, ty check sqlite_utils, black . --check and cog --check --diff README.md docs/*.rst all clean.

Testing

Six new cases in tests/test_analyze.py, all failing on 6bc1d33:

• most_common_includes_nulls — the shortcut case, with and without value_truncate
• most_common_ties_with_nulls and least_common_ties_with_nulls — the two sorts
• does_not_render_null_as_the_string_none — the truncate() case
• num_distinct_ignores_nulls — pins the current SQL semantics, passes both before and after, so the decision above is documented either way

Suite: 1506 passed, 16 skipped. mypy sqlite_utils tests, pyright sqlite_utils tests, flake8, ty check sqlite_utils, black . --check and cog --check --diff README.md docs/*.rst all clean.

---

📚 Documentation preview 📚: https://sqlite-utils--888.org.readthedocs.build/en/888/