Skip to content

Forward ignore_extras= and extras_key= from the TSV and auto-detect paths - #893

Open
feiiiiii5 wants to merge 1 commit into
simonw:mainfrom
feiiiiii5:fix/rows-from-file-tsv-ignore-extras
Open

feiiiiii5 wants to merge 1 commit into
simonw:mainfrom
feiiiiii5:fix/rows-from-file-tsv-ignore-extras

Conversation

@feiiiiii5

@feiiiiii5 feiiiiii5 commented Sep 30, 2026 •

Copy link
Copy Markdown

Fixes #892.

rows_from_file() documents that its extra-field options apply to "a CSV or TSV file", but they only ever worked for format=Format.CSV.

Both branches that recurse into the CSV branch called rows_from_file() without forwarding the two arguments:

    elif format == Format.TSV:
        rows, _ = rows_from_file(
            fp,
            format=Format.CSV,
            dialect=csv.excel_tab,
            encoding=encoding,
            # ignore_extras= and extras_key= were missing here
        )

So the inner call applied its own defaults — ignore_extras=False, extras_key=None — and raised RowError before the outer _extra_key_strategy() wrapper, which turned out to be dead code, ever saw the row. The caller had explicitly asked to ignore or capture the extras and got an exception instead.

The fix forwards both arguments and drops the redundant outer wrapper, so the strategy is applied once, on the shared CSV path that already implements it. With no options passed nothing changes: RowError is still raised, as documented.

Scope

I fixed the auto-detect branch (format omitted) in the same commit: it is the same root cause — same dropped-argument recursion, same dead wrapper — and would otherwise be a one-line follow-up. It does interact with #891 in a way worth stating so the two do not look contradictory: once the sniffer picks the right delimiter for a ragged file, the auto-detected path will honour ignore_extras as well. If you would rather split them, that is easy.

I did not add an auto-detect regression test, because csv.Sniffer returns the wrong delimiter on a small ragged input and pinning today's sniffing behaviour would bake in the bug #891 is about. The new test passes an explicit format, so it does not depend on the sniffer.

Tests

test_rows_from_file_tsv_extra_fields_strategies is a TSV copy of the existing CSV test, with the same three cases: ignore_extras=True drops the extra value, extras_key="_rest" collects it, and no options still raises.

Before the fix, two of the three fail with:

sqlite_utils.utils.RowError: Row {'id': '1', 'name': 'Cleo'} contained these extra values: ['oops']
sqlite_utils/utils.py:280: RowError

and the frame shows ignore_extras = False, extras_key = None inside _extra_key_strategy despite the caller passing True / "_rest".

I also checked the three neighbouring cases by hand on 6bc1d33 and on this branch. TSV with ignore_extras=True and TSV with extras_key both raise before and return the documented result after; TSV with no options raises in both; the auto-detected path is unchanged here because the sniffer mis-detects this small ragged input (that is #891).

The existing test hardcoded format=Format.CSV at tests/test_rows_from_file.py:44, which is why this was never caught.

git log -S"_extra_key_strategy" points at d379f43 ("rows_from_file(... ignore_extras: bool, restkey: str), refs #440"), which introduced the strategy and both branches together. The recursion has never forwarded the arguments, so this is an oversight in the original commit rather than a decision.

Checks

gate result
pytest 1500 passed, 16 skipped vs 1497 passed, 16 skipped on 6bc1d33 — +3 new cases, no regressions
mypy sqlite_utils tests Success: no issues found in 64 source files
flake8 clean
black . --check 64 files would be left unchanged
cog --check --diff README.md docs/*.rst exit 0, no regeneration needed

Not run: pyright sqlite_utils tests and ty check sqlite_utils — neither tool is available in this environment and I did not install them. The change adds no annotations and no new call shapes, so I do not expect them to fire, but I have not verified that.


📚 Documentation preview 📚: https://sqlite-utils--893.org.readthedocs.build/en/893/

…aths

Both branches that recurse into the CSV branch called rows_from_file()
without forwarding the two arguments, so the inner call applied its own
defaults and raised RowError even when the caller had explicitly asked
to ignore or capture the extras. The outer _extra_key_strategy wrapper
in each branch was dead code as a result.

Forward both arguments and drop the redundant wrapper, so the strategy
is applied once on the shared CSV path. With no options passed the
behaviour is unchanged.

Fixes simonw#892
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

rows_from_file() ignores ignore_extras= and extras_key= for anything that isn't Format.CSV

2 participants