Conversation
…aths Both branches that recurse into the CSV branch called rows_from_file() without forwarding the two arguments, so the inner call applied its own defaults and raised RowError even when the caller had explicitly asked to ignore or capture the extras. The outer _extra_key_strategy wrapper in each branch was dead code as a result. Forward both arguments and drop the redundant wrapper, so the strategy is applied once on the shared CSV path. With no options passed the behaviour is unchanged. Fixes simonw#892
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #892.
rows_from_file()documents that its extra-field options apply to "a CSV or TSV file", but they only ever worked forformat=Format.CSV.Both branches that recurse into the CSV branch called
rows_from_file()without forwarding the two arguments:So the inner call applied its own defaults —
ignore_extras=False,extras_key=None— and raisedRowErrorbefore the outer_extra_key_strategy()wrapper, which turned out to be dead code, ever saw the row. The caller had explicitly asked to ignore or capture the extras and got an exception instead.The fix forwards both arguments and drops the redundant outer wrapper, so the strategy is applied once, on the shared CSV path that already implements it. With no options passed nothing changes:
RowErroris still raised, as documented.Scope
I fixed the auto-detect branch (
formatomitted) in the same commit: it is the same root cause — same dropped-argument recursion, same dead wrapper — and would otherwise be a one-line follow-up. It does interact with #891 in a way worth stating so the two do not look contradictory: once the sniffer picks the right delimiter for a ragged file, the auto-detected path will honourignore_extrasas well. If you would rather split them, that is easy.I did not add an auto-detect regression test, because
csv.Snifferreturns the wrong delimiter on a small ragged input and pinning today's sniffing behaviour would bake in the bug #891 is about. The new test passes an explicit format, so it does not depend on the sniffer.Tests
test_rows_from_file_tsv_extra_fields_strategiesis a TSV copy of the existing CSV test, with the same three cases:ignore_extras=Truedrops the extra value,extras_key="_rest"collects it, and no options still raises.Before the fix, two of the three fail with:
and the frame shows
ignore_extras = False,extras_key = Noneinside_extra_key_strategydespite the caller passingTrue/"_rest".I also checked the three neighbouring cases by hand on
6bc1d33and on this branch. TSV withignore_extras=Trueand TSV withextras_keyboth raise before and return the documented result after; TSV with no options raises in both; the auto-detected path is unchanged here because the sniffer mis-detects this small ragged input (that is #891).The existing test hardcoded
format=Format.CSVattests/test_rows_from_file.py:44, which is why this was never caught.git log -S"_extra_key_strategy"points atd379f43("rows_from_file(... ignore_extras: bool, restkey: str), refs #440"), which introduced the strategy and both branches together. The recursion has never forwarded the arguments, so this is an oversight in the original commit rather than a decision.Checks
pytest6bc1d33— +3 new cases, no regressionsmypy sqlite_utils testsflake8black . --checkcog --check --diff README.md docs/*.rstNot run:
pyright sqlite_utils testsandty check sqlite_utils— neither tool is available in this environment and I did not install them. The change adds no annotations and no new call shapes, so I do not expect them to fire, but I have not verified that.📚 Documentation preview 📚: https://sqlite-utils--893.org.readthedocs.build/en/893/