rows_from_file() ignores ignore_extras= and extras_key= for anything that isn't explicitly format=Format.CSV.
The docstring says:
If a CSV or TSV file includes rows with more fields than are declared in the header a sqlite_utils.utils.RowError exception will be raised when you loop over the generator.
You can instead ignore the extra data by passing ignore_extras=True.
Or pass extras_key="rest" to put those additional values in a list in a key called rest.
In practice those two options only work for format=Format.CSV. For TSV — which the docstring names explicitly — and for the auto-detected path (format omitted) they are silently discarded, and the caller's own explicit request is contradicted by a RowError.
Both branches that recurse into the CSV branch call rows_from_file() without forwarding the two arguments, so the inner call uses its own defaults (ignore_extras=False, extras_key=None) and raises before the outer _extra_key_strategy() wrapper — which is dead code — ever sees the row:
>>> from io import BytesIO
>>> from sqlite_utils.utils import rows_from_file, Format
>>> rows, _ = rows_from_file(
... BytesIO(b"id\tname\r\n1\tCleo\toops"),
... format=Format.TSV,
... ignore_extras=True,
... )
>>> list(rows)
Traceback (most recent call last):
...
sqlite_utils.utils.RowError: Row {'id': '1', 'name': 'Cleo'} contained these extra values: ['oops']
The same happens with extras_key="_rest":
>>> rows, _ = rows_from_file(
... BytesIO(b"id\tname\r\n1\tCleo\toops"),
... format=Format.TSV,
... extras_key="_rest",
... )
>>> list(rows)
Traceback (most recent call last):
...
sqlite_utils.utils.RowError: Row {'id': '1', 'name': 'Cleo'} contained these extra values: ['oops']
With no options passed the behaviour is correct and should not change — RowError is still raised, as documented.
git log -S"_extra_key_strategy" points at d379f43 ("rows_from_file(... ignore_extras: bool, restkey: str), refs #440"), which introduced the strategy and both branches in one commit. The recursion has never forwarded the arguments, so this looks like an oversight in the original commit rather than a decision.
The existing test hardcoded format=Format.CSV (tests/test_rows_from_file.py:44), which is why it was never caught.
Proposed fix
Forward ignore_extras= and extras_key= to the inner call in both branches, and drop the now-redundant outer _extra_key_strategy wrapper so the strategy is applied once, on the shared CSV path that already implements it.
Note on scope
I would fix the auto-detect branch in the same commit, since it is the same root cause (same dropped-argument recursion, same dead wrapper) and would otherwise be a one-line follow-up. It interacts with #891: once the sniffer picks the right delimiter for a ragged file, the auto-detected path will honour ignore_extras too.
I did not add an auto-detect regression test, because csv.Sniffer returns the wrong delimiter on a small ragged input and pinning today's sniffing behaviour would bake in the bug #891 is about. The TSV test passes an explicit format, so it does not depend on the sniffer.
rows_from_file()ignoresignore_extras=andextras_key=for anything that isn't explicitlyformat=Format.CSV.The docstring says:
In practice those two options only work for
format=Format.CSV. For TSV — which the docstring names explicitly — and for the auto-detected path (formatomitted) they are silently discarded, and the caller's own explicit request is contradicted by aRowError.Both branches that recurse into the CSV branch call
rows_from_file()without forwarding the two arguments, so the inner call uses its own defaults (ignore_extras=False,extras_key=None) and raises before the outer_extra_key_strategy()wrapper — which is dead code — ever sees the row:The same happens with
extras_key="_rest":With no options passed the behaviour is correct and should not change —
RowErroris still raised, as documented.git log -S"_extra_key_strategy"points atd379f43("rows_from_file(... ignore_extras: bool, restkey: str), refs #440"), which introduced the strategy and both branches in one commit. The recursion has never forwarded the arguments, so this looks like an oversight in the original commit rather than a decision.The existing test hardcoded
format=Format.CSV(tests/test_rows_from_file.py:44), which is why it was never caught.Proposed fix
Forward
ignore_extras=andextras_key=to the inner call in both branches, and drop the now-redundant outer_extra_key_strategywrapper so the strategy is applied once, on the shared CSV path that already implements it.Note on scope
I would fix the auto-detect branch in the same commit, since it is the same root cause (same dropped-argument recursion, same dead wrapper) and would otherwise be a one-line follow-up. It interacts with #891: once the sniffer picks the right delimiter for a ragged file, the auto-detected path will honour
ignore_extrastoo.I did not add an auto-detect regression test, because
csv.Snifferreturns the wrong delimiter on a small ragged input and pinning today's sniffing behaviour would bake in the bug #891 is about. The TSV test passes an explicit format, so it does not depend on the sniffer.