Skip to content

fix(catalog): count a bulk-file line that isn't valid UTF-8 as malformed and go on - #423

Merged
plainprogrammer merged 4 commits into
mainfrom
093-catalog-invalid-utf8
Nov 14, 2026
Merged

plainprogrammer merged 4 commits into
mainfrom
093-catalog-invalid-utf8

Conversation

@plainprogrammer

Copy link
Copy Markdown
Owner

Closes #411

A direct fix, routed with the coordinator (no spec). A bulk-file line that isn't valid UTF-8 is now counted as one malformed line and the refresh goes on, as for any other bad line.

Cause

MTG::Scryfall::Source#entry_or_malformed called String#strip first. On a line that isn't valid UTF-8 it raises Encoding::CompatibilityError (or ArgumentError when the bad byte comes first, which was already rescued). The first wasn't rescued, so each_entry aborted mid-file and the whole refresh failed on one bad byte. Low impact (Scryfall's files should never hold such a line, and it failed safe), but one such line would have stopped every refresh until Scryfall fixed the file. Found in the implementation review of spec 085 (#364); the price source already has the fix.

Change

  • app/models/mtg/scryfall/source.rb: the first line of entry_or_malformed returns Catalog::Sources::Malformed.new(external_key: nil, error: "the line is not valid UTF-8") when line.valid_encoding? is false, before any strip or parse. Same shape as MTG::Scryfall::PriceQuotes#each_card_record. No shared helper: Catalog::Sources::Malformed and Prices::Sources::Malformed are different types and each guard is one line, so the two stay as they are (the coordinator approved this).
  • spec/models/mtg/scryfall/source_spec.rb: a gzip file with a 0xFF byte in a line before a good card, expecting a Malformed and then the good record.
  • docs/architecture/catalog-operations.md: one sentence saying what counts as a malformed line now, A catalog refresh stops on one line with invalid UTF-8 instead of counting it as malformed #411 in Sources and a log row. Nothing else in the docs stated what counts as malformed.

Evidence

  • Failing first: the new example failed with Encoding::CompatibilityError: invalid byte sequence in UTF-8 at source.rb:142 (String#strip); it passes with the guard. The whole source_spec.rb: 22 examples, 0 failures (also after merging main).
  • bin/ci checks green; the pre-push hook ran the full bin/ci at 95b29867 (5048 examples) and passed.

Implementation review

Opus 5.5, read-only: SPEC-ALIGNED. It confirmed that valid_encoding? tests UTF-8 (the reader's lines come out in Encoding.default_external, as in the price source), that a Malformed with no key is counted like the existing {not json case (Catalog::Refresh#sync_entry), that the new example fails without the fix for the right reason, and that no wider rescue is justified. It asked for one comment correction (done: a bad first byte raises ArgumentError) and raised a possible gap, a lone surrogate escape making the refresh's digest encoding raise. I checked it with bin/rails runner in the test environment: json 3.1.0 rejects a lone \ud800 as JSON::ParserError, which the source already rescues, so there is no gap.

🤖 Generated with Claude Code

plainprogrammer and others added 4 commits November 13, 2026 21:04
…med and go on

MTG::Scryfall::Source#entry_or_malformed called String#strip first, which raises Encoding::CompatibilityError on a line that isn't valid UTF-8. Nothing rescued it, so each_entry aborted mid-file and the whole refresh failed on one bad byte instead of counting one malformed line as it does for any other bad line. It now checks valid_encoding? first and yields a Malformed with no key ("the line is not valid UTF-8"), the same first guard MTG::Scryfall::PriceQuotes has for the same file (spec 085, 38780d3). The two sources return different Malformed types, so each keeps its own one-line guard and no helper is shared.

Failing first: the new example in source_spec (a gzip file with a 0xFF byte in a line before a good card, expecting a Malformed and then the good record) failed with Encoding::CompatibilityError at source.rb:142; the whole spec file passes after (22 examples).

Closes #411

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…alformed (#411)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…heck's comment (#411)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@brunnojob

Copy link
Copy Markdown

The guard is placed before both strip and JSON.parse, so malformed UTF-8 is classified as one malformed record instead of escaping the per-line parser path and aborting the whole refresh. It also keeps the existing blank-line behavior for valid input unchanged. The regression in this diff exercises the malformed-line count and continuation behavior. Static review; I did not run the Ruby suite.

@plainprogrammer
plainprogrammer marked this pull request as ready for review November 14, 2026 02:36
@plainprogrammer plainprogrammer self-assigned this Nov 14, 2026
@plainprogrammer
plainprogrammer merged commit c15a465 into main Nov 14, 2026
9 checks passed
@plainprogrammer
plainprogrammer deleted the 093-catalog-invalid-utf8 branch November 14, 2026 02:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A catalog refresh stops on one line with invalid UTF-8 instead of counting it as malformed

2 participants