Repository navigation
fix(catalog): count a bulk-file line that isn't valid UTF-8 as malformed and go on - #423
Merged
Merged
Conversation
…med and go on
MTG::Scryfall::Source#entry_or_malformed called String#strip first, which raises Encoding::CompatibilityError on a line that isn't valid UTF-8. Nothing rescued it, so each_entry aborted mid-file and the whole refresh failed on one bad byte instead of counting one malformed line as it does for any other bad line. It now checks valid_encoding? first and yields a Malformed with no key ("the line is not valid UTF-8"), the same first guard MTG::Scryfall::PriceQuotes has for the same file (spec 085, 38780d3). The two sources return different Malformed types, so each keeps its own one-line guard and no helper is shared.
Failing first: the new example in source_spec (a gzip file with a 0xFF byte in a line before a good card, expecting a Malformed and then the good record) failed with Encoding::CompatibilityError at source.rb:142; the whole spec file passes after (22 examples).
Closes #411
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…alformed (#411) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…heck's comment (#411) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
The guard is placed before both |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #411
A direct fix, routed with the coordinator (no spec). A bulk-file line that isn't valid UTF-8 is now counted as one malformed line and the refresh goes on, as for any other bad line.
Cause
MTG::Scryfall::Source#entry_or_malformedcalledString#stripfirst. On a line that isn't valid UTF-8 it raisesEncoding::CompatibilityError(orArgumentErrorwhen the bad byte comes first, which was already rescued). The first wasn't rescued, soeach_entryaborted mid-file and the whole refresh failed on one bad byte. Low impact (Scryfall's files should never hold such a line, and it failed safe), but one such line would have stopped every refresh until Scryfall fixed the file. Found in the implementation review of spec 085 (#364); the price source already has the fix.Change
app/models/mtg/scryfall/source.rb: the first line ofentry_or_malformedreturnsCatalog::Sources::Malformed.new(external_key: nil, error: "the line is not valid UTF-8")whenline.valid_encoding?is false, before any strip or parse. Same shape asMTG::Scryfall::PriceQuotes#each_card_record. No shared helper:Catalog::Sources::MalformedandPrices::Sources::Malformedare different types and each guard is one line, so the two stay as they are (the coordinator approved this).spec/models/mtg/scryfall/source_spec.rb: a gzip file with a0xFFbyte in a line before a good card, expecting aMalformedand then the good record.docs/architecture/catalog-operations.md: one sentence saying what counts as a malformed line now, A catalog refresh stops on one line with invalid UTF-8 instead of counting it as malformed #411 in Sources and a log row. Nothing else in the docs stated what counts as malformed.Evidence
Encoding::CompatibilityError: invalid byte sequence in UTF-8atsource.rb:142(String#strip); it passes with the guard. The wholesource_spec.rb: 22 examples, 0 failures (also after mergingmain).bin/ci checksgreen; the pre-push hook ran the fullbin/ciat95b29867(5048 examples) and passed.Implementation review
Opus 5.5, read-only: SPEC-ALIGNED. It confirmed that
valid_encoding?tests UTF-8 (the reader's lines come out inEncoding.default_external, as in the price source), that aMalformedwith no key is counted like the existing{not jsoncase (Catalog::Refresh#sync_entry), that the new example fails without the fix for the right reason, and that no wider rescue is justified. It asked for one comment correction (done: a bad first byte raisesArgumentError) and raised a possible gap, a lone surrogate escape making the refresh's digest encoding raise. I checked it withbin/rails runnerin the test environment: json 3.1.0 rejects a lone\ud800asJSON::ParserError, which the source already rescues, so there is no gap.🤖 Generated with Claude Code