Fix bulk card image downloader to use Scryfall CDN - #10928
Conversation
c1f6b16 to
e4dd5ad
Compare
Introduces CdnUuidCache which lazily loads res/cdn_uuid/{setCode}/{cn}.json
files and resolves CDN URLs (cards.scryfall.io, no rate limit) for card image
fetching. ImageFetcher and GuiDownloadFilteredCardImages prefer CDN URLs when
the asset files are present, falling back to the Scryfall API and then the
cardforge server as before.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
e4dd5ad to
6430183
Compare
- Add package-private `cdnBaseDirOverride` and `clearCacheForTesting()` so tests can supply a temp directory without triggering ForgeConstants/GUI init - Switch ensureSetLoaded to use the override when set (production path unchanged) - Log DEBUG with absolute path when a set directory is not found, making path-resolution failures visible in runtime logs - Add CdnUuidCacheTest covering happy path, language fallback, DFC front/back, same-UUID DFC, missing set (MISSING_SET sentinel caching), missing collector number, null inputs, and set-code case normalisation (16 tests total, all pass) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Both SwingImageFetcher and LibGDXImageFetcher only applied the .full → .fullborder path transform for api.scryfall.com URLs. CDN URLs (cards.scryfall.io) were saved as .full.jpg, which the game's image display code doesn't find — causing repeated re-download attempts. Add URL_SCRYFALL_CDN constant to ForgeConstants and use it in both fetchers so CDN downloads are treated identically to API downloads for file-path purposes. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
CdnUuidCache now loads per-set JSON from {cacheDir}/cdn_uuid/{set}.json.
On first lookup for a set the file is fetched from forge-extras (no rate
limit) and written to the local cache for all subsequent resolutions.
Returns null on any failure — callers fall back to the Scryfall API as before.
ForgeConstants: replace CDN_UUID_DIR (res/) with CACHE_CDN_UUID_DIR (cache/)
and FORGE_EXTRAS_CDN_UUID_URL.
Tests use localCacheDirOverride + remoteBaseUrlOverride (file:// in tests)
to verify the full local-hit and remote-fetch paths without network access.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
This PR has not been updated in a while nad has been marked on stale. Stale PRs will be auto closed |
|
Will tackle tonight or tomorrow night |
- Request Accept-Encoding: gzip when fetching a set's UUID JSON from forge-extras; raw.githubusercontent.com honors it (~48% smaller on the wire for this UUID-heavy JSON), decompressed transparently. - Store the local disk cache gzip-compressed too (.json.gz), saving space for whichever sets a user actually triggers a lookup for. The files committed to forge-extras itself stay plain, sorted text — compressing those would turn every regeneration into a full binary rewrite instead of the minimal diff the CLI now produces. Verified against the live forge-extras branch: server returns Content-Encoding: gzip (110,709 bytes) and decompresses back to the full 214,541-byte file. All 17 CdnUuidCacheTest cases still pass.
…est API Addresses the Discord discussion on forge-owned vs. user-owned CDN data: this adds a user-triggered path that doesn't depend on forge-extras at all, alongside (not instead of) the existing hosted data. Scryfall's /cards/manifest endpoint returns just id/set/collector_number /lang per card -- everything this lookup needs -- in ~15k-entry pages, without the ~100MB-2.5GB cost of a full bulk-data export. English coverage is currently ~8 pages (~25-30MB). It doesn't expose the actual CDN image URL, so a double-faced card's back face is assumed to share the front's id; verified against live Scryfall data that this holds for the overwhelming majority of DFCs (the CDN URL only differs by a front/back path segment), and the existing CDN-miss fallback already covers the rare exceptions. ScryfallManifestSync paginates the manifest (respecting its documented 10/minute rate limit), and supports incremental resync: entries are requested newest-image-update-first, with a persisted per-language watermark so a repeat sync typically stops after one page instead of re-walking the whole catalog. CdnUuidCache gains mergeSetEntries() (fills gaps without ever overwriting existing entries -- forge-extras/CLI data can carry a double-faced card's real distinct front/back UUIDs, which a manifest- derived guess must not clobber) and a public clearCache(), covering the simpler "let users purge it from settings" alternative to per-file expiry that came up in the same discussion. Both actions are wired into the mobile card-image-download screen, which is where this whole feature already lives (desktop never exposed "Download Card Images" -- only mobile/CardImageBrowserScreen does). 10 new tests in ScryfallManifestSyncTest (embedded HTTP server, no live network dependency) cover pagination, the incremental watermark, the never-overwrite merge policy, and cache clearing; all 27 CDN-related tests pass together.
Addresses tool4ever's review comment on PR Card-Forge#10928: a single-method class wrapping a URL format string didn't warrant its own file. CdnUuidCache.getCdnUrl() was the only production caller, so cdnUrl() moves there as a public static method; test call sites updated to match, and ScryfallBulkDataTest's formula assertion is folded into CdnUuidCacheTest instead of being dropped.
CdnUuidCache no longer fetches per-set JSON from the forge-extras repo. On a cache miss it now asks the new ScryfallSetSync to build that set's mapping on the spot from Scryfall's card search API (scoped to that one set, all languages), the client-side equivalent of what the old forge-extras/CLI generator did against a full bulk-data export. Unlike ScryfallManifestSync (which only ever sees a card's own id), the search API exposes each face's own image URL, so a double-faced card's rare genuinely-distinct back-face UUID is captured precisely via the new CdnUuidCache.mergeSetEntriesWithFaces, instead of assumed to match the front. This replaces the forge-scryfall-uuid-map CLI and forge-extras cdn_uuid data entirely -- both PRs are being closed in favor of this.
…age-downloader # Conflicts: # forge-gui-desktop/src/main/java/forge/util/SwingImageFetcher.java
…age-downloader # Conflicts: # forge-gui/res/languages/ko-KR.properties
No behavior change: condenses multi-paragraph javadoc down to one or two lines per method/class, drops inline comments that just restated an adjacent assert message or code branch, and removes a duplicated CDN/API/cardforge priority list that appeared in both a class and method javadoc.
|
For context
|
|
I'll happily rewrite my code #11572 once your PR gets merged — at the end of the day, all I wanted was for users to be able to download cards in their own language. I also think the unique cards behavior I mentioned is interesting: when "unique cards" is on and that same edition has the card printed in more than one language, I'd like the one downloaded to be the one in the user's language. And having the download language as its own setting is interesting too — a user might have the UI in one language but prefer to download cards in a different one, so it makes sense to let them choose that separately. |
Fetch card images from the Scryfall CDN instead of the rate-limited API
Summary
Card image downloads (desktop, mobile, and the bulk image browser) currently go through
api.scryfall.com, which rate-limits and is the main source of failed/slow downloads. Scryfallalso serves the same files unthrottled from
cards.scryfall.io, but that requires alreadyknowing the card's UUID.
This PR adds a local, disk-backed cache (
CdnUuidCache) that maps(set, collector number, lang)→ Scryfall UUID, populated on demand from Scryfall's search API (
ScryfallSetSync), and prefersCDN URLs built from that cache over the old API path. The API and the existing cardforge-hosted
server remain as fallbacks, so nothing regresses if a card isn't in the cache yet.
How image resolution works now
Request timeline for one card face (cache miss on first call, hit on a later call once the
background sync has landed):
sequenceDiagram participant Fetcher as ImageFetcher participant Cache as CdnUuidCache participant Disk as cdn_uuid/*.json.gz participant Sync as ScryfallSetSync (bg thread) participant API as api.scryfall.com Fetcher->>Cache: getCdnUrl(set, cn, lang, face) Cache->>Disk: read set file Disk-->>Cache: no entry for cn/lang Cache->>Disk: record timestamped miss Cache->>Sync: queue set for sync (async) Cache-->>Fetcher: null Fetcher->>Fetcher: fall back to Scryfall API / cardforge URL Sync->>API: GET /cards/search?q=set:... API-->>Sync: card pages (paginated) Sync->>Disk: merge cn/lang → uuid Note over Fetcher,Disk: next request for the same face Fetcher->>Cache: getCdnUrl(set, cn, lang, face) Cache->>Disk: read set file Disk-->>Cache: uuid found Cache-->>Fetcher: cards.scryfall.io URL Fetcher->>Fetcher: download from CDNLookups never block on network: a miss is served from fallback immediately, and the sync runs on
a background thread pool. Unresolved
(cn, lang)pairs are negative-cached with a timestamp sothey aren't retried on every lookup, only after a day.
How the pieces connect
flowchart LR subgraph Callers IF[ImageFetcher] GD[GuiDownloadFilteredCardImages] UI[CardImageBrowserScreen] end subgraph Cache Layer CUC[CdnUuidCache] SSS[ScryfallSetSync] Disk[(cdn_uuid/*.json.gz)] end subgraph Downloaders SW[SwingImageFetcher] LG[LibGDXImageFetcher] end IF -->|getCdnUrl| CUC GD -->|getCdnUrl| CUC UI -->|clearCache| CUC CUC <--> Disk CUC -->|queueSync| SSS SSS -->|merge results| CUC SSS --> ScryfallAPI[api.scryfall.com/cards/search] IF --> SW IF --> LG SW --> CDN[cards.scryfall.io] SW --> ScryfallDL[api.scryfall.com/cards/*] SW --> Cardforge[cardforge hosted server] LG --> CDN LG --> ScryfallDL LG --> CardforgeChanges
CdnUuidCache(new) — thread-safe, gzip-JSON-backed cache; resolves a UUID, or queues abackground sync and returns
null.ScryfallSetSync(new) — pulls one set's cards from Scryfall's search API (paginated,rate-limit-respecting), reading each face's own
image_urisso double-faced cards with distinctfront/back art resolve correctly.
ImageFetcher.addScryfallUrl/GuiDownloadFilteredCardImages.buildUrl— try CDN first,then Scryfall API, then cardforge.
SwingImageFetcher/LibGDXImageFetcher— recognize CDN URLs alongside API URLs soexisting border/rounding fixups still apply.
CardImageBrowserScreen— adds a "Clear CDN Image Lookup Cache" button.gsondependency inforge-gui/pom.xmlfor cache (de)serialization.Testing
CdnUuidCacheTest— URL formula, language fallback, DFC face resolution, miss recording/retry,cache upgrade on real data.
ScryfallSetSyncTest— single/double-faced resolution, pagination, 404 handling, neveroverwriting existing entries, miss retry staleness.
For reviewers — long-term maintenance concerns
ScryfallSetSyncis a second, independentintegration point with Scryfall beyond the existing per-card download API. If Scryfall changes
its search response shape or CDN URL format, both
ScryfallSetSync.addCard/uuidFromUrlandCdnUuidCache.cdnUrl's hand-built URL formula need to be checked —cdnUrldoes not verifythe URL it builds actually resolves.
mergeSetEntriesWithFacesnever overwriting real entries: thisis the invariant that keeps a bad/stale sync from clobbering a working UUID. Any future change
to the merge logic should preserve "real data always wins over a miss, and existing real data is
never replaced."
{cn: {lang: uuid | [front, back] | {miss}}}isread directly with no migration path. A future format change needs either a version bump or to
stay a strict superset of the current shape, otherwise old cache files silently parse wrong.
{cacheDir}/cdn_uuid/and only removed via the manual "Clear CDN Image Lookup Cache" button.There's no automatic pruning or staleness eviction beyond the per-entry miss-retry window —
worth confirming this is acceptable for long-running installs.
ThreadUtilservice pool:syncPendingSetsisfire-and-forget with no cap on concurrent set syncs; a user browsing many unseen sets in quick
succession could queue many simultaneous Scryfall requests. Rate limiting is per-request
(
REQUEST_INTERVAL_MS) within a single sync, not across syncs.