diff --git a/.claude/memory/MEMORY.md b/.claude/memory/MEMORY.md index 66aab3b5..8f899ac5 100644 --- a/.claude/memory/MEMORY.md +++ b/.claude/memory/MEMORY.md @@ -90,3 +90,4 @@ - [No git stash or sqlite3 in briefs](no-git-stash-in-briefs.md) — forbid git stash/reset/checkout and the sqlite3 command (it prompts the maintainer) in every subagent brief (except restoring a mutation-tested file); a stash/pop on a clean tree applied an unrelated old stash in spec 080 - [Unset `config.x` answers nil](rails-config-x-unset.md) — an unset `Rails.configuration.x` key is an empty OrderedOptions; assert a boot reading's class, not only nil attributes - [Mutation proof in place of failing-first](mutation-proof-when-behaviour-already-built.md) — only where an earlier task already built the behaviour; each mutation and its failure go in the commit body (coordinator, spec 085 Gate B) +- [Harness changes the model mid-run](harness-changes-the-model-mid-run.md) — a worker's main session can change model without being asked; state the model per commit at gates, answer from what the harness reports diff --git a/.claude/memory/harness-changes-the-model-mid-run.md b/.claude/memory/harness-changes-the-model-mid-run.md new file mode 100644 index 00000000..e7dd93e7 --- /dev/null +++ b/.claude/memory/harness-changes-the-model-mid-run.md @@ -0,0 +1,12 @@ +--- +name: harness-changes-the-model-mid-run +description: a worker's main session can change model mid-run without being asked; state the model per commit at gates +metadata: + type: feedback +--- + +During spec 089's run the harness moved a worker's main session from Sonnet 5.5 to Opus 5.5 and back (the commit attribution reminder changed with it), without the worker or the coordinator asking; a sibling worker saw the same. + +**Why:** the maintainer chose the model split for the run and wants it visible while it goes, not at the end (coordinator's brief, 2026-11-13). + +**How to apply:** at every gate and in `worker_done`, say which model wrote what, by commit range; when asked which model the session is on, answer from what the harness reports now and whether it changed, and change nothing. diff --git a/CLAUDE.md b/CLAUDE.md index 5ae5f6bd..d2c4e4f4 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -4,7 +4,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co ## Project Status -Collector is a self-hostable, multi-tenant web app for tracking collectibles, starting with Magic: The Gathering cards. The Rails app skeleton exists (stack, RSpec, `bin/ci`, Compose and Kamal deployment). Feature 002 added a collectible-agnostic catalog core (`app/models/catalog/`) with an MTG extension (`app/models/mtg/`), Scryfall bulk-data ingestion via a background refresh, and a card search at `/catalog/entries`, which has needed sign-in since accounts (spec 004) unless the operator opens it with `COLLECTOR_PUBLIC_CATALOG` (spec 023). Accounts (row-level tenancy through `Current.account`) and collections of lots (spec 004, list view spec 006) exist, and the card scanner (specs 005, 007–009) adds scanned cards to the collection through its confirm flow. Admins start and watch catalog refreshes and art index builds at `/admin/catalog`, and see, retry and discard background jobs at `/admin/jobs` (spec 015). A collector's data has a documented export format, and a collector downloads, previews and applies an export file from the account menu or More (`CollectionExport`, `CollectionImport`, `CollectionImport::Apply`; [`docs/architecture/data-portability.md`](docs/architecture/data-portability.md)). Every page's footer ends with Collector's licence line, AGPL-3.0 with a link to the source of the running build, and shows the running version and short SHA, which the image's build passes as `COLLECTOR_VERSION`, `COLLECTOR_REVISION` and `COLLECTOR_SOURCE_URL` and `Collector::BuildFacts` reads once at boot; `bin/image-smoke` fails an image without them (spec 084). The line ends with a link to `/licences`, which anyone can open and which lists the licences of the third-party software and assets in the image: the gems from the running bundle, the OCR engine, the importmap JavaScript, the fonts and the icons from `config/third_party_licences.yml`, kept honest by a guard spec (`Collector::ThirdPartyLicences`, `LicencesController`, spec 087), and `bin/ci checks` fails a gem or a manifest entry under a licence that `config/licence_policy.yml` doesn't allow or name as an exception (`Collector::ThirdPartyLicences::Policy` and `Check`, `bin/rails licences:check`, spec 091, [ADR 0031](docs/adr/0031-a-dependency-licence-is-a-decision.md)). A collector adds cards from a set by collector number at `/collection/number_entry` ("Add by number", spec 026): one line per number, quantity and finish, each added at once to a per-account list with Undo (`NumberEntry`, `NumberEntry::Line`), which Done clears; the list is working state, left out of the export. A collector manages the places their collection is kept in at `/locations` (`Location`, spec 029) and records which location a lot is in when adding or editing it, or moves some or all of a lot's copies with Move copies (`Lot::Move`, spec 032; a location that holds copies can't be removed; export format version 3) and moves every copy of many lots at once from bulk mode with Move to… (`Lot::BulkMove`, spec 041), opens a location's lots in bulk mode from its removal refusal page with one button (`Locations::SelectionsController`, spec 059), and the collection's grid tiles and table rows show it (spec 033) and filter by it (spec 037). A collector creates, renames and removes decks at `/decks` (`Deck`, `Deck::Creation`, `Catalog::Formats`, spec 062, [ADR 0028](docs/adr/0028-decks-are-locations-with-zones.md)): a deck is a location of kind `deck` whose zones are child locations that hold its copies, the collectible extension registers the formats, a lot in a zone is shown as "Deck › Zone", the export format (version 4) carries decks, a collector changes a deck's format at `/decks/:id/format/edit` (`Deck::FormatChange`, spec 064), opens a deck's lots in bulk mode from its removal refusal page (`Decks::SelectionsController`, spec 067), and opens a deck at `/decks/:id` to read the cards they own in it, zone by zone, with their cost and legality (`Deck::Page`, the seam `Catalog::DeckCards`, spec 066), and the deck's shape, its mana curve, card types and coloured mana symbols, in a panel beside the zones (`Catalog::DeckShape`, `MTG::DeckShape`, spec 069), and records the cards a deck's zone needs and they don't own as proxies, shown beside the owned copies and counted in a deck's cards but never as owned (`Deck::Proxy`, spec 068): a change of format moves them into Considering and the export carries them (format version 4 has `proxies`). A collector swaps a proxy for a free copy they own (Use my copy) or a new copy (Add a new copy) from the row's menu, and the proxy shrinks or goes in the same write (`Deck::Proxy::UseCopies`, `Deck::Proxy::AddCopy`, `Lot.free`, `Lot::MoveIntoZone`, spec 070). A collector adds a card to a deck from its card page, their free copies or a proxy (`Deck::Addition`, `Lot::FreeCopies`, spec 071). The card page shows how many copies are free to use and which decks hold the rest, in its stats and an "In decks" table (`Catalog::CardOverview`, `Deck::CardUses`, spec 073). A collector checks a deck on demand from its page, its size, copy limits and the cards with owned copies and proxies in one zone, and nothing about the check is stored (`Catalog::DeckChecks`, `MTG::Collecting.deck_rules`, `Decks::ChecksController`, spec 079). An operator backs up a running instance and restores it onto a fresh volume with two rake tasks, `backup:create[DIRECTORY]` and `backup:restore[FILE]` (`Collector::Backup`, `lib/collector/backup/`, spec 030, ADR 0024). Market prices will live in a fifth SQLite database, `prices` (`PricesRecord`, `db/prices_migrate/`, `db/prices_structure.sql`; spec 081, ADR 0029; [prices](docs/architecture/prices.md)), whose five tables exist, with a price source contract and Scryfall's prices through it (`Prices::Sources`, `MTG::Scryfall::PriceQuotes`, `PriceSource.sync!`; spec 085, #364) and no refresh or reader yet, and the backup holds it beside the primary (backup format version 2); a server start refuses a volume that holds the prices database and no primary (`Collector::StartCheck`, `start:check`, spec 086). CI proves an upgrade from the previous release's published image (`bin/upgrade-smoke`, `script/upgrade/`, spec 031, #42; [release and deploy](docs/architecture/release-and-deploy.md)). License: AGPL-3.0. +Collector is a self-hostable, multi-tenant web app for tracking collectibles, starting with Magic: The Gathering cards. The Rails app skeleton exists (stack, RSpec, `bin/ci`, Compose and Kamal deployment). Feature 002 added a collectible-agnostic catalog core (`app/models/catalog/`) with an MTG extension (`app/models/mtg/`), Scryfall bulk-data ingestion via a background refresh, and a card search at `/catalog/entries`, which has needed sign-in since accounts (spec 004) unless the operator opens it with `COLLECTOR_PUBLIC_CATALOG` (spec 023). Accounts (row-level tenancy through `Current.account`) and collections of lots (spec 004, list view spec 006) exist, and the card scanner (specs 005, 007–009) adds scanned cards to the collection through its confirm flow. Admins start and watch catalog refreshes and art index builds at `/admin/catalog`, and see, retry and discard background jobs at `/admin/jobs` (spec 015). A collector's data has a documented export format, and a collector downloads, previews and applies an export file from the account menu or More (`CollectionExport`, `CollectionImport`, `CollectionImport::Apply`; [`docs/architecture/data-portability.md`](docs/architecture/data-portability.md)). Every page's footer ends with Collector's licence line, AGPL-3.0 with a link to the source of the running build, and shows the running version and short SHA, which the image's build passes as `COLLECTOR_VERSION`, `COLLECTOR_REVISION` and `COLLECTOR_SOURCE_URL` and `Collector::BuildFacts` reads once at boot; `bin/image-smoke` fails an image without them (spec 084). The line ends with a link to `/licences`, which anyone can open and which lists the licences of the third-party software and assets in the image: the gems from the running bundle, the OCR engine, the importmap JavaScript, the fonts and the icons from `config/third_party_licences.yml`, kept honest by a guard spec (`Collector::ThirdPartyLicences`, `LicencesController`, spec 087), and `bin/ci checks` fails a gem or a manifest entry under a licence that `config/licence_policy.yml` doesn't allow or name as an exception (`Collector::ThirdPartyLicences::Policy` and `Check`, `bin/rails licences:check`, spec 091, [ADR 0031](docs/adr/0031-a-dependency-licence-is-a-decision.md)). A collector adds cards from a set by collector number at `/collection/number_entry` ("Add by number", spec 026): one line per number, quantity and finish, each added at once to a per-account list with Undo (`NumberEntry`, `NumberEntry::Line`), which Done clears; the list is working state, left out of the export. A collector manages the places their collection is kept in at `/locations` (`Location`, spec 029) and records which location a lot is in when adding or editing it, or moves some or all of a lot's copies with Move copies (`Lot::Move`, spec 032; a location that holds copies can't be removed; export format version 3) and moves every copy of many lots at once from bulk mode with Move to… (`Lot::BulkMove`, spec 041), opens a location's lots in bulk mode from its removal refusal page with one button (`Locations::SelectionsController`, spec 059), and the collection's grid tiles and table rows show it (spec 033) and filter by it (spec 037). A collector creates, renames and removes decks at `/decks` (`Deck`, `Deck::Creation`, `Catalog::Formats`, spec 062, [ADR 0028](docs/adr/0028-decks-are-locations-with-zones.md)): a deck is a location of kind `deck` whose zones are child locations that hold its copies, the collectible extension registers the formats, a lot in a zone is shown as "Deck › Zone", the export format (version 4) carries decks, a collector changes a deck's format at `/decks/:id/format/edit` (`Deck::FormatChange`, spec 064), opens a deck's lots in bulk mode from its removal refusal page (`Decks::SelectionsController`, spec 067), and opens a deck at `/decks/:id` to read the cards they own in it, zone by zone, with their cost and legality (`Deck::Page`, the seam `Catalog::DeckCards`, spec 066), and the deck's shape, its mana curve, card types and coloured mana symbols, in a panel beside the zones (`Catalog::DeckShape`, `MTG::DeckShape`, spec 069), and records the cards a deck's zone needs and they don't own as proxies, shown beside the owned copies and counted in a deck's cards but never as owned (`Deck::Proxy`, spec 068): a change of format moves them into Considering and the export carries them (format version 4 has `proxies`). A collector swaps a proxy for a free copy they own (Use my copy) or a new copy (Add a new copy) from the row's menu, and the proxy shrinks or goes in the same write (`Deck::Proxy::UseCopies`, `Deck::Proxy::AddCopy`, `Lot.free`, `Lot::MoveIntoZone`, spec 070). A collector adds a card to a deck from its card page, their free copies or a proxy (`Deck::Addition`, `Lot::FreeCopies`, spec 071). The card page shows how many copies are free to use and which decks hold the rest, in its stats and an "In decks" table (`Catalog::CardOverview`, `Deck::CardUses`, spec 073). A collector checks a deck on demand from its page, its size, copy limits and the cards with owned copies and proxies in one zone, and nothing about the check is stored (`Catalog::DeckChecks`, `MTG::Collecting.deck_rules`, `Decks::ChecksController`, spec 079). An operator backs up a running instance and restores it onto a fresh volume with two rake tasks, `backup:create[DIRECTORY]` and `backup:restore[FILE]` (`Collector::Backup`, `lib/collector/backup/`, spec 030, ADR 0024). Market prices will live in a fifth SQLite database, `prices` (`PricesRecord`, `db/prices_migrate/`, `db/prices_structure.sql`; spec 081, ADR 0029; [prices](docs/architecture/prices.md)), whose five tables exist, with a price source contract and Scryfall's prices through it (`Prices::Sources`, `MTG::Scryfall::PriceQuotes`, `PriceSource.sync!`; spec 085, #364) a daily refresh that records each price change once (`Prices::Refresh`, `Prices::RefreshJob`, `PriceRefreshRun`, `CurrentPrice`, `PriceChange`; spec 089, #365) and no reader yet (#366 to #369 are still to come), and the backup holds it beside the primary (backup format version 2); a server start refuses a volume that holds the prices database and no primary (`Collector::StartCheck`, `start:check`, spec 086). CI proves an upgrade from the previous release's published image (`bin/upgrade-smoke`, `script/upgrade/`, spec 031, #42; [release and deploy](docs/architecture/release-and-deploy.md)). License: AGPL-3.0. Mission and principles: `.claude/memory/foundation.md`. Goal and spec rules: `docs/north-star.md`. Order of work: `docs/roadmap.md`. Decisions: `docs/adr/` (index in its README). Current rules per subsystem: `docs/architecture/` (area docs). Detailed coding rules: `.claude/rules/`. Feature specs: `docs/specs/` (index: `docs/specs/README.md`). diff --git a/README.md b/README.md index fef868fe..cb308236 100644 --- a/README.md +++ b/README.md @@ -454,7 +454,7 @@ scp root@192.168.0.1:/srv/collector-backups/collector-backup-20261021T142100Z.ta The file is mode 0600 and owned by uid 1000 on the server; Kamal logs in as root by default (`ssh:` in `config/deploy.yml`), which can read it. Any other login needs `sudo`. -**When to back up:** before every upgrade, and at least daily for a collection that changes daily. From a host's cron, for example (02:17 every day, clear of Monday 03:00 to 05:00 in the container's time zone, UTC unless you set one, when the weekly catalog refresh writes the most and a backup would hold the database's write-ahead log back): +**When to back up:** before every upgrade, and at least daily for a collection that changes daily. From a host's cron, for example (02:17 every day, clear of Monday 03:00 to 05:00 in the container's time zone, UTC unless you set one, when the weekly catalog refresh writes the most and a backup would hold the database's write-ahead log back, and clear of 11:30 in the same time zone, when the daily price refresh starts writing the prices database, which a backup's read transaction holds back in the same way): ```cron 17 2 * * * cd /srv/collector && docker compose run --rm -T -v /srv/collector-backups:/backups web bin/rails "backup:create[/backups]" >> /var/log/collector-backup.log 2>&1 diff --git a/app/jobs/prices/refresh_job.rb b/app/jobs/prices/refresh_job.rb new file mode 100644 index 00000000..7abb404e --- /dev/null +++ b/app/jobs/prices/refresh_job.rb @@ -0,0 +1,29 @@ +# Runs the daily price refresh for a price source, in the background (spec 089). Queued daily by config/recurring.yml, +# or by hand from `bin/rails runner`; there is no button or task until the admin page (#367). +# +# No error is discarded: a failed job shows under Failed on `/admin/jobs`, which is how an operator learns of a guard that +# fired before #367 shows the runs. A departure from `background-jobs.md`'s `discard_on` advice, whose example is a +# `DeserializationError`: the argument is a string, and a discarded job would hide a guard that fired (spec 089 decision 7). +# +# @see file:prices.md +class Prices::RefreshJob < ApplicationJob + # How long the queue holds a second refresh of the source back when the first never reports its end: a waiting refresh + # must never start while another's job may still be working (the catalog refresh's value, spec 089 decision 7). The + # run's own stall rule is `PriceRefreshRun::STALL_AFTER`, in minutes. + LOCK_FOR = 6.hours + + queue_as :sync + + limits_concurrency to: 1, key: ->(source_key, *) { source_key }, duration: LOCK_FOR + + # The download's size error is retried too: a file republished mid-download gives exactly it, and the retry asks the + # index again (spec 089 decision 7). + retry_on Catalog::Sources::TransientError, Catalog::Sources::IntegrityError, ActiveRecord::StatementTimeout, + wait: :polynomially_longer, attempts: 3 + # After retry_on, so that it is tried first: a rate-limited source is a TransientError too, but the job comes back when + # the source asked. It reports no progress: a rate limit ends a refresh before any row is written. + include Catalog::RateLimitRetry + + # @param source_key [String] a key of `Prices.sources` + def perform(source_key) = Prices::Refresh.new(source_key, job_id:).call +end diff --git a/app/models/catalog/sources.rb b/app/models/catalog/sources.rb index df705483..b1c411ba 100644 --- a/app/models/catalog/sources.rb +++ b/app/models/catalog/sources.rb @@ -8,7 +8,9 @@ # .alternate_names(entries) optional: [[identity id, name], …] the name index adds to identity names # #languages sorted language codes to ingest (raises ConfigurationError) # #current_version(languages:) opaque String naming the current source data -# #download(version, dir:) Pathname of the verified local copy (IntegrityError, TransientError) +# #download(version, dir:) Pathname of the verified local copy (IntegrityError, TransientError). Two callers +# (the catalog refresh and a price refresh) may call it for one version at once, so it +# must tolerate concurrent calls: a partial download is never shared between calls # #each_set { |SetRecord| } # #each_entry(path, languages:) { |EntryRecord or Malformed| } streamed, filtered to languages # #reapply? optional: true to apply a version again although it was applied (asked before a skip) diff --git a/app/models/current_price.rb b/app/models/current_price.rb new file mode 100644 index 00000000..df3bf60b --- /dev/null +++ b/app/models/current_price.rb @@ -0,0 +1,11 @@ +# The latest known price of one printing in one series from one source, in the prices database (spec 081; spec 089). +# Keyed by printing (a catalog entry's id, with no foreign key: `prices.md`), source and series; the table is +# WITHOUT ROWID with no other index, so every lookup goes by printing, the key's first column. +# +# The daily refresh writes it with `upsert_all` and `delete_all` by key, which skip validations on purpose: each row comes +# from a validated `Prices::Sources::Quote`, and the table's own checks refuse the rest. +# +# @see file:prices.md +class CurrentPrice < PricesRecord + self.primary_key = %i[catalog_entry_id price_source_id price_series_id] +end diff --git a/app/models/mtg/scryfall/source.rb b/app/models/mtg/scryfall/source.rb index e462abde..afab4dfe 100644 --- a/app/models/mtg/scryfall/source.rb +++ b/app/models/mtg/scryfall/source.rb @@ -5,6 +5,10 @@ class MTG::Scryfall::Source LANGUAGES = %w[en es fr de it pt ja ko ru zhs zht he la grc ar sa ph qya].freeze LANGUAGES_ENV = "COLLECTOR_MTG_LANGUAGES" KEEP_DOWNLOADS = 2 + # A partial file this old belongs to a killed download: a live one is far younger (a job that never reports ends at its + # 6-hour lock). Each download writes its own partial file, so a killed one leaves an orphan for `prune` to remove + # (spec 089 decision 1, which supersedes spec 002 FR-3's "a partial file"). + STALE_PARTIAL_AFTER = 1.day def self.entry_extension_model = MTG::Printing def self.identity_extension_model = MTG::Card @@ -122,8 +126,12 @@ def artwork_ids_missing? = MTG::Printing.exists? && !MTG::Printing.where.not(ill def mana_values_missing? = MTG::Card.exists? && !MTG::Card.where.not(mana_value: nil).exists? + # Each call writes its own partial file, so two downloads of one version (the catalog refresh and a price refresh share + # the directory) never share one; the verified file is renamed into place, which replaces any earlier copy atomically. + # A failed download deletes its own partial file: an orphan of up to 408 MB would otherwise wait for a later + # successful download to prune it. def fetch(url, path, expected) - partial = Pathname("#{path}.part") + partial = Pathname("#{path}.#{SecureRandom.hex(4)}.part") File.open(partial, "wb") do |io| @client.download(url, to: io) { |received| @progress&.call(received, expected) } end @@ -132,10 +140,19 @@ def fetch(url, path, expected) actual = partial.size partial.delete raise Catalog::Sources::IntegrityError, "#{path.basename}: downloaded #{actual} bytes, Scryfall published #{expected}" + rescue StandardError + FileUtils.rm_f(partial) if partial + raise end + # Keeps the newest KEEP_DOWNLOADS files and drops stale partial files. A concurrent download may rename or delete a + # file between the listing and its use, so each file is judged, and removed, tolerating that (spec 089 decision 1). def prune(dir) - dir.glob("*.jsonl.gz").sort_by(&:mtime).reverse.drop(KEEP_DOWNLOADS).each(&:delete) + aged = ->(paths) { paths.filter_map { |path| [ path.mtime, path ] rescue nil } } # a file may vanish between glob and stat + aged.call(dir.glob("*.jsonl.gz")).sort_by(&:first).reverse.drop(KEEP_DOWNLOADS).each { |_mtime, path| FileUtils.rm_f(path) } + aged.call(dir.glob("*.part")).select { |mtime, _path| mtime < STALE_PARTIAL_AFTER.ago }.each { |_mtime, path| FileUtils.rm_f(path) } + rescue Errno::ENOENT + nil # the directory itself was removed under us end # The first check, before any strip or parse: `String#strip` raises Encoding::CompatibilityError on a line that isn't diff --git a/app/models/price_change.rb b/app/models/price_change.rb new file mode 100644 index 00000000..8f9f9555 --- /dev/null +++ b/app/models/price_change.rb @@ -0,0 +1,10 @@ +# One row of price history, in the prices database: the price of a printing in a series from a source on the day it +# differed from the last one recorded; a row with no cents records that the price went away that day (spec 081; spec +# 089). WITHOUT ROWID, keyed by printing, source, series and day, with no other index. Written by the daily refresh with +# `upsert_all` on that key, so a second write for the same day replaces the first (validations are skipped on purpose, +# as for `CurrentPrice`). +# +# @see file:prices.md +class PriceChange < PricesRecord + self.primary_key = %i[catalog_entry_id price_source_id price_series_id recorded_on] +end diff --git a/app/models/price_refresh_run.rb b/app/models/price_refresh_run.rb new file mode 100644 index 00000000..768ac224 --- /dev/null +++ b/app/models/price_refresh_run.rb @@ -0,0 +1,113 @@ +# One run of the daily price refresh, in the prices database (spec 089 decision 2): its identity (source, the file's +# day, version and languages), its times, its status and the counts of what it did to rows. The run is also the guard +# against overlap and against a run left behind (decision 7). Measurements of cost are #366's. +# +# `run_on` is nil until the file's date is known; `error` holds why a run failed or was skipped. +# +# @see file:prices.md +class PriceRefreshRun < PricesRecord + # What a run counts (spec 089 decision 2); the columns have these names and default to 0. + COUNTS = %i[quotes_seen unchanged repriced added removed removals_withheld unknown_printings malformed].freeze + # A running run with no progress for this long is left behind. The catalog refresh's own threshold (spec 089 decision 7). + STALL_AFTER = 15.minutes + # The most often the run row is touched while a download reports progress, so a long download never reads as stalled. + HEARTBEAT_EVERY = 30.seconds + + belongs_to :price_source + + scope :running, -> { where(status: "running") } + scope :finished, -> { where(status: "finished") } + + # Opens a run for the source, or records that none could be opened. In one transaction it closes each running run that + # was left behind (`#left_behind?`) as `failed` with `interrupted`; then, if a run still runs, it records the new run + # as `skipped` with `already running` (nothing else is touched), otherwise as `running`. + # + # @param source [PriceSource] + # @param job_id [String, nil] the starting job's Active Job id + # @param now [Time] + # @return [PriceRefreshRun] `running`, or `skipped` when another run is working + def self.start!(source, job_id:, now: Time.current) + transaction do + running.where(price_source: source).select { it.left_behind?(job_id:, now:) } + .each { it.update!(status: "failed", error: "interrupted", finished_at: now) } + if running.exists?(price_source_id: source.id) + create!(price_source: source, status: "skipped", error: "already running", job_id:, started_at: now, finished_at: now) + else + create!(price_source: source, status: "running", job_id:, started_at: now) + end + end + end + + # The finished runs of a source for one file and language set: what "applied" means (spec 089 decision 5). + # + # @return [ActiveRecord::Relation] + def self.applied(source, version, languages) = finished.where(price_source: source, source_version: version, languages:) + + # The latest day a finished or failed run of the source read a file for (a failed run may have written rows); skipped + # and running runs don't count (spec 089 decision 3). + # + # @return [Date, nil] + def self.latest_run_on(source) = where(price_source: source, status: %w[finished failed]).maximum(:run_on) + + # Whether nothing works on this running run any more: its last progress is older than STALL_AFTER, or it carries the + # starting job's own id (the queue re-ran it after a restart). + # + # @param job_id [String, nil] the Active Job id of the job now starting a run + # @param now [Time] the time the stall is measured against + # @return [Boolean] + def left_behind?(job_id:, now: Time.current) = updated_at < now - STALL_AFTER || (job_id.present? && self.job_id == job_id) + + # Whether the run is still open (not finished, skipped or failed). + # + # @return [Boolean] + def running? = status == "running" + + # Records the file this run is for. + # + # @param version [String] the catalog source's version (the file's name without its extension) + # @param languages [String] the sorted language set, comma-joined + # @return [true] + def record_file!(version:, languages:) = update!(source_version: version, languages:) + + # Ends the run as skipped, with the reason in `error`; `run_on` when it is known. + # + # @param reason [String] + # @param run_on [Date, nil] + # @return [true] + def skip!(reason, run_on: nil) = update!(status: "skipped", error: reason, run_on: run_on || self.run_on, finished_at: Time.current) + + # Ends the run as finished with its counts (keys outside COUNTS are ignored). + # + # @param counts [Hash{Symbol => Integer}] + # @return [true] + def finish!(counts) = update!(status: "finished", finished_at: Time.current, **counted(counts)) + + # Ends the run as failed with `": "` and the counts so far. + # + # @param error [StandardError] + # @param counts [Hash{Symbol => Integer}] + # @return [true] + def fail!(error, counts) = update!(status: "failed", error: "#{error.class}: #{error.message}", finished_at: Time.current, **counted(counts)) + + # Stores the counts so far and counts as progress (the stall rule reads `updated_at`). Written outside any batch's + # transaction. + # + # @param counts [Hash{Symbol => Integer}] + # @return [true] + def progress!(counts) = update!(**counted(counts), updated_at: Time.current) + + # A progress callback for a download: touches the run at most once per HEARTBEAT_EVERY. It ignores its arguments (a + # download's progress callback passes the bytes done and the total). + # + # @return [Boolean, nil] true when it touched the run, nil when it was too soon to touch it again + def heartbeat(*) + now = Time.current + return if @last_beat && now - @last_beat < HEARTBEAT_EVERY + + @last_beat = now + touch + end + + private + def counted(counts) = counts.slice(*COUNTS) +end diff --git a/app/models/prices/refresh.rb b/app/models/prices/refresh.rb new file mode 100644 index 00000000..3e968fd2 --- /dev/null +++ b/app/models/prices/refresh.rb @@ -0,0 +1,253 @@ +# One run of the daily price refresh for one price source (spec 089): asks the collectible's catalog source for the bulk +# file it already downloads, judges each quote against the current price and writes only what changed, and records the +# run (`PriceRefreshRun`). It makes no request outside that catalog source, never joins the prices database with the +# primary, and holds no network call or primary read inside a transaction on the prices database. +# +# Side effects: rows of `current_prices`, `price_changes`, `price_sources`, `price_series` and `price_refresh_runs`; +# the catalog source's download directory. It needs no `Current.account` (prices are global). Safe to run twice on one +# file: the second run is skipped as already applied, and a rerun of a failed run changes nothing it already wrote. +# +# @see file:prices.md +class Prices::Refresh + # Printings per write transaction, and printings (their current prices) per removal slice and transaction: a few + # thousand rows each, so the database is held briefly (spec 089 decision 7). + BATCH_SIZE = 1_000 + + # The largest share of the source's candidate current prices a run may remove in one day (spec 089 decision 4, ruled + # by the maintainer on 2026-11-13). A GUESS until #366's run records measure the real daily share of prices that + # disappear: a file that loses a tenth of the market in a day is far more likely truncated or damaged than a market + # event. A real drop above it is refused, every day, until this constant moves. + MAX_REMOVED_SHARE = Rational(1, 10) + + # The file can't be trusted to say prices are gone (decision 4, items 2 and 5); the run fails and nothing is removed. + class GuardError < StandardError; end + + # @param source_key [String] a key of `Prices.sources` + # @param job_id [String, nil] the Active Job id, so a re-run after a restart closes its own left-behind run + def initialize(source_key, job_id: nil) + @source_key = source_key + @job_id = job_id + @counts = Hash.new(0) + end + + # Runs the refresh. The source is looked up and synced before any run exists (a run's source row is required), so an + # unregistered key or a source that fails verification raises with no run recorded; every later failure ends the run + # `failed` with its error and re-raises it for the job's retry rules. + # + # @return [PriceRefreshRun] finished, skipped or (when another run works) skipped as already running + # @raise [ArgumentError, Prices::Sources::ConfigurationError] before a run exists + # @raise [GuardError, Prices::Sources::IntegrityError, Zlib::Error, Catalog::Sources::Error, StandardError] after the run is recorded failed + def call + @source_class = Prices.source_class(@source_key) + @source = PriceSource.sync!(@source_class) + @series = PriceSeries.ensure!(@source_class.series) + @run = PriceRefreshRun.start!(@source, job_id: @job_id) + return @run unless @run.running? + + perform + @run + rescue StandardError => error + @run.fail!(error, @counts) if @run&.running? + raise + end + + private + def collectible_type = @source_class.collectible_type + + def perform + return @run.skip!("the catalog has no printing of the type #{collectible_type} yet") unless Catalog::Entry.exists?(collectible_type:) + + catalog = Catalog.source_for(collectible_type) + @languages = catalog.languages + version = catalog.current_version(languages: @languages) + @run.record_file!(version:, languages: @languages.join(",")) + applied = PriceRefreshRun.applied(@source, version, @run.languages).first + return @run.skip!("#{version} already applied", run_on: applied.run_on) if applied + + catalog.progress = @run.method(:heartbeat) if catalog.respond_to?(:progress=) + path = catalog.download(version, dir: Rails.configuration.x.catalog_download_dir.join(collectible_type)) + reader = @source_class.new + @run_on = reader.reported_on(path) + @run.update!(run_on: @run_on) + latest = PriceRefreshRun.latest_run_on(@source) + return @run.skip!("a later file (#{latest}) was already read; this one is dated #{@run_on}") if latest && @run_on < latest + + apply(reader, path) + @run.finish!(@counts) + end + + # Reads the file once, flushing a batch of printings at a time (decision 6), then handles absences (decision 4). + def apply(reader, path) + @seen = Set.new # [entry id, series id] judged this run: a duplicate quote is ignored, an unseen candidate is absent + @unknown = Set.new + @held_entries = Set.new + @held_series = Set.new + @untrusted = false + @quotes_yielded = 0 + @pending = {} + @pending_malformed = [] + + reader.each_quote(path, languages: @languages) do |record| + case record + when Prices::Sources::Quote then queue_quote(record) + when Prices::Sources::Malformed then queue_malformed(record) + end + end + flush + remove_absent + end + + def queue_quote(quote) + @quotes_yielded += 1 + flush if !@pending.key?(quote.external_key) && pending_size >= BATCH_SIZE # earlier batches stand when the next raises + @series.fetch(quote.series) do + raise Prices::Sources::ConfigurationError, "#{@source_class} yielded a quote of #{quote.series.inspect}, which it did not declare" + end + # A repeat of a pending printing and series is ignored on arrival (the first is judged, AC-2.10): the batch counts + # printings, so a file repeating one key would otherwise grow its list without limit. + return if @pending[quote.external_key]&.any? { |queued| queued.series == quote.series } + + (@pending[quote.external_key] ||= []) << quote + end + + def queue_malformed(record) + @counts[:malformed] += 1 + Rails.logger.warn("prices.refresh skipped malformed #{record.external_key.inspect}: #{record.error}") + flush if pending_size >= BATCH_SIZE + @pending_malformed << record + end + + # Printings and unreadable records waiting for the next flush. Both count toward BATCH_SIZE: a file of mostly + # unreadable lines would otherwise pile up in memory and in one huge `IN` lookup of keys. + def pending_size = @pending.size + @pending_malformed.size + + # Judges and writes the pending printings. The primary is read here, outside any transaction; the prices database + # is read and written in one short transaction; the counts are tallied after it commits. + def flush + return if @pending.empty? && @pending_malformed.empty? + + keys = (@pending.keys + @pending_malformed.filter_map(&:external_key)).uniq + ids = Catalog::Entry.where(collectible_type:, external_key: keys).pluck(:external_key, :id).to_h + hold_malformed(ids) + judged = [] + @pending.each do |key, quotes| + id = ids[key] + next @unknown << key unless id + + quotes.each do |quote| + series_id = @series.fetch(quote.series).id + judged << [ id, series_id, quote.cents ] if @seen.add?([ id, series_id ]) + end + end + write_judged(judged) + @pending = {} + @pending_malformed = [] + @counts[:unknown_printings] = @unknown.size + @run.progress!(@counts) + end + + # What a Malformed holds back from removal (decision 4, items 3 and 4): no key, the whole run; a key and no series (or + # a series the source didn't declare), every series of the printing; a key and a series, that series. + def hold_malformed(ids) + @pending_malformed.each do |record| + entry_id = ids[record.external_key] + if record.external_key.nil? then @untrusted = true + elsif entry_id.nil? then next + elsif record.series.nil? || @series[record.series].nil? then @held_entries << entry_id + else @held_series << [ entry_id, @series[record.series].id ] + end + end + end + + def write_judged(judged) + return if judged.empty? + + tally = Hash.new(0) + PricesRecord.transaction do + existing = CurrentPrice.where(price_source_id: @source.id, catalog_entry_id: judged.map(&:first).uniq) + .pluck(:catalog_entry_id, :price_series_id, :cents).to_h { |entry_id, series_id, cents| [ [ entry_id, series_id ], cents ] } + currents = [] + changes = [] + judged.each do |entry_id, series_id, cents| + had = existing[[ entry_id, series_id ]] + next tally[:unchanged] += 1 if had == cents + + tally[had ? :repriced : :added] += 1 + keys = { catalog_entry_id: entry_id, price_source_id: @source.id, price_series_id: series_id } + currents << keys.merge(cents:, reported_on: @run_on, changed_on: @run_on) + changes << keys.merge(recorded_on: @run_on, cents:) + end + CurrentPrice.upsert_all(currents, unique_by: CurrentPrice.primary_key) if currents.any? + PriceChange.upsert_all(changes, unique_by: PriceChange.primary_key) if changes.any? + end + tally.each { |name, count| @counts[name] += count } + @counts[:quotes_seen] += judged.size + end + + # Absence is trusted only as far as the file is (spec 089 decision 4). The file has been read to the end here (a + # reader's error would have left `#apply` already). Two walks over the source's candidate current prices by printing + # id: the first counts the candidates and the absences, so the ceiling is judged on the whole; the second removes. + def remove_absent + raise GuardError, "the file yielded no valid quotes (#{@counts[:malformed]} malformed), so nothing was written or removed" if @quotes_yielded.zero? + + candidates = absent_clear = absent_held = 0 + each_candidate_slice do |rows| + rows.each do |entry_id, series_id| + candidates += 1 + next if @seen.include?([ entry_id, series_id ]) + + held?(entry_id, series_id) ? absent_held += 1 : absent_clear += 1 + end + end + @counts[:removals_withheld] = absent_held + return if absent_clear.zero? + + if absent_clear > candidates * MAX_REMOVED_SHARE + @counts[:removals_withheld] += absent_clear + raise GuardError, "#{absent_clear} of #{candidates} current prices are missing from the file, more than the #{(MAX_REMOVED_SHARE * 100).to_i} percent " \ + "a day may lose, so nothing was removed. A file that loses that much in a day is usually truncated or damaged; a real, " \ + "lasting drop is refused until Prices::Refresh::MAX_REMOVED_SHARE is raised." + end + + each_candidate_slice { |rows| remove(rows.reject { |entry_id, series_id| @seen.include?([ entry_id, series_id ]) || held?(entry_id, series_id) }) } + end + + def held?(entry_id, series_id) = @untrusted || @held_entries.include?(entry_id) || @held_series.include?([ entry_id, series_id ]) + + # Yields the source's candidate current prices as [entry id, series id], a slice of BATCH_SIZE printings at a time, + # by keyset on the printing id. A candidate's printing is a catalog entry of this type in a language of the run, + # retired or not; one the primary lacks, or in a language not read, isn't asked about and is left out. So is a price + # of a series the source doesn't declare today: the file can't speak for a series the source no longer reads, so its + # absence says nothing (spec 089 v0.5.0, AC-3.7). The primary is read outside any transaction, and each slice + # touches the run as progress. + def each_candidate_slice + last = 0 + declared = @series.values.map(&:id) + loop do + ids = CurrentPrice.where(price_source_id: @source.id).where("catalog_entry_id > ?", last).distinct.order(:catalog_entry_id).limit(BATCH_SIZE).pluck(:catalog_entry_id) + break if ids.empty? + + last = ids.last + languages = Catalog::Entry.where(collectible_type:, id: ids).pluck(:id, :language).to_h + rows = CurrentPrice.where(price_source_id: @source.id, catalog_entry_id: ids, price_series_id: declared).pluck(:catalog_entry_id, :price_series_id) + yield rows.select { |entry_id, _series_id| @languages.include?(languages[entry_id]) } + @run.progress!(@counts) + end + end + + # Deletes the current prices and records their blank history rows in ONE transaction: a price deleted without its row + # would no longer be a candidate, and the removal would be lost for good. `delete_all` is bulk maintenance of + # global rows by key; no callbacks or validations apply. + def remove(rows) + return if rows.empty? + + PricesRecord.transaction do + rows.group_by(&:first).each do |entry_id, pairs| + CurrentPrice.where(price_source_id: @source.id, catalog_entry_id: entry_id, price_series_id: pairs.map(&:last)).delete_all + end + PriceChange.upsert_all(rows.map { |entry_id, series_id| { catalog_entry_id: entry_id, price_source_id: @source.id, price_series_id: series_id, recorded_on: @run_on, cents: nil } }, + unique_by: PriceChange.primary_key) + end + @counts[:removed] += rows.size + end +end diff --git a/config/recurring.yml b/config/recurring.yml index 7eb801bc..a0b4e90d 100644 --- a/config/recurring.yml +++ b/config/recurring.yml @@ -28,6 +28,11 @@ production: args: [ "mtg", "scheduled" ] queue: sync schedule: every monday at 3:15am + refresh_prices: + class: Prices::RefreshJob + args: [ "scryfall" ] + queue: sync + schedule: at 11:30am every day sweep_import_uploads: class: ImportUpload::SweepJob queue: default diff --git a/db/prices_migrate/20261113190001_prepare_price_refresh_runs.rb b/db/prices_migrate/20261113190001_prepare_price_refresh_runs.rb new file mode 100644 index 00000000..53174e77 --- /dev/null +++ b/db/prices_migrate/20261113190001_prepare_price_refresh_runs.rb @@ -0,0 +1,25 @@ +# What the daily refresh needs of price_refresh_runs (spec 089 decision 2; supersedes the run_on column of spec 081). +# +# run_on becomes nullable: a run opens before its file's date is known, and some end before it is (already running, an +# empty catalog, a transient error). job_id is for the stall rule; languages and the counts say what the run did. +# +# price_refresh_runs is a rowid table, so Rails' table rebuild for change_column_null is safe here, unlike for +# current_prices and price_changes (see the first prices migration's warning). The columns are added first and the +# rebuild comes last, so the stored table SQL is the canonical one. `down` gives a nil run_on the date of started_at, +# so it never loses a row. +class PreparePriceRefreshRuns < ActiveRecord::Migration[8.1] + COUNTS = %i[quotes_seen unchanged changed added removed removals_withheld unknown_printings malformed].freeze + + def up + add_column :price_refresh_runs, :job_id, :string + add_column :price_refresh_runs, :languages, :string + COUNTS.each { |name| add_column :price_refresh_runs, name, :integer, null: false, default: 0 } + change_column_null :price_refresh_runs, :run_on, true + end + + def down + execute "UPDATE price_refresh_runs SET run_on = date(started_at) WHERE run_on IS NULL" + change_column_null :price_refresh_runs, :run_on, false + remove_columns :price_refresh_runs, *COUNTS, :languages, :job_id + end +end diff --git a/db/prices_migrate/20261113190002_rename_price_refresh_runs_changed.rb b/db/prices_migrate/20261113190002_rename_price_refresh_runs_changed.rb new file mode 100644 index 00000000..764946f8 --- /dev/null +++ b/db/prices_migrate/20261113190002_rename_price_refresh_runs_changed.rb @@ -0,0 +1,8 @@ +# Active Record reserves the attribute name `changed` (ActiveRecord::DangerousAttributeError), so the count of repriced +# quotes that 20261113190001 added under that name is `repriced` (spec 089 decision 2, v0.4.3). A rowid table, so the +# table rebuild of rename_column is safe; the migration before it is unreleased and stays as built. +class RenamePriceRefreshRunsChanged < ActiveRecord::Migration[8.1] + def change + rename_column :price_refresh_runs, :changed, :repriced + end +end diff --git a/db/prices_structure.sql b/db/prices_structure.sql index bad1a93f..8d5f3328 100644 --- a/db/prices_structure.sql +++ b/db/prices_structure.sql @@ -18,11 +18,13 @@ FOREIGN KEY ("price_source_id") FOREIGN KEY ("price_series_id") REFERENCES "price_series" ("id") ON DELETE RESTRICT, CONSTRAINT price_changes_cents_non_negative CHECK (cents IS NULL OR cents >= 0)) WITHOUT ROWID /*application='Collector'*/; -CREATE TABLE "price_refresh_runs" ("id" integer PRIMARY KEY AUTOINCREMENT NOT NULL, "price_source_id" integer NOT NULL, "run_on" date NOT NULL, "source_version" varchar, "status" varchar NOT NULL, "error" text, "started_at" datetime(6) NOT NULL, "finished_at" datetime(6), "created_at" datetime(6) NOT NULL, "updated_at" datetime(6) NOT NULL, CONSTRAINT "fk_rails_c95c78b8b6" +CREATE TABLE "price_refresh_runs" ("id" integer PRIMARY KEY AUTOINCREMENT NOT NULL, "price_source_id" integer NOT NULL, "run_on" date, "source_version" varchar, "status" varchar NOT NULL, "error" text, "started_at" datetime(6) NOT NULL, "finished_at" datetime(6), "created_at" datetime(6) NOT NULL, "updated_at" datetime(6) NOT NULL, "job_id" varchar, "languages" varchar, "quotes_seen" integer DEFAULT 0 NOT NULL, "unchanged" integer DEFAULT 0 NOT NULL, "repriced" integer DEFAULT 0 NOT NULL, "added" integer DEFAULT 0 NOT NULL, "removed" integer DEFAULT 0 NOT NULL, "removals_withheld" integer DEFAULT 0 NOT NULL, "unknown_printings" integer DEFAULT 0 NOT NULL, "malformed" integer DEFAULT 0 NOT NULL, CONSTRAINT "fk_rails_c95c78b8b6" FOREIGN KEY ("price_source_id") REFERENCES "price_sources" ("id") ON DELETE RESTRICT, CONSTRAINT price_refresh_runs_status_known CHECK (status IN ('running', 'finished', 'skipped', 'failed'))); CREATE INDEX "index_price_refresh_runs_on_price_source_id_and_run_on" ON "price_refresh_runs" ("price_source_id", "run_on") /*application='Collector'*/; INSERT INTO "schema_migrations" (version) VALUES +('20261113190002'), +('20261113190001'), ('20261113100001'); diff --git a/docs/architecture/catalog-operations.md b/docs/architecture/catalog-operations.md index 6cdb2a21..f9488098 100644 --- a/docs/architecture/catalog-operations.md +++ b/docs/architecture/catalog-operations.md @@ -18,6 +18,7 @@ How the catalog changes, and how admins start, watch and recover that work: the - What the artwork table holds, the fingerprint, which image an artwork uses, the index file's format, serving it and matching in the browser are **[scanner](scanner.md)** (spec 011 apart from FR-2's build run; ADRs 0006, 0007, 0012). - The admin flag and the `AdminOnly` guard are **[tenancy and accounts](tenancy-and-accounts.md)** (spec 004). - Running Solid Queue inside Puma (`SOLID_QUEUE_IN_PUMA`) and the volume that holds `storage/catalog/` are in CLAUDE.md and **release and deploy**. +- The daily price refresh reads the file this refresh's source downloads, through the same `#languages`, `#current_version` and `#download`, and is not a `Catalog::Operation`: its job, run record, guards and rules are in **[prices](prices.md)** (spec 089 decision 1). What the sharing means for the download is under "What the source does for a refresh" below. ## Glossary @@ -123,7 +124,8 @@ A start request while a build is in flight queues nothing and answers "An art in ### What the source does for a refresh (`MTG::Scryfall::Source`) - The source version is the name of the current bulk file in Scryfall's bulk-data index: `default_cards` when only English is configured, `all_cards` otherwise (spec 002 FR-3). -- The download goes to `storage/catalog//` through a partial file, renamed into place only when its size is the one Scryfall publishes (otherwise the partial is deleted and the run fails with `Catalog::Sources::IntegrityError`); a file of that size already there is used again, and only the `KEEP_DOWNLOADS` newest are kept (spec 002 FR-3, Error Scenarios). It reports bytes received while downloading and compressed bytes read while the file is read, through `#progress=` (spec 015 FR-2). +- The download goes to `storage/catalog//` through a partial file of its own (`..part`, a name per download, so two downloads of one version never share one: the price refresh is a second caller), renamed into place only when its size is the one Scryfall publishes (otherwise the partial is deleted and the run fails with `Catalog::Sources::IntegrityError`; a download that fails for any other reason deletes its own partial file too); a file of that size already there is used again, and only the `KEEP_DOWNLOADS` newest are kept; `prune` also deletes a `*.part` file whose mtime is older than `STALE_PARTIAL_AFTER`, so a killed download's orphan is cleared, and tolerates a file that vanishes under it (a concurrent download). The contract's comment says `#download` must tolerate concurrent calls for one version (spec 002 FR-3, Error Scenarios; spec 089 decision 1, which supersedes spec 002 FR-3's wording "a partial file"). It reports bytes received while downloading and compressed bytes read while the file is read, through `#progress=` (spec 015 FR-2). +- **One file for two refreshes.** An English-only instance's weekly refresh finds the day's file already on disk when the price refresh downloaded it first (the source reuses a file of the published size); an instance with languages beyond English reads `all_cards` (408 MB) and downloads it daily whichever job runs first, a cost the price refresh adds. The catalog refresh knows nothing of prices and records no analytics (design decision 10; spec 089 decision 1). - The file is read record by record; printings in languages not configured, and digital-only ones, are left out before anything is written (spec 002 FR-3). - Languages come from `COLLECTOR_MTG_LANGUAGES` (comma-separated codes); English is always included and is the default; a code Scryfall doesn't use fails the run before downloading, naming the code (spec 002 FR-6, Error Scenarios). - Every request sends the app's `User-Agent` and an `Accept` header (`application/json`, or `image/jpeg` for an artwork image), waits `MIN_INTERVAL` after the previous one, and has explicit open and read timeouts; a refresh requests no single cards (spec 002 FR-3; spec 011 AC-3.4). @@ -175,6 +177,7 @@ Filter chips Failed (the default), Running, Queued and Scheduled, each with its | `MTG::Scryfall::Client::RATE_LIMITED_FOR` | 30 (seconds) | `app/models/mtg/scryfall/client.rb` | How long Scryfall says a 429 limits access for; the client never asks again inside it on its own guess | #81 | | `MTG::Scryfall::Client::OPEN_TIMEOUT`, `READ_TIMEOUT` | 10, 60 (seconds) | `app/models/mtg/scryfall/client.rb` | Explicit timeouts on every request | spec 002 FR-3; spec 011 AC-3.4 | | `MTG::Scryfall::Source::KEEP_DOWNLOADS` | 2 | `app/models/mtg/scryfall/source.rb` | Downloaded bulk files kept, newest first | spec 002 FR-3 | +| `MTG::Scryfall::Source::STALE_PARTIAL_AFTER` | 1 day | `app/models/mtg/scryfall/source.rb` | A partial download file this old belongs to a killed download: a live one is far younger, since a job that never reports ends at its 6-hour lock. Each download writes its own partial file, so a killed one leaves an orphan for `prune` to remove | spec 089 decision 1 | | `BackgroundJobs::List::PER_PAGE` | 25 | `app/models/background_jobs/list.rb` | Jobs per page | spec 015 FR-5, AC-6.2 | | `poll` controller `interval` | 2,000 ms | `app/javascript/controllers/poll_controller.js` | About 2 seconds, so a change shows within 5 | spec 015 FR-4, AC-2.10; ADR 0015 | @@ -190,6 +193,7 @@ Filter chips Failed (the default), Running, Queued and Scheduled, each with its - **Times** are absolute UTC in the form `9 Oct 2026 03:15 UTC`, with a relative form ("2 minutes ago") for last progress or heartbeat and for list times (spec 015 FR-8; `AdminHelper#utc_time`). - **A graceful restart re-runs, a crash fails.** A graceful restart puts a running job back in the queue at once, and the job closes its own run and carries on; after a crash the queue fails the job once it notices the worker is gone (about 5 minutes), and the run reads interrupted by the job rule (spec 015 Error Scenarios; spec 011 Error Scenarios). - **Progress is judged on the attempt the rate limit ended, alone.** In the order rate limit, a busy-database retry that did work, rate limit with no new work, the second rate limit counts 2, not 1: the job fails sooner than a perfect rule would, never later than before; the count stays in one place, the job's retry record (spec 043 decision 2). +- **A partial download file is per download (supersedes spec 002 FR-3's "a partial file").** A fixed `.part` is shared by two downloads of one version, which the price refresh makes possible; a unique name makes a second download wasteful but harmless, and costs an orphan when a download is killed, which `prune` clears after `STALE_PARTIAL_AFTER` by mtime (a fixed-name partial file from an earlier build is caught by the same glob). The contract's comment carries the concurrent-call line (spec 089 decision 1; the implementation review found the vanishing-file race in `prune`). **Accepted limits and open issues:** @@ -215,6 +219,7 @@ Filter chips Failed (the default), Running, Queued and Scheduled, each with its - [Spec 043](../specs/043-rate-limit-retry-progress/spec.md): Problem Statement (the image origins' rate limits), decisions 1 to 4, FR-1 to FR-8, Stories 1 to 4. - [Spec 055](../specs/055-rate-limit-retry-now/spec.md): all. - [Spec 058](../specs/058-mtg-mana-value/spec.md) decision 3 (`#reapply?`'s second reason). +- [Spec 089](../specs/089-daily-price-refresh/spec.md) decision 1 (the shared download: the unique partial file, the stale partial pruning, the concurrent-call line of the contract; the rest of the spec is in [prices](prices.md)). - ADRs [0014](../adr/0014-hand-built-admin-jobs-console.md), [0015](../adr/0015-admin-progress-by-polling.md), [0018](../adr/0018-collectible-agnostic-catalog-core.md). - Issues [#60](https://github.com/plainprogrammer/Collector/issues/60), [#61](https://github.com/plainprogrammer/Collector/issues/61), [#73](https://github.com/plainprogrammer/Collector/issues/73), [#74](https://github.com/plainprogrammer/Collector/issues/74), [#78](https://github.com/plainprogrammer/Collector/issues/78), [#79](https://github.com/plainprogrammer/Collector/issues/79), [#80](https://github.com/plainprogrammer/Collector/issues/80), [#81](https://github.com/plainprogrammer/Collector/issues/81), [#84](https://github.com/plainprogrammer/Collector/issues/84), [#86](https://github.com/plainprogrammer/Collector/issues/86), [#111](https://github.com/plainprogrammer/Collector/issues/111), [#122](https://github.com/plainprogrammer/Collector/issues/122), [#176](https://github.com/plainprogrammer/Collector/issues/176), [#77](https://github.com/plainprogrammer/Collector/issues/77) (discarding a waiting retry stays out of scope), [#411](https://github.com/plainprogrammer/Collector/issues/411); open: [#41](https://github.com/plainprogrammer/Collector/issues/41), [#283](https://github.com/plainprogrammer/Collector/issues/283). @@ -229,4 +234,5 @@ Filter chips Failed (the default), Running, Queued and Scheduled, each with its | 2026-11-09 | Spec 053 (the art index's "Out of date" state for a build made with other settings and its status line, the "Warning" fact and line suffix for a build made for an older catalog, what the scanner does with each kind of drift, the accepted limits) and issue #176 | [#241](https://github.com/plainprogrammer/Collector/pull/241) | | 2026-11-09 | Spec 055 (Retry now for a failed run's own job waiting to retry: what it moves and how, one winner, the offer in one place, the refresh's "Queued to retry" tied to its own job, wording, admin only) and issue #77 | [#244](https://github.com/plainprogrammer/Collector/pull/244) | | 2026-11-10 | Spec 058 (decision 3: `#reapply?`'s second reason, an upgraded instance filling the new column on its next refresh; the partial-failure and no-`cmc` limits) and issue #250 | [#279](https://github.com/plainprogrammer/Collector/pull/279) | +| 2026-11-13 | Spec 089 decision 1 (the download shared with the daily price refresh: a partial file per download, stale partial files pruned, `#download` tolerating concurrent calls; supersedes spec 002 FR-3's "a partial file"; the price refresh's own rules are in `prices.md`) | spec 089's pull request | | 2026-11-13 | Issue #411 (a bulk-file line that isn't valid UTF-8 counts as one malformed line and the refresh goes on); no spec | #411's pull request | diff --git a/docs/architecture/prices.md b/docs/architecture/prices.md index b7bbe592..03cb5ce1 100644 --- a/docs/architecture/prices.md +++ b/docs/architecture/prices.md @@ -4,7 +4,7 @@ Where market prices live, how their database is built and what may never cross f ## Scope -The prices database: its configuration, its tables, the connection class later models inherit, and the rules that keep prices global and apart from collectors' data (spec 081). The price source contract, with Scryfall's source and the rows of sources and series that a source's declaration writes (spec 085, #364). The daily refresh (#365), the run record (#366), the admin log (#367), the history levels (#368), the display (#369) and the operator's cost documentation (#370) will extend this doc. +The prices database: its configuration, its tables, the connection class later models inherit, and the rules that keep prices global and apart from collectors' data (spec 081). The price source contract, with Scryfall's source and the rows of sources and series that a source's declaration writes (spec 085, #364). The daily refresh: its job, run record, guards and the three models it writes (spec 089, #365). The run's measurements (#366), the admin log (#367), the history levels (#368), the display (#369) and the operator's cost documentation (#370) will extend this doc. Neighbouring areas: the backup that holds the prices database, its format and its restore are in [release and deploy](release-and-deploy.md). The printings that prices name are the catalog's ([catalog core](catalog-core.md)). The export format holds no price ([data portability](data-portability.md)). @@ -16,7 +16,10 @@ Neighbouring areas: the backup that holds the prices database, its format and it - **Price series:** what a price means: a finish, an ISO 4217 currency and a kind such as `market`. Series are global, not per source (`price_series`; spec 081 decision 2). - **Current price:** the latest known price of one printing in one series from one source (`current_prices`). - **Price change:** one row of history, recorded on the day a price differs from the last one recorded; a row with no cents records that the price went away that day (`price_changes`; design decision 7). -- **Price refresh run:** one run of the daily refresh (`price_refresh_runs`). Its identity and times only, so far; #366 adds the measurements (spec 081 decision 2). +- **Price refresh run:** one run of the daily refresh (`price_refresh_runs`, `PriceRefreshRun`): its identity, times, status and the counts of what it did to rows; #366 adds the measurements (spec 081 decision 2; spec 089 decision 2). +- **Applied:** a run that finished. A file is applied once: a finished run of the same source has the same `source_version` and `languages` (spec 089 decision 5). +- **Candidate:** a current price the file may speak for: its printing is a catalog entry of the type in a language the run read (retired or not) and its series is one the source declares today. An **absent** candidate has no quote in the file; a **withheld** removal is an absent candidate the run did not remove because the file could not be trusted (spec 089 decision 4). +- **Repriced:** a quote whose price differs from the current price. The run's count is `repriced`, not `changed`: Active Record reserves that attribute name (spec 089 v0.4.3). - **Quote:** one price of one printing in one series from one source on one reported day, as a source yields it (`Prices::Sources::Quote`: the source's id for the printing, finish, currency, kind, whole cents and the day). Not a row: nothing stores a quote until #365 (spec 085 decision 2). - **Reported on:** the day a source's data is from, not the day it was read. For Scryfall it is the UTC date in the bulk file's name (spec 085 decisions 2 and 4). - **Three things named alike:** `PriceSource` is the model, a row of `price_sources`; `Prices::Sources` is the contract a source class implements; `Prices.sources` is the registry of source classes by key (spec 085 decisions 1, 7 and 11). @@ -33,19 +36,36 @@ Neighbouring areas: the backup that holds the prices database, its format and it - **The prices schema names no collectible**: its migrations, its structure file and `PricesRecord` never name a game or a source (spec 081 decision 9, AC-3.5; `spec/prices_core_spec.rb`). - **The backup holds the prices database** beside the primary; how is in [release and deploy](release-and-deploy.md) (spec 081 decision 6). - **The contract's comment is the contract.** `Prices::Sources` (`app/models/prices/sources.rb`) is the one place what a source provides and what each part returns or raises is written; this doc points to it and does not restate it. There is no ADR for it (spec 085 decisions 1 and 12; Gate A, 2026-11-13). Core files (`app/models/prices.rb`, `app/models/prices/**`, `PriceSource`, `PriceSeries`) name no collectible, source or marketplace, comments included, and `spec/price_source_core_spec.rb` fails on one (spec 085 decisions 1 and 12, AC-4.1). -- **A source yields only the prices its file carries.** A price the file leaves empty has no quote. A `Malformed` that names a series is not an absence (the price exists and was not understood); one with a key and no series means no series of that printing may be read as absent; one with no key makes the run's absences untrustworthy. The reason: a refresh that deletes a price when its quote is absent must never delete one the source failed to parse, because the deletion is recorded in history that can't be fetched again. What #365 does with each is its rule (spec 085 decision 2). +- **A source yields only the prices its file carries.** A price the file leaves empty has no quote. A `Malformed` that names a series is not an absence (the price exists and was not understood); one with a key and no series means no series of that printing may be read as absent; one with no key makes the run's absences untrustworthy. The reason: a refresh that deletes a price when its quote is absent must never delete one the source failed to parse, because the deletion is recorded in history that can't be fetched again. What the refresh does with each is the "Gone is not bad" invariant below (spec 085 decision 2; spec 089 decision 4). - **`each_quote` makes no network call, writes nothing and reads no database.** It reads the local file its caller gives and streams it one line at a time. The contract has no members that fetch the file: acquiring it is #365's (spec 085 decision 3). - **The date reported is the UTC date in the file's name.** A name with no date, or digits that are not a date, is an `IntegrityError` from `#reported_on` and from `#each_quote` before any quote is yielded, because a guessed date would put a wrong day into history that can't be fetched again (spec 085 decision 4, AC-2.4). - **A price becomes cents from its digits, never through a float.** Anything that isn't one to nine digits with at most two decimals is a `Malformed` that names its series; `"0.00"` is a quote of 0 cents, and whether to show it is the display's (#369) (spec 085 decision 6, AC-2.6). - **Quotes store what the file carries, twins included.** The EUR a non-English printing copies from its English twin is yielded as a quote of that printing; choosing the twin's price is a display rule that #369 owns (ruling 41, [research 083](../specs/083-non-english-printing-prices/research.md); spec 085 decision 5, AC-2.8). - **Source and series rows are written by `PriceSource.sync!` and `PriceSeries.ensure!`** from a source's declaration, never seeded by a migration and never deleted. A corrected attribution takes effect at the next sync; a series a source stops declaring keeps its row; series are global, so sources that quote the same finish, currency and kind share one row. Both calls are idempotent and create through the unique indexes, so the caller that loses a race reads the winner's row (spec 085 decision 7, AC-3.1 to AC-3.6). -- **Only `PriceSource` and `PriceSeries` are models.** `CurrentPrice`, `PriceChange` and `PriceRefreshRun` are #365's and #366's (spec 085 decision 8). +- **Every price table has a model**, all inheriting `PricesRecord`: `PriceSource`, `PriceSeries`, and, with the daily refresh, `CurrentPrice` and `PriceChange` (composite primary keys, bulk writes by key: `upsert_all(rows, unique_by: Model.primary_key)` and `delete_all`, which skip validations on purpose because each row comes from a validated `Quote`) and `PriceRefreshRun` (spec 085 decision 8; spec 089 decisions 2, 6 and 8). +- **The daily refresh reads the catalog source's file and nothing else.** `Prices::Refresh` asks `Catalog.source_for(type)` for `#languages`, `#current_version` and `#download`, so the catalog refresh and the price refresh read one file in `storage/catalog//`; a file one has downloaded the other reuses when both see the same version. `Prices::Sources` has no member for fetching. The run's `source_version` is the string the catalog keeps as its version. It is not a `Catalog::Operation`: it has its own job (`Prices::RefreshJob`, queue `sync`), its own run record and its own guard, because the catalog refresh must not learn of prices and the page for prices is #367's (spec 089 decision 1; roadmap ruling of 2026-11-13). +- **The source is looked up and synced before a run exists.** `Prices.source_class`, then `PriceSource.sync!`, then `PriceRefreshRun.start!`, because a run's source row is required (`price_source_id` is `NOT NULL`). An unregistered key or a source that fails verification fails the job with no run row: a defect of the build, not of a day (spec 089 decision 5, step 0). +- **One run at a time; a run left behind is closed.** `Prices::RefreshJob` holds a queue lock per source key (`LOCK_FOR`). `PriceRefreshRun.start!` closes each running run with no progress for `STALL_AFTER`, or carrying the starting job's own id, as `failed` with `interrupted`, then records the new run `skipped` (`already running`) if another still runs. Progress is a touch of the run row at every batch read (even one that writes nothing), every removal slice, and, while downloading, at most every `HEARTBEAT_EVERY` through the catalog source's optional `#progress=` hook (spec 089 decision 7). +- **A file is applied once, and the date is the file's.** The order: the source synced; the run opened; the catalog has no printing of the type (skipped, no date); the catalog source's current version and languages, and a finished run of the same source, version and languages (skipped ` already applied`, with that run's `run_on`); the download; the file's UTC date (`#reported_on`), written to the run; a file older than the latest `run_on` of the source's finished or failed runs (a failed run may have written rows) is skipped after the download, and a file of the same day with another version or language set is applied. `run_on` is nil until the date is known (spec 089 decisions 3 and 5; supersedes spec 081's `NOT NULL`). +- **Unchanged writes nothing; freshness is the latest finished run.** A quote equal to the current price writes no row, so `current_prices.reported_on` and `changed_on` are both the run date of the last write. A price is current as of the source's latest finished run's `run_on`, which a page labels "as of" (#369). Prices the run leaves alone (a language not read, a printing the primary lacks, a series no longer declared) go stale with no marker on their row. While the ceiling below fails every run, no run finishes and the label lags (spec 089 decision 3). +- **The four cases of a quote.** Per printing and series, in one short transaction per batch of `BATCH_SIZE` printings: a printing the catalog lacks is skipped and counted once (`unknown_printings`; its quotes are in no other count); a price equal to the current one writes nothing (`unchanged`); a different one updates the current price and writes a history row for the run's date (`repriced`); one with no current price inserts both (`added`); the first quote of a printing and series in a file is judged and a repeat is ignored. History rows are upserted on their key, so a second file of one day replaces that day's row and a rerun can't duplicate. `quotes_seen = unchanged + repriced + added`. A quote of a series the source didn't declare raises `Prices::Sources::ConfigurationError` before its batch is written; earlier batches stand (spec 089 decision 6). +- **Gone is not bad.** A price is removed (the current price deleted, a history row with no cents for the run's date) only if its printing is a catalog entry in a language the run read, its series is declared, the file has no quote for it, and the file can be trusted: it was read to the end (a `Zlib::Error` fails the run and removes nothing; earlier batches stand), the source yielded at least one `Quote` (else `no valid quotes`, nothing written), no `Malformed` names no printing (a keyless one withholds every removal), none names the printing (withholds all its series) or its series (withholds that one), and the absent prices are at most `MAX_REMOVED_SHARE` of all candidate rows (compared exactly, after a first walk that counts them all; a breach withholds every removal and fails the run, saying how many of how many). An unreadable price is not an absent one. A removal batch deletes the current prices and upserts their blank history rows in one transaction, so a failure leaves both or neither. `removals_withheld` counts every absent candidate not removed. Failing items 1, 2 and the ceiling fail the run; the others finish it, because the next day's file removes the price if it is really gone (spec 089 decision 4; spec 085 decision 2). +- **No transaction holds the network or the primary.** The download, the file read and the lookup of entry ids on the primary (by `external_key`, per batch; no join across the files) happen outside any transaction on the prices database; the run row's progress write is outside the batch transaction; counts are tallied after a batch commits and persisted with each progress touch, so an `interrupted` run keeps its last counts. Pending printings and unreadable records both count toward `BATCH_SIZE` (spec 089 decisions 2 and 7; an automated security review). +- **Errors and their fates, with no discard.** Transient errors (`Catalog::Sources::TransientError`, `ActiveRecord::StatementTimeout`) and the download's size error (`Catalog::Sources::IntegrityError`) are retried (`polynomially_longer`, 3 attempts); a rate limit is rescheduled by `Catalog::RateLimitRetry`, included after `retry_on`; everything else (`Prices::Sources::IntegrityError`, a bad language setting, `Zlib::Error`, `Prices::Refresh::GuardError`, any other error) fails the job, with the run `failed` and its error. Nothing is `discard_on`: a failed job shows under Failed on `/admin/jobs`, a deliberate departure from `background-jobs.md`'s advice, because #367 doesn't exist yet and a discarded job would hide a guard that fired (spec 089 decision 7). - **`hosted_paid_allowed?` is stated and stored, and nothing refuses a source by it yet.** No setting for the kind of instance exists; hosted mode (#392) adds one. Scryfall's is `true`, ruled by the maintainer on 2026-11-13 (roadmap rulings 27, 37 and 38). Issue #362 is open and its answer may change the value; the value is written only in `MTG::Scryfall::PriceQuotes.hosted_paid_allowed?` and in its spec (spec 085 decision 9). ## States and transitions - **The prices database comes into being with the instance's start.** `bin/rails db:prepare` creates it on a new volume, loading `db/prices_structure.sql`, and on an upgraded one that lacks it. A missing `sqlite3` command at that point leaves the load undone and two `command not found` lines in the log, and `db:prepare` then runs every prices migration from scratch, which builds the same tables (spec 081 decision 3). -- **A run's status** is one of `running`, `finished`, `skipped` and `failed`, fixed by a check so that #365 needs no rebuild of a released table to use them; what moves a run between them is #365's (spec 081 decision 2). +- **A run's status** is one of `running`, `finished`, `skipped` and `failed`, fixed by a check. A run opens `running` (`PriceRefreshRun.start!`) or, when another works, `skipped` with `already running`; it ends `finished` with its counts, `skipped` with its reason in `error` (no printing yet, already applied, an older file), or `failed` with `": "` and the counts so far, and the error re-raised for the job's rules. A skipped run is never "applied" (spec 089 decisions 2, 5 and 7). + +### What an operator sees when the job fails every day + +There is no prices page until #367, so a daily failure shows in three places and has one standing cause worth knowing (spec 089 decision 7, Gate A, 2026-11-13): + +- **One Failed job a day** under `/admin/jobs` (`Prices::RefreshJob`, queue `sync`), whose exception says what happened: `Prices::Refresh::GuardError` with how many of how many prices the file lacked, or `no valid quotes`; `Zlib::Error` for a cut-short or corrupt file; `Catalog::Sources::ConfigurationError` naming an unsupported `COLLECTOR_MTG_LANGUAGES` code. Solid Queue's hourly clearing removes only finished jobs, so an admin discards a failed job by hand. +- **A failed run row** in `price_refresh_runs` (read with `bin/rails runner`) with the same text in `error` and the counts so far; changes written before the failure stand, and nothing was removed. +- **A standing ceiling breach.** A real, lasting drop above `MAX_REMOVED_SHARE` of the market's prices is refused every day until that constant moves; it is a first guess until #366's run records measure the daily share of prices that disappear. A truncated or damaged file fixes itself with the next day's file. The "as of" label lags while no run finishes. ## Named constants @@ -59,6 +79,12 @@ Neighbouring areas: the backup that holds the prices database, its format and it | `MTG::Scryfall::PriceQuotes::FILE_STAMP` | `-(\d{4})(\d{2})(\d{2})\d{6}\.jsonl(?:\.gz)?\z` | `app/models/mtg/scryfall/price_quotes.rb` | The UTC timestamp at the end of a bulk file's name, the string the catalog keeps as the version; its date is the quotes' `reported_on` | spec 085 decision 4 | | `Prices::Sources::CURRENCY` | `\A[A-Z]{3}\z` | `app/models/prices/sources.rb` | An ISO 4217 code is three upper-case letters; the same rule as the `price_series` table's check, so a source can't build what the database would refuse | spec 085 decision 2; spec 081 | | `Prices::Sources::KEY` | `\A[a-z0-9_]+\z` | `app/models/prices/sources.rb` | A source's key is stored in `price_sources.key` and compared, so it is lower case letters, digits and underscores only; `verify!` refuses another | spec 085 decision 1 | +| `Prices::Refresh::BATCH_SIZE` | 1,000 | `app/models/prices/refresh.rb` | Printings per write transaction and per removal slice: a few thousand rows, so the database is held briefly | spec 089 decision 7 | +| `Prices::Refresh::MAX_REMOVED_SHARE` | `Rational(1, 10)` | `app/models/prices/refresh.rb` | The largest share of candidate current prices a run may remove in a day. **A guess until #366's run records measure the real daily share of prices that disappear**: a file that loses a tenth of the market in a day is far more likely truncated or damaged than a market event. A real drop above it is refused, every day, until the constant moves | spec 089 decision 4 (maintainer, 2026-11-13) | +| `PriceRefreshRun::STALL_AFTER` | 15 minutes | `app/models/price_refresh_run.rb` | A running run with no progress for this long is left behind; the catalog refresh's own threshold | spec 089 decision 7 | +| `PriceRefreshRun::HEARTBEAT_EVERY` | 30 seconds | `app/models/price_refresh_run.rb` | The most often a download's progress touches the run row, so a long download never reads as stalled | spec 089 decision 7 | +| `Prices::RefreshJob::LOCK_FOR` | 6 hours | `app/jobs/prices/refresh_job.rb` | The queue's concurrency lock per source key, as the catalog refresh's: a waiting refresh must never start while another's job may still be working | spec 089 decision 7 | +| Daily schedule | `at 11:30am every day`, production only | `config/recurring.yml` (`refresh_prices`) | After the 10:06 UTC publication of the file seen on 2026-11-13 (research 083); the run's date comes from the file whenever it runs; a one-line change if the pattern differs | spec 089 decision 8 | ## Decisions not in an ADR @@ -66,7 +92,10 @@ Neighbouring areas: the backup that holds the prices database, its format and it - **`PricesRecord` has the writing role only**: nothing reads a replica, and a reading pool that the suite's restored-instance helper doesn't re-point would be a trap (spec 081 decision 5; plan review). - **The structure file carries query-log comments.** Statements run in development end with `/*application='Collector'*/` (`config.active_record.query_log_tags_enabled`), SQLite stores them with the definition, and the dump keeps them. Production stores none. `spec/prices_schema_spec.rb` strips comments before it compares definitions (spec 081 execution, batch A). - **The structure file needs the `sqlite3` command** where it is dumped (development, after a migration) and loaded (the test database, a new instance). It is in the image's runtime stage and on CI's runners; CI installs none itself (spec 081 decision 3). -- **Two models exist, three tables have none yet.** `PriceSource` and `PriceSeries` were built with the contract because it needs them; `current_prices`, `price_changes` and `price_refresh_runs` have no model until #365 and #366, each of which writes the models, readers and writers it owns (spec 085 decision 8; spec 081 decision 5, Non-Goals). +- **The count is `repriced`, in a second migration.** Spec 089's first migration (`20261113190001`) added the count as `changed`; Active Record forbids that attribute name (`DangerousAttributeError`), found when the model was built. The unreleased migration stayed as built and `20261113190002` renames the column, so a database that applied either is right. The same two migrations make `run_on` nullable (a rebuild of the rowid table `price_refresh_runs`, which keeps its status check, foreign key and index) and add `job_id`, `languages` and the counts (spec 089 decision 2, v0.4.3). +- **Accepted limits of the refresh.** The `@seen` set of judged `[printing, series]` pairs (about 30 MB for 300,000 quotes), the unknown printings' keys and the held printings are held in memory for the run, bounded by the file; a removal slice sends one `DELETE` per printing; `quote.series` builds a validated `Series` per quote; `progress!` and `finish!` don't check the run is still `running` (a process resuming after the 6-hour lock lapsed could overwrite an `interrupted` closing). The measured first run is 18.4 s and a typical day 7.4 s for 110,000 cards and 275,000 quotes (spec 089 AC-5.6, the development machine, no threshold); #366's run records can reopen any of these (spec 089 implementation review). +- **A series the source no longer declares is left alone**, as a language not read is: a later build that drops one would otherwise fail every run on the ceiling or blank prices the file was never asked about (spec 089 v0.5.0; implementation review). +- **No ADR for the refresh.** The decisions are spec 089's, and this doc carries the rules (spec 089; Gate A, 2026-11-13). - **The attribution wording and the hosted flag are the source's declaration**, code that ships with the build and takes effect at the next sync, not a migration's seed: a migration is never edited once released, and the text is a statement the project may need to correct (spec 085 decisions 7, 9 and 10). - **No ADR for the contract.** The decisions are spec 085's, and the contract's comment and this doc carry the rules (spec 085 decision 12; Gate A, 2026-11-13). - **Reading cost.** `each_quote` read a generated file of 50,000 cards (300,000 quotes) in 1.2 s on the development machine; the figure is a measurement with no threshold, for #365 and #366 to plan from (spec 085 execution, T2.3). @@ -75,10 +104,11 @@ Neighbouring areas: the backup that holds the prices database, its format and it - [Spec 081](../specs/081-prices-database-in-backup/spec.md): decisions 1 to 5, 9 and 12, Stories 1 to 3 (the backup's part is in [release and deploy](release-and-deploy.md)). - [Spec 085](../specs/085-price-source-contract/spec.md): decisions 1 to 12, Stories 1 to 4 (the contract, Scryfall's source, the source and series rows, the guards). +- [Spec 089](../specs/089-daily-price-refresh/spec.md): decisions 1 to 8, Stories 1 to 5 (the daily refresh, its run record and guards, the removal rules, the schedule). - [Research 083](../specs/083-non-english-printing-prices/research.md) and [ruling 41](../roadmap.md#2026-11-13-prices-on-non-english-printings-research-083) (quotes store what the file carries). - [ADR 0029](../adr/0029-prices-database-held-by-the-backup.md). - [Price data design](../superpowers/specs/2026-11-13-price-data-design.md) (2026-11-13); roadmap rulings 35 to 40; rulings 27, 37 and 38 (the hosted flag). -- Issues [#363](https://github.com/plainprogrammer/Collector/issues/363) (this database), [#375](https://github.com/plainprogrammer/Collector/issues/375) (a start that refuses a prices file with no primary: built by [spec 086](../specs/086-start-refuses-orphan-prices/spec.md), whose rules, the check and its message, are in [release and deploy](release-and-deploy.md)), [#364](https://github.com/plainprogrammer/Collector/issues/364) (the contract), [#362](https://github.com/plainprogrammer/Collector/issues/362) (Scryfall's written answer, open; it may change the hosted flag). +- Issues [#363](https://github.com/plainprogrammer/Collector/issues/363) (this database), [#375](https://github.com/plainprogrammer/Collector/issues/375) (a start that refuses a prices file with no primary: built by [spec 086](../specs/086-start-refuses-orphan-prices/spec.md), whose rules, the check and its message, are in [release and deploy](release-and-deploy.md)), [#364](https://github.com/plainprogrammer/Collector/issues/364) (the contract), [#362](https://github.com/plainprogrammer/Collector/issues/362) (Scryfall's written answer, open; it may change the hosted flag), [#365](https://github.com/plainprogrammer/Collector/issues/365) (the daily refresh). ## Distillation log @@ -87,3 +117,4 @@ Neighbouring areas: the backup that holds the prices database, its format and it | 2026-11-13 | Spec 081 (decisions 1 to 5, 9 and 12: the prices database, its five tables, `WITHOUT ROWID` with the SQL schema file, the index exception, `PricesRecord`, the rules that keep prices global and apart; this doc's first distillation) | [#405](https://github.com/plainprogrammer/Collector/pull/405) | | 2026-11-13 | Spec 085 (decisions 1 to 12: the price source contract and its registry, the Scryfall source's quotes, the rows of sources and series written from a declaration, the hosted flag, the core guard; ruling 41 distilled) | [#413](https://github.com/plainprogrammer/Collector/pull/413) | | 2026-11-13 | Spec 086 (a start refuses a volume that holds the prices file and no primary: the pointer to release and deploy, which holds the rules; #375) | [#415](https://github.com/plainprogrammer/Collector/pull/415) | +| 2026-11-13 | Spec 089 (decisions 1 to 8: the daily refresh over the catalog source's file, its job, run record and guard, the date and freshness rules, the four cases, the removal rules and their guards, the errors table, the operator's view of a daily failure, the run model's migrations and the models of the three tables) | spec 089's pull request | diff --git a/docs/lessons/2026-11-13-a-column-named-changed-is-reserved-by-active-record.md b/docs/lessons/2026-11-13-a-column-named-changed-is-reserved-by-active-record.md new file mode 100644 index 00000000..e64b02e2 --- /dev/null +++ b/docs/lessons/2026-11-13-a-column-named-changed-is-reserved-by-active-record.md @@ -0,0 +1,24 @@ +--- +date: 2026-11-13 +spec: 089 (#365) +tags: [migrations, active-record, specs, reviews] +--- + +# Lesson: a column named `changed` can't be a model's attribute + +## Context + +Spec 089's decision 2 named the run's counts `quotes_seen`, `unchanged`, `changed`, `added`, `removed` and more, as columns of `price_refresh_runs`; four acceptance criteria repeated the name. The spec went through two review rounds, the plan through two more, and the first migration was written and committed. + +## What happened + +The first model of the table, written in Phase 2, raised `ActiveRecord::DangerousAttributeError: changed is defined by Active Record` on every `create!`. `changed`, `changed?` and `changed=` are Active Record's dirty-tracking methods, so the name can't be an attribute, and overriding the check would break dirty tracking. Nothing in four review rounds noticed, because no reviewer builds a model. + +## What to do next time + +- When a spec or a plan names columns, check each name against Active Record's reserved methods (`changed`, `changes`, `save`, `errors`, `attributes`, `new_record?` and the like) before the spec is approved; a one-line `ActiveRecord::Base.dangerous_attribute_method?(name)` in a scratch runner call settles it. Put the check in the review brief for any spec that adds a table. +- A fix after a migration is committed but unreleased is a second migration (`rename_column`) when the development database has already applied the first and `db:rollback` is off limits; the area doc says why there are two. + +## Signals to watch for + +- A count or a state named after a verb's participle (`changed`, `saved`, `destroyed`, `errors`) in a column table. diff --git a/docs/lessons/2026-11-13-deleting-by-absence-needs-a-candidate-rule-per-dimension.md b/docs/lessons/2026-11-13-deleting-by-absence-needs-a-candidate-rule-per-dimension.md new file mode 100644 index 00000000..83f870c7 --- /dev/null +++ b/docs/lessons/2026-11-13-deleting-by-absence-needs-a-candidate-rule-per-dimension.md @@ -0,0 +1,24 @@ +--- +date: 2026-11-13 +spec: 089 (#365) +tags: [data-safety, history, specs, reviews] +--- + +# Lesson: deleting by absence needs a candidate rule for every dimension a file can be silent about + +## Context + +Spec 089's daily refresh removes a price (and records the removal in history that can't be downloaded again) when the source's file has no quote for it. The spec's decision 4 limited "absent" to prices the file was asked about: a printing in a language the run read, whose entry exists. Two spec review rounds and two plan reviews accepted that. + +## What happened + +The implementation review found the dimension nobody listed: the series. A price in a series the source no longer declares (a later build drops one) has no quote in any file, so every run would see it absent, and either fail on the ceiling every day or, for a small series, blank prices the file was never asked about. The fix is the same reasoning as the language rule: only what the source declares today is a candidate (spec 089 v0.5.0). + +## What to do next time + +- When a rule deletes on absence, list every dimension in which the data could be silent for a reason that isn't "gone": language, series or kind, a printing the other database lacks, a source that stopped declaring something, a file that was truncated. Write the candidate rule per dimension in the spec, and add one example per dimension that shows the price kept. +- Prefer a ceiling and a counted, withheld removal to an unconditional delete; the ceiling is what would have shown this one in production. + +## Signals to watch for + +- A removal rule stated only in terms of the data (what the file lacks) and not of what the file was asked for. diff --git a/docs/lessons/2026-11-13-transaction-scope-in-a-transactional-suite-is-savepoint-spans.md b/docs/lessons/2026-11-13-transaction-scope-in-a-transactional-suite-is-savepoint-spans.md new file mode 100644 index 00000000..2cc92635 --- /dev/null +++ b/docs/lessons/2026-11-13-transaction-scope-in-a-transactional-suite-is-savepoint-spans.md @@ -0,0 +1,24 @@ +--- +date: 2026-11-13 +spec: 089 (#365) +tags: [testing, transactions, sqlite] +--- + +# Lesson: in a transactional suite, measure savepoint spans, not transaction depth + +## Context + +Spec 089 AC-4.4 first said that every write to the price tables is at transaction depth 1 and every write to the run row at depth 0, "counted through the connection's open transactions". + +## What happened + +The suite wraps each example in a non-joinable fixture transaction, so every `update!` or `create!` outside an explicit transaction opens a savepoint of its own: a run-row write and a batch's write both read depth 1, and depth could not tell them apart (a plan review found it by reading Active Record). The proof that holds cuts the connection's `sql.active_record` statements into spans from `SAVEPOINT` to its `RELEASE` or `ROLLBACK TO`, and asserts what each span contains. Rails opens the savepoint lazily, at the transaction's first statement on that connection, so "no read of the other database inside a transaction" has to compare that connection's open transactions to a baseline taken before the run: a read made before the first statement is in no span, and a first version of the check passed on a deliberately wrong implementation. + +## What to do next time + +- Specify and write a transaction-scope proof as spans of statements, and prove each assertion can fail by breaking the property in a throwaway way (a write outside the transaction; a read of the other database inside it) before trusting it. +- Never assert a depth in a spec that runs under transactional fixtures. + +## Signals to watch for + +- An acceptance criterion that says "at depth N", or a check that counts `open_transactions` without a baseline. diff --git a/docs/roadmap.md b/docs/roadmap.md index dbf78279..e734e44a 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -110,7 +110,7 @@ Runs beside the main line, when the maintainer starts it (rulings 35–41; [desi - [#363](https://github.com/plainprogrammer/Collector/issues/363) A prices database that the backup holds. - [#375](https://github.com/plainprogrammer/Collector/issues/375) A start refuses a volume that holds a prices database and no primary (after #363; spec 081 decision 8). Done: [spec 086](specs/086-start-refuses-orphan-prices/spec.md). - [#364](https://github.com/plainprogrammer/Collector/issues/364) A price source contract, and Scryfall's prices through it (after #361 and #363). Done: [spec 085](specs/085-price-source-contract/spec.md). -- [#365](https://github.com/plainprogrammer/Collector/issues/365) Prices refresh once a day, and each change is recorded once. +- [#365](https://github.com/plainprogrammer/Collector/issues/365) Prices refresh once a day, and each change is recorded once. Done: [spec 089](specs/089-daily-price-refresh/spec.md). - [#366](https://github.com/plainprogrammer/Collector/issues/366) Each price refresh records what it did and how large the database is. - [#367](https://github.com/plainprogrammer/Collector/issues/367) An admin reads the price refresh log. - [#368](https://github.com/plainprogrammer/Collector/issues/368) An operator chooses how much price history the instance keeps (after #365). diff --git a/docs/specs/089-daily-price-refresh/plan.md b/docs/specs/089-daily-price-refresh/plan.md new file mode 100644 index 00000000..55487e0c --- /dev/null +++ b/docs/specs/089-daily-price-refresh/plan.md @@ -0,0 +1,1833 @@ +# Implementation Plan: Prices Refresh Once a Day, and Each Change Is Recorded Once + +**Spec:** docs/specs/089-daily-price-refresh/spec.md (v0.4.3: the count `changed` is `repriced`; v0.4.2: v0.4.1 approved at Gate A, 2026-11-13; 0.4.2 patches AC-4.4's method after plan review) +**Decisions:** none in an ADR (the decisions are the spec's; the area doc carries the rules) +**Created:** 2026-11-13 +**Status:** Approved at Gate B by the coordinator under the maintainer's delegation (2026-11-13), after two plan review rounds (Fable 5.2); the coordinator read the round 2 fix diff (4c65fdb9) itself + +## Global Constraints + +- Prices are global: no `account_id`, no `Current.account`, nothing in the export (spec NFR Security). +- A printing is keyed by `catalog_entries.id` with no foreign key; **no join or attach across the two database files** (the suite has both, so one fails). +- `current_prices` and `price_changes` stay `WITHOUT ROWID` with **no index beyond the primary key**; the migration of this plan touches only `price_refresh_runs`, a rowid table (`prices.md`, `sqlite-and-migrations.md`). +- A released migration is never edited: this plan adds `db/prices_migrate/20261113190001_prepare_price_refresh_runs.rb` and leaves `20261113100001` alone. +- A `Malformed` quote is unreadable, not absent: the refresh never deletes or blanks a price it failed to parse (spec 085; spec decision 4). +- No network call, and no read of the primary database, inside a transaction on the prices database; the run row's progress write is outside the batch transaction (spec decision 7). +- Core files name no collectible, source or marketplace, comments included: `mtg`, `magic`, `scryfall`, `tcgplayer`, `cardmarket` as substrings, so not even "magical" (`spec/price_source_core_spec.rb`; spec decision 8). The only place the source's key is written is `config/recurring.yml`. +- No real network in the suite (WebMock/VCR). No agent runs the `sqlite3` command, `bin/rails console`, `db:rollback`, `db:migrate:down`, `db:seed`, or anything with `RAILS_ENV=production`; never `bin/jobs`. +- Public code gets a YARD comment (summary, `@param`/`@return`/`@raise`, side effects, idempotency); constants say why they have their value; `{…}` stays out of comments; the core never names a collectible (`.claude/rules/code-documentation.md`). +- Array and hash literals use RuboCop omakase spacing (`[ a, b ]`). Tag spec types explicitly (`type: :model`, `type: :job`); multi-expectation examples use `:aggregate_failures`. +- RuboCop's RSpec department is on with `ExampleLength` at 15 lines: an example longer than that carries `# rubocop:disable RSpec/ExampleLength -- ` on its `it` line (the repo's precedent); an example with several expectations carries `:aggregate_failures`; use `describe`, not `context`, for a group named without "when/with/without". +- Targeted runs only while implementing (`bin/rspec `); one example at a time on the shared test database; no full `bin/ci` until the coordinator gives the slot. +- Every implementer's prompt says: never run the `sqlite3` command or anything else on the permission-ask list. +- The area docs (`prices.md`, `catalog-operations.md`) are written by `distill-feature` after the implementation review (the run's finish order); Phase 9 lists what they must say, and the other documents are written in Phase 9. + +--- + +## Goal + +A Solid Queue job, scheduled daily, reads the catalog source's bulk file, judges each price against the current price, writes a history row only where a price changed (or went away, when the file can be trusted to say so), and records the run, in the prices database, without ever overlapping itself. + +## File map + +| File | Responsibility | +|---|---| +| `db/prices_migrate/20261113190001_prepare_price_refresh_runs.rb`, `db/prices_structure.sql` | the run table's nullable `run_on`, `job_id`, `languages` and counts (decision 2) | +| `app/models/current_price.rb`, `app/models/price_change.rb` | composite-key models of the two large tables; bulk writes by key | +| `app/models/price_refresh_run.rb` | the run record and its guard: `start!`, `skip!`, `finish!`, `fail!`, `progress!`, `heartbeat`, the lookups for "applied" and "older" | +| `app/models/prices/refresh.rb` | the decisions: skip rules, reading, the four cases, absence and its guards | +| `app/jobs/prices/refresh_job.rb` | the queue's face: lock, retries, rate-limit module | +| `app/models/mtg/scryfall/source.rb`, `app/models/catalog/sources.rb` | unique partial download name, stale partial pruning, the contract's comment (one change, one commit) | +| `config/recurring.yml` | the schedule | +| `spec/support/price_refresh_helpers.rb`, `spec/support/fake_price_source.rb` | the specs' fakes and builders | + +Three components (the run record, the refresh, the job) plus two thin models: within the Simplicity Gate. No wrapper types: Active Record's `upsert_all` and `delete_all` are used directly (Anti-Abstraction Gate). + +--- + +## Phase 0: Specs' fakes and the failing contract of the job + +**Implements:** FR-1, FR-2 (the shape only) | **Satisfies:** none yet (it makes the later phases' tests writable) +**Files:** `spec/support/price_refresh_helpers.rb`, `spec/support/fake_price_source.rb` +**Interfaces:** Consumes: `FakePriceSource`, `FakeCatalogSource`, the `:catalog_entry` factory. Produces: `use_widget_sources(records:, reported_on:, series:, version:, languages:)` (registers a price source `fake_source` and a catalog source `widgets`; undone after the example), `widget_quote(key, cents, series: PLAIN)`, `widget_malformed(key = nil, series: nil)`, `widget_entry(key, **attributes)`, `price_source_row`, `series_row(series = PLAIN)`, `seed_price(entry, cents, series: PLAIN, on:)`, the constant `PriceRefreshHelpers::PLAIN`. + +- [ ] **T0.1** Edit `spec/support/fake_price_source.rb`: a record that responds to `call` is called instead of yielded (so an example can raise after N quotes). Replace `@records.each(&block)` in `#each_quote` with: + + ```ruby + @records.each { |record| record.respond_to?(:call) ? record.call : block.call(record) } + ``` + + Update its header comment: "A record that responds to `call` is called, not yielded, so an example can raise partway." + +- [ ] **T0.2** Create `spec/support/price_refresh_helpers.rb`: + + ```ruby + # Builders and fakes for the daily price refresh's specs (spec 089): a fake price source and a fake catalog source for + # a made-up collectible, "widgets", registered for one example with `use_widget_sources`. + # + # @api private + module PriceRefreshHelpers + PLAIN = Prices::Sources::Series.new(finish: "plain", currency: "USD", kind: "market") + DAY = Date.new(2026, 11, 13) + + # Registers `fake_source` (prices) and `widgets` (catalog) for this example; both are undone after it. + # + # @param records [Array] what the price source yields from `each_quote` (a record that responds to `call` is called) + # @param reported_on [Date, Exception] what `#reported_on` answers + # @param series [Array, nil] the series it declares, or the one plain series + # @param version [String] the catalog source's current version + # @param languages [Array] the catalog source's languages + def use_widget_sources(records: [], reported_on: DAY, series: nil, version: "v1", languages: [ "en" ]) + price_class = Class.new(FakePriceSource) do + define_singleton_method(:series) { series } if series + define_method(:initialize) { |*| super(records:, reported_on:) } + end + catalog_class = Class.new(FakeCatalogSource) do + define_method(:initialize) { |*| super(version:, languages:) } + end + stub_const("WidgetPrices", price_class) + stub_const("WidgetCatalog", catalog_class) + Prices.sources["fake_source"] = "WidgetPrices" + Catalog.sources["widgets"] = "WidgetCatalog" + @widget_sources = true + price_class + end + + def widget_quote(key, cents, series: PLAIN, on: DAY) + Prices::Sources::Quote.new(external_key: key, finish: series.finish, currency: series.currency, kind: series.kind, cents:, reported_on: on) + end + + def widget_malformed(key = nil, series: nil) = Prices::Sources::Malformed.new(external_key: key, error: "unreadable", series:) + + def widget_entry(key, **attributes) = create(:catalog_entry, collectible_type: "widgets", external_key: key, **attributes) + + def price_source_row = PriceSource.sync!(FakePriceSource) # the same row `fake_source` has, before an example registers its own + + def series_row(series = PLAIN) = PriceSeries.ensure!([ series ]).fetch(series) + + # A current price and its history row, as an earlier run would have left them. + def seed_price(entry, cents, series: PLAIN, on: Date.new(2026, 11, 12)) + keys = { catalog_entry_id: entry.id, price_source_id: price_source_row.id, price_series_id: series_row(series).id } + CurrentPrice.insert_all!([ keys.merge(cents:, reported_on: on, changed_on: on) ]) + PriceChange.insert_all!([ keys.merge(recorded_on: on, cents:) ]) + end + end + + RSpec.configure do |config| + config.include PriceRefreshHelpers + config.after do + next unless @widget_sources + + Prices.sources.delete("fake_source") + Catalog.sources.delete("widgets") + end + end + ``` + +- [ ] **T0.3** Run `bin/rspec spec/models/prices_spec.rb spec/models/prices/sources_spec.rb` — expect PASS (the fake's change breaks nothing). Commit: `test: add the daily price refresh's fakes and builders (spec 089)`. + +--- + +## Phase 1: The run table fits the refresh + +**Implements:** Decision 2, FR-3 | **Satisfies:** AC-5.2, AC-1.9 (the table accepts a nil `run_on`) +**Files:** `db/prices_migrate/20261113190001_prepare_price_refresh_runs.rb`, `db/prices_structure.sql`, `spec/prices_schema_spec.rb` +**Interfaces:** Consumes: the table of `20261113100001_create_price_tables.rb`. Produces: `price_refresh_runs` with `run_on` nullable, `job_id` and `languages` (nullable strings) and the integer counts `quotes_seen`, `unchanged`, `repriced` (T1.6 renames the `changed` that T1.3 adds), `added`, `removed`, `removals_withheld`, `unknown_printings`, `malformed` (`null: false, default: 0`). + +The rebuild touches a rowid table (`price_refresh_runs`), so the `WITHOUT ROWID` trap of the two large tables doesn't apply; the schema spec proves they are untouched. + +- [ ] **T1.1** Failing test. In `spec/prices_schema_spec.rb`, give `migrated_database` an optional target and an existing path (the child script reads `ARGV[2]` as the version to migrate to), and add the examples. Replace the helper with: + + ```ruby + # A database built only by running db/prices_migrate's migrations, in a child Ruby process with Active Record alone. + # `through` stops at a version (nil: all); `path` continues a database already begun. + def migrated_database(through: nil, path: nil) + path ||= Pathname(Dir.mktmpdir("prices-migrated")).join("prices.sqlite3") + script = <<~RUBY + require "bundler/setup" + require "active_record" + ActiveRecord::Base.establish_connection(adapter: "sqlite3", database: ARGV[0]) + ActiveRecord::Migration.verbose = false + target = ARGV[2].to_s.empty? ? nil : ARGV[2].to_i + ActiveRecord::MigrationContext.new(ARGV[1]).migrate(target) + RUBY + _out, err, status = Open3.capture3(RbConfig.ruby, "-e", script, path.to_s, Rails.root.join("db/prices_migrate").to_s, through.to_s, chdir: Rails.root.to_s) + raise "migrating failed: #{err}" unless status.success? + + path + end + + def execute_in(path, sql) + database = SQLite3::Database.new(path.to_s) + database.execute(sql) + ensure + database&.close + end + ``` + + Add inside the describe: + + ```ruby + context "with spec 089's migration (AC-5.2, AC-1.9)" do + let(:released) { "20261113100001" } + let(:count_columns) { %w[quotes_seen unchanged changed added removed removals_withheld unknown_printings malformed] } + + def columns_of(path) + execute_in(path, "PRAGMA table_info(price_refresh_runs)").to_h { |_cid, name, type, notnull, default, _pk| [ name, { type:, notnull:, default: } ] } + end + + it "keeps a run row of a database migrated as main has it, and its table's check, key and index", :aggregate_failures do + path = migrated_database(through: released) + execute_in(path, "INSERT INTO price_sources (key, name, attribution, created_at, updated_at) VALUES ('s', 'S', 'a', '2026-11-13', '2026-11-13')") + execute_in(path, "INSERT INTO price_refresh_runs (price_source_id, run_on, status, started_at, created_at, updated_at) VALUES (1, '2026-11-12', 'finished', '2026-11-12 10:00:00', '2026-11-12', '2026-11-12')") + + migrated_database(path:) + columns = columns_of(path) + master = execute_in(path, "SELECT name, sql FROM sqlite_master").to_h + + expect(execute_in(path, "SELECT run_on, quotes_seen, job_id FROM price_refresh_runs")).to eq([ [ "2026-11-12", 0, nil ] ]) + expect(columns["run_on"][:notnull]).to eq(0) + expect(columns.keys).to include("job_id", "languages", *count_columns) + expect(count_columns.map { |name| columns[name].values_at(:notnull, :default) }.uniq).to eq([ [ 1, "0" ] ]) + expect(master["price_refresh_runs"]).to include("price_refresh_runs_status_known", "REFERENCES \"price_sources\"") + expect(master).to have_key("index_price_refresh_runs_on_price_source_id_and_run_on") + expect(rowid_problems(definitions(master_of(path)))).to be_empty + end + + it "accepts a run with no date and refuses an unknown status", :aggregate_failures do + path = migrated_database + execute_in(path, "INSERT INTO price_sources (key, name, attribution, created_at, updated_at) VALUES ('s', 'S', 'a', '2026-11-13', '2026-11-13')") + + expect { execute_in(path, "INSERT INTO price_refresh_runs (price_source_id, status, started_at, created_at, updated_at) VALUES (1, 'skipped', '2026-11-13', '2026-11-13', '2026-11-13')") }.not_to raise_error + expect { execute_in(path, "INSERT INTO price_refresh_runs (price_source_id, status, started_at, created_at, updated_at) VALUES (1, 'bogus', '2026-11-13', '2026-11-13', '2026-11-13')") }.to raise_error(SQLite3::ConstraintException) + end + + it "goes down without losing a row: a nil run_on takes the date of started_at", :aggregate_failures do + path = migrated_database + execute_in(path, "INSERT INTO price_sources (key, name, attribution, created_at, updated_at) VALUES ('s', 'S', 'a', '2026-11-13', '2026-11-13')") + execute_in(path, "INSERT INTO price_refresh_runs (price_source_id, status, started_at, created_at, updated_at) VALUES (1, 'skipped', '2026-11-13 09:00:00', '2026-11-13', '2026-11-13')") + + migrated_database(through: released, path:) + + expect(execute_in(path, "SELECT run_on FROM price_refresh_runs")).to eq([ [ "2026-11-13" ] ]) + expect(columns_of(path)["run_on"][:notnull]).to eq(1) + expect(columns_of(path).keys).not_to include("job_id", "quotes_seen") + end + end + ``` + +- [ ] **T1.2** Run `bin/rspec spec/prices_schema_spec.rb` — expect FAIL (the new examples: no such columns). + +- [ ] **T1.3** Create `db/prices_migrate/20261113190001_prepare_price_refresh_runs.rb`: + + ```ruby + # What the daily refresh needs of price_refresh_runs (spec 089 decision 2; supersedes the run_on column of spec 081). + # + # run_on becomes nullable: a run opens before its file's date is known, and some end before it is (already running, an + # empty catalog, a transient error). job_id is for the stall rule; languages and the counts say what the run did. + # + # price_refresh_runs is a rowid table, so Rails' table rebuild for change_column_null is safe here, unlike for + # current_prices and price_changes (see the first prices migration's warning). The columns are added first and the + # rebuild comes last, so the stored table SQL is the canonical one. `down` gives a nil run_on the date of started_at, + # so it never loses a row. + class PreparePriceRefreshRuns < ActiveRecord::Migration[8.1] + COUNTS = %i[quotes_seen unchanged changed added removed removals_withheld unknown_printings malformed].freeze + + def up + add_column :price_refresh_runs, :job_id, :string + add_column :price_refresh_runs, :languages, :string + COUNTS.each { |name| add_column :price_refresh_runs, name, :integer, null: false, default: 0 } + change_column_null :price_refresh_runs, :run_on, true + end + + def down + execute "UPDATE price_refresh_runs SET run_on = date(started_at) WHERE run_on IS NULL" + change_column_null :price_refresh_runs, :run_on, false + remove_columns :price_refresh_runs, *COUNTS, :languages, :job_id + end + end + ``` + +- [ ] **T1.4** Run `bin/rails db:migrate` (development; it re-dumps `db/prices_structure.sql`; the `sqlite3` command is run by Rails, not by you), then `bin/rails db:prepare`. `db:migrate` migrates and dumps every development database: stage only `db/prices_structure.sql` of what it generates, and check `git status` that `db/schema.rb` and the other structure files didn't change. Check `git diff db/prices_structure.sql`: `price_refresh_runs` has the new columns, `run_on` without `NOT NULL`, the status check, the foreign key and the index, a new `schema_migrations` row, and **the two `WITHOUT ROWID` tables unchanged**. If the test database is stale, `bin/rails db:test:prepare`. + +- [ ] **T1.5** Run `bin/rspec spec/prices_schema_spec.rb spec/backup_guards_spec.rb` and `ruby script/upgrade/guard.rb` — expect PASS. Commit (stage the migration, the structure file and the spec; the structure file is generated, never hand-edited): `feat(prices): make price_refresh_runs fit the daily refresh (spec 089 decision 2)`. + +- [ ] **T1.6** (Added at execution, spec v0.4.3.) Active Record reserves the attribute name `changed`, so a model of `price_refresh_runs` can't be built with the column T1.3 added. A second migration renames it: `db/prices_migrate/20261113190002_rename_price_refresh_runs_changed.rb`: + + ```ruby + # Active Record reserves the attribute name `changed` (ActiveRecord::DangerousAttributeError), so the count of repriced + # quotes that 20261113190001 added under that name is `repriced` (spec 089 decision 2, v0.4.3). A rowid table, so the + # table rebuild of rename_column is safe; the migration before it is unreleased and stays as built. + class RenamePriceRefreshRunsChanged < ActiveRecord::Migration[8.1] + def change + rename_column :price_refresh_runs, :changed, :repriced + end + end + ``` + + Run `bin/rails db:migrate` (stage only `db/prices_structure.sql` of what it generates), `bin/rails db:test:prepare`; the schema spec's `count_columns` list uses `repriced`; `spec/prices_core_spec.rb`'s file count (`expect(files.size).to eq(...)`, which says "a new migration raises this") becomes 5 (two migrations, the structure file and the class). Commit: `fix(prices): count repriced quotes under a name Active Record allows (spec 089 v0.4.3)`. + +--- + +## Phase 2: The models of the three tables, and the run's guard + +**Implements:** Decisions 2, 3, 6 (the keys), 7 (the guard), FR-2 | **Satisfies:** AC-4.1, AC-4.2, AC-1.9, AC-2.8 (keys) +**Files:** `app/models/current_price.rb`, `app/models/price_change.rb`, `app/models/price_refresh_run.rb`, `spec/models/current_price_spec.rb`, `spec/models/price_change_spec.rb`, `spec/models/price_refresh_run_spec.rb`, `spec/price_source_core_spec.rb` +**Interfaces:** Consumes: `PricesRecord`, `PriceSource`, Phase 1's columns. Produces: +`CurrentPrice` and `PriceChange` (composite primary keys; `upsert_all(rows, unique_by: Model.primary_key)` works on them), +`PriceRefreshRun::COUNTS`, `STALL_AFTER` (15 minutes), `HEARTBEAT_EVERY` (30 seconds), scopes `running`, `finished`, +`PriceRefreshRun.start!(source, job_id:, now: Time.current)` → a `running` run, or a `skipped` one with `error: "already running"` (after closing runs left behind as `failed`/`interrupted`), +`#left_behind?(job_id:, now:)`, `#running?`, `#record_file!(version:, languages:)`, `#skip!(reason, run_on: nil)`, `#finish!(counts)`, `#fail!(error, counts)`, `#progress!(counts)`, `#heartbeat(*)`, +`PriceRefreshRun.applied(source, version, languages)` (finished runs), `PriceRefreshRun.latest_run_on(source)` (the latest `run_on` of finished or failed runs). + +- [ ] **T2.1** Failing tests. `spec/models/current_price_spec.rb`: + + ```ruby + require "rails_helper" + + RSpec.describe CurrentPrice, type: :model do + it "is keyed by printing, source and series, and an upsert on that key replaces the row", :aggregate_failures do + source = PriceSource.sync!(FakePriceSource) + series = PriceSeries.ensure!(FakePriceSource.series).values.first + keys = { catalog_entry_id: 7, price_source_id: source.id, price_series_id: series.id } + + described_class.upsert_all([ keys.merge(cents: 100, reported_on: Date.new(2026, 11, 12), changed_on: Date.new(2026, 11, 12)) ], unique_by: described_class.primary_key) + described_class.upsert_all([ keys.merge(cents: 150, reported_on: Date.new(2026, 11, 13), changed_on: Date.new(2026, 11, 13)) ], unique_by: described_class.primary_key) + + expect(described_class.primary_key).to eq(%w[catalog_entry_id price_source_id price_series_id]) + expect(described_class.pluck(:cents, :changed_on)).to eq([ [ 150, Date.new(2026, 11, 13) ] ]) + end + + it "lives in the prices database and has no account" do + expect([ described_class.connection_db_config.name, described_class.column_names & %w[account_id user_id] ]).to eq([ "prices", [] ]) + end + end + ``` + + `spec/models/price_change_spec.rb` is the same shape: key `catalog_entry_id price_source_id price_series_id recorded_on`; upsert twice with cents `100` then `nil` (a blank row) and expect `pluck(:cents)` to eq `[ nil ]`. + + `spec/models/price_refresh_run_spec.rb`: + + ```ruby + require "rails_helper" + + RSpec.describe PriceRefreshRun, type: :model do + let(:source) { PriceSource.sync!(FakePriceSource) } + let(:now) { Time.utc(2026, 11, 13, 12, 0, 0) } + + def run!(status: "running", job_id: "job-1", updated_at: now, **attributes) + described_class.create!(price_source: source, status:, job_id:, started_at: updated_at, updated_at:, **attributes) + end + + describe ".start!" do + it "opens a running run with its job id and the time", :aggregate_failures do + run = described_class.start!(source, job_id: "job-2", now:) + + expect(run).to have_attributes(status: "running", job_id: "job-2", started_at: now, run_on: nil, finished_at: nil) + end + + it "records a skipped run, 'already running', while another run has recent progress (AC-4.1)", :aggregate_failures do + busy = run!(updated_at: now - 5.minutes) + + skipped = described_class.start!(source, job_id: "job-2", now:) + + expect(skipped).to have_attributes(status: "skipped", error: "already running", finished_at: now) + expect(busy.reload.status).to eq("running") + end + + it "closes a run with no progress for STALL_AFTER as interrupted and goes ahead (AC-4.2)", :aggregate_failures do + left = run!(updated_at: now - described_class::STALL_AFTER - 1.minute) + + run = described_class.start!(source, job_id: "job-2", now:) + + expect(left.reload).to have_attributes(status: "failed", error: "interrupted", finished_at: now) + expect(run.status).to eq("running") + end + + it "closes a running run that carries the starting job's own id (the queue re-ran it) (AC-4.2)", :aggregate_failures do + left = run!(job_id: "job-1", updated_at: now - 1.minute) + + run = described_class.start!(source, job_id: "job-1", now:) + + expect(left.reload).to have_attributes(status: "failed", error: "interrupted") + expect(run.status).to eq("running") + end + + it "leaves another source's runs alone" do + other = PriceSource.create!(key: "other", name: "Other", attribution: "text") + elsewhere = described_class.create!(price_source: other, status: "running", started_at: now, updated_at: now - 1.hour) + + described_class.start!(source, job_id: "job-2", now:) + + expect(elsewhere.reload.status).to eq("running") + end + end + + describe "#skip!, #finish!, #fail!" do + it "ends a run with its reason or counts", :aggregate_failures do + a = run!.tap { it.skip!("v1 already applied", run_on: Date.new(2026, 11, 13)) } + b = run!(job_id: "x").tap { it.finish!(unchanged: 3, repriced: 1, bogus: 9) } + c = run!(job_id: "y").tap { it.fail!(RuntimeError.new("boom"), added: 2) } + + expect(a).to have_attributes(status: "skipped", error: "v1 already applied", run_on: Date.new(2026, 11, 13)) + expect(b).to have_attributes(status: "finished", unchanged: 3, repriced: 1) + expect(c).to have_attributes(status: "failed", error: "RuntimeError: boom", added: 2) + expect([ a, b, c ].map(&:finished_at)).to all(be_present) + end + end + + describe ".applied and .latest_run_on" do + it "finds a finished run of the same version and languages only", :aggregate_failures do + finished = run!(status: "finished", source_version: "v1", languages: "en", run_on: Date.new(2026, 11, 13)) + run!(status: "failed", job_id: "a", source_version: "v1", languages: "en") + run!(status: "finished", job_id: "b", source_version: "v1", languages: "en,ja") + + expect(described_class.applied(source, "v1", "en")).to eq([ finished ]) + expect(described_class.applied(source, "v2", "en")).to be_empty + end + + it "takes the latest run_on of finished and failed runs, never of skipped or running ones" do + run!(status: "finished", job_id: "a", run_on: Date.new(2026, 11, 10)) + run!(status: "failed", job_id: "b", run_on: Date.new(2026, 11, 12)) + run!(status: "skipped", job_id: "c", run_on: Date.new(2026, 11, 14)) + run!(status: "running", job_id: "d", run_on: Date.new(2026, 11, 15)) + + expect(described_class.latest_run_on(source)).to eq(Date.new(2026, 11, 12)) + end + end + + describe "#progress! and #heartbeat" do + it "stores the counts, and a heartbeat touches the run at most once per HEARTBEAT_EVERY", :aggregate_failures do + run = run!(updated_at: now - 1.hour) + + travel_to(now) { run.heartbeat(1, 10) } + first = run.reload.updated_at + travel_to(now + 10.seconds) { run.heartbeat } + second = run.reload.updated_at + travel_to(now + 31.seconds) do + run.heartbeat + run.progress!(unchanged: 4, malformed: 1) + end + + expect([ first, second ]).to eq([ now, now ]) + expect(run.reload.updated_at).to be >= now + 31.seconds + expect(run).to have_attributes(unchanged: 4, malformed: 1) + end + end + end + ``` + + In `spec/price_source_core_spec.rb` extend the file list: `%w[app/models/prices.rb app/models/prices/**/*.rb app/models/price_source.rb app/models/price_series.rb app/models/current_price.rb app/models/price_change.rb app/models/price_refresh_run.rb app/jobs/prices/**/*.rb]` (spec 089 decision 8). + +- [ ] **T2.2** Run `bin/rspec spec/models/current_price_spec.rb spec/models/price_change_spec.rb spec/models/price_refresh_run_spec.rb` — expect FAIL (`uninitialized constant`). + +- [ ] **T2.3** Implement. `app/models/current_price.rb`: + + ```ruby + # The latest known price of one printing in one series from one source, in the prices database (spec 081; spec 089). + # Keyed by printing (a catalog entry's id, with no foreign key: `prices.md`), source and series; the table is + # WITHOUT ROWID with no other index, so every lookup goes by printing, the key's first column. + # + # The daily refresh writes it with `upsert_all` and `delete_all` by key, which skip validations on purpose: each row comes + # from a validated `Prices::Sources::Quote`, and the table's own checks refuse the rest. + # + # @see file:prices.md + class CurrentPrice < PricesRecord + self.primary_key = %i[catalog_entry_id price_source_id price_series_id] + end + ``` + + `app/models/price_change.rb`: + + ```ruby + # One row of price history, in the prices database: the price of a printing in a series from a source on the day it + # differed from the last one recorded; a row with no cents records that the price went away that day (spec 081; spec + # 089). WITHOUT ROWID, keyed by printing, source, series and day, with no other index. Written by the daily refresh with + # `upsert_all` on that key, so a second write for the same day replaces the first (validations are skipped on purpose, + # as for `CurrentPrice`). + # + # @see file:prices.md + class PriceChange < PricesRecord + self.primary_key = %i[catalog_entry_id price_source_id price_series_id recorded_on] + end + ``` + + `app/models/price_refresh_run.rb`: + + ```ruby + # One run of the daily price refresh, in the prices database (spec 089 decision 2): its identity (source, the file's + # day, version and languages), its times, its status and the counts of what it did to rows. The run is also the guard + # against overlap and against a run left behind (decision 7). Measurements of cost are #366's. + # + # `run_on` is nil until the file's date is known; `error` holds why a run failed or was skipped. + # + # @see file:prices.md + class PriceRefreshRun < PricesRecord + # What a run counts (spec 089 decision 2); the columns have these names and default to 0. + COUNTS = %i[quotes_seen unchanged repriced added removed removals_withheld unknown_printings malformed].freeze + # A running run with no progress for this long is left behind. The catalog refresh's own threshold (spec 089 decision 7). + STALL_AFTER = 15.minutes + # The most often the run row is touched while a download reports progress, so a long download never reads as stalled. + HEARTBEAT_EVERY = 30.seconds + + belongs_to :price_source + + scope :running, -> { where(status: "running") } + scope :finished, -> { where(status: "finished") } + + # Opens a run for the source, or records that none could be opened. In one transaction it closes each running run that + # was left behind (`#left_behind?`) as `failed` with `interrupted`; then, if a run still runs, it records the new run + # as `skipped` with `already running` (nothing else is touched), otherwise as `running`. + # + # @param source [PriceSource] + # @param job_id [String, nil] the starting job's Active Job id + # @param now [Time] + # @return [PriceRefreshRun] `running`, or `skipped` when another run is working + def self.start!(source, job_id:, now: Time.current) + transaction do + running.where(price_source: source).select { it.left_behind?(job_id:, now:) } + .each { it.update!(status: "failed", error: "interrupted", finished_at: now) } + if running.exists?(price_source_id: source.id) + create!(price_source: source, status: "skipped", error: "already running", job_id:, started_at: now, finished_at: now) + else + create!(price_source: source, status: "running", job_id:, started_at: now) + end + end + end + + # The finished runs of a source for one file and language set: what "applied" means (spec 089 decision 5). + # + # @return [ActiveRecord::Relation] + def self.applied(source, version, languages) = finished.where(price_source: source, source_version: version, languages:) + + # The latest day a finished or failed run of the source read a file for (a failed run may have written rows); skipped + # and running runs don't count (spec 089 decision 3). + # + # @return [Date, nil] + def self.latest_run_on(source) = where(price_source: source, status: %w[finished failed]).maximum(:run_on) + + # Whether nothing works on this running run any more: its last progress is older than STALL_AFTER, or it carries the + # starting job's own id (the queue re-ran it after a restart). + # + # @return [Boolean] + def left_behind?(job_id:, now: Time.current) = updated_at < now - STALL_AFTER || (job_id.present? && self.job_id == job_id) + + # Whether the run is still open (not finished, skipped or failed). + # + # @return [Boolean] + def running? = status == "running" + + # Records the file this run is for. + # + # @param version [String] the catalog source's version (the file's name without its extension) + # @param languages [String] the sorted language set, comma-joined + # @return [true] + def record_file!(version:, languages:) = update!(source_version: version, languages:) + + # Ends the run as skipped, with the reason in `error`; `run_on` when it is known. + # + # @param reason [String] + # @param run_on [Date, nil] + # @return [true] + def skip!(reason, run_on: nil) = update!(status: "skipped", error: reason, run_on: run_on || self.run_on, finished_at: Time.current) + + # Ends the run as finished with its counts (keys outside COUNTS are ignored). + # + # @param counts [Hash{Symbol => Integer}] + # @return [true] + def finish!(counts) = update!(status: "finished", finished_at: Time.current, **counted(counts)) + + # Ends the run as failed with `": "` and the counts so far. + # + # @param error [StandardError] + # @param counts [Hash{Symbol => Integer}] + # @return [true] + def fail!(error, counts) = update!(status: "failed", error: "#{error.class}: #{error.message}", finished_at: Time.current, **counted(counts)) + + # Stores the counts so far and counts as progress (the stall rule reads `updated_at`). Written outside any batch's + # transaction. + # + # @param counts [Hash{Symbol => Integer}] + # @return [true] + def progress!(counts) = update!(**counted(counts), updated_at: Time.current) + + # A progress callback for a download (called with bytes done and total): touches the run at most once per + # HEARTBEAT_EVERY. + def heartbeat(*) + now = Time.current + return if @last_beat && now - @last_beat < HEARTBEAT_EVERY + + @last_beat = now + touch + end + + private + def counted(counts) = counts.slice(*COUNTS) + end + ``` + + No status validation is added: the table's check constraint is the guard (as the other prices models). + +- [ ] **T2.4** Run the three model specs and `bin/rspec spec/price_source_core_spec.rb` — expect PASS. Run `bin/rubocop -a` on the files. Commit: `feat(prices): add the models of current prices, price changes and refresh runs (spec 089)`. + +--- + +## Phase 3: A download two jobs can make at once + +**Implements:** Decision 1's shared directory | **Satisfies:** AC-4.8 +**Files:** `app/models/mtg/scryfall/source.rb`, `app/models/catalog/sources.rb`, `spec/models/mtg/scryfall/source_spec.rb` +**Interfaces:** Consumes: the existing `#fetch`, `#prune`. Produces: `MTG::Scryfall::Source::STALE_PARTIAL_AFTER` (1 day); a partial file named `..part` per download; `prune` deletes `*.part` files older than that by mtime; the `Catalog::Sources` contract's comment says `#download` must tolerate concurrent calls for one version. + +This is a contract change (`#download` now has two callers that can run together) and a behaviour change of the catalog's source, so the comment and the behaviour go in **one commit** (`code-documentation.md`, "Extension contracts"). It supersedes spec 002 FR-3's "a partial file". + +- [ ] **T3.1** Failing tests, in `spec/models/mtg/scryfall/source_spec.rb`, inside `describe "#current_version and #download"` (the file's `dir`, `source`, `stub_scryfall` helpers are used by its neighbours): + + ```ruby + it "deletes a partial file left by a killed download once it is a day old, and keeps a fresh one (spec 089 AC-4.8)", :aggregate_failures do + old = dir.join("default-cards-1.jsonl.gz.dead.part") + fresh = dir.join("default-cards-1.jsonl.gz.live.part") + fixed = dir.join("default-cards-0.jsonl.gz.part") # a fixed-name partial file from an earlier build + [ old, fresh, fixed ].each { |path| path.write("x") } + FileUtils.touch(old, mtime: Time.now - 2.days) + FileUtils.touch(fixed, mtime: Time.now - 2.days) + stub_scryfall(cards: [ scryfall_card ]) + version = source.current_version(languages: [ "en" ]) + + source.download(version, dir:) + + expect(dir.children.map { |path| path.basename.to_s }).to contain_exactly("#{version}.jsonl.gz", fresh.basename.to_s) + end + + it "writes each download to a partial file of its own, so a second download of the version, started inside the first, is harmless (spec 089 AC-4.8)", :aggregate_failures do # rubocop:disable RSpec/ExampleLength -- one scenario + cards = [ scryfall_card ] + stub_scryfall(cards:) + version = source.current_version(languages: [ "en" ]) + body = gzip_jsonl(cards) + url = "https://data.scryfall.io/default-cards/#{version}.jsonl.gz" + other = described_class.new(client: MTG::Scryfall::Client.new, env: {}) + other.current_version(languages: [ "en" ]) + calls = 0 + partials_while_nested = nil + stub_request(:get, url).to_return do |_request| + calls += 1 + if calls == 1 + partials_while_nested = dir.glob("*.part").size # the first download's own partial file is open + other.download(version, dir:) + end + { body:, headers: { "Content-Type" => "application/gzip" } } + end + + path = source.download(version, dir:) + + expect(partials_while_nested).to eq(1) + expect(path.size).to eq(body.bytesize) + expect(dir.glob("*.part")).to be_empty + end + ``` + +- [ ] **T3.2** Run `bin/rspec spec/models/mtg/scryfall/source_spec.rb -e "partial file"` — expect FAIL (the old partial is not pruned; the nested download renames the first's partial file away: `Errno::ENOENT` or a wrong size). + +- [ ] **T3.3** Implement in `app/models/mtg/scryfall/source.rb`. Add the constant under `KEEP_DOWNLOADS`: + + ```ruby + # A partial file this old belongs to a killed download: a live one is far younger (a job that never reports ends at its + # 6-hour lock). Each download writes its own partial file, so a killed one leaves an orphan for `prune` to remove + # (spec 089 decision 1, which supersedes spec 002 FR-3's "a partial file"). + STALE_PARTIAL_AFTER = 1.day + ``` + + Change `fetch`'s first line, and `prune`: + + ```ruby + def fetch(url, path, expected) + partial = Pathname("#{path}.#{SecureRandom.hex(4)}.part") + # (rest unchanged, except that the download is wrapped so a failed one deletes its own partial file: + # `rescue StandardError; partial.delete if partial.exist?; raise` around the File.open block; an orphan of + # up to 408 MB would otherwise wait for a later successful download to prune it) + ``` + + ```ruby + # Keeps the newest KEEP_DOWNLOADS files and drops stale partial files. A concurrent download may rename or delete a + # file between the listing and its use, so each file is judged, and removed, tolerating that (spec 089 decision 1). + def prune(dir) + aged = ->(paths) { paths.filter_map { |path| [ path.mtime, path ] rescue nil } } # a file may vanish between glob and stat + aged.call(dir.glob("*.jsonl.gz")).sort_by(&:first).reverse.drop(KEEP_DOWNLOADS).each { |_mtime, path| FileUtils.rm_f(path) } + aged.call(dir.glob("*.part")).select { |mtime, _path| mtime < STALE_PARTIAL_AFTER.ago }.each { |_mtime, path| FileUtils.rm_f(path) } + rescue Errno::ENOENT + nil # the directory itself was removed under us + end + ``` + + In `app/models/catalog/sources.rb`, change the `#download` line of the contract's comment to: + + ``` + # #download(version, dir:) Pathname of the verified local copy (IntegrityError, TransientError). Two callers + # (the catalog refresh and a price refresh) may call it for one version at once, so it + # must tolerate concurrent calls: a partial download is never shared between calls + ``` + +- [ ] **T3.4** Run `bin/rspec spec/models/mtg/scryfall/source_spec.rb spec/jobs/catalog/refresh_job_spec.rb spec/models/catalog/refresh_spec.rb` — expect PASS (the existing size-mismatch example still leaves `dir.children` empty). Commit: `feat(catalog): a download writes a partial file of its own and stale ones are pruned (spec 089 decision 1)`. + +--- + +## Phase 4: The refresh's lifecycle: skips, the date, failure + +**Implements:** Decisions 1 (file), 5, 7 (guard, errors), FR-1, FR-2 | **Satisfies:** AC-1.2, AC-1.3, AC-1.4, AC-1.5, AC-1.8, AC-1.9 (a run that ends early), AC-4.1 (through the refresh), AC-4.2 (through the refresh), AC-4.7 (the download's heartbeat) +**Files:** `app/models/prices/refresh.rb`, `spec/models/prices/refresh_spec.rb` +**Interfaces:** Consumes: Phase 2's `PriceRefreshRun`, `PriceSource.sync!`, `PriceSeries.ensure!`, `Catalog.source_for`, `Prices.source_class`, `Rails.configuration.x.catalog_download_dir`. Produces: `Prices::Refresh.new(source_key, job_id: nil).call` → the `PriceRefreshRun`; `Prices::Refresh::BATCH_SIZE` (1,000), `MAX_REMOVED_SHARE` (`Rational(1, 10)`), `Prices::Refresh::GuardError`. Phase 4 builds `#call`, `#perform` and a stub `#apply` that reads nothing; Phases 5 and 6 fill `#apply`. + +- [ ] **T4.1** Failing tests, `spec/models/prices/refresh_spec.rb` (the first half; later phases add describes to this file): + + ```ruby + require "rails_helper" + + RSpec.describe Prices::Refresh, type: :model do + def refresh(job_id: "job-1") = described_class.new("fake_source", job_id:) + def last_run = PriceRefreshRun.order(:id).last + + describe "before it applies anything" do + it "skips with a reason and no date when the catalog has no printing of the type, asking the catalog's source nothing (AC-1.5)", :aggregate_failures do + use_widget_sources(records: [ widget_quote("w-1", 100) ]) + allow(Catalog).to receive(:source_for).and_call_original + + run = refresh.call + + expect(Catalog).not_to have_received(:source_for) + expect(run).to have_attributes(status: "skipped", run_on: nil) + expect(run.error).to include("no printing") + expect([ CurrentPrice.count, PriceChange.count ]).to eq([ 0, 0 ]) + end + + it "syncs the source before a run exists, so a source that fails verification fails the job with no run (AC-1.8)", :aggregate_failures do + use_widget_sources + Prices.sources["fake_source"] = "WidgetPrices" + broken = Class.new(FakePriceSource) { def self.key = "BAD KEY" } + stub_const("WidgetPrices", broken) + + expect { refresh.call }.to raise_error(Prices::Sources::ConfigurationError, /key/) + expect(PriceRefreshRun.count).to eq(0) + expect { described_class.new("nobody").call }.to raise_error(ArgumentError, /unknown price source/) + expect(PriceRefreshRun.count).to eq(0) + end + + it "syncs the source and its series first, so their rows exist when the run opens (AC-1.8)", :aggregate_failures do + use_widget_sources + widget_entry("w-1") + + expect { refresh.call }.not_to raise_error # Phase 6 makes a file with no quote a GuardError; T6.3 updates this example + + expect(PriceSource.pluck(:key)).to eq([ "fake_source" ]) + expect(PriceSeries.count).to eq(1) + expect(last_run.price_source.key).to eq("fake_source") + end + + it "skips a file already applied for the same version and languages, copying its date (AC-1.2; the end-to-end example proves nothing is downloaded again)", :aggregate_failures do + use_widget_sources(version: "v1") + widget_entry("w-1") + PriceRefreshRun.create!(price_source: price_source_row, status: "finished", source_version: "v1", languages: "en", + run_on: Date.new(2026, 11, 13), started_at: 1.hour.ago) + + run = refresh.call + + expect(run).to have_attributes(status: "skipped", error: "v1 already applied", run_on: Date.new(2026, 11, 13), + source_version: "v1", languages: "en") + end + + it "applies a file of the same version when the language set differs (AC-1.3)", :aggregate_failures do + use_widget_sources(version: "all-1", languages: %w[de en], records: [ widget_quote("w-1", 100) ]) + widget_entry("w-1") + PriceRefreshRun.create!(price_source: price_source_row, status: "finished", source_version: "all-1", languages: "en,ja", + run_on: Date.new(2026, 11, 13), started_at: 1.hour.ago) + + run = refresh.call + + expect(run).to have_attributes(status: "finished", languages: "de,en") + end + + it "skips a file older than the latest finished or failed run (AC-1.4)", :aggregate_failures do + use_widget_sources(reported_on: Date.new(2026, 11, 12), records: [ widget_quote("w-1", 100) ]) + widget_entry("w-1") + PriceRefreshRun.create!(price_source: price_source_row, status: "failed", source_version: "other", languages: "en", + run_on: Date.new(2026, 11, 13), started_at: 2.hours.ago) + + older = refresh.call + + expect(older).to have_attributes(status: "skipped", run_on: Date.new(2026, 11, 12)) + expect(older.error).to include("2026-11-13") + expect([ CurrentPrice.count, PriceChange.count ]).to eq([ 0, 0 ]) + end + + it "skips a file older than a finished run, and applies another version of the finished run's own day (AC-1.4)", :aggregate_failures do + widget_entry("w-1") + PriceRefreshRun.create!(price_source: price_source_row, status: "finished", source_version: "v1", languages: "en", + run_on: Date.new(2026, 11, 13), started_at: 2.hours.ago) + use_widget_sources(reported_on: Date.new(2026, 11, 12), records: [ widget_quote("w-1", 100) ], version: "old") + older = refresh.call + use_widget_sources(reported_on: Date.new(2026, 11, 13), records: [ widget_quote("w-1", 100) ], version: "v2") + + same_day = refresh.call + + expect(older.status).to eq("skipped") + expect(same_day).to have_attributes(status: "finished", run_on: Date.new(2026, 11, 13)) + end + + it "ends the run failed, with its error, when the source's file has no date, and nothing is written (AC-1.9)", :aggregate_failures do + use_widget_sources(reported_on: Prices::Sources::IntegrityError.new("no date")) + widget_entry("w-1") + + expect { refresh.call }.to raise_error(Prices::Sources::IntegrityError) + expect(last_run).to have_attributes(status: "failed", run_on: nil) + expect(last_run.error).to eq("Prices::Sources::IntegrityError: no date") + end + + it "records a skipped run, 'already running', for a run with recent progress, and writes nothing (AC-4.1)", :aggregate_failures do + use_widget_sources(records: [ widget_quote("w-1", 100) ]) + widget_entry("w-1") + PriceRefreshRun.create!(price_source: price_source_row, status: "running", job_id: "other-job", started_at: 1.minute.ago) + + run = refresh(job_id: "job-2").call + + expect(run).to have_attributes(status: "skipped", error: "already running") + expect([ CurrentPrice.count, PriceChange.count ]).to eq([ 0, 0 ]) + end + + it "closes a run left behind as interrupted and goes ahead (AC-4.2)", :aggregate_failures do + use_widget_sources(records: [ widget_quote("w-1", 100) ]) + widget_entry("w-1") + left = PriceRefreshRun.create!(price_source: price_source_row, status: "running", job_id: "dead-job", + started_at: 1.hour.ago, updated_at: 1.hour.ago) + + run = refresh(job_id: "job-2").call + + expect(left.reload).to have_attributes(status: "failed", error: "interrupted") + expect(run.status).to eq("finished") + end + + it "touches the run while the download reports progress, at most once per HEARTBEAT_EVERY (AC-4.7)", :aggregate_failures do + use_widget_sources(records: [ widget_quote("w-1", 100) ]) + widget_entry("w-1") + touches = [] + during = nil + stub_const("WidgetCatalog", Class.new(ReportingCatalogSource) do + define_method(:initialize) { |*| super(version: "v1", languages: [ "en" ]) } + define_method(:download) { |version, dir:| before = touches.size; super(version, dir:).tap { during = touches.size - before } } + end) + callback = ->(*, payload) { touches << payload[:sql] if payload[:sql].to_s.match?(/UPDATE "price_refresh_runs" SET "updated_at" = [^,]+ WHERE/) } + + ActiveSupport::Notifications.subscribed(callback, "sql.active_record") { refresh.call } + + expect(during).to eq(1) # ReportingCatalogSource reports twice within one instant while downloading: one touch, which sets only updated_at + end + end + end + ``` + +- [ ] **T4.2** Run `bin/rspec spec/models/prices/refresh_spec.rb` — expect FAIL (`uninitialized constant Prices::Refresh`). + +- [ ] **T4.3** Implement `app/models/prices/refresh.rb` (Phases 5 and 6 add to the private section; the skeleton below is complete for this phase): + + ```ruby + # One run of the daily price refresh for one price source (spec 089): asks the collectible's catalog source for the bulk + # file it already downloads, judges each quote against the current price and writes only what changed, and records the + # run (`PriceRefreshRun`). It makes no request outside that catalog source, never joins the prices database with the + # primary, and holds no network call or primary read inside a transaction on the prices database. + # + # Side effects: rows of `current_prices`, `price_changes`, `price_sources`, `price_series` and `price_refresh_runs`; + # the catalog source's download directory. It needs no `Current.account` (prices are global). Safe to run twice on one + # file: the second run is skipped as already applied, and a rerun of a failed run changes nothing it already wrote. + # + # @see file:prices.md + class Prices::Refresh + # Printings per write transaction, and printings (their current prices) per removal slice and transaction: a few + # thousand rows each, so the database is held briefly (spec 089 decision 7). + BATCH_SIZE = 1_000 + + # The largest share of the source's candidate current prices a run may remove in one day (spec 089 decision 4, ruled + # by the maintainer on 2026-11-13). A GUESS until #366's run records measure the real daily share of prices that + # disappear: a file that loses a tenth of the market in a day is far more likely truncated or damaged than a market + # event. A real drop above it is refused, every day, until this constant moves. + MAX_REMOVED_SHARE = Rational(1, 10) + + # The file can't be trusted to say prices are gone (decision 4, items 2 and 5); the run fails and nothing is removed. + class GuardError < StandardError; end + + # @param source_key [String] a key of `Prices.sources` + # @param job_id [String, nil] the Active Job id, so a re-run after a restart closes its own left-behind run + def initialize(source_key, job_id: nil) + @source_key = source_key + @job_id = job_id + @counts = Hash.new(0) + end + + # Runs the refresh. The source is looked up and synced before any run exists (a run's source row is required), so an + # unregistered key or a source that fails verification raises with no run recorded; every later failure ends the run + # `failed` with its error and re-raises it for the job's retry rules. + # + # @return [PriceRefreshRun] finished, skipped or (when another run works) skipped as already running + # @raise [ArgumentError, Prices::Sources::ConfigurationError] before a run exists + # @raise [GuardError, Prices::Sources::IntegrityError, Zlib::Error, Catalog::Sources::Error, StandardError] after the run is recorded failed + def call + @source_class = Prices.source_class(@source_key) + @source = PriceSource.sync!(@source_class) + @series = PriceSeries.ensure!(@source_class.series) + @run = PriceRefreshRun.start!(@source, job_id: @job_id) + return @run unless @run.running? + + perform + @run + rescue StandardError => error + @run.fail!(error, @counts) if @run&.running? + raise + end + + private + def collectible_type = @source_class.collectible_type + + def perform + return @run.skip!("the catalog has no printing of the type #{collectible_type} yet") unless Catalog::Entry.exists?(collectible_type:) + + catalog = Catalog.source_for(collectible_type) + @languages = catalog.languages + version = catalog.current_version(languages: @languages) + @run.record_file!(version:, languages: @languages.join(",")) + applied = PriceRefreshRun.applied(@source, version, @run.languages).first + return @run.skip!("#{version} already applied", run_on: applied.run_on) if applied + + catalog.progress = @run.method(:heartbeat) if catalog.respond_to?(:progress=) + path = catalog.download(version, dir: Rails.configuration.x.catalog_download_dir.join(collectible_type)) + reader = @source_class.new + @run_on = reader.reported_on(path) + @run.update!(run_on: @run_on) + latest = PriceRefreshRun.latest_run_on(@source) + return @run.skip!("a later file (#{latest}) was already read; this one is dated #{@run_on}") if latest && @run_on < latest + + apply(reader, path) + @run.finish!(@counts) + end + + # Filled in by the next phases. + def apply(_reader, _path); end + end + ``` + + Why the order holds: `latest_run_on` runs after the new run's `run_on` was written; the run is `running`, so it is not among the statuses counted. + +- [ ] **T4.4** Run `bin/rspec spec/models/prices/refresh_spec.rb` — expect PASS. `bin/rubocop -a app/models/prices/refresh.rb spec/models/prices/refresh_spec.rb`. Commit: `feat(prices): the daily refresh's lifecycle, skips and run guard (spec 089)`. + +--- + +## Phase 5: Each quote is judged and written + +**Implements:** Decisions 2 (counts), 3, 6, FR-1 | **Satisfies:** AC-1.1, AC-1.6, AC-1.7, AC-2.1, AC-2.2, AC-2.3, AC-2.5, AC-2.6, AC-2.7, AC-2.8, AC-2.10, AC-2.11, AC-2.12, AC-4.7 (batches), AC-5.3 +**Files:** `app/models/prices/refresh.rb`, `spec/models/prices/refresh_spec.rb` +**Interfaces:** Consumes: Phase 4's `Prices::Refresh`, `CurrentPrice`, `PriceChange`, `Catalog::Entry`. Produces: `#apply` reads `reader.each_quote(path, languages: @languages)`, batches by distinct printing (`BATCH_SIZE`), looks up entry ids on the primary by `external_key` (outside any transaction), judges against current prices inside one prices transaction, writes `upsert_all`, tallies `@counts` after the commit, calls `@run.progress!(@counts)` after each batch. It records, for Phase 6: `@seen` (a `Set` of `[entry_id, series_id]`), `@held_entries`, `@held_series`, `@untrusted`, `@quotes_yielded`, `@unknown`. + +- [ ] **T5.1** Failing tests: add to `spec/models/prices/refresh_spec.rb` (a `describe "applying a file"`, with `before { use_widget_sources(records:) }`-style setup per example; helper `refresh`, `last_run` from Phase 4): + + ```ruby + describe "judging quotes" do + def rows = CurrentPrice.pluck(:catalog_entry_id, :cents, :reported_on, :changed_on) + def history = PriceChange.order(:catalog_entry_id, :recorded_on).pluck(:catalog_entry_id, :recorded_on, :cents) + + it "adds a price no one has, with a history row for the run's date, and finishes the run (AC-1.1, AC-2.1)", :aggregate_failures do + use_widget_sources(records: [ widget_quote("w-1", 142) ], reported_on: Date.new(2026, 11, 13)) + entry = widget_entry("w-1") + + run = refresh.call + + expect(run).to have_attributes(status: "finished", run_on: Date.new(2026, 11, 13), source_version: "v1", languages: "en", + job_id: "job-1", added: 1, quotes_seen: 1, unchanged: 0, repriced: 0) + expect(run.started_at).to be_present + expect(run.finished_at).to be_present + expect(rows).to eq([ [ entry.id, 142, Date.new(2026, 11, 13), Date.new(2026, 11, 13) ] ]) + expect(history).to eq([ [ entry.id, Date.new(2026, 11, 13), 142 ] ]) + end + + it "writes nothing for an unchanged price, and the current price's dates stay (AC-2.2)", :aggregate_failures do + use_widget_sources(records: [ widget_quote("w-1", 100) ]) + entry = widget_entry("w-1") + seed_price(entry, 100, on: Date.new(2026, 11, 12)) + + run = refresh.call + + expect(run).to have_attributes(unchanged: 1, repriced: 0, added: 0, quotes_seen: 1) + expect(rows).to eq([ [ entry.id, 100, Date.new(2026, 11, 12), Date.new(2026, 11, 12) ] ]) + expect(history.size).to eq(1) + end + + it "updates a changed price and adds one history row, leaving the earlier ones (AC-2.3)", :aggregate_failures do + use_widget_sources(records: [ widget_quote("w-1", 150) ]) + entry = widget_entry("w-1") + seed_price(entry, 100, on: Date.new(2026, 11, 12)) + + run = refresh.call + + expect(run).to have_attributes(repriced: 1, added: 0, unchanged: 0) + expect(rows).to eq([ [ entry.id, 150, Date.new(2026, 11, 13), Date.new(2026, 11, 13) ] ]) + expect(history).to eq([ [ entry.id, Date.new(2026, 11, 12), 100 ], [ entry.id, Date.new(2026, 11, 13), 150 ] ]) + end + + it "judges each series of a printing on its own (AC-2.6)", :aggregate_failures do + series = [ PriceRefreshHelpers::PLAIN, + Prices::Sources::Series.new(finish: "foil", currency: "USD", kind: "market"), + Prices::Sources::Series.new(finish: "etched", currency: "USD", kind: "market") ] + use_widget_sources(series:, records: [ widget_quote("w-1", 100, series: series[0]), widget_quote("w-1", 300, series: series[1]), widget_quote("w-1", 500, series: series[2]) ]) + entry = widget_entry("w-1") + seed_price(entry, 100, series: series[0]) # unchanged + seed_price(entry, 250, series: series[1]) # changed + + run = refresh.call + + expect(run).to have_attributes(quotes_seen: 3, unchanged: 1, repriced: 1, added: 1) + expect(CurrentPrice.count).to eq(3) + end + + it "skips a printing the catalog doesn't have, counting it once however many series it has (AC-2.5)", :aggregate_failures do + series = [ PriceRefreshHelpers::PLAIN, Prices::Sources::Series.new(finish: "foil", currency: "USD", kind: "market") ] + use_widget_sources(series:, records: [ widget_quote("ghost", 1, series: series[0]), widget_quote("ghost", 2, series: series[1]), widget_quote("w-1", 100) ]) + widget_entry("w-1") + + run = refresh.call + + expect(run).to have_attributes(unknown_printings: 1, quotes_seen: 1, added: 1) + expect(CurrentPrice.count).to eq(1) + end + + it "judges the first of two quotes for one printing and series, ignoring the second (AC-2.10)", :aggregate_failures do + use_widget_sources(records: [ widget_quote("w-1", 100), widget_quote("w-2", 5), widget_quote("w-1", 999) ]) + entry = widget_entry("w-1") + widget_entry("w-2") + + run = refresh.call + + expect(run).to have_attributes(quotes_seen: 2, added: 2) + expect(CurrentPrice.find_by(catalog_entry_id: entry.id).cents).to eq(100) + end + + it "replaces a history row of the same day when a second file of that day differs (AC-2.7)", :aggregate_failures do + use_widget_sources(records: [ widget_quote("w-1", 150) ], version: "v2") + entry = widget_entry("w-1") + seed_price(entry, 100, on: Date.new(2026, 11, 13)) + + refresh.call + + expect(history).to eq([ [ entry.id, Date.new(2026, 11, 13), 150 ] ]) + end + + it "gives every written row the file's date, whatever the clock says (AC-1.6)", :aggregate_failures do + use_widget_sources(records: [ widget_quote("w-1", 100) ], reported_on: Date.new(2026, 11, 13)) + widget_entry("w-1") + + travel_to(Time.utc(2026, 12, 25, 12)) { refresh.call } + + expect(CurrentPrice.pluck(:reported_on, :changed_on)).to eq([ [ Date.new(2026, 11, 13), Date.new(2026, 11, 13) ] ]) + expect(PriceChange.pluck(:recorded_on)).to eq([ Date.new(2026, 11, 13) ]) + end + + it "writes the same rows when the job is performed twice on one file (AC-1.7)", :aggregate_failures do + use_widget_sources(records: [ widget_quote("w-1", 100), widget_quote("w-2", 200) ]) + widget_entry("w-1") + widget_entry("w-2") + + refresh.call + first = [ CurrentPrice.pluck(:catalog_entry_id, :cents, :changed_on).sort, PriceChange.pluck(:catalog_entry_id, :cents, :recorded_on).sort ] + second = refresh.call + + expect(second).to have_attributes(status: "skipped", error: "v1 already applied") + expect([ CurrentPrice.pluck(:catalog_entry_id, :cents, :changed_on).sort, PriceChange.pluck(:catalog_entry_id, :cents, :recorded_on).sort ]).to eq(first) + expect(PriceRefreshRun.count).to eq(2) + end + + it "keeps rows keyed by the entry's id, in the prices database (AC-2.8)", :aggregate_failures do + use_widget_sources(records: [ widget_quote("w-1", 100) ]) + entry = widget_entry("w-1") + + refresh.call + + expect(CurrentPrice.first.catalog_entry_id).to eq(entry.id) + expect([ CurrentPrice, PriceChange, PriceRefreshRun ].map { it.connection_db_config.name }.uniq).to eq([ "prices" ]) + end + + it "raises, fails the run and keeps earlier batches when a quote's series was not declared (AC-2.11)", :aggregate_failures do + undeclared = Prices::Sources::Series.new(finish: "gold", currency: "USD", kind: "market") + stub_const("Prices::Refresh::BATCH_SIZE", 1) + use_widget_sources(records: [ widget_quote("w-1", 100), widget_quote("w-2", 5, series: undeclared) ]) + widget_entry("w-1") + widget_entry("w-2") + + expect { refresh.call }.to raise_error(Prices::Sources::ConfigurationError, /gold/) + + expect(last_run).to have_attributes(status: "failed", added: 1) + expect(CurrentPrice.count).to eq(1) + end + + it "logs a malformed record's key and error, and counts it (AC-2.12)", :aggregate_failures do + use_widget_sources(records: [ widget_malformed("w-1", series: PriceRefreshHelpers::PLAIN), widget_quote("w-2", 5) ]) + widget_entry("w-1") + widget_entry("w-2") + allow(Rails.logger).to receive(:warn) + + run = refresh.call + + expect(run.malformed).to eq(1) + expect(Rails.logger).to have_received(:warn).with(/prices\.refresh skipped malformed .*w-1.*unreadable/) + end + + it "touches the run after every batch, even one that writes nothing (AC-4.7)", :aggregate_failures do + stub_const("Prices::Refresh::BATCH_SIZE", 2) + entries = Array.new(5) { |index| widget_entry("w-#{index}") } + entries.each { |entry| seed_price(entry, 100) } + use_widget_sources(records: entries.map { widget_quote(it.external_key, 100) }) # all unchanged: no batch writes a price + progress = [] + callback = ->(*, payload) { progress << payload[:sql] if payload[:sql].to_s.start_with?('UPDATE "price_refresh_runs" SET') && payload[:sql].include?('"quotes_seen"') } + + ActiveSupport::Notifications.subscribed(callback, "sql.active_record") { refresh.call } + + expect(CurrentPrice.pluck(:cents).uniq).to eq([ 100 ]) # nothing was written + expect(progress.size).to eq(3) # a progress write per batch (2, 2 and 1 printings): only they put quotes_seen in an UPDATE's SET + end + end + ``` + +- [ ] **T5.2** Run `bin/rspec spec/models/prices/refresh_spec.rb -e "judging quotes"` — expect FAIL (no rows are written; `#apply` is empty). + +- [ ] **T5.3** Implement. In `app/models/prices/refresh.rb` replace the empty `apply` with the following (inside `private`): + + ```ruby + # Reads the file once, flushing a batch of printings at a time (decision 6), then handles absences (Phase 6). + def apply(reader, path) + @seen = Set.new # [entry id, series id] judged this run: a duplicate quote is ignored, an unseen candidate is absent + @unknown = Set.new + @held_entries = Set.new + @held_series = Set.new + @untrusted = false + @quotes_yielded = 0 + @pending = {} + @pending_malformed = [] + + reader.each_quote(path, languages: @languages) do |record| + case record + when Prices::Sources::Quote then queue_quote(record) + when Prices::Sources::Malformed then queue_malformed(record) + end + end + flush + end + + def queue_quote(quote) + @quotes_yielded += 1 + flush if !@pending.key?(quote.external_key) && @pending.size >= BATCH_SIZE # earlier batches stand when the next raises + @series.fetch(quote.series) do + raise Prices::Sources::ConfigurationError, "#{@source_class} yielded a quote of #{quote.series.inspect}, which it did not declare" + end + (@pending[quote.external_key] ||= []) << quote + end + + def queue_malformed(record) + @counts[:malformed] += 1 + Rails.logger.warn("prices.refresh skipped malformed #{record.external_key.inspect}: #{record.error}") + @pending_malformed << record + end + + # Judges and writes the pending printings. The primary is read here, outside any transaction; the prices database + # is read and written in one short transaction; the counts are tallied after it commits. + def flush + return if @pending.empty? && @pending_malformed.empty? + + keys = (@pending.keys + @pending_malformed.filter_map(&:external_key)).uniq + ids = Catalog::Entry.where(collectible_type:, external_key: keys).pluck(:external_key, :id).to_h + hold_malformed(ids) + judged = [] + @pending.each do |key, quotes| + id = ids[key] + next @unknown << key unless id + + quotes.each do |quote| + series_id = @series.fetch(quote.series).id + judged << [ id, series_id, quote.cents ] if @seen.add?([ id, series_id ]) + end + end + write_judged(judged) + @pending = {} + @pending_malformed = [] + @counts[:unknown_printings] = @unknown.size + @run.progress!(@counts) + end + + # What a Malformed holds back from removal (decision 4, items 3 and 4): no key, the whole run; a key and no series (or + # a series the source didn't declare), every series of the printing; a key and a series, that series. + def hold_malformed(ids) + @pending_malformed.each do |record| + entry_id = ids[record.external_key] + if record.external_key.nil? then @untrusted = true + elsif entry_id.nil? then next + elsif record.series.nil? || @series[record.series].nil? then @held_entries << entry_id + else @held_series << [ entry_id, @series[record.series].id ] + end + end + end + + def write_judged(judged) + return if judged.empty? + + tally = Hash.new(0) + PricesRecord.transaction do + existing = CurrentPrice.where(price_source_id: @source.id, catalog_entry_id: judged.map(&:first).uniq) + .pluck(:catalog_entry_id, :price_series_id, :cents).to_h { |entry_id, series_id, cents| [ [ entry_id, series_id ], cents ] } + currents = [] + changes = [] + judged.each do |entry_id, series_id, cents| + had = existing[[ entry_id, series_id ]] + next tally[:unchanged] += 1 if had == cents + + tally[had ? :repriced : :added] += 1 + keys = { catalog_entry_id: entry_id, price_source_id: @source.id, price_series_id: series_id } + currents << keys.merge(cents:, reported_on: @run_on, changed_on: @run_on) + changes << keys.merge(recorded_on: @run_on, cents:) + end + CurrentPrice.upsert_all(currents, unique_by: CurrentPrice.primary_key) if currents.any? + PriceChange.upsert_all(changes, unique_by: PriceChange.primary_key) if changes.any? + end + tally.each { |name, count| @counts[name] += count } + @counts[:quotes_seen] += judged.size + end + ``` + + If `upsert_all(..., unique_by: Model.primary_key)` is refused for a composite key on this Rails (8.1.4), use `unique_by: %i[catalog_entry_id price_source_id price_series_id]` (and the four columns for `PriceChange`): the `WITHOUT ROWID` primary key is the conflict target. T2.1's model specs prove which form works; use that form here. + +- [ ] **T5.4** Run `bin/rspec spec/models/prices/refresh_spec.rb` — expect PASS. `bin/rubocop -a`. Commit: `feat(prices): judge each quote and record a change once (spec 089)`. + +--- + +## Phase 6: Gone is not bad: absences and their guards + +**Implements:** Decision 4, decision 6 (removal), FR-1 must-nots | **Satisfies:** AC-2.4, AC-2.9, AC-3.1 to AC-3.9, AC-4.7 (removal slices), AC-5.3 (withheld) +**Files:** `app/models/prices/refresh.rb`, `spec/models/prices/refresh_spec.rb` +**Interfaces:** Consumes: Phase 5's `@seen`, `@held_entries`, `@held_series`, `@untrusted`, `@quotes_yielded`, `@languages`, `@run_on`, `@counts`. Produces: after the whole file is read, `#remove_absent` raises `GuardError` when the file yielded no `Quote`; walks the source's current prices twice by printing id in slices of `BATCH_SIZE` (the first walk counts candidates and absences, the second removes, only if the ceiling holds); a removal slice deletes the current prices and upserts blank history rows in one transaction; `@counts[:removed]` and `[:removals_withheld]`. + +- [ ] **T6.1** Failing tests: add to `spec/models/prices/refresh_spec.rb`. Each example that removes a price seeds **at least ten candidate rows** (the ceiling is a share): a helper makes them. + + ```ruby + describe "absences" do + def seed_candidates(count, prefix: "keep") = Array.new(count) { |index| widget_entry("#{prefix}-#{index}").tap { seed_price(it, 100) } } + def keep_quotes(count, prefix: "keep") = Array.new(count) { |index| widget_quote("#{prefix}-#{index}", 100) } + def blank_history(entry) = PriceChange.where(catalog_entry_id: entry.id, cents: nil).pluck(:recorded_on) + + it "removes a price the file leaves out and writes a blank history row for the run's date (AC-2.4)", :aggregate_failures do + kept = seed_candidates(10) + gone = widget_entry("gone").tap { seed_price(it, 100) } + use_widget_sources(records: keep_quotes(10)) + + run = refresh.call + + expect(run).to have_attributes(status: "finished", removed: 1, removals_withheld: 0, unchanged: 10) + expect(CurrentPrice.where(catalog_entry_id: gone.id)).to be_empty + expect(blank_history(gone)).to eq([ Date.new(2026, 11, 13) ]) + expect(CurrentPrice.where(catalog_entry_id: kept.map(&:id)).count).to eq(10) + end + + it "treats a price that comes back as an added quote with its own history row (AC-2.4)", :aggregate_failures do + seed_candidates(10) + gone = widget_entry("gone").tap { seed_price(it, 100) } + use_widget_sources(records: keep_quotes(10), version: "v1") + refresh.call + use_widget_sources(records: keep_quotes(10) + [ widget_quote("gone", 120) ], version: "v2", reported_on: Date.new(2026, 11, 14)) + + run = refresh.call + + expect(run).to have_attributes(added: 1, unchanged: 10) + expect(CurrentPrice.find_by(catalog_entry_id: gone.id).cents).to eq(120) + expect(PriceChange.where(catalog_entry_id: gone.id).order(:recorded_on).pluck(:recorded_on, :cents)).to eq([ [ Date.new(2026, 11, 12), 100 ], [ Date.new(2026, 11, 13), nil ], [ Date.new(2026, 11, 14), 120 ] ]) + end + + it "removes a price of a printing absent from the file altogether, and of a retired entry (AC-2.9)", :aggregate_failures do + seed_candidates(20) + absent = widget_entry("absent").tap { seed_price(it, 100) } + retired = widget_entry("retired", retired_at: 1.day.ago).tap { seed_price(it, 100) } + use_widget_sources(records: keep_quotes(20)) + + run = refresh.call + + expect(run).to have_attributes(removed: 2) + expect(CurrentPrice.where(catalog_entry_id: [ absent.id, retired.id ])).to be_empty + end + + it "leaves a price whose printing is in a language the run did not read, or the primary lacks, and does not count it (AC-3.7)", :aggregate_failures do + seed_candidates(10) + japanese = widget_entry("ja-1", language: "ja").tap { seed_price(it, 100) } + CurrentPrice.insert_all!([ { catalog_entry_id: 9_999_999, price_source_id: price_source_row.id, price_series_id: series_row.id, cents: 5, reported_on: Date.new(2026, 11, 12), changed_on: Date.new(2026, 11, 12) } ]) + use_widget_sources(records: keep_quotes(10)) + + run = refresh.call + + expect(run).to have_attributes(removed: 0, removals_withheld: 0) + expect(CurrentPrice.where(catalog_entry_id: [ japanese.id, 9_999_999 ]).count).to eq(2) + end + + it "does not remove a price whose quote was unreadable, and the printing's other series are judged as usual (AC-3.1)", :aggregate_failures do + other = Prices::Sources::Series.new(finish: "foil", currency: "USD", kind: "market") + seed_candidates(10) + entry = widget_entry("w-x") + seed_price(entry, 100) + seed_price(entry, 100, series: other) + use_widget_sources(series: [ PriceRefreshHelpers::PLAIN, other ], records: keep_quotes(10) + [ widget_malformed("w-x", series: PriceRefreshHelpers::PLAIN) ]) + + run = refresh.call + + expect(run).to have_attributes(malformed: 1, removals_withheld: 1, removed: 1) + expect(CurrentPrice.where(catalog_entry_id: entry.id, price_series_id: series_row.id).count).to eq(1) # kept + expect(CurrentPrice.where(catalog_entry_id: entry.id, price_series_id: series_row(other).id)).to be_empty # absent and trusted + expect(blank_history(entry)).to eq([ Date.new(2026, 11, 13) ]) + end + + it "removes none of a printing's prices when its line could be read only as far as its key (AC-3.2)", :aggregate_failures do + other = Prices::Sources::Series.new(finish: "foil", currency: "USD", kind: "market") + seed_candidates(10) + entry = widget_entry("w-x") + seed_price(entry, 100) + seed_price(entry, 100, series: other) + use_widget_sources(series: [ PriceRefreshHelpers::PLAIN, other ], records: keep_quotes(10) + [ widget_malformed("w-x") ]) + + run = refresh.call + + expect(run).to have_attributes(removed: 0, removals_withheld: 2) + expect(CurrentPrice.where(catalog_entry_id: entry.id).count).to eq(2) + end + + it "removes nothing at all when a line names no printing, applying the quotes around it (AC-3.3)", :aggregate_failures do + seed_candidates(10) + gone = widget_entry("gone").tap { seed_price(it, 100) } + new_entry = widget_entry("fresh") + use_widget_sources(records: keep_quotes(10).first(5) + [ widget_malformed ] + keep_quotes(10).last(5) + [ widget_quote("fresh", 77) ]) + + run = refresh.call + + expect(run).to have_attributes(status: "finished", removed: 0, removals_withheld: 1, added: 1) + expect(CurrentPrice.where(catalog_entry_id: gone.id).count).to eq(1) + expect(CurrentPrice.where(catalog_entry_id: new_entry.id).count).to eq(1) + end + + it "fails the run, with the reader's error, when the file is cut short; earlier batches stand, nothing is removed, and a whole file of the day finishes without a duplicate row (AC-3.4)", :aggregate_failures do # rubocop:disable RSpec/ExampleLength -- two runs of one scenario + stub_const("Prices::Refresh::BATCH_SIZE", 2) + seed_candidates(10) + gone = widget_entry("gone").tap { seed_price(it, 100) } + fresh = Array.new(4) { |index| widget_entry("new-#{index}") } + use_widget_sources(records: keep_quotes(10) + fresh.map { widget_quote(it.external_key, 7) } + [ -> { raise Zlib::GzipFile::Error, "unexpected end of file" } ]) + + expect { refresh.call }.to raise_error(Zlib::GzipFile::Error) + + expect(last_run).to have_attributes(status: "failed", removed: 0) + expect(last_run.error).to include("unexpected end of file") + expect(CurrentPrice.where(catalog_entry_id: gone.id).count).to eq(1) + written = CurrentPrice.where(catalog_entry_id: fresh.map(&:id)).count + expect(written).to be_between(2, 4) + + use_widget_sources(records: keep_quotes(10) + fresh.map { widget_quote(it.external_key, 7) }, version: "v1") + second = refresh.call + + expect(second.status).to eq("finished") + expect(CurrentPrice.where(catalog_entry_id: fresh.map(&:id)).count).to eq(4) + expect(PriceChange.where(catalog_entry_id: fresh.map(&:id)).count).to eq(4) + end + + it "fails the run, 'no valid quotes', and writes nothing when the file yields no quote (AC-3.5)", :aggregate_failures do + seed_candidates(10) + use_widget_sources(records: [ widget_malformed ]) + + expect { refresh.call }.to raise_error(Prices::Refresh::GuardError, /no valid quotes/) + + expect(last_run).to have_attributes(status: "failed", removed: 0) + expect(CurrentPrice.count).to eq(10) + expect(PriceChange.count).to eq(10) + end + + it "finishes a file whose quotes are all for printings the catalog lacks, counting them and writing or removing nothing (AC-3.5)", :aggregate_failures do + widget_entry("w-1") # the catalog has a printing of the type, so the run isn't skipped + use_widget_sources(records: [ widget_quote("ghost-1", 1), widget_quote("ghost-2", 1) ]) + + run = refresh.call + + expect(run).to have_attributes(status: "finished", unknown_printings: 2, quotes_seen: 0, removed: 0) + expect([ CurrentPrice.count, PriceChange.count ]).to eq([ 0, 0 ]) + end + + it "refuses every removal and fails the run when absences exceed the share, counting them as withheld (AC-3.6)", :aggregate_failures do + seed_candidates(10) + gone = Array.new(2) { |index| widget_entry("gone-#{index}").tap { seed_price(it, 100) } } + use_widget_sources(records: keep_quotes(10)) # 2 of 12 candidates absent: over 1/10 + + expect { refresh.call }.to raise_error(Prices::Refresh::GuardError, /2 of 12/) + + expect(last_run).to have_attributes(status: "failed", removed: 0, removals_withheld: 2) + expect(CurrentPrice.where(catalog_entry_id: gone.map(&:id)).count).to eq(2) + end + + it "removes absences exactly at the share (AC-3.6)", :aggregate_failures do + seed_candidates(9) + gone = widget_entry("gone").tap { seed_price(it, 100) } # 1 of 10 candidates: exactly 1/10 + use_widget_sources(records: keep_quotes(9)) + + run = refresh.call + + expect(run).to have_attributes(status: "finished", removed: 1) + expect(CurrentPrice.where(catalog_entry_id: gone.id)).to be_empty + end + + it "reads a failed file again on the next run (AC-3.8)", :aggregate_failures do + seed_candidates(10) + widget_entry("gone").tap { seed_price(it, 100) } + widget_entry("gone-2").tap { seed_price(it, 100) } + use_widget_sources(records: keep_quotes(10)) + expect { refresh.call }.to raise_error(Prices::Refresh::GuardError) + + expect { refresh.call }.to raise_error(Prices::Refresh::GuardError) # read again, not skipped as applied + + expect(PriceRefreshRun.where(status: "failed").count).to eq(2) + expect(PriceRefreshRun.where(status: "skipped")).to be_empty + end + + it "touches the run for each slice of the removal walk (AC-4.7)" do + stub_const("Prices::Refresh::BATCH_SIZE", 2) + seed_candidates(10) + widget_entry("gone").tap { seed_price(it, 100) } + use_widget_sources(records: keep_quotes(10)) + progress = [] + callback = ->(*, payload) { progress << payload[:sql] if payload[:sql].to_s.start_with?('UPDATE "price_refresh_runs" SET') && payload[:sql].include?('"updated_at"') } + + ActiveSupport::Notifications.subscribed(callback, "sql.active_record") { refresh.call } + + expect(progress.size).to be >= 12 # eleven candidates in slices of two: six slices in each of the two walks + end + + it "leaves both or neither of a removal batch's current price and blank history row when the batch fails (AC-3.9)", :aggregate_failures do + seed_candidates(10) + gone = widget_entry("gone").tap { seed_price(it, 100) } + use_widget_sources(records: keep_quotes(10)) + PricesRecord.lease_connection.execute("CREATE TRIGGER no_blank_history BEFORE INSERT ON price_changes WHEN NEW.cents IS NULL BEGIN SELECT RAISE(ABORT, 'injected'); END") + + expect { refresh.call }.to raise_error(ActiveRecord::StatementInvalid, /injected/) + + expect(CurrentPrice.where(catalog_entry_id: gone.id).count).to eq(1) + expect(blank_history(gone)).to be_empty + + PricesRecord.lease_connection.execute("DROP TRIGGER no_blank_history") + run = refresh.call + + expect(run).to have_attributes(status: "finished", removed: 1) + expect(CurrentPrice.where(catalog_entry_id: gone.id)).to be_empty + expect(blank_history(gone)).to eq([ Date.new(2026, 11, 13) ]) + end + end + ``` + +- [ ] **T6.2** Run `bin/rspec spec/models/prices/refresh_spec.rb -e "absences"` — expect FAIL. + +- [ ] **T6.3** Implement. In `#apply`, after the final `flush` (the last statement of Phase 5's `#apply`), add `remove_absent`; add to the `private` section: + + ```ruby + # Absence is trusted only as far as the file is (spec 089 decision 4). The file has been read to the end here (a + # reader's error would have left `#apply` already). Two walks over the source's candidate current prices by printing + # id: the first counts the candidates and the absences, so the ceiling is judged on the whole; the second removes. + def remove_absent + raise GuardError, "the file yielded no valid quotes (#{@counts[:malformed]} malformed), so nothing was written or removed" if @quotes_yielded.zero? + + candidates = absent_clear = absent_held = 0 + each_candidate_slice do |rows| + rows.each do |entry_id, series_id| + candidates += 1 + next if @seen.include?([ entry_id, series_id ]) + + held?(entry_id, series_id) ? absent_held += 1 : absent_clear += 1 + end + end + @counts[:removals_withheld] = absent_held + return if absent_clear.zero? + + if absent_clear > candidates * MAX_REMOVED_SHARE + @counts[:removals_withheld] += absent_clear + raise GuardError, "#{absent_clear} of #{candidates} current prices are missing from the file, more than the #{(MAX_REMOVED_SHARE * 100).to_i} percent " \ + "a day may lose, so nothing was removed. A file that loses that much in a day is usually truncated or damaged; a real, " \ + "lasting drop is refused until Prices::Refresh::MAX_REMOVED_SHARE is raised." + end + + each_candidate_slice { |rows| remove(rows.reject { |entry_id, series_id| @seen.include?([ entry_id, series_id ]) || held?(entry_id, series_id) }) } + end + + def held?(entry_id, series_id) = @untrusted || @held_entries.include?(entry_id) || @held_series.include?([ entry_id, series_id ]) + + # Yields the source's candidate current prices as [entry id, series id], a slice of BATCH_SIZE printings at a time, + # by keyset on the printing id. A candidate's printing is a catalog entry of this type in a language of the run, + # retired or not; one the primary lacks, or in a language not read, isn't asked about and is left out. The primary is + # read outside any transaction, and each slice touches the run as progress. + def each_candidate_slice + last = 0 + loop do + ids = CurrentPrice.where(price_source_id: @source.id).where("catalog_entry_id > ?", last).distinct.order(:catalog_entry_id).limit(BATCH_SIZE).pluck(:catalog_entry_id) + break if ids.empty? + + last = ids.last + languages = Catalog::Entry.where(collectible_type:, id: ids).pluck(:id, :language).to_h + rows = CurrentPrice.where(price_source_id: @source.id, catalog_entry_id: ids).pluck(:catalog_entry_id, :price_series_id) + yield rows.select { |entry_id, _series_id| @languages.include?(languages[entry_id]) } + @run.progress!(@counts) + end + end + + # Deletes the current prices and records their blank history rows in ONE transaction: a price deleted without its row + # would no longer be a candidate, and the removal would be lost for good. `delete_all` is bulk maintenance of + # global rows by key; no callbacks or validations apply. + def remove(rows) + return if rows.empty? + + PricesRecord.transaction do + rows.group_by(&:first).each do |entry_id, pairs| + CurrentPrice.where(price_source_id: @source.id, catalog_entry_id: entry_id, price_series_id: pairs.map(&:last)).delete_all + end + PriceChange.upsert_all(rows.map { |entry_id, series_id| { catalog_entry_id: entry_id, price_source_id: @source.id, price_series_id: series_id, recorded_on: @run_on, cents: nil } }, + unique_by: PriceChange.primary_key) + end + @counts[:removed] += rows.size + end + ``` + + A `GuardError` from `remove_absent` happens inside `#apply`, so `#call`'s rescue marks the run failed with the counts so far (`removals_withheld` included) — `@counts` is the same hash. `@run.progress!(@counts)` runs per slice; `@counts[:removals_withheld]` set before the second walk is persisted by it. + + Phase 4's example `syncs the source and its series first` has no quote in its file, which is now a `GuardError` (`no valid quotes`): in `spec/models/prices/refresh_spec.rb` change its expectation to `raise_error(Prices::Refresh::GuardError, /no valid quotes/)`. + +- [ ] **T6.4** Run `bin/rspec spec/models/prices/refresh_spec.rb` — expect PASS. `bin/rubocop -a`. Commit: `feat(prices): remove a gone price only when the file can be trusted (spec 089)`. + +--- + +## Phase 7: The job and the schedule + +**Implements:** Decisions 7 (lock, retries, errors), 8 (schedule) | **Satisfies:** AC-4.3, AC-4.6, AC-5.1 +**Files:** `app/jobs/prices/refresh_job.rb`, `config/recurring.yml`, `spec/jobs/prices/refresh_job_spec.rb`, `spec/config/recurring_spec.rb` +**Interfaces:** Consumes: `Prices::Refresh.new(key, job_id:).call`, `Catalog::RateLimitRetry`. Produces: `Prices::RefreshJob.perform_later("scryfall")`-shaped jobs on queue `sync`; `Prices::RefreshJob::LOCK_FOR` (6 hours); `refresh_prices` in `config/recurring.yml`. + +- [ ] **T7.1** Failing tests. `spec/jobs/prices/refresh_job_spec.rb`: + + ```ruby + require "rails_helper" + + RSpec.describe Prices::RefreshJob, type: :job do + it "allows one refresh per source key at a time, on the sync queue (AC-4.3)", :aggregate_failures do + expect(described_class.queue_name).to eq("sync") + expect(described_class.concurrency_limit).to eq(1) + expect(described_class.concurrency_on_conflict).to eq(:block) + expect(described_class.concurrency_duration).to eq(6.hours) + expect(described_class.new("fake_source").concurrency_key).to eq("Prices::RefreshJob/fake_source") + end + + it "runs the refresh for the source key, under its own job id" do + refresh = instance_double(Prices::Refresh, call: nil) + allow(Prices::Refresh).to receive(:new).and_return(refresh) + job = described_class.new("fake_source") + + job.perform_now + + expect(Prices::Refresh).to have_received(:new).with("fake_source", job_id: job.job_id) + end + + # decision 7's table (AC-4.6): the job's fate for each error; no error is discarded + { + "Catalog::Sources::TransientError" => [ Catalog::Sources::TransientError, "timeout", :retried ], + "ActiveRecord::StatementTimeout" => [ ActiveRecord::StatementTimeout, "busy", :retried ], + "Catalog::Sources::IntegrityError" => [ Catalog::Sources::IntegrityError, "downloaded 1 bytes", :retried ], + "Catalog::Sources::RateLimitedError" => [ Catalog::Sources::RateLimitedError, "slow down", :retried ], + "Prices::Sources::IntegrityError" => [ Prices::Sources::IntegrityError, "no date", :failed ], + "Catalog::Sources::ConfigurationError" => [ Catalog::Sources::ConfigurationError, "bad language", :failed ], + "Zlib::Error" => [ Zlib::Error, "cut short", :failed ], + "Prices::Refresh::GuardError" => [ Prices::Refresh::GuardError, "over the ceiling", :failed ], + "RuntimeError" => [ RuntimeError, "anything else", :failed ] + }.each do |name, (error, message, fate)| + it "#{fate == :retried ? "retries" : "fails, without a retry or a discard,"} on #{name} (AC-4.6)", :aggregate_failures do + allow(Prices::Refresh).to receive(:new).and_raise(error, message) + + if fate == :retried + expect { described_class.perform_now("fake_source") }.to have_enqueued_job(described_class).with("fake_source") + else + expect { described_class.perform_now("fake_source") }.to raise_error(error, message) + end + end + end + + it "fails on the third attempt of a transient error, not retried again (AC-4.6)" do + allow(Prices::Refresh).to receive(:new).and_raise(Catalog::Sources::TransientError, "timeout") + job = described_class.new("fake_source") + 2.times { job.perform_now } # two retries are queued (retry_on counts per exception class, in the job's exception_executions) + + expect { job.perform_now }.to raise_error(Catalog::Sources::TransientError, "timeout") + end + end + ``` + + Add to `spec/config/recurring_spec.rb`: + + ```ruby + it "refreshes prices daily in production, at another time than the catalog's refresh, and not in development (spec 089 AC-5.1)", :aggregate_failures do + config = YAML.load_file(Rails.root.join("config/recurring.yml"), aliases: true) + task = config.dig("production", "refresh_prices") + + expect(task).to include("class" => "Prices::RefreshJob", "args" => [ "scryfall" ], "queue" => "sync") + expect(Fugit.parse_cronish(task["schedule"]).to_cron_s).to eq("30 11 * * *") + expect(config.dig("development", "refresh_prices")).to be_nil + end + ``` + +- [ ] **T7.2** Run `bin/rspec spec/jobs/prices/refresh_job_spec.rb spec/config/recurring_spec.rb` — expect FAIL. + +- [ ] **T7.3** Implement `app/jobs/prices/refresh_job.rb`: + + ```ruby + # Runs the daily price refresh for a price source, in the background (spec 089). Queued daily by config/recurring.yml, + # or by hand from `bin/rails runner`; there is no button or task until the admin page (#367). + # + # No error is discarded: a failed job shows under Failed on `/admin/jobs`, which is how an operator learns of a guard that + # fired before #367 shows the runs. A departure from `background-jobs.md`'s `discard_on` advice, whose example is a + # `DeserializationError`: the argument is a string, and a discarded job would hide a guard that fired (spec 089 decision 7). + # + # @see file:prices.md + class Prices::RefreshJob < ApplicationJob + # How long the queue holds a second refresh of the source back when the first never reports its end: a waiting refresh + # must never start while another's job may still be working (the catalog refresh's value, spec 089 decision 7). The + # run's own stall rule is `PriceRefreshRun::STALL_AFTER`, in minutes. + LOCK_FOR = 6.hours + + queue_as :sync + + limits_concurrency to: 1, key: ->(source_key, *) { source_key }, duration: LOCK_FOR + + # The download's size error is retried too: a file republished mid-download gives exactly it, and the retry asks the + # index again (spec 089 decision 7). + retry_on Catalog::Sources::TransientError, Catalog::Sources::IntegrityError, ActiveRecord::StatementTimeout, + wait: :polynomially_longer, attempts: 3 + # After retry_on, so that it is tried first: a rate-limited source is a TransientError too, but the job comes back when + # the source asked. It reports no progress: a rate limit ends a refresh before any row is written. + include Catalog::RateLimitRetry + + # @param source_key [String] a key of `Prices.sources` + def perform(source_key) = Prices::Refresh.new(source_key, job_id:).call + end + ``` + + In `config/recurring.yml`, add to the `production:` block after `refresh_mtg_catalog`: + + ```yaml + refresh_prices: + class: Prices::RefreshJob + args: [ "scryfall" ] + queue: sync + schedule: at 11:30am every day + ``` + +- [ ] **T7.4** Run `bin/rspec spec/jobs/prices/refresh_job_spec.rb spec/config/recurring_spec.rb spec/price_source_core_spec.rb` — expect PASS (the core guard now covers `app/jobs/prices/**`; the schedule's `scryfall` is in YAML, outside it). If `retry_on … IntegrityError` and the rate-limit module don't compose as the example for `RateLimitedError` expects, put `Catalog::Sources::RateLimitedError`'s example under `have_enqueued_job` with `retry_job`'s wait (the module reschedules it itself). Commit: `feat(prices): the daily refresh job, its errors' fates and its schedule (spec 089)`. + +--- + +## Phase 8: Proofs: transactions, the network, the real source, the timing + +**Implements:** Decisions 6, 7, NFR | **Satisfies:** AC-4.4, AC-4.5, AC-5.6, AC-1.1 and AC-1.7 end to end +**Files:** `spec/models/prices/refresh_spec.rb`, `spec/models/prices/refresh_end_to_end_spec.rb`, `spec/models/prices/refresh_timing_spec.rb` +**Interfaces:** Consumes: everything above, `stub_scryfall`, `scryfall_card`, `gzip_jsonl`. Produces: evidence only (specs); the timing numbers go in `verification.md`. + +- [ ] **T8.1** AC-4.4, in `spec/models/prices/refresh_spec.rb`: + + ```ruby + describe "transactions (AC-4.4)" do + it "writes in batches of BATCH_SIZE printings inside savepoint spans, the run row in spans of its own, and reads the primary outside them", :aggregate_failures do # rubocop:disable RSpec/ExampleLength -- one subscribed run, one set of assertions + stub_const("Prices::Refresh::BATCH_SIZE", 2) + entries = Array.new(5) { |index| widget_entry("w-#{index}") } + use_widget_sources(records: entries.map { widget_quote(it.external_key, 100) }) + entries.each { |entry| seed_price(entry, 90) } # five changes, so every batch writes both tables + prices = PricesRecord.lease_connection + stack = [] # open spans on the prices connection: each a list of its statements + spans = [] + outside = [] + primary_inside = [] + callback = lambda do |_name, _start, _finish, _id, payload| + sql = payload[:sql].to_s + if payload[:connection].equal?(prices) + if sql.start_with?("SAVEPOINT") then stack.push([]) + elsif sql.start_with?("RELEASE SAVEPOINT", "ROLLBACK TO SAVEPOINT") then spans << stack.pop + elsif stack.any? then stack.last << sql + else outside << sql + end + elsif stack.any? + primary_inside << sql + end + end + + ActiveSupport::Notifications.subscribed(callback, "sql.active_record") { refresh.call } + + write = ->(sql, table) { sql.match?(/\A\s*(INSERT|UPDATE|DELETE)\b.*?\b#{table}\b/m) } + data = ->(sql) { write.call(sql, "current_prices") || write.call(sql, "price_changes") } + batch_spans = spans.select { |statements| statements.any?(&data) } + expect(outside.grep(/\A\s*(INSERT|UPDATE|DELETE)/)).to all(include("price_refresh_runs")) # no price write outside a span + expect(batch_spans.size).to eq(3) # five printings in batches of two: 2, 2 and 1 + expect(batch_spans.flat_map { |statements| statements.select { write.call(it, "price_refresh_runs") } }).to be_empty # the run row is not written inside a batch + expect(batch_spans.map { |statements| statements.find { write.call(it, "current_prices") }.scan(/\),\s*\(/).size + 1 }).to eq([ 2, 2, 1 ]) # tuples of each batch's insert + expect(primary_inside).to be_empty + expect(CurrentPrice.count).to eq(5) + end + end + ``` + +- [ ] **T8.2** AC-4.5 and the real source end to end. `spec/models/prices/refresh_end_to_end_spec.rb`: + + ```ruby + require "rails_helper" + + # The refresh through the real catalog source and the real price source, over a stubbed Scryfall (spec 089 AC-4.5, AC-1.1, + # AC-1.7): the file's quotes become rows, and no request happens inside a transaction. + RSpec.describe "the daily price refresh over the real sources", type: :model do + let(:stamp) { "20261113100623" } + + def cards(prices_by_id) = prices_by_id.map { |id, prices| scryfall_card("id" => id, "prices" => prices) } + + around do |example| + example.run + ensure + FileUtils.rm_rf(Rails.configuration.x.catalog_download_dir) + end + + it "downloads once, applies the file's quotes dated by its name, and makes no request inside a transaction (AC-4.5)", :aggregate_failures do # rubocop:disable RSpec/ExampleLength -- one scenario + one = create(:catalog_entry, collectible_type: "mtg", external_key: "card-1") + two = create(:catalog_entry, collectible_type: "mtg", external_key: "card-2") + stub_scryfall(cards: cards("card-1" => { "usd" => "1.42", "usd_foil" => "3.10", "eur" => nil }, "card-2" => { "usd" => "0.05" }), stamp:) + connections = [ ApplicationRecord.lease_connection, PricesRecord.lease_connection ] + baselines = connections.map(&:open_transactions) + inside = [] + seen = 0 + callback = lambda do |request, _response| + seen += 1 + inside << request.uri.to_s if connections.zip(baselines).any? { |connection, baseline| connection.open_transactions > baseline } + end + WebMock.after_request(&callback) + + run = Prices::Refresh.new("scryfall").call + + expect(seen).to be >= 2 # the bulk-data index and the file + expect(inside).to be_empty + expect(run).to have_attributes(status: "finished", run_on: Date.new(2026, 11, 13), source_version: "default-cards-#{stamp}", added: 3, quotes_seen: 3) + expect(CurrentPrice.where(catalog_entry_id: one.id).order(:price_series_id).pluck(:cents)).to contain_exactly(142, 310) + expect(CurrentPrice.where(catalog_entry_id: two.id).pluck(:cents)).to eq([ 5 ]) + expect(PriceChange.pluck(:recorded_on).uniq).to eq([ Date.new(2026, 11, 13) ]) + ensure + WebMock::CallbackRegistry.callbacks.reject! { |entry| entry[:block].equal?(callback) } # only this example's, never VCR's + end + + it "changes nothing when the same file is read again (AC-1.7)", :aggregate_failures do + create(:catalog_entry, collectible_type: "mtg", external_key: "card-1") + stub_scryfall(cards: cards("card-1" => { "usd" => "1.42" }), stamp:) + Prices::Refresh.new("scryfall").call + before = [ CurrentPrice.pluck(:cents, :changed_on), PriceChange.pluck(:cents, :recorded_on) ] + + run = Prices::Refresh.new("scryfall").call + + expect(run).to have_attributes(status: "skipped", error: "default-cards-#{stamp} already applied") + expect(a_request(:get, /data\.scryfall\.io/)).to have_been_made.once # the file is not downloaded again + expect([ CurrentPrice.pluck(:cents, :changed_on), PriceChange.pluck(:cents, :recorded_on) ]).to eq(before) + end + end + ``` + + The example removes only its own callback: VCR hooks into WebMock with an `after_request` callback of its own, which `WebMock.reset_callbacks` would drop. `PriceSource.sync!` of `"scryfall"` writes the real source row; the example needs no registration. + +- [ ] **T8.3** AC-5.6, `spec/models/prices/refresh_timing_spec.rb` (excluded unless `SCANNER_TIMING` is set; run by hand: `SCANNER_TIMING=1 bin/rspec spec/models/prices/refresh_timing_spec.rb`): + + ```ruby + require "rails_helper" + + # A day's file at realistic size (spec 089 AC-5.6): 110,000 cards, about 300,000 quotes, entries in the test database. + # A measurement, not a gate; the numbers are recorded in verification.md (#366 replaces them with the run's own). + RSpec.describe "the daily price refresh's time", :timing, type: :model do + let(:count) { 110_000 } + let(:stamp) { "20261113100623" } + + def card_lines(shift) + Array.new(count) do |index| + scryfall_card("id" => "card-#{index}", "prices" => { "usd" => format("%.2f", 1 + index / 100.0 + shift.call(index)), "usd_foil" => (index.even? ? format("%.2f", 2 + index / 100.0) : nil), "eur" => format("%.2f", 1 + index / 120.0) }) + end + end + + it "applies a first run and then a typical day", :aggregate_failures do # rubocop:disable RSpec/ExampleLength -- one measured scenario + set = create(:catalog_set, collectible_type: "mtg") + identity = create(:catalog_identity, collectible_type: "mtg") + now = Time.current + Array.new(count) do |index| + { collectible_type: "mtg", external_key: "card-#{index}", catalog_set_id: set.id, catalog_identity_id: identity.id, number: index.to_s, language: "en", + name: "Card #{index}", kind: "card", content_digest: "d", created_at: now, updated_at: now } + end.each_slice(1_000) { |rows| Catalog::Entry.insert_all(rows) } # one statement per 1,000: a single one would pass SQLite's statement size + timings = {} + + stub_scryfall(cards: card_lines(->(_) { 0 }), stamp: stamp) + timings[:first_run] = Benchmark.realtime { Prices::Refresh.new("scryfall").call } + first = PriceRefreshRun.order(:id).last + + next_stamp = "20261114100623" + stub_scryfall(cards: card_lines(->(index) { (index % 6).zero? ? 0.5 : 0 }), stamp: next_stamp) # about 17 percent change + timings[:typical_day] = Benchmark.realtime { Prices::Refresh.new("scryfall").call } + second = PriceRefreshRun.order(:id).last + + # rubocop:disable RSpec/Output -- the measurement is the point of this example + puts "price refresh timing: #{timings.transform_values { it.round(1) }.inspect}, first #{first.attributes.slice(*PriceRefreshRun::COUNTS.map(&:to_s)).inspect}, second #{second.attributes.slice(*PriceRefreshRun::COUNTS.map(&:to_s)).inspect}" + # rubocop:enable RSpec/Output + expect(first.status).to eq("finished") + expect(second).to have_attributes(status: "finished") + expect(second.repriced).to be > 0 + ensure + FileUtils.rm_rf(Rails.configuration.x.catalog_download_dir) + end + end + ``` + + The cards' `prices` carry `usd`, `usd_foil` and `eur`; the file is built in memory by `stub_scryfall` (110,000 lines, a few tens of MB). If building it in memory is too slow or too large, write it under `tmp/` with `gzip_jsonl` and stub the download to return that file. Do not add the example to the default suite (it is excluded without `SCANNER_TIMING`). + +- [ ] **T8.4** (If a command below asks for permission, bring the prompt to the coordinator; never answer it.) Run `bin/rspec spec/models/prices/` — expect PASS; run `SCANNER_TIMING=1 bin/rspec spec/models/prices/refresh_timing_spec.rb` once, **alone**, and keep its printed line for `verification.md`. Commit: `test(prices): prove the refresh's transactions, its requests and its time (spec 089)`. + +--- + +## Phase 9: Guards, documents and the index + +**Implements:** FR-1 to FR-3's documentation, Decision 8, NFR | **Satisfies:** AC-5.4, AC-5.5, AC-5.7 +**Files:** `README.md` (the backup advice only), `docs/roadmap.md`, `CLAUDE.md` (the status paragraph), `docs/specs/README.md`, `docs/specs/089-daily-price-refresh/verification.md`; `docs/architecture/prices.md` and `docs/architecture/catalog-operations.md` by `distill-feature` +**Interfaces:** Consumes: the finished code. Produces: documents that say what the job does, each statement with its source. + +- [ ] **T9.1** Run `bin/ci checks` (RuboCop, YARD with no warning, the comment ratchet, Brakeman, bundler-audit, importmap audit, the migration guard; it needs no CI slot) and `bin/rspec spec/price_source_core_spec.rb spec/prices_core_spec.rb spec/prices_schema_spec.rb spec/backup_guards_spec.rb spec/config/recurring_spec.rb`. Fix what they say (a missing comment, a `{…}` in a comment, a collectible name in a core comment). Commit any fixes: `chore(prices): satisfy the comment ratchet and the core guard (spec 089)`. + +- [ ] **T9.2** `README.md`, "Back up": the advice reads "before every upgrade and daily, never Monday 03:00–05:00". Add the daily window: the price refresh runs daily at 11:30 server time and writes the prices database, which a backup's read transaction holds back (ADR 0029, Cost): "never Monday 03:00–05:00, and not around 11:30 (the daily price refresh)". Keep the table row in `release-and-deploy.md` in step (`Backup schedule advice`) — that area doc is `distill-feature`'s, so only `README.md` here. Source: spec 089 decision 8; ADR 0029. + +- [ ] **T9.3** `docs/roadmap.md`: the prices lane's #365 row becomes "Done: [spec 089](specs/089-daily-price-refresh/spec.md)" in the form #364's row uses. `CLAUDE.md`'s status paragraph: where it says the prices database "whose five tables exist, with a price source contract and Scryfall's prices through it … and no refresh or reader yet", say the daily refresh exists (`Prices::Refresh`, `Prices::RefreshJob`, `PriceRefreshRun`, spec 089, #365), leaving "no reader yet" and #366 to #369 as they are. Do not restate the area doc's rules (`documentation.md`). + +- [ ] **T9.4** `docs/specs/README.md`: the spec's row already exists (status "Approved", plan "present"); leave its status for `distill-feature`. + +- [ ] **T9.5** Create `docs/specs/089-daily-price-refresh/verification.md`: one row per AC with the evidence (the spec file and example, or the command and its output), the timing line of T8.4, the document criteria verified by reading (AC-5.7), and an empty `## Implementation review` heading for the review's verdict. + +- [ ] **T9.6** What `distill-feature` must write into the area docs (it runs after the implementation review; this list is the plan's commitment, spec AC-5.7 and the coordinator's Gate A note): + - **`prices.md`:** the daily refresh's rules (decisions 3 to 7): the run's date, "unchanged writes nothing" and what freshness means (the latest finished `run_on`, with the lag known limit); the four cases and the blank row; the absence rules and the five conditions, the candidate rule and what a withheld removal is; the guards and the errors table; the models and their keys; the run record's columns, with where #366 starts; `MAX_REMOVED_SHARE` in Named constants **as a guess until #366 measures the daily share of disappearing prices, and that a real drop above it is refused until the constant moves**; `BATCH_SIZE`, `STALL_AFTER`, `HEARTBEAT_EVERY`, `LOCK_FOR` rows; the "Only `PriceSource` and `PriceSeries` are models" invariant and the "Two models exist" decision replaced; spec 081's `run_on` supersession; **what an operator sees and does when a guard or a bad file fails the job every day, since there is no prices page until #367**: one Failed job a day under `/admin/jobs` (`Prices::RefreshJob`), whose exception says what happened (`Prices::Refresh::GuardError` with how many of how many prices, `Zlib::Error` for a cut-short file, `Catalog::Sources::ConfigurationError` naming a bad language code); the failed run row in `price_refresh_runs`; a standing ceiling breach is refused until `MAX_REMOVED_SHARE` is raised; Solid Queue's hourly clearing removes only finished jobs, so an admin discards a failed job by hand; and the as-of label lags while runs fail. + - **`catalog-operations.md`:** the shared download (the price refresh asks the catalog source for the same file; a file one job downloaded is reused by the other; an instance with more languages downloads the large file daily), the unique partial file name and `STALE_PARTIAL_AFTER`, the contract's concurrent-call line, the supersession of spec 002 FR-3's "a partial file", and that the price refresh does not use `Catalog::Operation` (a pointer to `prices.md`). + - Both: the spec's sources inline, and the distillation log rows. + +- [ ] **T9.7** Commit the documents (stage and commit separately): `docs: the daily price refresh in the roadmap, the status paragraph and the backup advice (spec 089)`. + +--- + +## Phase 10: Integration Verification + +**Implements:** All FRs | **Satisfies:** All ACs + +- [ ] Run `bin/ci checks` — expect all steps pass (RuboCop, YARD, the comment ratchet, Brakeman, bundler-audit, importmap audit, the migration guard). +- [ ] Run, one at a time on the shared test database: `bin/rspec spec/models/prices spec/models/price_refresh_run_spec.rb spec/models/current_price_spec.rb spec/models/price_change_spec.rb spec/jobs/prices spec/config spec/models/mtg/scryfall spec/jobs/catalog spec/models/catalog spec/prices_schema_spec.rb spec/prices_core_spec.rb spec/price_source_core_spec.rb spec/backup_guards_spec.rb` — expect PASS. +- [ ] Do **not** run a separate full `bin/ci`: the one full run is the push's, with the coordinator's slot, after the implementation review and the wrap. +- [ ] Verify each AC against `verification.md`; commit: `docs(specs): verification of spec 089`. + +--- + +## Quickstart Validation + +On a development database with the catalog loaded and Scryfall reachable (the maintainer's machine, not the suite): + +1. `bin/rails runner 'Prices::RefreshJob.perform_later("scryfall")'` (the dev server runs Solid Queue inside Puma; never `bin/jobs`). +2. Watch `/admin/jobs`: the job runs on `sync`. +3. `bin/rails runner 'p PriceRefreshRun.order(:id).last.attributes'` — a `finished` run with `run_on` the file's UTC date and counts; `p CurrentPrice.count, PriceChange.count`. +4. Queue it again: the new run is `skipped` with ` already applied` and no row changes. diff --git a/docs/specs/089-daily-price-refresh/spec.md b/docs/specs/089-daily-price-refresh/spec.md new file mode 100644 index 00000000..dc493ddc --- /dev/null +++ b/docs/specs/089-daily-price-refresh/spec.md @@ -0,0 +1,312 @@ +# Feature 089: Prices Refresh Once a Day, and Each Change Is Recorded Once + +**Status:** Approved by the coordinator under the maintainer's delegation (2026-11-13), after two Mode A review rounds (Fable 5.2). The maintainer ruled decisions 1 to 4 as recommended on 2026-11-13 (relayed by the coordinator) +**Version:** 0.5.0 +**Created:** 2026-11-13 +**Last Updated:** 2026-11-13 +**Branch:** `089-daily-price-refresh` +**Issue:** [#365](https://github.com/plainprogrammer/Collector/issues/365) +**Builds on:** [`prices.md`](../../architecture/prices.md) (the prices database, the `WITHOUT ROWID` rules, the contract's absence rules), [spec 081](../081-prices-database-in-backup/spec.md), [spec 085](../085-price-source-contract/spec.md), [ADR 0029](../../adr/0029-prices-database-held-by-the-backup.md), [research 083](../083-non-english-printing-prices/research.md) and [ruling 41](../../roadmap.md#2026-11-13-prices-on-non-english-printings-research-083), the [price data design](../../superpowers/specs/2026-11-13-price-data-design.md) ("The daily refresh"), [`catalog-operations.md`](../../architecture/catalog-operations.md) (the refresh lifecycle this one follows in part), `.claude/rules/background-jobs.md` +**Supersedes:** spec 002 FR-3's wording that a download goes through "a partial file" (decision 1); spec 081's table for `price_refresh_runs`, whose `run_on` was `NOT NULL` (decision 2) +**Frozen:** the current rules are in [`prices.md`](../../architecture/prices.md) (whole spec) and [`catalog-operations.md`](../../architecture/catalog-operations.md) (parts: decision 1's shared download); see the [specs index](../README.md) + +--- + +## Changelog + +| Version | Date | Change | +|---------|------|--------| +| 0.5.0 | 2026-11-13 | Implementation review: a current price whose series the source no longer declares is not a candidate for removal, as for a language not read (decision 4, AC-3.7) (MINOR) | +| 0.4.3 | 2026-11-13 | Execution: the count column `changed` is `repriced`, because Active Record reserves the attribute name `changed` and `PriceRefreshRun` can't be built with it; decision 2's table, the identity of AC-5.3 and the history-rows sentence follow; no behaviour changes (PATCH) | +| 0.4.2 | 2026-11-13 | Plan review round 1: AC-4.4 measures savepoint spans, not transaction depth, because the suite's non-joinable transaction makes every write a savepoint (a testing method; no behaviour changes) (PATCH) | +| 0.4.1 | 2026-11-13 | Approved (Gate A). Status line only (PATCH) | +| 0.4.0 | 2026-11-13 | Review round 2 fixes (listed in the 0.3.0 row's second half); not re-reviewed (two rounds is the limit) | +| 0.3.0 | 2026-11-13 | Review round 1 (Fable 5.2): `run_on` becomes nullable and `job_id` is added (a run opens before its file's date is known); stale partial downloads are pruned; a removal batch is atomic; errors and their fates are one table, with no `discard_on` (a guard failure shows under Failed on `/admin/jobs`); the older-file guard counts finished and failed runs (a failed run may have written rows); `quotes_seen` counts known printings only; constants have values; duplicate quotes and an undeclared series are decided; ACs added for the uncovered rules; AC-4.4 and AC-4.5 rebuilt so they can be written; the index row. Review round 2 (Fable 5.2): the source is looked up and synced before a run exists; "at least one quote" is any `Quote` yielded; the ceiling is two-phase and needs ten candidates in examples; counts are tallied after commit; a catch-all row and the no-discard rationale in the errors table; the download-size error is retried; stale partials by mtime; `#download` must tolerate concurrent calls (the catalog contract's comment); `down` of the migration; AC wording (MINOR) | +| 0.2.0 | 2026-11-13 | The maintainer's rulings on decisions 1 to 4 recorded (as recommended); decision 4's ceiling is a named constant documented as a guess until #366 measures it; the supersession of spec 002 FR-3's partial-file wording is named (MINOR) | +| 0.1.0 | 2026-11-13 | First draft | + +--- + +## Problem Statement + +The prices database has its five tables (spec 081) and Scryfall's prices come out of the bulk file as quotes (spec 085), but nothing reads a quote into a row. No price is stored, so no price history is being recorded, and history can't be downloaded again: every day without this job is a gap for good (design, "Constraints"). The job also has to be safe to leave running unattended for years. It writes the history, so a bad file, a double run or a half-parsed price must never delete or duplicate a row there. + +## Goals + +- Current prices refreshed once a day, by a scheduled job on the `sync` queue, separate from the weekly catalog refresh. +- A history row only where a price changed, dated by the bulk file's UTC date, so the same file twice changes nothing. +- A price the file no longer carries is removed from the current prices and recorded as a blank history row, and only when the file can be trusted to say so. +- A printing the catalog doesn't have yet is skipped and counted. +- Short write batches, no network inside a transaction, and never two runs at once. +- The run's identity, times, status and the counts this issue names, in `price_refresh_runs`. + +## Non-Goals + +- **The run's measurements** (#366): seconds downloading and processing, the file's size, lines read, printings matched, the prices database's size, WAL, free pages and row counts, and peak memory. This spec records what the job did to rows; #366 records what it cost. The line is decision 2. (The design's "Prices" and "Writes" groups and its unknown ids moved here; #366 owns the rest.) +- **The admin page, and any start button or rake task** (#367). A run is started by the schedule, or by queuing the job from `bin/rails runner`. +- **History levels** (#368). Every priced printing gets history here; #368 narrows it. +- **Any reader or display of a price** (#369), including ruling 41's twin lookup. +- A source other than Scryfall, a price by condition, and prices that follow a merged printing (#371). +- An operator override that forces a withheld removal (decision 4): nothing asks for it yet. + +## Users and Context + +**Primary users:** the scheduler, and the later issues that read the rows (#366 to #369). +**Secondary users:** an operator, who sees one more scheduled job on `/admin/jobs` and a database that grows. A collector sees nothing. +**Usage context:** a Solid Queue job, once a day, on a volume with the catalog's downloaded file; and from specs with generated files. No request calls it. +**Vocabulary:** a *quote* and a *series* are as in [`prices.md`](../../architecture/prices.md). A *candidate* is a current price the file may speak for: its printing is a catalog entry in a configured language (decision 4). An *absent* candidate has no quote in the file. *Withheld* is a removal the run didn't apply because the file couldn't be trusted to say the price was gone. + +## Decisions + +### 1. A job of its own on the catalog's file (ruled by the maintainer, 2026-11-13) + +**The question (the issue's open point):** does the daily refresh share the catalog refresh's download and its operation lifecycle (`Catalog::Operation`)? **Ruled:** share the file, not the lifecycle. + +- **The file.** The refresh asks the catalog's source for it, as the contract's comment allows (`Catalog.source_for(type)`, a fresh instance per call: `#languages`, `#current_version`, `#download`), so the two refreshes read one file. A file one has downloaded is reused by the other when both see the same version (the source reuses a file of the published size). That is about once a week for an English-only instance, where the weekly catalog refresh finds the day's file already on disk; an instance with more languages reads `all_cards` (408 MB) and downloads it daily whichever job runs first, a cost the price refresh adds (Non-Functional Requirements). No member is added to `Prices::Sources`. The run's `source_version` is the string the catalog keeps as its version (`default-cards-20261113100623`). +- **The lifecycle.** The price refresh has its own job (`Prices::RefreshJob`, queue `sync`), its own run record (`PriceRefreshRun`, in the prices database) and its own small guard (decision 7). It does not use `Catalog::Operation`, `Catalog::RefreshRun`, the admin catalog page or `Catalog::Operation::Queue`: that lifecycle is built around the catalog's records and stages and the page that shows them, the catalog refresh must not learn of prices (design decision 10), and the page for prices is #367's. What it copies is the pattern: a run opened first, a stall rule for a run left behind, `retry_on` for transient errors, and the rate-limit module (`Catalog::RateLimitRetry`, answering no progress, since a rate limit comes before any row is written). +- **The shared directory.** Both jobs download into `storage/catalog//` through a partial file, and the source prunes to the newest `KEEP_DOWNLOADS` files. Two downloads of one file at once would write one partial file, so the partial file's name is unique per download (`..part`) and a second download is wasteful but harmless. A unique name leaves an orphan when a download is killed, where the fixed name was overwritten by the next one, so `prune` also deletes a partial file (`*.part`, so a fixed-name one from an earlier build is caught too) whose mtime is older than `STALE_PARTIAL_AFTER` (1 day: a live download is far younger, since the jobs' 6 hour locks end a job that never reports, so a partial file a day old belongs to a killed download). Both changes are in `MTG::Scryfall::Source` (`#fetch` and `#prune`) and supersede spec 002 FR-3's "a partial file". The contract's comment (`app/models/catalog/sources.rb`) is the binding statement, and `#download` now has two callers that can run together, so it gains one line: `#download` must tolerate concurrent calls for one version. + +**Alternatives:** (a) a fully shared lifecycle (one operation, two stages) couples the catalog refresh to prices and makes a prices failure look like a catalog failure; (b) a separate download (a second request for the bulk-data index and a second 81 to 408 MB file a day) is the simplest and the most wasteful. + +### 2. What a run records at this stage, and where the line is (ruled, 2026-11-13) + +`price_refresh_runs` has the run's identity (source, `run_on`, `source_version`, status, error) and its times (`started_at`, `finished_at`). Two of its facts don't fit this job and change: + +- **`run_on` becomes nullable.** A run opens before its file is known, and some end before it is: `already running`, an empty catalog, a transient error, a rate limit, a bad language setting. The date can't be read from the version before download (the file's name, and the date in it, is the source's: spec 085 decision 4). So `run_on` is nil until the file's date is known; a run skipped as already applied copies the `run_on` of the finished run it matched; every run that reads the date (after the download) records it, whatever its outcome. This supersedes spec 081's table (`run_on` `NOT NULL`). +- **`job_id` is added** (a nullable string, the Active Job id), for the stall rule of decision 7. + +A migration (`db/prices_migrate/`, a new one: released migrations are never edited) makes `run_on` nullable by rebuilding `price_refresh_runs`, a rowid table (the `WITHOUT ROWID` trap is for the two large tables), keeping its status check, its foreign key and its index, adds `job_id` and `languages` (nullable strings) and the counts below (integers, `null: false, default: 0`), and `db/prices_structure.sql` is re-dumped with it: + +| Column | Counts | +|---|---| +| `languages` | the sorted language set read, as the catalog's run keeps it (`"en,ja"`) | +| `quotes_seen` | quotes of printings the catalog has, read from the file: `unchanged + repriced + added`. A duplicate of a quote already read in this file is not counted | +| `unchanged` | quotes whose price equals the current price: nothing written | +| `repriced` | quotes whose price differs from the current price: current price and history row written (the column is not `changed`: Active Record reserves that attribute name, and a model of the table can't be built with it) | +| `added` | quotes of a printing and series with no current price: written the same way | +| `removed` | prices whose absence was applied: current price deleted, blank history row written | +| `removals_withheld` | absent candidates that were not removed for any reason in decision 4 (a `Malformed`, a keyless `Malformed`, or the ceiling) | +| `unknown_printings` | printings in the file the catalog doesn't have, each counted once; their quotes are in no other count | +| `malformed` | `Malformed` records, of any kind | + +**The line.** Everything above counts a decision the job took. Left to #366: time spent downloading and processing, the file's size, lines read, printings matched, the prices database's size, WAL, free pages and row counts, and peak memory. History rows written is `repriced + added + removed`, so it has no column. + +`error` holds the reason a run didn't finish: the error of a failed run (`": "`), or why it was skipped. A skipped run is never "applied". A failed run's counts are those so far: the counts are tallied after a batch commits (a rolled-back batch adds nothing) and persisted with each progress touch, so a run closed as `interrupted` keeps the counts of its last touch. A run skipped as already applied records its `source_version` and `languages`. + +The migration's `down` fills a nil `run_on` with the date of the run's `started_at`, restores `NOT NULL`, and drops the added columns (it never loses a row). The plan adds the columns first and rebuilds the table last, so the stored SQL is canonical. + +### 3. The run's date and what "unchanged" means for `reported_on` (ruled, 2026-11-13) + +The run's date is the UTC date of the bulk file (`#reported_on`, spec 085 decision 4), never the clock's. A quote is *unchanged* when its cents equal the current price's; nothing is written for it, so `current_prices.reported_on` and `changed_on` are both the run date of the last write. The two columns agree for every row this job writes; both are written because spec 081 made both, and a later source that reports per quote may separate them. + +**The consequence to hand to #369:** a price's freshness is not on its row. A price is current as of the source's latest *finished* run's `run_on`, which is what a page labels "as of". Ruled: accept this; the alternative (rewriting `reported_on` on every unchanged price) writes the whole table daily, which the design's anti-goal and the issue's "nothing is written" rule exclude. Prices the run leaves alone (a language not configured, a printing the primary lacks) are not touched and go stale with no marker on their row. **Known limit:** while the ceiling of decision 4 fails every run (a day's worth of one currency vanishing, say), the other prices' changes are still written daily and dated, but no run finishes, so the "as of" label lags until one does. #367 shows the failed runs. + +**Older files.** A file whose date is earlier than the latest `run_on` of the source's finished or failed runs (a failed run may have written rows) is skipped after the download, with a reason that names the later date: a history row dated before a later one would reorder history that readers take in date order. A file of the same date and another version (a second publication that day, or a changed language set) is applied: its unchanged prices write nothing, and its changes write one history row per series and date, replacing a row of that date (decision 6). A same-date file that returns a price to its value of the day before leaves a history row equal to the row before it; accepted. + +### 4. "Gone" is not "the file is bad" (ruled, 2026-11-13) + +A source yields no quote for a price the file leaves empty (spec 085 decision 2), and it can't say whether a printing with no quote was in the file at all. So "gone" is told from absence, and absence is trusted only as far as the file is. + +**A current price is a candidate for removal** when its printing is a catalog entry of the collectible type, in a language of the run's `languages`, whether or not the entry is retired, and its series is one the source declares today (`.series`). A price whose printing the primary doesn't have (a restore's lag, spec 081), whose language isn't configured (an operator dropped `ja`) or whose series the source no longer declares (a later build dropped one) is left alone: the file wasn't asked about it, and it is not counted as withheld. A printing missing from the file altogether, and a retired entry the file no longer lists, are absent like any other. (A printing whose language changed to one not configured, and a card whose `lang` is missing or garbled and so is filtered out by the source without a `Malformed`, are removed as absent: rare, the source's contract behaviour, and bounded by the ceiling.) A candidate with no quote in this file is *absent*. + +**An absent candidate is removed** (the current price is deleted and a history row with no cents is written for the run's date) only when all of these hold; otherwise it is withheld and counted in `removals_withheld`: + +1. **The file was read to the end.** A corrupt or cut-short gzip raises, the run fails, and no removal is made. Changes and additions already written stand: each was a valid quote, and each batch commits whole. +2. **The source yielded at least one `Quote`**, for a known printing or not (this tally is not `quotes_seen`, which counts known printings only: a catalog that lags the file, a language just added, must not fail a sound file). An empty file, or one with nothing readable, fails the run (`no valid quotes`), as the catalog's refresh fails one with no valid records. Nothing is written. +3. **No `Malformed` names no printing** (a line that isn't JSON or has no `id`): such a line could have been any card, so the run's absences can't be trusted, and every removal is withheld (spec 085 decision 2). +4. **No `Malformed` names this printing** (a card whose `prices` isn't an object, or whose price in this series couldn't be read): that series, or every series of the printing, is withheld. An unreadable price is not an absent one (spec 085 decision 2; the contract's comment). +5. **The removals are a small share of the candidates.** The candidates are the rows of `current_prices` of the source that meet the rule above (printing and series, not printings). The run is two-phase here: the walk over the candidates counts all of them and their absences before the first removal batch, so the ceiling is judged on the whole. If the absent candidates left after items 3 and 4 exceed `MAX_REMOVED_SHARE` of all candidates (compared exactly: `absent > candidates * MAX_REMOVED_SHARE` with a `Rational`), none is applied and the run fails, `error` saying how many of how many, and every absent candidate is counted in `removals_withheld`. A file that lost a tenth of the market's prices in one day is a truncated or damaged publication far more often than a market event. + +**What fails the run and what finishes it.** Items 1, 2 and 5 fail it, and the job's failure shows under Failed on `/admin/jobs` (decision 7); 3 and 4 finish it with `removals_withheld` counted, because the next day's file will remove the price if it is really gone, and the quotes that were readable are applied. A failed run is not "applied" (decision 5), so the same file is read again by the next run. + +**Ruled (maintainer, 2026-11-13):** 10 percent, and a breach fails the run. `MAX_REMOVED_SHARE` is `Rational(1, 10)`, a named constant whose comment and whose row in `prices.md`'s Named constants table say it is a guess until #366 measures the daily share of disappearing prices, and that a real drop above it is refused until the constant moves. + +**Limit.** A real, permanent drop above the ceiling is refused every day until the constant moves. Nothing forces it now (Non-Goals). The failed run says so in plain words. Because the ceiling is a share, an example that removes a price needs at least ten candidate rows (or sets the constant with `stub_const`); the examples of AC-2.4, AC-2.9 and AC-3.9 say which. + +### 5. A file is applied once + +A run is *applied* when it finished. The job decides in this order: + +0. **The source is looked up and synced** (`Prices.source_class(key)`, then `PriceSource.sync!`), before any run exists: a run's `price_source_id` is `NOT NULL` with a foreign key, so there can be no run row without the source's row. An unregistered key, or a class that fails verification, fails the job with no run (the one failure with no run record; it is a defect of the build, not of a day). Then `PriceRefreshRun.start!` opens the run (decision 7: `already running`, or a run left behind closed). +1. **The catalog has no printing of the type:** the run is skipped with a reason (`run_on` nil), before anything is asked of the catalog's source: there is nothing to attach a price to. +2. **Before downloading**, the job asks the catalog's source for the current version and languages. It records the run `skipped` with `" already applied"` (`run_on` copied from the matched run, `source_version` and `languages` recorded) when a finished run of the same source has the same `source_version` and the same `languages`. The language set matters: `en,de` and `en,ja` both read `all_cards`, and the second adds printings the first never stored. +3. **After the download**, the file's date is read. A file older than decision 3's rule is skipped with its reason. + +A skipped run changes nothing but its own row. A file the job did not skip is read once, in one run. + +### 6. Writes: the four cases + +Quotes are read in file order. The first quote of a printing and series in a file is the one judged; a later duplicate is ignored and not counted. The job collects the printings of a batch (`BATCH_SIZE` printings), finds their catalog ids with one query on the primary (by `external_key`, per batch; no join across files), then reads their current prices from the prices database and writes the batch in one transaction on the prices database. A quote whose series the source did not declare (and so has no `price_series` row) raises `Prices::Sources::ConfigurationError` before the batch is written: that is a defect of the source, and earlier batches stand. + +| The quote | Current price | Written | +|---|---|---| +| the printing is unknown to the catalog | none | nothing; the printing is counted once in `unknown_printings` | +| equals the current price | one | nothing | +| differs from the current price | one | the current price updated (`cents`, `reported_on` and `changed_on` the run date) and a history row for the run date with the new cents | +| the printing and series have none | none | the current price inserted and a history row, the same | +| no quote, a trusted absence (decision 4) | one | the current price deleted and a history row for the run date with no cents | + +A history row is written with an upsert on its primary key: if a row for the same printing, series and date exists (a second file of that date), the later cents replace it, so the key is never violated and a rerun is safe. A removal batch deletes the current prices and upserts their blank history rows **in one transaction**: a failure leaves both or neither, because a price deleted without its row would no longer be a candidate and the removal would be lost for good. Removals run after the whole file was read, in their own batches of `BATCH_SIZE` current prices, from a keyset walk over the source's current prices by printing id (reading the printings' language from the primary in slices, outside the transaction). + +A price that was removed and returns is an `added` quote. `Malformed` records are counted and logged (`prices.refresh skipped malformed …`, the key and the error), as the catalog does. + +### 7. One run at a time, a run left behind, batches, errors and the network + +- **Overlap.** `Prices::RefreshJob` holds a queue concurrency lock per source key (`LOCK_FOR`, 6 hours, as the catalog's: a waiting refresh must never start while another's job may still be working), so a second job waits its turn. `PriceRefreshRun.start!` records a new run `skipped` (`already running`) while another run of the source is running and not left behind. +- **A run left behind.** A running run whose last progress (`updated_at`) is older than `STALL_AFTER` (15 minutes, the catalog's), or whose `job_id` is the starting job's own (the queue re-ran it after a restart), is closed `failed` with `interrupted` when the next run starts. Progress is a touch of the run row at every batch read (even one that writes nothing, as a typical day is mostly unchanged), at every slice of the removal walk, and, while downloading, at most every `HEARTBEAT_EVERY` (30 seconds) through the catalog source's optional `#progress=` hook, so a long download or walk never reads as stalled. This is the catalog refresh's rule without its queue reading. A waiting rate-limit retry holds no lock and its run is already failed; a job that starts meanwhile finds nothing running, and finds the file applied or not as it stands. That is accepted: the worst outcome is one extra request to the bulk-data index. +- **Batches.** `BATCH_SIZE` is 1,000 printings (a few thousand rows). A batch's transaction holds only writes to the prices database and reads of it; the primary's lookup, the file read and the download happen outside any transaction. The run row's progress write is outside the batch transaction. +- **Errors and their fates.** One table, one meaning each. No error is discarded: a failed job shows under Failed on `/admin/jobs`, where an admin can retry it, which is how an operator learns of a guard that fired before #367 shows the runs. + + | Error | Run | Job | + |---|---|---| + | `Catalog::Sources::TransientError`, `ActiveRecord::StatementTimeout` (which is what the SQLite adapter makes of `SQLite3::BusyException`) | `failed` | retried, `polynomially_longer`, 3 attempts, then failed | + | `Catalog::Sources::RateLimitedError` | `failed` | rescheduled by `Catalog::RateLimitRetry` after the wait asked for, then by its rule failed | + | `Catalog::Sources::IntegrityError` (the download's size: a file republished mid-download gives exactly this, and the retry asks the index again) | `failed` | retried like the transient errors, 3 attempts, then failed | + | `Prices::Sources::IntegrityError` (a name with no date), `Catalog::Sources::ConfigurationError` (a bad language setting), `Prices::Sources::ConfigurationError` from an undeclared series (decision 6) | `failed` | failed, not retried | + | `Prices::Sources::ConfigurationError` from `PriceSource.sync!`, or an unregistered key (decision 5, step 0) | none: no run can exist | failed, not retried | + | `Zlib::Error` (a corrupt or cut-short file) | `failed`, no removal | failed, not retried | + | a guard of decision 4 (items 2 and 5): `Prices::Refresh::GuardError` | `failed`, nothing removed | failed, not retried | + | any other `StandardError` (`Catalog::Sources::Error` in general, a `KeyError` from the bulk index) | `failed` | failed, not retried | + + A persistent cause (a bad language code, a standing ceiling breach) leaves one Failed job a day: Solid Queue's hourly clearing removes only finished jobs, and discarding a failed job is the admin's, on `/admin/jobs`. That is a deliberate departure from `background-jobs.md`'s `discard_on` advice, whose example is a `DeserializationError`: the job's argument is a string, and a discarded job would hide a guard that fired before #367 shows the runs. + +### 8. The job, the schedule and the files + +`Prices::RefreshJob` (`app/jobs/prices/refresh_job.rb`, queue `sync`, argument: the source's key) calls `Prices::Refresh` (`app/models/prices/refresh.rb`), which owns the decisions above. The models `CurrentPrice`, `PriceChange` and `PriceRefreshRun` are added (spec 085 left them out) on composite primary keys; writes are bulk (`insert_all`, `upsert_all`, `delete_all` by key), which skips validations on purpose, as the rows come from a validated `Quote` (a header comment says so, as `Location.upsert_all` does). The job's schedule is one line in `config/recurring.yml`, production only like the catalog's: `refresh_prices`, `args: ["scryfall"]`, `queue: sync`, `at 11:30am every day` (the file seen on 2026-11-13 was published at 10:06 UTC, research 083; the host's time zone is the container's, UTC unless the operator sets one, and the run's date comes from the file whenever it runs). The backup advice ("never Monday 03:00 to 05:00") gains the daily window (ADR 0029, Cost). + +Core files name no collectible, comments included: `spec/price_source_core_spec.rb` lists the new ones (its glob already covers `app/models/prices/**`; the three models and the job are added by name). The schedule line is the one place the source's key is written outside its own class and the initializer. + +## User Stories + +### Story 1: A file is applied once, and the date is the file's + +**As** an operator **I want** the daily job to apply each day's file once **So that** the history doesn't grow when nothing changed + +- [ ] **AC-1.1** Given a bulk file for 2026-11-13 and no earlier run When the job runs Then a run is `finished` with `run_on` 2026-11-13, the file's `source_version`, the language set and its `job_id`, and `started_at` and `finished_at` set. +- [ ] **AC-1.2** Given AC-1.1 has finished When the job runs again on the same file and languages Then the new run is `skipped` with ` already applied` and the earlier run's `run_on`, no row of `current_prices` or `price_changes` changes, and the file is not downloaded again. +- [ ] **AC-1.3** Given a finished run of `all_cards` with `en,ja` When the job runs with `en,de` Then the file is applied, because the language set differs. +- [ ] **AC-1.4** Given a finished run dated 2026-11-13 and a file dated 2026-11-12 When the job runs Then the run is `skipped` after the download with a reason naming the later date, and nothing else changes; Given a second file dated 2026-11-13 with another version Then it is applied; and Given a failed run that read a file dated 2026-11-13 and then a file dated 2026-11-12 Then the older one is skipped. +- [ ] **AC-1.5** Given no catalog entry of the type When the job runs Then the run is `skipped` with a reason and a nil `run_on`, the catalog's source is asked nothing, and no row of either price table is written. +- [ ] **AC-1.6** Given the clock is a different UTC day from the file's When the job runs Then every written row carries the file's date. +- [ ] **AC-1.7** Given the job is performed twice on one file When both have finished Then the rows of both price tables are the same as after the first (idempotency), and a run is recorded for each. +- [ ] **AC-1.8** Given each run When it starts Then `PriceSource.sync!` has been called for the source (its rows exist and are current) before the run row is opened and before any quote is written; and Given an unregistered key, or a source that fails verification When the job runs Then it fails with the error, and no run row is written. +- [ ] **AC-1.9** Given a run that ended before its file's date was known (already running, a transient error) When it is read Then its `run_on` is nil and the table accepts it. + +### Story 2: Each quote is judged, and a change is recorded once + +**As** the history **I want** a row only where a price changed **So that** it stays small + +- [ ] **AC-2.1** Given a quote whose printing and series have no current price When it is applied Then a current price exists with those cents and the run date as `reported_on` and `changed_on`, one history row for the run date has those cents, and `added` is 1. +- [ ] **AC-2.2** Given a quote equal to the current price When it is applied Then no row of either table changes (the current price's dates stay), and `unchanged` is 1. +- [ ] **AC-2.3** Given a quote that differs When it is applied Then the current price has the new cents and the run date, one history row for the run date has the new cents, the earlier history rows are untouched, and `changed` is 1. +- [ ] **AC-2.4** Given a current price whose printing the file carries with no quote for that series, and a trusted file When it is applied Then the current price is gone, one history row for the run date has no cents, and `removed` is 1 (among at least ten candidate rows, decision 4's limit); and Given the price comes back in a later file Then it is an added quote with its own history row. +- [ ] **AC-2.5** Given a quote of a printing the catalog doesn't have When it is applied Then no row is written for it, `unknown_printings` counts the printing once however many series it has, and its quotes are in no other count. +- [ ] **AC-2.6** Given one printing with a quote in each of three series, one new, one equal and one changed When applied Then each is judged on its own series, and `quotes_seen` is 3 with `added`, `unchanged` and `changed` 1 each. +- [ ] **AC-2.7** Given a history row already exists for a printing, series and date, and a later file of that date differs When it is applied Then the row's cents are replaced and no second row exists. +- [ ] **AC-2.8** Given rows written by a run When they are read Then each is keyed by the catalog entry's id, and the example passes in the suite's two-database setup, where a query that joins or attaches the two files would fail. +- [ ] **AC-2.9** Given a current price whose printing is absent from the file altogether (not one card of it), and one whose entry is retired and unlisted When the run is trusted (among at least ten candidate rows) Then both are removed like AC-2.4's. +- [ ] **AC-2.10** Given a file with two quotes for one printing and series When it is applied Then the first is judged, the second is ignored, and `quotes_seen` counts one. +- [ ] **AC-2.11** Given a quote whose series the source didn't declare When it is reached Then `Prices::Sources::ConfigurationError` is raised before that batch is written, the run is `failed`, and earlier batches stand. +- [ ] **AC-2.12** Given a `Malformed` record When the file is read Then a log line names its key and error (`prices.refresh skipped malformed …`), and `malformed` counts it. + +### Story 3: A bad file changes nothing it shouldn't + +**As** an operator **I want** a truncated or unreadable file to leave history alone **So that** a bad day isn't recorded as a market crash + +- [ ] **AC-3.1** Given a `Malformed` that names a series for a printing with a current price in that series, and no quote for it When the run finishes Then that price is not removed and no history row is written for it, the other series of the printing are judged as usual, and `malformed` and `removals_withheld` count it. +- [ ] **AC-3.2** Given a `Malformed` with a printing's key and no series When the run finishes Then none of that printing's prices is removed. +- [ ] **AC-3.3** Given a `Malformed` with no key When the run finishes Then no price is removed at all, quotes before and after it are applied, and `removals_withheld` counts the absent candidates. +- [ ] **AC-3.4** Given a gzip file cut short after some cards When the job runs Then the run is `failed` with the reader's error in `error`, no price is removed, the prices written before the cut stand, and the next run of a whole file of that date applies the rest and writes no row twice. +- [ ] **AC-3.5** Given an empty file, or one in which nothing is readable When the job runs Then the run is `failed` with `no valid quotes`, and no row of either price table is written; and Given a file whose quotes are all for printings the catalog doesn't have Then the run finishes with `quotes_seen` 0, `unknown_printings` counted and nothing removed or written. +- [ ] **AC-3.6** Given absent candidates above `MAX_REMOVED_SHARE` of all candidate rows When the run reaches its removals Then none is applied, the run is `failed` and `error` names how many of how many, `removals_withheld` counts them all, and the quotes' changes already written stand; and Given absent candidates exactly at the share Then they are removed (the comparison is exact, with a `Rational`). +- [ ] **AC-3.7** Given a current price whose printing is in a language the run didn't read, or that the primary doesn't have, or whose series the source doesn't declare When the file lacks it Then it is not removed and is not counted as withheld. +- [ ] **AC-3.8** Given a failed run When the job runs again on the same file Then the file is read again (a failed run is not applied). +- [ ] **AC-3.9** Given a failure injected in a removal batch When it happens Then that batch's current prices and blank history rows are both still there or both gone, never one, and a later run removes what is left (among at least ten candidate rows). + +### Story 4: One run at a time, in short batches + +**As** the instance **I want** runs that never overlap and never hold the database long **So that** the app stays responsive and the history isn't written twice + +- [ ] **AC-4.1** Given a run that is running with recent progress When another job starts Then its run is `skipped` with `already running`, and nothing is downloaded or written. +- [ ] **AC-4.2** Given a running run whose last progress is older than `STALL_AFTER`, or whose `job_id` is the starting job's When a job starts Then that run is closed `failed` with `interrupted` and the new run proceeds. +- [ ] **AC-4.3** Given `Prices::RefreshJob` When its concurrency limit is read Then it is one job per source key for `LOCK_FOR`, on queue `sync`. +- [ ] **AC-4.4** Given a file of five printings and `BATCH_SIZE` stubbed to 2 When it is applied, with a subscriber on `sql.active_record` that cuts the prices connection's statements into spans from `SAVEPOINT` to its release or rollback (the suite's own transaction is non-joinable, so every transaction, a batch's or a single run-row write, is a savepoint, and a depth count can't tell them apart) Then every write to `current_prices` and `price_changes` is inside a span; no span holds both such a write and a write to `price_refresh_runs` (the run row's progress is written outside the batch transactions); three spans hold data writes, each of at most 2 printings; and no statement on the primary's connection arrives while a prices span is open. +- [ ] **AC-4.5** Given the run at the catalog source's level (`MTG::Scryfall::Source` with stubbed requests, since the fake catalog source makes no HTTP call and would make the check vacuous) When it downloads and applies a file, with a `WebMock` `after_request` hook that records any request made while either connection's open transactions exceed the baseline Then the recorded list is empty and at least one request was seen. +- [ ] **AC-4.6** Given each error of decision 7's table When the job meets it Then the run ends `failed` with the error (none for the errors of step 0), and the job is retried, rescheduled or failed as the table says; and no error is discarded. +- [ ] **AC-4.7** Given a download in progress When the source reports progress Then the run row's `updated_at` is touched at most once per `HEARTBEAT_EVERY`, a batch read that writes nothing also touches it, and so does each slice of the removal walk. +- [ ] **AC-4.8** Given a partial download file older than `STALE_PARTIAL_AFTER` in the download directory and a fresh one When a download finishes Then the old one is deleted and the fresh one is kept; and Given a second `download` of the same version started from inside the first's stubbed stream (no threads: the example is deterministic) Then each writes its own partial file and the final file is the published size. + +### Story 5: A schedule, run records and documents + +**As** a maintainer **I want** the job scheduled and the records and documents current **So that** the next issues build on them + +- [ ] **AC-5.1** Given `config/recurring.yml` When the production block is read Then `refresh_prices` runs `Prices::RefreshJob` with `scryfall` on queue `sync`, at a time of day other than the catalog refresh's, and the development block has none. +- [ ] **AC-5.2** Given the migration When the prices database is built from the migrations and from `db/prices_structure.sql` Then both have the columns of decision 2 and a nullable `run_on`, the status check, foreign key and index of `price_refresh_runs` survive, and `current_prices` and `price_changes` are still `WITHOUT ROWID` (`spec/prices_schema_spec.rb`); `spec/backup_guards_spec.rb` and `ruby script/upgrade/guard.rb` pass; and `bin/upgrade-smoke` passes in CI's image job (it is not a `bin/ci` step). +- [ ] **AC-5.3** Given a finished run When it is read Then its counts are those of the file, with `quotes_seen = unchanged + repriced + added`. +- [ ] **AC-5.4** Given the new core files When the guard reads them, comments included Then none names a collectible, source or marketplace. +- [ ] **AC-5.5** Given `bin/ci checks` When it runs Then new public code has comments, YARD builds, and RuboCop and Brakeman pass. +- [ ] **AC-5.6** Given a generated file of 110,000 cards with about 300,000 quotes, built under `tmp/` by the example, and the test databases When the job applies it as a first run, and again as a typical day (10 to 20 percent of prices changed) Then the example records both durations (`:timing`, no threshold; it is excluded unless `SCANNER_TIMING` is set, so it is run by hand and the numbers are recorded in `verification.md`). +- [ ] **AC-5.7** Given `prices.md`, `catalog-operations.md`, `README.md`'s backup advice, `docs/roadmap.md`, `docs/specs/README.md` and `CLAUDE.md` When read Then they state what the job does, with the source of each statement (verified by reading, recorded in `verification.md`). + +## Functional Requirements + +### FR-1: The refresh + +**Must:** apply decisions 3 to 6 and 8; write the history only through the cases of decision 6; record the run (decision 2); verify and sync the source (`PriceSource.sync!`) at the start of every run. +**Must not:** write a price whose quote was a `Malformed`, remove one the file can't be trusted about, call the network inside a transaction, read the primary inside a prices transaction, join the two files, name a collectible in a core file, or touch the catalog's tables. + +### FR-2: The job and its guard + +**Must:** decisions 1, 5 and 7: one job per source at a time, a run record for every attempt, the stall rule, the errors' table. +Also: the catalog contract's comment (`app/models/catalog/sources.rb`) gains the line that `#download` must tolerate concurrent calls for one version (decision 1). +**Must not:** use `Catalog::Operation` or `Catalog::RefreshRun`, or change what the catalog refresh does (apart from the partial file's name and its pruning, decision 1). + +### FR-3: The schema change + +**Must:** add the changes of decision 2 with a migration of its own (never edit a released one), update `db/prices_structure.sql`, and keep both large tables `WITHOUT ROWID`. + +## Non-Functional Requirements + +### Performance + +- The first run writes about 300,000 current prices and as many history rows (English only) in batches of 1,000 printings; a typical day writes the changes only. AC-5.6 records the durations: they are a measurement, not a gate, and #366 replaces them with the run's own numbers. +- Memory holds the set of series judged this run (for the absences) and the unknown printings' keys, not the file: one batch of quotes at a time otherwise. Peak memory has no portable cheap reading and is #366's. +- **A cost the price refresh adds to the catalog's:** an instance with languages beyond English downloads `all_cards` (408 MB) every day, where the catalog refresh downloaded it once a week. + +### Security + +- Prices are global data: no account, no `Current.account`, nothing in the export. The job reads a file its source downloaded and a key it was queued with; nothing user-supplied reaches a query (bind parameters throughout). + +### Reliability + +- The job is at-least-once safe: every batch commits whole and is idempotent on its keys, and removals are computed from the file read to the end. +- A killed run leaves its row `running`, which the next run closes (decision 7). + +## Error Scenarios + +| Scenario | Expected Behavior | +|----------|-------------------| +| The bulk-data index lists no such file, or Scryfall times out | `TransientError`: the run fails, the job retries a few times, seconds apart (decision 7's table) | +| Scryfall answers 429 | The rate-limit module reschedules the job after the wait it asked for; the run is failed, the retry records its own | +| The download is the wrong size | The catalog source's `IntegrityError`; the run fails; the job fails and shows under Failed; the next day's run tries again | +| The file's name carries no date | `Prices::Sources::IntegrityError`; the run fails, nothing is written, the job fails | +| `COLLECTOR_MTG_LANGUAGES` has an unsupported code | `Catalog::Sources::ConfigurationError` from the catalog source; the run fails each day, the job fails and shows under Failed, naming the code | +| The gzip is cut short | The run fails; no removal; what was written stands (AC-3.4) | +| A card line is unreadable | Counted and logged; the readable lines apply; every removal is withheld | +| The ceiling is breached | The run fails with how many of how many; nothing is removed; the job fails and shows under Failed | +| The database is busy | The batch rolls back; the job retries; the next attempt finds the earlier batches written and judges them unchanged | +| The process is killed mid-run | The row stays `running` until the next run closes it as `interrupted` | +| Two jobs start together | The queue holds one back; if both start, the second is `skipped` | +| An operator removes a language | Prices of that language's printings are left alone, not removed | + +## Open Questions + +None are open. The maintainer ruled decisions 1 to 4 on 2026-11-13: (1) the file is shared and the lifecycle isn't, with a unique partial-download name; (2) the run's columns and the line with #366; (3) a price's freshness is the source's latest finished `run_on`, not a column, and earlier-dated files are skipped; (4) the removal conditions, a ceiling of 10 percent, and a failed run on breach. Decisions 5 to 8 and the changes review round 1 made to 1 to 4 (a nullable `run_on`, `job_id`, pruning, the older-file rule over failed runs, the errors' table) follow from the rulings and are Gate A's to confirm. The schedule's 11:30 is a guess from one day's publication time and is a one-line change. + +**Supersession (decision 1):** this spec supersedes spec 002 FR-3's wording that a download goes through "a partial file": the partial file's name is unique per download, and partial files left behind are pruned. It also supersedes spec 081's column table for `price_refresh_runs` (`run_on` was `NOT NULL`, no `job_id`). It is recorded here, in `docs/specs/README.md` (this spec's row and the 002 and 081 rows' superseded-by cells), and in [`catalog-operations.md`](../../architecture/catalog-operations.md) and [`prices.md`](../../architecture/prices.md). + +## Out of Scope (Future Considerations) + +- The measurements (#366), the admin log with a view of the queue (#367), the history levels (#368), the display (#369), the cost documentation (#370). +- A manual start (button or rake task) and an operator override of a withheld removal. +- Following a merged printing (#371). diff --git a/docs/specs/089-daily-price-refresh/verification.md b/docs/specs/089-daily-price-refresh/verification.md new file mode 100644 index 00000000..d63160de --- /dev/null +++ b/docs/specs/089-daily-price-refresh/verification.md @@ -0,0 +1,115 @@ +# Verification: spec 089, prices refresh once a day, and each change is recorded once (#365) + +**Spec:** [`spec.md`](spec.md) (0.4.3) · **Plan:** [`plan.md`](plan.md) + +## Evidence + +Commands run in the worktree after the last code commit (`5c85705f`), on the test databases: + +| Command | Result | +|---|---| +| `bin/ci checks` (AC-5.5) | `Continuous Integration passed in 28.64s`: Setup, Upgrade (released migrations unchanged), RuboCop (875 files, no offenses), YARD without warnings, the comment ratchet (`18 added or changed public objects, all with a comment`), Brakeman (0 warnings), bundler-audit (no vulnerabilities), importmap audit (no vulnerable packages) | +| `ruby script/upgrade/guard.rb` (AC-5.2) | `v0.3.0: its 34 migrations are unchanged` | +| `bin/rspec spec/price_source_core_spec.rb spec/prices_core_spec.rb spec/prices_schema_spec.rb spec/backup_guards_spec.rb spec/config/recurring_spec.rb` (AC-5.2, AC-5.4) | `23 examples, 0 failures` | +| `SCANNER_TIMING=1 bin/rspec spec/models/prices/refresh_timing_spec.rb` (AC-5.6) | see the timing below | + +`bin/upgrade-smoke` (AC-5.2's last clause) needs a container engine and runs in CI's image job; it is not run here. + +**Timing (AC-5.6)**, run on the development machine, once, alone: a generated file of 110,000 cards with 275,000 quotes, first run all added (275,000 added), then a typical day of 256,666 unchanged and 18,334 repriced. The example printed: + +``` +price refresh timing: {first_run: 18.4, typical_day: 7.4} +``` + +in seconds; the whole example took 36.55 s (it builds the file and the catalog entries too). A measurement with no threshold (the spec's wording); #366 replaces it with the run's own numbers. + +## Acceptance criteria + +| AC | Evidence | +|---|---| +| AC-1.1 | `spec/models/prices/refresh_spec.rb` "adds a price no one has, with a history row for the run's date, and finishes the run (AC-1.1, AC-2.1)"; `refresh_end_to_end_spec.rb` "downloads once, applies the file's quotes dated by its name, and makes no request inside a transaction (AC-4.5)" (run, `source_version`, counts) | +| AC-1.2 | `refresh_spec.rb` "skips a file already applied for the same version and languages, copying its date (AC-1.2; …)"; `refresh_end_to_end_spec.rb` "changes nothing when the same file is read again (AC-1.7)" proves the second run is `skipped`, the file is requested once and the rows are unchanged | +| AC-1.3 | `refresh_spec.rb` "applies a file of the same version when the language set differs (AC-1.3)" | +| AC-1.4 | `refresh_spec.rb` "skips a file older than the latest finished or failed run (AC-1.4)" and "skips a file older than a finished run, and applies another version of the finished run's own day (AC-1.4)" | +| AC-1.5 | `refresh_spec.rb` "skips with a reason and no date when the catalog has no printing of the type, asking the catalog's source nothing (AC-1.5)" | +| AC-1.6 | `refresh_spec.rb` "gives every written row the file's date, whatever the clock says (AC-1.6)" | +| AC-1.7 | `refresh_spec.rb` "writes the same rows when the job is performed twice on one file (AC-1.7)"; `refresh_end_to_end_spec.rb` "changes nothing when the same file is read again (AC-1.7)" | +| AC-1.8 | `refresh_spec.rb` "syncs the source before a run exists, so a source that fails verification fails the job with no run (AC-1.8)" and "syncs the source and its series first, so their rows exist when the run opens (AC-1.8)" | +| AC-1.9 | `refresh_spec.rb` "ends the run failed, with its error, when the source's file has no date, and nothing is written (AC-1.9)"; `spec/prices_schema_spec.rb` "accepts a run with no date and refuses an unknown status" (the table accepts a nil `run_on`) | +| AC-2.1 | `refresh_spec.rb` "adds a price no one has, with a history row for the run's date, and finishes the run (AC-1.1, AC-2.1)" | +| AC-2.2 | `refresh_spec.rb` "writes nothing for an unchanged price, and the current price's dates stay (AC-2.2)" | +| AC-2.3 | `refresh_spec.rb` "updates a changed price and adds one history row, leaving the earlier ones (AC-2.3)" (the count is `repriced` since 0.4.3) | +| AC-2.4 | `refresh_spec.rb` "removes a price the file leaves out and writes a blank history row for the run's date (AC-2.4)" and "treats a price that comes back as an added quote with its own history row (AC-2.4)" | +| AC-2.5 | `refresh_spec.rb` "skips a printing the catalog doesn't have, counting it once however many series it has (AC-2.5)" | +| AC-2.6 | `refresh_spec.rb` "judges each series of a printing on its own (AC-2.6)" | +| AC-2.7 | `refresh_spec.rb` "replaces a history row of the same day when a second file of that day differs (AC-2.7)"; `spec/models/price_change_spec.rb` "is keyed by printing, source, series and day, and an upsert on that key replaces the row" | +| AC-2.8 | `refresh_spec.rb` "keeps rows keyed by the entry's id, in the prices database (AC-2.8)"; the suite has both databases, so a join or an attach would fail every example of the file | +| AC-2.9 | `refresh_spec.rb` "removes a price of a printing absent from the file altogether, and of a retired entry (AC-2.9)" | +| AC-2.10 | `refresh_spec.rb` "judges the first of two quotes for one printing and series, ignoring the second (AC-2.10)" | +| AC-2.11 | `refresh_spec.rb` "raises, fails the run and keeps earlier batches when a quote's series was not declared (AC-2.11)" | +| AC-2.12 | `refresh_spec.rb` "logs a malformed record's key and error, and counts it (AC-2.12)" | +| AC-3.1 | `refresh_spec.rb` "does not remove a price whose quote was unreadable, and the printing's other series are judged as usual (AC-3.1)" | +| AC-3.2 | `refresh_spec.rb` "removes none of a printing's prices when its line could be read only as far as its key (AC-3.2)" | +| AC-3.3 | `refresh_spec.rb` "removes nothing at all when a line names no printing, applying the quotes around it (AC-3.3)" | +| AC-3.4 | `refresh_spec.rb` "fails the run, with the reader's error, when the file is cut short; earlier batches stand, nothing is removed, and a whole file of the day finishes …" (AC-3.4) | +| AC-3.5 | `refresh_spec.rb` "fails the run, 'no valid quotes', and writes nothing when the file yields no quote (AC-3.5)" and "finishes a file whose quotes are all for printings the catalog lacks, counting them and writing or removing nothing (AC-3.5)" | +| AC-3.6 | `refresh_spec.rb` "refuses every removal and fails the run when absences exceed the share, counting them as withheld (AC-3.6)" and "removes absences exactly at the share (AC-3.6)" | +| AC-3.7 | `refresh_spec.rb` "leaves a price whose printing is in a language the run did not read, or the primary lacks, and does not count it (AC-3.7)" | +| AC-3.8 | `refresh_spec.rb` "reads a failed file again on the next run (AC-3.8)" | +| AC-3.9 | `refresh_spec.rb` "leaves both or neither of a removal batch's current price and blank history row when the batch fails (AC-3.9)" | +| AC-4.1 | `spec/models/price_refresh_run_spec.rb` ".start!" "records a skipped run, 'already running', while another run has recent progress (AC-4.1)"; `refresh_spec.rb` "records a skipped run, 'already running', for a run with recent progress, and writes nothing (AC-4.1)" | +| AC-4.2 | `price_refresh_run_spec.rb` "closes a run with no progress for STALL_AFTER as interrupted and goes ahead (AC-4.2)" and "closes a running run that carries the starting job's own id … (AC-4.2)"; `refresh_spec.rb` "closes a run left behind as interrupted and goes ahead (AC-4.2)" | +| AC-4.3 | `spec/jobs/prices/refresh_job_spec.rb` "allows one refresh per source key at a time, on the sync queue (AC-4.3)" | +| AC-4.4 | `refresh_spec.rb` describe "transactions (AC-4.4)": "writes in batches of BATCH_SIZE printings inside savepoint spans, the run row in spans of its own, and reads the primary outside them" (savepoint spans, spec 0.4.2; deviation c below) | +| AC-4.5 | `refresh_end_to_end_spec.rb` "downloads once, applies the file's quotes dated by its name, and makes no request inside a transaction (AC-4.5)" (`MTG::Scryfall::Source` over stubbed requests; the `after_request` hook compares open transactions to a baseline, deviation e below) | +| AC-4.6 | `refresh_job_spec.rb` the per-error examples "retries on …" and "fails, without a retry or a discard, on …" (AC-4.6), and "fails on the third attempt of a transient error, not retried again (AC-4.6)" | +| AC-4.7 | `refresh_spec.rb` "touches the run while the download reports progress, at most once per HEARTBEAT_EVERY (AC-4.7)", "touches the run after every batch, even one that writes nothing (AC-4.7)" and "touches the run for each slice of the removal walk (AC-4.7)"; `price_refresh_run_spec.rb` "#progress! and #heartbeat" | +| AC-4.8 | `spec/models/mtg/scryfall/source_spec.rb` "deletes a partial file left by a killed download once it is a day old, and keeps a fresh one (spec 089 AC-4.8)" and "writes each download to a partial file of its own, so a second download of the version, started inside the first, is harmless (spec 089 AC-4.8)" | +| AC-5.1 | `spec/config/recurring_spec.rb` "refreshes prices daily in production, at another time than the catalog's refresh, and not in development (spec 089 AC-5.1)" | +| AC-5.2 | `spec/prices_schema_spec.rb` context "with spec 089's migration (AC-5.2, AC-1.9)" (a database migrated as main has it keeps its rows, check, key and index; the migration goes down without losing a row) and the file's examples for `WITHOUT ROWID` on both builds; `spec/backup_guards_spec.rb`; `ruby script/upgrade/guard.rb`; commands and output in Evidence; `bin/upgrade-smoke` is CI's image job | +| AC-5.3 | `refresh_spec.rb` "judges each series of a printing on its own (AC-2.6)" (`quotes_seen` 3 = 1 + 1 + 1) and `refresh_end_to_end_spec.rb` (the finished run's counts) | +| AC-5.4 | `spec/price_source_core_spec.rb` "names no collectible, source or marketplace in the files it adds" (its list names `current_price.rb`, `price_change.rb`, `price_refresh_run.rb`, `app/models/prices/**` and `app/jobs/prices/**`) and "has a matcher that flags a name, so the guard can fail"; `spec/prices_core_spec.rb` | +| AC-5.5 | Evidence: `bin/ci checks` passed (YARD without warnings, the comment ratchet with 18 objects, RuboCop, Brakeman) | +| AC-5.6 | Evidence: `spec/models/prices/refresh_timing_spec.rb` "applies a first run and then a typical day" (excluded without `SCANNER_TIMING`); the line above | +| AC-5.7 | Verified by reading, see below | + +## Document criteria (AC-5.7, verified by reading) + +| File | What it gained or holds | Source | +|---|---|---| +| `README.md` ("Back up", "When to back up") | the sentence now also says to keep backups clear of 11:30 in the container's time zone, when the daily price refresh writes the prices database, which a backup's read transaction holds back like the catalog refresh's window | spec 089 decision 8; ADR 0029, Cost | +| `docs/roadmap.md` (prices lane) | the #365 row ends "Done: [spec 089](specs/089-daily-price-refresh/spec.md)", in the form of #364's row | spec 089; issue #365 | +| `CLAUDE.md` (status paragraph) | the prices database now has "a daily refresh that records each price change once (`Prices::Refresh`, `Prices::RefreshJob`, `PriceRefreshRun`, `CurrentPrice`, `PriceChange`; spec 089, #365) and no reader yet (#366 to #369 are still to come)" | spec 089; issue #365 | +| `docs/specs/README.md` | the spec's row exists (status Approved, plan present); left for `distill-feature` (plan T9.4) | plan T9.4 | +| `docs/architecture/prices.md`, `docs/architecture/catalog-operations.md` | **Not yet written**: `distill-feature` writes them after the implementation review, from the list in plan T9.6; AC-5.7 stays open for these two until then | plan T9.6; `.claude/rules/documentation.md` | + +## Deviations of the run so far + +- (a) The count column `changed` became `repriced`, by a second migration, because Active Record reserves the attribute name `changed` and `PriceRefreshRun` can't be built with it (spec 0.4.3; commits `e4f4e5b0`, `d8f3add3`). +- (b) `MTG::Scryfall::Source#fetch` deletes its own partial file when a download fails, and `#prune` tolerates a file vanishing under it (a concurrent download's cleanup). +- (c) AC-4.4 measures savepoint spans, not transaction depth, because the suite's non-joinable transaction makes every write a savepoint (spec 0.4.2). +- (d) The pending unreadable records are bounded by the batch size: an automated security review found the pending set in `queue_malformed` unbounded; fixed in commit `3e8a6615`. +- (e) T8.1's check that the primary is not read inside a prices transaction compares the prices connection's open transactions to a baseline, because Rails opens a savepoint lazily. + +## Implementation review + +SPEC-ALIGNED + +Mode B review of commits `eb205c8f` to `39cc2c08`, by a read-only reviewer on **Opus 5.5** (spec v0.4.3; it read the code and the specs and ran nothing that writes). It found no blocking and no major finding, and every invariant held against a hunt for counterexamples: a price whose quote was unreadable, or whose printing, language or series the file was not asked about, is never deleted or blanked; only a trusted absence removes a price, with its blank history row in the same transaction; no history row for an unchanged price; an older file is never applied; two runs never both write; the core files name no collectible; no join across the two files, no new index, both large tables still `WITHOUT ROWID`; the released migration is untouched. Open by design: AC-5.2's `bin/upgrade-smoke` clause (CI's image job) and AC-5.7's two area docs, which `distill-feature` writes after this review. + +What the review found, and what was done (by Sonnet 5.5 implementers, test first, commits `ea0c6eb2`, `2dc0896a`, `0f2a9161`, `3aa57018`; the spec change by the main session): + +| Finding | Outcome | +|---|---| +| A current price whose series the source no longer declares would read as absent: a later build that drops a series would fail every run on the ceiling, or blank prices the file was never asked about | **Fixed, and the spec changed** (v0.5.0: decision 4's candidate rule and AC-3.7); `each_candidate_slice` filters to the declared series (`ea0c6eb2`) | +| Quotes repeated while their printing is pending pile up in `@pending[key]` without a bound (the class of the `queue_malformed` finding) | Fixed: a repeat of a pending printing and series is ignored when queued (`2dc0896a`) | +| The rate-limit row of the job spec passed without `Catalog::RateLimitRetry`; the slice-touch example (`>= 12`) was too loose; AC-2.10 covered only a same-batch duplicate; AC-3.4 `be_between(2, 4)`; AC-3.6 did not assert that changes already written stand | Fixed, each proved able to fail (`0f2a9161`) | +| `left_behind?` and `heartbeat` lacked `@param`/`@return`; the core guard's note said "two migrations" | Fixed (`3aa57018`) | +| `@seen` as two-element arrays (about 30 MB for 300,000), `#remove` sending one `DELETE` per printing, `quote.series` rebuilt and validated per quote, `progress!` and `finish!` not checking the run is still `running`, `PriceSeries.ensure!` called twice | **Accepted, not changed**: the measured first run (18.4 s for 110,000 cards) and a typical day (7.4 s) are well inside a daily window, and the hairline of a process resuming after the 6-hour lock lapsed needs two failures at once; the area doc records them so #366's measurements can reopen them | +| `fetch` also deletes its own partial file on failure, beyond FR-2's wording; two migrations (190001 adds `changed`, 190002 renames it) where FR-3 says one | Accepted: benign, recorded as deviations (a) and (b); `catalog-operations.md` says it | + +Fresh evidence after the fixes: `bin/rspec spec/models/prices spec/jobs/prices spec/prices_core_spec.rb spec/price_source_core_spec.rb spec/models/price_refresh_run_spec.rb`: 85 examples, 0 failures; `bin/rubocop` on the touched files: no offenses; `bin/yard doc`: no warnings. The wider targeted run (prices, catalog, scryfall, config, backup guards: 630 examples) passed before the fixes; the full suite and `bin/ci` run once, with the coordinator's slot, at the push. + +## Distillation + +Written by `distill-feature` after the implementation review, in `docs/architecture/prices.md` (the refresh's invariants and states, the operator's view of a daily failure, the run model's migrations and accepted limits, the named constants including `MAX_REMOVED_SHARE` as a guess until #366, the sources and the log row) and `docs/architecture/catalog-operations.md` (the shared download, the partial file per download, `STALE_PARTIAL_AFTER`, the supersession of spec 002 FR-3's "a partial file", the sources and the log row). AC-5.7's two area documents are now written; the log rows name "spec 089's pull request" until it has a number. The index row is Frozen with its plan "retirement owed". diff --git a/docs/specs/README.md b/docs/specs/README.md index 23e62c6d..be3889a1 100644 --- a/docs/specs/README.md +++ b/docs/specs/README.md @@ -22,7 +22,7 @@ A plan is deleted from the tree in a commit of its own after its feature's pull | Spec | Status | Areas | Plan | Supersedes | Superseded by | |---|---|---|---|---|---| | [001 Rails Application Skeleton](001-rails-app-skeleton/spec.md) | Frozen, by [release and deploy](../architecture/release-and-deploy.md) | Release and deploy (container, Compose, Kamal); CLAUDE.md (R5): stack, CI, hooks | retired in `96d5e651ea8a2696f6372ecc41907a2d20a2efe1`; read it with `git show 96d5e651ea8a2696f6372ecc41907a2d20a2efe1^:docs/specs/001-rails-app-skeleton/plan.md` | | The Compose file that builds the image from a checkout and Kamal's placeholder registry (FR-5, FR-6, AC-3.1, AC-4.1): 012 (FR-4, FR-5; AC-1.5, AC-5.1). AC-2.5's single check job, and Story 2's "CI passes" meaning the same in the pre-push hook for documentation-only branches: 035 (decisions 3, 5) | -| [002 Scryfall Catalog Ingestion and Card Search](002-scryfall-catalog-search/spec.md) | Frozen, by [catalog operations](../architecture/catalog-operations.md) (FR-3–FR-6 and their Error Scenarios rows) and [catalog core](../architecture/catalog-core.md) (FR-1, FR-2, FR-7–FR-10; FR-9's convention stays in the rules) | [Catalog operations](../architecture/catalog-operations.md) (FR-3–FR-6); [Catalog core](../architecture/catalog-core.md) (FR-1, FR-2, FR-7–FR-10) | retired in `7876bfd8c0126b232043cc6968693b4184c86256`; read it with `git show 7876bfd8c0126b232043cc6968693b4184c86256^:docs/specs/002-scryfall-catalog-search/plan.md` | | Non-Goals and FR-7, public search: 004 (sign-in), 023 (an operator setting). AC-1.5, AC-1.7, AC-1.8 and AC-1.11's messages, AC-10.1, FR-8's search link: 004 (AC-6.5, AC-8.2). AC-3.2, the card page's attribution: 018 (the footer). AC-1.11, the not-loaded notice: 015 (per catalog type, AC-1.1, AC-1.2). FR-4, the 6-hour "already running" rule: 015 (15-minute stall, job rules, AC-3.3, AC-3.4). FR-4 and its two 429 Error Scenarios rows, the wait for another job's retry: 024 (amended in place, v1.4.0). FR-3, FR-4 and Error Scenarios, rate-limit waits: amended in place for #74 and #81 (v1.2.0, v1.3.0). 043: FR-4's "more than 3 such retries of one job" and the Error Scenarios row "rate limits the same job a fourth time" (the fourth rate limit in a row with no progress between them) | +| [002 Scryfall Catalog Ingestion and Card Search](002-scryfall-catalog-search/spec.md) | Frozen, by [catalog operations](../architecture/catalog-operations.md) (FR-3–FR-6 and their Error Scenarios rows) and [catalog core](../architecture/catalog-core.md) (FR-1, FR-2, FR-7–FR-10; FR-9's convention stays in the rules) | [Catalog operations](../architecture/catalog-operations.md) (FR-3–FR-6); [Catalog core](../architecture/catalog-core.md) (FR-1, FR-2, FR-7–FR-10) | retired in `7876bfd8c0126b232043cc6968693b4184c86256`; read it with `git show 7876bfd8c0126b232043cc6968693b4184c86256^:docs/specs/002-scryfall-catalog-search/plan.md` | | Non-Goals and FR-7, public search: 004 (sign-in), 023 (an operator setting). AC-1.5, AC-1.7, AC-1.8 and AC-1.11's messages, AC-10.1, FR-8's search link: 004 (AC-6.5, AC-8.2). AC-3.2, the card page's attribution: 018 (the footer). AC-1.11, the not-loaded notice: 015 (per catalog type, AC-1.1, AC-1.2). FR-4, the 6-hour "already running" rule: 015 (15-minute stall, job rules, AC-3.3, AC-3.4). FR-4 and its two 429 Error Scenarios rows, the wait for another job's retry: 024 (amended in place, v1.4.0). FR-3, FR-4 and Error Scenarios, rate-limit waits: amended in place for #74 and #81 (v1.2.0, v1.3.0). 043: FR-4's "more than 3 such retries of one job" and the Error Scenarios row "rate limits the same job a fourth time" (the fourth rate limit in a row with no progress between them); [089](089-daily-price-refresh/spec.md) (FR-3 "a partial file": the partial file is named uniquely per download and stale ones are pruned) | | [003 Worktree Environment Setup](003-worktree-environment-setup/spec.md) | Frozen, by [release and deploy](../architecture/release-and-deploy.md) | CLAUDE.md (R5) | retired in `8498f1b0e3a07b8ec0e933f7c16df1e6339e335a`; read it with `git show 8498f1b0e3a07b8ec0e933f7c16df1e6339e335a^:docs/specs/003-worktree-environment-setup/plan.md` | | | | [004 Design System, Accounts, Adding Cards and the Collection Grid](004-design-system-collection/spec.md) | Frozen, by [collection and lots](../architecture/collection-and-lots.md) (the collection and adding cards), [catalog core](../architecture/catalog-core.md) (search, the card page's catalog content and the vocabulary), [tenancy and accounts](../architecture/tenancy-and-accounts.md) (accounts, users, admin users) and the [design system](../design-system/README.md) (its design-system parts) | [Collection and lots](../architecture/collection-and-lots.md) (collection, adding cards); [Catalog core](../architecture/catalog-core.md) (search, the card page's catalog content, the collecting vocabulary); [Tenancy and accounts](../architecture/tenancy-and-accounts.md) (accounts, users, admin users); [design system](../design-system/README.md) | retired in `528c0c22c08486e63c3452148f7a5440f6c71755`; read it with `git show 528c0c22c08486e63c3452148f7a5440f6c71755^:docs/specs/004-design-system-collection/plan.md` | 002 Non-Goals and FR-7, public search: search needs sign-in (FR-2). 002 AC-1.5, AC-1.7, AC-1.8 and AC-1.11's messages, AC-10.1, FR-8's search link (AC-6.5, AC-8.2) | FR-2, sign-in for the search, card and printings pages: 023 (not needed when the operator opens the catalog). AC-6.5, the not-loaded message: 015 (AC-1.1, AC-1.2). AC-6.3, unowned tiles shown faded: 021 (a marker, the image unfaded). AC-8.2, the card page's attribution: 018 (the footer). AC-11.7, FR-10's filter bar, the return target of Edit copy and Remove (AC-2.7, AC-10.1, AC-10.4), AC-10.4 and bulk Undo: 006 (AC-1.7). AC-11.6, the empty collection's sentence, and the collection page head: 026 (a link and an "Add by number" action, its Supersedes list). The header's navigation for a collector (Collection, Search, Scan): 029 (gains Locations after Scan, decision 8). FR-6 and the merge criteria (AC-7.4, AC-9.3, AC-10.2), the add and edit forms, the card page's Your copies table: 032 (a lot's identity includes its location; Location field, column and Move copies). The tile gains a place line (no location line, AC-11.2, the Non-Goal): 033; Story 11's filter bar having only the name search: 037 (the Location select). FR-6's price paid in one instance-wide currency and its "Currency: an instance setting" line, AC-9.1's "in the instance currency": 038 (each account's own; `COLLECTOR_CURRENCY` is the default a new account starts with). FR-1's `CLAUDE.md` guidance that the repo copy of the design system is updated by re-exporting: 057 (decision 12; the design system is the project's own, in code); 071 (AC-9.5 in part: Add a copy stays visible and is the flexible control of the head row beside Add to deck and the menu, as ItemPagePhone documents); 073 (FR-9's single stat, its Non-Goal and Out of Scope entries, AC-8.1's order in part) | | [005 Card Scanner Phase 0 — Feasibility Spikes](005-card-scanner-phase-0/spec.md) | Frozen, by [scanner](../architecture/scanner.md) | [Scanner](../architecture/scanner.md) | retired in `a5e252f9273889df360dd759e5bffb5380963cc1`; read it with `git show a5e252f9273889df360dd759e5bffb5380963cc1^:docs/specs/005-card-scanner-phase-0/plan.md` | | | @@ -96,11 +96,12 @@ A plan is deleted from the tree in a commit of its own after its feature's pull | [078 A Mana Bar Label Fits Its Segment or Isn't Drawn](078-mana-bar-label-fit/spec.md) | Frozen, by [collection and lots](../architecture/collection-and-lots.md) (the panel's paragraph); the panel's markup and CSS rule are `DeckShape` in the [design system](../design-system/README.md) | [Design system](../design-system/README.md) (`DeckShape`), [collection and lots](../architecture/collection-and-lots.md) (the panel's paragraph) | retired in `b275dc7e20ef03355d825eec48569c2eb7ce8df0`; `git show b275dc7e20ef03355d825eec48569c2eb7ce8df0^:docs/specs/078-mana-bar-label-fit/plan.md` | In part 077 decision 2 (the mana-label band as a documented limit, ruled A on 2026-11-11), its Goal 1's qualification, Open Questions, Error Scenarios row for the band, AC-5.1 and AC-5.3, and its Supersedes row's exception for 060 AC-2.4 ([#330](https://github.com/plainprogrammer/Collector/issues/330)); 060 decision 4 and 069 decision 9 are kept | | | [079 Deck Checks (Deck Size, Copy Limits, and Owned Plus Proxy of One Card)](079-deck-checks/spec.md) | Frozen, by [collection and lots](../architecture/collection-and-lots.md) (the checks' rules); the results section's markup and CSS rule are `DeckCheck` in the [design system](../design-system/README.md) | [Collection and lots](../architecture/collection-and-lots.md) (decks and the deck page), [DeckPage](../design-system/components/DeckPage.md) | retired in `2086fd143fdd0a99fb5ade70f050e15ab2b0650d`; `git show 2086fd143fdd0a99fb5ade70f050e15ab2b0650d^:docs/specs/079-deck-checks/plan.md` | Nothing. It does in part (size and copy limits) the "deck rules" that 069 left as a non-goal | | | [080 Design System: the Phone-First Scanner](080-phone-first-scanner-components/spec.md) | Frozen, by [design system](../design-system/README.md) (the README's rules for focused pages, sheets, the status band and motion; `Sheet`, `StatusBand`, `Scanner`; the previews; the tokens and `additions.css`) | [Design system](../design-system/README.md) (focused pages, sheets, the status band, motion; `Sheet`, `StatusBand`, `Scanner`; `additions.css`, the tokens) | retired in `93580784d80713b5b1ca8151b442d29c256fa38a`; `git show 93580784d80713b5b1ca8151b442d29c256fa38a^:docs/specs/080-phone-first-scanner-components/plan.md` | Nothing shipped; under `prefers-reduced-motion` every component loses its transitions (decision 14) | | -| [081 A Prices Database That the Backup Holds](081-prices-database-in-backup/spec.md) | Frozen, by [prices](../architecture/prices.md), [release and deploy](../architecture/release-and-deploy.md) and [data portability](../architecture/data-portability.md) | Prices (the prices database); release and deploy (backup and restore, the upgrade proof); data portability (an export holds no price) | retired in `9a9e0f5e32766dc072eb642522ca38ca24a12275`; `git show 9a9e0f5e32766dc072eb642522ca38ca24a12275^:docs/specs/081-prices-database-in-backup/plan.md` | Spec 030 decision 2 in part (the backup holds the prices database too), decision 5 in part (backup format version 2), decision 6 (a fresh volume has none of five databases) decision 13 and AC-1.11 (free space for both databases), AC-1.4 (the archive's files and manifest) and AC-5.5's four production databases; spec 031 AC-2.1, AC-2.4, AC-2.5, AC-3.2, AC-4.5, AC-6.2 and AC-8.1 in part (the proof covers the prices database); ADR 0024 decisions 2, 3 and 4 in part (ADR 0029) | [086](086-start-refuses-orphan-prices/spec.md) in part (decision 8's "never start" advice and AC-5.9's advice: a start now refuses the volume) | +| [081 A Prices Database That the Backup Holds](081-prices-database-in-backup/spec.md) | Frozen, by [prices](../architecture/prices.md), [release and deploy](../architecture/release-and-deploy.md) and [data portability](../architecture/data-portability.md) | Prices (the prices database); release and deploy (backup and restore, the upgrade proof); data portability (an export holds no price) | retired in `9a9e0f5e32766dc072eb642522ca38ca24a12275`; `git show 9a9e0f5e32766dc072eb642522ca38ca24a12275^:docs/specs/081-prices-database-in-backup/plan.md` | Spec 030 decision 2 in part (the backup holds the prices database too), decision 5 in part (backup format version 2), decision 6 (a fresh volume has none of five databases) decision 13 and AC-1.11 (free space for both databases), AC-1.4 (the archive's files and manifest) and AC-5.5's four production databases; spec 031 AC-2.1, AC-2.4, AC-2.5, AC-3.2, AC-4.5, AC-6.2 and AC-8.1 in part (the proof covers the prices database); ADR 0024 decisions 2, 3 and 4 in part (ADR 0029) | [086](086-start-refuses-orphan-prices/spec.md) in part (decision 8's "never start" advice and AC-5.9's advice: a start now refuses the volume); [089](089-daily-price-refresh/spec.md) (the `price_refresh_runs` table: `run_on` is nullable, `job_id`, `languages` and counts are added) | | [082 Scanner: Capture Always in View](082-scanner-capture-always-in-view/spec.md) | Frozen, by [scanner](../architecture/scanner.md) and [catalog core](../architecture/catalog-core.md) (the notice on the scanner); Story 7's paired re-measure withdrawn by the maintainer (1.1.0); the markup is `Scanner`, `Sheet` and `Footer` in the [design system](../design-system/README.md) | [Scanner](../architecture/scanner.md) (the page, the sheets, the lines, the live guide); [catalog core](../architecture/catalog-core.md) (the notice on the scanner); [design system](../design-system/README.md) (`Scanner`, `Sheet`, `Footer`) | retired in `92b6ce71eb4809135cbc24c0abe03cd3c088f9ad`; `git show 92b6ce71eb4809135cbc24c0abe03cd3c088f9ad^:docs/specs/082-scanner-capture-always-in-view/plan.md` | 018 FR-2 for the scanner page; 009 AC-1.3, 040's notice and 042 AC-1.4 (the add's sentence); the sitting panel's place (009 AC-3.2, AC-3.5; 042); 040 AC-7.1 and 042 AC-7.1's "no script is added" as far as reaching the control; 047's status region for "Set" and Undo; 007 AC-1.1's stage for live capture; 009 AC-7.2, AC-7.4 (the framing advice on the page) and AC-8.1 ("Scan" marked current); 011 AC-5.2's art line as a live region on the page; 084 decisions 1 and 11, AC-1.1 and AC-4.3 for the scanner and measurement pages (the focused-page exception: no footer and no `c-shell--footed`, the small print in the sheets' foot, patch 1.0.3) | | | [083 Prices on Non-English Printings](083-non-english-printing-prices/research.md) (research) | Approved (findings ruled on 2026-11-13, [ruling 41](../roadmap.md#2026-11-13-prices-on-non-english-printings-research-083): option C); ruling 41 is distilled into [prices](../architecture/prices.md) with spec 085 | [Prices](../architecture/prices.md); the prices lane ([price data design](../superpowers/specs/2026-11-13-price-data-design.md)); [catalog operations](../architecture/catalog-operations.md) (the bulk file the catalog reads) | none (no plan) | Settles open question 1 of the [price data design](../superpowers/specs/2026-11-13-price-data-design.md) (non-English printings); the design's six price series are five in the 2026-11-13 file (no etched EUR) | | | [084 The Running Version, Its Commit and the Licence, Shown in the App](084-version-and-licence-in-app/spec.md) | Frozen, by [catalog core](../architecture/catalog-core.md) (the licence line, the always-present footer, the shared notice partial) and [release and deploy](../architecture/release-and-deploy.md) (the image's build facts); the footer's markup is `Footer` in the [design system](../design-system/README.md) | [Catalog core](../architecture/catalog-core.md) (the footer), [release and deploy](../architecture/release-and-deploy.md) (the image's build facts), [design system](../design-system/README.md) (`Footer`, `Sheet`, `VisitorHeader`) | retired in `8c6af54dac8197b04d0019b075026337bb3f4230`; `git show 8c6af54dac8197b04d0019b075026337bb3f4230^:docs/specs/084-version-and-licence-in-app/plan.md` | 018 AC-2.3 and FR-2's "be absent (no element) when no type supplies a notice"; in part, 018 Goal 2 and FR-2's opening sentence (the footer also holds the core's licence line) ([#374](https://github.com/plainprogrammer/Collector/issues/374)) | 087: AC-1.4's "no other part of the line is a link", the full wording of AC-2.1, AC-2.3, AC-2.4 and AC-2.14 and decision 2's wording table, AC-2.13's two links, FR-1's "with two links", the Accessibility NFR's and Users and Context's two links, decision 12's no-script notice (it gains the link), the Non-Goals "a licence page in the app" and "Listing third-party licences" (the line gains a third sentence and link); decision 8 is carried out there. Decisions 1 and 11, AC-1.1 and AC-4.3, for the scanner and measurement pages: 082 (decision 16, patch 1.0.3; a one-screen focused page has no footer and carries the small print in its sheets' foot) | | [085 A Price Source Contract, and Scryfall's Prices Through It](085-price-source-contract/spec.md) | Frozen, by [prices](../architecture/prices.md) | [Prices](../architecture/prices.md) (the source contract, Scryfall's source, the source and series rows) | retired in `e78c902bc0f9f6158bfddc1fbfffdf1edf9da3e4`; `git show e78c902bc0f9f6158bfddc1fbfffdf1edf9da3e4^:docs/specs/085-price-source-contract/plan.md` | | | | [086 A Start Refuses a Volume That Holds a Prices Database and No Primary](086-start-refuses-orphan-prices/spec.md) | Frozen, by [release and deploy](../architecture/release-and-deploy.md) | [Release and deploy](../architecture/release-and-deploy.md) (the start check before `db:prepare`, its message, the restore's advice and the smoke leg); [prices](../architecture/prices.md) (a pointer only) | retired in `2c5939d4604cc1221534677c3e4bcc4aa5bdc916`; `git show 2c5939d4604cc1221534677c3e4bcc4aa5bdc916^:docs/specs/086-start-refuses-orphan-prices/plan.md` | In part, spec 081 decision 8's "Collector must never be started on such a volume" and AC-5.9's advice (the start now refuses it; the restore's advice says so) ([#375](https://github.com/plainprogrammer/Collector/issues/375)) | | | [087 The Licences of the Third-Party Software and Assets in the Image, Readable from the App](087-third-party-licences-in-app/spec.md) | Frozen, by [catalog core](../architecture/catalog-core.md) (the licence line's third sentence, the page, the reader, the manifest, the guard), [release and deploy](../architecture/release-and-deploy.md) (the smoke check) and [tenancy and accounts](../architecture/tenancy-and-accounts.md) (the endpoint); the `Licences` pattern is in the [design system](../design-system/README.md) | [Catalog core](../architecture/catalog-core.md) (the footer and the third-party licences page); [release and deploy](../architecture/release-and-deploy.md) (the smoke check); [tenancy and accounts](../architecture/tenancy-and-accounts.md) (the endpoint that skips sign-in); [design system](../design-system/README.md) (`Licences`, `Footer`) | retired in `11de80918d734d3b6e001d16b013f8a77c820ffc`; `git show 11de80918d734d3b6e001d16b013f8a77c820ffc^:docs/specs/087-third-party-licences-in-app/plan.md` | 084: AC-1.4's "no other part of the line is a link", the full wording of AC-2.1, AC-2.3, AC-2.4 and AC-2.14 and decision 2's wording table, AC-2.13's two links, FR-1's "with two links", the Accessibility NFR's and Users and Context's two links, decision 12's no-script notice (it gains the link), the Non-Goals "a licence page in the app" and "Listing third-party licences" ([#376](https://github.com/plainprogrammer/Collector/issues/376)) | | +| [089 Prices Refresh Once a Day, and Each Change Is Recorded Once](089-daily-price-refresh/spec.md) | Frozen, by [prices](../architecture/prices.md) and [catalog operations](../architecture/catalog-operations.md) (decision 1's shared download) | [Prices](../architecture/prices.md) (the daily refresh, its run record and guards, the removal rules, the models of the three tables); [catalog operations](../architecture/catalog-operations.md) (the shared download: the partial file per download, the stale partial pruning) | retirement owed | 002 FR-3's "a partial file" (the partial file is unique per download and stale ones are pruned); 081's `price_refresh_runs` table (`run_on` nullable, `job_id`, `languages`, counts) | | | [091 CI Fails on a Dependency Whose Licence Isn't Allowed](091-licence-check-in-ci/spec.md) | Frozen, by [catalog core](../architecture/catalog-core.md) (the policy file, the check and its rules) and [release and deploy](../architecture/release-and-deploy.md) (the `bin/ci checks` step); [ADR 0031](../adr/0031-a-dependency-licence-is-a-decision.md) | [Catalog core](../architecture/catalog-core.md) (the third-party licences page, its guard, the policy and the check); [release and deploy](../architecture/release-and-deploy.md) (the step) | retired in `9748579451730adc98fb0db5b626a9492fdd4598`; `git show 9748579451730adc98fb0db5b626a9492fdd4598^:docs/specs/091-licence-check-in-ci/plan.md` | | | diff --git a/spec/config/recurring_spec.rb b/spec/config/recurring_spec.rb index 290b9027..c1be3002 100644 --- a/spec/config/recurring_spec.rb +++ b/spec/config/recurring_spec.rb @@ -9,6 +9,15 @@ expect(Fugit.parse_cronish(task["schedule"]).to_cron_s).to eq("15 3 * * 1") end + it "refreshes prices daily in production, at another time than the catalog's refresh, and not in development (spec 089 AC-5.1)", :aggregate_failures do + config = YAML.load_file(Rails.root.join("config/recurring.yml"), aliases: true) + task = config.dig("production", "refresh_prices") + + expect(task).to include("class" => "Prices::RefreshJob", "args" => [ "scryfall" ], "queue" => "sync") + expect(Fugit.parse_cronish(task["schedule"]).to_cron_s).to eq("30 11 * * *") + expect(config.dig("development", "refresh_prices")).to be_nil + end + %w[development production].each do |environment| it "schedules the import upload sweep every 15 minutes in #{environment} (spec 028 AC-5.5)" do task = YAML.load_file(Rails.root.join("config/recurring.yml"), aliases: true).dig(environment, "sweep_import_uploads") diff --git a/spec/jobs/prices/refresh_job_spec.rb b/spec/jobs/prices/refresh_job_spec.rb new file mode 100644 index 00000000..d032e79d --- /dev/null +++ b/spec/jobs/prices/refresh_job_spec.rb @@ -0,0 +1,62 @@ +require "rails_helper" + +RSpec.describe Prices::RefreshJob, type: :job do + it "allows one refresh per source key at a time, on the sync queue (AC-4.3)", :aggregate_failures do + expect(described_class.queue_name).to eq("sync") + expect(described_class.concurrency_limit).to eq(1) + expect(described_class.concurrency_on_conflict).to eq(:block) + expect(described_class.concurrency_duration).to eq(6.hours) + expect(described_class.new("fake_source").concurrency_key).to eq("Prices::RefreshJob/fake_source") + end + + it "runs the refresh for the source key, under its own job id" do + refresh = instance_double(Prices::Refresh, call: nil) + allow(Prices::Refresh).to receive(:new).and_return(refresh) + job = described_class.new("fake_source") + + job.perform_now + + expect(Prices::Refresh).to have_received(:new).with("fake_source", job_id: job.job_id) + end + + # decision 7's table (AC-4.6): the job's fate for each error; no error is discarded + { + "Catalog::Sources::TransientError" => [ Catalog::Sources::TransientError, "timeout", :retried ], + "ActiveRecord::StatementTimeout" => [ ActiveRecord::StatementTimeout, "busy", :retried ], + "Catalog::Sources::IntegrityError" => [ Catalog::Sources::IntegrityError, "downloaded 1 bytes", :retried ], + "Catalog::Sources::RateLimitedError" => [ Catalog::Sources::RateLimitedError, "slow down", :retried ], + "Prices::Sources::IntegrityError" => [ Prices::Sources::IntegrityError, "no date", :failed ], + "Catalog::Sources::ConfigurationError" => [ Catalog::Sources::ConfigurationError, "bad language", :failed ], + "Zlib::Error" => [ Zlib::Error, "cut short", :failed ], + "Prices::Refresh::GuardError" => [ Prices::Refresh::GuardError, "over the ceiling", :failed ], + "RuntimeError" => [ RuntimeError, "anything else", :failed ] + }.each do |name, (error, message, fate)| + it "#{fate == :retried ? "retries" : "fails, without a retry or a discard,"} on #{name} (AC-4.6)", :aggregate_failures do + allow(Prices::Refresh).to receive(:new).and_raise(error, message) + + if fate == :retried + expect { described_class.perform_now("fake_source") }.to have_enqueued_job(described_class).with("fake_source") + else + expect { described_class.perform_now("fake_source") }.to raise_error(error, message) + end + end + end + + it "waits at least Catalog::RateLimitRetry::SHORTEST_WAIT before it tries a rate-limited source again, which retry_on's own backoff would not (AC-4.6)", :aggregate_failures do + allow(Prices::Refresh).to receive(:new).and_raise(Catalog::Sources::RateLimitedError, "slow down") + job = described_class.new("fake_source") + + job.perform_now + + expect(ActiveJob::Base.queue_adapter.enqueued_jobs.last[:at]).to be >= Time.now.to_f + Catalog::RateLimitRetry::SHORTEST_WAIT.to_i - 1 + expect(job.exception_executions).to include(Catalog::RateLimitRetry::COUNTED_AS => 1) + end + + it "fails on the third attempt of a transient error, not retried again (AC-4.6)" do + allow(Prices::Refresh).to receive(:new).and_raise(Catalog::Sources::TransientError, "timeout") + job = described_class.new("fake_source") + 2.times { job.perform_now } # two retries are queued (retry_on counts per exception class, in the job's exception_executions) + + expect { job.perform_now }.to raise_error(Catalog::Sources::TransientError, "timeout") + end +end diff --git a/spec/models/current_price_spec.rb b/spec/models/current_price_spec.rb new file mode 100644 index 00000000..46882a8b --- /dev/null +++ b/spec/models/current_price_spec.rb @@ -0,0 +1,19 @@ +require "rails_helper" + +RSpec.describe CurrentPrice, type: :model do + it "is keyed by printing, source and series, and an upsert on that key replaces the row", :aggregate_failures do + source = PriceSource.sync!(FakePriceSource) + series = PriceSeries.ensure!(FakePriceSource.series).values.first + keys = { catalog_entry_id: 7, price_source_id: source.id, price_series_id: series.id } + + described_class.upsert_all([ keys.merge(cents: 100, reported_on: Date.new(2026, 11, 12), changed_on: Date.new(2026, 11, 12)) ], unique_by: described_class.primary_key) + described_class.upsert_all([ keys.merge(cents: 150, reported_on: Date.new(2026, 11, 13), changed_on: Date.new(2026, 11, 13)) ], unique_by: described_class.primary_key) + + expect(described_class.primary_key).to eq(%w[catalog_entry_id price_source_id price_series_id]) + expect(described_class.pluck(:cents, :changed_on)).to eq([ [ 150, Date.new(2026, 11, 13) ] ]) + end + + it "lives in the prices database and has no account" do + expect([ described_class.connection_db_config.name, described_class.column_names & %w[account_id user_id] ]).to eq([ "prices", [] ]) + end +end diff --git a/spec/models/mtg/scryfall/source_spec.rb b/spec/models/mtg/scryfall/source_spec.rb index 3a7f0cb6..a1122ae2 100644 --- a/spec/models/mtg/scryfall/source_spec.rb +++ b/spec/models/mtg/scryfall/source_spec.rb @@ -123,6 +123,47 @@ expect(WebMock).to have_requested(:get, /data\.scryfall\.io/).once expect(dir.children.map { |path| path.basename.to_s }).to contain_exactly("c.jsonl.gz", "#{version}.jsonl.gz") end + + it "deletes a partial file left by a killed download once it is a day old, and keeps a fresh one (spec 089 AC-4.8)", :aggregate_failures do + old = dir.join("default-cards-1.jsonl.gz.dead.part") + fresh = dir.join("default-cards-1.jsonl.gz.live.part") + fixed = dir.join("default-cards-0.jsonl.gz.part") # a fixed-name partial file from an earlier build + [ old, fresh, fixed ].each { |path| path.write("x") } + FileUtils.touch(old, mtime: Time.now - 2.days) + FileUtils.touch(fixed, mtime: Time.now - 2.days) + stub_scryfall(cards: [ scryfall_card ]) + version = source.current_version(languages: [ "en" ]) + + source.download(version, dir:) + + expect(dir.children.map { |path| path.basename.to_s }).to contain_exactly("#{version}.jsonl.gz", fresh.basename.to_s) + end + + it "writes each download to a partial file of its own, so a second download of the version, started inside the first, is harmless (spec 089 AC-4.8)", :aggregate_failures do # rubocop:disable RSpec/ExampleLength -- one scenario + cards = [ scryfall_card ] + stub_scryfall(cards:) + version = source.current_version(languages: [ "en" ]) + body = gzip_jsonl(cards) + url = "https://data.scryfall.io/default-cards/#{version}.jsonl.gz" + other = described_class.new(client: MTG::Scryfall::Client.new, env: {}) + other.current_version(languages: [ "en" ]) + calls = 0 + partials_while_nested = nil + stub_request(:get, url).to_return do |_request| + calls += 1 + if calls == 1 + partials_while_nested = dir.glob("*.part").size # the first download's own partial file is open + other.download(version, dir:) + end + { body:, headers: { "Content-Type" => "application/gzip" } } + end + + path = source.download(version, dir:) + + expect(partials_while_nested).to eq(1) + expect(path.size).to eq(body.bytesize) + expect(dir.glob("*.part")).to be_empty + end end describe "#each_entry" do diff --git a/spec/models/price_change_spec.rb b/spec/models/price_change_spec.rb new file mode 100644 index 00000000..134c9c29 --- /dev/null +++ b/spec/models/price_change_spec.rb @@ -0,0 +1,19 @@ +require "rails_helper" + +RSpec.describe PriceChange, type: :model do + it "is keyed by printing, source, series and day, and an upsert on that key replaces the row", :aggregate_failures do + source = PriceSource.sync!(FakePriceSource) + series = PriceSeries.ensure!(FakePriceSource.series).values.first + keys = { catalog_entry_id: 7, price_source_id: source.id, price_series_id: series.id, recorded_on: Date.new(2026, 11, 13) } + + described_class.upsert_all([ keys.merge(cents: 100) ], unique_by: described_class.primary_key) + described_class.upsert_all([ keys.merge(cents: nil) ], unique_by: described_class.primary_key) + + expect(described_class.primary_key).to eq(%w[catalog_entry_id price_source_id price_series_id recorded_on]) + expect(described_class.pluck(:cents)).to eq([ nil ]) + end + + it "lives in the prices database and has no account" do + expect([ described_class.connection_db_config.name, described_class.column_names & %w[account_id user_id] ]).to eq([ "prices", [] ]) + end +end diff --git a/spec/models/price_refresh_run_spec.rb b/spec/models/price_refresh_run_spec.rb new file mode 100644 index 00000000..96b1e535 --- /dev/null +++ b/spec/models/price_refresh_run_spec.rb @@ -0,0 +1,106 @@ +require "rails_helper" + +RSpec.describe PriceRefreshRun, type: :model do + let(:source) { PriceSource.sync!(FakePriceSource) } + let(:now) { Time.utc(2026, 11, 13, 12, 0, 0) } + + def run!(status: "running", job_id: "job-1", updated_at: now, **attributes) + described_class.create!(price_source: source, status:, job_id:, started_at: updated_at, updated_at:, **attributes) + end + + describe ".start!" do + it "opens a running run with its job id and the time", :aggregate_failures do + run = described_class.start!(source, job_id: "job-2", now:) + + expect(run).to have_attributes(status: "running", job_id: "job-2", started_at: now, run_on: nil, finished_at: nil) + end + + it "records a skipped run, 'already running', while another run has recent progress (AC-4.1)", :aggregate_failures do + busy = run!(updated_at: now - 5.minutes) + + skipped = described_class.start!(source, job_id: "job-2", now:) + + expect(skipped).to have_attributes(status: "skipped", error: "already running", finished_at: now) + expect(busy.reload.status).to eq("running") + end + + it "closes a run with no progress for STALL_AFTER as interrupted and goes ahead (AC-4.2)", :aggregate_failures do + left = run!(updated_at: now - described_class::STALL_AFTER - 1.minute) + + run = described_class.start!(source, job_id: "job-2", now:) + + expect(left.reload).to have_attributes(status: "failed", error: "interrupted", finished_at: now) + expect(run.status).to eq("running") + end + + it "closes a running run that carries the starting job's own id (the queue re-ran it) (AC-4.2)", :aggregate_failures do + left = run!(job_id: "job-1", updated_at: now - 1.minute) + + run = described_class.start!(source, job_id: "job-1", now:) + + expect(left.reload).to have_attributes(status: "failed", error: "interrupted") + expect(run.status).to eq("running") + end + + it "leaves another source's runs alone" do + other = PriceSource.create!(key: "other", name: "Other", attribution: "text") + elsewhere = described_class.create!(price_source: other, status: "running", started_at: now, updated_at: now - 1.hour) + + described_class.start!(source, job_id: "job-2", now:) + + expect(elsewhere.reload.status).to eq("running") + end + end + + describe "#skip!, #finish!, #fail!" do + it "ends a run with its reason or counts", :aggregate_failures do + a = run!.tap { it.skip!("v1 already applied", run_on: Date.new(2026, 11, 13)) } + b = run!(job_id: "x").tap { it.finish!(unchanged: 3, repriced: 1, bogus: 9) } + c = run!(job_id: "y").tap { it.fail!(RuntimeError.new("boom"), added: 2) } + + expect(a).to have_attributes(status: "skipped", error: "v1 already applied", run_on: Date.new(2026, 11, 13)) + expect(b).to have_attributes(status: "finished", unchanged: 3, repriced: 1) + expect(c).to have_attributes(status: "failed", error: "RuntimeError: boom", added: 2) + expect([ a, b, c ].map(&:finished_at)).to all(be_present) + end + end + + describe ".applied and .latest_run_on" do + it "finds a finished run of the same version and languages only", :aggregate_failures do + finished = run!(status: "finished", source_version: "v1", languages: "en", run_on: Date.new(2026, 11, 13)) + run!(status: "failed", job_id: "a", source_version: "v1", languages: "en") + run!(status: "finished", job_id: "b", source_version: "v1", languages: "en,ja") + + expect(described_class.applied(source, "v1", "en")).to eq([ finished ]) + expect(described_class.applied(source, "v2", "en")).to be_empty + end + + it "takes the latest run_on of finished and failed runs, never of skipped or running ones" do + run!(status: "finished", job_id: "a", run_on: Date.new(2026, 11, 10)) + run!(status: "failed", job_id: "b", run_on: Date.new(2026, 11, 12)) + run!(status: "skipped", job_id: "c", run_on: Date.new(2026, 11, 14)) + run!(status: "running", job_id: "d", run_on: Date.new(2026, 11, 15)) + + expect(described_class.latest_run_on(source)).to eq(Date.new(2026, 11, 12)) + end + end + + describe "#progress! and #heartbeat" do + it "stores the counts, and a heartbeat touches the run at most once per HEARTBEAT_EVERY", :aggregate_failures do + run = run!(updated_at: now - 1.hour) + + travel_to(now) { run.heartbeat(1, 10) } + first = run.reload.updated_at + travel_to(now + 10.seconds) { run.heartbeat } + second = run.reload.updated_at + travel_to(now + 31.seconds) do + run.heartbeat + run.progress!(unchanged: 4, malformed: 1) + end + + expect([ first, second ]).to eq([ now, now ]) + expect(run.reload.updated_at).to be >= now + 31.seconds + expect(run).to have_attributes(unchanged: 4, malformed: 1) + end + end +end diff --git a/spec/models/prices/refresh_end_to_end_spec.rb b/spec/models/prices/refresh_end_to_end_spec.rb new file mode 100644 index 00000000..9f0f207a --- /dev/null +++ b/spec/models/prices/refresh_end_to_end_spec.rb @@ -0,0 +1,54 @@ +require "rails_helper" + +# The refresh through the real catalog source and the real price source, over a stubbed Scryfall (spec 089 AC-4.5, AC-1.1, +# AC-1.7): the file's quotes become rows, and no request happens inside a transaction. +RSpec.describe "the daily price refresh over the real sources", type: :model do + let(:stamp) { "20261113100623" } + + def cards(prices_by_id) = prices_by_id.map { |id, prices| scryfall_card("id" => id, "prices" => prices) } + + around do |example| + example.run + ensure + FileUtils.rm_rf(Rails.configuration.x.catalog_download_dir) + end + + it "downloads once, applies the file's quotes dated by its name, and makes no request inside a transaction (AC-4.5)", :aggregate_failures do # rubocop:disable RSpec/ExampleLength -- one scenario + one = create(:catalog_entry, collectible_type: "mtg", external_key: "card-1") + two = create(:catalog_entry, collectible_type: "mtg", external_key: "card-2") + stub_scryfall(cards: cards("card-1" => { "usd" => "1.42", "usd_foil" => "3.10", "eur" => nil }, "card-2" => { "usd" => "0.05" }), stamp:) + connections = [ ApplicationRecord.lease_connection, PricesRecord.lease_connection ] + baselines = connections.map(&:open_transactions) + inside = [] + seen = 0 + callback = lambda do |request, _response| + seen += 1 + inside << request.uri.to_s if connections.zip(baselines).any? { |connection, baseline| connection.open_transactions > baseline } + end + WebMock.after_request(&callback) + + run = Prices::Refresh.new("scryfall").call + + expect(seen).to be >= 2 # the bulk-data index and the file + expect(inside).to be_empty + expect(run).to have_attributes(status: "finished", run_on: Date.new(2026, 11, 13), source_version: "default-cards-#{stamp}", added: 3, quotes_seen: 3) + expect(CurrentPrice.where(catalog_entry_id: one.id).order(:price_series_id).pluck(:cents)).to contain_exactly(142, 310) + expect(CurrentPrice.where(catalog_entry_id: two.id).pluck(:cents)).to eq([ 5 ]) + expect(PriceChange.pluck(:recorded_on).uniq).to eq([ Date.new(2026, 11, 13) ]) + ensure + WebMock::CallbackRegistry.callbacks.reject! { |entry| entry[:block].equal?(callback) } # only this example's, never VCR's + end + + it "changes nothing when the same file is read again (AC-1.7)", :aggregate_failures do + create(:catalog_entry, collectible_type: "mtg", external_key: "card-1") + stub_scryfall(cards: cards("card-1" => { "usd" => "1.42" }), stamp:) + Prices::Refresh.new("scryfall").call + before = [ CurrentPrice.pluck(:cents, :changed_on), PriceChange.pluck(:cents, :recorded_on) ] + + run = Prices::Refresh.new("scryfall").call + + expect(run).to have_attributes(status: "skipped", error: "default-cards-#{stamp} already applied") + expect(a_request(:get, /data\.scryfall\.io/)).to have_been_made.once # the file is not downloaded again + expect([ CurrentPrice.pluck(:cents, :changed_on), PriceChange.pluck(:cents, :recorded_on) ]).to eq(before) + end +end diff --git a/spec/models/prices/refresh_spec.rb b/spec/models/prices/refresh_spec.rb new file mode 100644 index 00000000..2f13e3d8 --- /dev/null +++ b/spec/models/prices/refresh_spec.rb @@ -0,0 +1,615 @@ +require "rails_helper" + +RSpec.describe Prices::Refresh, type: :model do + def refresh(job_id: "job-1") = described_class.new("fake_source", job_id:) + def last_run = PriceRefreshRun.order(:id).last + + describe "before it applies anything" do + it "skips with a reason and no date when the catalog has no printing of the type, asking the catalog's source nothing (AC-1.5)", :aggregate_failures do + use_widget_sources(records: [ widget_quote("w-1", 100) ]) + allow(Catalog).to receive(:source_for).and_call_original + + run = refresh.call + + expect(Catalog).not_to have_received(:source_for) + expect(run).to have_attributes(status: "skipped", run_on: nil) + expect(run.error).to include("no printing") + expect([ CurrentPrice.count, PriceChange.count ]).to eq([ 0, 0 ]) + end + + it "syncs the source before a run exists, so a source that fails verification fails the job with no run (AC-1.8)", :aggregate_failures do + use_widget_sources + Prices.sources["fake_source"] = "WidgetPrices" + broken = Class.new(FakePriceSource) { def self.key = "BAD KEY" } + stub_const("WidgetPrices", broken) + + expect { refresh.call }.to raise_error(Prices::Sources::ConfigurationError, /key/) + expect(PriceRefreshRun.count).to eq(0) + expect { described_class.new("nobody").call }.to raise_error(ArgumentError, /unknown price source/) + expect(PriceRefreshRun.count).to eq(0) + end + + it "syncs the source and its series first, so their rows exist when the run opens (AC-1.8)", :aggregate_failures do + use_widget_sources + widget_entry("w-1") + + expect { refresh.call }.to raise_error(Prices::Refresh::GuardError, /no valid quotes/) # a file with no quote is a GuardError (AC-3.5) + + expect(PriceSource.pluck(:key)).to eq([ "fake_source" ]) + expect(PriceSeries.count).to eq(1) + expect(last_run.price_source.key).to eq("fake_source") + end + + it "skips a file already applied for the same version and languages, copying its date (AC-1.2; the end-to-end example proves nothing is downloaded again)", :aggregate_failures do + use_widget_sources(version: "v1") + widget_entry("w-1") + PriceRefreshRun.create!(price_source: price_source_row, status: "finished", source_version: "v1", languages: "en", + run_on: Date.new(2026, 11, 13), started_at: 1.hour.ago) + + run = refresh.call + + expect(run).to have_attributes(status: "skipped", error: "v1 already applied", run_on: Date.new(2026, 11, 13), + source_version: "v1", languages: "en") + end + + it "applies a file of the same version when the language set differs (AC-1.3)", :aggregate_failures do + use_widget_sources(version: "all-1", languages: %w[de en], records: [ widget_quote("w-1", 100) ]) + widget_entry("w-1") + PriceRefreshRun.create!(price_source: price_source_row, status: "finished", source_version: "all-1", languages: "en,ja", + run_on: Date.new(2026, 11, 13), started_at: 1.hour.ago) + + run = refresh.call + + expect(run).to have_attributes(status: "finished", languages: "de,en") + end + + it "skips a file older than the latest finished or failed run (AC-1.4)", :aggregate_failures do + use_widget_sources(reported_on: Date.new(2026, 11, 12), records: [ widget_quote("w-1", 100) ]) + widget_entry("w-1") + PriceRefreshRun.create!(price_source: price_source_row, status: "failed", source_version: "other", languages: "en", + run_on: Date.new(2026, 11, 13), started_at: 2.hours.ago) + + older = refresh.call + + expect(older).to have_attributes(status: "skipped", run_on: Date.new(2026, 11, 12)) + expect(older.error).to include("2026-11-13") + expect([ CurrentPrice.count, PriceChange.count ]).to eq([ 0, 0 ]) + end + + it "skips a file older than a finished run, and applies another version of the finished run's own day (AC-1.4)", :aggregate_failures do + widget_entry("w-1") + PriceRefreshRun.create!(price_source: price_source_row, status: "finished", source_version: "v1", languages: "en", + run_on: Date.new(2026, 11, 13), started_at: 2.hours.ago) + use_widget_sources(reported_on: Date.new(2026, 11, 12), records: [ widget_quote("w-1", 100) ], version: "old") + older = refresh.call + use_widget_sources(reported_on: Date.new(2026, 11, 13), records: [ widget_quote("w-1", 100) ], version: "v2") + + same_day = refresh.call + + expect(older.status).to eq("skipped") + expect(same_day).to have_attributes(status: "finished", run_on: Date.new(2026, 11, 13)) + end + + it "ends the run failed, with its error, when the source's file has no date, and nothing is written (AC-1.9)", :aggregate_failures do + use_widget_sources(reported_on: Prices::Sources::IntegrityError.new("no date")) + widget_entry("w-1") + + expect { refresh.call }.to raise_error(Prices::Sources::IntegrityError) + expect(last_run).to have_attributes(status: "failed", run_on: nil) + expect(last_run.error).to eq("Prices::Sources::IntegrityError: no date") + end + + it "records a skipped run, 'already running', for a run with recent progress, and writes nothing (AC-4.1)", :aggregate_failures do + use_widget_sources(records: [ widget_quote("w-1", 100) ]) + widget_entry("w-1") + PriceRefreshRun.create!(price_source: price_source_row, status: "running", job_id: "other-job", started_at: 1.minute.ago) + + run = refresh(job_id: "job-2").call + + expect(run).to have_attributes(status: "skipped", error: "already running") + expect([ CurrentPrice.count, PriceChange.count ]).to eq([ 0, 0 ]) + end + + it "closes a run left behind as interrupted and goes ahead (AC-4.2)", :aggregate_failures do + use_widget_sources(records: [ widget_quote("w-1", 100) ]) + widget_entry("w-1") + left = PriceRefreshRun.create!(price_source: price_source_row, status: "running", job_id: "dead-job", + started_at: 1.hour.ago, updated_at: 1.hour.ago) + + run = refresh(job_id: "job-2").call + + expect(left.reload).to have_attributes(status: "failed", error: "interrupted") + expect(run.status).to eq("finished") + end + + it "touches the run while the download reports progress, at most once per HEARTBEAT_EVERY (AC-4.7)", :aggregate_failures do + use_widget_sources(records: [ widget_quote("w-1", 100) ]) + widget_entry("w-1") + touches = [] + during = nil + stub_const("WidgetCatalog", Class.new(ReportingCatalogSource) do + define_method(:initialize) { |*| super(version: "v1", languages: [ "en" ]) } + define_method(:download) { |version, dir:| before = touches.size; super(version, dir:).tap { during = touches.size - before } } + end) + callback = ->(*, payload) { touches << payload[:sql] if payload[:sql].to_s.match?(/UPDATE "price_refresh_runs" SET "updated_at" = [^,]+ WHERE/) } + + ActiveSupport::Notifications.subscribed(callback, "sql.active_record") { refresh.call } + + expect(during).to eq(1) # ReportingCatalogSource reports twice within one instant while downloading: one touch, which sets only updated_at + end + end + + describe "judging quotes" do + def rows = CurrentPrice.pluck(:catalog_entry_id, :cents, :reported_on, :changed_on) + def history = PriceChange.order(:catalog_entry_id, :recorded_on).pluck(:catalog_entry_id, :recorded_on, :cents) + + it "adds a price no one has, with a history row for the run's date, and finishes the run (AC-1.1, AC-2.1)", :aggregate_failures do + use_widget_sources(records: [ widget_quote("w-1", 142) ], reported_on: Date.new(2026, 11, 13)) + entry = widget_entry("w-1") + + run = refresh.call + + expect(run).to have_attributes(status: "finished", run_on: Date.new(2026, 11, 13), source_version: "v1", languages: "en", + job_id: "job-1", added: 1, quotes_seen: 1, unchanged: 0, repriced: 0) + expect(run.started_at).to be_present + expect(run.finished_at).to be_present + expect(rows).to eq([ [ entry.id, 142, Date.new(2026, 11, 13), Date.new(2026, 11, 13) ] ]) + expect(history).to eq([ [ entry.id, Date.new(2026, 11, 13), 142 ] ]) + end + + it "writes nothing for an unchanged price, and the current price's dates stay (AC-2.2)", :aggregate_failures do + use_widget_sources(records: [ widget_quote("w-1", 100) ]) + entry = widget_entry("w-1") + seed_price(entry, 100, on: Date.new(2026, 11, 12)) + + run = refresh.call + + expect(run).to have_attributes(unchanged: 1, repriced: 0, added: 0, quotes_seen: 1) + expect(rows).to eq([ [ entry.id, 100, Date.new(2026, 11, 12), Date.new(2026, 11, 12) ] ]) + expect(history.size).to eq(1) + end + + it "updates a changed price and adds one history row, leaving the earlier ones (AC-2.3)", :aggregate_failures do + use_widget_sources(records: [ widget_quote("w-1", 150) ]) + entry = widget_entry("w-1") + seed_price(entry, 100, on: Date.new(2026, 11, 12)) + + run = refresh.call + + expect(run).to have_attributes(repriced: 1, added: 0, unchanged: 0) + expect(rows).to eq([ [ entry.id, 150, Date.new(2026, 11, 13), Date.new(2026, 11, 13) ] ]) + expect(history).to eq([ [ entry.id, Date.new(2026, 11, 12), 100 ], [ entry.id, Date.new(2026, 11, 13), 150 ] ]) + end + + it "judges each series of a printing on its own (AC-2.6)", :aggregate_failures do + series = [ PriceRefreshHelpers::PLAIN, + Prices::Sources::Series.new(finish: "foil", currency: "USD", kind: "market"), + Prices::Sources::Series.new(finish: "etched", currency: "USD", kind: "market") ] + use_widget_sources(series:, records: [ widget_quote("w-1", 100, series: series[0]), widget_quote("w-1", 300, series: series[1]), widget_quote("w-1", 500, series: series[2]) ]) + entry = widget_entry("w-1") + seed_price(entry, 100, series: series[0]) # unchanged + seed_price(entry, 250, series: series[1]) # changed + + run = refresh.call + + expect(run).to have_attributes(quotes_seen: 3, unchanged: 1, repriced: 1, added: 1) + expect(CurrentPrice.count).to eq(3) + end + + it "skips a printing the catalog doesn't have, counting it once however many series it has (AC-2.5)", :aggregate_failures do + series = [ PriceRefreshHelpers::PLAIN, Prices::Sources::Series.new(finish: "foil", currency: "USD", kind: "market") ] + use_widget_sources(series:, records: [ widget_quote("ghost", 1, series: series[0]), widget_quote("ghost", 2, series: series[1]), widget_quote("w-1", 100) ]) + widget_entry("w-1") + + run = refresh.call + + expect(run).to have_attributes(unknown_printings: 1, quotes_seen: 1, added: 1) + expect(CurrentPrice.count).to eq(1) + end + + it "judges the first of two quotes for one printing and series, ignoring the second (AC-2.10)", :aggregate_failures do + use_widget_sources(records: [ widget_quote("w-1", 100), widget_quote("w-2", 5), widget_quote("w-1", 999) ]) + entry = widget_entry("w-1") + widget_entry("w-2") + + run = refresh.call + + expect(run).to have_attributes(quotes_seen: 2, added: 2) + expect(CurrentPrice.find_by(catalog_entry_id: entry.id).cents).to eq(100) + end + + it "ignores a second quote for a printing and series already judged in an earlier batch (AC-2.10)", :aggregate_failures do + stub_const("Prices::Refresh::BATCH_SIZE", 1) + use_widget_sources(records: [ widget_quote("w-1", 100), widget_quote("w-2", 5), widget_quote("w-1", 999) ]) + entry = widget_entry("w-1") + widget_entry("w-2") + + run = refresh.call + + expect(run).to have_attributes(quotes_seen: 2, added: 2) + expect(CurrentPrice.find_by(catalog_entry_id: entry.id).cents).to eq(100) + expect(PriceChange.where(catalog_entry_id: entry.id).pluck(:cents)).to eq([ 100 ]) + end + + it "ignores a quote repeated while its printing is pending, so a file that repeats one cannot grow the batch without limit (AC-2.10)", :aggregate_failures do + stub_const("Prices::Refresh::BATCH_SIZE", 1) + run_refresh = refresh + waiting = nil + peek = -> { waiting = run_refresh.instance_variable_get(:@pending).transform_values(&:size) } # the batch's memory, read mid-file + use_widget_sources(records: [ *Array.new(5) { |index| widget_quote("w-1", 100 + index) }, peek, widget_quote("w-2", 5) ]) + entry = widget_entry("w-1") + widget_entry("w-2") + + run = run_refresh.call + + expect(waiting).to eq({ "w-1" => 1 }) + expect(run).to have_attributes(status: "finished", quotes_seen: 2, added: 2) + expect(CurrentPrice.find_by(catalog_entry_id: entry.id).cents).to eq(100) + end + + it "replaces a history row of the same day when a second file of that day differs (AC-2.7)", :aggregate_failures do + use_widget_sources(records: [ widget_quote("w-1", 150) ], version: "v2") + entry = widget_entry("w-1") + seed_price(entry, 100, on: Date.new(2026, 11, 13)) + + refresh.call + + expect(history).to eq([ [ entry.id, Date.new(2026, 11, 13), 150 ] ]) + end + + it "gives every written row the file's date, whatever the clock says (AC-1.6)", :aggregate_failures do + use_widget_sources(records: [ widget_quote("w-1", 100) ], reported_on: Date.new(2026, 11, 13)) + widget_entry("w-1") + + travel_to(Time.utc(2026, 12, 25, 12)) { refresh.call } + + expect(CurrentPrice.pluck(:reported_on, :changed_on)).to eq([ [ Date.new(2026, 11, 13), Date.new(2026, 11, 13) ] ]) + expect(PriceChange.pluck(:recorded_on)).to eq([ Date.new(2026, 11, 13) ]) + end + + it "writes the same rows when the job is performed twice on one file (AC-1.7)", :aggregate_failures do + use_widget_sources(records: [ widget_quote("w-1", 100), widget_quote("w-2", 200) ]) + widget_entry("w-1") + widget_entry("w-2") + + refresh.call + first = [ CurrentPrice.pluck(:catalog_entry_id, :cents, :changed_on).sort, PriceChange.pluck(:catalog_entry_id, :cents, :recorded_on).sort ] + second = refresh.call + + expect(second).to have_attributes(status: "skipped", error: "v1 already applied") + expect([ CurrentPrice.pluck(:catalog_entry_id, :cents, :changed_on).sort, PriceChange.pluck(:catalog_entry_id, :cents, :recorded_on).sort ]).to eq(first) + expect(PriceRefreshRun.count).to eq(2) + end + + it "keeps rows keyed by the entry's id, in the prices database (AC-2.8)", :aggregate_failures do + use_widget_sources(records: [ widget_quote("w-1", 100) ]) + entry = widget_entry("w-1") + + refresh.call + + expect(CurrentPrice.first.catalog_entry_id).to eq(entry.id) + expect([ CurrentPrice, PriceChange, PriceRefreshRun ].map { it.connection_db_config.name }.uniq).to eq([ "prices" ]) + end + + it "raises, fails the run and keeps earlier batches when a quote's series was not declared (AC-2.11)", :aggregate_failures do + undeclared = Prices::Sources::Series.new(finish: "gold", currency: "USD", kind: "market") + stub_const("Prices::Refresh::BATCH_SIZE", 1) + use_widget_sources(records: [ widget_quote("w-1", 100), widget_quote("w-2", 5, series: undeclared) ]) + widget_entry("w-1") + widget_entry("w-2") + + expect { refresh.call }.to raise_error(Prices::Sources::ConfigurationError, /gold/) + + expect(last_run).to have_attributes(status: "failed", added: 1) + expect(CurrentPrice.count).to eq(1) + end + + it "logs a malformed record's key and error, and counts it (AC-2.12)", :aggregate_failures do + use_widget_sources(records: [ widget_malformed("w-1", series: PriceRefreshHelpers::PLAIN), widget_quote("w-2", 5) ]) + widget_entry("w-1") + widget_entry("w-2") + allow(Rails.logger).to receive(:warn) + + run = refresh.call + + expect(run.malformed).to eq(1) + expect(Rails.logger).to have_received(:warn).with(/prices\.refresh skipped malformed .*w-1.*unreadable/) + end + + it "touches the run after every batch, even one that writes nothing (AC-4.7)", :aggregate_failures do + stub_const("Prices::Refresh::BATCH_SIZE", 2) + entries = Array.new(5) { |index| widget_entry("w-#{index}") } + entries.each { |entry| seed_price(entry, 100) } + use_widget_sources(records: entries.map { widget_quote(it.external_key, 100) }) # all unchanged: no batch writes a price + progress = [] + callback = ->(*, payload) { progress << payload[:sql] if payload[:sql].to_s.start_with?('UPDATE "price_refresh_runs" SET') && payload[:sql].include?('"quotes_seen"') } + + ActiveSupport::Notifications.subscribed(callback, "sql.active_record") { refresh.call } + + expect(CurrentPrice.pluck(:cents).uniq).to eq([ 100 ]) # nothing was written + expect(progress.size).to eq(3) # a progress write per batch (2, 2 and 1 printings): only they put quotes_seen in an UPDATE's SET + end + + it "flushes unreadable records by the batch size too, so a file of mostly unreadable lines isn't held whole (spec 089 decision 6)" do + stub_const("Prices::Refresh::BATCH_SIZE", 2) + keys = Array.new(5) { |index| "m-#{index + 1}" } + use_widget_sources(records: keys.map { widget_malformed(it, series: PriceRefreshHelpers::PLAIN) } + [ widget_quote("w-1", 100) ]) + (keys + [ "w-1" ]).each { widget_entry(it) } + lookups = 0 + callback = ->(*, payload) { lookups += 1 if payload[:sql].to_s.match?(/FROM "catalog_entries".*external_key/m) && payload[:sql].start_with?("SELECT") } + + ActiveSupport::Notifications.subscribed(callback, "sql.active_record") { refresh.call } + + expect(lookups).to eq(3) # after two unreadable records, after two more, and the final flush (the quote and the fifth) + end + end + + describe "absences" do + def seed_candidates(count, prefix: "keep") = Array.new(count) { |index| widget_entry("#{prefix}-#{index}").tap { seed_price(it, 100) } } + def keep_quotes(count, prefix: "keep") = Array.new(count) { |index| widget_quote("#{prefix}-#{index}", 100) } + def blank_history(entry) = PriceChange.where(catalog_entry_id: entry.id, cents: nil).pluck(:recorded_on) + + it "removes a price the file leaves out and writes a blank history row for the run's date (AC-2.4)", :aggregate_failures do + kept = seed_candidates(10) + gone = widget_entry("gone").tap { seed_price(it, 100) } + use_widget_sources(records: keep_quotes(10)) + + run = refresh.call + + expect(run).to have_attributes(status: "finished", removed: 1, removals_withheld: 0, unchanged: 10) + expect(CurrentPrice.where(catalog_entry_id: gone.id)).to be_empty + expect(blank_history(gone)).to eq([ Date.new(2026, 11, 13) ]) + expect(CurrentPrice.where(catalog_entry_id: kept.map(&:id)).count).to eq(10) + end + + it "treats a price that comes back as an added quote with its own history row (AC-2.4)", :aggregate_failures do + seed_candidates(10) + gone = widget_entry("gone").tap { seed_price(it, 100) } + use_widget_sources(records: keep_quotes(10), version: "v1") + refresh.call + use_widget_sources(records: keep_quotes(10) + [ widget_quote("gone", 120) ], version: "v2", reported_on: Date.new(2026, 11, 14)) + + run = refresh.call + + expect(run).to have_attributes(added: 1, unchanged: 10) + expect(CurrentPrice.find_by(catalog_entry_id: gone.id).cents).to eq(120) + expect(PriceChange.where(catalog_entry_id: gone.id).order(:recorded_on).pluck(:recorded_on, :cents)).to eq([ [ Date.new(2026, 11, 12), 100 ], [ Date.new(2026, 11, 13), nil ], [ Date.new(2026, 11, 14), 120 ] ]) + end + + it "removes a price of a printing absent from the file altogether, and of a retired entry (AC-2.9)", :aggregate_failures do + seed_candidates(20) + absent = widget_entry("absent").tap { seed_price(it, 100) } + retired = widget_entry("retired", retired_at: 1.day.ago).tap { seed_price(it, 100) } + use_widget_sources(records: keep_quotes(20)) + + run = refresh.call + + expect(run).to have_attributes(removed: 2) + expect(CurrentPrice.where(catalog_entry_id: [ absent.id, retired.id ])).to be_empty + end + + it "leaves a price whose printing is in a language the run did not read, or the primary lacks, and does not count it (AC-3.7)", :aggregate_failures do + seed_candidates(10) + japanese = widget_entry("ja-1", language: "ja").tap { seed_price(it, 100) } + CurrentPrice.insert_all!([ { catalog_entry_id: 9_999_999, price_source_id: price_source_row.id, price_series_id: series_row.id, cents: 5, reported_on: Date.new(2026, 11, 12), changed_on: Date.new(2026, 11, 12) } ]) + use_widget_sources(records: keep_quotes(10)) + + run = refresh.call + + expect(run).to have_attributes(removed: 0, removals_withheld: 0) + expect(CurrentPrice.where(catalog_entry_id: [ japanese.id, 9_999_999 ]).count).to eq(2) + end + + it "leaves a price of a series the source no longer declares, and does not count it (AC-3.7)", :aggregate_failures do + retired_series = Prices::Sources::Series.new(finish: "gold", currency: "USD", kind: "market") + seed_candidates(10) + legacy = widget_entry("legacy") + seed_price(legacy, 100, series: retired_series) # the source declares only the plain series today + use_widget_sources(records: keep_quotes(10)) + + run = refresh.call + + expect(run).to have_attributes(status: "finished", removed: 0, removals_withheld: 0) + expect(CurrentPrice.where(catalog_entry_id: legacy.id, price_series_id: series_row(retired_series).id).count).to eq(1) + expect(PriceChange.where(catalog_entry_id: legacy.id, cents: nil)).to be_empty + end + + it "does not remove a price whose quote was unreadable, and the printing's other series are judged as usual (AC-3.1)", :aggregate_failures do + other = Prices::Sources::Series.new(finish: "foil", currency: "USD", kind: "market") + seed_candidates(10) + entry = widget_entry("w-x") + seed_price(entry, 100) + seed_price(entry, 100, series: other) + use_widget_sources(series: [ PriceRefreshHelpers::PLAIN, other ], records: keep_quotes(10) + [ widget_malformed("w-x", series: PriceRefreshHelpers::PLAIN) ]) + + run = refresh.call + + expect(run).to have_attributes(malformed: 1, removals_withheld: 1, removed: 1) + expect(CurrentPrice.where(catalog_entry_id: entry.id, price_series_id: series_row.id).count).to eq(1) # kept + expect(CurrentPrice.where(catalog_entry_id: entry.id, price_series_id: series_row(other).id)).to be_empty # absent and trusted + expect(blank_history(entry)).to eq([ Date.new(2026, 11, 13) ]) + end + + it "removes none of a printing's prices when its line could be read only as far as its key (AC-3.2)", :aggregate_failures do + other = Prices::Sources::Series.new(finish: "foil", currency: "USD", kind: "market") + seed_candidates(10) + entry = widget_entry("w-x") + seed_price(entry, 100) + seed_price(entry, 100, series: other) + use_widget_sources(series: [ PriceRefreshHelpers::PLAIN, other ], records: keep_quotes(10) + [ widget_malformed("w-x") ]) + + run = refresh.call + + expect(run).to have_attributes(removed: 0, removals_withheld: 2) + expect(CurrentPrice.where(catalog_entry_id: entry.id).count).to eq(2) + end + + it "removes nothing at all when a line names no printing, applying the quotes around it (AC-3.3)", :aggregate_failures do + seed_candidates(10) + gone = widget_entry("gone").tap { seed_price(it, 100) } + new_entry = widget_entry("fresh") + use_widget_sources(records: keep_quotes(10).first(5) + [ widget_malformed ] + keep_quotes(10).last(5) + [ widget_quote("fresh", 77) ]) + + run = refresh.call + + expect(run).to have_attributes(status: "finished", removed: 0, removals_withheld: 1, added: 1) + expect(CurrentPrice.where(catalog_entry_id: gone.id).count).to eq(1) + expect(CurrentPrice.where(catalog_entry_id: new_entry.id).count).to eq(1) + end + + it "fails the run, with the reader's error, when the file is cut short; earlier batches stand, nothing is removed, and a whole file of the day finishes without a duplicate row (AC-3.4)", :aggregate_failures do # rubocop:disable RSpec/ExampleLength -- two runs of one scenario + stub_const("Prices::Refresh::BATCH_SIZE", 2) + seed_candidates(10) + gone = widget_entry("gone").tap { seed_price(it, 100) } + fresh = Array.new(4) { |index| widget_entry("new-#{index}") } + use_widget_sources(records: keep_quotes(10) + fresh.map { widget_quote(it.external_key, 7) } + [ -> { raise Zlib::GzipFile::Error, "unexpected end of file" } ]) + + expect { refresh.call }.to raise_error(Zlib::GzipFile::Error) + + expect(last_run).to have_attributes(status: "failed", removed: 0) + expect(last_run.error).to include("unexpected end of file") + expect(CurrentPrice.where(catalog_entry_id: gone.id).count).to eq(1) + written = CurrentPrice.where(catalog_entry_id: fresh.map(&:id)).count + expect(written).to eq(2) # the batches flushed before the error: the last two printings were still pending + + use_widget_sources(records: keep_quotes(10) + fresh.map { widget_quote(it.external_key, 7) }, version: "v1") + second = refresh.call + + expect(second.status).to eq("finished") + expect(CurrentPrice.where(catalog_entry_id: fresh.map(&:id)).count).to eq(4) + expect(PriceChange.where(catalog_entry_id: fresh.map(&:id)).count).to eq(4) + end + + it "fails the run, 'no valid quotes', and writes nothing when the file yields no quote (AC-3.5)", :aggregate_failures do + seed_candidates(10) + use_widget_sources(records: [ widget_malformed ]) + + expect { refresh.call }.to raise_error(Prices::Refresh::GuardError, /no valid quotes/) + + expect(last_run).to have_attributes(status: "failed", removed: 0) + expect(CurrentPrice.count).to eq(10) + expect(PriceChange.count).to eq(10) + end + + it "finishes a file whose quotes are all for printings the catalog lacks, counting them and writing or removing nothing (AC-3.5)", :aggregate_failures do + widget_entry("w-1") # the catalog has a printing of the type, so the run isn't skipped + use_widget_sources(records: [ widget_quote("ghost-1", 1), widget_quote("ghost-2", 1) ]) + + run = refresh.call + + expect(run).to have_attributes(status: "finished", unknown_printings: 2, quotes_seen: 0, removed: 0) + expect([ CurrentPrice.count, PriceChange.count ]).to eq([ 0, 0 ]) + end + + it "refuses every removal and fails the run when absences exceed the share, counting them as withheld (AC-3.6)", :aggregate_failures do + seed_candidates(10) + gone = Array.new(2) { |index| widget_entry("gone-#{index}").tap { seed_price(it, 100) } } + repriced = seed_candidates(1, prefix: "moved").first + use_widget_sources(records: keep_quotes(10) + [ widget_quote("moved-0", 150) ]) # 2 of 13 candidates absent: over 1/10 + + expect { refresh.call }.to raise_error(Prices::Refresh::GuardError, /2 of 13/) + + expect(last_run).to have_attributes(status: "failed", removed: 0, removals_withheld: 2, repriced: 1) + expect(CurrentPrice.where(catalog_entry_id: gone.map(&:id)).count).to eq(2) + expect(CurrentPrice.find_by(catalog_entry_id: repriced.id).cents).to eq(150) # changes already written stand + expect(PriceChange.where(catalog_entry_id: repriced.id).count).to eq(2) + end + + it "removes absences exactly at the share (AC-3.6)", :aggregate_failures do + seed_candidates(9) + gone = widget_entry("gone").tap { seed_price(it, 100) } # 1 of 10 candidates: exactly 1/10 + use_widget_sources(records: keep_quotes(9)) + + run = refresh.call + + expect(run).to have_attributes(status: "finished", removed: 1) + expect(CurrentPrice.where(catalog_entry_id: gone.id)).to be_empty + end + + it "reads a failed file again on the next run (AC-3.8)", :aggregate_failures do + seed_candidates(10) + widget_entry("gone").tap { seed_price(it, 100) } + widget_entry("gone-2").tap { seed_price(it, 100) } + use_widget_sources(records: keep_quotes(10)) + expect { refresh.call }.to raise_error(Prices::Refresh::GuardError) + + expect { refresh.call }.to raise_error(Prices::Refresh::GuardError) # read again, not skipped as applied + + expect(PriceRefreshRun.where(status: "failed").count).to eq(2) + expect(PriceRefreshRun.where(status: "skipped")).to be_empty + end + + it "touches the run for each slice of the removal walk (AC-4.7)" do + stub_const("Prices::Refresh::BATCH_SIZE", 2) + seed_candidates(10) + widget_entry("gone").tap { seed_price(it, 100) } + use_widget_sources(records: keep_quotes(10)) + progress = [] + callback = ->(*, payload) { progress << payload[:sql] if payload[:sql].to_s.start_with?('UPDATE "price_refresh_runs" SET') && payload[:sql].include?('"updated_at"') } + + ActiveSupport::Notifications.subscribed(callback, "sql.active_record") { refresh.call } + + progress.reject! { it.include?('"status"') } # the run's end is not a touch + walks = progress.drop(progress.rindex { it.include?('"quotes_seen"') } + 1) # only the read's flushes write quotes_seen; what follows is the two walks + expect(walks.size).to eq(12) # eleven candidates in slices of two: six slices in each of the two walks + end + + it "leaves both or neither of a removal batch's current price and blank history row when the batch fails (AC-3.9)", :aggregate_failures do + seed_candidates(10) + gone = widget_entry("gone").tap { seed_price(it, 100) } + use_widget_sources(records: keep_quotes(10)) + PricesRecord.lease_connection.execute("CREATE TRIGGER no_blank_history BEFORE INSERT ON price_changes WHEN NEW.cents IS NULL BEGIN SELECT RAISE(ABORT, 'injected'); END") + + expect { refresh.call }.to raise_error(ActiveRecord::StatementInvalid, /injected/) + + expect(CurrentPrice.where(catalog_entry_id: gone.id).count).to eq(1) + expect(blank_history(gone)).to be_empty + + PricesRecord.lease_connection.execute("DROP TRIGGER no_blank_history") + run = refresh.call + + expect(run).to have_attributes(status: "finished", removed: 1) + expect(CurrentPrice.where(catalog_entry_id: gone.id)).to be_empty + expect(blank_history(gone)).to eq([ Date.new(2026, 11, 13) ]) + end + end + + describe "transactions (AC-4.4)" do + it "writes in batches of BATCH_SIZE printings inside savepoint spans, the run row in spans of its own, and reads the primary outside them", :aggregate_failures do # rubocop:disable RSpec/ExampleLength -- one subscribed run, one set of assertions + stub_const("Prices::Refresh::BATCH_SIZE", 2) + entries = Array.new(5) { |index| widget_entry("w-#{index}") } + use_widget_sources(records: entries.map { widget_quote(it.external_key, 100) }) + entries.each { |entry| seed_price(entry, 90) } # five changes, so every batch writes both tables + prices = PricesRecord.lease_connection + baseline = prices.open_transactions # the example's own wrapping transaction + stack = [] # open spans on the prices connection: each a list of its statements + spans = [] + outside = [] + primary_inside = [] + callback = lambda do |_name, _start, _finish, _id, payload| + sql = payload[:sql].to_s + if payload[:connection].equal?(prices) + if sql.start_with?("SAVEPOINT") then stack.push([]) + elsif sql.start_with?("RELEASE SAVEPOINT", "ROLLBACK TO SAVEPOINT") then spans << stack.pop + elsif stack.any? then stack.last << sql + else outside << sql + end + elsif prices.open_transactions > baseline # not stack.any?: Rails opens a savepoint lazily, at the transaction's first statement + primary_inside << sql + end + end + + ActiveSupport::Notifications.subscribed(callback, "sql.active_record") { refresh.call } + + write = ->(sql, table) { sql.match?(/\A\s*(INSERT|UPDATE|DELETE)\b.*?\b#{table}\b/m) } + data = ->(sql) { write.call(sql, "current_prices") || write.call(sql, "price_changes") } + batch_spans = spans.select { |statements| statements.any?(&data) } + expect(outside.grep(/\A\s*(INSERT|UPDATE|DELETE)/)).to all(include("price_refresh_runs")) # no price write outside a span + expect(batch_spans.size).to eq(3) # five printings in batches of two: 2, 2 and 1 + expect(batch_spans.flat_map { |statements| statements.select { write.call(it, "price_refresh_runs") } }).to be_empty # the run row is not written inside a batch + expect(batch_spans.map { |statements| statements.find { write.call(it, "current_prices") }.scan(/\),\s*\(/).size + 1 }).to eq([ 2, 2, 1 ]) # tuples of each batch's insert + expect(primary_inside).to be_empty + expect(CurrentPrice.count).to eq(5) + end + end +end diff --git a/spec/models/prices/refresh_timing_spec.rb b/spec/models/prices/refresh_timing_spec.rb new file mode 100644 index 00000000..b6a1e485 --- /dev/null +++ b/spec/models/prices/refresh_timing_spec.rb @@ -0,0 +1,50 @@ +require "rails_helper" + +# A day's file at realistic size (spec 089 AC-5.6): 110,000 cards, about 300,000 quotes, entries in the test database. +# A measurement, not a gate; the numbers are recorded in verification.md (#366 replaces them with the run's own). +RSpec.describe "the daily price refresh's time", :timing, type: :model do + let(:count) { 110_000 } + let(:stamp) { "20261113100623" } + + # Seconds one block takes. Not Benchmark: it is a bundled gem on this Ruby and the bundle doesn't list it. + def realtime + started = Process.clock_gettime(Process::CLOCK_MONOTONIC) + yield + Process.clock_gettime(Process::CLOCK_MONOTONIC) - started + end + + def card_lines(shift) + Array.new(count) do |index| + scryfall_card("id" => "card-#{index}", "prices" => { "usd" => format("%.2f", 1 + index / 100.0 + shift.call(index)), "usd_foil" => (index.even? ? format("%.2f", 2 + index / 100.0) : nil), "eur" => format("%.2f", 1 + index / 120.0) }) + end + end + + it "applies a first run and then a typical day", :aggregate_failures do # rubocop:disable RSpec/ExampleLength -- one measured scenario + set = create(:catalog_set, collectible_type: "mtg") + identity = create(:catalog_identity, collectible_type: "mtg") + now = Time.current + Array.new(count) do |index| + { collectible_type: "mtg", external_key: "card-#{index}", catalog_set_id: set.id, catalog_identity_id: identity.id, number: index.to_s, language: "en", + name: "Card #{index}", kind: "card", content_digest: "d", created_at: now, updated_at: now } + end.each_slice(1_000) { |rows| Catalog::Entry.insert_all(rows) } # one statement per 1,000: a single one would pass SQLite's statement size + timings = {} + + stub_scryfall(cards: card_lines(->(_) { 0 }), stamp: stamp) + timings[:first_run] = realtime { Prices::Refresh.new("scryfall").call } + first = PriceRefreshRun.order(:id).last + + next_stamp = "20261114100623" + stub_scryfall(cards: card_lines(->(index) { (index % 6).zero? ? 0.5 : 0 }), stamp: next_stamp) # about 17 percent change + timings[:typical_day] = realtime { Prices::Refresh.new("scryfall").call } + second = PriceRefreshRun.order(:id).last + + # rubocop:disable RSpec/Output -- the measurement is the point of this example + puts "price refresh timing: #{timings.transform_values { it.round(1) }.inspect}, first #{first.attributes.slice(*PriceRefreshRun::COUNTS.map(&:to_s)).inspect}, second #{second.attributes.slice(*PriceRefreshRun::COUNTS.map(&:to_s)).inspect}" + # rubocop:enable RSpec/Output + expect(first.status).to eq("finished") + expect(second).to have_attributes(status: "finished") + expect(second.repriced).to be > 0 + ensure + FileUtils.rm_rf(Rails.configuration.x.catalog_download_dir) + end +end diff --git a/spec/models/prices_record_spec.rb b/spec/models/prices_record_spec.rb index 54388d76..5382ccc2 100644 --- a/spec/models/prices_record_spec.rb +++ b/spec/models/prices_record_spec.rb @@ -81,7 +81,9 @@ def series!(finish = "plain") = insert("INSERT INTO price_series (finish, curren "price_series" => %w[id finish currency kind created_at updated_at], "current_prices" => %w[catalog_entry_id price_source_id price_series_id cents reported_on changed_on], "price_changes" => %w[catalog_entry_id price_source_id price_series_id recorded_on cents], - "price_refresh_runs" => %w[id price_source_id run_on source_version status error started_at finished_at created_at updated_at] + # spec 089 decision 2 added job_id, languages and the counts after spec 081's columns (`repriced` is v0.4.3's name) + "price_refresh_runs" => %w[id price_source_id run_on source_version status error started_at finished_at created_at updated_at + job_id languages quotes_seen unchanged repriced added removed removals_withheld unknown_printings malformed] } keys = { "current_prices" => %w[catalog_entry_id price_source_id price_series_id], "price_changes" => %w[catalog_entry_id price_source_id price_series_id recorded_on] } diff --git a/spec/price_source_core_spec.rb b/spec/price_source_core_spec.rb index 96fec3d0..fff0573b 100644 --- a/spec/price_source_core_spec.rb +++ b/spec/price_source_core_spec.rb @@ -5,7 +5,7 @@ RSpec.describe "the price source contract is core", type: :model do let(:matcher) { /mtg|magic|scryfall|tcgplayer|cardmarket/i } let(:core_files) do - %w[app/models/prices.rb app/models/prices/**/*.rb app/models/price_source.rb app/models/price_series.rb] + %w[app/models/prices.rb app/models/prices/**/*.rb app/models/price_source.rb app/models/price_series.rb app/models/current_price.rb app/models/price_change.rb app/models/price_refresh_run.rb app/jobs/prices/**/*.rb] .flat_map { |pattern| Rails.root.glob(pattern) } end diff --git a/spec/prices_core_spec.rb b/spec/prices_core_spec.rb index 4857b14d..720c47e6 100644 --- a/spec/prices_core_spec.rb +++ b/spec/prices_core_spec.rb @@ -4,7 +4,7 @@ it "names no collectible or source in its migrations, structure file or connection class, comments included", :aggregate_failures do files = Dir[Rails.root.join("db/prices_migrate/*.rb").to_s] + [ Rails.root.join("db/prices_structure.sql").to_s, Rails.root.join("app/models/prices_record.rb").to_s ] - expect(files.size).to eq(3) # one migration so far, the structure file and the class; a new migration raises this + expect(files.size).to eq(5) # three migrations (the released one and the two of spec 089), the structure file and the class; a new migration raises this expect(files.select { File.read(it).match?(/\b(mtg|scryfall|magic)\b/i) }).to be_empty end end diff --git a/spec/prices_schema_spec.rb b/spec/prices_schema_spec.rb index 3e58878e..966c891d 100644 --- a/spec/prices_schema_spec.rb +++ b/spec/prices_schema_spec.rb @@ -33,21 +33,30 @@ def rowid_problems(definitions) end # A database built only by running db/prices_migrate's migrations, in a child Ruby process with Active Record alone. - def migrated_database - path = Pathname(Dir.mktmpdir("prices-migrated")).join("prices.sqlite3") + # `through` stops at a version (nil: all); `path` continues a database already begun. + def migrated_database(through: nil, path: nil) + path ||= Pathname(Dir.mktmpdir("prices-migrated")).join("prices.sqlite3") script = <<~RUBY require "bundler/setup" require "active_record" ActiveRecord::Base.establish_connection(adapter: "sqlite3", database: ARGV[0]) ActiveRecord::Migration.verbose = false - ActiveRecord::MigrationContext.new(ARGV[1]).migrate + target = ARGV[2].to_s.empty? ? nil : ARGV[2].to_i + ActiveRecord::MigrationContext.new(ARGV[1]).migrate(target) RUBY - _out, err, status = Open3.capture3(RbConfig.ruby, "-e", script, path.to_s, Rails.root.join("db/prices_migrate").to_s, chdir: Rails.root.to_s) + _out, err, status = Open3.capture3(RbConfig.ruby, "-e", script, path.to_s, Rails.root.join("db/prices_migrate").to_s, through.to_s, chdir: Rails.root.to_s) raise "migrating failed: #{err}" unless status.success? path end + def execute_in(path, sql) + database = SQLite3::Database.new(path.to_s) + database.execute(sql) + ensure + database&.close + end + def master_of(path) database = SQLite3::Database.new(path.to_s, readonly: true) database.execute("SELECT name, type, sql FROM sqlite_master WHERE type IN ('table', 'index') ORDER BY name") @@ -77,4 +86,51 @@ def loaded_master expect(rowid_problems(definitions(rows))).to eq([ "current_prices: isn't WITHOUT ROWID; #{advice}" ]) end + + context "with spec 089's migration (AC-5.2, AC-1.9)" do + let(:released) { "20261113100001" } + let(:count_columns) { %w[quotes_seen unchanged repriced added removed removals_withheld unknown_printings malformed] } + + def columns_of(path) + execute_in(path, "PRAGMA table_info(price_refresh_runs)").to_h { |_cid, name, type, notnull, default, _pk| [ name, { type:, notnull:, default: } ] } + end + + it "keeps a run row of a database migrated as main has it, and its table's check, key and index", :aggregate_failures do + path = migrated_database(through: released) + execute_in(path, "INSERT INTO price_sources (key, name, attribution, created_at, updated_at) VALUES ('s', 'S', 'a', '2026-11-13', '2026-11-13')") + execute_in(path, "INSERT INTO price_refresh_runs (price_source_id, run_on, status, started_at, created_at, updated_at) VALUES (1, '2026-11-12', 'finished', '2026-11-12 10:00:00', '2026-11-12', '2026-11-12')") + + migrated_database(path:) + columns = columns_of(path) + master = execute_in(path, "SELECT name, sql FROM sqlite_master").to_h + + expect(execute_in(path, "SELECT run_on, quotes_seen, job_id FROM price_refresh_runs")).to eq([ [ "2026-11-12", 0, nil ] ]) + expect(columns["run_on"][:notnull]).to eq(0) + expect(columns.keys).to include("job_id", "languages", *count_columns) + expect(count_columns.map { |name| columns[name].values_at(:notnull, :default) }.uniq).to eq([ [ 1, "0" ] ]) + expect(master["price_refresh_runs"]).to include("price_refresh_runs_status_known", "REFERENCES \"price_sources\"") + expect(master).to have_key("index_price_refresh_runs_on_price_source_id_and_run_on") + expect(rowid_problems(definitions(master_of(path)))).to be_empty + end + + it "accepts a run with no date and refuses an unknown status", :aggregate_failures do + path = migrated_database + execute_in(path, "INSERT INTO price_sources (key, name, attribution, created_at, updated_at) VALUES ('s', 'S', 'a', '2026-11-13', '2026-11-13')") + + expect { execute_in(path, "INSERT INTO price_refresh_runs (price_source_id, status, started_at, created_at, updated_at) VALUES (1, 'skipped', '2026-11-13', '2026-11-13', '2026-11-13')") }.not_to raise_error + expect { execute_in(path, "INSERT INTO price_refresh_runs (price_source_id, status, started_at, created_at, updated_at) VALUES (1, 'bogus', '2026-11-13', '2026-11-13', '2026-11-13')") }.to raise_error(SQLite3::ConstraintException) + end + + it "goes down without losing a row: a nil run_on takes the date of started_at", :aggregate_failures do + path = migrated_database + execute_in(path, "INSERT INTO price_sources (key, name, attribution, created_at, updated_at) VALUES ('s', 'S', 'a', '2026-11-13', '2026-11-13')") + execute_in(path, "INSERT INTO price_refresh_runs (price_source_id, status, started_at, created_at, updated_at) VALUES (1, 'skipped', '2026-11-13 09:00:00', '2026-11-13', '2026-11-13')") + + migrated_database(through: released, path:) + + expect(execute_in(path, "SELECT run_on FROM price_refresh_runs")).to eq([ [ "2026-11-13" ] ]) + expect(columns_of(path)["run_on"][:notnull]).to eq(1) + expect(columns_of(path).keys).not_to include("job_id", "quotes_seen") + end + end end diff --git a/spec/support/fake_price_source.rb b/spec/support/fake_price_source.rb index e2319ebb..79b2aa5e 100644 --- a/spec/support/fake_price_source.rb +++ b/spec/support/fake_price_source.rb @@ -1,4 +1,5 @@ # A price source for the contract's own specs (spec 085): it knows no real collectible and yields the records it is given. +# A record that responds to `call` is called, not yielded, so an example can raise partway. # # @api private class FakePriceSource @@ -30,7 +31,7 @@ def reported_on(_path) = @reported_on.is_a?(Exception) ? raise(@reported_on) : @ def each_quote(path, languages:, &block) return enum_for(:each_quote, path, languages:) unless block - @records.each(&block) + @records.each { |record| record.respond_to?(:call) ? record.call : block.call(record) } end end diff --git a/spec/support/price_refresh_helpers.rb b/spec/support/price_refresh_helpers.rb new file mode 100644 index 00000000..7e653088 --- /dev/null +++ b/spec/support/price_refresh_helpers.rb @@ -0,0 +1,60 @@ +# Builders and fakes for the daily price refresh's specs (spec 089): a fake price source and a fake catalog source for +# a made-up collectible, "widgets", registered for one example with `use_widget_sources`. +# +# @api private +module PriceRefreshHelpers + PLAIN = Prices::Sources::Series.new(finish: "plain", currency: "USD", kind: "market") + DAY = Date.new(2026, 11, 13) + + # Registers `fake_source` (prices) and `widgets` (catalog) for this example; both are undone after it. + # + # @param records [Array] what the price source yields from `each_quote` (a record that responds to `call` is called) + # @param reported_on [Date, Exception] what `#reported_on` answers + # @param series [Array, nil] the series it declares, or the one plain series + # @param version [String] the catalog source's current version + # @param languages [Array] the catalog source's languages + def use_widget_sources(records: [], reported_on: DAY, series: nil, version: "v1", languages: [ "en" ]) + price_class = Class.new(FakePriceSource) do + define_singleton_method(:series) { series } if series + define_method(:initialize) { |*| super(records:, reported_on:) } + end + catalog_class = Class.new(FakeCatalogSource) do + define_method(:initialize) { |*| super(version:, languages:) } + end + stub_const("WidgetPrices", price_class) + stub_const("WidgetCatalog", catalog_class) + Prices.sources["fake_source"] = "WidgetPrices" + Catalog.sources["widgets"] = "WidgetCatalog" + @widget_sources = true + price_class + end + + def widget_quote(key, cents, series: PLAIN, on: DAY) + Prices::Sources::Quote.new(external_key: key, finish: series.finish, currency: series.currency, kind: series.kind, cents:, reported_on: on) + end + + def widget_malformed(key = nil, series: nil) = Prices::Sources::Malformed.new(external_key: key, error: "unreadable", series:) + + def widget_entry(key, **attributes) = create(:catalog_entry, collectible_type: "widgets", external_key: key, **attributes) + + def price_source_row = PriceSource.sync!(FakePriceSource) # the same row `fake_source` has, before an example registers its own + + def series_row(series = PLAIN) = PriceSeries.ensure!([ series ]).fetch(series) + + # A current price and its history row, as an earlier run would have left them. + def seed_price(entry, cents, series: PLAIN, on: Date.new(2026, 11, 12)) + keys = { catalog_entry_id: entry.id, price_source_id: price_source_row.id, price_series_id: series_row(series).id } + CurrentPrice.insert_all!([ keys.merge(cents:, reported_on: on, changed_on: on) ]) + PriceChange.insert_all!([ keys.merge(recorded_on: on, cents:) ]) + end +end + +RSpec.configure do |config| + config.include PriceRefreshHelpers + config.after do + next unless @widget_sources + + Prices.sources.delete("fake_source") + Catalog.sources.delete("widgets") + end +end