Skip to content

feat(prices): refresh prices once a day and record each change once (#365) - #425

Merged
plainprogrammer merged 36 commits into
mainfrom
089-daily-price-refresh
Nov 14, 2026
Merged

plainprogrammer merged 36 commits into
mainfrom
089-daily-price-refresh

Conversation

@plainprogrammer

Copy link
Copy Markdown
Owner

Closes #365.

What this does

Prices refresh once a day, and each change is recorded once. A scheduled job (Prices::RefreshJob, queue sync, at 11:30am every day, production only) reads the bulk file the catalog already downloads and, per quote:

  • unchanged: nothing is written;
  • changed or new: the current price is updated and a history row is written for the run's date (the UTC date of the file, so the same file twice changes nothing);
  • gone: the current price is removed and a blank history row is written, in one transaction, and only when the file can be trusted to say so;
  • a printing the catalog lacks: skipped and counted.

A Malformed quote is never read as absent. A file cut short, empty, with an unreadable line that names no printing, or that would remove more than 10 percent of the candidate prices (MAX_REMOVED_SHARE, a guess until #366 measures it) removes nothing. Writes go in batches of 1,000 printings, no network call or primary read happens inside a transaction on the prices database, and two runs never overlap.

Spec: docs/specs/089-daily-price-refresh/spec.md (v0.5.0), plan and verification.md beside it. The rules are distilled into docs/architecture/prices.md and catalog-operations.md.

The issue's open point, and the rulings

  • Shared download, separate lifecycle (maintainer, 2026-11-13): the price refresh asks the catalog source for the same file, but has its own job, run record and guard, not Catalog::Operation. One change to the catalog source: a partial download file per download, stale partial files pruned, and the Catalog::Sources contract comment says #download must tolerate concurrent calls (supersedes spec 002 FR-3's "a partial file"; recorded in the index both ways).
  • The run record holds identity, times, status and the counts this issue names (quotes_seen, unchanged, repriced, added, removed, removals_withheld, unknown_printings, malformed); Each price refresh records what it did and how large the database is #366 owns timings, sizes and memory. run_on becomes nullable and job_id and languages are added, by two new migrations in db/prices_migrate/ (the released one is untouched).
  • A price's freshness is the latest finished run's date, not a column; a file older than a run that wrote rows is skipped.

Deviations and things the reviews changed

  • The count column changed is repriced: Active Record reserves the name (found in Phase 2; spec v0.4.3; a second migration renames it).
  • AC-4.4 measures savepoint spans, not transaction depth, because the suite's non-joinable transaction makes every write a savepoint (spec v0.4.2).
  • The implementation review (SPEC-ALIGNED) found a gap: a series the source no longer declares would read as absent. Such a price is now left alone (spec v0.5.0, AC-3.7).
  • An automated security review found the pending unreadable records unbounded; fixed.
  • Accepted, recorded in prices.md and on Each price refresh records what it did and how large the database is #366: the judged-pairs set in memory (about 30 MB for 300,000 quotes), one DELETE per printing in a removal slice, and a process resuming after the 6-hour lock lapsed could overwrite an interrupted run.

Evidence

  • Full pre-push bin/ci passed at 8a035683 (RuboCop, YARD, the comment ratchet, Brakeman, bundler-audit, importmap audit, the licence check, the whole RSpec suite). A first push attempt was blocked by one failing example, spec 081's column-list spec (prices_record_spec.rb:78), which I then fixed.
  • Timing (hand run, development machine, no threshold): 110,000 cards, 275,000 quotes: first run 18.4 s, a typical day (about 17 percent repriced) 7.4 s.
  • Not run here: bin/upgrade-smoke (needs a container engine; CI's image job runs it). No UI change, so no screenshots.

Owed to the maintainer

  • The schedule time (11:30 server time, a guess from one day's publication time, 10:06 UTC on 2026-11-13) is a one-line change in config/recurring.yml.
  • A real, lasting drop in prices above 10 percent is refused every day until MAX_REMOVED_SHARE moves; there is no override and no prices page until An admin reads the price refresh log #367.
  • The README's backup advice now keeps backups clear of 11:30.

Model split: spec drafts, rulings and wiring by the worker's main session (Sonnet 5.5, with a stretch on Opus 5.5 that wrote the Gate A record and the plan draft); spec and plan reviews by read-only Fable 5.2 subagents; implementation by Sonnet 5.5 subagents; the implementation review by a read-only Opus 5.5 subagent.

🤖 Generated with Claude Code

plainprogrammer and others added 30 commits November 13, 2026 19:13
Draft v0.1.0 with four decisions marked pending for the coordinator: the
file is shared with the catalog refresh and the lifecycle is not, what a
run records and where #366 starts, what reported_on means when unchanged
prices write nothing, and how a gone price is told from a bad file.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…to 4

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Nullable run_on and job_id, stale partial downloads pruned, atomic removal
batches, one table of errors and fates, quotes_seen over known printings,
the older-file rule over runs that wrote rows, and buildable AC-4.4/4.5.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The source is synced before a run exists, any Quote counts for the
no-valid-quotes guard, the ceiling is judged on the whole walk, the
download-size error is retried, counts are tallied after commit, and the
errors table has a catch-all and the no-discard rationale.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ten phases: fakes, the run table migration, the three models and the run
guard, the unique partial download, the refresh's lifecycle, judging
quotes, absences and their guards, the job and schedule, the proofs, and
the documents.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The run model gains running?, the helpers sync the fake source before
seeding, AC-4.4 measures savepoint spans (the suite's non-joinable
transaction makes every write a savepoint), the undeclared-series check
follows the flush, and the examples and lint notes the review found.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The third-attempt example counts retries as retry_on does, the removal
walk is added after the final flush, the batch-touch example counts only
quotes_seen updates, three long examples carry their ExampleLength reason,
and the prune and YARD details the review found.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… decision 2)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ones are pruned (spec 089 decision 1)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Active Record reserves the attribute name changed, so PriceRefreshRun
cannot be built with a column of that name. Found at execution, in
Phase 2. Ruling: rename it with a second, unreleased migration; the plan
gains T1.6.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…(spec 089 v0.4.3)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…resh runs (spec 089)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…c 089)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
My rename of the changed count to repriced also hit :changed_on in the
plan's Phase 5 examples; the implementer used the real column name.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…spec 089)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…le (spec 089)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…spec 089)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…time (spec 089)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…d the backup advice (spec 089)

The prices lane's #365 row is Done, the status paragraph says the daily
refresh exists, the backup advice gains the refresh's 11:30 window (ADR 0029,
Cost), and verification.md records the evidence of each criterion, the timing
and the run's deviations.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…andidate

Implementation review: a series the source no longer declares would read
as absent, so a later build that drops one would fail every run on the
ceiling or blank prices the file was never asked about.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…removal candidates (spec 089 v0.5.0)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…pec 089)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… review (spec 089)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…note (spec 089)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The daily refresh's invariants, states, operator view of a daily failure,
constants (MAX_REMOVED_SHARE a guess until #366) and accepted limits go
into prices.md; the shared download, the partial file per download and
the stale partial pruning into catalog-operations.md. The index row is
Frozen with its plan retirement owed.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
plainprogrammer and others added 6 commits November 13, 2026 21:42
…s area docs (spec 027 D4)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Three lessons (a column named changed is reserved by Active Record;
savepoint spans, not depth, in a transactional suite; deleting by absence
needs a candidate rule per dimension) and a note that the harness can
change a worker's model mid-run.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Conflicts in CLAUDE.md (kept main's licence-check sentence and the status
paragraph's daily refresh) and docs/specs/README.md (kept both new rows).
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Conflicts in docs/architecture/catalog-operations.md (kept both
distillation log rows) and docs/specs/README.md (kept the 089 row and
main's 091 row, whose plan is retired).
@plainprogrammer
plainprogrammer marked this pull request as ready for review November 14, 2026 03:32
@plainprogrammer plainprogrammer self-assigned this Nov 14, 2026
@plainprogrammer
plainprogrammer merged commit b89ff31 into main Nov 14, 2026
9 checks passed
@plainprogrammer
plainprogrammer deleted the 089-daily-price-refresh branch November 14, 2026 03:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Prices refresh once a day, and each change is recorded once

1 participant