Repository navigation
feat(prices): refresh prices once a day and record each change once (#365) - #425
Merged
Merged
Conversation
Draft v0.1.0 with four decisions marked pending for the coordinator: the file is shared with the catalog refresh and the lifecycle is not, what a run records and where #366 starts, what reported_on means when unchanged prices write nothing, and how a gone price is told from a bad file. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…to 4 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Nullable run_on and job_id, stale partial downloads pruned, atomic removal batches, one table of errors and fates, quotes_seen over known printings, the older-file rule over runs that wrote rows, and buildable AC-4.4/4.5. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The source is synced before a run exists, any Quote counts for the no-valid-quotes guard, the ceiling is judged on the whole walk, the download-size error is retried, counts are tallied after commit, and the errors table has a catch-all and the no-discard rationale. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ten phases: fakes, the run table migration, the three models and the run guard, the unique partial download, the refresh's lifecycle, judging quotes, absences and their guards, the job and schedule, the proofs, and the documents. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The run model gains running?, the helpers sync the fake source before seeding, AC-4.4 measures savepoint spans (the suite's non-joinable transaction makes every write a savepoint), the undeclared-series check follows the flush, and the examples and lint notes the review found. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The third-attempt example counts retries as retry_on does, the removal walk is added after the final flush, the batch-touch example counts only quotes_seen updates, three long examples carry their ExampleLength reason, and the prune and YARD details the review found. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… decision 2) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ones are pruned (spec 089 decision 1) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Active Record reserves the attribute name changed, so PriceRefreshRun cannot be built with a column of that name. Found at execution, in Phase 2. Ruling: rename it with a second, unreleased migration; the plan gains T1.6. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…(spec 089 v0.4.3) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…resh runs (spec 089) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…c 089) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
My rename of the changed count to repriced also hit :changed_on in the plan's Phase 5 examples; the implementer used the real column name. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…spec 089) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…le (spec 089) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…spec 089) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…time (spec 089) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…d the backup advice (spec 089) The prices lane's #365 row is Done, the status paragraph says the daily refresh exists, the backup advice gains the refresh's 11:30 window (ADR 0029, Cost), and verification.md records the evidence of each criterion, the timing and the run's deviations. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…andidate Implementation review: a series the source no longer declares would read as absent, so a later build that drops one would fail every run on the ceiling or blank prices the file was never asked about. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…removal candidates (spec 089 v0.5.0) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…pec 089) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… review (spec 089) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…note (spec 089) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The daily refresh's invariants, states, operator view of a daily failure, constants (MAX_REMOVED_SHARE a guess until #366) and accepted limits go into prices.md; the shared download, the partial file per download and the stale partial pruning into catalog-operations.md. The index row is Frozen with its plan retirement owed. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…s area docs (spec 027 D4) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Three lessons (a column named changed is reserved by Active Record; savepoint spans, not depth, in a transactional suite; deleting by absence needs a candidate rule per dimension) and a note that the harness can change a worker's model mid-run. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Conflicts in CLAUDE.md (kept main's licence-check sentence and the status paragraph's daily refresh) and docs/specs/README.md (kept both new rows).
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Conflicts in docs/architecture/catalog-operations.md (kept both distillation log rows) and docs/specs/README.md (kept the 089 row and main's 091 row, whose plan is retired).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #365.
What this does
Prices refresh once a day, and each change is recorded once. A scheduled job (
Prices::RefreshJob, queuesync,at 11:30am every day, production only) reads the bulk file the catalog already downloads and, per quote:A
Malformedquote is never read as absent. A file cut short, empty, with an unreadable line that names no printing, or that would remove more than 10 percent of the candidate prices (MAX_REMOVED_SHARE, a guess until #366 measures it) removes nothing. Writes go in batches of 1,000 printings, no network call or primary read happens inside a transaction on the prices database, and two runs never overlap.Spec:
docs/specs/089-daily-price-refresh/spec.md(v0.5.0), plan andverification.mdbeside it. The rules are distilled intodocs/architecture/prices.mdandcatalog-operations.md.The issue's open point, and the rulings
Catalog::Operation. One change to the catalog source: a partial download file per download, stale partial files pruned, and theCatalog::Sourcescontract comment says#downloadmust tolerate concurrent calls (supersedes spec 002 FR-3's "a partial file"; recorded in the index both ways).quotes_seen,unchanged,repriced,added,removed,removals_withheld,unknown_printings,malformed); Each price refresh records what it did and how large the database is #366 owns timings, sizes and memory.run_onbecomes nullable andjob_idandlanguagesare added, by two new migrations indb/prices_migrate/(the released one is untouched).Deviations and things the reviews changed
changedisrepriced: Active Record reserves the name (found in Phase 2; spec v0.4.3; a second migration renames it).prices.mdand on Each price refresh records what it did and how large the database is #366: the judged-pairs set in memory (about 30 MB for 300,000 quotes), oneDELETEper printing in a removal slice, and a process resuming after the 6-hour lock lapsed could overwrite aninterruptedrun.Evidence
bin/cipassed at8a035683(RuboCop, YARD, the comment ratchet, Brakeman, bundler-audit, importmap audit, the licence check, the whole RSpec suite). A first push attempt was blocked by one failing example, spec 081's column-list spec (prices_record_spec.rb:78), which I then fixed.bin/upgrade-smoke(needs a container engine; CI's image job runs it). No UI change, so no screenshots.Owed to the maintainer
config/recurring.yml.MAX_REMOVED_SHAREmoves; there is no override and no prices page until An admin reads the price refresh log #367.Model split: spec drafts, rulings and wiring by the worker's main session (Sonnet 5.5, with a stretch on Opus 5.5 that wrote the Gate A record and the plan draft); spec and plan reviews by read-only Fable 5.2 subagents; implementation by Sonnet 5.5 subagents; the implementation review by a read-only Opus 5.5 subagent.
🤖 Generated with Claude Code