Repository navigation
dump: make app.dump timestamps deterministic (precondition for daily refresh) #18
Description
Why
Moving the full refresh/dump to a daily cadence is blocked by timestamp churn.
app.dump is a stateless rebuild: every run stamps created_at/updated_at with the build time, so even when the underlying data/ is unchanged it emits a ~1400-file diff. On a daily schedule this means a giant no-op PR to TechAPI every day, drowning real changes and defeating review. (This is the same churn that was hit and reverted during a manual dump.)
What
Make dump output deterministic so a re-run with unchanged data produces no diff:
- preserve
created_at(never rewrite once set), and - only bump
updated_atwhen the record's substantive fields actually change, or - drop build-time timestamps from the static dump entirely / pin them (e.g.
SOURCE_DATE_EPOCH).
Unblocks
Once dump diffs reflect only real data changes, weekly-refresh (or a split dump-only job) can safely move to daily. Until then it stays weekly.
Related
Scheduling split landed in b53bcab (coverage-report daily; scrape + ingest weekly).
Activity
Seungpyo1007 commentedon Sep 24, 2026
Status update: the determinism precondition described here is done. TechEngine #63 (slug-hash ids), #65 (git-history-derived created_at/updated_at), and #66 (fetch-depth: 0) landed 2026-08, and dump-refresh has produced zero-diff PRs on unchanged data since (confirmed across multiple runs).
app.dumpreruns are now stable.Leaving this open rather than closing it, since the actual "move weekly-refresh to daily" decision this was meant to unblock hasn't been made — that's a real call about crawl frequency/load, not a code fix, so it needs a decision rather than a PR.
Seungpyo1007 commentedon Sep 26, 2026
Closing: done. Deterministic dump landed in #63 (slug-hash ids), #65 (git-history timestamps) and #66 (fetch-depth: 0); dump-refresh has produced zero-diff PRs on unchanged data since.