Repository navigation
ci: scope every CI job to the changed data instead of the full dataset #350
Description
Why
The 2026-09-29/30 mass-agent push (18 concurrent workers on one develop) exposed how expensive our CI is: every PR re-reads the whole dataset (~190k records / ~292k dump files) no matter how little changed, dump refresh takes ~33 min and re-runs on every rebase, and the automation volume coincided with the TechEngineBot account suspension.
Goal
Make every CI job scale with the size of the change, not the size of the dataset.
Plan
- Unblock – drop the suspended TechEngineBot from
pr-metadataassignees (404s every PR). - Scope calculator (TechEngine) – changed paths ->
{category, files, full_run_required}; falls back to a full run when code/schema changed. - Scoped loading –
load_categoryaccepts a file list; brand/soc slug sets for FK checks stay full (small). - verify-comment –
fetch-depth: 1, readdata/_verify/status.jsoninstead of rescoring everything, score once. - validate – PRs validate changed files (+FK targets); full validation only on develop/main push and nightly.
- dump – per-category and per-path generation, category matrix; move dump refresh off the per-PR critical path (needs a decision on the "dump is the last commit of every data PR" rule).
- deploy-pages – precompute the homepage history timeline so the build no longer needs
fetch-depth: 0. - Cleanup – stale worktrees/branches (29 worktrees, 111 local branches), stale
dump-refresh/*PRs.
Metadata
Metadata
Assignees
Labels
Type
Projects
- Status
Activity
ci(pr-metadata): stop assigning the suspended TechEngineBot
perf(verify): load only the changed records for `score --changed`
ci(verify-comment): shallow TechAPI checkout, baseline from status.json
perf(validate): scope PR validation to the changed records
ci(validate-data): validate only changed records on PRs
perf(dump): dump only the collections a change can affect
perf(dump): dump only the collections a change can affect
ci(pr-validation): shallow checkouts and scoped app.validate
perf(integrity): scope the PR integrity gate to changed records
ci: dump after merge instead of per PR; full validation on push only
Seungpyo1007 commentedon Oct 31, 2026
Done in #374, #377 and #378. The dump is no longer committed or rebuilt per PR; pages.yml generates it at deploy. pr-checks runs the engine validator on the changed records only, and the full run only when the TechEngine pin moves. The old bot workflows are gone, and the stale branches and worktrees were cleaned up on 2026-10-31. Item 7 is left as it is: pages.yml keeps fetch-depth: 0, because the dump stamps created_at/updated_at from git history, and it only runs on a release.