Repository navigation
fix(data): backfill 37 confirmed GSMArena URLs via verify tool - #339
Merged
Merged
Conversation
Upgrades 37 kaggle-only smartphone records (Tier 3, source_category gsmarena-kaggle) to a confirmed GSMArena Tier-2 spec-page URL. Each URL was confirmed by app.verify.gsmarena_backfill's strict multi-signal gate (sitemap identity + heading match + liveness + spec agreement); AMBIGUOUS/NOTFOUND/CONTRADICT results were reported, not written. Refs #296
Regenerates the published static dump (site/public/v1/smartphones) for the 37 records upgraded in the previous commit, via python -m app.dump from a TechEngine checkout against this worktree's data. Only the source_urls of the affected detail pages changed; unrelated dump churn was excluded. Refs #296
github-actions
Bot
added
bug
Member
🔎 Data verification — Tier 0 (offline existence/trust)Scored by 12/12 categories ? 191,819 records ? diff base develop@761729a Changed records in this PR37 record(s) assessed. %%{init: {"theme":"base","themeVariables":{"pie1":"#3fb950","pie2":"#d29922","pie3":"#f85149","pieStrokeWidth":"0px","pieOpacity":"1"}}}%%
pie showData
title Verification bands — all records
"Green" : 35
"Yellow" : 2
"Red" : 0
Full-dataset baseline191819 record(s) assessed. %%{init: {"theme":"base","themeVariables":{"pie1":"#3fb950","pie2":"#d29922","pie3":"#f85149","pieStrokeWidth":"0px","pieOpacity":"1"}}}%%
pie showData
title Verification bands — all records
"Green" : 32807
"Yellow" : 157957
"Red" : 1055
Hard violations (forced red):
green = authoritative source + complete + consistent · yellow = plausible, needs confirmation · red = sparse/weak source or a hard contradiction. Promotion to |
Member
TechEngine change review: PASS12/12 categories ? 191,819 records ? diff base develop@761729a
Changed data
Changed record examplessmartphone modified
Heuristic review
|
Member
TechEngine validation stats: PASS12/12 categories ? 191,819 records ? diff base develop@761729a
Data summary
Warning Tracked verified coverage is below 50% for monitor 0.0% (0/882), software 0.8% (358/42493), laptop 5.3% (105/1995), tablet 8.3% (288/3459), soc 13.4% (284/2117), all 16.8% (32257/191819), smartphone 18.8% (17581/93407), website 19.0% (7635/40084), and 1 more. Validation notes
Key output:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Retry of the GSMArena backfill (the
2026-09-23rate limit had long expired).Upgrades kaggle-only smartphone records (Tier 3,
variant.source_category == "gsmarena-kaggle") to a confirmed GSMArena Tier-2 spec-page URL, using theexisting
app.verify.gsmarena_backfilltool in TechEngine. The tool's strictmulti-signal gate (sitemap identity + heading match + liveness + spec agreement
on battery/RAM/display/weight) decides every write; AMBIGUOUS / NOTFOUND /
CONTRADICT are reported, never written. Matches were not hand-authored and the
gate was not loosened.
37 records received a confirmed
source_urlsupgrade before the run wasstopped by a fresh rate limit (see below). Commits:
fix(data): backfill 37 confirmed GSMArena URLs via verify tool— thedata/smartphone/**source_urlsedits.chore(dump): refresh smartphone dump pages for GSMArena URL backfill— themandatory dump refresh (
python -m app.dumpfrom a TechEngine checkout, freshscratch
DATABASE_URL,TECHAPI_DATA_DIRpointed at this worktree). Only the37 affected
site/public/v1/smartphones/<slug>/index.jsondetail pageschanged; unrelated score-percentile/schema churn from a fresh full dump was
excluded so this PR carries only the backfill's effect.
Rate limit (run stopped early, not retried)
The apply run made real progress (the 37 confirmed writes above) and then hit a
GSMArena 429 with
Retry-After: 36000s(10 hours). The tool stopped cleanlybefore sending any further requests — no retry spinning — and left the remaining
kaggle-only records eligible for a later invocation. A dry-run of 20 immediately
beforehand was clean (19 CONFIRM / 1 NOTFOUND, no 429), so the limit was tripped
during this apply run rather than being a pre-existing block.
Unresolved this run (correctly not written):
Caterpillar B15(NOTFOUND,no-index-hit),
Lenovo A60+(AMBIGUOUS, ambiguous-index),Sharp Aquos Crystal 2(NOTFOUND, not-live).Tool summary
(54 CONFIRM proposed across this run's window; 37 resulted in a new working-tree
write — the remainder already carried the confirmed URL from a prior session's
cache, so the idempotent write guard left them untouched.)
Verification
python -m app.validate(bundled validator) —✅ Data validation passed.python -m app.dump; the 37 committed detail pages areclean
source_urlsappends matching the data edits.Closes #296