Repository navigation
fix(data): resolve 364 smartphone brand-prefix conflicts - #330
Merged
Merged
Conversation
…uplicates) Follow-up to #329, which merged the 50 identical-core brand-prefix pairs and left 717 pairs that disagreed on a core field (soc / release_date / ram_gb / battery_mah / weight_g / os) documented but unresolved. This resolves the largest, highest- confidence slice of those. Classification of the 717 (scripted, then spot-verified against sources): - The soc "conflict" in 572 pairs is two DIFFERENT slugs that both resolve to a real soc record. For 414 of those the two slugs are the SAME physical chip with a different spelling (e.g. "qualcomm-msm7227-snapdragon-s1" vs "qualcomm-msm7227", "qualcomm-msm8260a-snapdragon-s4-plus" vs "qualcomm-snapdragon-s4-msm8260a") — not a real spec difference. - The other conflicts on those same-chip pairs are the fabricated import placeholders (weight_g=150.0, battery_mah=1500, and the generic YYYY-01-01 launch date) that #329 already flagged as a systemic bulk-import bug. This commit merges the 364 same-chip pairs whose two sides are a GSMArena scrape (bare slug, Kaggle `gsmarena-mobile-devices`, carries a `raw_gsmarena` block) and a phonedb.net twin (brand-prefixed slug). Keeper = the GSMArena record: - Its specs are the real ones. Spot-verified: Acer DX650 weighs 133 g (GSMArena, imei.org, softpedia) — the phonedb twin's 150.0 g is the stamped placeholder; Acer CloudMobile S500 is 122 g (GSMArena) vs phonedb's 150.0. - soc kept as the keeper's slug (confirmed FK-valid for all 364; same physical chip). - name upgraded to the brand-inclusive form ("Acer DX650" over "DX650"). - source_urls unioned so the phonedb.net reference is retained. - verified left unchanged — NOT flipped to true, because the phonedb side's scalar values proved unreliable (fabricated weights), so it is not a trustworthy source. - release_date left as the keeper's value and NOT resolved here: the phonedb twin's specific dates are unreliable too (phonedb says Acer DX650 = 2009-03 but GSMArena/imei say 2009-06), so importing them would be guessing. Dates are reported as still-unresolved rather than changed. The redundant phonedb twin file is deleted. No scalar value was invented; in every one of these 364 pairs the keeper already held the real GSMArena value and only the deleted twin carried the placeholder. Left for follow-up (documented on #296): 37 same-chip pairs sourced against vendor /Wikipedia pages rather than phonedb, 157 pairs with a genuine ram/weight/battery question needing per-record sourcing, and 158 pairs whose soc slugs are two different chips — several of which are genuinely different devices that share a name (e.g. Honor Magic V2 SD8Gen2 vs Kirin 9000S, Huawei P50 Pro SD888 vs Kirin 9000, HTC One M9 Helio X10 vs Snapdragon 810) and will be un-paired, not merged. Both gates pass: `python -m app.validate` and `integrity_check.py <data> --strict` (0 hard anomalies). Closes #296
…flicts Regenerates site/public/v1 from the current data with no re-scrape (python -m app.dump). Removes the 364 deleted phonedb-twin slugs and reconciles score percentiles across the scored catalog. `app.validate` passes on the dumped tree.
github-actions
Bot
added
bug
Member
🔎 Data verification — Tier 0 (offline existence/trust)Scored by 12/12 categories ? 191,346 records ? diff base develop@2360ec8 Changed records in this PR364 record(s) assessed. %%{init: {"theme":"base","themeVariables":{"pie1":"#3fb950","pie2":"#d29922","pie3":"#f85149","pieStrokeWidth":"0px","pieOpacity":"1"}}}%%
pie showData
title Verification bands — all records
"Green" : 357
"Yellow" : 7
"Red" : 0
Full-dataset baseline191346 record(s) assessed. %%{init: {"theme":"base","themeVariables":{"pie1":"#3fb950","pie2":"#d29922","pie3":"#f85149","pieStrokeWidth":"0px","pieOpacity":"1"}}}%%
pie showData
title Verification bands — all records
"Green" : 32949
"Yellow" : 157343
"Red" : 1054
Hard violations (forced red):
green = authoritative source + complete + consistent · yellow = plausible, needs confirmation · red = sparse/weak source or a hard contradiction. Promotion to |
Member
TechEngine change review: PASS12/12 categories ? 191,346 records ? diff base develop@2360ec8
Changed data
Changed record examplessmartphone modified
smartphone deleted
Heuristic review
|
Member
TechEngine validation stats: PASS12/12 categories ? 191,346 records ? diff base develop@2360ec8
Data summary
Warning Tracked verified coverage is below 50% for monitor 0.0% (0/882), software 0.8% (358/42493), laptop 5.3% (105/1995), tablet 8.6% (285/3305), soc 13.4% (284/2117), all 16.7% (31952/191346), smartphone 18.6% (17280/92994), website 19.0% (7635/40084), and 1 more. Validation notes
Key output:
|
Seungpyo1007
added a commit
that referenced
this pull request
Sep 29, 2026
Continues the brand-prefix dedup from #329/#330. Those resolved 414 pairs and carried forward a residual of 353. This resolves 170 of that residual, source- verified, and explicitly un-pairs 3 that are genuinely different devices. Three sub-slices resolved (Refs #296): 1. 37 vendor/Wikipedia-sourced same-chip pairs. Same device, same physical chip (soc slugs differ only in spelling, e.g. `qualcomm-msm8974-snapdragon-800` vs `snapdragon-800`, `hisilicon-kirin-980` vs `kirin-980`), with battery/weight/ram already matching. Unlike the phonedb twins in #330, the twin here carries a Wikipedia or vendor page as its source, so its specific release_date is reliable and was adopted over the bare side's generic YYYY-01-01 placeholder (spot-checked: Huawei P30 Pro 2019-03-26 = GSMArena; LG G2 September 2013 = Wikipedia). Kept the canonical bare slug, unioned source_urls, set verified=true only where the twin was a vendor/Wikipedia source. 2. 7 same-model pairs whose two sides named different chips — verified against a cited source which side is correct, fixed the soc, then merged: - Oppo Find X8 Pro -> Dimensity 9400 (oppo.com launch; GSMArena) - Motorola Edge 50 Ultra -> Snapdragon 8s Gen 3 (motorola.com; phonearena) - Honor Magic V3 -> Snapdragon 8 Gen 3 (phonearena SM8650-AB; honor.com) - OnePlus Nord CE 2 Lite (Snapdragon 695), Nord CE 2 (Dimensity 900), Sony Xperia Z (APQ8064 S4 Pro), Honor Magic V2 (Snapdragon 8 Gen 2) — the bare side already held the correct chip; merged and cited GSMArena/vendor. 3. 126 pairs where the soc slugs are the same physical chip via a spelling variant, or one side's soc was a vague descriptor (`dual-core 1.0 GHz`, `<brand> mobile platform`, `n-a`) and the twin named the specific FK-valid part. Kept the GSMArena (bare) record, adopted the specific soc when the keeper's was vague, rejected the phonedb stamped placeholders (weight 150.0 / battery 1500), left the (unreliable) phonedb dates alone, unioned source_urls, deleted the twin. Un-paired as genuinely different devices (left as separate records, no merge): - HTC: `one-m9` holds the One M9+ (Helio X10) vs the One M9 (Snapdragon 810). - Motorola: `razr` holds the 2023 Razr (SD 8+ Gen 1) vs the 2019 Razr (SD 710). - Sony: `xperia-z3` holds the Xperia Z3+ (SD 810) vs the Xperia Z3 (SD 801). No value invented; every repaired field cites its evidence URL in source_urls. Left unresolved for a later pass (documented on #296): 21 same-model pairs whose correct chip I could not source this round (includes the Huawei P50 Pro, which genuinely shipped both Kirin 9000 and Snapdragon 888 SKUs), 157 pairs needing a per-record ram/weight/battery source, and 2 odd/other pairs. Both gates pass: `python -m app.validate` and `integrity_check.py <data> --strict` (0 hard anomalies). Refs #296
This was referenced Sep 29, 2026
Seungpyo1007
added a commit
that referenced
this pull request
Sep 29, 2026
Continues the brand-prefix dedup from #329/#330. Those resolved 414 pairs and carried forward a residual of 353. This resolves 170 of that residual, source- verified, and explicitly un-pairs 3 that are genuinely different devices. Three sub-slices resolved (Refs #296): 1. 37 vendor/Wikipedia-sourced same-chip pairs. Same device, same physical chip (soc slugs differ only in spelling, e.g. `qualcomm-msm8974-snapdragon-800` vs `snapdragon-800`, `hisilicon-kirin-980` vs `kirin-980`), with battery/weight/ram already matching. Unlike the phonedb twins in #330, the twin here carries a Wikipedia or vendor page as its source, so its specific release_date is reliable and was adopted over the bare side's generic YYYY-01-01 placeholder (spot-checked: Huawei P30 Pro 2019-03-26 = GSMArena; LG G2 September 2013 = Wikipedia). Kept the canonical bare slug, unioned source_urls, set verified=true only where the twin was a vendor/Wikipedia source. 2. 7 same-model pairs whose two sides named different chips — verified against a cited source which side is correct, fixed the soc, then merged: - Oppo Find X8 Pro -> Dimensity 9400 (oppo.com launch; GSMArena) - Motorola Edge 50 Ultra -> Snapdragon 8s Gen 3 (motorola.com; phonearena) - Honor Magic V3 -> Snapdragon 8 Gen 3 (phonearena SM8650-AB; honor.com) - OnePlus Nord CE 2 Lite (Snapdragon 695), Nord CE 2 (Dimensity 900), Sony Xperia Z (APQ8064 S4 Pro), Honor Magic V2 (Snapdragon 8 Gen 2) — the bare side already held the correct chip; merged and cited GSMArena/vendor. 3. 126 pairs where the soc slugs are the same physical chip via a spelling variant, or one side's soc was a vague descriptor (`dual-core 1.0 GHz`, `<brand> mobile platform`, `n-a`) and the twin named the specific FK-valid part. Kept the GSMArena (bare) record, adopted the specific soc when the keeper's was vague, rejected the phonedb stamped placeholders (weight 150.0 / battery 1500), left the (unreliable) phonedb dates alone, unioned source_urls, deleted the twin. Un-paired as genuinely different devices (left as separate records, no merge): - HTC: `one-m9` holds the One M9+ (Helio X10) vs the One M9 (Snapdragon 810). - Motorola: `razr` holds the 2023 Razr (SD 8+ Gen 1) vs the 2019 Razr (SD 710). - Sony: `xperia-z3` holds the Xperia Z3+ (SD 810) vs the Xperia Z3 (SD 801). No value invented; every repaired field cites its evidence URL in source_urls. Left unresolved for a later pass (documented on #296): 21 same-model pairs whose correct chip I could not source this round (includes the Huawei P50 Pro, which genuinely shipped both Kirin 9000 and Snapdragon 888 SKUs), 157 pairs needing a per-record ram/weight/battery source, and 2 odd/other pairs. Both gates pass: `python -m app.validate` and `integrity_check.py <data> --strict` (0 hard anomalies). Refs #296
Seungpyo1007
added a commit
that referenced
this pull request
Sep 29, 2026
Scanned the pda category (140 records, reviewed every match by hand) for the three duplicate patterns the tablet/watch (#324) and smartphone (#329/#330) passes turned up. Merged 28 redundant files into 21 keepers (140 -> 112), keeping the better-sourced side field by field and unioning source_urls. Pattern 1 - identical scrape imported twice under different row-id slugs (100% identical specs, differ only by source_row): - motorola/2002/accompli-388: kept ...-scrapegsma-344, deleted ...-1227. - mitac/2005/mio-a701: kept ...-scrapegsma-278, deleted ...-1161. Pattern 2 - same device under a brand-prefixed slug and a bare slug in sibling dirs. Same convention as the #330 GSMArena/phonedb merge: kept the bare GSMArena/Kaggle record (real specs + raw_gsmarena, sometimes Wikipedia), upgraded name to the brand-inclusive form, unioned source_urls to retain the phonedb reference, kept the keeper's soc/release_date, left verified as-is, deleted the phonedb twin: - treo-680 <- palm-treo-680-palm-lowrider ("Palm Treo 680") - treo-750v <- palm-treo-750v-htc-cheetah ("Palm Treo 750v") - treo-750 <- palm-treo-750-htc-cheetah ("Palm Treo 750") - treo-500v <- palm-treo-500v-palm-otto ("Palm Treo 500v") Treo 750 vs 750v and Treo 500 vs 500v were kept as distinct SKUs (only the matching-model twins were merged). palm-treo-500-palm-otto (2008 Treo 500) is a different device and was left alone. Pattern 3 - cross-source duplicate: the same PDA imported as a clean short-slug record and as a cigarplug/scrape-gsma row in the same base dir. Kept the clean record (correct soc FK, canonical slug), deleted the scrape twin, and repaired the keeper's ram_gb from the twin's cited memory_internal (the clean side carried the fabricated ram_gb=1 placeholder; the scrape side had the real value). The github.com/cigarplug/scrape-gsma URL was unioned so the repaired RAM is sourced: - 16 HP iPAQ pairs: h6310/h6315/h6320/h6325/hw6510/hw6515 -> 0.0625 GB (64 MB); hw6910/hw6915/rw6815/rw6818/rw6828/514 -> 0.0625; 610c/910c/data-messenger/ voice-messenger -> 0.125 GB (128 MB). - i-mate pda2 (two identical scrape twins, -1004 and -121) -> 0.125 GB. - i-mate pda2k -> 0.125 GB; i-mate pdal -> 0.062 GB. - palm treo-pro (3-way: bare + scrape-gsma-7066 + phonedb palm-treo-pro): kept the bare Wikipedia-cited record, ram 1 -> 0.125 GB (128 MB), name -> "Palm Treo Pro", unioned all four source_urls. Left unmerged and documented (ambiguous, per "leave ambiguous unmerged"): the GSM webOS palm-pre-scrapegsma-7065 and palm-pre-plus-scrapegsma-7063 overlap a region-split family (palm-pre[-plus]-palm-castle-c CDMA and palm-pre[-plus]-gsm-*-palm-castle-g EU/NA) but do not map 1:1, so no merge. The EU/NA regional variant pairs (pixi-gsm, pixi-plus-gsm, pre-2-gsm, pre[-plus]-gsm-castle-g) are legitimate separate SKUs and were left alone. No value was invented: every ram_gb repair took the real figure already present in the deleted twin's cited memory_internal. verified flags unchanged. Both gates pass: `python -m app.validate` and `integrity_check.py <data> --strict` (0 hard anomalies). Refs #296
This was referenced Sep 29, 2026
Seungpyo1007
added a commit
that referenced
this pull request
Sep 29, 2026
… base slug The brand-prefix scan across #329/#330/#334 kept flagging some pairs as duplicates that are actually two DIFFERENT products which happen to share a base name — the mechanical slug-matching couldn't tell them apart. A focused pass for that specific pattern (name divergence, not the "same device / data conflict" or "needs a source" patterns already handled) found 7, each confirmed against a source as a distinct real device. Un-paired by giving the mis-slugged variant a distinct slug/filename /name so it is no longer a false duplicate of the base model. No merges; both records stay as separate devices. - HTC U11+ vs U11 — distinct fall-2017 model, smaller bezels, SD835/6GB/3930mAh vs the summer U11 (androidcentral). `u11` -> `u11-plus` (name "HTC U11+"). - HTC One M9+ vs One M9 — the M9+ uses a Helio X10 and QHD display, separate from the SD810 One M9. `one-m9` -> `one-m9-plus`. - Huawei P40 Pro+ vs P40 Pro — distinct higher model (pocket-lint, CNET). `p40-pro` -> `p40-pro-plus`. - Sony Xperia Z3+ vs Xperia Z3 — the Z3+ (a.k.a. Xperia Z4) is a distinct 2015 Snapdragon 810 model with its own Wikipedia page, vs the 2014 Z3 (Snapdragon 801). `xperia-z3` -> `xperia-z3-plus`. - Motorola Razr (2023) vs Razr (2019) — different foldable generations (SD 8+ Gen 1 vs SD 710). `razr` -> `razr-2023`. - Fairphone 3+ (2020) vs Fairphone 3 (2019) — successor model. The 3+ record -> `fairphone-3-plus-dual-sim-lte-fp3plus`. - Gigabyte GSmart GS202+ vs GS202 — distinct + variant. The + record -> `gsmart-gs202-plus`. Each renamed record keeps its own specs and sources untouched (nothing merged, nothing invented); only slug, base_model_slug, name and filename change so the two devices no longer collide as a false pair. New slugs are collision-free. Both gates pass: `python -m app.validate` and `integrity_check.py <data> --strict` (0 hard anomalies). Residual brand-prefix pairs drop from 183 to 176. Refs #296
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to #329. That PR merged the 50 brand-prefix pairs whose core specs were
already identical and left 717 pairs that disagreed on a core field
(
soc/release_date/ram_gb/battery_mah/weight_g/os) documentedbut unresolved. This PR resolves the largest, highest-confidence slice.
How the 717 break down
Every pair is a bare slug (
dx650, from the Kagglegsmarena-mobile-devicesdump,carrying a
raw_gsmarenablock) vs a brand-prefixed twin (acer-dx650). Conflictfields across the 717: soc 572, release_date 498, weight_g 159, battery_mah 83,
ram_gb 62, os 14.
The
socconflict is almost never a real spec difference — both slugs resolve to areal soc record, and in 414 pairs they are the same physical chip under a
different slug spelling (
qualcomm-msm7227-snapdragon-s1vsqualcomm-msm7227,qualcomm-msm8260a-snapdragon-s4-plusvsqualcomm-snapdragon-s4-msm8260a). Theremaining conflicts on those pairs are the fabricated bulk-import placeholders
(
weight_g=150.0,battery_mah=1500, genericYYYY-01-01date) that #329 flagged.What this PR changes: 364 merges
The 364 same-chip pairs sourced Kaggle-GSMArena (bare) vs phonedb.net (prefixed).
Keeper = the GSMArena record (real specs, richer, has the
raw_gsmarenaprovenanceblock). For each:
150.0 g/1500 mAhare the stamped placeholders. In all 364 the keeper alreadyheld the real value, so no scalar was rewritten — the placeholder only ever
lived on the deleted twin. Spot-verified: Acer DX650 = 133 g (GSMArena,
imei.org, softpedia) not phonedb's 150.0; Acer CloudMobile S500 = 122 g.
Acer DX650overDX650).phonedb side's scalars proved unreliable, so it isn't a trustworthy source.
phonedb's specific dates are unreliable (it lists Acer DX650 as 2009-03; GSMArena
and imei.org say 2009-06), so importing them would be guessing. Reported as
still-unresolved rather than changed.
No value was invented.
Counts (this PR)
pairs, which include the genuine variants, are being triaged separately)
sourced vs vendor/Wikipedia pages, 157 needing a per-record ram/weight/battery
source, 158 different-chip pairs (some genuinely different devices to un-pair),
plus the release_date field on the 364 merged here.
Validation
python -m app.validate→ passed.integrity_check.py <data> --strict→ no hard anomalies (0 duplicate slugs,0 slug/file mismatches, 0 verified-without-source).
Dump
Last commit refreshes
site/public/v1from the merged data (python -m app.dump,no re-scrape): removes the 364 deleted slugs and reconciles score percentiles.
Closes #296