Repository navigation
fix(data): merge 29 confirmed laptop near-duplicate records - #332
Merged
Merged
Conversation
Each removed record was a byte-identical re-import of a device that already exists under a different Ammok row-id slug (Pattern 1: same scrape imported twice). Within each group the two files differed only in `slug` and `variant.source_row`; every real spec field (name, brand, cpu, release_date, device_category, cpu_name, gpu_name, gpu_type, ram_gb, storage_gb, display, weight_g, os) plus `source_urls` matched exactly, so the merge kept the file with the lower `source_row` and dropped the redundant one. `source_urls` were identical, so the union is a no-op and no field values were altered or invented. Configuration variants (same model, different CPU/RAM/storage) were left untouched -- those are distinct real SKUs, not duplicates. 29 groups across acer(1), dell(13), hp(3), lenovo(8), msi(2), toshiba(2). 1995 -> 1966 laptop records.
Regenerated the static laptop dump from the deduped dataset (python -m app.dump against a fresh scratch database seeded from this worktree). Drops the 29 orphaned per-slug pages for the removed duplicates and updates the laptop collection count in the list and manifest (1995 -> 1966). Scope limited to the laptop collection touched by this change.
github-actions
Bot
added
bug
Member
🔎 Data verification — Tier 0 (offline existence/trust)Scored by 12/12 categories ? 191,790 records ? diff base develop@ea3f31e Changed records in this PRNo records scored. Full-dataset baseline191790 record(s) assessed. %%{init: {"theme":"base","themeVariables":{"pie1":"#3fb950","pie2":"#d29922","pie3":"#f85149","pieStrokeWidth":"0px","pieOpacity":"1"}}}%%
pie showData
title Verification bands — all records
"Green" : 32772
"Yellow" : 157963
"Red" : 1055
Hard violations (forced red):
green = authoritative source + complete + consistent · yellow = plausible, needs confirmation · red = sparse/weak source or a hard contradiction. Promotion to |
Member
TechEngine change review: PASS12/12 categories ? 191,790 records ? diff base develop@ea3f31e
Changed data
Changed record exampleslaptop deleted
Heuristic review
|
Member
TechEngine validation stats: PASS12/12 categories ? 191,790 records ? diff base develop@ea3f31e
Data summary
Warning Tracked verified coverage is below 50% for monitor 0.0% (0/882), software 0.8% (358/42493), laptop 5.3% (105/1966), tablet 8.3% (288/3459), soc 13.4% (284/2117), all 16.8% (32257/191790), smartphone 18.8% (17581/93407), website 19.0% (7635/40084), and 1 more. Validation notes
Key output:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Scanned
data/laptop/(1,995 records) for near-duplicates using the same method as the tablet/watch pass (#324) and smartphone pass (#329): slug/name normalization (punctuation- and brand-prefix-agnostic) plus a spec cross-check (release_date, cpu ref, cpu_name, ram_gb, storage_gb, display size/resolution, gpu, weight, os). Merged 29 confirmed duplicate groups and left all configuration variants untouched.1995 → 1966laptop records.Per-pattern breakdown
Pattern 1 — identical scrape imported twice under different row-id slugs: 29 groups (all merged).
Each group was two files that were byte-identical except
slugandvariant.source_row— i.e. the same device row imported twice fromAmmok/laptop_price_predictionunder two different row ids. Merge rule: keep the file with the lowersource_row(deterministic), delete the other.source_urlswere identical in every group, so the union is a no-op; no field value was changed or invented.Groups by brand: acer (1), dell (13), hp (3), lenovo (8), msi (2), toshiba (2).
All 29 merged groups (kept ← removed)
acer-aspire-7-ammok-211acer-aspire-7-ammok-380dell-alienware-17-ammok-531dell-alienware-17-ammok-553dell-alienware-17-ammok-842dell-alienware-17-ammok-969dell-inspiron-7567-ammok-152dell-inspiron-7567-ammok-774dell-xps-15-ammok-187dell-xps-15-ammok-351dell-xps-15-ammok-294dell-xps-15-ammok-630dell-inspiron-5370-ammok-72dell-inspiron-5370-ammok-216dell-inspiron-5570-ammok-28dell-inspiron-5570-ammok-537dell-inspiron-5570-ammok-356dell-inspiron-5570-ammok-427dell-inspiron-5570-ammok-514dell-inspiron-5570-ammok-658dell-inspiron-5570-ammok-73dell-inspiron-5570-ammok-657dell-inspiron-5770-ammok-61dell-inspiron-5770-ammok-129dell-inspiron-5770-ammok-190dell-inspiron-5770-ammok-261dell-xps-13-ammok-209dell-xps-13-ammok-345hp-elitebook-x360-ammok-881hp-elitebook-x360-ammok-886hp-probook-430-ammok-170hp-probook-430-ammok-249hp-probook-470-ammok-30hp-probook-470-ammok-80lenovo-ideapad-320-17ikb-ammok-281lenovo-ideapad-320-17ikb-ammok-558lenovo-legion-y720-15ikb-ammok-204lenovo-legion-y720-15ikb-ammok-233lenovo-legion-y720-15ikb-ammok-296lenovo-legion-y720-15ikb-ammok-521lenovo-thinkpad-t470s-ammok-723lenovo-thinkpad-t470s-ammok-894lenovo-thinkpad-x270-ammok-650lenovo-thinkpad-x270-ammok-927lenovo-thinkpad-yoga-ammok-597lenovo-thinkpad-yoga-ammok-793lenovo-yoga-910-13ikb-ammok-245lenovo-yoga-910-13ikb-ammok-799lenovo-yoga-920-13ikb-ammok-183lenovo-yoga-920-13ikb-ammok-344msi-ge72mvr-7rg-ammok-149msi-ge72mvr-7rg-ammok-812msi-gl72m-7rex-ammok-331msi-gl72m-7rex-ammok-725toshiba-satellite-pro-ammok-144toshiba-satellite-pro-ammok-807toshiba-satellite-pro-ammok-174toshiba-satellite-pro-ammok-838Pattern 2 — same device in two sibling dirs via brand-prefix path variance (e.g.
dell-xps-13vsxps-13): none found.Every record is stored under a consistent brand-prefixed
base_model_slug; there were zero cases of the same brand+name living under two different base directories.Pattern 3 — cross-source duplicates under different slug conventions: none found.
No device name appears in both bulk datasets (
Ammok/laptop_price_predictionandabhinavflac/laptops-specs-dataset). The two datasets cover disjoint eras/models. The only cross-checkable identical-spec collisions were different SKUs that happen to share the Ammok dataset's coarse specs (e.g.Acer Aspire ES1-572vsAcer Extensa EX2540) — those were correctly not merged because their names differ.Placeholder / fabricated-defaults check
No systemic placeholder-value bug (the equivalent of the phone
battery_mah=1500/weight_g=150.0stamping) is present in laptop data.ram_gb,storage_gb,weight_g, anddisplay.size_inchall show organic distributions with distinct-name diversity proportional to frequency (e.g.ram_gb=16across 735 distinct model names;storage_gb=512across 739; the most commonweight_g=2200.0spans 41 distinct models). No single value is stamped across many unrelated devices. The two lowest-diversity weight values (1860.0and2180.0, ~3 distinct names each) are simply config variants of a few models, not fabricated defaults.Conflicts found / resolved
None. Because every merged pair was byte-identical outside
slug/variant.source_row, there were no field disagreements to reconcile — nothing was repaired, so no source citations were required beyond the existingsource_urls(which matched within each pair).Validation
python -m app.validate→ passed.python integrity_check.py <data> --strict→ no hard anomalies (only the pre-existing advisory CPU/GPU benchmark ratio outliers, which the gate treats as review-only).python -m app.dumprun from the TechEngine checkout against a fresh scratch database withTECHAPI_DATA_DIRpointing at this worktree. Scope kept to the laptop collection (removed the 29 orphaned per-slug pages; updated the laptop list + manifest count1995 → 1966). Unrelated pre-existing dump drift in other collections was deliberately excluded to keep the diff reviewable.Closes #296