Skip to content

TechAPI dataset roadmap and status: all categories (1989-2026) #1

Open
2 of 2 issues completed
Open

Description

@Seungpyo1007

Purpose

Roadmap and status board for the TechAPI dataset: every category from 1989 to 2026, published as a static JSON API at https://gettechapi.github.io/TechAPI/.

This issue stays open. The body is the current status, updated each release. Comments are one summary per release. Day-to-day work is tracked in the workstream issues below and on Project #2.

How work is tracked (since 2026-09-27)

Workstream Issue Which PRs link there
Roadmap & releases #1 (this issue) develop → main release PRs and the batched dump PRs
New records #295 feat(data…): new CPUs/GPUs/SoCs/devices, benchmark backfill, new sources
Data accuracy #296 fix(data…): corrections, duplicate merges, dates, placeholder cleanup
Engine, bot & automation #297 ci/app PRs in TechAPI; TechEngine PRs reference GetTechAPI/TechAPI#297
Site #19 Homepage and site PRs

pr-metadata adds the matching Closes #N line automatically. Auto-close is disabled, so trackers stay open. Project #2 has a Workstream field with the same split.

Current status (release #315, 2026-09-27)

Category Records Verified Verified % Notes
smartphone 93,407 17,581 18.8% 39,622 models; Wikipedia/Wikidata matching confirms only ~1.5% of Tier-3-only models
software 42,493 358 0.8% Domain scoring since TechEngine #103; most records lack dates/developers even on Wikidata
website 40,084 7,635 19.0% Domain scoring + Wikidata liveness since TechEngine #103
cpu 4,507 4,211 93.4%
tablet 3,459 288 8.3%
soc 2,117 280 13.2% 996 red = placeholder imports with no failing check (weak source only)
gpu 2,076 1,493 71.9%
laptop 1,995 105 5.3% Domain scoring since TechEngine #102
monitor 882 0 0.0% Domain scoring since TechEngine #102; no T1/T2 sources yet
watch 447 94 21.0%
brand 207 123 59.4%
pda 140 74 52.9%
devices (excl. software, website) 109,237 24,249 22.2% Comparable to the pre-09-27 figure (20.8%)
all 12 categories 191,814 32,242 16.8%

Two figures are shown because software and website (82,577 records) entered the denominator on 2026-09-27. Their verification is a same-source check: the cited Wikidata item still exists and the record passes domain rules. Most software records are too sparse for green, even after filling from Wikidata. Verified records have only grown.

Goals

  1. Verification: 30% of records in the scored categories (smartphone, tablet, watch, pda, cpu, gpu, soc, brand); laptop and monitor scoring is done (TechEngine data: add laptop category — 970 laptops (first non-mobile class) #102).
  2. Coverage: keep new releases (2025-2026 phones, SoCs, CPUs, GPUs, laptops, watches) within one release of launch, when a citable source exists.
  3. Accuracy: no known impossible values (storage-as-RAM, placeholder dates, estimate-only benchmarks); duplicate-free slugs.
  4. Automation: every PR is checked on all 12 categories; weekly refresh and ingest run green.

Recent releases

Release Date Main content
#315 2026-09-27 8,839 verified promotions (first software), Wikidata-filled software/website fields, check-urls fixes
#308 2026-09-27 Dump with 312 accuracy fixes (storage, battery, monitor fields, SoC years), watch SoC links
#304 2026-09-27 Wearable SoCs, Galaxy Watch9 / Ultra 2, Pixel Watch 4, laptop/monitor scoring, 1,306 promotions
#292 2026-09-26 Galaxy Tab S11 / iPad Pro M5, 13 Surface laptops, benchmark backfill (~1,900 records), 182 SoC dates, GPU duplicate merge
#284 2026-09-25 2025-26 watches and phones, 83 Xeon thread counts, RAM and duplicate fixes
#274 2026-09-25 470 new CPUs/GPUs/SoCs/MacBooks, Intel and GPU cross-check fixes
#252 2026-09-25 SoC dedup, Commons images, tablet citations, 146 verified promotions
#249 2026-09-25 TechPicks image fixes, base_model_slug fill

Data rules (summary)

  • Every record cites source_urls. Nothing is invented; if no source states a value, it stays null.
  • Bulk imports land as verified: false. The verify layer promotes a record only when a live T1/T2 source agrees.
  • Sources: Wikipedia, Wikidata, manufacturer pages and CC datasets are fine. GSMArena and PhoneDB may only be used to correct existing values (ADR-013). Namuwiki is banned. DeviceSpecifications is paused.
  • PRs target develop. Before each release, develop gets one batched dump (site/public/v1); main only moves through release PRs.

History

This issue began on 2026-05-28 as the "Massive dataset rebuild" tracker for 8 categories (51,783 records at PR #36). The original description is preserved in the comment that announced this restructure.

Activity

  1. Seungpyo1007 commented on May 28, 2026

    @Seungpyo1007
    MemberAuthor

    Rolled out in 11 commits on main

    # Commit Subject
    1 4dc4b57 chore(scripts): widen validator bounds and refresh seed loader
    2 9708ee3 feat(app): expand GPU model and schema for richer hardware specs
    3 7a9df9e feat(data/brand): restructure into country folders and expand catalog
    4 de9b32c feat(data/cpu): expand AMD lineup from K5 to Zen 6 with segment split
    5 b08e2cf feat(data/cpu): expand Intel lineup from i486 to Panther Lake with segment split
    6 9781670 feat(data/cpu): add IBM POWER, z mainframe, PowerPC, Cell, Motorola G3/G4
    7 5cd4b5b feat(data/cpu): add historic and niche vendors across 11 manufacturers
    8 2a434bf feat(data/gpu): rebuild discrete GPU catalog across 10 vendors with segment split
    9 fc7bcd1 feat(data/smartphone): rebuild smartphone catalog across major brands
    10 54306c8 feat(data/soc): rebuild SoC catalog covering Apple, Qualcomm, MediaTek, Samsung, Google, HiSilicon
    11 f64a74a docs: update README and data/README for expanded dataset and new layout

    (The site homepage update was already on main as ef15e00 feat(site): add interactive API playground and redesign homepage, so the rebase deduplicated it.)

    Result

    • ✅ python -m scripts.validate PASS across brand / cpu / gpu / smartphone / soc
    • 464 CPU records, 330 GPU records, ~366 smartphone records, ~100+ SoC records, 129 brand records
    • New layouts: brand/<country>/<slug>.json + cpu/<mfr>/<year>/<consumer|enterprise>/<slug>.json
    • Tracked in Project: TechAPI Dataset Rebuild
  2. 601 remaining items

  3. github-actions commented on Sep 26, 2026

    @github-actions
    Contributor

    Linked PR update: Add 13 Microsoft Surface laptop models (2015-2023) from Wikipedia #286

    Changed Data

    Category Added Modified Deleted
    none 0 0 0

    Changed Areas

    Area Files
    data 13

    Notes

    • This is an automatic tracking comment for the long-running issue.
    • PR validation details are posted on the PR by TechEngineBot.
    • The tracker issue remains open even when the PR uses Closes #... for GitHub Development linking.
  4. github-actions commented on Sep 26, 2026

    @github-actions
    Contributor

    Linked PR update: Fix 65 SoC release dates from Wikipedia launch tables #287

    Changed Data

    Category Added Modified Deleted
    soc 0 65 0

    Changed Areas

    Area Files
    data 65

    Notes

    • This is an automatic tracking comment for the long-running issue.
    • PR validation details are posted on the PR by TechEngineBot.
    • The tracker issue remains open even when the PR uses Closes #... for GitHub Development linking.
  5. github-actions commented on Sep 26, 2026

    @github-actions
    Contributor

    Linked PR update: Backfill benchmark scores for ~1,000 CPUs and ~900 GPUs #288

    Changed Data

    Category Added Modified Deleted
    cpu 0 500 0

    Changed Areas

    Area Files
    data 500

    Notes

    • This is an automatic tracking comment for the long-running issue.
    • Large PR detected: changed-area tables are based on the first sampled files, while total file/line counts come from GitHub PR metadata.
    • PR validation details are posted on the PR by TechEngineBot.
    • The tracker issue remains open even when the PR uses Closes #... for GitHub Development linking.
  6. github-actions commented on Sep 26, 2026

    @github-actions
    Contributor

    Linked PR update: Fix 117 MediaTek, Exynos and Kirin SoC release dates #289

    Changed Data

    Category Added Modified Deleted
    soc 0 117 0

    Changed Areas

    Area Files
    data 117

    Notes

    • This is an automatic tracking comment for the long-running issue.
    • PR validation details are posted on the PR by TechEngineBot.
    • The tracker issue remains open even when the PR uses Closes #... for GitHub Development linking.
  7. github-actions commented on Sep 26, 2026

    @github-actions
    Contributor

    Linked PR update: fix(data): resolve GPU conflicts against manufacturer specifications #290

    Changed Data

    Category Added Modified Deleted
    gpu 0 3 0

    Changed Areas

    Area Files
    data 3

    Notes

    • This is an automatic tracking comment for the long-running issue.
    • PR validation details are posted on the PR by TechEngineBot.
    • The tracker issue remains open even when the PR uses Closes #... for GitHub Development linking.
  8. github-actions commented on Sep 26, 2026

    @github-actions
    Contributor

    Linked PR update: chore(dump): refresh static API for 2026-09-26 data #291

    Changed Data

    Category Added Modified Deleted
    none 0 0 0

    Changed Areas

    Area Files
    site 500

    Notes

    • This is an automatic tracking comment for the long-running issue.
    • Large PR detected: changed-area tables are based on the first sampled files, while total file/line counts come from GitHub PR metadata.
    • PR validation details are posted on the PR by TechEngineBot.
    • The tracker issue remains open even when the PR uses Closes #... for GitHub Development linking.
  9. github-actions commented on Sep 26, 2026

    @github-actions
    Contributor

    Linked PR update: release: tablets, Surface laptops, benchmark backfill, SoC dates (2026-09-26) #292

    Changed Data

    Category Added Modified Deleted
    cpu 0 499 0

    Changed Areas

    Area Files
    data 500

    Notes

    • This is an automatic tracking comment for the long-running issue.
    • Large PR detected: changed-area tables are based on the first sampled files, while total file/line counts come from GitHub PR metadata.
    • PR validation details are posted on the PR by TechEngineBot.
    • The tracker issue remains open even when the PR uses Closes #... for GitHub Development linking.
  10. Seungpyo1007 commented on Sep 26, 2026

    @Seungpyo1007
    MemberAuthor

    2026-09-26 progress

    New records (collection)

    Corrections (accuracy)

    Release

  11. github-actions commented on Sep 26, 2026

    @github-actions
    Contributor

    Linked PR update: fix(data): correct storage-as-RAM on 12 older devices from sourced specs #293

    Changed Data

    Category Added Modified Deleted
    smartphone 0 35 0
    watch 0 2 0

    Changed Areas

    Area Files
    data 365

    Notes

    • This is an automatic tracking comment for the long-running issue.
    • PR validation details are posted on the PR by TechEngineBot.
    • The tracker issue remains open even when the PR uses Closes #... for GitHub Development linking.
  12. github-actions commented on Sep 26, 2026

    @github-actions
    Contributor

    Linked PR update: ci: send the PR base to TechEngine checks and stop tracking scores.jsonl #294

    Changed Data

    Category Added Modified Deleted
    none 0 0 0

    Changed Areas

    Area Files
    .github 3
    .gitignore 1
    data 1

    Notes

    • This is an automatic tracking comment for the long-running issue.
    • PR validation details are posted on the PR by TechEngineBot.
    • The tracker issue remains open even when the PR uses Closes #... for GitHub Development linking.
  13. Seungpyo1007 commented on Sep 26, 2026

    @Seungpyo1007
    MemberAuthor

    2026-09-27: PR/issue/automation audit and fixes

    Audit: TechAPI PRs #173–#292, all TechEngine PRs and issues, Project #2 across the org, and TechEngineBot coverage.

    TechEngineBot coverage (TechEngine #95 + TechAPI #294)

    • Before:
      • The integrity gate checked CPU/GPU only: 6,583 of 191,797 records (3.4%).
      • Comments and scores covered 8 of 12 categories (55%). Laptop, monitor, software and website were missing.
      • Release PRs reported "no data changes" because the diff base was hard-coded.
      • Dump PRs weren't checked at all.
    • After:
      • Every comment states its scope: 12/12 categories · 191,797 records · diff base develop@<sha>.
      • Integrity structure checks run on all categories; the PR's real base is used; dump PRs get a JSON/count check.
      • Categories without domain rules no longer get free completeness credit.
    • data/_verify/state/scores.jsonl (43.5 MB generated cache) is no longer tracked.

    Rules and process

    Data

  14. Seungpyo1007 commented on Sep 26, 2026

    @Seungpyo1007
    MemberAuthor

    Tracking restructured (2026-09-27)

    This issue is now the roadmap and status board. The body above is rewritten and will be updated each release; comments here are one summary per release. Ongoing work moves to the workstream issues:

    Existing PR links and comments stay as they are. pr-metadata will route new PRs to the matching issue.

    Original description (2026-05-28, last edited before PR #36)

    Purpose

    Long-running tracker for the TechAPI dataset rebuild across brand, cpu, gpu, smartphone, soc, tablet, watch, and pda.

    This issue intentionally stays open while bulk imports, validation hardening, public dump refreshes, and manual verification work continue. PRs may use Closes #1 so GitHub Development links the PR back to this tracker. Repository auto-close is disabled for this workflow, so merging linked PRs should not finish this issue.

    Current Status

    Latest Data Snapshot

    Category Total Verified Unverified Missing verified Verified %
    brand 189 0 60 129 0.0%
    soc 2,079 58 2,021 0 2.8%
    smartphone 42,051 184 41,867 0 0.4%
    tablet 1,171 0 1,171 0 0.0%
    watch 176 0 176 0 0.0%
    pda 110 0 110 0 0.0%
    gpu 2,030 0 2,030 0 0.0%
    cpu 3,977 976 3,001 0 24.5%
    all 51,783 1,218 50,436 129 2.4%

    Recent PR History

    PR Status Main change
    #36 Open Import Phones 2024 smartphones, tablets, watches, and refresh public dump
    #35 Merged Import Global Smartphone Database variants, add SoC stubs, normalize duplicate brand slugs, and refresh public dump
    #34 Merged Import GSMArena Kaggle smartphones, tablets, watches, and refresh public dump
    #33 Merged Import PhoneDB/Kaggle smartphone variants, tablets, SoC stubs, and public dump refresh
    #32 Merged Add tablet/watch/PDA/API/site category support and prior mobile dump refresh
    #25 Merged Add 5,000 PhoneDB raw smartphone variants plus 45 Mobiles 2025 records
    #24 Merged Add smartphone and SoC records, improve PR metadata and project automation
    #23 Merged Import a larger smartphone batch
    #22 Merged Add smartphone and SoC records from Kaggle-derived sources
    #17 Merged Expand GPU imports and public data refresh
    #16 Merged Expand CPU imports and public data refresh

    Sources Currently Used

    Validation Policy

    Every data PR should include TechEngineBot comments for:

    • Changed data summary: added, modified, deleted, verified/unverified source counts, and examples
    • Validation stats: category totals, verified coverage, warning callouts, and key validation output
    • Checks: python -m app.validate, python integrity_check.py TechAPI/data --strict, and site build when site files change
    • Heuristic review: naming, typo-like patterns, duplicate-looking fields, and data-quality warnings

    Low verified coverage is allowed for bulk import PRs, but should be called out as a follow-up warning instead of failing validation.

    Remaining Work

    • Continue large unverified imports where source coverage is useful
    • Rebase data/import-staging before each push and keep commits split by source, brand, era, or category
    • Backfill manual verification for imported smartphone, tablet, watch, PDA, GPU, CPU, and SoC records
    • Add or repair brand verified flags so the brand category no longer has missing verification metadata
    • Dedupe or collapse raw mobile variants where a source creates excessive regional/storage duplicates
    • Improve source attribution and audit notes for records imported from broad datasets
    • Keep public v1/index.json and category dumps refreshed after each data batch

    Operational Notes

    • Assignees: @Seungpyo1007 and @TechEngineBot
    • Labels: data, enhancement
    • Milestone: Massive dataset rebuild (1989-2026)
    • Projects: TechEngine work and TechAPI-Project
    • Priority: High for bulk data PRs
    • Start date: 2026-06-20
    • Target date: 2026-09-30

    TechEngineBot should add or update a tracking comment on this issue whenever a linked data PR is opened or synchronized.

    Latest linked PR: #36

  15. Seungpyo1007 commented on Sep 26, 2026

    @Seungpyo1007
    MemberAuthor

    Release 2026-09-27 ① (#304): deployed and live.

  16. Seungpyo1007 commented on Sep 27, 2026

    @Seungpyo1007
    MemberAuthor

    Release 2026-09-27 ② (#308): 1,306 verified promotions published, and the static API now carries the #300–#302 and #306 accuracy fixes. Verified: 21.4% of device categories and 12.2% of all 12 categories (table in the body).

  17. Seungpyo1007 commented on Sep 27, 2026

    @Seungpyo1007
    MemberAuthor

    Release 2026-09-27 ③ (#315): 8,839 verified promotions (websites 7,635, the first 358 software, 762 CPUs), Wikidata-filled software/website fields, and check-urls fixes. Verified: 22.2% of device categories and 16.8% of all 12 categories (table in the body).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

dataDataset changesenhancementNew feature or requesttrackingLong-running tracker issue (stays open)

Projects

Relationships

None yet

Development

No branches or pull requests

Issue actions