Skip to content

Ingest and sources (workstream) #99

Open

Description

@Seungpyo1007

Purpose

Workstream tracker for ingest and sources, under the roadmap #97. PRs in this area link here with Closes #<this>.

Scope

  • app/ingest collectors and parsers (Wikipedia CPU/GPU/smartphone tables, PassMark, notebookcheck, topcpu, SPEC, technical.city, cgdirector)
  • app/ingest/enrich.py benchmark backfill runner (fill-only-nulls)
  • weekly-ingest workflow (Monday, opens ingest/* PRs to TechAPI develop)

Rules

  • Never invent values. A parser that can't read a field leaves it null; e.g. a bare "Cores" column doesn't imply a thread count (Parse multi-row headers in the Wikipedia GPU ingest; stop guessing threads #93).
  • Dedup new slugs against existing ones in both directions and ignoring punctuation.
  • Source policy follows TechAPI SPEC.md: GSMArena and PhoneDB only for corrections; Namuwiki banned; no bypassing anti-bot protection (Cloudflare, WAF).

Open items

  • Wikipedia AMD Ryzen/EPYC list parsers (models live in column 1 under a family column 0); the importers currently live outside the engine.
  • Transposed spec tables (MacBook/Surface-style "model per column") as a reusable parser.
  • DX12 guard for topcpu GPU scores by architecture, not year (Time Spy estimates on TeraScale/Tesla-era cards).
  • Xeon thread counts for about 390 rows (Intel spec pages; the tables list cores only).

Activity

  1. Seungpyo1007 commented on Sep 26, 2026

    @Seungpyo1007
    MemberAuthor

    New open item (2026-09-27): AMD legacy CPU list pages need a dedicated parser. A dry run of WikipediaCpuIngest on 15 AMD list pages (Athlon/Athlon 64/X2/XP, Duron, FX, K5/K6, Opteron, Phenom, Sempron, Turion, mobile, APUs) yields ~87 "complete" rows, but the values are wrong:

    • Voltage cells are read as core/thread counts (1.25 V → cores 1, threads 25).
    • Bulldozer/Piledriver modules (cores) is read as cores (threads) (FX-8350 → 4C/8T).
    • Most rows on Opteron, Phenom, Turion, Duron, K6 and the APU list miss cores entirely (~1,100 rows).

    Nothing was imported. Needed: per-page column mapping (Cores/Modules/Threads headers), a voltage-column exclusion, and a module-aware core count.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

enhancementNew feature or requesttrackingLong-running tracker issue (stays open)

Type

No type

Projects

Relationships

None yet

Development

No branches or pull requests

Issue actions