Repository navigation
Verification and TechEngineBot (workstream) #98
Description
Purpose
Workstream tracker for verification and TechEngineBot, under the roadmap #97. PRs in this area link here with Closes #<this>.
Scope
app/verify(offline scoring, signals, hosts, crossref, network verify, promotion ledger)integrity_check.py(structural checks for all categories; benchmark sanity for CPU/GPU)- TechEngineBot PR comments:
techapi-pr-validation-comment,techapi-verify-comment,/verify, dump check verify-network,verify-status,coverage-reportworkflows
Status (2026-09-27)
- Every category is scored since feat: analyze all dataset categories and actual PR diffs #95. Categories without domain rules (laptop, monitor, software, website) get structural checks only and can't be green.
- Verified: 11.5% of records, 16.3% of models, across TechAPI.
Open items
- Source qualifying T1/T2 citations for the remaining yellow laptop/monitor records: current Tier 0 laptop 107 green / 1,888 yellow / 0 red and monitor 0 green / 882 yellow / 0 red (feat(verify): add laptop and monitor domain scoring #102). Of the yellow records, laptop 0/1,888 and monitor 0/882 have a qualifying source_url; all 2,770 need sourcing (no pipeline gap found).
- Domain scoring rules (RICH_FIELDS and signals) for laptop and monitor.
- 18 pre-existing CPU anomalies from the full integrity scan.
- Next promotion pass (verify-network) after the release of #292 data.
Metadata
Metadata
Assignees
Type
Projects
- Status
Activity
Seungpyo1007 commentedon Sep 26, 2026
Reopened: this tracker was auto-closed by the
Closes #98keyword in #101's commit message. Tracker issues stay open. #101 (laptop/monitor Tier 0 normalization) is done: laptop 0/1,995/0 and monitor 0/882/0 (green/yellow/red), up from 1,888 and 882 red.feat(verify): add laptop and monitor domain scoring
feat(verify): add laptop and monitor domain scoring
feat(verify): score software and websites with batched Wikidata liveness
fix(verify): skip fully cached records before capping the check-urls …
fix(verify): keep the query in URL dedupe and cache every citation of…
Seungpyo1007 commentedon Sep 28, 2026
Live laptop/monitor audit against TechAPI develop
bf4a0597381cdf208c1fdeceedc1559f2b770bb7using TechEngine main303890871897f50bf214ca4943fb569b9073ddd6and the sameapp.verify score/statustooling used for #102. The old "red in Tier 0" checklist text is stale after #101/#102.This is a data gap: all 2,770 yellow records need qualifying citations. Among yellow laptops, 965 cite only
https://huggingface.co/datasets/Ammok/laptop_price_prediction(unclassified host) and 923 cite onlyhttps://github.com/abhinavflac/laptops-specs-dataset(T3); all 882 monitors cite onlyhttps://www.kaggle.com/datasets/durjoychandrapaul/amazon-products-sales-monitor-dataset(T3). These are existing dataset citations, not model-specific authoritative sources.verify-networkscores every category without a category filter;check-urlsandpromotealso default to every category, including laptop and monitor.promoterecomputesoffline.score_recordfor each unverified record and accepts live classified T1/T2 citations. None of these paths excludes laptops or monitors, and the green band still correctly requires a T1/T2 source. No sources were invented or reclassified, no data files or code were changed, and no code PR is warranted for this outcome.Before/after bands are unchanged: laptop 107/1,888/0; monitor 0/882/0 (green/yellow/red). Validation: full pytest suite 499 passed in 258.79s against live TechAPI develop data;
mypy appandruff check app testspass.Seungpyo1007 commentedon Sep 28, 2026
CPU integrity review, 2026-09-28
Full
integrity_check.pyrun: TechEngine main303890871897f50bf214ca4943fb569b9073ddd6, TechAPI developbf4a0597381cdf208c1fdeceedc1559f2b770bb7. Scope: 12/12 categories, 191,814 records, 4,507 CPUs, 2,076 GPUs. This uses the current full scan, with no legacy CPU/GPU-only mode.18 hard CPU anomalies remain, exactly matching the tracker count: 18 duplicate-name pairs (36 records). Additionally, 139 advisory CPU findings affect 113 other records: 8 era findings, 24 PassMark/R23 ratios, 11 PassMark/Geekbench ratios, 90 R23/Geekbench ratios, and 6 Cinebench 2024/R23 ratios. Total: 149 distinct flagged CPU records, represented below as 18 pair rows plus 113 advisory rows.
All flagged records were read, including populated fields, citations, and both sides of each duplicate. No data or check code changed, and no PR or dump refresh is warranted. This is an unresolved conflict/provenance report, not a declaration that the stored benchmark values are correct.
Hard findings: genuine duplicate records with conflicting source-derived fields. The
atom-*records originate in Wikipedia weekly ingest (4b49f32e464f);intel-atom-*originate in the Kaggle import (2b9e286d2bd3). The checker correctly detects the repeated exact names. Every pair disagrees on release date; many also disagree on threads, cache level, clocks, or segment. Merging by copying the richer record would silently choose between conflicts, which #296 forbids; previous #282 also left pairs with conflicting dates/specifications unresolved. Both records are retained.The cited Atom list labels cache as L2 and describes Hyper-Threading separately from the cores column. Existing Wikipedia-derived records put that cache in
l3_cache_mband often copy cores into threads; current ingestion already guards these cases inapp/ingest/sources/wikipedia_cpu.py/app/ingest/normalize.py. This is pre-existing ingestion data, not a bug inintegrity_check.py. For Z3530, Intel gives Q2 2014, not an exact day; substituting either stored full date would invent precision. The hard flag is duplicate identity/name, and no unambiguous source-backed keeper reconciliation was established while the paired source conflicts remain. Individual legacy field errors are documented rather than bundled into a partial merge.Values in the conflict column are ordered
atom-*/intel-atom-*. Common to all pairs: architecture text differs and populatedl3_cache_mbin the Wikipedia import versus null in the Kaggle import.Advisory findings: classification and limits. Era checks use only a year and a fixed R23 ceiling (3,000 before 2011), ignoring cores/threads. Ratio checks flag
abs(log(a/b) - median) > 4*MADacross the entire mixed catalog: the current ranges are PM/R23 0.8761 to 2.9344 (811 pairs), PM/GB 1.0213 to 4.3864 (693), R23/GB 0.9866 to 1.8817 (739), and CB2024/R23 0.04302 to 0.07201 (178). These are statistical review boundaries, not physical limits or scoring validity constraints. The unusedlo/hiarguments inmad_outliersare not applied ranges; changing them would not change the scan.For high-core records, different workload scaling is a plausible cause, not proof that a value is correct: Geekbench 6 internals, pp. 7 to 10, documents its shared-task model and weighted workload scores. Benchmark version, machine/power configuration, socket count, and result-level provenance are absent from these records. Generic vendor/spec/list citations cannot establish an exact Cinebench or Geekbench replacement, so these findings remain advisory rather than being relabeled confirmed false positives.
Primary-source spot checks: Opteron 6386 SE PassMark confirms stored 8,291; Xeon 6980P PassMark confirms stored 74,324 with one sample and high error. Neither resolves the other benchmark in the ratio. The cited 5950X Geekbench page currently shows 15,293 multi versus stored 12,018; its header and body mix version labels, so a live aggregate is not a version-locked correction. The cited 8452Y Geekbench page currently shows 0/0, not an authoritative replacement for stored 9,863. TopCPU could not be retrieved during review and is a ranking root, not an exact result URL. These source discrepancies/gaps stay unresolved.
Abbreviations: PM =
passmark_cpu_mark; R23 =cinebench_r23_multi; GB =geekbench_multi(record does not identify its version); CB24 =cinebench_2024_multi. Score tuples are PM / R23 / GB / CB24, with-for null. All ratio values below match the checker rounding.All 113 advisory CPU records (139 findings)
Validation:
python integrity_check.py DATA_ROOT --strict --hard-report hard-anomalies.jsoncompleted the full scan and exited 1, solely for the 18 duplicate-name hard anomalies. CPU name/tier and single>multi sections are clean; no CPU slug/filename, duplicate-slug, or verified-without-source findings. GPU advisory output was preserved but is outside this CPU task. No thresholds were widened or hard findings suppressed. Before/after CPU findings are unchanged; the 18 hard items in #98 remain open pending source-conflict reconciliation, and the advisory records need version/configuration-specific benchmark citations before correction.Seungpyo1007 commentedon Sep 28, 2026
The 18 hard Atom CPU duplicate-name anomalies reported earlier (integrity_check.py --strict) are resolved: GetTechAPI/TechAPI#321 merges each pair into one Wikipedia-verified record. `integrity_check.py --strict` now reports 0 hard anomalies. This was also the root cause of weekly-refresh's last 5-6 failures/cancellations — it should be able to complete once this merges.
Seungpyo1007 commentedon Sep 29, 2026
CPU advisory re-check, 2026-09-29
Prompted by today's successes elsewhere (Atom/tablet/watch/smartphone dedups that found real, fixable problems a prior pass had written off), I took a deliberately fresh, source-first pass at the 139 advisory CPU findings rather than trusting the earlier "pre-existing" classification. Fresh
integrity_check.pyrun: TechEngineSeungpyo1007/cpu-advisory-recheck(checker unchanged from main), TechAPI develop2360ec8cb17c, in a dedicated TechAPI worktree. Scope reproduced exactly: 12/12 categories, 191,710 records, CPU=4,628, GPU=2,083; 139 advisory CPU findings across 113 records (8 era, 24 PM/R23, 11 PM/GB, 90 R23/GB, 6 CB2024/R23). No hard anomalies (the 18 Atom dups were resolved by GetTechAPI/TechAPI#321).Outcome: after genuine re-verification, the "pre-existing" conclusion holds. No advisory record has a citable, authoritative replacement value, so no data PR is warranted (opening one would mean inventing values, which #296 forbids). What follows is the new primary-source work behind that, kept distinct from "I didn't look hard enough."
Abbreviations: PM =
passmark_cpu_mark, R23 =cinebench_r23_multi, GB =geekbench_multi(GB6 by the ingest convention inapp/ingest/sources/topcpu.pyandnotebookcheck.py), CB24 =cinebench_2024_multi.Source accessibility (what is and isn't citable right now)
browser.geekbench.com) returns 403, and its aggregate isn't version-locked anyway (matchesapp/verify/http_check.py).So for the R23 and GB sides of the ratio pairs there is generally no fetchable, version-locked, per-SKU source. That's the honest ceiling, not a shortcut.
PM verification (the citable half): stored values confirmed correct
I re-pulled PM for the extreme PM-pair outliers directly from cpubenchmark.net:
These, plus the prior pass's Opteron 6386 SE (8,291) and Xeon 6980P (74,324) confirmations, mean the PM side of the flagged EPYC/Xeon ratios is right. The PM/GB and PM/R23 flags on these parts come from the other side: GB6's multi-core score does not scale to 64–256-core server parts (its shared-task model saturates — Geekbench 6 internals, pp. 7–10), while R23 and PM keep scaling. That is a real methodology divergence, not a corrupt value.
The 90-record R23/GB cluster is a benchmark-methodology signature, not contamination
The R23/GB flags are almost perfectly monotonic in core count: high-core parts (Threadripper/EPYC/Xeon Platinum/Max, i9-X) cluster at ratio ~1.9–2.6; low-core parts (i3/Pentium/Celeron/dual- and quad-core mobile) at ~0.8–0.99. R23 multi keeps scaling with cores; GB6 multi compresses. A MAD test over a catalog that mixes 2-core and 288-core parts will light these up. This is exactly what the checker docstring already anticipates ("a heterogeneous catalog … legitimately produces many ratio outliers"), now confirmed against the live data rather than assumed.
Era findings (8) are checker false positives, and worth fixing in the checker separately
The era rule
year<2011 and r23_multi>3000uses release year and a fixed ceiling only — it ignores cores/threads. But Cinebench R23 runs on the physical silicon regardless of when the chip launched, so a 6-core/12-thread Gulftown legitimately posts ~5,000–6,500 R23 today, and a 4c/8t Bloomfield ~3,000–3,800 (consistent with published R23 tables, e.g. cgdirector R23 results, and cpu-monkey's per-chip pages for i7-980X / i7-920). All 8 era rows (i7-920/965/870/970/980X/990X, Phenom II X6 1090T/1100T) are pre-2011 high-core enthusiast chips — precisely the population this heuristic mis-flags. This is a checker limitation, not a data error. Filed as a separateintegrity_check.pynote under this workstream rather than a data change; a cores-aware era bound (or dropping the year gate for R23) would remove these false positives without weakening the wrong-variant detector.CB2024/R23 findings (6): individually-correct values sitting at a tight distribution edge
I could source-lock one of these end-to-end via NanoReview (which cites its Cinebench/PassMark sources with sample counts): Core i9-13900F — CB2024 multi 1601 vs stored 1622 (correct); its R23 39,200 and PM 48,155-vs-stored-48,391 are also correct for a 24-core Raptor Lake. Both sides of the CB24/R23 ratio are individually right; the flag is only because the whole-catalog CB24/R23 band is narrow (0.043–0.072) and this chip sits at its edge. No fixed CB24↔R23 conversion exists, so an "outlier" here isn't evidence of a wrong number.
Genuinely suspicious, but not fixable without inventing
Three records look internally off, and I want to be explicit that these are "checked, no citable authoritative source available," not "skipped for effort":
geekbench_single1051,geekbench_multi9863; R23/GB=5.48). The stored single of 1051 matches the Geekbench 5 single (cputronic's structured data: GB5 1049/2072), i.e. this is very likely a GB5 value mislabeled into the GB6 field. But the cited Geekbench page is 403 / shows 0/0, and cputronic's "GB6" prose is self-contradictory and unusable. There is no fetchable, version-locked GB6 figure to replace it with, so it stays as-is.I did not blanket-relabel these as "correct" — they are plausible data errors I couldn't correct within the rule "never invent a value; cite the evidence URL."
Bottom line
The earlier conclusion was right, and this pass upgrades it from "left as pre-existing" to source-verified where a citable source exists: PM values on the flagged EPYC/Xeon parts confirmed exact on cpubenchmark.net; the R23/GB and PM/GB clusters explained as GB6-scaling methodology, not contamination; the 8 era findings identified as a fixable checker heuristic bug; one CB24/R23 case (i9-13900F) confirmed correct end-to-end. A small residual (8452Y GB5-mislabel; 5500GT and i5-1140G7 low R23) is genuinely suspect but has no accessible authoritative source for a corrected value, so per #296's rules it stays documented rather than guessed. No data files changed; no dump refresh needed.
fix(verify): make era advisory rule cores-aware
fix(verify): make cross-source ratio outliers cores-aware
Seungpyo1007 commentedon Sep 29, 2026
Verified-promotion re-check after the 2026-09-29 data fixes
Re-ran the Tier 3 promotion step against a fresh
developcheckout (synced to8b2811d, which includes today's CPU/GPU/smartphone/laptop/monitor/PDA and citation-hunting passes) to see whether any of today's source/data fixes pushed borderline records over the verification threshold.Dry-run result (
python -m app.verify promote, dry-run is the default):0 records are newly promotable from a standalone promote run. 518 records score green offline but are blocked as
green-needs-live-t1— they need a live Tier-1 authoritative source confirmed in the URL-liveness cache beforepromotewill flip them.Why I did not apply anything. The standalone
promotestep can't reach eligibility on its own: promotion depends on the Tier-1 URL-liveness cache and the Tier-2 crossref cache, which are (re)built by thecheck-urls/check-wikidata/crossrefsteps run immediately before promotion. Those network steps hit rate-limited external sites, and the established workflow (.github/workflows/verify-network.yml) is explicit that they run only in that controlled, scheduled environment ("NEVER on pull_request — external sites, rate-limited"), commit as TechEngineBot, and open a human-gated PR on TechAPIdevelopguarded so onlyverifiedtoggles change. That pipeline — not an ad-hoc localpromote --apply— is how promotions are meant to land, so forcing a flip here would be wrong on three counts: it bypasses the established actor/guard/review path, it would run against caches not freshly re-confirmed against today's changedsource_urls, and the dry-run shows nothing eligible to apply anyway. Averified:false→trueflip is a trust signal that's expensive to earn back, so I'm reporting rather than forcing it.Recommendation. To actually convert any of the 518 green-blocked records off the back of today's fixes, run the full
verify-networkworkflow (workflow_dispatchwithapply=true, or wait for the Monday 04:00 UTC schedule). That will re-check source liveness against the currentsource_urls, cross-reference, and — if anything genuinely clears the bar — open the usual TechEngineBot promotion PR for review.Refs #98
feat(verify): gate new-device candidates before they enter device_cat…
fix(verify): reject maker marketing shots in the Commons photo backfill