feat(evalboard): multi-harness colour-coded charts + generated cost table by uipreliga · Pull Request #70 · UiPath/coder_eval

uipreliga · 2026-07-30T23:53:41Z

Goal

Make the overview charts plot every harness at once by default with one colour-coded line each (clicking a harness name scopes the analytics block to it), and fix the evalboard's drifted cost mirror so displayed USD is correct and cannot silently drift again.

Six phases, one commit each, plus a review-fix commit and a deferred-guards commit.

The cost bug this fixes

lib/pricing.ts declared itself a hand-copy of src/coder_eval/pricing.py and had drifted three ways:


7 models carried WRONG rates	Opus 4.6/4.7/4.8 read Opus 4.1's legacy $15/$75, overstating cost 3× on every rendered estimate. Haiku 4.5 read Haiku 3.5's $0.80/$4. `claude-opus-4-6` appears in real run data.
17 priced models were missing	rendered "—" for cost
2 entries were dead	absent from the Python table, and redundant given the undated fallback

The DELIBERATELY_UNMIRRORED allowlist meant to hold the line had failed twice over: 17 models drifted past it, and 2 of its own 8 entries were models real runs use (gpt-5.6-terra is the single most-run model in the recorded data), so it was actively suppressing cost for runs the board does execute.

Rather than hand-copy better — which would convert a one-time copy error into a recurring obligation to retype 56 rows × 4 numbers across two languages — rates are now written in exactly one file in one language and generated into a checked-in artifact. pnpm gen:pricing after editing pricing.py. No Python file is modified by this PR; if a rate is wrong there, that is a separate backend change.

What else changed

One HARNESSES table replaces four parallel sources of harness knowledge (id, display order, label, logo path, series colour). Row order is display and palette order, so there is no slot map to hand-align. delegate-sdk promoted to a known harness.
One chart implementation. daily-chart.tsx and turn-budget-chart.tsx were ~117 lines each and identical apart from a dataKey, a connectNulls value and one tooltip sentence. Now one RunPointLineChart and two ~30-line wrappers, rendering one <Line> per harness over a shared numeric time axis.
/ defaults to all harnesses. An absent ?h= no longer silently means claude-code. A ?h= scope narrows charts and rails together, so the view is internally consistent and shareable. /trends, /watchlist, /path-to-ga keep exactly-one-harness semantics with no edits.
Two bugs: tag chips dropped an explicit ?h=claude-code (a scope reset under the new semantics); HarnessBadge printed the raw id that callers then printed again ("delegate-sdk delegate-sdk").

Colour choices

Each harness wears its vendor's brand colour, chosen by validator sweep on the strict all-pairs gate rather than by eye — which changed two of the four. UiPath orange is unusable next to Anthropic coral (CVD ΔE 5.1, normal-vision ΔE 9.5 — hard to tell apart even with full colour vision), so Delegate takes violet. OpenAI black fails two proxy gates (lightness band, chroma floor) and is shipped deliberately: every check that measures distinguishability directly passes, black separates on the one channel colour-vision deficiency does not touch, and there is no black element anywhere in the plot area. The rationale is recorded above the table so the next person to run the validator does not "fix" a decision.

Notable bugs found during review

Two silent-misattribution bugs in the charts, neither anticipated by the plan:

shared={false} is silently ignored by LineChart in recharts 2.15.4. getTooltipEventType() computes 'item', LineChart only accepts 'axis', so it falls back — the tooltip stayed nearest-x across every harness.
With the default allowDuplicatedCategory, recharts builds its categorical domain from the concatenation of all series, then each <Line> indexes its own points with that combined index. Hovering Claude Code's Jul 5 dot rendered a second, bogus active dot on Codex's Jul 10 — a date Claude Code has no run for.

Both fixed by allowDuplicatedCategory={false}; four new tests hover every dot and fail without it. Nothing in the suite mounted a tooltip before, which is how this survived a spec review.

Also fixed: URLSearchParams.size is Safari 17+, so on iOS 16 clicking a harness chip navigated to a bare /, discarding window/tag/q; the pricing drift guard compared two products of the same regex and so was blind to a row that regex cannot match; and describe() was decoupled from dataKey so a wrapper can no longer narrate one metric while charting another.

Verification

434 tests pass (baseline was 341 passed / 3 failed — those 3 were the pricing drift the first commit fixes)
pnpm tsc --noEmit clean · pnpm next build clean · pnpm gen:pricing idempotent
Bundles held at baseline: /runs/[id] 163 kB, /trends 120 kB First Load JS. LegendStrip/HarnessSwatch live in a recharts-free module on purpose — importing them from the chart module measured +105 kB on both those chart-less pages. Two new guards enforce that and the lib/harness.ts zero-import leaf rule, both previously only comments.
Rendered and inspected at EVALBOARD_EDITION=internal pnpm dev:local: All chip, brand-coloured lines and chip swatches, correct tooltip attribution, scope surviving a tag click, single labels in the run table.

⚠️ CI does not exercise any of this

Nothing in .github/workflows/ runs the evalboard/ suite — grep -rn evalboard .github/workflows/ yields one hit, a comment saying evalboard/pnpm-lock.yaml is out of scope. So the 434 tests and the pricing drift guard are green-by-absence on this PR, and the drift class this PR removes stays undetected going forward: a rate edited in pricing.py without pnpm gen:pricing will pass CI. That is precisely how the 3× Opus rates accumulated.

Deliberately not fixed here — adding a required check to pr-checks.yml is a repo-wide CI policy change, outside a change scoped to evalboard/. The concrete job to add is written up as the top item in .claude/harness-candidates.md, alongside three other deferred guards.

Not in scope

Non-Claude rates (gpt-5.x, gemini-3.x, OpenRouter, Bedrock open-weight) are generated faithfully but were never verified against their vendors — generation makes the mirror faithful, not the source right. Bedrock -v1 suffixes still resolve to no price on both sides. Introductory pricing (Sonnet 5 is $2/$10 through 2026-08-31 against a $3/$15 list) would need time-dependent rates in the Python schema.

🤖 Generated with Claude Code

… SSOT lib/pricing.ts declared itself a hand-copy of src/coder_eval/pricing.py and had drifted three ways: 7 shared models carried WRONG rates (Opus 4.6/4.7/4.8 read Opus 4.1's legacy $15/$75, overstating cost 3x on every rendered estimate; Haiku 4.5 read Haiku 3.5's $0.80/$4), 17 priced models were missing, and 2 entries were dead. The DELIBERATELY_UNMIRRORED allowlist meant to hold the line had failed twice over — 17 models drifted past it, and 2 of its own 8 entries were models real runs use, so the board rendered "—" for their cost. Replace the hand-copy with a checked-in artifact generated from the Python table, so rates are written in exactly one file in one language and the check becomes "is the artifact current?" rather than 56 rows x 4 numbers retyped across two languages forever. Rates are unchanged on the Python side. - scripts/gen-pricing.mjs emits lib/pricing.generated.ts (56 models, pricing.py order preserved). It is also the single home of the Python-table parser: the parity test imports it rather than re-declaring the regex. - Because that parser runs on both sides of the drift test, a row ROW_RE cannot match would be missing from the artifact AND from the expectation — passing deep-equal while silently unpricing the model. readTable() cross-checks matched rows against the count of ModelPricing( constructions and refuses to emit on mismatch, closing that hole rather than only documenting it. - The main-vs-import guard uses realpathSync, not a lexical compare: Node realpaths the main module, so any symlink in the invocation path made `pnpm gen:pricing` a silent no-op that wrote nothing and exited 0. - Parser uses a null-prototype accumulator (a __proto__ key would otherwise set the prototype and drop the row) and rejects non-finite rates (a malformed one would emit `inputPerMTok: NaN`, typecheck, and render "$NaN"). - Pricing moves to lib/pricing-types.ts so the generated file and pricing.ts share it without a cycle; pricing.ts re-exports it, so no importer changes. Everything below PRICING is byte-identical — resolvePricing, normalizeModel, tokenBucketUsd and messageCostUsd are untouched. Rates re-verified against Anthropic's published per-MTok card: every Claude row in pricing.py matches, and each cache rate is exactly the documented multiplier (write 1.25x input, read 0.1x). Non-Claude rows are generated faithfully but remain unverified against their vendors — generation makes the mirror faithful, not the source right. One line of lib/__tests__/pricing.test.ts had to change (75 -> 25): it asserted claude-opus-4-8 output at $75, encoding the very bug this fixes. The other 21 assertions in that file pass untouched. Suite: 341 passed / 3 failed -> 346 passed / 0 failed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…d colour Harness knowledge lived in two parallel lists in two files: display order in lib/harness.ts and labels + logo paths in harness-badge.tsx. Adding a series colour plus a 4th harness would have made that four lists whose indices had to be hand-aligned — an implicit slot map. Collapse them into one ordered table in the leaf module and derive everything from it, so row order IS display order and colour order, with no alignment to maintain. Adds delegate-sdk as a known harness (it is a real production harness: coder_eval_uipath declares type: Literal["delegate-sdk"]). Derived from the table: KnownHarness, KNOWN_HARNESSES, knownIndex, harnessRow, harnessShortLabel (moved from harness-badge.tsx and re-exported there, so all seven existing importers are untouched). New for the multi-series charts: harnessColor (pure, total, prototype-safe by construction — a linear find, not a Record lookup, so harnessColor("toString") returns a real hex), orderHarnesses (the one known-first-then-alphabetical comparator), and groupByHarness. groupByHarness lives here rather than in lib/overview.ts on purpose: overview.ts reaches node:fs and @azure/storage-blob transitively, and a *value* import from a "use client" chart would pull that graph into the browser bundle. It is generic over { harness: string } so this module needs nothing from the data layer, and lib/harness.ts remains a zero-import leaf. parseHarnessScope takes over the `?h=` body and returns null for absent/malformed, which is what "all harnesses" will mean on the home page. parseHarnessParam becomes `parseHarnessScope(raw) ?? DEFAULT_HARNESS` — the charset and length bound now exist once and cannot drift apart. Observably identical for every input, so its six pre-existing tests pass byte-unedited. Each colour is its vendor's brand value, chosen by validator sweep on the strict all-pairs gate rather than by eye — which ruled out UiPath orange (CVD dE 5.1 vs Anthropic coral, indistinguishable even with full colour vision) in favour of violet for Delegate. The comment above the table records the validator command, its result, and why the two #000000 gate failures are waived, so the next person to run it does not "fix" a deliberate decision. One regression caught while reviewing and fixed here: because delegate-sdk is now a KNOWN row with logo: null, and KNOWN_HARNESSES feeds HarnessSelector on the /trends and /watchlist skeletons, the badge's raw-id fallback fired for a known harness and the chip read "delegate-sdkDelegate SDK". HarnessBadge now separates "unknown harness" (raw id, unchanged) from "known but no logo yet" (render nothing, since callers already print the label). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

getOverview pre-filters perRun to one harness, so it can only ever return a single RunPoint[] and both charts render one hardcoded-blue line. Stamping each point with its harness is what lets a later phase draw one coloured line per harness. The underlying fetch is already unfiltered and cached, and the harness filter is a pure in-memory .filter(), so an all-harness view costs zero extra IO. - RunPoint.harness is required, not optional: getOverview always has a value, so optionality would only invite `?? DEFAULT_HARNESS` fallbacks at consumers. tsc --noEmit is the proof every construction site was updated. - OverviewData.harnesses is the server-side answer to "which harnesses am I looking at", computed once from the windowed, filtered points — deliberately NOT from listRecentHarnesses(), which is a discovery list over a different slice and can name harnesses this window has no runs for. An empty window yields [], so the page cannot claim "all 1 harnesses" and the chart cannot draw a phantom series. - listRecentHarnessesInner drops its inline known-first-then-alphabetical sort for orderHarnesses. This removes a real divergence: the inline version sorted newcomers with .sort() (UTF-16 code units) while orderHarnesses uses localeCompare, so ids like Zebra-Agent / apex-agent would have ordered differently in the switcher than in the chart legend. The never-empty [DEFAULT_HARNESS] fallback stays at this call site, where it belongs. The perRun filter is untouched — it is already null-tolerant, so an all-harness view needs nothing more than passing null. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…w charts daily-chart.tsx and turn-budget-chart.tsx were ~117 lines each and identical apart from a dataKey, a connectNulls value and one tooltip sentence — including byte-identical shortLabel/fullLabel. Converting both to multi-series would have meant writing the same conversion, legend, tooltip and height change twice, with a wrong-dataKey copy-paste as the likeliest error. So there is now one RunPointLineChart and two ~30-line wrappers, and the possibility is removed rather than tested for. One <Line> per harness, each carrying its own data over a shared numeric time axis, coloured by harness id rather than series order — so a chart line and a selector chip can never disagree. Adds a per-chart legend (>=2 series only; a single line is already named by the subtitle) and a harness-naming tooltip. h-56 -> h-72 absorbs the legend and unpacks the top-30% crowding that a [0,100] domain gives four harnesses all sitting between 70 and 100%. The domain stays [0,100]: a zoomed [50,100] read better but misrepresents a percentage. Two real attribution bugs, both found by review and both fixed here: - shared={false} is IGNORED by LineChart in recharts 2.15.4 — getTooltipEventType computes 'item', LineChart only accepts 'axis', so it silently falls back. The tooltip stayed nearest-x across every harness. - With the default allowDuplicatedCategory, recharts builds its categorical domain from the CONCATENATION of all series and then each <Line> indexes its OWN points with that combined index. Hovering claude-code's Jul 5 dot rendered two active dots: the right one, plus a bogus highlight on codex's Jul 10 — a date claude-code has no run for. Four harnesses would light four. allowDuplicatedCategory={false} on the XAxis fixes both by switching lookup to findEntryInArray(series, "timestamp", activeLabel). It is load-bearing for correctness, not de-duplication, and the code says so. New tests hover every dot and assert harness + date + rate + exactly one active dot; they fail without the prop. Nothing in the suite mounted a tooltip before, which is how this survived. describe() now receives the plotted VALUE rather than the point, so a wrapper cannot narrate one metric while charting another — that combination previously typechecked and rendered. LegendStrip and HarnessSwatch live in app/_components/legend.tsx, not in chart-common.tsx as first planned: ChipLegend consumes LegendStrip and the tag rail renders on chart-less pages, so importing it from the chart module pulls recharts into /runs/[id] (163 -> 268 kB First Load JS) and /trends (119 -> 225 kB). Same bundle-correctness rule that keeps groupByHarness in the leaf module. ChipLegend keeps its zero-arg API and renders identically (the Tailwind-400 classes became their literal hexes), so its three consumers are untouched, and /path-to-ga needs no edits. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

… All chip An absent `?h=` on `/` now means ALL harnesses — one colour-coded line per harness in both charts — instead of silently meaning claude-code. An explicit `?h=` narrows the charts AND the rails together, so the view is always internally consistent and shareable. The selector gains a leading "All" chip behind a new `allowAll` prop, so /trends, /watchlist and /path-to-ga keep exactly-one-harness semantics with no edits. Includes bug B1, pulled forward from the next phase because THIS change is what makes it a defect: hrefForTag dropped `h` when it equalled the default, which was harmless while absent `?h=` resolved to claude-code, and is a scope reset now that absent means All. Without it, clicking any tag chip from an isolated Claude Code view silently re-widened the charts, the rails and the selector back to All — exactly the "silently resets the user's scope mid-navigation" failure this phase was warned about. The rule is now stated once, for every consumer: an explicit scope is always emitted; an absent `h` means all harnesses. On /trends the URL merely becomes explicit, with identical behaviour. Two further bugs found in review: - URLSearchParams.size is Safari 17+ only. Where it is undefined the ternary took the falsy branch and navigated to a bare pathname, so on iOS 16 clicking a harness chip would discard window, tag, q AND the scope being set. Uses the toString() idiom buildHref and SearchBox already use. jsdom implements .size, so no test could have caught this. - scopeLabel special-cased zero harnesses but not one, rendering "all 1 harnesses". It now names the single harness, which is also more useful since the chart legend hides itself below two series. The connector became "for" rather than "across", which reads correctly for both a set and one member. scopeLabel is computed once and used in BOTH subtitle branches — the tag/q branch previously never named the harness at all — and its count comes from OverviewData.harnesses, derived from the same points the chart groups, so "all N" always equals the number of lines drawn. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

HarnessBadge printed the raw id as text when it had no vendor logo, and every caller renders harnessShortLabel right after it — so the selector read "delegate-sdk delegate-sdk". Fixed by deleting the fallback rather than adding an accessor: the badge now returns null when its row has no logo, so "no logo → no image" lives in one place, and naming the harness is unambiguously the caller's job. No hasHarnessLogo export — "has a logo" is `logo != null` on one table row, read only by the badge. The run-table Harness column, which relied on that text fallback, now pairs the badge with the label. It normalizes first, because RunListingRow.harness is nullable and a legacy unstamped run would otherwise show the Claude Code logo beside a blank label. Unknown harnesses still render their id, so that cell looks the same as before; known ones gain a label. Also drops the logo's alt text. With both callers printing the label beside it, alt made screen readers announce the harness twice ("Codex · OpenAI Codex") — the selector test had a substring-regex workaround for precisely that, now replaced with exact accessible-name matching. The logo is decorative next to a text label; title still carries the vendor detail on hover. B1 shipped in the previous commit, where the all-harnesses default is what made it a live scope reset rather than a URL nicety. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Findings from the final cross-phase review (two of the first three were raised independently by more than one reviewer). The drift guard was structurally blind. pricing-parity.test.ts imported parsePythonTable, so it compared two products of the same regex — a row that regex cannot match (keyword arguments, scientific notation) is missing from the generated artifact AND from the expectation, so deep-equal passes while the board renders "—" for that model forever. The coverage cross-check existed but only inside the generator, i.e. not on the side that CI would run. readTable is now exported and the test uses it. The same test disclaimed copying the rate card and then copied four rates, so a legitimate vendor repricing would break the suite even after a correct regeneration — the two-places problem generation exists to remove. Every expectation is now derived from pricing.py via the parsed table, with an added guard that the float-fidelity case cannot pass vacuously if those rates ever become round numbers. Same for the one hardcoded rate in pricing.test.ts. The legend named harnesses that draw nothing. groupByHarness deliberately keeps null-metric points (grouping must not filter), but turnBudgetRate is legitimately null for a whole harness when none of its runs carry an expected_turns budget — so "Within Expected Turns" could list three harnesses above two lines, and a reader looking for the missing one would read its absence as 0%. The legend is now derived from series with at least one non-null value for the plotted metric. The chart could draw a line the selector could not isolate. The chart's series come from the windowed points; the selector's chips come from a fixed-count scan of recent runs. A harness that ran once three weeks ago appears in a 30d chart but not in that scan, so it had a visible series and no chip. The page now passes the union. Adds the two guards that were previously only comments, both worth a measured ~105 kB of First Load JS and both violated by adding one import: lib/harness.ts has no imports at all, and app/_components/legend.tsx never reaches recharts or the chart module (with the tag rail's import path pinned, since re-pointing it is the actual regression path). Verified both fail when violated. Also pins the contract this change is named for — the subtitle's "all N harnesses" equals the number of lines drawn — by asserting harnessesInPoints and the rendered curve count over the same input, since those are two derivations of what should be one set. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…cing run Four candidates, the first raised independently by two reviewers: CI never runs the evalboard suite, so the pricing drift guard the generated-SSOT design rests on is never executed. Deferred rather than applied because adding a required check to pr-checks.yml is a repo-wide CI-policy change outside a plan scoped to evalboard/ — recorded with the concrete job to add. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

github-actions · 2026-07-30T23:53:56Z

Claude finished @uipreliga's task in 1m 33s —— View job

📋 Review Task List

Read .github/code_review.md for review standards
Read CLAUDE.md for project conventions
Examine the full diff (git diff origin/main...HEAD)
Review key changed files with full context
Analyze for cross-file consistency issues
Check "what's missing" per review guidelines
Perform design-level scrutiny
Post comprehensive review feedback

Starting review now...

uipreliga and others added 8 commits July 30, 2026 14:46

uipreliga requested review from akshaylive, bai-uipath and tmatup as code owners July 30, 2026 23:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(evalboard): multi-harness colour-coded charts + generated cost table - #70

feat(evalboard): multi-harness colour-coded charts + generated cost table#70
uipreliga wants to merge 8 commits into
mainfrom
feat/evalboard-multi-harness-charts

uipreliga commented Jul 30, 2026

Uh oh!

github-actions Bot commented Jul 30, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

uipreliga commented Jul 30, 2026

Goal

The cost bug this fixes

What else changed

Colour choices

Notable bugs found during review

Verification

⚠️ CI does not exercise any of this

Not in scope

Uh oh!

github-actions Bot commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📋 Review Task List

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

github-actions Bot commented Jul 30, 2026 •

edited

Loading