feat(litellm): actual per-call cost + cache accounting for the open-weight backend

.env.example

@@ -58,6 +58,17 @@ LOG_TO_FILE=false  # Set to true to enable file logging
 # GEMINI_API_KEY="AIza..."
 # ANTIGRAVITY_MODEL="gemini-3.1-pro-preview"
 
+# LiteLLM (Anthropic-compatible) open-weight backend (API_BACKEND=litellm). Point
+# the agent at a self-hosted LiteLLM proxy (see litellm/start-litellm.sh, which
+# prints these). LITELLM_COST_LOG must be the SAME path the proxy writes its
+# per-call JSONL to — that is how the run joins ACTUAL OpenRouter cost + cache back
+# onto each turn; if it is unset (or points elsewhere) the run silently falls back
+# to static rate-card pricing with 0 cache reads.
+# LITELLM_BASE_URL="http://localhost:4000"
+# LITELLM_AUTH_TOKEN="sk-..."
+# LITELLM_MODEL="zai.glm-5"
+# LITELLM_COST_LOG="./tmp/litellm-costs.jsonl"
+
 # UiPath CLI plugin-discovery pin. When unset, the sandbox auto-derives
 # the canonical `node_modules/@uipath` from the resolved `uip` binary at setup
 # time. Operators on dedicated eval hosts can pin explicitly to override.

CLAUDE.md

@@ -25,6 +25,7 @@ coder_eval/
 ├── logging_config.py              # Structured logging setup
 ├── path_utils.py                  # Run ID generation, path utilities
 ├── pricing.py                     # Model pricing / cost calculation (ModelPricing, calculate_cost, register_pricing)
+├── litellm_cost.py                # Join proxy-captured ACTUAL per-call cost/cache onto turns (LiteLLM backend; apply_actual_cost)
 ├── utils.py                       # Version info helpers
 │
 ├── agents/
@@ -41,7 +42,7 @@ coder_eval/
 │   ├── routing.py                 # ApiRoute (DirectRoute/BedrockRoute)
 │   ├── sandbox.py                 # SandboxConfig, ResourceLimits
 │   ├── tasks.py                   # TaskDefinition, AgentConfig, Dataset (dataset fan-out + sample)
-│   ├── telemetry.py               # CommandTelemetry, CommandStatistics, TokenUsage, ReconciliationMessage, TranscriptMessage
+│   ├── telemetry.py               # CommandTelemetry, CommandStatistics, TokenUsage, ProviderCallCost, ReconciliationMessage, TranscriptMessage
 │   └── templates.py               # RepoSource, TemplateDirSource, StarterFilesSource
 │
 ├── criteria/                      # Criterion checker plugins (one file per type)
@@ -138,7 +139,7 @@ action.yml                         # Published composite GitHub Action (coder-ev
 - **Dataset fan-out**: `TaskDefinition.dataset` (inline rows or JSONL path) expands a single task into N row-tasks with `${row.<field>}` substitution in `initial_prompt` and `success_criteria` string fields. Expansion runs in `task_loader.expand_dataset` **before** variant resolution, so variants cannot override the dataset. Row sampling: CLI `--sample N` (fixed-seed uniform-random N over the whole dataset) overrides `--sample-per-stratum N` / `dataset.sample_per_stratum` (stratified random N-per-stratum, keyed on `stratify_field`, default `expected_skill` — for classification suites like activation). Stratified sampling (whether the N-per-stratum count comes from the **CLI** `--sample-per-stratum` flag or **YAML** `dataset.sample_per_stratum`) is **nondeterministic** by default — it re-draws each run (so the nightly activation suite broadens coverage over time). Set `dataset.sample_seed` to pin a reproducible sample; an explicit seed always wins. (Only `--sample N` uses a fixed seed, since a smoke test wants the same N rows each run.)
 - **Per-criterion aggregation**: Each `BaseCriterion` subclass exposes `aggregate(criterion, per_row_results) -> CriterionAggregate | None`. Default emits `count / mean / median / std / min / max` so every criterion is suite-thresholdable for free. Classification-style criteria return `ClassificationCriterionResult` (subclass of `CriterionResult`) and layer accuracy / P/R/F1 / confusion via the shared `overlay_classification_metrics` utility. `BaseSuccessCriterion.suite_thresholds` gates the suite on those metrics; CLI exits non-zero on any gate failure.
 - **Sub-agent token accounting**: There is NO separate per-sub-agent field. Every sub-agent generation is captured as a `parent_tool_use_id`-tagged `AssistantMessage` in the turn transcript, so per-sub-agent usage is derived by grouping those messages on that id (the evalboard's `aggregateSubAgentUsage` does exactly this). Claude bubbles its sub-agent's intermediate generations into the parent stream natively, and the **terminal** generation (delivered as the Agent tool result, never streamed) is synthesized into one via `_synthesize_subagent_terminal_message` from `tool_use_result.usage`. Codex reconstructs all child generations from the child rollout (`_recover_subagent_tool_calls`). The turn total already includes sub-agent cost — Claude via the SDK's cumulative `model_usage`; Codex via `_fold_subagent_tokens`, which folds the child messages (their real per-generation tokens) into the parent total. `CommandTelemetry.result_summary` is stored **untruncated** (no 200-char cap) so sub-agent returns are preserved whole. Set `CODER_EVAL_RAW_SDK_LOG=1` to dump every raw SDK event to the task log for inspection.
-- **Reconciliation message (stream self-reconciles to the turn total)**: The per-message stream consistently under-reports the authoritative turn total — a fixed prompt slice (~512 input tokens on Claude) is billed on no SDK-emitted message, and sub-agent input/cache only partially bubbles up. So `EventCollector.build_turn_record` appends one synthetic `ReconciliationMessage` (`role="reconciliation"`, in the `TranscriptMessage` union) per turn, carrying the per-bucket residual = `token_usage` − Σ(assistant message buckets). The invariant: **summing the four token buckets across `TurnRecord.messages` (assistant + reconciliation) equals `token_usage` exactly**, for both Claude and Codex (Codex's stream is already complete after `_recover_subagent_tool_calls`, so its residual is usually 0 and no entry is emitted). This is what lets the evalboard SUM the message stream as the source of truth instead of reading a separate aggregate ("agent tokens"): `selectTokenTotals` returns the stream sum whenever a reconciliation entry is present, and the timeline renders it as its own row. It is agent-agnostic (booked at the single `EventCollector` seam), carries no cost (cost stays on `token_usage`), and is excluded from generation/turn counts and the cost simulator. The Python `token_usage`/`total_token_usage` aggregate is unchanged and still authoritative for budget/judges/reports.
+- **Reconciliation message (stream self-reconciles to the turn total)**: The per-message stream consistently under-reports the authoritative turn total — a fixed prompt slice (~512 input tokens on Claude) is billed on no SDK-emitted message, and sub-agent input/cache only partially bubbles up. So `EventCollector.build_turn_record` appends one synthetic `ReconciliationMessage` (`role="reconciliation"`, in the `TranscriptMessage` union) per turn, carrying the per-bucket residual = `token_usage` − Σ(assistant message buckets). The invariant: **summing the four token buckets across `TurnRecord.messages` (assistant + reconciliation) equals `token_usage` exactly**, for both Claude and Codex (Codex's stream is already complete after `_recover_subagent_tool_calls`, so its residual is usually 0 and no entry is emitted). This is what lets the evalboard SUM the message stream as the source of truth instead of reading a separate aggregate ("agent tokens"): `selectTokenTotals` returns the stream sum whenever a reconciliation entry is present, and the timeline renders it as its own row. It is agent-agnostic (booked at the single `EventCollector` seam), carries no cost (cost stays on `token_usage`), and is excluded from generation/turn counts and the cost simulator. The LiteLLM open-weight actual-cost join (`litellm_cost.apply_actual_cost`) deliberately writes cost at the TURN level only (`token_usage.total_cost_usd` = the real OpenRouter bill) plus the per-call `TurnRecord.provider_call_costs` audit record; it does NOT touch the message token buckets, so `EventCollector` stays the single writer and this invariant holds on every backend. The Python `token_usage`/`total_token_usage` aggregate is unchanged and still authoritative for budget/judges/reports.
 - **sandbox isolation**: Tasks that don't need MCP servers should set `setting_sources: []` in their `agent:` block to isolate the sandbox from the host project's CLAUDE.md and settings. Without this, the host project's CLAUDE.md (often 20 KB+) is injected into every API call, inflating cache-creation tokens and cost significantly.
 - **Run-time caps (non-criterion enforcement)**: `TaskDefinition.run_limits` (`RunLimits` model) is the single namespace for all run-time caps — `max_turns` / `task_timeout` / `turn_timeout` (structural) and `max_input_tokens` / `max_output_tokens` / `max_total_tokens` / `max_usd` (cumulative budget). Token/USD breaches abort with `FinalStatus.TOKEN_BUDGET_EXCEEDED` or `COST_BUDGET_EXCEEDED` (both `category == "failed"`). Structural caps are set from the CLI via `-D run_limits.max_turns=…` / `-D run_limits.task_timeout=…` / `-D run_limits.turn_timeout=…` (field-merged into `run_limits`); budget caps via `-D run_limits.max_usd=…` etc. or YAML. Layered config uses field-merge — a variant block overrides individual keys without replacing the task's block.
 - **Early stop on criterion (opt-in)**: `run_limits.stop_early` (default off) ends a single-shot Claude run early once the run's **armed** criteria are decided, so a raised `max_turns` isn't wasted on the smoke flavor. A criterion is armed by `stop_when: pass|fail|decided|auto`; only criteria that can decide from a partial trajectory may arm (non-empty `live_stop_polarities` ClassVar + `live_verdict` override — currently `skill_triggered`, `command_executed`; CE025 keeps the two consistent). `decided` arms **both** polarities; `auto` arms whichever polarities **this instance** can decide — the value for dataset-fanned criteria whose positive/distractor role flips per row. Stop rule: the pass-stop fires when every **pass-armed** criterion live-passes (fail-armed distractors are not required to pass; zero pass-armed ⇒ never pass-stops); the fail-stop fires on the first fail-armed live-fail but is **deferred while any pass-armed criterion is undecided** — a distractor misfire must not truncate a positive row's recall signal, so the latched misfire fires once the positives resolve (or the run continues to the cap). A fail-stop is therefore verdict-preserving; a pass-stop can miss a *later* distractor misfire, so authoritative P/R/F1 comes from a `stop_early: false` run. Driven by `orchestration/early_stop.py::EarlyStopWatcher` through the Claude agent's cooperative `should_stop` seam (tool-call granularity, no SIGKILL); live verdicts only *trigger* the stop — the standard `check_all_async` on the frozen trajectory is authoritative. An early-stopped run gates on the **armed subset** (`EvaluationResult.armed_criteria_passed`); a completed run gates on the full set. Every unsupported use is a hard error at resolution (plan *and* run), and a runtime verdict bug **fails open** to a full run. Surfaces: `EarlyStopInfo`, report notes/badges, `stopped_early` run.json rows, `EarlyStopped`/`EarlyStopReason` telemetry dims. Worked rationale: docs/TASK_DEFINITION_GUIDE.md § `stop_early`. Defaults off ⇒ behavior byte-for-behavior unchanged.

docs/REPORT_SCHEMA.md

@@ -168,15 +168,22 @@ fields so subclass keys round-trip.
 `iteration`, `user_input`, `agent_output`, `commands` (`list[CommandTelemetry]`),
 `timestamp`, `duration_seconds`, `token_usage`, `model_used`, `assistant_turn_count`,
 `messages` (`list[TranscriptMessage]`, discriminated on `role`:
-`user`/`assistant`/`reconciliation`), `num_turns`, `max_turns_exhausted`,
+`user`/`assistant`/`reconciliation`), `provider_call_costs`
+(`list[ProviderCallCost]` — one row per real upstream call with its ACTUAL cost +
+cache buckets + routed `provider`, captured proxy-side on the LiteLLM open-weight
+backend and rendered by the evalboard as a per-call table; empty on every other
+backend), `num_turns`, `max_turns_exhausted`,
 `result_summary` (`{is_error, subtype, stop_reason, result}`), `crashed`,
 `crash_reason`.
 
 > **Token invariant.** Summing the four token buckets across `messages`
 > (assistant + the synthetic `reconciliation` entry) equals `token_usage` exactly.
 > The `reconciliation` message carries the residual the per-message stream
-> under-reports; it has no cost and is excluded from turn/generation counts. See the
-> [Claude Code guide](agents/CLAUDE_CODE.md#telemetry).
+> under-reports; it has no cost and is excluded from turn/generation counts. The
+> LiteLLM actual-cost join writes cost at the TURN level only (`token_usage.total_cost_usd`
+> = the real bill) plus the per-call `provider_call_costs` audit record — it does
+> NOT touch the message token buckets, so this invariant holds on every backend.
+> See the [Claude Code guide](agents/CLAUDE_CODE.md#telemetry).
 
 ### EarlyStopInfo
 

evalboard/app/runs/[id]/[...task]/_sections.tsx

@@ -9,6 +9,7 @@ import type {
     MessageEvent,
     MessageToolUse,
     SubAgentTotals,
+    TaskDetail,
     TokenTotals,
     ToolCall,
 } from "@/lib/runs";
@@ -533,6 +534,126 @@ export function MessageTimelineSection({
     );
 }
 
+// Short-form a call id for the table (full ids are long provider hashes). Keeps
+// the head/tail so a row is still identifiable at a glance. "—" when absent.
+function shortCallId(id: string | null): string {
+    if (!id) return "—";
+    return id.length > 14 ? `${id.slice(0, 6)}…${id.slice(-4)}` : id;
+}
+
+function fmtCallTok(n: number | null): string {
+    return n != null ? fmtCompact(n) : "—";
+}
+
+// Per-call actual cost + cache table, sourced from each turn's
+// provider_call_costs audit rows (LiteLLM/open-weight backend). One sub-table
+// per turn with calls, plus a per-turn total. Renders nothing on Claude/Bedrock
+// runs (empty providerCalls).
+export function ProviderCallTableSection({
+    providerCalls,
+}: {
+    providerCalls: TaskDetail["providerCalls"];
+}) {
+    if (providerCalls.length === 0) return null;
+    const totalCalls = providerCalls.reduce((s, t) => s + t.calls.length, 0);
+    return (
+        <section className="space-y-2">
+            <h2 className="text-sm font-semibold text-gray-900">
+                Provider calls ({totalCalls})
+            </h2>
+            <p className="text-[10px] text-gray-500">
+                Actual per-call cost + cache captured from the provider (LiteLLM
+                backend). The turn total is the real bill.
+            </p>
+            {providerCalls.map(({ iteration, calls }) => {
+                const turnTotal = calls.reduce(
+                    (s, c) => s + (c.costUsd ?? 0),
+                    0,
+                );
+                return (
+                    <div key={iteration} className="space-y-1">
+                        <div className="text-xs text-gray-500">
+                            Turn {iteration + 1}
+                        </div>
+                        <TableScroll>
+                            <table className="w-full text-sm">
+                                <thead>
+                                    <tr className="bg-gray-50 border-b border-gray-200 text-left text-gray-600">
+                                        <th className="py-2 px-3 font-medium">
+                                            Call
+                                        </th>
+                                        <th className="py-2 px-3 font-medium">
+                                            Provider
+                                        </th>
+                                        <th className="py-2 px-3 font-medium text-right">
+                                            Input
+                                        </th>
+                                        <th className="py-2 px-3 font-medium text-right">
+                                            Cache read
+                                        </th>
+                                        <th className="py-2 px-3 font-medium text-right">
+                                            Cache write
+                                        </th>
+                                        <th className="py-2 px-3 font-medium text-right">
+                                            Output
+                                        </th>
+                                        <th className="py-2 px-3 font-medium text-right">
+                                            Cost ($)
+                                        </th>
+                                    </tr>
+                                </thead>
+                                <tbody>
+                                    {calls.map((c, i) => (
+                                        <tr
+                                            key={c.callId ?? i}
+                                            className="border-b border-gray-100 last:border-b-0"
+                                        >
+                                            <td className="py-2 px-3 font-mono text-xs text-gray-900">
+                                                {shortCallId(c.callId)}
+                                            </td>
+                                            <td className="py-2 px-3 text-xs text-gray-700">
+                                                {c.provider ?? "—"}
+                                            </td>
+                                            <td className="py-2 px-3 text-right tabular-nums text-gray-700">
+                                                {fmtCallTok(c.inputTokens)}
+                                            </td>
+                                            <td className="py-2 px-3 text-right tabular-nums text-amber-700">
+                                                {fmtCallTok(c.cacheReadTokens)}
+                                            </td>
+                                            <td className="py-2 px-3 text-right tabular-nums text-amber-700">
+                                                {fmtCallTok(c.cacheWriteTokens)}
+                                            </td>
+                                            <td className="py-2 px-3 text-right tabular-nums text-gray-700">
+                                                {fmtCallTok(c.outputTokens)}
+                                            </td>
+                                            <td className="py-2 px-3 text-right tabular-nums text-gray-900">
+                                                {c.costUsd != null
+                                                    ? fmtUsd(c.costUsd)
+                                                    : "—"}
+                                            </td>
+                                        </tr>
+                                    ))}
+                                    <tr className="border-t border-gray-200 bg-gray-50 font-medium">
+                                        <td
+                                            className="py-2 px-3 text-xs text-gray-600"
+                                            colSpan={6}
+                                        >
+                                            Turn total
+                                        </td>
+                                        <td className="py-2 px-3 text-right tabular-nums text-gray-900">
+                                            {fmtUsd(turnTotal)}
+                                        </td>
+                                    </tr>
+                                </tbody>
+                            </table>
+                        </TableScroll>
+                    </div>
+                );
+            })}
+        </section>
+    );
+}
+
 // Owns the cost-simulator lever state and renders BOTH the message timeline and
 // the simulator beneath it. Lifting the levers here lets the timeline show each
 // message's projected token Δ inline (the "message view" of the diff) driven by