Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 43 additions & 15 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -706,17 +706,17 @@ narrower question: “At published API text-token prices, roughly what
would this observed mix of model work cost when mapped onto the movement in my
quota meter?”

#### Fast mode and service-tier uncertainty
#### Service tiers and pricing uncertainty

Codexometer prices each newly observed response separately. Standard token cost
and the requested-tier premium are maintained independently; quota learning uses
and the signed requested-tier adjustment are maintained independently; quota learning uses
their sum. For example, $1 at standard prices plus a $1 Fast premium over **5
percentage points** of quota movement implies roughly **$40 per 100%**, compared
with **$20 at standard prices**. It never multiplies the observed quota percentage
or applies today's `/fast` setting retroactively.

- `TIER*` means the estimate includes a **requested**, not billing-confirmed,
Fast premium. The current Codex sources expose persisted
tier adjustment (a premium for Fast/Ultrafast or a discount for Flex). The current Codex sources expose persisted
`thread_settings_applied` settings, but neither local token events nor the
shared app-server's response-usage notifications expose the actual billed tier.
Settings are kept per session and captured for each turn; an in-flight turn
Expand All @@ -736,13 +736,17 @@ or applies today's `/fast` setting retroactively.
comparison** for the same sample windows. Narrower layouts omit that comparison.
- A known default setting has no premium. An explicitly unsupported tier/model
combination is unpriced and interrupts clean learning rather than silently
applying an invented multiplier. Standard benchmark rankings are unchanged.
applying an invented multiplier. Benchmark comparisons continue to use Standard rates.

Fast pricing was updated on **30 September 2026**: **GPT-6.1 Sol, GPT-6 (Astra, Sol, Luna) and GPT-5.6
Tier pricing was reviewed on **12 October 2026**: **GPT-6.1 Sol, GPT-6 (Astra, Sol, Luna) and GPT-5.6
(Sol, Terra, Luna)** use **2× applicable standard API prices** for `fast` / the
legacy `priority` alias. The premium is applied after the existing per-response
long-context and cache-read/cache-write calculations; reasoning output is not
counted twice. Other models' Fast prices are not yet maintained. See the official
legacy `priority` alias. Astra and GPT-6.1 Sol **Ultrafast** use **6×** applicable standard API prices.
**Flex** uses **½×** on the maintained GPT-6/GPT-5.6 models, GPT-5.2 and GPT-5.5.
GPT-5.2 fallback pricing and published short-context Fast rates for GPT-5.2,
GPT-5.3 Codex, GPT-5.4/Mini and GPT-5.5 are also maintained; GPT-5.5 Fast is 2.5×.
Other explicit model/tier combinations and unpublished long-context Fast rates
remain unavailable. Adjustments follow per-response long-context and cache
calculations; reasoning output is not counted twice. See the official
[Astra pricing](https://developers.openai.com/api/docs/models/gpt-6-astra),
[Codex speed guidance](https://learn.chatgpt.com/docs/agent-configuration/speed),
and [API Fast-mode guide](https://developers.openai.com/api/docs/guides/fast-mode).
Expand All @@ -758,9 +762,21 @@ new evidence accumulates. Future improvement: consume the actual response
`service_tier` if Codex exposes it with matching per-response token usage, then
prefer that evidence to requested settings.

Codexometer normally prices each newly completed local model call using the
requested model durably recorded in its turn context. Current Codex rollout
files do not persist transient model-reroute events, so a reroute cannot be
Codexometer prefers Codex's durable **`token_usage_record`** entries for each
newly completed response. Exact records are counted independently of cumulative
context counters, with bounded response-ID deduplication; inherited parent
records are excluded from a child's own usage. Startup establishes a historical
baseline without charging past responses, using a bounded 4 MiB tail scan.
An exact record switches that rollout to response accounting: subsequent
`token_count` snapshots remain context/attribution baselines and never create
another billable response. Older rollouts retain the legacy token-count reader. A replacement rollout for
an existing thread establishes a new baseline rather than charging copied
history again; session settings come from its newest observed rollout.

The requested model remains the fallback when actual per-response model evidence
is unavailable; durable usage records do not supply model or billed-tier fields.
Diagnostics explicitly distinguish requested-model attribution from a matching
live reroute. Current Codex rollout files do not persist transient model-reroute events, so a reroute cannot be
reconstructed from that source alone. When Codexometer and the CLI share the
managed daemon described above, Codexometer keeps a live subscription and
matches reroute and token-usage events to rollout calls by thread ID, turn ID,
Expand All @@ -779,7 +795,7 @@ cache-write input, and output are priced separately; requests above the
published 272,000-input-token threshold use the corresponding long-context
rates where OpenAI publishes them. Unknown models or missing price classes fail
closed as `UNPRICED USAGE` rather than being guessed or treated as free.
Codex's context-window bookkeeping updates are recognised only when the window
In the legacy reader, Codex's context-window bookkeeping updates are recognised only when the window
and both complete, explicit zero-token-class breakdowns match its synthetic fill record.
They rebase the local counter without adding response costs, token totals or
model-call graph pulses. Quota learning restarts as `LOCAL ACCOUNTING REBASED`.
Expand All @@ -788,20 +804,32 @@ the current baseline, including inherited child history and historical scans.
Missing input/output usage and inconsistent response/cumulative usage instead
restart as `MISSING RESPONSE USAGE` or `INCONSISTENT RESPONSE USAGE`; no costs
are guessed. Unknown model/tier or unavailable pricing remains `UNPRICED USAGE`.
The terminal Quota view retains a bounded issue panel with the reason, originating
session (including a child agent), model and its attribution basis, requested tier,
effort, response/legacy/benchmark source, response ID and timestamp. The panel
fits available terminal space; the browser Quota view shows the latest observed
issue with wrapping. Diagnostics retain accounting metadata, not prompt/reply
content, and are not persisted. A prior issue cannot be assigned to a later
learning restart. Reserve (`gpt-reserve`), custom/preview and provider-specific
model IDs stay unpriced where a price mapping is unverified; display aliases are
not billing evidence.
Core, Extended, and DigBench trials use ephemeral threads that intentionally do
not appear in normal persisted session telemetry. While a subscription-funded
benchmark suite is active, Quota views replace the numeric API-equivalent
estimate with `SUBSCRIPTION BENCHMARK ACTIVE` instead of presenting partial
accounting. When each trial finishes, its authoritative benchmark usage is
folded into the same process-local accounting used for quota learning; missing
or unpriceable benchmark usage fails closed and restarts the learning anchor.
folded into the same process-local accounting used for quota learning; missing,
inconsistent or unpriceable benchmark usage fails closed with separate reasons
and restarts the learning anchor. Complete raw response usage takes precedence
over cumulative/context snapshots, each response retains its observed/requested
model, and identical notification replays do not duplicate costs.
Benchmarks funded by `CODEXOMETER_BENCHMARK_API_KEY` are excluded because they
do not consume the displayed subscription quota. Benchmark threads remain
hidden from Sessions; hiding presentation does not exclude their aggregate
subscription impact from Quota views.
The embedded rates come from the
[official OpenAI API pricing page](https://developers.openai.com/api/docs/pricing)
and were last updated on **2026-09-30**.
and were last reviewed on **2026-10-12**.
Every Quota presentation repeats that retrieval date and a terminal hyperlink
to the source in its footer when the terminal is wide enough, matching the
Benchmark view and making stale compiled pricing conspicuous wherever a priced
Expand Down
92 changes: 67 additions & 25 deletions internal/codex/benchmark.go
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ const (
// the published rates compiled into this release. Keep both in sync whenever
// standardAPIPrices changes.
StandardAPIPricingSourceURL = "https://developers.openai.com/api/docs/pricing"
StandardAPIPricingRetrievedOn = "2026-09-30"
StandardAPIPricingRetrievedOn = "2026-10-12"
)

// BenchmarkUsage is the app-server token breakdown for one isolated turn.
Expand Down Expand Up @@ -111,9 +111,14 @@ func (c Client) BenchmarkBillingSource() BenchmarkBillingSource {

// BenchmarkResponseUsage is the exact upstream usage for one Responses API
// completion when the current Codex app-server exposes experimental raw events.
// Missing or conflicting usage retains response identity for diagnostics;
// UsageMissing distinguishes an omitted payload from valid all-zero usage.
type BenchmarkResponseUsage struct {
ResponseID string
Usage BenchmarkUsage
ResponseID string
Model string
Usage BenchmarkUsage
UsageMissing bool
UsageIssue string
}

type BenchmarkInteractionKind string
Expand Down Expand Up @@ -930,7 +935,7 @@ func (s *appServerSession) runBenchmark(
Usage *BenchmarkUsage `json:"usage"`
}
if json.Unmarshal(params, &event) == nil && event.ThreadID == startedThread.Thread.ID && event.TurnID == startedTurn.Turn.ID {
telemetry.recordRawResponse(event.ResponseID, event.Usage)
telemetry.recordRawResponseForModel(event.ResponseID, event.Usage, result.ActualModel)
}
case "model/rerouted":
var event struct {
Expand Down Expand Up @@ -1163,14 +1168,20 @@ func (s *appServerSession) interruptBenchmarkTurn(
}

func applyBenchmarkMeasurements(result *BenchmarkResult, telemetry *benchmarkTelemetry) {
result.CostUSD, result.CostKnown, result.CostIssue = 0, false, ""
result.UsageKnown, result.UsageIssue = false, ""
telemetry.apply(result)
if result.ToolUsed {
result.CostIssue = "tool use prohibited: " + result.ToolType
} else if result.UsageKnown {
if result.UsageSource == BenchmarkUsageRawResponses {
result.CostKnown = true
for _, response := range result.ResponseUsage {
cost, known, issue := estimateSingleResponseAPICostWithIssue(result.ActualModel, response.Usage)
model := response.Model
if model == "" {
model = result.ActualModel
}
cost, known, issue := estimateSingleResponseAPICostWithIssue(model, response.Usage)
if !known {
result.CostUSD, result.CostKnown, result.CostIssue = 0, false, issue
break
Expand All @@ -1191,54 +1202,79 @@ type benchmarkTelemetry struct {
lastCumulative BenchmarkUsage
hasLastCumulative bool
rawResponses []BenchmarkResponseUsage
rawIDs map[string]struct{}
rawIDs map[string]int
rawObserved bool
rawMissingUsage bool
issue string
cumulativeIssue string
}

func newBenchmarkTelemetry() *benchmarkTelemetry {
return &benchmarkTelemetry{rawIDs: make(map[string]struct{})}
return &benchmarkTelemetry{rawIDs: make(map[string]int)}
}

func (t *benchmarkTelemetry) recordCumulative(usage BenchmarkUsage) {
t.cumulativeObserved = true
if issue := validateBenchmarkUsage(usage); issue != "" {
t.setIssue("invalid cumulative usage: " + issue)
t.cumulativeIssue = "invalid cumulative usage: " + issue
}
if t.hasLastCumulative && !benchmarkUsageAtLeast(usage, t.lastCumulative) {
t.setIssue("cumulative usage regressed")
t.cumulativeIssue = "cumulative usage regressed"
}
t.cumulative = usage
t.lastCumulative = usage
t.hasLastCumulative = true
}

func (t *benchmarkTelemetry) recordRawResponse(responseID string, usage *BenchmarkUsage) {
t.recordRawResponseForModel(responseID, usage, "")
}

func (t *benchmarkTelemetry) recordRawResponseForModel(id string, usage *BenchmarkUsage, model string) {
t.rawObserved = true
if strings.TrimSpace(responseID) == "" {
t.setIssue("raw response usage omitted response id")
return
response := BenchmarkResponseUsage{ResponseID: id, Model: model, UsageMissing: usage == nil}
if usage == nil {
response.UsageIssue = "raw response omitted usage"
} else {
response.Usage = *usage
if issue := validateBenchmarkUsage(*usage); issue != "" {
response.UsageIssue = "invalid raw response usage: " + issue
}
}
if _, duplicate := t.rawIDs[responseID]; duplicate {
t.setIssue("duplicate raw response usage: " + responseID)
if strings.TrimSpace(id) == "" {
response.UsageIssue = "raw response usage omitted response id"
t.setIssue(response.UsageIssue)
t.rawResponses = append(t.rawResponses, response)
return
}
t.rawIDs[responseID] = struct{}{}
if usage == nil {
t.rawMissingUsage = true
if index, duplicate := t.rawIDs[id]; duplicate {
previous := &t.rawResponses[index]
if previous.Model == model && previous.UsageMissing == response.UsageMissing && previous.Usage == response.Usage {
return
}
previous.UsageIssue = "conflicting duplicate raw response usage: " + id
if previous.Model != model {
previous.UsageIssue = fmt.Sprintf("conflicting duplicate raw response model: %q versus %q", previous.Model, model)
}
t.setIssue(previous.UsageIssue)
return
}
if issue := validateBenchmarkUsage(*usage); issue != "" {
t.setIssue("invalid raw response usage: " + issue)
t.rawIDs[id] = len(t.rawResponses)
t.rawResponses = append(t.rawResponses, response)
if response.UsageMissing {
t.rawMissingUsage = true
} else if response.UsageIssue != "" {
t.setIssue(response.UsageIssue)
}
t.rawResponses = append(t.rawResponses, BenchmarkResponseUsage{ResponseID: responseID, Usage: *usage})
}

func (t *benchmarkTelemetry) apply(result *BenchmarkResult) {
result.UsageObserved = t.cumulativeObserved || t.rawObserved
result.UsageSource = BenchmarkUsageUnavailable
result.ResponseUsage = append([]BenchmarkResponseUsage(nil), t.rawResponses...)
if t.rawObserved {
result.UsageSource = BenchmarkUsageRawResponses
}
if t.issue != "" {
result.UsageIssue = t.issue
return
Expand All @@ -1257,13 +1293,18 @@ func (t *benchmarkTelemetry) apply(result *BenchmarkResult) {
result.UsageIssue = "invalid raw response total: " + issue
return
}
if t.cumulativeObserved && rawTotal != t.cumulative {
result.UsageIssue = "raw and cumulative usage disagree"
return
}

result.Usage, result.UsageKnown, result.UsageSource = rawTotal, true, BenchmarkUsageRawResponses
return
}
if t.rawObserved && t.rawMissingUsage {
result.UsageIssue = "raw response omitted usage"
return
}
if t.cumulativeIssue != "" {
result.UsageIssue = t.cumulativeIssue
return
}
if t.cumulativeObserved {
result.Usage, result.UsageKnown, result.UsageSource = t.cumulative, true, BenchmarkUsageCumulative
return
Expand Down Expand Up @@ -1609,6 +1650,7 @@ type apiPrice struct {
}

var standardAPIPrices = map[string]apiPrice{
"gpt-5.2": {input: 1.75, cached: 0.175, output: 14.00},
"gpt-6.1-sol": {input: 2.00, cached: 0.10, cacheWrite: 2.50, cacheWriteKnown: true, output: 10.00, longInput: 4.00, longCached: 0.20, longCacheWrite: 5.00, longOutput: 15.00, longKnown: true},
"gpt-6-sol": {input: 2.00, cached: 0.20, cacheWrite: 2.50, cacheWriteKnown: true, output: 10.00, longInput: 4.00, longCached: 0.40, longCacheWrite: 5.00, longOutput: 15.00, longKnown: true},
"gpt-6-luna": {input: 0.10, cached: 0.01, cacheWrite: 0.125, cacheWriteKnown: true, output: 0.50, longInput: 0.20, longCached: 0.02, longCacheWrite: 0.25, longOutput: 0.75, longKnown: true},
Expand Down Expand Up @@ -1640,7 +1682,7 @@ func estimateSingleResponseAPICostWithIssue(model string, usage BenchmarkUsage)
}
price, ok := priceForModel(model)
if !ok {
return 0, false, "no published price for model " + strings.TrimSpace(model)
return 0, false, "no maintained standard API price for model " + strings.TrimSpace(model)
}
if usage.CacheWriteInputTokens > 0 && !price.cacheWriteKnown {
return 0, false, "no published cache-write price for model " + strings.TrimSpace(model)
Expand Down
Loading
Loading