Skip to content

Price exact Codex responses and explain quota accounting failures - #84

Merged
merefield merged 2 commits into
mainfrom
fix/exact-usage-pricing-diagnostics
Oct 12, 2026
Merged

merefield merged 2 commits into
mainfrom
fix/exact-usage-pricing-diagnostics

Conversation

@merefield

@merefield merefield commented Oct 12, 2026 •

Copy link
Copy Markdown
Owner

Codexometer can report UNPRICED USAGE while the selected session still shows GPT-6 Astra at xhigh. Reasoning effort is not a price-lookup key: ordinary Astra Standard/Fast responses, cache writes and long contexts already price successfully. The restart is account-wide and can originate from another local session, a child agent, an unsupported requested tier, or a subscription benchmark. This change uses Codex's exact persisted response records, fills verified pricing gaps, and reports the originating failure rather than leaving the tester with a generic warning.

Resulting behaviour

  • Prefer durable token_usage_record entries for live response tokens and costs. Once a rollout uses exact records, its token_count snapshots only provide context/attribution checkpoints. Synthetic context fills, resets and differing context totals cannot manufacture another response or block an otherwise valid exact response. Older logs retain the existing legacy reader and its synthetic-fill/missing-total safeguards when the startup scan covers the whole history. If a bounded scan cannot establish the mode of a large rollout, context snapshots fail closed with a source-discovery diagnostic until an owned exact response arrives; changing checkpoints continue to invalidate learning, while identical checkpoints do not duplicate the warning.
  • Deduplicate recent response IDs, report conflicting replays, and exclude inherited parent history from child accounting. Establish a bounded historical startup baseline without charging old responses; keep concurrent appends and incomplete JSONL records outside that baseline. A replacement rollout for an already observed thread establishes a new baseline and restarts learning rather than billing copied history again. Existing delayed daemon reroute matching remains available.
  • Maintain published Astra/GPT-6.1 Sol Ultrafast at 6× and supported Flex at ½× Standard rates, including cache-read/write and long-context classes. Signed Flex discounts no longer look like accounting regressions. Add GPT-5.2 fallback/snapshot pricing and published short-context Fast prices for GPT-5.2, GPT-5.3 Codex, GPT-5.4/Mini and GPT-5.5; GPT-5.5 Fast uses 2.5×. Pricing reviewed against the official API tables on 2026-10-12.
  • Prefer complete exact benchmark response usage over cumulative/context snapshots; retain the model captured for each response, rather than applying the final model to every response. Identical notification replays are idempotent, conflicting usage or model replays fail closed, and repeated measurement does not add costs twice. Omitted usage retains its response ID/model. Count and diagnose only failing responses, retain verified costs for priceable responses, and report aggregate failures once without attributing them to every response. Missing/cancelled usage and inconsistent telemetry have separate counters from unavailable pricing.
  • Show a bounded API-EQ ISSUE panel in terminal Quota and a wrapped last usage accounting issue in browser Quota. Include the precise reason, originating session/child or benchmark, model and requested/live-reroute attribution, requested tier, reasoning effort, response/legacy/benchmark source, response ID and timestamp. Keep terminal controls within the available height. Diagnostic fields are sanitized, bounded, process-local accounting metadata; prompt/reply content is not retained. Stale benchmark/session diagnostics cannot be attributed to a newer restart.

Circumstances behind the original warning

Circumstance in v0.23.3 Handling in this PR
A missing or unmaintained model ID, including a server fallback GPT-5.2 is now maintained. Other missing/unknown IDs retain explicit model/source diagnostics. Dated snapshots of maintained models continue to resolve to their base rates.
An explicit unsupported tier/model combination Verified Ultrafast, Flex and additional Fast combinations are priced. Unverified combinations retain the requested tier and precise failure reason. Missing tier evidence still uses a qualified Standard estimate.
Cache-write or long-context usage without a maintained price class Astra's classes are already maintained; per-response boundaries now come from exact records. Unpublished classes remain unavailable rather than receiving inferred prices.
Missing, inconsistent, cancelled or unpriceable subscription benchmark results Separate missing/inconsistent/unpriced causes and retained trial/response metadata; complete exact usage is no longer rejected by synthetic cumulative snapshots.
Failure in another session or child The learning counters remain account-wide; diagnostics identify the originating session instead of implying the selected Astra session failed.

A restart reason can remain visible while the next clean interval accumulates. Ordinary missing or inconsistent response usage is not evidence of an unsupported model. There is no assumption that Pro/other subscription variants require a different Astra token price.

Codex source audit: 2026-07-12 through 2026-10-11

Audited the user's full-history clone at a9b0815167c27ab13719dd12d13b63e02b70d221. Relevant upstream changes:

Date Upstream change Implication
2026-07-14 #32985 / 1c9f890c0: exact rawResponse/completed notifications The protocol explicitly distinguishes exact responses from accumulated, estimated or replayed token counts. Benchmarks already receive these events; complete exact data is now authoritative.
2026-07-16 #33454 / 2edad72de: cache-write token usage The field and Astra price class are supported. Future unknown usage fields fail closed with a specific diagnostic.
2026-08-25 #40651 / 68301fa45: immutable settings per model step A turn may include different review/compaction models or tiers. Preserve available response-level model evidence; do not claim a turn-wide requested setting proves actual per-step billing.
2026-08-27 and 2026-09-01 #41087 / 2c4a95736, #41980 / e017e93ac: upstream usage metadata Extra metadata/amount is not assumed to be API dollars or a confirmed model/tier.
2026-08-31 #41912 / 5f79a92e3: durable response usage and response/thread/turn attribution Previously ignored by the live reader; this PR consumes the exact usage and response identity.
2026-09-02 #42372 / 503791977: Luna Reserve fallback The request wire ID is gpt-reserve; normal-model metadata is for presentation. Preserve explicit unverified-price diagnostics instead of mapping Reserve to Luna prices.
2026-10-03 and 2026-10-06 #50472 / 604061ce5, #51253 / 7ac954ea2: expanded Ultrafast support and independent speed policy Catalogue-backed command support can outpace static pricing. Verified OpenAI tier rates are now maintained; provider-specific IDs are not guessed from their display names.
2026-10-10 #52742 / 5ef96ab27: opt-in encrypted output replay No direct token-accounting change explaining the unpriced warning was found.

Evidence boundaries

Durable response usage does not include a verified model or delivered service-tier field. A matching live reroute is used where available; otherwise pricing keeps the explicitly identified requested-model/tier estimate. Late attachment, disconnection and compaction/review steps without matching model evidence retain this limitation. Reserve, preview/custom/provider IDs and unpublished price classes stay unpriced with richer detail. This PR does not infer billing from display aliases or subscription credit multipliers, change session controls, or claim knowledge of OpenAI's private quota formula.

Validation

  • All 95 browser tests pass, including narrow-screen wrapping, source metadata and text/HTML isolation for the new diagnostic.
  • Full Go race/coverage suite passes across packages excluding exhaustive locale-screen tests (CI retains its separate locale matrix); focused race reruns cover the final startup, replay and diagnostic refinements.
  • Exact/legacy migration, startup between response and snapshot, concurrent/partial appends, inherited history, malformed/unknown usage classes, duplicate/conflicting IDs and delayed daemon reroutes have regression coverage.
  • Pricing tests cover 272,000/272,001 input-token boundaries, cache classes, Astra Standard/Fast/Ultrafast/Flex, fallback snapshots, and rejection of unverified aliases. Learning tests cover signed discounts, old-diagnostic rejection and preserved terminal height.
  • Mixed-model benchmark costing, snapshot independence, idempotent measurement, missing/cancelled/invalid usage, model-only replay conflicts, per-response failure counts and aggregate failure attribution are covered. Large-rollout discovery tests cover exact records outside the 4 MiB startup tail, indistinguishable legacy histories, checkpoint deduplication and recovery when exact records arrive.
  • Svelte/type/format checks, Go vet and Windows amd64 application/UI-test cross-builds pass. Generated web assets are included; the English snapshot update reflects the intentional pricing-review date change.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Nine unresolved moderate findings affect replay handling, metadata attribution, exact-record fail-closed behavior, benchmark counting, and diagnostic layout or selection.

4 open findings
What changed in this PR

Updates Codexometer quota accounting to use exact response records, expand verified pricing, and surface bounded diagnostics in terminal and browser interfaces.

Changes:

  • Adds exact-response deduplication, attribution, baselining, and failure classification.
  • Expands model and service-tier pricing coverage.
  • Adds sanitized diagnostics, regression tests, documentation, and rebuilt web assets.
File Changes / review notes
web/​tests/​browser.spec.ts Tests responsive and safe browser diagnostics.
web/​src/​state.svelte.ts Adds diagnostic snapshot state.
web/​src/​Quota.svelte Displays wrapped usage issues.
README.md Documents pricing and accounting behavior.
internal/​web/​usage_diagnostics_test.go Tests safe diagnostic projection.
internal/​web/​state.go Projects latest diagnostics into web state.
internal/​web/​dist/​index.html Updates embedded asset references.
internal/​web/​dist/​assets/​index-GcuOkuuL.css Removes superseded stylesheet.
internal/​web/​dist/​assets/​index-Bjr_1drg.css Adds rebuilt diagnostic styles.
internal/​ui/​view.go Renders terminal quota diagnostics.
internal/​ui/​quota_api_eq.go Tracks quota issues and tier adjustments. Moderate findings: account for wrapped reset-control rows in height calculations (1 vote); select the newest diagnostic by timestamp rather than map iteration order (1 vote).
internal/​ui/​quota_api_eq_test.go Tests discounts, attribution, and terminal sizing.
internal/​ui/​model.go Integrates layout and cancellation handling.
internal/​ui/​english_snapshot_test.go Updates presentation snapshot.
internal/​ui/​benchmark_quota.go Classifies benchmark accounting failures. Moderate finding: avoid counting one unpriceable response once per response in aggregate failures (2 votes).
internal/​ui/​benchmark_quota_test.go Tests benchmark diagnostics and attribution.
internal/​codex/​usage_diagnostics.go Defines bounded diagnostic metadata and formatting.
internal/​codex/​service_tier.go Adds expanded tier pricing and long-context handling.
internal/​codex/​service_tier_test.go Tests tier and fallback pricing.
internal/​codex/​live_usage.go Integrates exact records with legacy accounting. Moderate finding: preserve pending call metadata after model-history eviction (1 vote).
internal/​codex/​live_usage_exact.go Reads, deduplicates, and attributes exact usage records. Moderate findings: fail closed on malformed recognized exact usage (1 vote); persist exact-mode knowledge when bounded startup scanning cannot establish it (2 votes).
internal/​codex/​live_usage_exact_test.go Tests exact usage lifecycle and edge cases.
internal/​codex/​live_usage_accounting_test.go Updates usage-failure classification coverage.
internal/​codex/​digbench.go Preserves per-response benchmark model data.
internal/​codex/​benchmark.go Adds pricing and benchmark telemetry handling. Moderate findings: include model in duplicate comparisons (2 votes); retain response ID/model for missing usage (2 votes); fail closed on valid-to-missing replays (1 vote).
internal/​codex/​benchmark_test.go Tests exact benchmark costing and replay handling.

🧠 Review effort: Lite


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread internal/codex/benchmark.go Outdated
Comment thread internal/codex/benchmark.go Outdated
Comment thread internal/codex/live_usage_exact.go
Comment thread internal/ui/benchmark_quota.go Outdated

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Rollout replacement can suppress later legacy usage, and mixed benchmark failures discard valid response costs.

0 open findings

4 resolved since last review
Previously missed (3)

In code that hasn't changed since last review

Medium severity Reset exact-usage state when replacing rollout files

internal/​codex/​live_usage.go:655

When an existing rollout file is truncated/replaced, this rescan inherits exactUsage, exactUsageUncertain, and the response-ID maps from the old contents. restoreExactUsageBaseline can set exact mode but never clears it, so replacing an exact-mode rollout with a legacy rollout causes every subsequent token_count response to be skipped by the exact-mode branch. Reset the exact-discovery state before rebuilding it from the replacement file.

Medium severity Preserve valid pricing when some responses have invalid usage

internal/​ui/​benchmark_quota.go:135

This early return discards every otherwise-priceable response whenever one raw response has missing or invalid usage. For example, a benchmark with one omitted response and one valid Astra response records zero cost and zero priced calls, contrary to the PR's stated requirement to retain verified costs and count only failing responses. Continue through the raw responses, skip/report only entries carrying UsageIssue/UsageMissing, and price the remaining entries.

Low severity Localize the new terminal frame title

internal/​ui/​quota_api_eq.go:584

The new terminal frame title bypasses the repository's localization path, so every non-English locale will show this heading in English. Existing frame titles use i18n.Text (for example, internal/ui/quota_reset.go:101 and internal/ui/view.go:135,269), while internal/i18n/i18n_test.go:93-124 requires complete matching catalogues. Translate this literal and add it to every locale catalogue.

🧠 Review effort: Balanced

@merefield
merefield merged commit c6c46f2 into main Oct 12, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants