Skip to content

feat(usage): Capture orchestrator usage through a source ledger - #103

Merged
4ndreello merged 42 commits into
mainfrom
feat/orchestrator-usage
Sep 23, 2026
Merged

4ndreello merged 42 commits into
mainfrom
feat/orchestrator-usage

Conversation

@4ndreello

Copy link
Copy Markdown
Owner

codedeck open sessions recorded no usage, and they are almost all of the real spend. Worker usage also lost cost across process restarts: harnesses report a cumulative value per native session or process, and the store replaced the row's usage with the latest one.

Usage now goes through a ledger:

  • usage_sources keeps a global high-water mark per source key (claude:<nativeId>#<process>, codex:<thread>, session:<id>, claude-open:<nativeId>). A new observation adds only its increase over the mark, so re-reading the same cumulative value adds nothing.
  • usage_attributions records what each row got from each source and is materialized into sessions.usage_*, so existing readers keep working.
  • session_native_links keeps every native id an open row saw. The session-id hook now appends ids to the sidecar, and open runs a link watcher that sends session.linkNative and flushes before finishOpenSession and session.release.
  • On release, at daemon start for stale rows, and when a second id links, the daemon reads the Claude transcript (last cost-state, else deduped assistant tokens priced per model) and observes it. Release re-reads every link of the row, which covers resuming an id on a revived row and /clear then /resume inside one open.
  • The statusline sends its live cost with codedeck usage <run> --json --observe <session_id>=<cost>. run $ is workers plus the orchestrator sources other than the current session plus the live payload cost.
  • RunUsageSummary gains orchestrator and total; the top level stays workers only. codedeck usage gains --by origin and --backfill, which imports past orchestrator transcripts as usage_legacy entries with a mark, so a later resume of the same id counts only the increase.
  • Pricing: cached tokens are subtracted from input for codex only (PRICE-01); cached above input has no known cost.
  • Rows that already had usage before the upgrade are seeded into the incoming source when it has no mark, or into seed:<sessionId> otherwise (and when a Claude row's previous usage came from another process), so the upgrade neither drops nor double counts them.

Spec, design and tasks: .specs/features/orchestrator-usage/ (ORCH, RUN, SRC, PRICE, BF requirements).

The daemon must be restarted after deploy to create the new tables; the migration only adds tables and indexes.

Not covered: the Claude statusline cost after --resume and --fork-session was not checked against the real binary, Antigravity and OMP counters across processes are assumed cumulative per session, and the TUI dashboard with --by origin was not exercised. Backfilled legacy entries have no row attribution by design.

Tests

Scoped batches, one at a time, on the final HEAD:

  • npx vitest run tests/orchestrator-usage-daemon.test.ts tests/usage-daemon.test.ts tests/release-interrupted.test.ts tests/session-adopt.test.ts: 56 passed
  • npx vitest run tests/usage-ledger.test.ts tests/native-links.test.ts tests/usage-query.test.ts tests/usage-backfill.test.ts tests/usage-cli.test.ts tests/session-store.test.ts: 39 passed
  • npx vitest run tests/pricing.test.ts tests/usage-source.test.ts tests/claude-transcript.test.ts tests/usage-schema.test.ts tests/power-database.test.ts tests/usage.test.ts: 45 passed
  • npx vitest run tests/statusline.test.ts tests/link-watcher.test.ts tests/session-id-hook.test.ts tests/session-runtime.test.ts tests/run-env.test.ts tests/usage-statusline-contract.test.ts: 36 passed
  • npx vitest run tests/open-action.test.ts tests/open-contract.test.ts: 54 passed
  • npx vitest run tests/open-pty.test.ts tests/open-args.test.ts: 130 passed
  • npx vitest run tests/power-recover.test.ts tests/run-linkage.test.ts tests/opencode-parser.test.ts tests/usage-charts.test.ts: 24 passed
  • npx tsc --noEmit: clean; npm run build: ok
  • codedeck usage <run> --json --observe <id>=<cost>: about 72 ms per call (62 ms before)
  • Mutation probes, 13 faults: the ledger delta, the negative-delta guard, the seed rule, the PRICE-01 subtraction, codex-only cached input, first vs last cost-state, ORCH-16 previous ids, the RUN-06 exclusion, sidecar append, last sidecar id, the close-path flush and the worker-only top level were all caught. sequence <= ? vs < in the process ordinal is equivalent (event sequences are unique).

4ndreello and others added 30 commits September 22, 2026 20:09
Count cached tokens inside Codex input once and reject inconsistent token totals.

Co-Authored-By: Codex <noreply@anthropic.com>
Store source high-water marks, row attributions, and native session links so usage can be reconciled after restart.

Co-Authored-By: Codex <noreply@openai.com>
Keep worker metrics separate from orchestrator usage while exposing a combined cost.

Co-Authored-By: Codex <noreply@anthropic.com>
Expose orchestrator and worker totals so usage views include open sessions.

Co-Authored-By: Codex <noreply@anthropic.com>
Give each harness a stable source key for usage attribution.

Co-Authored-By: Codex <noreply@openai.com>
Track source marks and per-session deltas so cumulative reports can move across rows without double counting.

Co-Authored-By: Codex <noreply@openai.com>
Keep every native transcript id linked to its open session and expose unreconciled links for recovery.

Co-Authored-By: Codex <noreply@openai.com>
Extract cumulative costs and token totals from Claude transcript files.

Co-Authored-By: Codex <noreply@openai.com>
Expose the associated run id to spawned workers for usage attribution.

Co-Authored-By: Codex <noreply@openai.com>
A first cost report of zero is known usage, not missing usage. Persist its high-water mark and attribution so run completeness remains accurate.

Co-Authored-By: Codex <noreply@openai.com>
Keep token updates on the shared session write path and clear nullable cost directly, which SessionStore.update cannot represent.

Co-Authored-By: Codex <noreply@openai.com>
Keep zero-token synthetic models from making priced fallback usage incomplete.

Measure memory while parsing the large transcript fixture.

Co-Authored-By: Codex <noreply@openai.com>
Keep headroom between the generated file size and the memory limit.

Co-Authored-By: Codex <noreply@openai.com>
Track each non-incremental worker source at its high-water mark while
keeping incremental events additive. Cover Claude process changes, Codex
thread totals, and event replay with hand-made stream fixtures.

Co-Authored-By: Codex <noreply@openai.com>
Accept native ids only for open rows and validate reported costs. Return
worker and orchestrator usage with source attributions in usage.get.

Co-Authored-By: Codex <noreply@openai.com>
Record transcript cost and token high-water values on release, and revisit
older links when an open row receives another native id. Keep release
status even when transcript reading fails.

Co-Authored-By: Codex <noreply@openai.com>
Schedule transcript reconciliation after recovery without delaying the
socket listener. Preserve dead working rows that recovery terminalizes,
and keep the reconciliation promise available to tests.

Co-Authored-By: Codex <noreply@openai.com>
Carry the stored run id through DriverSession.send so follow-up worker
processes keep CODEDECK_RUN_ID. Pass the run id to the initial driver start.

Co-Authored-By: Codex <noreply@openai.com>
Retry failed transcript reads for completed or failed open rows after a
daemon restart. Keep startup recovery non-blocking and verify the retry.

Co-Authored-By: Codex <noreply@openai.com>
Derive Claude source numbering from the session.started count at the
current event sequence, matching the worker usage model.

Co-Authored-By: Codex <noreply@openai.com>
Attribute pre-upgrade usage to the incoming source when it has no mark, so the next cumulative observation adds only its increase. Keep the seed source when the incoming source already has a mark on another row.

Co-Authored-By: Codex <noreply@openai.com>
4ndreello and others added 12 commits September 22, 2026 21:17
Prove last-valid sidecar selection and Claude close-time linking order.

Co-Authored-By: Codex <noreply@openai.com>
Refresh reconciled transcript totals when an open row is resumed or later
released. The ledger high-water marks absorb unchanged cumulative values.

Co-Authored-By: Codex <noreply@openai.com>
Place unclassified prior usage under a seed source when the latest
usage event came from an earlier Claude process.

Co-Authored-By: Codex <noreply@openai.com>
Treat open rows linked to missing or unpriced transcripts as incomplete
in usage analytics.

Co-Authored-By: Codex <noreply@openai.com>
Recover transcript usage when the watcher misses its final native link.

Co-Authored-By: Codex <noreply@openai.com>
@sonarqubecloud

Copy link
Copy Markdown

@4ndreello
4ndreello merged commit 80ec486 into main Sep 23, 2026
4 checks passed
@4ndreello
4ndreello deleted the feat/orchestrator-usage branch September 23, 2026 01:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant