Skip to content

feat: show token utilization to users in CLI and extension #223

Description

@arunSunnyKVS

Summary

opfor run makes multiple LLM calls per run (attacker generation, adaptive turns, judge) via the Vercel AI SDK, which returns token usage (inputTokens, outputTokens, totalTokens) on every generateText() result. Currently, all usage data is discarded — the user has no visibility into how many tokens a run consumed.

Motivation

  • Users running Opfor against paid LLM APIs need to understand and manage cost.
  • CI/CD pipelines need token usage for budgeting and alerting.
  • Helps users optimize: fewer evaluators, fewer turns, or cheaper models when costs are too high.

Proposed Behavior

CLI (opfor run)

Display token totals at the end of the run, after the results summary:

Results: 5 passed, 2 failed, 0 errors
Safety score: 71%

Token usage: 51,323 input / 6,057 output (57,380 total)

Report: .opfor/reports/run-report-.../target-report.html

Browser Extension

Show a token summary line on the "Done" screen below the pass/fail results:

Tokens used: 57.4K

Report (HTML + JSON)

  • JSON: summary.tokenUsage with totalInputTokens, totalOutputTokens, totalTokens on UnifiedRunReport. Per-evaluator tokenUsage on EvaluatorResult.
  • HTML: Token totals in the executive summary card. Per-evaluator breakdown in the evaluator accordion.

NDJSON events (--events)

run_finish event includes token usage in the summary.

Implementation Notes

  • Track at the per-evaluator level internally, aggregate to per-run for display.
  • Capture usage from every generateText() / generateObject() return value in the core engine.
  • The MCP judge path (chatCompletionJsonContent) uses raw fetch and currently discards usage from the response JSON — needs to be parsed and returned.
  • Thread a TokenTracker through RunAllOptions → evaluator loop → attack drivers (similar to how AbortSignal was threaded for graceful shutdown).
  • No attacker/judge role split in the initial implementation — total tokens only. Role-level split can be added later if users request it.
  • No cost estimation in v1 — token counts are the reliable metric. Model pricing is volatile and varies by provider.

Scope

  • CLI end-of-run display
  • Extension "Done" screen
  • HTML report (summary + per-evaluator)
  • JSON report fields
  • NDJSON --events output
  • Out of scope: cost estimation, attacker vs judge split, per-attack granularity

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions