Summary
opfor run makes multiple LLM calls per run (attacker generation, adaptive turns, judge) via the Vercel AI SDK, which returns token usage (inputTokens, outputTokens, totalTokens) on every generateText() result. Currently, all usage data is discarded — the user has no visibility into how many tokens a run consumed.
Motivation
- Users running Opfor against paid LLM APIs need to understand and manage cost.
- CI/CD pipelines need token usage for budgeting and alerting.
- Helps users optimize: fewer evaluators, fewer turns, or cheaper models when costs are too high.
Proposed Behavior
CLI (opfor run)
Display token totals at the end of the run, after the results summary:
Results: 5 passed, 2 failed, 0 errors
Safety score: 71%
Token usage: 51,323 input / 6,057 output (57,380 total)
Report: .opfor/reports/run-report-.../target-report.html
Browser Extension
Show a token summary line on the "Done" screen below the pass/fail results:
Report (HTML + JSON)
- JSON:
summary.tokenUsage with totalInputTokens, totalOutputTokens, totalTokens on UnifiedRunReport. Per-evaluator tokenUsage on EvaluatorResult.
- HTML: Token totals in the executive summary card. Per-evaluator breakdown in the evaluator accordion.
NDJSON events (--events)
run_finish event includes token usage in the summary.
Implementation Notes
- Track at the per-evaluator level internally, aggregate to per-run for display.
- Capture usage from every
generateText() / generateObject() return value in the core engine.
- The MCP judge path (
chatCompletionJsonContent) uses raw fetch and currently discards usage from the response JSON — needs to be parsed and returned.
- Thread a
TokenTracker through RunAllOptions → evaluator loop → attack drivers (similar to how AbortSignal was threaded for graceful shutdown).
- No attacker/judge role split in the initial implementation — total tokens only. Role-level split can be added later if users request it.
- No cost estimation in v1 — token counts are the reliable metric. Model pricing is volatile and varies by provider.
Scope
Summary
opfor runmakes multiple LLM calls per run (attacker generation, adaptive turns, judge) via the Vercel AI SDK, which returns token usage (inputTokens,outputTokens,totalTokens) on everygenerateText()result. Currently, all usage data is discarded — the user has no visibility into how many tokens a run consumed.Motivation
Proposed Behavior
CLI (
opfor run)Display token totals at the end of the run, after the results summary:
Browser Extension
Show a token summary line on the "Done" screen below the pass/fail results:
Report (HTML + JSON)
summary.tokenUsagewithtotalInputTokens,totalOutputTokens,totalTokensonUnifiedRunReport. Per-evaluatortokenUsageonEvaluatorResult.NDJSON events (
--events)run_finishevent includes token usage in the summary.Implementation Notes
generateText()/generateObject()return value in the core engine.chatCompletionJsonContent) uses rawfetchand currently discardsusagefrom the response JSON — needs to be parsed and returned.TokenTrackerthroughRunAllOptions→ evaluator loop → attack drivers (similar to howAbortSignalwas threaded for graceful shutdown).Scope
--eventsoutput