Bound per request AI cost (ORB-100)

[thomasluizon]

Implements ORB-100. Source: https://linear.app/useorbitai/issue/ORB-100/bound-the-per-request-ai-cost-cap-prompt-size-tool-payloads-and

Summary

• Caps the prompt habit index at 300 entries, prioritizes overdue and due today habits, preserves selected ancestor chains, and tells Astra when the index is partial.
• Caps the client history transcript at 24,000 characters using newest whole entries and drops oversized serialized tool payloads in favor of a valid JSON marker.
• Tracks reported input plus output tokens across model rounds and stops before another completion after the 120,000 token request ceiling is crossed.
• Adds a five minute, per-user chat lease in the existing rate limit bucket table. The action filter returns the existing 429 response shape and releases the exact acquired lease on success, failure, cancellation, and stream completion.
• Releases leases with an exact set-based delete, so cleanup cannot save or detach unrelated request mutations.
• Removes overflow history summarization after confirming the only production caller rejects history above 40 messages, so the extra billed call was unreachable.

No schema migration or mobile contract change is required.

Streaming lifetime verification

ProcessChatStream awaits StreamCommandResultAsync, which awaits the MediatR request and every SSE write before the action returns. The action filter awaits next(), so its finally runs after the stream body has finished or failed. Filter tests cover success, action exceptions, and client cancellation.

External interface evidence

The installed OpenAI .NET package is version 2.12.0. Its installed OpenAI.xml declares ChatTokenUsage.InputTokenCount, ChatTokenUsage.OutputTokenCount, and ChatTokenUsage.TotalTokenCount. The package changelog maps the first two properties to the REST fields prompt_tokens and completion_tokens.

The SDK transport test invokes the installed ChatClient with this complete redacted buffered response shape:

{"id":"chatcmpl-test","object":"chat.completion","created":1700000000,"model":"gpt-test","choices":[{"index":0,"message":{"role":"assistant","content":"Hi there"},"finish_reason":"stop"}],"usage":{"prompt_tokens":11,"completion_tokens":7,"total_tokens":18,"prompt_tokens_details":{"cached_tokens":3}}}

The resulting ReportedTokenCount is asserted as 18 for buffered and streaming responses. Re-derive with dotnet test tests/Orbit.Infrastructure.Tests --filter FullyQualifiedName~AiIntentServiceStreamingTests.

Verification

• dotnet build Orbit.slnx --no-restore: zero errors.
• dotnet test Orbit.slnx --no-build: 5,744 passed, zero failed.
• Focused prompt, payload, token, lease, filter, review regression, and analyzer coverage is included.

Assumptions

• The 300 entry ceiling counts required ancestors. I rejected allowing ancestors beyond the ceiling because that would leave prompt growth unbounded.
• The concurrent lease applies to ProcessChat and ProcessChatStream, not audio transcription. I rejected a controller-level filter because ORB-100 describes Astra chat completion work.
• The cumulative token guard stops when reported usage exceeds 120,000. I rejected stopping at equality because the ticket consistently says passes or crosses the ceiling.

[thomasluizon]

Approach: add the five safety constants in AppConstants, cap and prioritize the habit index while carrying a partial flag into prompt rendering, cap history entries and serialized tool payloads in AiIntentService, and expose cumulative reported usage so the existing tool loop can stop before another completion call. Add a dedicated database backed chat lease API using the existing unique rate limit bucket key and sentinel window, then wrap both chat actions with an action filter that acquires once and releases in finally. This keeps the existing per minute limiter unchanged, avoids a migration or tokenizer dependency, and places each bound at the layer that already owns the data. Tests will cover selection and hierarchy, transcript and payload bounds, loop termination, lease contention and expiry, and filter release on failures and cancellation.

[pullfrog]

✅ No new issues found.

Reviewed changes Reviewed the initial PR scaffold; the current head contains no file changes or behavioral surface.

• Initial scaffold: The sole commit establishes the ORB-100 branch without changing code, contracts, configuration, or tests.

  ^{｜ View workflow run ｜ Using GPT Sol ｜ 𝕏}

[pullfrog]

Caution

Lease cleanup can commit a destructive chat mutation after the request failed or was cancelled.

Reviewed changes Reviewed both implementation commits added since the prior Pullfrog review, covering the following behavioral surfaces.

• Bounded habit context: Capped the prompt index at 300 prioritized habits while preserving selected ancestor chains and marking truncated indexes as partial.
• Bounded history and tools: Limited history to 24,000 characters of newest whole entries, removed overflow summarization, and replaced oversized tool payloads with a JSON drop marker.
• Cumulative token guard: Propagated reported OpenAI usage through each round and stopped the tool loop before another completion after cumulative usage exceeds 120,000 tokens.
• Per-user chat lease: Added a database-backed lease around buffered and streaming chat actions, including exact lease identity on release to preserve successor requests.
• Focused verification: Added prompt, history, tool payload, token usage, action filter, and lease lifecycle tests.

  ^{｜ Fix all ➔ ｜ Fix 👍s ➔ ｜ View workflow run ｜ Using GPT Sol ｜ 𝕏}

[pullfrog]

✅ No new issues found.

Reviewed changes Reviewed the two commits added since the prior Pullfrog review, with full PR diff coverage for regression context.

• Isolated lease cleanup: Replaced tracked lease removal plus SaveChangesAsync with an exact set-based delete that cannot persist or detach unrelated request mutations.
• Added relational regression coverage: Moved lease lifecycle cases onto SQLite and verified cleanup leaves an unrelated pending entity tracked but unsaved.
• Refreshed architecture output: Updated the generated architecture report for the added code and tests without changing runtime behavior.

  ^{｜ View workflow run ｜ Using GPT Sol ｜ 𝕏}

[sonarqubecloud]

Quality Gate passed

Issues
4 New issues
0 Accepted issues

Measures
0 Security Hotspots
81.7% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud