Skip to content

feat(providers): add request pacing controls - #1584

Closed
kimrinking-cell wants to merge 13 commits into
lidge-jun:devfrom
kimrinking-cell:codex/provider-request-pacing
Closed

kimrinking-cell wants to merge 13 commits into
lidge-jun:devfrom
kimrinking-cell:codex/provider-request-pacing

Conversation

@kimrinking-cell

@kimrinking-cell kimrinking-cell commented Aug 13, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Add proactive outbound request-start pacing per provider and exact model id.
  • Add dashboard controls for RPM/minimum interval, live queue status, and model overrides.
  • Keep queue waiting outside the upstream response-header timeout and cover HTTP, Responses WebSocket, and adapter fetch transports.

Provider request pacing dashboard

Verification

  • bun run typecheck
  • bun run test — 11,475 passed, 8 environment-dependent skipped, 0 failed
  • cd gui && bun test tests — 774 passed, 0 failed
  • cd gui && bun run lint
  • cd gui && bun run lint:i18n
  • cd gui && bun run build
  • cd docs-site && bun run build — 265 pages built
  • bun run privacy:scan
  • Isolated browser verification of the NVIDIA NIM settings view at 38 provider RPM and a 10 RPM model override

Checklist

  • Scope stays focused and avoids unrelated cleanup.
  • Docs or release notes were updated when needed.
  • Security-sensitive changes were reviewed for secrets, auth, and unsafe defaults.

Review readiness checklist

This PR stays in draft until every box below is ticked. Tick all four boxes once the requirements are met:

  • All CI tests are green on my local testing.

  • I pushed my PR to the latest dev commit.

  • I resolved all correct Codex and CodeRabbit findings.

  • My PR is ready for review.

Summary by CodeRabbit

  • New Features
    • Added configurable provider request pacing to control outbound request-start rates.
    • Supports provider-wide and model-specific limits using requests-per-minute and minimum-interval settings.
    • Added queue status, next-slot visibility, cancellation handling, and pacing controls in provider settings.
    • Added validation, persistence, management API support, and localized interface text.
  • Documentation
    • Documented configuration options, supported transports, and queue timeout behavior.
  • Bug Fixes
    • Prevented queued time from consuming response-header timeout budgets.

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request intake: hygiene-blocked Deterministic PR hygiene checks failed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants