feat(auth): active credential health prober - #5162
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ecabebae22
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
ecabeba to
5e788a0
Compare
Default probe path is now /models, so base URLs that already contain an API version (e.g., /v1, /v1beta) append the endpoint correctly rather than duplicating a /v1 prefix. resolveProbeURL still deduplicates when an explicit /v1/models path is configured. Updated config example and added unit tests.
The ticker select used a bare break, which exited only the select and could continue into the next auth iteration with a canceled context. Use a loop flag set by the ctx.Done() case so the outer sweep loop exits before launching any further probes.
probe now creates a context without the parent's deadline while still propagating cancellation, so an inherited whole-request timeout cannot expire while reading response headers/body. Added TestProberDropsContextDeadline.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a4c52840f3
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Manager now accepts a prober parent context via SetProberParentContext; StartProber derives from that context so cancellation propagates from Service.Run. StopProber waits for the run goroutine to return before proceeding. Service.Run sets the prober parent to its lifecycle context.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 99e0e37dc9
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Prober failures now compute a RetryAfter from BackoffBase/BackoffMax using an exponential ladder stored on the auth (proberBackoff). applyAuthFailureState honors RetryAfter for ErrorCodeForceCooldown instead of applying the generic status-code cooldowns (e.g. 30 min for 401, 12 h for 404). Successful probes reset proberBackoff. Added tests for the ladder, override, and escalation.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: eedc550267
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Prober start/stop now share a dedicated proberLifecycleMu so that a config-driven restart cannot be observed as stopped-then-started by a concurrent shutdown. StartProber/StopProber lock the whole operation; internal start/stop helpers assume the lock is held. restartProberLocked acquires the lock for its stop-or-start sequence. Service shutdown now stops the file watcher before stopping the prober, preventing in-flight config updates from launching a new prober after shutdown begins. Added tests covering StopProber waiting on an in-flight probe and StartProber blocking while StopProber holds the lifecycle lock.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: cc0000fb87
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Credential-scoped probe failures set auth.Unavailable/auth.NextRetryAfter without updating ModelStates, so a credential with a clean model state remained selectable. Add an unexported authLevelCooldown flag set by applyAuthFailureState and checked by isAuthBlockedForModel before per-model state. This lets prober/invalid_grant/auth-level 5xx blocks precede model state, while aggregated model-only unavailability still allows sibling model selection. Added tests for manual auth-level block and end-to-end prober MarkResult.
The prober looked up executors by raw auth.Provider, so openai-compatibility credentials with compat_name/provider_key and mixed-case provider values were silently skipped. Use executorKeyFromAuth so the lookup matches the same canonical provider key used by Manager.HttpRequest and scheduling. Added TestProberUsesCanonicalExecutorKey and TestProberUsesCompatNameExecutorKey.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: dcb16b55d6
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
`SetConfig`, `SetConfigSnapshot`, and `ApplyConfigWithCooldownStateStore` previously held `configCooldownMu` while calling `restartProberLocked`. `restartProberLocked` stops the running prober and waits for its goroutines; those goroutines call `m.persistCooldownStates` and wait for `configCooldownMu`, producing a lock-order inversion. Move the stop-and-wait outside the config lock, and have `restartProberLocked` read the latest runtime config from `m.runtimeConfig` directly. Added `TestSetConfigDoesNotDeadlockWithFailingProbe`.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f74c58b315
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
`RestoreCooldownStates` restores auth-level `Unavailable`/`NextRetryAfter` from persisted records but did not set `authLevelCooldown`, so a credential with a clean `ModelStates` map was selectable despite the restored probe backoff. Set the marker when restoring an auth-level (no-model) record and clear it in `clearCooldownStateForAuth`. Added `TestManager_RestoreAuthLevelCooldownBlocksCleanModelState`.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 578d6b6660
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
`SetConfig`/`ApplyConfigWithCooldownStateStore` called from `Builder.Build` before `Service.Run`, starting the prober under `context.Background()` before auths/executors were loaded. `restartProberLocked` now checks `m.proberParent` and only starts when the service lifecycle context is installed. `SetProberParentContext` no longer starts the prober. `Service.Run` now calls `RestartProber` after auth/executor initialization so the first sweep targets real credentials. Added `RestartProber` public method for the service. Added `TestProberDoesNotStartBeforeParentContext`; updated prober tests to set parent context.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 52cf58bc85
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
The global /models default is only correct for OpenAI-compatible upstreams that embed the API version in base_url. Gemini, aistudio, xAI, and Kimi need versioned model-list paths; without them the prober receives 404 and force-cools healthy credentials. probe now asks the canonical executor for its identifier and maps it to the right versioned path (/v1beta/models for Gemini, /v1/models for OpenAI-compat/ xAI/Kimi). Unknown providers still use the configured default. resolveProbeURL continues to deduplicate the version segment when base_url already contains it. Added TestProberProbePathForProvider and TestProberUsesProviderSpecificProbePath.
When a config reload or shutdown cancels the prober while a probe is already in flight, the executor returned context.Canceled and the probe converted it into a forced cooldown. Check probeCtx.Err() for context.Canceled after the HTTP call and return without calling MarkResult, so lifecycle cancellation does not suspend healthy credentials. Added TestProberIgnoresCancellationResult.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8629894107
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
With more credentials than the rate limit, snapshotAuths holds auth pointers for long enough that an auto-refresh or watcher update can replace the credential before the probe runs. probe now re-fetches the auth under the manager lock immediately before dispatch and discards the result if the auth pointer has been replaced while the request was in flight, so a stale token does not force-cool the healthy replacement. Added TestProberDiscardsResultIfAuthReplaced.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 29d9891eee
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Probe results now stay out of stats, run the hook async, and are discarded when the auth is replaced during the probe. Transport errors are redacted, 204 responses are accepted, and OAuth credentials are refreshed once on a 401 before cooling. Prober lifecycle waits until executors and auths are registered. Added tests for each P1/P2 scenario.
Custom openai-compatible providers now fall back to the configured OpenAICompatibility.BaseURL when the auth has no explicit base_url.
SetConfig/SetConfigSnapshot now schedule prober restart in a background goroutine so hooks that call them from the prober's own callback path (e.g. OnAuthRegistered during a 401 refresh) cannot deadlock on proberWg.Wait. Added regression test.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2c7cb409c2
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
…ooldown in sweeps
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7b94a14736
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0a7bf032be
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
…h compat names case-insensitively
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d79b2a5cd0
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
…h hook context from probe lifetime
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: dda6f57871
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0e0946c636
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
…ze prober restarts; rate-limit post-refresh retries
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2a00e1974b
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
…r SourceAuth before hooks
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: aabadb403f
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3961c8f630
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 94d4c1e1f3
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 298e7e465a
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: aedeb2c298
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ec7d21fa3b
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
…or custom probe paths
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 243c4f18a9
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
…h; tie prober hook context to service lifecycle
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 67601d8bf9
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
…void race on proberCtx
1 participant
Summary
Port of Plus #220 to stock. Adds an opt-in
credential-proberconfig block and a periodic health prober insdk/cliproxy/auth.internal/config/prober.gowithCredentialProberConfig; wired intoConfig.sdk/cliproxy/auth/conductor_prober.go(+ tests) runs periodicGET {base_url}/v1/modelsprobes via the existingProviderExecutor.HttpRequest.Manager.MarkResultwithCredentialScope: trueandErrorCodeForceCooldown, reusing the existing cooldown/suspension machinery.conductor.gogains prober lifecycle fields;conductor_cooldown.gostarts/stops the loop on config changes.config.example.yaml.rate-limit-per-minute, concurrency-limited bymax-concurrency, timeout-gated.Relates to #220.