Index multi-account routing and probe one catalog per profile

[maiphucgiang]

Summary

Request-path scheduling

• Steady-state pick()/headers_for() no longer rescan the credential directory: CredentialPool keeps by_id/by_cm/by_account_key indexes plus a cached profile per entry, and a background _reconcile_loop applies file changes after a directory fingerprint move. Explicit imports, logins and management paths still reload immediately; get_headers() performs exactly one file-version check on the selected account, and reconcile-time reloads skip unchanged files with a single stat (a generation gap still forces the re-read).
• One routing decision resolves one policy view (model_policy.view), so credential_enabled/route_allowed/rule_for reuse the same revision instead of re-copying the policy per candidate. Published account_catalogs are indexed once per catalog revision (app/catalog_index.py); /v1/models tables are cached by pool/policy/catalog/ledger revision under the existing TTL, and the table key is taken after the catalog index is warm, so the first build after an invalidation does not rebuild twice.
• app/routing_scheduler.py serves static eligibility per (model, region, revision-set, alias scope) shard with an LRU cache; cooldowns, sticky, capacity leases and tried stay per request. routing_scheduler = legacy | shadow | indexed; the default is shadow for a conservative rollout — it returns the legacy answer while recording bounded, credential-free mismatches under GET /admin/performance (admin-auth) — and the startup path (argparse defaults, environment fallback and the settings schema) agrees, so a deployment without explicit flags gets shadow, not indexed.
• Shards are invalidated only when entry identities, the policy, the published catalogs or an account's solvency actually change: reload() bumps the pool revision only on a real identity/profile/order move, the ledger exposes a routing_revision that moves only when a balance crosses zero or an intl account gains its first balance row, catalog republishing is keyed by content (and still publishes when account_catalogs has never been set), and the stale table is dropped before the fresh index is installed. Deleted credential files surface through the 5 s reconcile window; a removed credential selected in that window fails fast in get_headers and records a cooldown.

Catalog probing and the manual full scan

• catalog_probe_mode = account (default, wired through the same startup paths) refreshes every synced account each cycle; representative probes one healthy credential per product profile per TTL cycle — accounts without a catalog first, then the current source, then the stalest — with exponential backoff after failures, and back-fills missing or dirty account catalogs under a bounded per-cycle quota (floor 16, converging within about twelve cycles, re-woken only on progress), so new accounts become routable without a manual scan. A cycle without candidates for a profile no longer resets that profile's retry state.
• The back-fill pass holds the pool lock only for a single policy snapshot (one model_policy.view for the whole loop), skips accounts still in breaker cooldown, and guards each header fetch individually: one broken account is recorded as failed without aborting the pass, so the model cache still publishes and the remaining accounts back-fill. The serial (housekeep_concurrency=1) branch materializes its results before checking for progress.
• Probes mark an account dirty only for models newly advertised since the last representative catalog (AggregateCatalog.model_ids), so mixed catalog versions stop re-queueing every cycle; a background token refresh neither clears a pending refresh marker (only a successful catalog store does) nor sets one on its own, while imports and logins force a refresh through the credential writer. /v1/models stays the union of current per-account catalogs, request-time networking stays zero, and account-unique models keep routing only through their own account.
• POST /admin/models/scan runs a user-triggered full scan in the background (single-flight, bounded progress payload, no credential material in the view); GET /admin/models/scan reports progress, per-account success/failure, catalog differences, the overall error and a per-profile breakdown of distinct catalog versions. A failed account keeps its previous catalog. The WebUI models page gains the scan button, restores an in-progress scan on page load, polls while running, reloads the catalog list when the scan finishes, and keeps passive polling errors silent.

Background maintenance concurrency

• Account syncs, catalog refreshes and usage sweeps run on bounded workers with one semaphore per endpoint family (credits/models/usage), initialized under a lock; housekeep_concurrency (1–16, default 4) restores serial behaviour at 1. Failed accounts retry with 60·2^n backoff capped at 900 s plus jitter, and cycles skip accounts synced within five minutes.

Benchmarks

• benchmarks/bench_routing.py measures pick/header/cooldown/capacity/tried/models cases for 10–1000 accounts; per-op pick medians drop from 8.5 ms / 413 ms / 10.0 s (10/100/500) to 0.10 / 0.6 / 2.8 / 5.8 ms (10/100/500/1000) with zero summary() calls, candidate checks growing with shard size only. First pick after a revision move rebuilds the shard (1000 accounts ≈ 250 ms). A dedicated pick + revision churn case replays a token-refresh-like reconcile plus a maintenance ledger write per pick: zero summary() calls, no shard rebuild, median 0.6 / 2.9 / 15.3 ms (10/100/500) — the cost of stat-only rechecks.

Commits

• fdc4b23 Add offline routing benchmark and process counters
• b9272af Move credential file scanning out of the request path
• 2e5240a Reuse one policy revision and indexed catalogs per routing decision
• 8184dfc Add revision-keyed routing shards with legacy and shadow modes
• a395d5d Probe one representative catalog per profile and add a manual full scan
• 068b3fb Bound background maintenance concurrency per endpoint
• d526b0d Serve published model tables from the catalog index
• 1b047dd Reuse one credential read per header fetch
• f00c18d Back-fill laggard catalogs and gate the indexed rollout on shadow
• 934a3c3 Keep passive scan polling silent in the models page
• bf96143 Apply rollout defaults on real startup and harden the laggard pass
• af082cb Keep pending catalog refresh markers across background reloads
• adaf4b5 Trim lock-held policy copies and keep the published catalog index warm

Validation

• Backend: every tests/test_*.py file exits 0 (75 files), including test_routing_scheduler.py (LRU eviction, semantic revision stability, conservative startup defaults via the real main() parse path, token-rotation shard stability, pending-marker preservation), test_routing_snapshot.py, test_catalog_probe.py (new-account routability, bounded-cycle convergence, broken-account survival, cooldown skipping, retry-backoff preservation, no re-marking on unchanged catalogs, cross-account catalog-version reporting), test_environment_config.py (rollout switches resolve to shadow/account with source default and explicit CLI still wins), test_housekeep.py, test_perf_metrics.py and test_identity_sync.py (relogin still forces the catalog refresh).
• Web: vp test run 153 passed; vp check formatting, lint and type checks clean; the full Playwright suite passes 18/18 (npm run test:browser), including the models page with restored scan polling.
• Benchmark reproduced on this machine (benchmarks/bench_routing.py --accounts 10,100,500,1000 --repeat 3): steady-state medians 0.097/0.575/2.91/5.93 ms, summary() calls 0, headers_for stat calls 1.0 per operation; legacy/shadow/indexed choose identical credentials across normal, sticky, cooldown, capacity-full and failover cases; the revision-churn case shows zero summary calls and a hot shard, and current_models medians stay at the sub-millisecond table-cache level after the warm-index change.
• CI on every pushed head of this PR (final head adaf4b5): test, web, image and all three CodeQL analyses passed; CodeQL open alerts on the PR head ref: 0.

[sourcery-ai]

Sorry @maiphucgiang, your pull request is larger than the review limit of 150,000 diff characters