From 5d8d67afd3ac76b3c8f83dfa46d0d764c5f9eb8d Mon Sep 17 00:00:00 2001 From: Valeriy Selitskiy <239034+iamwavecut@users.noreply.github.com> Date: Mon, 12 Oct 2026 16:27:29 +0200 Subject: [PATCH 01/18] Design backend lanes: per-backend queues, fallback on failure or budget --- .../specs/2026-10-12-backend-lanes-design.md | 163 ++++++++++++++++++ 1 file changed, 163 insertions(+) create mode 100644 docs/superpowers/specs/2026-10-12-backend-lanes-design.md diff --git a/docs/superpowers/specs/2026-10-12-backend-lanes-design.md b/docs/superpowers/specs/2026-10-12-backend-lanes-design.md new file mode 100644 index 00000000..9a2ebe53 --- /dev/null +++ b/docs/superpowers/specs/2026-10-12-backend-lanes-design.md @@ -0,0 +1,163 @@ +# Backend Lanes Design + +## Goal + +Load every remote inference backend at its physical width and no further. A request waits in +its own backend's queue instead of jumping to another backend because the first one is busy. +A fallback model serves a request only when the primary fails or when waiting would exhaust the +request's budget. + +## Current State + +- Capacity pools (`llm_capacity_pools`, `provider_models.pool_id`) are counters. `PoolRegistry` + keeps `in_flight` per pool and one registry-wide `Notify`. A release in any pool wakes every + waiter of every pool, and waiters race to re-acquire. There is no queue order and no priority. +- The walker (`RoutedAttemptWalker::run`) skips a candidate whose pool is full. A primary-role + candidate may first wait `primary_slot_wait` (dialog 12 s, media optimizer 15 s, YouTube and + history summary 60 s); then the walker takes the next candidate with a free slot. When a whole + pass is busy it waits for any release up to `max_slot_wait` (dialog 1 s, default 300 s). +- Unpooled models (GPT Luna) are always free, so a busy Gemma on GPU0 spills to Qwen on GPU2 and + then to Luna. In the 24 h before 2026-10-12 13:00 UTC the dialog recorded 3,365 primary slot + waits; 1,525 dialog runs started on GPU2 and Luna served 3,255 dialog requests. +- Dialog workers are derived as the sum of the pool widths of every model in the dialog chain + plus `PERSISTENT_QUEUE_DIALOG_UNPOOLED_SHARE` per unpooled model, capped by + `PERSISTENT_QUEUE_DIALOG_WORKERS_CAP`. Production computes Gemma 8 + Qwen 8 + Luna 10 = 26 and + runs 24 workers against 8 Gemma lanes, treating fallback capacity as extra throughput. +- Memory workers are `max(configured, min(memory pool width, cap 4))`. +- Pools attach to model rows, not to servers. A model row that reaches a server without a pool + bypasses its width (the vision fallback did until migration 198). +- Server widths are configured twice: in the bot's pools and in each service's Discovery + registration (`max_concurrent_jobs`). They drift apart without any signal. +- Walkers are constructed in 25 places, each with its own wait settings. + +Measured turn shape (2026-10-12): a dialog turn's median is about 9 s, of which one model +round is about 3–4 s; materialization, tools and Telegram sends take the rest. With 24 workers +GPU0 averaged about 2 busy lanes of 8 and peaked at 8. + +## Design + +### Backends and lanes + +A *lane pool* is the width of one physical backend: one server that accepts a fixed number of +concurrent requests. The existing `llm_capacity_pools` rows become lane pools; their meaning +changes from "budget shared by some models" to "the server's width". + +| Lane pool | Backend | Width | +| --- | --- | ---: | +| `aifarm-vllm-gpu0` | Gemma on GPU0 (dialog, vision, agents, media optimizer) | 8 | +| `aifarm-gpu2-qwen27b` | Qwen on GPU2 (memory, history summary, YouTube fallbacks, dialog fallback, vision fallback) | 8 | +| `aifarm-embedder` (new) | embedder (CPU) | 8 | +| `aifarm-privacy-filter` (new) | privacy filter (GPU1, under the arbiter) | 4 | +| `aifarm-asr` (new) | speech recognition (GPU1, under the arbiter) | 1 | +| `aifarm-draw-gpu1` | draw and music (GPU1, under the arbiter) | 1 | +| external APIs | OpenRouter and other hosted APIs | unpooled | + +Invariants, checked whenever the routing table loads: + +1. Every enabled model whose provider targets a Discovery service belongs to a lane pool. +2. All enabled models that target the same Discovery service belong to the same lane pool. +3. A lane pool's width equals the `max_concurrent_jobs` of its Discovery service. + +A violation emits a `lane_config_mismatch` routing event and a readiness warning; routing keeps +working with the bot's configured width. Invariant 3 is checked against Discovery's +`GET /v1/services/{name}/capacity`, so the farm registration stays the server-side truth and +drift becomes visible. + +### Lane queue + +`PoolRegistry` keeps one queue per lane pool. `acquire(pool, priority, deadline)` either takes a +free lane or enqueues a waiter. A released lane is handed directly to the head waiter: highest +priority first, then arrival order. There is no wake-all and no race. A waiter that gives up +(deadline, cancellation) leaves the queue; a handoff to a dropped waiter passes to the next one. +Resizing keeps in-flight accounting: a grown pool serves waiters at once, and a shrunk pool lets +in-flight work finish before new admissions. `try_acquire` stays for non-blocking checks. + +Priorities, stored per workflow in a new `workflows.lane_priority` column: + +| Priority | Workflows | +| --- | --- | +| 0 interactive | `dialog`, `agentic_*`, `vision`, `voice_transcription`, image and music workflows, `redaction` and `embedding` on the dialog path | +| 1 user tool | `history_summary`, `youtube_summary`, `media_prompt_optimizer` | +| 2 background | `memory_extraction` (including its resolution calls), `memory_subject_merge`, memory-write `embedding` and `redaction` | + +Embedding and redaction serve both the dialog path and memory writes. The caller passes its +priority in `RoutedRequestContext`; the workflow column is the default. + +Each pool also keeps an exponentially weighted mean of lane hold time, used for the forecast +below and exposed for observability. + +### Walker policy: fallback on failure or budget + +For each candidate in chain order, the walker: + +1. skips it if its circuit breaker is open (failure); +2. computes the forecast wait: `ceil((waiters ahead + 1) / width) × mean hold time`; +3. if the request has a deadline and the forecast exceeds the admission budget, moves to the + next candidate without waiting (budget); +4. otherwise waits in the candidate's queue until a lane is handed over or the admission budget + ends, then moves to the next candidate. + +The admission budget is the time left until the request deadline minus a fallback reserve: the +time the next candidate needs to answer. The reserve is a per-workflow value +(`workflows.fallback_reserve_ms`): 30 s for interactive and user-tool workflows, 0 for +background ones. A request without a deadline (background work) waits in its queue without a +limit and never falls back because of load. + +`primary_slot_wait`, the busy-skip pass and the per-construction `max_slot_wait` disappear; the +policy is the same for all 25 walker constructions. Unpooled candidates are admitted at once. + +### Workers + +Workers run whole requests and are no longer derived from fallback capacity. + +- Dialog workers = width of the dialog primary's lane pool × `DIALOG_WORKERS_PER_LANE` + (default 2), so 16 in production. Workers beyond the busy lanes wait in Gemma's queue during + their model rounds and run their non-model phases meanwhile. `PERSISTENT_QUEUE_DIALOG_UNPOOLED_SHARE` + and the fallback terms of `derived_worker_count` are removed. +- Memory workers = width of the memory primary's lane pool × `MEMORY_WORKERS_PER_LANE` + (default 2), so 16. Memory runs spend about 70% of their time outside the model; extra workers + keep GPU2's lanes busy at background priority, behind dialog fallbacks and user tools. +- Both resize when the routing table reloads. + +### Observability + +Per lane pool: width, in-flight, queue depth per priority, mean hold time, and recent wait +percentiles, in the admin routing status and the runtime GraphQL API. Routing events: +`lane_wait` (waited, position, priority) replaces `primary_slot_wait` and `capacity_pool_wait`; +`lane_bypass` records a budget skip with the forecast; `lane_config_mismatch` records invariant +violations. + +## Migrations + +1. Lane pools for the embedder (8), privacy filter (4) and ASR (1); attach their model rows. +2. `workflows.lane_priority SMALLINT NOT NULL DEFAULT 0` and + `workflows.fallback_reserve_ms INTEGER NOT NULL DEFAULT 30000`, set to the values above. + The down migration drops both columns and the new pools. + +## Error Handling + +- A lane wait that ends at the deadline returns the existing capacity-timeout error, so turn + outcome classification and user signals do not change. +- A dropped waiter or a cancelled request never holds or leaks a lane; permits stay RAII. +- If Discovery is unreachable during the width check, the check is skipped and retried on the + next reload; routing is unaffected. + +## Testing + +- Lane queue unit tests: FIFO within a priority, priority order, handoff on release, waiter + cancellation, deadline expiry, grow and shrink under load, no leaked permits. +- Walker tests: a busy primary is waited for instead of skipped; an open breaker falls back + at once; a forecast beyond the budget falls back without waiting; a request without a deadline + waits; unpooled candidates are immediate. +- Worker sizing tests for dialog and memory. +- Invariant tests for the routing table loader, including the vision fallback case. +- Production verification after deploy: dialog requests served by GPU2 and Luna drop to + failures and budget skips only; GPU0 lane occupancy and queue waits are visible; memory runs at + background priority without delaying dialog fallbacks. + +## Out of Scope + +- The GPU1 arbiter's internal scheduling. +- Hourly memory windows (the next change, after this one). +- Discovery server changes beyond the read-only width check. +- Startup catch-up and restart cost. From 31aad5206fe33178ecf65735f2eb698f774bb586 Mon Sep 17 00:00:00 2001 From: Valeriy Selitskiy <239034+iamwavecut@users.noreply.github.com> Date: Mon, 12 Oct 2026 16:32:57 +0200 Subject: [PATCH 02/18] Backend lanes: one sequential GPU1 lane for draw, music and ASR --- .../specs/2026-10-12-backend-lanes-design.md | 40 +++++++++++++------ 1 file changed, 28 insertions(+), 12 deletions(-) diff --git a/docs/superpowers/specs/2026-10-12-backend-lanes-design.md b/docs/superpowers/specs/2026-10-12-backend-lanes-design.md index 9a2ebe53..fe5b53e0 100644 --- a/docs/superpowers/specs/2026-10-12-backend-lanes-design.md +++ b/docs/superpowers/specs/2026-10-12-backend-lanes-design.md @@ -38,25 +38,34 @@ GPU0 averaged about 2 busy lanes of 8 and peaked at 8. ### Backends and lanes -A *lane pool* is the width of one physical backend: one server that accepts a fixed number of -concurrent requests. The existing `llm_capacity_pools` rows become lane pools; their meaning -changes from "budget shared by some models" to "the server's width". +A *lane pool* is the width of one physical resource: the number of requests it can run at once. +Usually that is one server; several services that must not run concurrently share one pool. The +existing `llm_capacity_pools` rows become lane pools; their meaning changes from "budget shared +by some models" to "the resource's width". | Lane pool | Backend | Width | | --- | --- | ---: | | `aifarm-vllm-gpu0` | Gemma on GPU0 (dialog, vision, agents, media optimizer) | 8 | | `aifarm-gpu2-qwen27b` | Qwen on GPU2 (memory, history summary, YouTube fallbacks, dialog fallback, vision fallback) | 8 | | `aifarm-embedder` (new) | embedder (CPU) | 8 | -| `aifarm-privacy-filter` (new) | privacy filter (GPU1, under the arbiter) | 4 | -| `aifarm-asr` (new) | speech recognition (GPU1, under the arbiter) | 1 | -| `aifarm-draw-gpu1` | draw and music (GPU1, under the arbiter) | 1 | +| `aifarm-gpu1` (renamed `aifarm-draw-gpu1`) | draw, music and speech recognition on GPU1, one at a time | 1 | +| `aifarm-privacy-filter` (new) | privacy filter on GPU1, beside the single lane | 4 | | external APIs | OpenRouter and other hosted APIs | unpooled | +GPU1 runs under the VRAM arbiter, which loads and offloads models on demand. Draw, music and +speech recognition share one sequential lane: while one of them runs, the others wait, so the +arbiter loads one heavy model set at a time instead of swapping between concurrent jobs. The +privacy filter keeps its own pool and runs beside that lane; the arbiter already keeps it +resident or moves it to the CPU. + Invariants, checked whenever the routing table loads: 1. Every enabled model whose provider targets a Discovery service belongs to a lane pool. -2. All enabled models that target the same Discovery service belong to the same lane pool. -3. A lane pool's width equals the `max_concurrent_jobs` of its Discovery service. +2. All enabled models that target the same Discovery service belong to the same lane pool. A + pool may span several services when they share one sequential resource (GPU1's draw, music + and speech recognition). +3. A lane pool's width does not exceed the `max_concurrent_jobs` of any Discovery service it + spans, and equals it when the pool spans one service. A violation emits a `lane_config_mismatch` routing event and a readiness warning; routing keeps working with the bot's configured width. Invariant 3 is checked against Discovery's @@ -76,13 +85,16 @@ Priorities, stored per workflow in a new `workflows.lane_priority` column: | Priority | Workflows | | --- | --- | -| 0 interactive | `dialog`, `agentic_*`, `vision`, `voice_transcription`, image and music workflows, `redaction` and `embedding` on the dialog path | -| 1 user tool | `history_summary`, `youtube_summary`, `media_prompt_optimizer` | +| 0 interactive | `dialog`, `agentic_*`, `vision`, `voice_transcription`, `redaction` and `embedding` on the dialog path | +| 1 user tool | `history_summary`, `youtube_summary`, `media_prompt_optimizer`, image and music workflows | | 2 background | `memory_extraction` (including its resolution calls), `memory_subject_merge`, memory-write `embedding` and `redaction` | Embedding and redaction serve both the dialog path and memory writes. The caller passes its priority in `RoutedRequestContext`; the workflow column is the default. +On the GPU1 lane, a voice message's transcription goes ahead of queued draws and songs, but a +running draw or song is never interrupted: the transcription waits for it to finish. + Each pool also keeps an exponentially weighted mean of lane hold time, used for the forecast below and exposed for observability. @@ -129,7 +141,8 @@ violations. ## Migrations -1. Lane pools for the embedder (8), privacy filter (4) and ASR (1); attach their model rows. +1. Rename `aifarm-draw-gpu1` to `aifarm-gpu1` and attach the speech recognition model to it; + add lane pools for the embedder (8) and the privacy filter (4) and attach their model rows. 2. `workflows.lane_priority SMALLINT NOT NULL DEFAULT 0` and `workflows.fallback_reserve_ms INTEGER NOT NULL DEFAULT 30000`, set to the values above. The down migration drops both columns and the new pools. @@ -150,7 +163,10 @@ violations. at once; a forecast beyond the budget falls back without waiting; a request without a deadline waits; unpooled candidates are immediate. - Worker sizing tests for dialog and memory. -- Invariant tests for the routing table loader, including the vision fallback case. +- Invariant tests for the routing table loader, including the vision fallback case and a pool + that spans several services. +- A GPU1 test: a transcription queued behind a running draw starts only after the draw releases + the lane, and ahead of a second queued draw. - Production verification after deploy: dialog requests served by GPU2 and Luna drop to failures and budget skips only; GPU0 lane occupancy and queue waits are visible; memory runs at background priority without delaying dialog fallbacks. From cb74245c05cdeb376d0956d0d2bd6a2482e54771 Mon Sep 17 00:00:00 2001 From: Valeriy Selitskiy <239034+iamwavecut@users.noreply.github.com> Date: Mon, 12 Oct 2026 16:36:08 +0200 Subject: [PATCH 03/18] Backend lanes: name pools by device and purpose --- .../specs/2026-10-12-backend-lanes-design.md | 22 ++++++++++++------- 1 file changed, 14 insertions(+), 8 deletions(-) diff --git a/docs/superpowers/specs/2026-10-12-backend-lanes-design.md b/docs/superpowers/specs/2026-10-12-backend-lanes-design.md index fe5b53e0..44b5de33 100644 --- a/docs/superpowers/specs/2026-10-12-backend-lanes-design.md +++ b/docs/superpowers/specs/2026-10-12-backend-lanes-design.md @@ -45,15 +45,18 @@ by some models" to "the resource's width". | Lane pool | Backend | Width | | --- | --- | ---: | -| `aifarm-vllm-gpu0` | Gemma on GPU0 (dialog, vision, agents, media optimizer) | 8 | -| `aifarm-gpu2-qwen27b` | Qwen on GPU2 (memory, history summary, YouTube fallbacks, dialog fallback, vision fallback) | 8 | +| `aifarm-gpu0-dialog` (was `aifarm-vllm-gpu0`) | dialog model on GPU0 (dialog, vision, agents, media optimizer) | 8 | +| `aifarm-gpu2-utility` (was `aifarm-gpu2-qwen27b`) | utility model on GPU2 (memory, history summary, YouTube fallbacks, dialog fallback, vision fallback) | 8 | | `aifarm-embedder` (new) | embedder (CPU) | 8 | -| `aifarm-gpu1` (renamed `aifarm-draw-gpu1`) | draw, music and speech recognition on GPU1, one at a time | 1 | -| `aifarm-privacy-filter` (new) | privacy filter on GPU1, beside the single lane | 4 | +| `aifarm-gpu1-media` (was `aifarm-draw-gpu1`) | draw, music and speech recognition on GPU1, one at a time | 1 | +| `aifarm-gpu1-privacy` (new) | privacy filter on GPU1, beside the media lane | 4 | | external APIs | OpenRouter and other hosted APIs | unpooled | +Pool names follow `aifarm--`, so they stay true when a model or a serving +engine changes. The unused `vram-cloud` and `boogu-gpu` pools are left as they are. + GPU1 runs under the VRAM arbiter, which loads and offloads models on demand. Draw, music and -speech recognition share one sequential lane: while one of them runs, the others wait, so the +speech recognition share the `aifarm-gpu1-media` lane: while one of them runs, the others wait, so the arbiter loads one heavy model set at a time instead of swapping between concurrent jobs. The privacy filter keeps its own pool and runs beside that lane; the arbiter already keeps it resident or moves it to the CPU. @@ -92,7 +95,7 @@ Priorities, stored per workflow in a new `workflows.lane_priority` column: Embedding and redaction serve both the dialog path and memory writes. The caller passes its priority in `RoutedRequestContext`; the workflow column is the default. -On the GPU1 lane, a voice message's transcription goes ahead of queued draws and songs, but a +On the GPU1 media lane, a voice message's transcription goes ahead of queued draws and songs, but a running draw or song is never interrupted: the transcription waits for it to finish. Each pool also keeps an exponentially weighted mean of lane hold time, used for the forecast @@ -141,8 +144,11 @@ violations. ## Migrations -1. Rename `aifarm-draw-gpu1` to `aifarm-gpu1` and attach the speech recognition model to it; - add lane pools for the embedder (8) and the privacy filter (4) and attach their model rows. +1. Rename the pools to `aifarm-gpu0-dialog`, `aifarm-gpu2-utility` and `aifarm-gpu1-media`, + attach the speech recognition model to `aifarm-gpu1-media`, and add `aifarm-embedder` (8) and + `aifarm-gpu1-privacy` (4) with their model rows. Pool ids do not change, so model rows keep + their `pool_id`; the code constant that names the GPU2 pool (`MEMORY_GPU2_POOL`) follows the + rename. Previous names are recorded for the down migration. 2. `workflows.lane_priority SMALLINT NOT NULL DEFAULT 0` and `workflows.fallback_reserve_ms INTEGER NOT NULL DEFAULT 30000`, set to the values above. The down migration drops both columns and the new pools. From 36a44797db0457da16fedc5893e064bcce282d0e Mon Sep 17 00:00:00 2001 From: Valeriy Selitskiy <239034+iamwavecut@users.noreply.github.com> Date: Mon, 12 Oct 2026 21:37:09 +0200 Subject: [PATCH 04/18] Give lane pools a priority queue that hands released lanes to the next waiter --- crates/openplotva-llm/src/router/capacity.rs | 587 ++++-- crates/openplotva-llm/src/router/mod.rs | 4 +- .../plans/2026-10-12-backend-lanes.md | 1776 +++++++++++++++++ 3 files changed, 2236 insertions(+), 131 deletions(-) create mode 100644 docs/superpowers/plans/2026-10-12-backend-lanes.md diff --git a/crates/openplotva-llm/src/router/capacity.rs b/crates/openplotva-llm/src/router/capacity.rs index 1d249455..498aaac1 100644 --- a/crates/openplotva-llm/src/router/capacity.rs +++ b/crates/openplotva-llm/src/router/capacity.rs @@ -1,25 +1,56 @@ -//! Shared capacity pools: a concurrency budget over one physical resource. +//! Lane pools: the width of one physical inference resource and its queue. //! -//! Models attached to the same pool contend for its slots; a model is -//! selectable while its pool has a free slot. The registry lives outside the -//! [`super::table::RoutingTable`] (like [`super::breaker::BreakerSet`]) so -//! in-flight accounting survives an `ArcSwap` reload: [`PoolRegistry::apply`] -//! resizes cells in place instead of swapping them, and a [`PoolPermit`] owns -//! its cell `Arc` directly so a permit outlives even a deleted pool. +//! A pool admits up to `max_concurrency` requests at once. Further requests +//! wait in a per-pool priority queue (lower [`LanePriority`] first, arrival +//! order within a priority), and a released lane is handed directly to the head +//! waiter as a [`PoolPermit`] sent through a oneshot channel: no wake-all, no +//! race. A waiter that gives up leaves the queue; a permit already sent to it +//! releases the lane again when the channel drops, so no lane can leak. //! -//! A deliberate non-semaphore design: resizing `max` downward under load must -//! not lose in-flight counts (tokio `Semaphore` shrink needs `forget_permits` -//! debt bookkeeping). All locks are short synchronous sections; only the RAII -//! permit — not a guard — crosses `.await`, per the project concurrency rules. - -use std::collections::HashMap; +//! The registry lives outside the [`super::table::RoutingTable`] (like +//! [`super::breaker::BreakerSet`]) so in-flight accounting survives an +//! `ArcSwap` reload: [`PoolRegistry::apply`] resizes cells in place, and a +//! [`PoolPermit`] owns its cell `Arc` so a permit outlives even a deleted pool. +//! All locks are short synchronous sections; only the RAII permit, never a +//! guard, crosses `.await`. + +use std::collections::{BTreeMap, HashMap}; use std::sync::{Arc, Mutex, MutexGuard}; -use std::time::Duration; +use std::time::{Duration, Instant}; -use tokio::sync::Notify; +use tokio::sync::oneshot; use super::table::{PoolId, RoutingTable}; +/// Order in which the waiters of one lane pool are served: lower first. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, PartialOrd, Ord, Hash)] +pub enum LanePriority { + #[default] + Interactive = 0, + UserTool = 1, + Background = 2, +} + +impl LanePriority { + #[must_use] + pub fn from_db(value: i16) -> Self { + match value { + i16::MIN..=0 => Self::Interactive, + 1 => Self::UserTool, + _ => Self::Background, + } + } + + #[must_use] + pub fn as_str(self) -> &'static str { + match self { + Self::Interactive => "interactive", + Self::UserTool => "user_tool", + Self::Background => "background", + } + } +} + /// Desired state of one pool, resolved from the routing table. /// `max_concurrency == None` means unlimited: gauge-only, never blocks. #[derive(Clone, Copy, Debug, PartialEq, Eq)] @@ -28,18 +59,55 @@ pub struct PoolSpec { pub max_concurrency: Option, } -/// Live occupancy of one pool, for the admin status endpoint. +/// Live state of one pool, for the admin status endpoint and the runtime API. #[derive(Clone, Copy, Debug, PartialEq, Eq)] pub struct PoolOccupancy { pub id: PoolId, pub max_concurrency: Option, pub in_flight: u32, + /// Waiters per priority: interactive, user tool, background. + pub queued: [u32; 3], + pub mean_hold_ms: Option, } +/// Weight of the newest sample in the mean lane hold time. +const HOLD_MEAN_WEIGHT: f64 = 0.2; + +type WaiterKey = (LanePriority, u64); + #[derive(Debug)] struct CellState { max: Option, in_flight: u32, + waiters: BTreeMap>, + next_seq: u64, + mean_hold: Option, +} + +impl CellState { + fn new(max: Option) -> Self { + Self { + max, + in_flight: 0, + waiters: BTreeMap::new(), + next_seq: 0, + mean_hold: None, + } + } + + fn has_free_lane(&self) -> bool { + self.max.is_none_or(|max| self.in_flight < max) + } + + fn record_hold(&mut self, held: Duration) { + self.mean_hold = Some(match self.mean_hold { + Some(mean) => { + let mean = mean.as_secs_f64(); + Duration::from_secs_f64(mean + HOLD_MEAN_WEIGHT * (held.as_secs_f64() - mean)) + } + None => held, + }); + } } #[derive(Debug)] @@ -55,19 +123,26 @@ impl PoolCell { } } -/// RAII slot in a capacity pool. Dropping the permit frees the slot and wakes -/// one waiter, so cancellation of an in-flight attempt can never leak a slot. -/// A permit for an unpooled model is a no-op. +/// RAII lane of a pool. Dropping it hands the lane to the next waiter, so +/// cancellation of an in-flight attempt can never leak a lane. A permit for an +/// unpooled model is a no-op. #[derive(Debug, Default)] pub struct PoolPermit { cell: Option>, - released: Option>, + acquired_at: Option, } impl PoolPermit { fn unpooled() -> Self { Self::default() } + + fn held(cell: Arc) -> Self { + Self { + cell: Some(cell), + acquired_at: Some(Instant::now()), + } + } } impl Drop for PoolPermit { @@ -75,25 +150,47 @@ impl Drop for PoolPermit { if let Some(cell) = self.cell.take() { let mut state = cell.lock(); state.in_flight = state.in_flight.saturating_sub(1); - drop(state); - if let Some(released) = self.released.take() { - // Waiters share one registry-wide `Notify` across pools, so wake - // all of them: a single wakeup could land on a waiter of another - // pool and leave this pool's waiter asleep with a free slot. - // `notify_one` additionally stores a wakeup for a waiter that is - // between its failed acquire scan and the wait call. - released.notify_waiters(); - released.notify_one(); + if let Some(acquired_at) = self.acquired_at { + state.record_hold(acquired_at.elapsed()); } + serve_waiters(&cell, &mut state); + } + } +} + +/// Hand free lanes to waiters in queue order. A send to a waiter that already +/// gave up returns the permit; it is disarmed (the lock is held, so its drop +/// must not release) and the lane goes to the next waiter. +fn serve_waiters(cell: &Arc, state: &mut CellState) { + while state.has_free_lane() { + let Some((_, waiter)) = state.waiters.pop_first() else { + return; + }; + state.in_flight = state.in_flight.saturating_add(1); + if let Err(mut unclaimed) = waiter.send(PoolPermit::held(Arc::clone(cell))) { + unclaimed.cell = None; + state.in_flight = state.in_flight.saturating_sub(1); } } } +/// Queue entry of one waiting `acquire`. Dropping it (budget end, cancellation) +/// leaves the queue; after a handoff the entry is already gone. +struct QueuedWaiter<'a> { + cell: &'a PoolCell, + key: WaiterKey, +} + +impl Drop for QueuedWaiter<'_> { + fn drop(&mut self) { + self.cell.lock().waiters.remove(&self.key); + } +} + /// Registry of live pool cells, keyed by pool id. #[derive(Debug, Default)] pub struct PoolRegistry { cells: Mutex>>, - released: Arc, } impl PoolRegistry { @@ -108,75 +205,123 @@ impl PoolRegistry { .unwrap_or_else(std::sync::PoisonError::into_inner) } + fn cell(&self, pool: Option) -> Option> { + pool.and_then(|pool| self.lock().get(&pool).cloned()) + } + /// Reconcile the registry with a freshly loaded routing table: insert new /// pools, update `max` in place (in-flight accounting preserved), drop - /// cells for removed pools (outstanding permits keep their cell alive and - /// still decrement it harmlessly). Wakes waiters so a grown pool is - /// noticed immediately. + /// cells of removed pools (outstanding permits keep their cell alive). A + /// grown pool serves its waiters at once; a shrunk one admits nothing new + /// until in-flight work drains below the new width. pub fn apply(&self, specs: &[PoolSpec]) { - { - let mut cells = self.lock(); - cells.retain(|id, _| specs.iter().any(|spec| spec.id == *id)); - for spec in specs { - match cells.get(&spec.id) { - Some(cell) => cell.lock().max = spec.max_concurrency, - None => { - cells.insert( - spec.id, - Arc::new(PoolCell { - state: Mutex::new(CellState { - max: spec.max_concurrency, - in_flight: 0, - }), - }), - ); - } + let mut cells = self.lock(); + cells.retain(|id, _| specs.iter().any(|spec| spec.id == *id)); + for spec in specs { + match cells.get(&spec.id) { + Some(cell) => { + let mut state = cell.lock(); + state.max = spec.max_concurrency; + serve_waiters(cell, &mut state); + } + None => { + cells.insert( + spec.id, + Arc::new(PoolCell { + state: Mutex::new(CellState::new(spec.max_concurrency)), + }), + ); } } } - self.released.notify_waiters(); } - /// Try to take a slot. `None` pool (unpooled model) and pools unknown to - /// the registry always succeed with a no-op permit; a full pool returns - /// `None`. Never blocks. + /// Take a free lane without waiting. Fails while the pool is full or while + /// others wait, so it never jumps the queue. Unpooled models and pools + /// unknown to the registry always succeed with a no-op permit. #[must_use] pub fn try_acquire(&self, pool: Option) -> Option { - let Some(pool) = pool else { + let Some(cell) = self.cell(pool) else { return Some(PoolPermit::unpooled()); }; - let cell = { - let cells = self.lock(); - match cells.get(&pool) { - Some(cell) => Arc::clone(cell), - None => return Some(PoolPermit::unpooled()), - } - }; { let mut state = cell.lock(); - if let Some(max) = state.max - && state.in_flight >= max - { + if !state.waiters.is_empty() || !state.has_free_lane() { return None; } state.in_flight = state.in_flight.saturating_add(1); } - Some(PoolPermit { - cell: Some(cell), - released: Some(Arc::clone(&self.released)), + Some(PoolPermit::held(cell)) + } + + /// Whether [`Self::try_acquire`] would succeed right now. + #[must_use] + pub fn has_free_lane(&self, pool: Option) -> bool { + self.cell(pool).is_none_or(|cell| { + let state = cell.lock(); + state.waiters.is_empty() && state.has_free_lane() }) } - /// Wait until any permit is released anywhere in the registry (or a pool - /// is resized), bounded by `timeout`. Returns `false` on timeout. Callers - /// re-run their acquire scan on wakeup; a spurious wake is harmless. - pub async fn wait_for_release(&self, timeout: Duration) -> bool { - tokio::time::timeout(timeout, self.released.notified()) - .await - .is_ok() + /// Wait in the pool's queue for a lane. Returns `None` once `budget` ends; + /// the waiter then leaves the queue. `budget == None` waits without limit. + pub async fn acquire( + &self, + pool: Option, + priority: LanePriority, + budget: Option, + ) -> Option { + let Some(cell) = self.cell(pool) else { + return Some(PoolPermit::unpooled()); + }; + let (receiver, _queued) = { + let mut state = cell.lock(); + if state.waiters.is_empty() && state.has_free_lane() { + state.in_flight = state.in_flight.saturating_add(1); + drop(state); + return Some(PoolPermit::held(cell)); + } + let (sender, receiver) = oneshot::channel(); + let key = (priority, state.next_seq); + state.next_seq = state.next_seq.wrapping_add(1); + state.waiters.insert(key, sender); + ( + receiver, + QueuedWaiter { + cell: cell.as_ref(), + key, + }, + ) + }; + match budget { + Some(budget) => tokio::time::timeout(budget, receiver).await.ok()?.ok(), + None => receiver.await.ok(), + } } - /// Current occupancy of every registered pool, sorted by id. + /// Expected wait for a new request of `priority`: the waiters it would + /// queue behind, in rounds of the pool width, times the mean lane hold + /// time. `Some(ZERO)` when a lane is free; `None` for unpooled or + /// unlimited pools and before the pool's first lane was released. + #[must_use] + pub fn forecast_wait(&self, pool: Option, priority: LanePriority) -> Option { + let cell = self.cell(pool)?; + let state = cell.lock(); + let width = state.max?; + if state.waiters.is_empty() && state.has_free_lane() { + return Some(Duration::ZERO); + } + let mean_hold = state.mean_hold?; + let ahead = state + .waiters + .keys() + .filter(|(waiting, _)| *waiting <= priority) + .count(); + let rounds = (ahead + 1).div_ceil(width.max(1) as usize); + Some(mean_hold.saturating_mul(u32::try_from(rounds).unwrap_or(u32::MAX))) + } + + /// Current state of every registered pool, sorted by id. #[must_use] pub fn occupancy(&self) -> Vec { let cells = self.lock(); @@ -184,10 +329,19 @@ impl PoolRegistry { .iter() .map(|(id, cell)| { let state = cell.lock(); + let mut queued = [0_u32; 3]; + for (priority, _) in state.waiters.keys() { + let slot = &mut queued[*priority as usize]; + *slot = slot.saturating_add(1); + } PoolOccupancy { id: *id, max_concurrency: state.max, in_flight: state.in_flight, + queued, + mean_hold_ms: state + .mean_hold + .map(|mean| u64::try_from(mean.as_millis()).unwrap_or(u64::MAX)), } }) .collect(); @@ -256,6 +410,7 @@ pub fn derived_worker_count( #[cfg(test)] mod tests { + use std::sync::Mutex; use std::time::Duration; use super::super::breaker::BreakerConfig; @@ -355,77 +510,249 @@ mod tests { assert!(registry.try_acquire(Some(1)).is_some()); } - #[tokio::test(start_paused = true)] - async fn wait_for_release_wakes_on_drop() { - let registry = Arc::new(registry_with(&[PoolSpec { + fn registry(max: u32) -> PoolRegistry { + registry_with(&[PoolSpec { id: 1, - max_concurrency: Some(1), - }])); - let permit = registry.try_acquire(Some(1)).expect("slot"); + max_concurrency: Some(max), + }]) + } + #[tokio::test] + async fn released_lane_goes_to_the_oldest_waiter_of_the_best_priority() { + let pools = Arc::new(registry(1)); + let held = pools.try_acquire(Some(1)).expect("free lane"); + let order = Arc::new(Mutex::new(Vec::new())); + let mut tasks = Vec::new(); + for (name, priority) in [ + ("background", LanePriority::Background), + ("interactive-1", LanePriority::Interactive), + ("interactive-2", LanePriority::Interactive), + ] { + let pools = Arc::clone(&pools); + let order = Arc::clone(&order); + tasks.push(tokio::spawn(async move { + let permit = pools.acquire(Some(1), priority, None).await.expect("lane"); + order.lock().expect("order").push(name); + drop(permit); + })); + tokio::task::yield_now().await; + } + drop(held); + for task in tasks { + task.await.expect("task"); + } + assert_eq!( + *order.lock().expect("order"), + vec!["interactive-1", "interactive-2", "background"] + ); + } + + #[tokio::test] + async fn try_acquire_does_not_jump_the_queue() { + let pools = Arc::new(registry(1)); + let held = pools.try_acquire(Some(1)).expect("free lane"); let waiter = { - let registry = Arc::clone(®istry); - tokio::spawn(async move { registry.wait_for_release(Duration::from_secs(60)).await }) + let pools = Arc::clone(&pools); + tokio::spawn( + async move { pools.acquire(Some(1), LanePriority::Background, None).await }, + ) }; tokio::task::yield_now().await; - drop(permit); - assert!(waiter.await.expect("join"), "waiter must wake on release"); + assert!(!pools.has_free_lane(Some(1))); + drop(held); + let handed = waiter + .await + .expect("task") + .expect("lane handed to the waiter"); + assert!(pools.try_acquire(Some(1)).is_none()); + drop(handed); + assert!(pools.try_acquire(Some(1)).is_some()); } - #[tokio::test(start_paused = true)] - async fn one_release_wakes_every_waiter() { - // Waiters of different pools share the registry Notify; a release must - // not be swallowed by a waiter that cannot use the freed slot. - let registry = Arc::new(registry_with(&[ - PoolSpec { - id: 1, - max_concurrency: Some(1), - }, - PoolSpec { - id: 2, - max_concurrency: Some(1), - }, - ])); - let permit = registry.try_acquire(Some(1)).expect("slot"); - let _other = registry.try_acquire(Some(2)).expect("slot"); - let waiters: Vec<_> = (0..3) - .map(|_| { - let registry = Arc::clone(®istry); - tokio::spawn( - async move { registry.wait_for_release(Duration::from_secs(60)).await }, - ) + #[tokio::test] + async fn a_waiter_that_gives_up_after_the_handoff_does_not_leak_the_lane() { + let pools = Arc::new(registry(1)); + let held = pools.try_acquire(Some(1)).expect("free lane"); + let waiter = { + let pools = Arc::clone(&pools); + tokio::spawn(async move { + pools + .acquire(Some(1), LanePriority::Interactive, None) + .await }) - .collect(); + }; tokio::task::yield_now().await; - drop(permit); - for waiter in waiters { - assert!( - waiter.await.expect("join"), - "every waiter must wake on release" - ); - } + drop(held); + waiter.abort(); + let _ = waiter.await; + assert!( + pools.try_acquire(Some(1)).is_some(), + "the handed lane came back" + ); + assert_eq!(pools.occupancy()[0].queued, [0, 0, 0]); } - #[tokio::test(start_paused = true)] - async fn wait_for_release_times_out() { - let registry = registry_with(&[PoolSpec { - id: 1, - max_concurrency: Some(1), - }]); - let _permit = registry.try_acquire(Some(1)).expect("slot"); - assert!(!registry.wait_for_release(Duration::from_millis(50)).await); + #[tokio::test] + async fn a_waiter_dropped_before_the_handoff_is_skipped() { + let pools = Arc::new(registry(1)); + let held = pools.try_acquire(Some(1)).expect("free lane"); + let gone = { + let pools = Arc::clone(&pools); + tokio::spawn(async move { + pools + .acquire(Some(1), LanePriority::Interactive, None) + .await + }) + }; + tokio::task::yield_now().await; + let next = { + let pools = Arc::clone(&pools); + tokio::spawn(async move { pools.acquire(Some(1), LanePriority::UserTool, None).await }) + }; + tokio::task::yield_now().await; + gone.abort(); + let _ = gone.await; + drop(held); + let handed = next.await.expect("task").expect("lane for the next waiter"); + assert_eq!(pools.occupancy()[0].in_flight, 1); + drop(handed); + assert_eq!(pools.occupancy()[0].in_flight, 0); } #[tokio::test(start_paused = true)] - async fn release_before_wait_is_not_lost() { - let registry = registry_with(&[PoolSpec { + async fn a_budget_ends_the_wait_and_leaves_the_queue() { + let pools = registry(1); + let _held = pools.try_acquire(Some(1)).expect("free lane"); + let waited = pools + .acquire( + Some(1), + LanePriority::Interactive, + Some(Duration::from_secs(2)), + ) + .await; + assert!(waited.is_none()); + assert_eq!(pools.occupancy()[0].queued, [0, 0, 0]); + } + + #[tokio::test] + async fn growing_a_pool_serves_waiters_and_shrinking_admits_nothing_new() { + let pools = Arc::new(registry(1)); + let held = pools.try_acquire(Some(1)).expect("free lane"); + let waiter = { + let pools = Arc::clone(&pools); + tokio::spawn(async move { + pools + .acquire(Some(1), LanePriority::Interactive, None) + .await + }) + }; + tokio::task::yield_now().await; + pools.apply(&[PoolSpec { + id: 1, + max_concurrency: Some(2), + }]); + let second = waiter + .await + .expect("task") + .expect("served by the grown pool"); + pools.apply(&[PoolSpec { id: 1, max_concurrency: Some(1), }]); - let permit = registry.try_acquire(Some(1)).expect("slot"); - drop(permit); - // The stored notify_one wakeup covers the scan-then-wait race window. - assert!(registry.wait_for_release(Duration::from_millis(1)).await); + drop(held); + assert!( + pools.try_acquire(Some(1)).is_none(), + "still at the new width" + ); + drop(second); + assert!(pools.try_acquire(Some(1)).is_some()); + } + + #[tokio::test] + async fn queued_counts_follow_priorities() { + let pools = Arc::new(registry(1)); + let _held = pools.try_acquire(Some(1)).expect("free lane"); + let mut waiters = Vec::new(); + for priority in [ + LanePriority::Background, + LanePriority::Background, + LanePriority::UserTool, + ] { + let pools = Arc::clone(&pools); + waiters.push(tokio::spawn(async move { + pools.acquire(Some(1), priority, None).await + })); + tokio::task::yield_now().await; + } + assert_eq!(pools.occupancy()[0].queued, [0, 1, 2]); + for waiter in &waiters { + waiter.abort(); + } + for waiter in waiters { + let _ = waiter.await; + } + assert_eq!(pools.occupancy()[0].queued, [0, 0, 0]); + } + + #[tokio::test] + async fn forecast_counts_waiters_ahead_and_mean_hold_time() { + let pools = Arc::new(registry(2)); + assert_eq!( + pools.forecast_wait(Some(1), LanePriority::Interactive), + Some(Duration::ZERO) + ); + let first = pools.try_acquire(Some(1)).expect("lane"); + let second = pools.try_acquire(Some(1)).expect("lane"); + assert_eq!( + pools.forecast_wait(Some(1), LanePriority::Interactive), + None, + "no hold history yet" + ); + tokio::time::sleep(Duration::from_millis(40)).await; + drop(first); + let first = pools.try_acquire(Some(1)).expect("lane"); + let one_round = pools + .forecast_wait(Some(1), LanePriority::Interactive) + .expect("history"); + assert!(one_round >= Duration::from_millis(30), "{one_round:?}"); + + let mut waiters = Vec::new(); + for _ in 0..2 { + let pools = Arc::clone(&pools); + waiters.push(tokio::spawn(async move { + pools + .acquire(Some(1), LanePriority::Interactive, None) + .await + })); + tokio::task::yield_now().await; + } + let two_rounds = pools + .forecast_wait(Some(1), LanePriority::Interactive) + .expect("history"); + assert_eq!( + two_rounds, + one_round * 2, + "two waiters ahead fill one round" + ); + assert_eq!( + pools.forecast_wait(Some(1), LanePriority::Background), + Some(two_rounds) + ); + for waiter in &waiters { + waiter.abort(); + } + drop((first, second)); + assert_eq!(pools.forecast_wait(None, LanePriority::Interactive), None); + } + + #[test] + fn priorities_parse_from_the_database() { + assert_eq!(LanePriority::from_db(0), LanePriority::Interactive); + assert_eq!(LanePriority::from_db(1), LanePriority::UserTool); + assert_eq!(LanePriority::from_db(2), LanePriority::Background); + assert_eq!(LanePriority::from_db(7), LanePriority::Background); + assert_eq!(LanePriority::from_db(-1), LanePriority::Interactive); + assert!(LanePriority::Interactive < LanePriority::Background); } fn candidate(model: i64) -> Candidate { diff --git a/crates/openplotva-llm/src/router/mod.rs b/crates/openplotva-llm/src/router/mod.rs index b1c137ed..15d3fdb0 100644 --- a/crates/openplotva-llm/src/router/mod.rs +++ b/crates/openplotva-llm/src/router/mod.rs @@ -13,7 +13,9 @@ pub mod table; pub mod triggers; pub use breaker::{BreakerConfig, BreakerLiveness, BreakerSet, BreakerStateView}; -pub use capacity::{PoolOccupancy, PoolPermit, PoolRegistry, PoolSpec, derived_worker_count}; +pub use capacity::{ + LanePriority, PoolOccupancy, PoolPermit, PoolRegistry, PoolSpec, derived_worker_count, +}; pub use handle::RouterHandle; pub use policy::{Attempt, Liveness, RetryBudget, TriggerView, select, select_chain}; pub use table::{ diff --git a/docs/superpowers/plans/2026-10-12-backend-lanes.md b/docs/superpowers/plans/2026-10-12-backend-lanes.md new file mode 100644 index 00000000..59182bfb --- /dev/null +++ b/docs/superpowers/plans/2026-10-12-backend-lanes.md @@ -0,0 +1,1776 @@ +# Backend Lanes Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Load every remote inference backend at its physical width: requests wait in their backend's priority queue, fall back only on failure or budget, and worker counts follow the primary backend's lanes. + +**Architecture:** `PoolRegistry` (openplotva-llm router) becomes a set of lane pools with per-pool priority queues that hand a released lane directly to the next waiter. `RoutedAttemptWalker` admits every candidate through its lane queue with a deadline-derived budget and a hold-time forecast instead of skipping busy candidates. Workflows carry `lane_priority` and `fallback_reserve_ms` from the database; worker counts derive from the primary lane pool; a loader check compares lane widths with Discovery. + +**Tech Stack:** Rust 1.99 (edition 2024), tokio (`oneshot`, `time`), sqlx/Postgres migrations, async-graphql runtime API. + +**Spec:** `docs/superpowers/specs/2026-10-12-backend-lanes-design.md` + +## Global Constraints + +- Lane pool names: `aifarm-gpu0-dialog`, `aifarm-gpu2-utility`, `aifarm-gpu1-media`, `aifarm-embedder`, `aifarm-gpu1-privacy`. +- Widths: GPU0 dialog 8, GPU2 utility 8, embedder 8, GPU1 media 1 (draw, music, speech recognition), GPU1 privacy 4; external APIs stay unpooled. +- Priorities: 0 interactive, 1 user tool, 2 background; lower is served first; FIFO within a priority. +- Fallback reserve: 30,000 ms for interactive and user-tool workflows, 0 for background workflows; the last candidate in a chain waits until the deadline. +- Workers: `PERSISTENT_QUEUE_DIALOG_WORKERS_PER_LANE` default 2, `PERSISTENT_QUEUE_MEMORY_WORKERS_PER_LANE` default 2, memory worker cap default 32. +- Lane config mismatches never fail readiness; they emit `lane_config_mismatch` routing events. +- Migrations are numbered up/down pairs; data changes record previous values for the down migration. +- Follow AGENTS.md: `cargo fmt --all`, `cargo clippy --workspace --all-targets -- -D warnings`, relevant `cargo test`, ready PR, review loop, deploy only on request. + +## Review Focus + +1. A waiter cancelled after a lane was handed to it must not leak the lane (the permit inside the channel releases it). Test in Task 1. +2. Shrinking a pool below its in-flight count admits nothing new until work drains; growing it serves queued waiters at once. Test in Task 1. +3. A request whose deadline already passed is not queued at all and fails with the capacity-timeout error immediately. Test in Task 3. +4. After a restart, background memory waiters must not take GPU2 lanes ahead of a dialog fallback that arrives later. Test in Task 1 (priority order). +5. A candidate with an open circuit breaker is tried only when its lane is free and is never waited for. Test in Task 3. + +--- + +### Task 1: Lane queue in `PoolRegistry` + +**Files:** +- Modify: `crates/openplotva-llm/src/router/capacity.rs` (whole file except `derived_worker_count`) +- Modify: `crates/openplotva-llm/src/router/mod.rs:16` (exports) + +**Interfaces:** +- Consumes: `PoolId`, `RoutingTable` from `router::table`. +- Produces: + - `pub enum LanePriority { Interactive = 0, UserTool = 1, Background = 2 }` with `from_db(i16) -> Self`, `as_str(self) -> &'static str`, `Default` = `Interactive`, `Ord` (lower first). + - `PoolRegistry::acquire(&self, pool: Option, priority: LanePriority, budget: Option) -> Option` (async; `None` when the budget ends). + - `PoolRegistry::try_acquire(&self, pool: Option) -> Option` (non-blocking; fails while waiters exist). + - `PoolRegistry::has_free_lane(&self, pool: Option) -> bool`. + - `PoolRegistry::forecast_wait(&self, pool: Option, priority: LanePriority) -> Option`. + - `PoolOccupancy { id, max_concurrency, in_flight, queued: [u32; 3], mean_hold_ms: Option }`. + - `wait_for_release` is removed. + +- [ ] **Step 1: Write the failing tests** (replace the pool tests in `capacity.rs`'s `mod tests` that call `wait_for_release`; keep the `derived_worker_count` tests untouched) + +```rust + fn registry(max: u32) -> PoolRegistry { + let registry = PoolRegistry::new(); + registry.apply(&[PoolSpec { id: 1, max_concurrency: Some(max) }]); + registry + } + + #[tokio::test] + async fn released_lane_goes_to_the_oldest_waiter_of_the_best_priority() { + let pools = Arc::new(registry(1)); + let held = pools.try_acquire(Some(1)).expect("free lane"); + let order = Arc::new(Mutex::new(Vec::new())); + let mut tasks = Vec::new(); + for (name, priority) in [ + ("background", LanePriority::Background), + ("interactive-1", LanePriority::Interactive), + ("interactive-2", LanePriority::Interactive), + ] { + let pools = Arc::clone(&pools); + let order = Arc::clone(&order); + tasks.push(tokio::spawn(async move { + let permit = pools.acquire(Some(1), priority, None).await.expect("lane"); + order.lock().expect("order").push(name); + drop(permit); + })); + tokio::task::yield_now().await; + } + drop(held); + for task in tasks { + task.await.expect("task"); + } + assert_eq!( + *order.lock().expect("order"), + vec!["interactive-1", "interactive-2", "background"] + ); + } + + #[tokio::test] + async fn try_acquire_does_not_jump_the_queue() { + let pools = Arc::new(registry(1)); + let held = pools.try_acquire(Some(1)).expect("free lane"); + let waiter = { + let pools = Arc::clone(&pools); + tokio::spawn(async move { pools.acquire(Some(1), LanePriority::Background, None).await }) + }; + tokio::task::yield_now().await; + drop(held); + let handed = waiter.await.expect("task").expect("lane handed to the waiter"); + assert!(pools.try_acquire(Some(1)).is_none()); + drop(handed); + assert!(pools.try_acquire(Some(1)).is_some()); + } + + #[tokio::test] + async fn a_waiter_that_gives_up_after_the_handoff_does_not_leak_the_lane() { + let pools = Arc::new(registry(1)); + let held = pools.try_acquire(Some(1)).expect("free lane"); + let waiter = { + let pools = Arc::clone(&pools); + tokio::spawn(async move { pools.acquire(Some(1), LanePriority::Interactive, None).await }) + }; + tokio::task::yield_now().await; + drop(held); + waiter.abort(); + let _ = waiter.await; + tokio::task::yield_now().await; + assert!(pools.try_acquire(Some(1)).is_some(), "the handed lane came back"); + assert_eq!(pools.occupancy()[0].queued, [0, 0, 0]); + } + + #[tokio::test(start_paused = true)] + async fn a_budget_ends_the_wait_and_leaves_the_queue() { + let pools = registry(1); + let _held = pools.try_acquire(Some(1)).expect("free lane"); + let waited = pools + .acquire(Some(1), LanePriority::Interactive, Some(Duration::from_secs(2))) + .await; + assert!(waited.is_none()); + assert_eq!(pools.occupancy()[0].queued, [0, 0, 0]); + } + + #[tokio::test] + async fn growing_a_pool_serves_waiters_and_shrinking_admits_nothing_new() { + let pools = Arc::new(registry(1)); + let held = pools.try_acquire(Some(1)).expect("free lane"); + let waiter = { + let pools = Arc::clone(&pools); + tokio::spawn(async move { pools.acquire(Some(1), LanePriority::Interactive, None).await }) + }; + tokio::task::yield_now().await; + pools.apply(&[PoolSpec { id: 1, max_concurrency: Some(2) }]); + let second = waiter.await.expect("task").expect("served by the grown pool"); + pools.apply(&[PoolSpec { id: 1, max_concurrency: Some(1) }]); + drop(held); + assert!(pools.try_acquire(Some(1)).is_none(), "still at the new width"); + drop(second); + assert!(pools.try_acquire(Some(1)).is_some()); + } + + #[tokio::test] + async fn forecast_counts_waiters_ahead_and_mean_hold_time() { + let pools = Arc::new(registry(2)); + assert_eq!(pools.forecast_wait(Some(1), LanePriority::Interactive), Some(Duration::ZERO)); + let first = pools.try_acquire(Some(1)).expect("lane"); + let second = pools.try_acquire(Some(1)).expect("lane"); + assert_eq!(pools.forecast_wait(Some(1), LanePriority::Interactive), None, "no history yet"); + tokio::time::sleep(Duration::from_millis(20)).await; + drop(first); + let first = pools.try_acquire(Some(1)).expect("lane"); + let forecast = pools.forecast_wait(Some(1), LanePriority::Interactive).expect("history"); + assert!(forecast >= Duration::from_millis(15)); + drop((first, second)); + assert_eq!(pools.forecast_wait(None, LanePriority::Interactive), None); + } + + #[test] + fn priorities_parse_from_the_database() { + assert_eq!(LanePriority::from_db(0), LanePriority::Interactive); + assert_eq!(LanePriority::from_db(1), LanePriority::UserTool); + assert_eq!(LanePriority::from_db(2), LanePriority::Background); + assert_eq!(LanePriority::from_db(7), LanePriority::Background); + assert!(LanePriority::Interactive < LanePriority::Background); + } +``` + +- [ ] **Step 2: Run the tests to verify they fail** + +Run: `cargo test -p openplotva-llm --lib router::capacity` +Expected: compile errors (`LanePriority`, `acquire`, `forecast_wait`, `queued` not defined). + +- [ ] **Step 3: Implement the lane queue** (replace everything above `derived_worker_count` in `capacity.rs`) + +```rust +//! Lane pools: the width of one physical inference resource and its queue. +//! +//! A pool admits up to `max_concurrency` requests. Further requests wait in a +//! priority queue (lower priority value first, arrival order within a +//! priority), and a released lane is handed directly to the next waiter as a +//! [`PoolPermit`] sent through a oneshot channel. A waiter that gives up drops +//! its receiver; a permit already inside the channel then releases the lane on +//! drop, so no lane can leak. The registry lives outside the +//! [`super::table::RoutingTable`] so in-flight accounting survives a reload: +//! [`PoolRegistry::apply`] resizes cells in place. + +use std::collections::{BTreeMap, HashMap}; +use std::sync::{Arc, Mutex, MutexGuard}; +use std::time::{Duration, Instant}; + +use tokio::sync::oneshot; + +use super::table::{PoolId, RoutingTable}; + +/// Order in which waiters of one lane pool are served: lower first. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, PartialOrd, Ord, Hash)] +pub enum LanePriority { + #[default] + Interactive = 0, + UserTool = 1, + Background = 2, +} + +impl LanePriority { + #[must_use] + pub fn from_db(value: i16) -> Self { + match value { + i16::MIN..=0 => Self::Interactive, + 1 => Self::UserTool, + _ => Self::Background, + } + } + + #[must_use] + pub fn as_str(self) -> &'static str { + match self { + Self::Interactive => "interactive", + Self::UserTool => "user_tool", + Self::Background => "background", + } + } +} + +/// Desired state of one pool, resolved from the routing table. +/// `max_concurrency == None` means unlimited: gauge-only, never blocks. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct PoolSpec { + pub id: PoolId, + pub max_concurrency: Option, +} + +/// Live state of one pool, for the admin status endpoint and the runtime API. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct PoolOccupancy { + pub id: PoolId, + pub max_concurrency: Option, + pub in_flight: u32, + /// Waiters per priority: interactive, user tool, background. + pub queued: [u32; 3], + pub mean_hold_ms: Option, +} + +/// Weight of the newest sample in the mean lane hold time. +const HOLD_MEAN_WEIGHT: f64 = 0.2; + +#[derive(Debug)] +struct CellState { + max: Option, + in_flight: u32, + waiters: BTreeMap<(LanePriority, u64), oneshot::Sender>, + next_seq: u64, + mean_hold_ms: Option, +} + +impl CellState { + fn has_free_lane(&self) -> bool { + self.max.is_none_or(|max| self.in_flight < max) + } + + fn record_hold(&mut self, held: Duration) { + let sample = held.as_secs_f64() * 1000.0; + self.mean_hold_ms = Some(match self.mean_hold_ms { + Some(mean) => mean + HOLD_MEAN_WEIGHT * (sample - mean), + None => sample, + }); + } +} + +#[derive(Debug)] +struct PoolCell { + state: Mutex, +} + +impl PoolCell { + fn lock(&self) -> MutexGuard<'_, CellState> { + self.state + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner) + } +} + +/// RAII lane of a pool. Dropping it frees the lane for the next waiter, so +/// cancellation of an in-flight attempt can never leak a lane. A permit for an +/// unpooled model is a no-op. +#[derive(Debug, Default)] +pub struct PoolPermit { + cell: Option>, + acquired_at: Option, +} + +impl PoolPermit { + fn unpooled() -> Self { + Self::default() + } + + fn held(cell: Arc) -> Self { + Self { + cell: Some(cell), + acquired_at: Some(Instant::now()), + } + } +} + +impl Drop for PoolPermit { + fn drop(&mut self) { + if let Some(cell) = self.cell.take() { + let mut state = cell.lock(); + state.in_flight = state.in_flight.saturating_sub(1); + if let Some(acquired_at) = self.acquired_at { + state.record_hold(acquired_at.elapsed()); + } + serve_waiters(&cell, &mut state); + } + } +} + +/// Hand free lanes to waiters in queue order. A waiter that already gave up +/// returns the permit; it is disarmed and the lane goes to the next waiter. +fn serve_waiters(cell: &Arc, state: &mut CellState) { + while state.has_free_lane() { + let Some((_, waiter)) = state.waiters.pop_first() else { + return; + }; + state.in_flight = state.in_flight.saturating_add(1); + if let Err(mut unclaimed) = waiter.send(PoolPermit::held(Arc::clone(cell))) { + unclaimed.cell = None; + state.in_flight = state.in_flight.saturating_sub(1); + } + } +} + +/// Registry of live pool cells, keyed by pool id. +#[derive(Debug, Default)] +pub struct PoolRegistry { + cells: Mutex>>, +} + +impl PoolRegistry { + #[must_use] + pub fn new() -> Self { + Self::default() + } + + fn lock(&self) -> MutexGuard<'_, HashMap>> { + self.cells + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner) + } + + fn cell(&self, pool: Option) -> Option> { + pool.and_then(|pool| self.lock().get(&pool).cloned()) + } + + /// Reconcile the registry with a freshly loaded routing table: insert new + /// pools, update `max` in place (in-flight accounting preserved), drop cells + /// of removed pools (outstanding permits keep their cell alive). A grown + /// pool serves its waiters at once. + pub fn apply(&self, specs: &[PoolSpec]) { + let mut cells = self.lock(); + cells.retain(|id, _| specs.iter().any(|spec| spec.id == *id)); + for spec in specs { + match cells.get(&spec.id) { + Some(cell) => { + let mut state = cell.lock(); + state.max = spec.max_concurrency; + serve_waiters(cell, &mut state); + } + None => { + cells.insert( + spec.id, + Arc::new(PoolCell { + state: Mutex::new(CellState { + max: spec.max_concurrency, + in_flight: 0, + waiters: BTreeMap::new(), + next_seq: 0, + mean_hold_ms: None, + }), + }), + ); + } + } + } + } + + /// Take a free lane without waiting. Fails while the pool is full or while + /// others wait, so it never jumps the queue. Unpooled models and pools + /// unknown to the registry always succeed with a no-op permit. + #[must_use] + pub fn try_acquire(&self, pool: Option) -> Option { + let Some(cell) = self.cell(pool) else { + return Some(PoolPermit::unpooled()); + }; + let mut state = cell.lock(); + if !state.waiters.is_empty() || !state.has_free_lane() { + return None; + } + state.in_flight = state.in_flight.saturating_add(1); + drop(state); + Some(PoolPermit::held(cell)) + } + + /// Whether `try_acquire` would succeed right now. + #[must_use] + pub fn has_free_lane(&self, pool: Option) -> bool { + self.cell(pool).is_none_or(|cell| { + let state = cell.lock(); + state.waiters.is_empty() && state.has_free_lane() + }) + } + + /// Wait in the pool's queue for a lane. `None` once `budget` ends; the + /// waiter then leaves the queue. `budget == None` waits without a limit. + pub async fn acquire( + &self, + pool: Option, + priority: LanePriority, + budget: Option, + ) -> Option { + let Some(cell) = self.cell(pool) else { + return Some(PoolPermit::unpooled()); + }; + let (key, receiver) = { + let mut state = cell.lock(); + if state.waiters.is_empty() && state.has_free_lane() { + state.in_flight = state.in_flight.saturating_add(1); + drop(state); + return Some(PoolPermit::held(cell)); + } + let (sender, receiver) = oneshot::channel(); + let key = (priority, state.next_seq); + state.next_seq = state.next_seq.wrapping_add(1); + state.waiters.insert(key, sender); + (key, receiver) + }; + let handed = match budget { + Some(budget) => match tokio::time::timeout(budget, receiver).await { + Ok(handed) => handed.ok(), + Err(_) => None, + }, + None => receiver.await.ok(), + }; + if handed.is_none() { + cell.lock().waiters.remove(&key); + } + handed + } + + /// Expected wait for a new request of `priority`: the waiters it would + /// queue behind, in rounds of the pool width, times the mean lane hold + /// time. `Some(ZERO)` when a lane is free; `None` for unpooled or unlimited + /// pools and before the first lane was released. + #[must_use] + pub fn forecast_wait(&self, pool: Option, priority: LanePriority) -> Option { + let cell = self.cell(pool)?; + let mut state = cell.lock(); + let width = state.max?; + state.waiters.retain(|_, waiter| !waiter.is_closed()); + if state.waiters.is_empty() && state.has_free_lane() { + return Some(Duration::ZERO); + } + let mean_hold_ms = state.mean_hold_ms?; + let ahead = state + .waiters + .keys() + .filter(|(waiting, _)| *waiting <= priority) + .count() as u64; + let rounds = (ahead + 1).div_ceil(u64::from(width.max(1))); + Some(Duration::from_secs_f64(mean_hold_ms * rounds as f64 / 1000.0)) + } + + /// Current state of every registered pool, sorted by id. + #[must_use] + pub fn occupancy(&self) -> Vec { + let cells = self.lock(); + let mut out: Vec = cells + .iter() + .map(|(id, cell)| { + let mut state = cell.lock(); + state.waiters.retain(|_, waiter| !waiter.is_closed()); + let mut queued = [0_u32; 3]; + for (priority, _) in state.waiters.keys() { + queued[*priority as usize] = queued[*priority as usize].saturating_add(1); + } + PoolOccupancy { + id: *id, + max_concurrency: state.max, + in_flight: state.in_flight, + queued, + mean_hold_ms: state.mean_hold_ms.map(|mean| mean.round() as u64), + } + }) + .collect(); + drop(cells); + out.sort_by_key(|occupancy| occupancy.id); + out + } +} +``` + +Update the export in `crates/openplotva-llm/src/router/mod.rs:16`: + +```rust +pub use capacity::{ + LanePriority, PoolOccupancy, PoolPermit, PoolRegistry, PoolSpec, derived_worker_count, +}; +``` + +Add `"sync"` is already in the crate's tokio features (`crates/openplotva-llm/Cargo.toml:27`), so `oneshot` is available. + +- [ ] **Step 4: Run the tests to verify they pass** + +Run: `cargo test -p openplotva-llm --lib router::capacity` +Expected: PASS. The crate's other code that called `wait_for_release` does not exist inside `openplotva-llm`; the app crate is fixed in Task 3. + +- [ ] **Step 5: Commit** + +```bash +git add crates/openplotva-llm/src/router/capacity.rs crates/openplotva-llm/src/router/mod.rs +git commit -m "Give lane pools a priority queue that hands released lanes to the next waiter" +``` + +--- + +### Task 2: Workflow lane settings in the database and the routing table + +**Files:** +- Create: `migrations/199_workflow_lane_settings.up.sql`, `migrations/199_workflow_lane_settings.down.sql` +- Modify: `crates/openplotva-storage/src/llm_routing.rs` (`WorkflowRecord`, `SQL_LIST_WORKFLOWS`, `workflow_from_row`, two test literals) +- Modify: `crates/openplotva-llm/src/router/table.rs` (`WorkflowRoute`) +- Modify: `crates/openplotva-app/src/model_routing.rs` (`build_route`, memory workflow seeding inserts, 7 `WorkflowRecord` test literals) +- Modify: `crates/openplotva-app/src/routed_attempts.rs` (`RoutedRequestContext`, 2 `WorkflowRecord` test literals) +- Modify: `WorkflowRecord`/`WorkflowRoute` literals in `crates/openplotva-app/src/dialog_runtime.rs`, `crates/openplotva-app/src/image_jobs.rs`, `crates/openplotva-llm/src/router/{capacity.rs,mod.rs,table.rs}` +- Test: `crates/openplotva-storage/src/lib.rs` (migration contract test), `crates/openplotva-app/src/model_routing.rs` (route mapping test) + +**Interfaces:** +- Consumes: `LanePriority` (Task 1). +- Produces: `WorkflowRecord { .., lane_priority: i16, fallback_reserve_ms: i32 }`; `WorkflowRoute { .., lane_priority: LanePriority, fallback_reserve: Duration }`; `RoutedRequestContext { .., lane_priority: Option }`. + +- [ ] **Step 1: Write the migration contract test** in `crates/openplotva-storage/src/lib.rs` next to `vision_fallback_joins_the_gpu2_pool_and_is_reversible` + +```rust + #[test] + fn workflow_lane_settings_default_and_are_reversible() { + const UP: &str = include_str!("../../../migrations/199_workflow_lane_settings.up.sql"); + const DOWN: &str = include_str!("../../../migrations/199_workflow_lane_settings.down.sql"); + + for value in [ + "ADD COLUMN lane_priority SMALLINT NOT NULL DEFAULT 0", + "ADD COLUMN fallback_reserve_ms INTEGER NOT NULL DEFAULT 30000", + "SET lane_priority = 1", + "'history_summary', 'youtube_summary', 'media_prompt_optimizer'", + "'image_generation', 'image_generation_flux', 'image_generation_boogu_turbo'", + "'image_edit', 'image_edit_flux', 'image_edit_boogu_turbo', 'music'", + "SET lane_priority = 2, fallback_reserve_ms = 0", + "WHERE key IN ('memory_extraction', 'memory_subject_merge')", + ] { + assert!(UP.contains(value), "missing lane settings contract {value}"); + } + assert!(DOWN.contains("DROP COLUMN lane_priority")); + assert!(DOWN.contains("DROP COLUMN fallback_reserve_ms")); + } +``` + +- [ ] **Step 2: Run it to verify it fails** + +Run: `cargo test -p openplotva-storage --lib workflow_lane_settings` +Expected: FAIL (`include_str!` cannot find the files). + +- [ ] **Step 3: Write the migrations** + +`migrations/199_workflow_lane_settings.up.sql`: + +```sql +-- Each workflow's place in its lane queues and the time it keeps for a fallback. +-- Priority 0 is interactive, 1 a user-facing tool, 2 background work; lower is +-- served first. Background work waits for its lane without falling back. + +ALTER TABLE workflows + ADD COLUMN lane_priority SMALLINT NOT NULL DEFAULT 0, + ADD COLUMN fallback_reserve_ms INTEGER NOT NULL DEFAULT 30000; + +UPDATE workflows SET lane_priority = 1 +WHERE key IN ('history_summary', 'youtube_summary', 'media_prompt_optimizer', + 'image_generation', 'image_generation_flux', 'image_generation_boogu_turbo', + 'image_edit', 'image_edit_flux', 'image_edit_boogu_turbo', 'music'); + +UPDATE workflows SET lane_priority = 2, fallback_reserve_ms = 0 +WHERE key IN ('memory_extraction', 'memory_subject_merge'); +``` + +`migrations/199_workflow_lane_settings.down.sql`: + +```sql +ALTER TABLE workflows + DROP COLUMN lane_priority, + DROP COLUMN fallback_reserve_ms; +``` + +- [ ] **Step 4: Run the contract test to verify it passes** + +Run: `cargo test -p openplotva-storage --lib workflow_lane_settings` +Expected: PASS. + +- [ ] **Step 5: Carry the columns through storage** + +In `crates/openplotva-storage/src/llm_routing.rs`: + +```rust +#[derive(Clone, Debug, PartialEq)] +pub struct WorkflowRecord { + pub key: String, + pub kind: String, + pub full_routing: bool, + pub retry_max_hops: i32, + pub retry_wall_ms: i32, + pub enabled: bool, + pub lane_priority: i16, + pub fallback_reserve_ms: i32, +} +``` + +```rust +const SQL_LIST_WORKFLOWS: &str = "SELECT key, kind, full_routing, retry_max_hops, retry_wall_ms, enabled, lane_priority, fallback_reserve_ms FROM workflows ORDER BY key ASC"; +``` + +```rust +fn workflow_from_row(row: PgRow) -> Result { + Ok(WorkflowRecord { + key: row.try_get("key")?, + kind: row.try_get("kind")?, + full_routing: row.try_get("full_routing")?, + retry_max_hops: row.try_get("retry_max_hops")?, + retry_wall_ms: row.try_get("retry_wall_ms")?, + enabled: row.try_get("enabled")?, + lane_priority: row.try_get("lane_priority")?, + fallback_reserve_ms: row.try_get("fallback_reserve_ms")?, + }) +} +``` + +Every other `WorkflowRecord { .. }` literal in the workspace gets `lane_priority: 0, fallback_reserve_ms: 30_000,` (13 literals; find them with `rg -n "WorkflowRecord \{" crates`). + +- [ ] **Step 6: Carry the settings into `WorkflowRoute` and the request context** + +In `crates/openplotva-llm/src/router/table.rs`, add to `WorkflowRoute` (after `retry`): + +```rust + /// Place of this workflow's requests in their lane queues. + pub lane_priority: super::capacity::LanePriority, + /// Time kept for the next candidate when a lane wait has a deadline. + pub fallback_reserve: std::time::Duration, +``` + +Every `WorkflowRoute { .. }` literal (5; `rg -n "WorkflowRoute \{" crates`) gets +`lane_priority: LanePriority::Interactive, fallback_reserve: Duration::from_secs(30),` with the imports the file needs. + +In `crates/openplotva-app/src/model_routing.rs` `build_route`, extend the returned route: + +```rust + retry: RetryBudget { + max_hops: workflow.retry_max_hops.max(1) as u32, + wall_clock: Duration::from_millis(workflow.retry_wall_ms.max(0) as u64), + }, + lane_priority: openplotva_llm::router::LanePriority::from_db(workflow.lane_priority), + fallback_reserve: Duration::from_millis(workflow.fallback_reserve_ms.max(0) as u64), +``` + +In the memory cascade seeding (`model_routing.rs`, the two `INSERT INTO workflows` statements for memory workflows), insert the background settings so fresh installs match the migration: + +```rust + "INSERT INTO workflows (key, kind, full_routing, retry_max_hops, retry_wall_ms, enabled, lane_priority, fallback_reserve_ms) \ + VALUES ($1, 'chat', FALSE, 2, 900000, TRUE, 2, 0) \ + ON CONFLICT (key) DO UPDATE SET kind = EXCLUDED.kind, \ + full_routing = EXCLUDED.full_routing, retry_max_hops = EXCLUDED.retry_max_hops, \ + retry_wall_ms = EXCLUDED.retry_wall_ms, enabled = EXCLUDED.enabled, \ + lane_priority = EXCLUDED.lane_priority, fallback_reserve_ms = EXCLUDED.fallback_reserve_ms", +``` + +(the extraction statement keeps its own `retry_max_hops`/`retry_wall_ms` values and gains the same two columns with `2, 0`). + +In `crates/openplotva-app/src/routed_attempts.rs` `RoutedRequestContext`, add: + +```rust + /// Lane queue priority; `None` takes the workflow's `lane_priority`. + pub lane_priority: Option, +``` + +and import `LanePriority` from `openplotva_llm::router`. + +- [ ] **Step 7: Write the route mapping test** in `crates/openplotva-app/src/model_routing.rs` tests + +```rust + #[test] + fn workflow_lane_settings_reach_the_route() { + let mut snapshot = minimal_dialog_snapshot(); + snapshot.workflows[0].lane_priority = 2; + snapshot.workflows[0].fallback_reserve_ms = 0; + let table = build_routing_table(&snapshot); + let route = table.resolve("dialog", false).expect("route"); + assert_eq!(route.lane_priority, openplotva_llm::router::LanePriority::Background); + assert_eq!(route.fallback_reserve, Duration::ZERO); + } +``` + +If `minimal_dialog_snapshot` does not exist in that test module, build the snapshot inline from the module's existing helper that the route tests use (search the module for `fn snapshot`/`RoutingSnapshot {`), with one provider, one model, the `dialog` workflow and one primary assignment. + +- [ ] **Step 8: Run the tests** + +Run: `cargo test -p openplotva-storage --lib && cargo test -p openplotva-llm --lib && cargo test -p openplotva-app --lib model_routing` +Expected: PASS. + +- [ ] **Step 9: Commit** + +```bash +git add migrations/199_workflow_lane_settings.up.sql migrations/199_workflow_lane_settings.down.sql crates +git commit -m "Store each workflow's lane priority and fallback reserve" +``` + +--- + +### Task 3: Walker admits candidates through their lane queues + +**Files:** +- Modify: `crates/openplotva-app/src/routed_attempts.rs` (`RoutedAttemptWalker`, `run`, tests) + +**Interfaces:** +- Consumes: `PoolRegistry::{try_acquire, has_free_lane, acquire, forecast_wait}`, `WorkflowRoute::{lane_priority, fallback_reserve}`, `RoutedRequestContext::lane_priority`. +- Produces: routing events `lane_wait` (info: `waited_ms`, `acquired`, `priority`, `budget_ms`) and `lane_bypass` (info: `forecast_ms`, `budget_ms`, `priority`); `capacity_wait_timeout` stays the event and error when no candidate got a lane. Removes `with_primary_slot_wait`, `with_max_slot_wait`, `wait_for_primary_slot`, `DEFAULT_MAX_SLOT_WAIT`. + +- [ ] **Step 1: Write the failing walker tests** (in `routed_attempts.rs` tests; they reuse `pooled_snapshot(max_hops)`: model 10 in pool 1 width 1 as the first primary, model 20 as the second candidate) + +```rust + fn fallback_snapshot() -> RoutingSnapshot { + let mut snapshot = pooled_snapshot(3); + snapshot.assignments[0].weight = Some(100); + snapshot.assignments[1].role = "fallback".to_owned(); + snapshot.assignments[1].weight = None; + snapshot.assignments[1].fallback_order = Some(1); + snapshot.models[1].pool_id = None; + snapshot + } + + async fn run_recording( + walker: &RoutedAttemptWalker, + deadline: Option, + ) -> (Result<&'static str, RoutedAttemptRunError>, Vec) { + let executed = Arc::new(std::sync::Mutex::new(Vec::new())); + let models = Arc::clone(&executed); + let result = walker + .run( + RoutedRequestContext { + workflow_key: "dialog".to_owned(), + deadline, + ..RoutedRequestContext::default() + }, + move |attempt| { + let models = Arc::clone(&models); + async move { + models.lock().expect("models").push(attempt.model_id); + Ok::<_, std::io::Error>("ok") + } + }, + |_error| None, + ) + .await; + let executed = executed.lock().expect("models").clone(); + (result, executed) + } + + #[tokio::test] + async fn a_busy_primary_is_waited_for_instead_of_falling_back() { + let snapshot = fallback_snapshot(); + let pools = Arc::new(PoolRegistry::new()); + let walker = walker_with_pools(&snapshot, Arc::clone(&pools)); + let held = pools.try_acquire(Some(1)).expect("occupy the primary lane"); + let release = tokio::spawn(async move { + tokio::time::sleep(Duration::from_millis(50)).await; + drop(held); + }); + let deadline = std::time::Instant::now() + Duration::from_secs(120); + + let (result, executed) = run_recording(&walker, Some(deadline)).await; + + release.await.expect("release"); + assert_eq!(result.expect("primary output"), "ok"); + assert_eq!(executed, vec![10]); + } + + #[tokio::test] + async fn a_forecast_beyond_the_budget_falls_back_without_waiting() { + let snapshot = fallback_snapshot(); + let pools = Arc::new(PoolRegistry::new()); + let walker = walker_with_pools(&snapshot, Arc::clone(&pools)); + let warmup = pools.try_acquire(Some(1)).expect("lane"); + tokio::time::sleep(Duration::from_millis(300)).await; + drop(warmup); + let _held = pools.try_acquire(Some(1)).expect("occupy the primary lane"); + // 30 s reserve leaves 100 ms of budget, below the 300 ms mean hold. + let deadline = std::time::Instant::now() + Duration::from_millis(30_100); + + let (result, executed) = run_recording(&walker, Some(deadline)).await; + + assert_eq!(result.expect("fallback output"), "ok"); + assert_eq!(executed, vec![20]); + } + + #[tokio::test] + async fn an_expired_deadline_is_not_queued() { + let mut snapshot = pooled_snapshot(3); + snapshot.assignments.truncate(1); + let pools = Arc::new(PoolRegistry::new()); + let walker = walker_with_pools(&snapshot, Arc::clone(&pools)); + let _held = pools.try_acquire(Some(1)).expect("occupy the only lane"); + let started = std::time::Instant::now(); + + let (result, executed) = run_recording(&walker, Some(started)).await; + + assert!(executed.is_empty()); + assert!(started.elapsed() < Duration::from_millis(500)); + assert!(matches!( + result, + Err(RoutedAttemptRunError::Routing(error)) + if error.kind() == RoutedRoutingErrorKind::CapacityWaitTimeout + || error.kind() == RoutedRoutingErrorKind::DeadlineExceeded + )); + } + + #[tokio::test] + async fn a_candidate_with_an_open_breaker_is_never_waited_for() { + let snapshot = fallback_snapshot(); + let pools = Arc::new(PoolRegistry::new()); + let breakers = Arc::new(BreakerSet::new()); + let table = crate::model_routing::build_routing_table(&snapshot); + pools.apply(&table.pool_specs()); + let breaker = table + .resolve("dialog", false) + .expect("route") + .primary[0] + .breaker; + for _ in 0..breaker.failure_threshold.max(1) { + breakers.record_failure(1, 10, breaker); + } + let walker = RoutedAttemptWalker::new( + RouterHandle::new(table), + Arc::clone(&breakers), + Arc::new(TriggerState::new()), + Arc::clone(&pools), + ); + let _held = pools.try_acquire(Some(1)).expect("occupy the primary lane"); + + let (result, executed) = run_recording(&walker, None).await; + + assert_eq!(result.expect("fallback output"), "ok"); + assert_eq!(executed, vec![20]); + } + + #[tokio::test] + async fn a_free_primary_sibling_is_taken_before_waiting() { + let mut snapshot = pooled_snapshot(3); + snapshot.assignments[0].weight = Some(100); + snapshot.assignments[1].weight = Some(1); + snapshot.models[1].pool_id = None; + let pools = Arc::new(PoolRegistry::new()); + let walker = walker_with_pools(&snapshot, Arc::clone(&pools)); + let _held = pools.try_acquire(Some(1)).expect("occupy the heavy primary"); + + let (result, executed) = run_recording(&walker, None).await; + + assert_eq!(result.expect("sibling output"), "ok"); + assert_eq!(executed, vec![20]); + } +``` + +Use the field name of the breaker threshold that `BreakerConfig` actually has (`rg -n "pub struct BreakerConfig" -A8 crates/openplotva-llm/src/router/breaker.rs`). Delete the old tests that encode skipping a busy candidate: `busy_pool_skips_to_free_candidate_without_breaker_mutation`, `busy_skips_do_not_consume_the_hop_budget`, and any test that relies on `with_primary_slot_wait`, `with_max_slot_wait` or a 1 s capacity timeout; keep `failed_target_is_not_reselected_while_fallback_pool_is_busy` after adapting it to queue waits (it must still never re-select the failed target). + +- [ ] **Step 2: Run them to verify they fail** + +Run: `cargo test -p openplotva-app --lib routed_attempts` +Expected: FAIL (the busy primary is skipped, the forecast bypass does not exist). + +- [ ] **Step 3: Replace admission in the walker** + +Remove from `RoutedAttemptWalker`: the fields `max_slot_wait` and `primary_slot_wait`, their builder methods, `wait_for_primary_slot`, and `DEFAULT_MAX_SLOT_WAIT`. Add this method: + +```rust + /// Admit one candidate through its lane queue. A free lane is taken at once. + /// A candidate with an open breaker is never waited for. Otherwise the + /// request waits in the lane queue until `deadline - reserve`, where the + /// reserve is kept only when another candidate follows; it skips the wait + /// when the forecast already exceeds that budget. + async fn admit_lane( + &self, + pool: Option, + live: bool, + has_next_candidate: bool, + route: &WorkflowRoute, + context: &RoutedRequestContext, + attempt: &Attempt, + total_slot_wait: &mut Duration, + ) -> Option { + if let Some(permit) = self.pools.try_acquire(pool) { + return Some(permit); + } + if !live { + return None; + } + let priority = context.lane_priority.unwrap_or(route.lane_priority); + let reserve = if has_next_candidate { + route.fallback_reserve + } else { + Duration::ZERO + }; + let budget = context.deadline.map(|deadline| { + deadline + .saturating_duration_since(Instant::now()) + .saturating_sub(reserve) + }); + if let Some(budget) = budget { + let forecast = self.pools.forecast_wait(pool, priority); + if budget.is_zero() || forecast.is_some_and(|forecast| forecast > budget) { + self.record_event(routing_event_with_severity( + "lane_bypass", + "info", + context, + Some(attempt.provider), + Some(attempt.model), + "lane wait would exceed the request budget; trying the next candidate", + json!({ + "priority": priority.as_str(), + "budget_ms": budget.as_millis(), + "forecast_ms": forecast.map(|forecast| forecast.as_millis()), + }), + )); + return None; + } + } + let started = Instant::now(); + let permit = self.pools.acquire(pool, priority, budget).await; + let waited = started.elapsed(); + *total_slot_wait += waited; + self.record_event(routing_event_with_severity( + "lane_wait", + "info", + context, + Some(attempt.provider), + Some(attempt.model), + if permit.is_some() { + "lane acquired after waiting in its queue" + } else { + "lane wait reached the request budget" + }, + json!({ + "priority": priority.as_str(), + "waited_ms": waited.as_millis(), + "budget_ms": budget.map(|budget| budget.as_millis()), + "acquired": permit.is_some(), + }), + )); + permit + } +``` + +In `run`: + +1. After `let mut attempts = { ... }` and the `preferred_target` reordering, move candidates whose lane is free ahead of busy candidates of the same role, keeping the order otherwise: + +```rust + attempts.sort_by_key(|attempt| { + let pool = table.model(attempt.model).and_then(|row| row.pool_id); + (attempt.role != Role::Primary, !self.pools.has_free_lane(pool)) + }); +``` + + Apply this sort before the `preferred_target` block so a preferred target still goes first. +2. Iterate by index so the admission knows whether another candidate follows: + +```rust + let candidates = attempts; + for (index, attempt) in candidates.iter().enumerate() { + let attempt = attempt.clone(); + let has_next_candidate = candidates[index + 1..].iter().any(|next| { + !attempted_targets.contains(&(next.provider, next.model)) + }); + let live = self.breakers.is_live_at(attempt.provider, attempt.model, Instant::now()); +``` + +3. Replace the permit block (`let mut permit = self.pools.try_acquire(pool_id); if permit.is_none() && attempt.role == Role::Primary ...`) with: + +```rust + let Some(permit) = self + .admit_lane( + pool_id, + live, + has_next_candidate, + route, + &context, + &attempt, + &mut total_slot_wait, + ) + .await + else { + if first_execution { + busy_skips += 1; + } + break; + }; +``` + +4. Replace the block after the candidate loop (from `if busy_skips == 0 || started_attempts >= max_hops` to the end of the `'passes` loop body) with: + +```rust + if executed_this_pass == 0 && busy_skips > 0 && last_error.is_none() { + self.record_event(routing_event( + "capacity_wait_timeout", + &context, + None, + None, + "workflow capacity unavailable: no candidate lane within the request budget", + json!({ + "busy_candidates": busy_skips, + "waited_ms": total_slot_wait.as_millis(), + }), + )); + return Err(RoutedAttemptRunError::Routing(RoutedRoutingError::new( + RoutedRoutingErrorKind::CapacityWaitTimeout, + format!( + "workflow capacity unavailable: no candidate lane after {}ms", + total_slot_wait.as_millis() + ), + ))); + } + break; +``` + + Remove the now-unused `wait_reported` variable. `quota_skips` stays (OpenRouter free gate). + +- [ ] **Step 4: Run the walker tests** + +Run: `cargo test -p openplotva-app --lib routed_attempts` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add crates/openplotva-app/src/routed_attempts.rs +git commit -m "Admit routed attempts through lane queues: fall back on failure or budget" +``` + +--- + +### Task 4: Remove the per-site wait knobs + +**Files:** +- Modify: `crates/openplotva-app/src/dialog_runtime.rs` (`INTERACTIVE_CAPACITY_WAIT_TIMEOUT`, `RouterChatProvider::new`, `with_primary_slot_wait`, the router construction near line 516, the test near line 1040) +- Modify: `crates/openplotva-app/src/lib.rs` (three `.with_primary_slot_wait(..)` calls) +- Modify: `crates/openplotva-app/src/history_summary.rs`, `crates/openplotva-app/src/youtube.rs`, `crates/openplotva-app/src/media.rs` (slot-wait constants) +- Modify: `crates/openplotva-config/src/lib.rs` (`DEFAULT_DIALOG_PRIMARY_CAPACITY_WAIT_MS`, `aifarm_pool_primary_capacity_wait_ms`, its raw field and env read) + +**Interfaces:** +- Consumes: walker without wait knobs (Task 3). +- Produces: nothing new; `DIALOG_AIFARM_POOL_PRIMARY_CAPACITY_WAIT_MS` is no longer read. + +- [ ] **Step 1: Rewrite the dialog capacity-timeout test** + +The test in `dialog_runtime.rs` that asserts `started.elapsed() == INTERACTIVE_CAPACITY_WAIT_TIMEOUT` becomes: with every dialog lane held and a request deadline 2 s ahead, the provider returns the capacity-timeout error after about 2 s and records `capacity_wait_timeout`: + +```rust + assert!(started.elapsed() >= Duration::from_secs(2)); + assert!( + events + .iter() + .any(|event| event.event_type == "capacity_wait_timeout") + ); +``` + +(keep the rest of the test's setup; replace `INTERACTIVE_CAPACITY_WAIT_TIMEOUT` with the 2 s deadline the test passes in its context; use `#[tokio::test(start_paused = true)]` if the test is not paused already). + +- [ ] **Step 2: Remove the knobs and constants** + +Delete `INTERACTIVE_CAPACITY_WAIT_TIMEOUT` and the `.with_max_slot_wait(..)` call in `RouterChatProvider::new`, the `with_primary_slot_wait` method, and the `primary_slot_wait` read and call near line 516. Delete `.with_primary_slot_wait(..)` in `lib.rs` (history summary, YouTube, media optimizer) and the constants `HISTORY_SUMMARY_PRIMARY_SLOT_WAIT`, `YOUTUBE_PRIMARY_SLOT_WAIT`, `MEDIA_OPTIMIZER_PRIMARY_SLOT_WAIT`. In `openplotva-config`, delete `DEFAULT_DIALOG_PRIMARY_CAPACITY_WAIT_MS`, the `aifarm_pool_primary_capacity_wait_ms` field, the `dialog_aifarm_pool_primary_capacity_wait_ms` raw field, its parse and env read, and the config tests that set them. + +- [ ] **Step 3: Build and test** + +Run: `cargo test -p openplotva-config && cargo test -p openplotva-app --lib dialog_runtime` +Expected: PASS; `cargo build -p openplotva-app` has no unused-import warnings. + +- [ ] **Step 4: Commit** + +```bash +git add crates +git commit -m "Drop per-call slot waits now that lanes queue requests" +``` + +--- + +### Task 5: Callers pass background priority for memory work + +**Files:** +- Modify: `crates/openplotva-app/src/embedder.rs` (`RoutedDiscoveryEmbedder`, `embedding_context`) +- Modify: `crates/openplotva-app/src/memory_runtime.rs` (`RoutedDiscoveryRedactor::redact_text` context) +- Modify: `crates/openplotva-app/src/lib.rs` (memory write embedder construction) + +**Interfaces:** +- Consumes: `RoutedRequestContext::lane_priority`, `LanePriority`. +- Produces: `RoutedDiscoveryEmbedder::with_lane_priority(self, LanePriority) -> Self`. + +- [ ] **Step 1: Write the failing test** in `embedder.rs` tests + +```rust + #[test] + fn memory_writes_embed_at_background_priority() { + assert_eq!(embedding_context(None).lane_priority, None); + assert_eq!( + embedding_context(Some(LanePriority::Background)).lane_priority, + Some(LanePriority::Background) + ); + } +``` + +- [ ] **Step 2: Run it to verify it fails** + +Run: `cargo test -p openplotva-app --lib memory_writes_embed_at_background_priority` +Expected: FAIL (`embedding_context` takes no argument). + +- [ ] **Step 3: Implement** + +```rust +pub struct RoutedDiscoveryEmbedder { + walker: RoutedAttemptWalker, + base_config: DiscoveryEmbedderConfig, + lane_priority: Option, +} +``` + +`new` sets `lane_priority: None`; add: + +```rust + /// Queue this embedder's requests at `priority` instead of the workflow default. + #[must_use] + pub fn with_lane_priority(mut self, priority: LanePriority) -> Self { + self.lane_priority = Some(priority); + self + } +``` + +```rust +fn embedding_context(lane_priority: Option) -> RoutedRequestContext { + RoutedRequestContext { + workflow_key: "embedding".to_owned(), + lane_priority, + ..RoutedRequestContext::default() + } +} +``` + +and pass `self.lane_priority` at both call sites. In `lib.rs`, the memory write embedder (`memory_write_embedder`) gets `.with_lane_priority(openplotva_llm::router::LanePriority::Background)`. In `memory_runtime.rs`, `RoutedDiscoveryRedactor::redact_text`'s context gets `lane_priority: Some(LanePriority::Background),`. + +- [ ] **Step 4: Run the tests** + +Run: `cargo test -p openplotva-app --lib embedder && cargo test -p openplotva-app --lib memory_runtime` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add crates/openplotva-app/src/embedder.rs crates/openplotva-app/src/memory_runtime.rs crates/openplotva-app/src/lib.rs +git commit -m "Queue memory embeddings and redactions at background priority" +``` + +--- + +### Task 6: Lane pools named by device and purpose + +**Files:** +- Create: `migrations/200_backend_lane_pools.up.sql`, `migrations/200_backend_lane_pools.down.sql` +- Modify: `crates/openplotva-app/src/model_routing.rs:1816` (`MEMORY_GPU2_POOL`) and its tests that name the pool +- Test: `crates/openplotva-storage/src/lib.rs` (contract test) + +**Interfaces:** +- Produces: pool rows `aifarm-gpu0-dialog`, `aifarm-gpu2-utility`, `aifarm-gpu1-media` (renamed, same ids), `aifarm-embedder` (8), `aifarm-gpu1-privacy` (4); the ASR, embedder and privacy filter model rows attached. + +- [ ] **Step 1: Write the contract test** + +```rust + #[test] + fn backend_lane_pools_are_renamed_attached_and_reversible() { + const UP: &str = include_str!("../../../migrations/200_backend_lane_pools.up.sql"); + const DOWN: &str = include_str!("../../../migrations/200_backend_lane_pools.down.sql"); + + for value in [ + "('aifarm-vllm-gpu0', 'aifarm-gpu0-dialog'", + "('aifarm-gpu2-qwen27b', 'aifarm-gpu2-utility'", + "('aifarm-draw-gpu1', 'aifarm-gpu1-media'", + "('aifarm-embedder', 8,", + "('aifarm-gpu1-privacy', 4,", + "('aifarm-asr', 'aifarm-gpu1-media')", + "('aifarm-embed', 'aifarm-embedder')", + "('privacy-filter', 'aifarm-gpu1-privacy')", + "backend_lanes_v1_previous_name", + "backend_lanes_v1_previous_pool_id", + "created_by_backend_lanes_v1", + ] { + assert!(UP.contains(value), "missing lane pool contract {value}"); + } + assert!(DOWN.contains("WHERE config ? 'backend_lanes_v1_previous_pool_id'")); + assert!(DOWN.contains("DELETE FROM llm_capacity_pools WHERE config ? 'created_by_backend_lanes_v1'")); + assert!(DOWN.contains("WHERE config ? 'backend_lanes_v1_previous_name'")); + } +``` + +- [ ] **Step 2: Run it to verify it fails** + +Run: `cargo test -p openplotva-storage --lib backend_lane_pools` +Expected: FAIL (files missing). + +- [ ] **Step 3: Write the migrations** + +`migrations/200_backend_lane_pools.up.sql`: + +```sql +-- Lane pools describe physical resources and are named aifarm--. +-- GPU1's draw, music and speech recognition share one sequential lane under the +-- VRAM arbiter; its privacy filter runs beside that lane. Renamed pools keep their +-- ids, so model rows keep their pool_id. Every change records what it replaced. + +LOCK TABLE llm_capacity_pools, llm_providers, provider_models IN SHARE ROW EXCLUSIVE MODE; + +DO $$ +BEGIN + IF EXISTS ( + SELECT 1 FROM llm_capacity_pools + WHERE config ?| ARRAY['backend_lanes_v1_previous_name', 'created_by_backend_lanes_v1'] + ) OR EXISTS ( + SELECT 1 FROM provider_models WHERE config ? 'backend_lanes_v1_previous_pool_id' + ) THEN + RAISE EXCEPTION 'backend lanes marker collision'; + END IF; +END $$; + +UPDATE llm_capacity_pools AS pool +SET config = pool.config || jsonb_build_object( + 'backend_lanes_v1_previous_name', pool.name, + 'backend_lanes_v1_previous_description', pool.description + ), + name = renamed.new_name, + description = renamed.description, + updated_at = now() +FROM ( + VALUES + ('aifarm-vllm-gpu0', 'aifarm-gpu0-dialog', 'GPU0 dialog model: eight lanes.'), + ('aifarm-gpu2-qwen27b', 'aifarm-gpu2-utility', 'GPU2 utility model: eight lanes.'), + ('aifarm-draw-gpu1', 'aifarm-gpu1-media', 'GPU1 draw, music and speech recognition: one lane under the VRAM arbiter.') +) AS renamed (old_name, new_name, description) +WHERE pool.name = renamed.old_name; + +INSERT INTO llm_capacity_pools (name, max_concurrency, description, config) +SELECT new_pool.name, new_pool.width, new_pool.description, + '{"created_by_backend_lanes_v1": true}'::jsonb +FROM ( + VALUES + ('aifarm-embedder', 8, 'Embedder: eight lanes.'), + ('aifarm-gpu1-privacy', 4, 'GPU1 privacy filter: four lanes beside the media lane.') +) AS new_pool (name, width, description) +WHERE NOT EXISTS (SELECT 1 FROM llm_capacity_pools WHERE name = new_pool.name); + +UPDATE provider_models AS model +SET config = model.config || jsonb_build_object( + 'backend_lanes_v1_previous_pool_id', model.pool_id + ), + pool_id = target_pool.id +FROM llm_providers AS provider, + ( + VALUES + ('aifarm-asr', 'aifarm-gpu1-media'), + ('aifarm-embed', 'aifarm-embedder'), + ('privacy-filter', 'aifarm-gpu1-privacy') + ) AS mapping (provider_name, pool_name), + llm_capacity_pools AS target_pool +WHERE provider.id = model.provider_id + AND provider.name = mapping.provider_name + AND target_pool.name = mapping.pool_name + AND model.pool_id IS DISTINCT FROM target_pool.id; +``` + +`migrations/200_backend_lane_pools.down.sql`: + +```sql +LOCK TABLE llm_capacity_pools, provider_models IN SHARE ROW EXCLUSIVE MODE; + +UPDATE provider_models +SET pool_id = (config->>'backend_lanes_v1_previous_pool_id')::BIGINT, + config = config - 'backend_lanes_v1_previous_pool_id' +WHERE config ? 'backend_lanes_v1_previous_pool_id'; + +DELETE FROM llm_capacity_pools WHERE config ? 'created_by_backend_lanes_v1'; + +UPDATE llm_capacity_pools +SET name = config->>'backend_lanes_v1_previous_name', + description = config->>'backend_lanes_v1_previous_description', + config = config - 'backend_lanes_v1_previous_name' - 'backend_lanes_v1_previous_description', + updated_at = now() +WHERE config ? 'backend_lanes_v1_previous_name'; +``` + +- [ ] **Step 4: Follow the rename in code** + +`crates/openplotva-app/src/model_routing.rs:1816`: + +```rust +const MEMORY_GPU2_POOL: &str = "aifarm-gpu2-utility"; +``` + +Update the test SQL and assertions in `model_routing.rs` that name `aifarm-gpu2-qwen27b` as the memory pool (`rg -n "aifarm-gpu2-qwen27b" crates/openplotva-app/src/model_routing.rs`). Leave the historical migration contract tests in `openplotva-storage` untouched. + +- [ ] **Step 5: Run the tests** + +Run: `cargo test -p openplotva-storage --lib backend_lane_pools && cargo test -p openplotva-app --lib model_routing` +Expected: PASS. + +- [ ] **Step 6: Verify the migration on a copy of production routing data** + +Run the up and down SQL against the production database inside one transaction that ends with `ROLLBACK`, as done for migrations 197 and 198 (copy both files into the Postgres container, `BEGIN; CREATE TEMP TABLE ... AS SELECT ... ; \i up; SELECT ...; \i down; SELECT count(*) of rows differing; ROLLBACK;`). +Expected: after up the five pool names exist with widths 8, 8, 1, 8, 4 and the ASR, embedder and privacy filter models point at their pools; after down 0 rows differ. + +- [ ] **Step 7: Commit** + +```bash +git add migrations/200_backend_lane_pools.up.sql migrations/200_backend_lane_pools.down.sql crates +git commit -m "Name lane pools by device and purpose and pool GPU1, embedder and privacy filter" +``` + +--- + +### Task 7: Workers per lane + +**Files:** +- Modify: `crates/openplotva-llm/src/router/capacity.rs` (`derived_worker_count` → `lane_worker_count`, tests) +- Modify: `crates/openplotva-llm/src/router/mod.rs:16` (export) +- Modify: `crates/openplotva-config/src/lib.rs` (`dialog_unpooled_share` → `dialog_workers_per_lane`, add `memory_workers_per_lane`, memory cap default 32) +- Modify: `crates/openplotva-app/src/model_routing.rs` (`RouterRuntime` fields and `apply_published_table`) +- Modify: `crates/openplotva-app/src/lib.rs` (dialog and memory worker sizing near lines 12172 and 12971) + +**Interfaces:** +- Produces: `pub fn lane_worker_count(table: &RoutingTable, workflow_key: &str, per_lane: u32, cap: u32) -> Option`; config fields `dialog_workers_per_lane: i32` (env `PERSISTENT_QUEUE_DIALOG_WORKERS_PER_LANE`, default 2) and `memory_workers_per_lane: i32` (env `PERSISTENT_QUEUE_MEMORY_WORKERS_PER_LANE`, default 2); `DEFAULT_PERSISTENT_QUEUE_MEMORY_WORKERS_CAP = 32`; `RouterRuntime { .., dialog_workers_per_lane: u32, dialog_worker_cap: u32 }`. + +- [ ] **Step 1: Write the failing test** (replace the `derived_worker_count` tests in `capacity.rs`) + +```rust + #[test] + fn workers_follow_the_primary_lane_pool_only() { + let table = table_with_dialog_chain(); + assert_eq!(lane_worker_count(&table, "dialog", 2, 24), Some(16)); + assert_eq!(lane_worker_count(&table, "dialog", 2, 10), Some(10)); + assert_eq!(lane_worker_count(&table, "missing", 2, 24), None); + } +``` + +where `table_with_dialog_chain()` builds (inside the test module) a `RoutingTable` with a dialog route whose primary model is in pool 1 (width 8) and whose fallback models are in pool 2 (width 8) and unpooled — the same shape the old `derived_worker_count` tests used; reuse and rename their setup helper. + +- [ ] **Step 2: Run it to verify it fails** + +Run: `cargo test -p openplotva-llm --lib workers_follow_the_primary_lane_pool_only` +Expected: FAIL (`lane_worker_count` not defined). + +- [ ] **Step 3: Implement** + +```rust +/// Workers for a workflow: the width of its heaviest primary's lane pool times +/// `per_lane`, clamped to `1..=cap`. Fallback capacity is not throughput. +/// `None` when the workflow has no route or its primary has no bounded pool. +#[must_use] +pub fn lane_worker_count( + table: &RoutingTable, + workflow_key: &str, + per_lane: u32, + cap: u32, +) -> Option { + let route = table.resolve(workflow_key, false)?; + let primary = route.primary.iter().max_by_key(|candidate| candidate.weight)?; + let pool = table.model(primary.model)?.pool_id?; + let width = table.pool(pool)?.max_concurrency?; + Some(width.saturating_mul(per_lane.max(1)).clamp(1, cap.max(1))) +} +``` + +Delete `derived_worker_count` and export `lane_worker_count` instead. In `openplotva-config`: replace `dialog_unpooled_share` (field, raw field, env read, default const) with `dialog_workers_per_lane` (`PERSISTENT_QUEUE_DIALOG_WORKERS_PER_LANE`, `DEFAULT_PERSISTENT_QUEUE_DIALOG_WORKERS_PER_LANE: i32 = 2`); add `memory_workers_per_lane` (`PERSISTENT_QUEUE_MEMORY_WORKERS_PER_LANE`, default 2); set `DEFAULT_PERSISTENT_QUEUE_MEMORY_WORKERS_CAP` to 32. In `RouterRuntime` rename `dialog_unpooled_share` to `dialog_workers_per_lane` and call `lane_worker_count(&table, "dialog", self.dialog_workers_per_lane, self.dialog_worker_cap)` in `apply_published_table`. In `lib.rs` use `lane_worker_count` for the initial dialog workers (with `dialog_workers_per_lane`) and for memory workers: + +```rust + let derived_memory_workers = openplotva_llm::router::lane_worker_count( + &router_handle.snapshot(), + memory_runtime::MEMORY_EXTRACTION_WORKFLOW_KEY, + u32::try_from(config.persistent_queue.memory_workers_per_lane.max(1)).unwrap_or(1), + u32::try_from(memory_workers_cap).unwrap_or(1), + ); +``` + +Update `effective_memory_consolidation_workers` tests if they assume the cap of 4. + +- [ ] **Step 4: Run the tests** + +Run: `cargo test -p openplotva-llm --lib && cargo test -p openplotva-config && cargo test -p openplotva-app --lib` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add crates +git commit -m "Size dialog and memory workers from their primary lane pool" +``` + +--- + +### Task 8: Lane configuration check against Discovery + +**Files:** +- Create: `crates/openplotva-app/src/lane_checks.rs` +- Modify: `crates/openplotva-llm/src/router/table.rs` (`RoutingTable::routed_models`, `RoutingTable::pool_ids`) +- Modify: `crates/openplotva-app/src/lib.rs` (module declaration, startup check, readiness entry) +- Modify: `crates/openplotva-app/src/model_routing.rs` (`reload_router_runtime` schedules the check) + +**Interfaces:** +- Consumes: `RoutingTable`, the routing event reporter (`crate::runtime_routing::RoutingEventReporter`), Discovery `GET /v1/services/{name}/capacity` → `{"capacity": {"max_concurrent_jobs": N, ..}}`. +- Produces: + - `pub struct LaneConfigIssue { pub kind: &'static str, pub pool_id: Option, pub detail: String }` + - `pub fn lane_config_issues(table: &RoutingTable, discovery_widths: &HashMap) -> Vec` + - `pub async fn fetch_discovery_widths(http: &reqwest::Client, base_url: &str, services: &[String]) -> HashMap` + - `pub async fn check_lane_config(http, base_url, table, reporter) -> Vec` + +- [ ] **Step 1: Write the failing tests** in `lane_checks.rs` + +```rust +#[cfg(test)] +mod tests { + use super::*; + + fn table(asr_pool: Option, gpu1_width: u32) -> RoutingTable { + // providers: 1 draw-api (draw-api service), 2 asr (asr-api service), 3 openrouter (no service) + // models: 10 draw in pool 3, 20 asr in `asr_pool`, 30 luna unpooled + // routes: image_generation -> 10, voice_transcription -> 20, dialog -> 30 + // pools: 3 width `gpu1_width` + crate::lane_checks::tests_support::routing_table(asr_pool, gpu1_width) + } + + fn widths() -> HashMap { + HashMap::from([("draw-api".to_owned(), 1), ("asr-api".to_owned(), 1)]) + } + + #[test] + fn a_shared_sequential_lane_spanning_two_services_is_valid() { + assert!(lane_config_issues(&table(Some(3), 1), &widths()).is_empty()); + } + + #[test] + fn a_farm_model_outside_any_pool_is_reported() { + let issues = lane_config_issues(&table(None, 1), &widths()); + assert_eq!(issues.len(), 1); + assert_eq!(issues[0].kind, "unpooled_farm_model"); + } + + #[test] + fn a_pool_wider_than_its_service_is_reported() { + let issues = lane_config_issues(&table(Some(3), 2), &widths()); + assert!(issues.iter().any(|issue| issue.kind == "width_mismatch")); + } + + #[test] + fn unknown_discovery_widths_are_not_reported() { + assert!(lane_config_issues(&table(Some(3), 2), &HashMap::new()).is_empty()); + } +} +``` + +and a `#[cfg(test)] pub(crate) mod tests_support` in the same file that builds that table with `RoutingTable::new`, `set_pools`, and `set_route` (one `Candidate` per route, `Role::Primary`, weight 100, default `BreakerConfig`, `InferenceOverrides::default()`, `RetryBudget::default()`, `LanePriority::Interactive`, 30 s reserve). + +- [ ] **Step 2: Run them to verify they fail** + +Run: `cargo test -p openplotva-app --lib lane_checks` +Expected: FAIL (module missing). + +- [ ] **Step 3: Implement** + +In `table.rs` add: + +```rust + /// Models any route can reach: primaries, fallbacks, canaries, shadows and + /// trigger targets of every workflow and scope. + pub fn routed_models(&self) -> impl Iterator { + let mut ids: Vec = self + .routes + .values() + .flat_map(|scoped| scoped.global.iter().chain(scoped.vip.iter())) + .flat_map(|route| { + route + .primary + .iter() + .chain(route.fallback_tail.iter()) + .chain(route.canary.iter()) + .chain(route.shadow.iter()) + .chain(route.triggers.iter().map(|trigger| &trigger.engage)) + .map(|candidate| candidate.model) + }) + .collect(); + ids.sort_unstable(); + ids.dedup(); + ids.into_iter().filter_map(move |id| self.models.get(&id)) + } +``` + +`lane_checks.rs`: + +```rust +//! Checks that lane pools match the farm: every routed farm model sits in a +//! pool, a Discovery service never splits across pools, and a pool is not wider +//! than the services it spans. Mismatches are reported, never enforced. + +use std::collections::{BTreeMap, BTreeSet, HashMap}; +use std::time::Duration; + +use openplotva_llm::router::RoutingTable; + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct LaneConfigIssue { + pub kind: &'static str, + pub pool_id: Option, + pub detail: String, +} + +#[must_use] +pub fn lane_config_issues( + table: &RoutingTable, + discovery_widths: &HashMap, +) -> Vec { + let mut issues = Vec::new(); + let mut pools_by_service: BTreeMap> = BTreeMap::new(); + let mut services_by_pool: BTreeMap> = BTreeMap::new(); + for model in table.routed_models().filter(|model| model.enabled) { + let Some(service) = table + .provider(model.provider) + .filter(|provider| provider.enabled) + .and_then(|provider| provider.discovery_service_name.clone()) + .filter(|service| !service.trim().is_empty()) + else { + continue; + }; + match model.pool_id { + Some(pool) => { + pools_by_service.entry(service.clone()).or_default().insert(pool); + services_by_pool.entry(pool).or_default().insert(service); + } + None => issues.push(LaneConfigIssue { + kind: "unpooled_farm_model", + pool_id: None, + detail: format!("model {} reaches {service} without a lane pool", model.model_name), + }), + } + } + for (service, pools) in &pools_by_service { + if pools.len() > 1 { + issues.push(LaneConfigIssue { + kind: "split_service", + pool_id: None, + detail: format!("{service} is split across pools {pools:?}"), + }); + } + } + for (pool, services) in &services_by_pool { + let Some(width) = table.pool(*pool).and_then(|spec| spec.max_concurrency) else { + continue; + }; + let known: Vec = services + .iter() + .filter_map(|service| discovery_widths.get(service).copied()) + .collect(); + let Some(narrowest) = known.iter().copied().min() else { + continue; + }; + let mismatch = if services.len() == 1 { width != narrowest } else { width > narrowest }; + if mismatch { + issues.push(LaneConfigIssue { + kind: "width_mismatch", + pool_id: Some(*pool), + detail: format!("pool {pool} has {width} lanes; Discovery allows {narrowest} for {services:?}"), + }); + } + } + issues +} + +/// `max_concurrent_jobs` per Discovery service; services that do not answer are left out. +pub async fn fetch_discovery_widths( + http: &reqwest::Client, + base_url: &str, + services: &[String], +) -> HashMap { + let mut widths = HashMap::new(); + for service in services { + let url = format!("{}/v1/services/{service}/capacity", base_url.trim_end_matches('/')); + let Ok(response) = http.get(url).timeout(Duration::from_secs(5)).send().await else { + continue; + }; + let Ok(body) = response.json::().await else { + continue; + }; + if let Some(width) = body["capacity"]["max_concurrent_jobs"] + .as_u64() + .and_then(|width| u32::try_from(width).ok()) + { + widths.insert(service.clone(), width); + } + } + widths +} +``` + +Plus `pub async fn check_lane_config(http: &reqwest::Client, base_url: &str, table: &RoutingTable, reporter: Option<&crate::runtime_routing::RoutingEventReporter>) -> Vec` that collects the routed services from `table.routed_models()`, calls `fetch_discovery_widths`, runs `lane_config_issues`, records one `lane_config_mismatch` routing event (severity `warn`, workflow key `lanes`, `detail = {"kind": .., "pool_id": .., "detail": ..}`) per issue and logs `tracing::warn!` for each. Build the event with the same constructor the walker's events use (`crate::runtime_routing::RoutingEvent`, see `routing_event_with_severity` in `routed_attempts.rs`). + +In `lib.rs`: declare `mod lane_checks;`; after the router table is loaded at startup, run `check_lane_config` once with a `reqwest::Client` and `config.llm.discovery.base_url`, and push `ReadinessCheck::ok("lane_config", ..)` whose detail is `"lane pools match Discovery"` or `" lane config mismatches: "` (never `error`, so readiness stays green). In `reload_router_runtime`, spawn the same check after `apply_published_table` when the runtime carries a Discovery base URL and HTTP client (add those two fields to `RouterRuntime`). + +- [ ] **Step 4: Run the tests** + +Run: `cargo test -p openplotva-app --lib lane_checks && cargo test -p openplotva-llm --lib` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add crates +git commit -m "Report lane pools that do not match the farm's Discovery limits" +``` + +--- + +### Task 9: Lane observability + +**Files:** +- Modify: `crates/openplotva-app/src/lib.rs` (admin routing status JSON near line 8843; runtime API wiring near line 11266) +- Modify: `crates/openplotva-server/src/runtime_graphql.rs` (`RuntimeLaneInspector`, `RuntimeLaneData`, `llmLanes` query, diagnostics field) +- Test: `crates/openplotva-server/src/runtime_graphql.rs` tests + +**Interfaces:** +- Consumes: `PoolRegistry::occupancy()` (Task 1), pool names from the routing table. +- Produces: GraphQL `llmLanes: [RuntimeLane!]!` with fields `id`, `name`, `maxConcurrency`, `inFlight`, `queuedInteractive`, `queuedUserTool`, `queuedBackground`, `meanHoldMs`. + +- [ ] **Step 1: Write the failing GraphQL test** next to the existing runtime GraphQL tests that use stub inspectors + +```rust + struct LaneInspectorStub; + + impl RuntimeLaneInspector for LaneInspectorStub { + fn lanes(&self) -> Vec { + vec![RuntimeLaneData { + id: 1, + name: "aifarm-gpu0-dialog".to_owned(), + max_concurrency: Some(8), + in_flight: 8, + queued_interactive: 2, + queued_user_tool: 0, + queued_background: 0, + mean_hold_ms: Some(3400), + }] + } + } + + #[tokio::test] + async fn llm_lanes_reports_occupancy_and_queues() { + let schema = runtime_schema_with(|diagnostics| { + diagnostics.lane_inspector = Some(Arc::new(LaneInspectorStub)); + }); + let response = schema + .execute("{ llmLanes { name maxConcurrency inFlight queuedInteractive meanHoldMs } }") + .await; + assert!(response.errors.is_empty(), "{:?}", response.errors); + let data = response.data.into_json().expect("json"); + assert_eq!(data["llmLanes"][0]["name"], "aifarm-gpu0-dialog"); + assert_eq!(data["llmLanes"][0]["inFlight"], 8); + assert_eq!(data["llmLanes"][0]["queuedInteractive"], 2); + } +``` + +Use the test module's existing schema builder with diagnostics (search the module for how `routing_event_inspector` is set in tests and follow that helper; name it `runtime_schema_with` only if no equivalent exists). + +- [ ] **Step 2: Run it to verify it fails** + +Run: `cargo test -p openplotva-server --lib llm_lanes_reports_occupancy_and_queues` +Expected: FAIL (types missing). + +- [ ] **Step 3: Implement** + +```rust +/// Runtime API lane pool read boundary. +pub trait RuntimeLaneInspector: Send + Sync { + fn lanes(&self) -> Vec; +} + +#[derive(Clone, Debug, async_graphql::SimpleObject)] +#[graphql(name = "RuntimeLane")] +pub struct RuntimeLaneData { + pub id: i64, + pub name: String, + pub max_concurrency: Option, + pub in_flight: u32, + pub queued_interactive: u32, + pub queued_user_tool: u32, + pub queued_background: u32, + pub mean_hold_ms: Option, +} +``` + +Add `pub lane_inspector: Option>` to the diagnostics struct and the query root (mirroring `routing_event_inspector`), and the resolver: + +```rust + async fn llm_lanes(&self) -> Vec { + self.lane_inspector + .as_deref() + .map(RuntimeLaneInspector::lanes) + .unwrap_or_default() + } +``` + +In `lib.rs`, implement the inspector over `Arc` and `Arc` (name from the snapshot's pool rows; add `RoutingTable::pool_name(id) -> Option<&str>` by storing names next to specs in `set_pools`, or carry names in `PoolSpec` as `name: String` if simpler — keep `PoolSpec: Clone` and update its constructors). Extend the admin routing status JSON with `"queued": {"interactive": .., "user_tool": .., "background": ..}`, `"mean_hold_ms"` and `"name"` per pool. + +- [ ] **Step 4: Run the tests** + +Run: `cargo test -p openplotva-server --lib && cargo test -p openplotva-app --lib` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add crates +git commit -m "Expose lane occupancy, queues and hold times" +``` + +--- + +### Task 10: Verification and delivery + +**Files:** none new. + +- [ ] **Step 1: Full local gate** + +Run: +```bash +cargo fmt --all +cargo clippy --workspace --all-targets -- -D warnings +cargo test -p openplotva-llm -p openplotva-storage -p openplotva-app -p openplotva-server -p openplotva-config --lib +``` +Expected: all clean and green. + +- [ ] **Step 2: Storage live tests on a fresh Postgres 17** (as CI runs them) + +Run a throwaway `pgvector/pgvector:pg17` container, set `OPENPLOTVA_TEST_POSTGRES_DSN`, and run +`cargo test -p openplotva-storage --lib -- --test-threads=1 outbox_roundtrip_is_idempotent_ordered_and_fenced live_`. +Expected: all pass (migrations 199 and 200 apply through the migrator). Remove the container, its image and volume afterwards. + +- [ ] **Step 3: Pull request and review loop** + +Push `feat/backend-lanes`, open a ready PR with the summary, migrations, verification and deployment risk (dialog turns may wait in Gemma's queue instead of moving to GPU2; Luna traffic drops). Handle every review finding per AGENTS.md before merging. + +- [ ] **Step 4: Deploy when the owner asks, then verify in production** + +After `deploy-production.yml` succeeds: +- `_sqlx_migrations` has 199 and 200; `llm_capacity_pools` has the five lane names; the ASR, embedder and privacy filter models point at their pools. +- The startup log shows dialog workers 16 and memory workers 16; readiness detail for `lane_config` says the pools match Discovery. +- Over the first hour: dialog requests served by GPU2 and OpenRouter are only failures and budget bypasses (`lane_bypass`), `lane_wait` events show queueing on GPU0, `llmLanes` shows queues per priority. From f4b208aadf39af3aa9368061809ff43cef362426 Mon Sep 17 00:00:00 2001 From: Valeriy Selitskiy <239034+iamwavecut@users.noreply.github.com> Date: Mon, 12 Oct 2026 21:39:36 +0200 Subject: [PATCH 05/18] Store each workflow's lane priority and fallback reserve --- crates/openplotva-app/src/dialog_runtime.rs | 2 + crates/openplotva-app/src/image_jobs.rs | 2 + crates/openplotva-app/src/model_routing.rs | 57 ++++++++++++++++--- crates/openplotva-app/src/routed_attempts.rs | 10 +++- crates/openplotva-llm/src/router/capacity.rs | 2 + crates/openplotva-llm/src/router/mod.rs | 2 + crates/openplotva-llm/src/router/table.rs | 6 ++ crates/openplotva-storage/src/lib.rs | 21 +++++++ crates/openplotva-storage/src/llm_routing.rs | 6 +- .../199_workflow_lane_settings.down.sql | 3 + migrations/199_workflow_lane_settings.up.sql | 16 ++++++ 11 files changed, 115 insertions(+), 12 deletions(-) create mode 100644 migrations/199_workflow_lane_settings.down.sql create mode 100644 migrations/199_workflow_lane_settings.up.sql diff --git a/crates/openplotva-app/src/dialog_runtime.rs b/crates/openplotva-app/src/dialog_runtime.rs index 99bb52ce..ca0de0fe 100644 --- a/crates/openplotva-app/src/dialog_runtime.rs +++ b/crates/openplotva-app/src/dialog_runtime.rs @@ -1925,6 +1925,8 @@ mod tests { retry_max_hops: 1, retry_wall_ms: 60_000, enabled: true, + lane_priority: 0, + fallback_reserve_ms: 30_000, }], assignments: vec![AssignmentRecord { id: 100, diff --git a/crates/openplotva-app/src/image_jobs.rs b/crates/openplotva-app/src/image_jobs.rs index ad865cea..190f5123 100644 --- a/crates/openplotva-app/src/image_jobs.rs +++ b/crates/openplotva-app/src/image_jobs.rs @@ -7647,6 +7647,8 @@ mod tests { retry_max_hops: 3, retry_wall_ms: 60_000, enabled: true, + lane_priority: 0, + fallback_reserve_ms: 30_000, }], assignments: vec![AssignmentRecord { id: 100, diff --git a/crates/openplotva-app/src/model_routing.rs b/crates/openplotva-app/src/model_routing.rs index 37df1463..34e07fcf 100644 --- a/crates/openplotva-app/src/model_routing.rs +++ b/crates/openplotva-app/src/model_routing.rs @@ -13,8 +13,8 @@ use openplotva_config::AppConfig; use openplotva_llm::router::table::Role; use openplotva_llm::router::triggers::{TriggerCondition, TriggerSpec}; use openplotva_llm::router::{ - BreakerConfig, Candidate, InferenceOverrides, Kind, ModelRow, PoolSpec, ProviderRow, - RetryBudget, RouterHandle, RoutingTable, Scope, WorkflowRoute, + BreakerConfig, Candidate, InferenceOverrides, Kind, LanePriority, ModelRow, PoolSpec, + ProviderRow, RetryBudget, RouterHandle, RoutingTable, Scope, WorkflowRoute, }; use openplotva_storage::StorageError; use openplotva_storage::llm_routing::{ @@ -1956,11 +1956,14 @@ pub async fn backfill_memory_routing_cascades(pool: &PgPool) -> Result Result, + /// Lane queue priority; `None` takes the workflow's `lane_priority`. + pub lane_priority: Option, pub suppress_all_attempts_exhausted_admin_report: bool, pub suppressed_all_attempts_exhausted_reason: Option, } @@ -1016,6 +1018,8 @@ mod tests { retry_max_hops: 1, retry_wall_ms: 60_000, enabled: true, + lane_priority: 0, + fallback_reserve_ms: 30_000, }], assignments: vec![assignment(assignment_overrides)], triggers: vec![], @@ -1644,6 +1648,8 @@ mod tests { retry_max_hops: max_hops, retry_wall_ms: 60_000, enabled: true, + lane_priority: 0, + fallback_reserve_ms: 30_000, }], assignments: vec![first, second], triggers: vec![], diff --git a/crates/openplotva-llm/src/router/capacity.rs b/crates/openplotva-llm/src/router/capacity.rs index 498aaac1..5d6a08fd 100644 --- a/crates/openplotva-llm/src/router/capacity.rs +++ b/crates/openplotva-llm/src/router/capacity.rs @@ -821,6 +821,8 @@ mod tests { triggers: Vec::new(), shadow: Vec::new(), retry: RetryBudget::default(), + lane_priority: LanePriority::Interactive, + fallback_reserve: Duration::from_secs(30), }, ); table diff --git a/crates/openplotva-llm/src/router/mod.rs b/crates/openplotva-llm/src/router/mod.rs index 15d3fdb0..7c92d0e9 100644 --- a/crates/openplotva-llm/src/router/mod.rs +++ b/crates/openplotva-llm/src/router/mod.rs @@ -58,6 +58,8 @@ mod tests { triggers: Vec::new(), shadow: Vec::new(), retry: RetryBudget::default(), + lane_priority: LanePriority::Interactive, + fallback_reserve: std::time::Duration::from_secs(30), } } diff --git a/crates/openplotva-llm/src/router/table.rs b/crates/openplotva-llm/src/router/table.rs index 18d7f960..bf3aff70 100644 --- a/crates/openplotva-llm/src/router/table.rs +++ b/crates/openplotva-llm/src/router/table.rs @@ -5,10 +5,12 @@ //! control plane is engine-agnostic policy over abstract provider/model ids. use std::collections::HashMap; +use std::time::Duration; use serde_json::Value; use super::breaker::BreakerConfig; +use super::capacity::LanePriority; use super::policy::RetryBudget; use super::triggers::TriggerSpec; @@ -134,6 +136,10 @@ pub struct WorkflowRoute { pub triggers: Vec, pub shadow: Vec, pub retry: RetryBudget, + /// Place of this workflow's requests in their lane queues. + pub lane_priority: LanePriority, + /// Time kept for the next candidate when a lane wait has a deadline. + pub fallback_reserve: Duration, } /// Per-workflow holder of the global default and optional VIP override. diff --git a/crates/openplotva-storage/src/lib.rs b/crates/openplotva-storage/src/lib.rs index 38147151..24fa2528 100644 --- a/crates/openplotva-storage/src/lib.rs +++ b/crates/openplotva-storage/src/lib.rs @@ -10066,6 +10066,27 @@ mod tests { assert!(DOWN.contains("WHERE config ? 'textual_tool_calls_v1_previous'")); } + #[test] + fn workflow_lane_settings_default_and_are_reversible() { + const UP: &str = include_str!("../../../migrations/199_workflow_lane_settings.up.sql"); + const DOWN: &str = include_str!("../../../migrations/199_workflow_lane_settings.down.sql"); + + for value in [ + "ADD COLUMN lane_priority SMALLINT NOT NULL DEFAULT 0", + "ADD COLUMN fallback_reserve_ms INTEGER NOT NULL DEFAULT 30000", + "SET lane_priority = 1", + "'history_summary', 'youtube_summary', 'media_prompt_optimizer'", + "'image_generation', 'image_generation_flux', 'image_generation_boogu_turbo'", + "'image_edit', 'image_edit_flux', 'image_edit_boogu_turbo', 'music'", + "SET lane_priority = 2, fallback_reserve_ms = 0", + "WHERE key IN ('memory_extraction', 'memory_subject_merge')", + ] { + assert!(UP.contains(value), "missing lane settings contract {value}"); + } + assert!(DOWN.contains("DROP COLUMN lane_priority")); + assert!(DOWN.contains("DROP COLUMN fallback_reserve_ms")); + } + #[test] fn vision_fallback_joins_the_gpu2_pool_and_is_reversible() { const UP: &str = include_str!("../../../migrations/198_vision_fallback_gpu2_pool.up.sql"); diff --git a/crates/openplotva-storage/src/llm_routing.rs b/crates/openplotva-storage/src/llm_routing.rs index 4baa73e5..125a452d 100644 --- a/crates/openplotva-storage/src/llm_routing.rs +++ b/crates/openplotva-storage/src/llm_routing.rs @@ -70,6 +70,8 @@ pub struct WorkflowRecord { pub retry_max_hops: i32, pub retry_wall_ms: i32, pub enabled: bool, + pub lane_priority: i16, + pub fallback_reserve_ms: i32, } #[derive(Clone, Debug, PartialEq)] @@ -422,7 +424,7 @@ const SQL_GET_POOL_BY_NAME: &str = "SELECT id, name, max_concurrency, descriptio const SQL_UPDATE_POOL: &str = "UPDATE llm_capacity_pools SET name = $2, max_concurrency = $3, description = $4, config = $5::jsonb, updated_at = now() WHERE id = $1"; const SQL_DELETE_POOL: &str = "DELETE FROM llm_capacity_pools WHERE id = $1"; -const SQL_LIST_WORKFLOWS: &str = "SELECT key, kind, full_routing, retry_max_hops, retry_wall_ms, enabled FROM workflows ORDER BY key ASC"; +const SQL_LIST_WORKFLOWS: &str = "SELECT key, kind, full_routing, retry_max_hops, retry_wall_ms, enabled, lane_priority, fallback_reserve_ms FROM workflows ORDER BY key ASC"; const SQL_INSERT_WORKFLOW_IF_MISSING: &str = "INSERT INTO workflows (key, kind, full_routing, retry_max_hops, retry_wall_ms, enabled) VALUES ($1, $2, $3, $4, $5, $6) ON CONFLICT (key) DO NOTHING"; const SQL_UPDATE_WORKFLOW: &str = "UPDATE workflows SET full_routing = $2, retry_max_hops = $3, retry_wall_ms = $4, enabled = $5 WHERE key = $1"; const SQL_SET_WORKFLOW_ENABLED: &str = "UPDATE workflows SET enabled = $2 WHERE key = $1"; @@ -894,6 +896,8 @@ fn workflow_from_row(row: PgRow) -> Result { retry_max_hops: row.try_get("retry_max_hops")?, retry_wall_ms: row.try_get("retry_wall_ms")?, enabled: row.try_get("enabled")?, + lane_priority: row.try_get("lane_priority")?, + fallback_reserve_ms: row.try_get("fallback_reserve_ms")?, }) } diff --git a/migrations/199_workflow_lane_settings.down.sql b/migrations/199_workflow_lane_settings.down.sql new file mode 100644 index 00000000..28dbbaa1 --- /dev/null +++ b/migrations/199_workflow_lane_settings.down.sql @@ -0,0 +1,3 @@ +ALTER TABLE workflows + DROP COLUMN lane_priority, + DROP COLUMN fallback_reserve_ms; diff --git a/migrations/199_workflow_lane_settings.up.sql b/migrations/199_workflow_lane_settings.up.sql new file mode 100644 index 00000000..d5e4104d --- /dev/null +++ b/migrations/199_workflow_lane_settings.up.sql @@ -0,0 +1,16 @@ +-- Each workflow's place in its lane queues and the time it keeps for a fallback. +-- Priority 0 is interactive, 1 a user-facing tool, 2 background work; a lane pool +-- serves lower values first. Background work keeps no fallback reserve: without a +-- deadline it waits for its own lane instead of moving to another backend. + +ALTER TABLE workflows + ADD COLUMN lane_priority SMALLINT NOT NULL DEFAULT 0, + ADD COLUMN fallback_reserve_ms INTEGER NOT NULL DEFAULT 30000; + +UPDATE workflows SET lane_priority = 1 +WHERE key IN ('history_summary', 'youtube_summary', 'media_prompt_optimizer', + 'image_generation', 'image_generation_flux', 'image_generation_boogu_turbo', + 'image_edit', 'image_edit_flux', 'image_edit_boogu_turbo', 'music'); + +UPDATE workflows SET lane_priority = 2, fallback_reserve_ms = 0 +WHERE key IN ('memory_extraction', 'memory_subject_merge'); From f53b4daad8b709b028f2fd9cba2c9722b6d6c05d Mon Sep 17 00:00:00 2001 From: Valeriy Selitskiy <239034+iamwavecut@users.noreply.github.com> Date: Mon, 12 Oct 2026 21:47:58 +0200 Subject: [PATCH 06/18] Admit routed attempts through lane queues: fall back on failure or budget --- crates/openplotva-app/src/routed_attempts.rs | 543 ++++++++++++------- 1 file changed, 344 insertions(+), 199 deletions(-) diff --git a/crates/openplotva-app/src/routed_attempts.rs b/crates/openplotva-app/src/routed_attempts.rs index 95d671fe..2695fff6 100644 --- a/crates/openplotva-app/src/routed_attempts.rs +++ b/crates/openplotva-app/src/routed_attempts.rs @@ -10,13 +10,6 @@ use openplotva_llm::router::{ }; use serde_json::{Value, json}; -/// Upper bound on how long one walker run waits for a capacity-pool slot when -/// every candidate's pool is full. Slot waiting deliberately does not count -/// against the workflow's `retry_wall_ms` (which budgets attempt execution): -/// pool queueing replaces the server-side queueing some backends did before, -/// so it is bounded only by the request deadline and this sanity cap. -const DEFAULT_MAX_SLOT_WAIT: Duration = Duration::from_secs(300); - /// Minimum share of the caller's budget an attempt must have received for a /// deadline cut to count against the provider's circuit breaker. const ATTEMPT_BREAKER_MIN_SLICE: Duration = Duration::from_secs(30); @@ -31,8 +24,6 @@ pub struct RoutedAttemptWalker { triggers: Arc, pools: Arc, openrouter_free_gate: Option>, - max_slot_wait: Duration, - primary_slot_wait: Duration, model_output_retries: usize, reporter: Option, } @@ -51,8 +42,6 @@ impl RoutedAttemptWalker { triggers, pools, openrouter_free_gate: None, - max_slot_wait: DEFAULT_MAX_SLOT_WAIT, - primary_slot_wait: Duration::ZERO, model_output_retries: 0, reporter: None, } @@ -91,12 +80,6 @@ impl RoutedAttemptWalker { self } - #[must_use] - pub fn with_max_slot_wait(mut self, max_slot_wait: Duration) -> Self { - self.max_slot_wait = max_slot_wait; - self - } - /// How many extra samples the same model gets when a validator rejects what it /// wrote, before the chain walks on to another provider. Zero (the default) keeps /// the immediate walk. @@ -106,15 +89,6 @@ impl RoutedAttemptWalker { self } - /// How long a busy pool of a primary-role candidate is waited for before - /// the walker moves on to the fallback tail. Zero (the default) keeps the - /// instant skip. - #[must_use] - pub fn with_primary_slot_wait(mut self, primary_slot_wait: Duration) -> Self { - self.primary_slot_wait = primary_slot_wait; - self - } - /// Config of the workflow's heaviest global primary, for callers that shape /// the request to the model before the walk starts. #[must_use] @@ -130,53 +104,118 @@ impl RoutedAttemptWalker { .map(|model| model.config.clone()) } - // The fallback tail serves the request without the primary's tools or - // quality, so a bounded queue on a busy primary beats an instant downgrade. - // Slot time is charged to the caller's deadline, not to the retry wall. - async fn wait_for_primary_slot( + /// Admit one candidate through its lane queue. A free lane is taken at once. + /// A candidate with an open breaker is never waited for. Otherwise the + /// request waits in the lane queue until `deadline - reserve`, keeping the + /// workflow's fallback reserve only while another candidate follows, and + /// moves on at once when the forecast wait already exceeds that budget. + /// Lane time is charged to the caller's deadline, not to the retry wall. + #[allow(clippy::too_many_arguments)] + async fn admit_lane( &self, pool: Option, + live: bool, + has_next_candidate: bool, + route: &WorkflowRoute, context: &RoutedRequestContext, attempt: &Attempt, total_slot_wait: &mut Duration, ) -> Option { - let started = Instant::now(); - let budget = match context.deadline { - Some(deadline) => deadline - .saturating_duration_since(started) - .min(self.primary_slot_wait), - None => self.primary_slot_wait, + if let Some(permit) = self.pools.try_acquire(pool) { + return Some(permit); + } + if !live { + return None; + } + let priority = context.lane_priority.unwrap_or(route.lane_priority); + let reserve = if has_next_candidate { + route.fallback_reserve + } else { + Duration::ZERO }; - let mut permit = None; - while permit.is_none() { - let remaining = budget.saturating_sub(started.elapsed()); - if remaining.is_zero() || !self.pools.wait_for_release(remaining).await { - break; + let budget = context.deadline.map(|deadline| { + deadline + .saturating_duration_since(Instant::now()) + .saturating_sub(reserve) + }); + if let Some(budget) = budget { + let forecast = self.pools.forecast_wait(pool, priority); + if budget.is_zero() || forecast.is_some_and(|forecast| forecast > budget) { + self.record_event(routing_event_with_severity( + "lane_bypass", + "info", + context, + Some(attempt.provider), + Some(attempt.model), + "lane wait would exceed the request budget; trying the next candidate", + json!({ + "priority": priority.as_str(), + "budget_ms": budget.as_millis(), + "forecast_ms": forecast.map(|forecast| forecast.as_millis()), + }), + )); + return None; } - permit = self.pools.try_acquire(pool); } + let started = Instant::now(); + let permit = self.pools.acquire(pool, priority, budget).await; let waited = started.elapsed(); *total_slot_wait += waited; self.record_event(routing_event_with_severity( - "primary_slot_wait", + "lane_wait", "info", context, Some(attempt.provider), Some(attempt.model), if permit.is_some() { - "primary pool slot acquired after waiting" + "lane acquired after waiting in its queue" } else { - "primary pool stayed busy for the whole wait budget; falling back" + "lane wait reached the request budget" }, json!({ + "priority": priority.as_str(), "waited_ms": waited.as_millis(), - "budget_ms": budget.as_millis(), + "budget_ms": budget.map(|budget| budget.as_millis()), "acquired": permit.is_some(), }), )); permit } + /// Among the live weighted head of the chain (primaries and engaged + /// overflow), move candidates with a free lane ahead of busy ones. Every + /// other position keeps its order. + fn prefer_free_lanes( + &self, + attempts: &mut [Attempt], + table: &openplotva_llm::router::RoutingTable, + now: Instant, + ) { + let positions: Vec = attempts + .iter() + .enumerate() + .filter(|(_, attempt)| { + matches!(attempt.role, Role::Primary | Role::Overflow) + && self + .breakers + .is_live_at(attempt.provider, attempt.model, now) + }) + .map(|(index, _)| index) + .collect(); + let mut head: Vec = positions + .iter() + .map(|index| attempts[*index].clone()) + .collect(); + head.sort_by_key(|attempt| { + !self + .pools + .has_free_lane(table.model(attempt.model).and_then(|model| model.pool_id)) + }); + for (index, attempt) in positions.into_iter().zip(head) { + attempts[index] = attempt; + } + } + pub async fn run( &self, context: RoutedRequestContext, @@ -209,7 +248,6 @@ impl RoutedAttemptWalker { let max_hops = route.retry.max_hops.max(1) as usize; let mut started_attempts = 0usize; let mut total_slot_wait = Duration::ZERO; - let mut wait_reported = false; let mut first_pass = true; let mut last_error = None; let mut last_provider_model = None; @@ -221,11 +259,13 @@ impl RoutedAttemptWalker { // first, and whatever is left still covers a fallback that answers badly. let mut model_output_retry = 0usize; - // Selection-pass loop: walk a fresh chain, skipping candidates whose - // capacity pool is full. A busy skip touches neither the breaker nor - // the hop budget (busy is not dead). When a whole pass is busy, wait - // for any slot to free and re-select; only execution time counts - // against the wall clock, so pool queueing never eats the retry budget. + // Selection-pass loop: walk a fresh chain, admitting each candidate + // through its lane queue. A candidate passed over for its lane (open + // breaker, budget) touches neither the breaker nor the hop budget. + // After an executed attempt fails, re-select so a passed-over + // candidate can be tried with the time that is left; only execution + // time counts against the wall clock, so lane queueing never eats the + // retry budget. 'passes: loop { let selection_now = Instant::now(); let mut attempts = { @@ -242,6 +282,9 @@ impl RoutedAttemptWalker { }) .collect::>() }; + self.prefer_free_lanes(&mut attempts, &table, selection_now); + // The previous step's target is preferred only while its lane is + // free: the preference saves a warm cache, not a queue position. if let Some((provider_name, model_name)) = &context.preferred_target && let Some(index) = attempts.iter().position(|attempt| { table @@ -253,6 +296,9 @@ impl RoutedAttemptWalker { && self .breakers .is_live_at(attempt.provider, attempt.model, selection_now) + && self.pools.has_free_lane( + table.model(attempt.model).and_then(|model| model.pool_id), + ) }) { let preferred = attempts.remove(index); @@ -296,7 +342,7 @@ impl RoutedAttemptWalker { let mut busy_skips = 0usize; let mut quota_skips = 0usize; let mut executed_this_pass = 0usize; - for attempt in attempts { + for (index, attempt) in attempts.iter().enumerate() { if attempted_targets.contains(&(attempt.provider, attempt.model)) { continue; } @@ -315,26 +361,29 @@ impl RoutedAttemptWalker { let provider = table.provider(attempt.provider); let model = table.model(attempt.model); let pool_id = model.and_then(|row| row.pool_id); + let live = + self.breakers + .is_live_at(attempt.provider, attempt.model, Instant::now()); + let has_next_candidate = attempts[index + 1..] + .iter() + .any(|next| !attempted_targets.contains(&(next.provider, next.model))); // A rejected model output is not a sick provider: the same model gets a // bounded number of fresh samples before the chain walks on, so a guard // rejection does not hand a tool-shaped turn to a tool-less fallback. let mut first_execution = true; loop { - let mut permit = self.pools.try_acquire(pool_id); - if permit.is_none() - && attempt.role == Role::Primary - && !self.primary_slot_wait.is_zero() - { - permit = self - .wait_for_primary_slot( - pool_id, - &context, - &attempt, - &mut total_slot_wait, - ) - .await; - } - let Some(permit) = permit else { + let Some(permit) = self + .admit_lane( + pool_id, + live, + has_next_candidate, + route, + &context, + attempt, + &mut total_slot_wait, + ) + .await + else { if first_execution { busy_skips += 1; } @@ -603,62 +652,34 @@ impl RoutedAttemptWalker { break; } if executed_this_pass > 0 { - // Something ran and failed while other candidates were busy: - // re-select immediately, a slot may have freed meanwhile. + // Something ran and failed while other candidates were passed + // over: re-select, a passed-over candidate may fit now. continue; } - - // Every candidate in the pass was pool-busy: wait for any slot. - let wait_budget = match context.deadline { - Some(deadline) => deadline - .checked_duration_since(Instant::now()) - .unwrap_or(Duration::ZERO) - .min(self.max_slot_wait), - None => self.max_slot_wait, - }; - if !wait_reported { - wait_reported = true; - self.record_event(routing_event_with_severity( - "capacity_pool_wait", - "info", - &context, - None, - None, - "all candidate capacity pools are busy; waiting for a slot", - json!({ - "busy_candidates": busy_skips, - "quota_skips": quota_skips, - "wait_budget_ms": wait_budget.as_millis(), - }), - )); - } - let wait_started = Instant::now(); - let released = if wait_budget.is_zero() { - false - } else { - self.pools.wait_for_release(wait_budget).await - }; - total_slot_wait += wait_started.elapsed(); - if !released { - self.record_event(routing_event( - "capacity_wait_timeout", - &context, - None, - None, - "workflow capacity unavailable: slot wait timed out", - json!({ - "busy_candidates": busy_skips, - "waited_ms": total_slot_wait.as_millis(), - }), - )); - return Err(RoutedAttemptRunError::Routing(RoutedRoutingError::new( - RoutedRoutingErrorKind::CapacityWaitTimeout, - format!( - "workflow capacity unavailable: all pools busy after {}ms slot wait", - total_slot_wait.as_millis() - ), - ))); + if last_error.is_some() || deadline_cut { + break; } + + // No candidate got a lane within the request budget. + self.record_event(routing_event( + "capacity_wait_timeout", + &context, + None, + None, + "workflow capacity unavailable: no candidate lane within the request budget", + json!({ + "busy_candidates": busy_skips, + "quota_skips": quota_skips, + "waited_ms": total_slot_wait.as_millis(), + }), + )); + return Err(RoutedAttemptRunError::Routing(RoutedRoutingError::new( + RoutedRoutingErrorKind::CapacityWaitTimeout, + format!( + "workflow capacity unavailable: no candidate lane after {}ms lane wait", + total_slot_wait.as_millis() + ), + ))); } let (provider_id, model_id) = last_provider_model.unwrap_or_default(); @@ -1678,7 +1699,7 @@ mod tests { } #[tokio::test] - async fn busy_pool_skips_to_free_candidate_without_breaker_mutation() { + async fn a_free_primary_sibling_is_taken_before_a_busy_one() { let snapshot = pooled_snapshot(3); let pools = Arc::new(PoolRegistry::new()); let breakers = Arc::new(BreakerSet::new()); @@ -1692,68 +1713,22 @@ mod tests { ); let _busy = pools.try_acquire(Some(1)).expect("occupy the gpu pool"); - let executed = Arc::new(std::sync::Mutex::new(Vec::new())); - let executed_models = Arc::clone(&executed); - let output = walker - .run( - RoutedRequestContext { - workflow_key: "dialog".to_owned(), - ..RoutedRequestContext::default() - }, - move |attempt| { - let executed_models = Arc::clone(&executed_models); - async move { - executed_models - .lock() - .expect("models") - .push(attempt.model_id); - Ok::<_, std::io::Error>("ok") - } - }, - |_error| None, - ) - .await - .expect("free candidate output"); - - assert_eq!(output, "ok"); - assert_eq!(*executed.lock().expect("models"), vec![20]); + // The 50/50 weighted order puts the busy primary first in about half + // of the runs; every run must still go to the free sibling. + for _ in 0..16 { + let (result, executed) = + tokio::time::timeout(Duration::from_secs(2), run_recording(&walker, None, None)) + .await + .expect("a free sibling must not wait behind a busy one"); + assert_eq!(result.expect("free candidate output"), "ok"); + assert_eq!(executed, vec![20]); + } assert!( breakers.is_live_at(1, 10, std::time::Instant::now()), - "a busy skip must not trip the breaker" + "a busy lane must not trip the breaker" ); } - #[tokio::test] - async fn busy_skips_do_not_consume_the_hop_budget() { - let snapshot = pooled_snapshot(1); - let pools = Arc::new(PoolRegistry::new()); - let walker = walker_with_pools(&snapshot, Arc::clone(&pools)); - let _busy = pools.try_acquire(Some(1)).expect("occupy the gpu pool"); - - let calls = Arc::new(AtomicUsize::new(0)); - let call_count = Arc::clone(&calls); - let output = walker - .run( - RoutedRequestContext { - workflow_key: "dialog".to_owned(), - ..RoutedRequestContext::default() - }, - move |_attempt| { - let call_count = Arc::clone(&call_count); - async move { - call_count.fetch_add(1, Ordering::Relaxed); - Ok::<_, std::io::Error>("ok") - } - }, - |_error| None, - ) - .await - .expect("the single hop must go to the free candidate"); - - assert_eq!(output, "ok"); - assert_eq!(calls.load(Ordering::Relaxed), 1); - } - #[tokio::test(start_paused = true)] async fn failed_target_is_not_reselected_while_fallback_pool_is_busy() { let mut snapshot = pooled_snapshot(3); @@ -1846,7 +1821,7 @@ mod tests { .expect("fallback succeeds after the transient events"); let events = reporter.buffer().routing_events(10); - for event_type in ["attempt_failed", "capacity_pool_wait"] { + for event_type in ["attempt_failed", "lane_wait"] { let event = events .iter() .find(|event| event.event_type == event_type) @@ -1856,7 +1831,7 @@ mod tests { } #[tokio::test(start_paused = true)] - async fn all_busy_waits_for_release_then_succeeds() { + async fn a_busy_lane_is_waited_for_then_serves() { let mut snapshot = pooled_snapshot(3); snapshot.models.truncate(1); snapshot.assignments.truncate(1); @@ -1897,8 +1872,7 @@ mod tests { async fn busy_primary_is_waited_for_before_the_fallback_tail() { let snapshot = primary_with_fallback_snapshot(); let pools = Arc::new(PoolRegistry::new()); - let walker = walker_with_pools(&snapshot, Arc::clone(&pools)) - .with_primary_slot_wait(std::time::Duration::from_millis(200)); + let walker = walker_with_pools(&snapshot, Arc::clone(&pools)); let busy = pools.try_acquire(Some(1)).expect("occupy the gpu pool"); tokio::spawn(async move { tokio::time::sleep(std::time::Duration::from_millis(50)).await; @@ -1916,54 +1890,61 @@ mod tests { |_error| None, ) .await - .expect("the primary slot frees inside the wait budget"); + .expect("the primary lane frees while the request waits"); assert_eq!(served_by, 1, "the primary must serve once its slot frees"); assert_eq!(started.elapsed(), std::time::Duration::from_millis(50)); } #[tokio::test(start_paused = true)] - async fn busy_primary_falls_back_only_after_the_wait_budget() { + async fn busy_primary_falls_back_when_its_lane_wait_reaches_the_budget() { let snapshot = primary_with_fallback_snapshot(); let pools = Arc::new(PoolRegistry::new()); - let walker = walker_with_pools(&snapshot, Arc::clone(&pools)) - .with_primary_slot_wait(std::time::Duration::from_millis(200)); + let walker = walker_with_pools(&snapshot, Arc::clone(&pools)); let _busy = pools.try_acquire(Some(1)).expect("occupy the gpu pool"); let started = tokio::time::Instant::now(); + // The 30 s fallback reserve leaves 200 ms of lane budget. + let deadline = Instant::now() + Duration::from_millis(30_200); let served_by = walker .run( RoutedRequestContext { workflow_key: "dialog".to_owned(), + deadline: Some(deadline), ..RoutedRequestContext::default() }, |attempt| async move { Ok::<_, std::io::Error>(attempt.provider_id) }, |_error| None, ) .await - .expect("the fallback serves after the primary wait budget"); + .expect("the fallback serves after the primary lane budget"); assert_eq!( served_by, 2, "the fallback serves a permanently busy primary" ); - assert_eq!(started.elapsed(), std::time::Duration::from_millis(200)); + let waited = started.elapsed(); + assert!( + waited > Duration::from_millis(150) && waited <= Duration::from_millis(200), + "{waited:?}" + ); } #[tokio::test(start_paused = true)] - async fn slot_wait_timeout_classifies_as_capacity_unavailable() { + async fn a_lane_wait_that_reaches_the_deadline_classifies_as_capacity_unavailable() { let mut snapshot = pooled_snapshot(3); snapshot.models.truncate(1); snapshot.assignments.truncate(1); let pools = Arc::new(PoolRegistry::new()); - let walker = walker_with_pools(&snapshot, Arc::clone(&pools)) - .with_max_slot_wait(std::time::Duration::from_millis(50)); + let walker = walker_with_pools(&snapshot, Arc::clone(&pools)); let _busy = pools.try_acquire(Some(1)).expect("occupy the gpu pool"); + // The only candidate keeps no fallback reserve: it waits until the deadline. let result = walker .run( RoutedRequestContext { workflow_key: "dialog".to_owned(), + deadline: Some(Instant::now() + Duration::from_millis(50)), ..RoutedRequestContext::default() }, |_attempt| async { Ok::<_, std::io::Error>("unreachable") }, @@ -2010,17 +1991,181 @@ mod tests { "run must still be executing when the timeout drops it" ); + let occupancy = pools.occupancy(); + assert_eq!(occupancy.len(), 1); + assert_eq!(occupancy[0].in_flight, 0); + assert_eq!(occupancy[0].queued, [0, 0, 0]); assert!( pools.try_acquire(Some(1)).is_some(), - "cancelling the in-flight attempt must free its slot" + "cancelling the in-flight attempt must free its lane" + ); + } + + fn lane_reporter() -> crate::runtime_routing::RoutingEventReporter { + crate::runtime_routing::RoutingEventReporter::new( + crate::runtime_routing::RoutingEventBuffer::new(16), + None, + None, + ) + } + + async fn run_recording( + walker: &RoutedAttemptWalker, + deadline: Option, + lane_priority: Option, + ) -> ( + Result<&'static str, RoutedAttemptRunError>, + Vec, + ) { + let executed = Arc::new(std::sync::Mutex::new(Vec::new())); + let models = Arc::clone(&executed); + let result = walker + .run( + RoutedRequestContext { + workflow_key: "dialog".to_owned(), + deadline, + lane_priority, + ..RoutedRequestContext::default() + }, + move |attempt| { + let models = Arc::clone(&models); + async move { + models.lock().expect("models").push(attempt.model_id); + Ok::<_, std::io::Error>("ok") + } + }, + |_error| None, + ) + .await; + let executed = executed.lock().expect("models").clone(); + (result, executed) + } + + #[tokio::test] + async fn a_forecast_beyond_the_budget_falls_back_without_waiting() { + let snapshot = primary_with_fallback_snapshot(); + let pools = Arc::new(PoolRegistry::new()); + let reporter = lane_reporter(); + let walker = + walker_with_pools(&snapshot, Arc::clone(&pools)).with_reporter(reporter.clone()); + let warmup = pools.try_acquire(Some(1)).expect("lane"); + tokio::time::sleep(Duration::from_millis(200)).await; + drop(warmup); + let _busy = pools.try_acquire(Some(1)).expect("occupy the primary lane"); + // The 30 s reserve leaves about 50 ms, below the 200 ms mean hold time. + let started = Instant::now(); + let deadline = started + Duration::from_millis(30_050); + + let (result, executed) = run_recording(&walker, Some(deadline), None).await; + + assert_eq!(result.expect("fallback output"), "ok"); + assert_eq!(executed, vec![20]); + assert!( + started.elapsed() < Duration::from_millis(40), + "no lane wait" + ); + assert!( + reporter + .buffer() + .routing_events(16) + .iter() + .any(|event| event.event_type == "lane_bypass") ); + } + + #[tokio::test] + async fn a_candidate_with_an_open_breaker_is_never_waited_for() { + let mut snapshot = primary_with_fallback_snapshot(); + snapshot.workflows[0].retry_max_hops = 3; + let pools = Arc::new(PoolRegistry::new()); + let breakers = Arc::new(BreakerSet::new()); + let open = BreakerConfig { + fail_threshold: 1, + cooldown: Duration::from_secs(60), + }; + breakers.record_failure(1, 10, open); + breakers.record_failure(2, 20, open); + let table = crate::model_routing::build_routing_table(&snapshot); + pools.apply(&table.pool_specs()); + let walker = RoutedAttemptWalker::new( + RouterHandle::new(table), + breakers, + Arc::new(TriggerState::new()), + Arc::clone(&pools), + ); + let _busy = pools.try_acquire(Some(1)).expect("occupy the primary lane"); + + // Both candidates are in cooldown, so the chain probes them in order: + // the busy primary is passed over instead of waited for. + let (result, executed) = + tokio::time::timeout(Duration::from_secs(5), run_recording(&walker, None, None)) + .await + .expect("an open breaker must not queue the request"); + + assert_eq!(result.expect("probe output"), "ok"); + assert_eq!(executed, vec![20]); + } + + #[tokio::test] + async fn an_expired_deadline_is_not_queued() { + let mut snapshot = pooled_snapshot(3); + snapshot.models.truncate(1); + snapshot.assignments.truncate(1); + let pools = Arc::new(PoolRegistry::new()); + let walker = walker_with_pools(&snapshot, Arc::clone(&pools)); + let _busy = pools.try_acquire(Some(1)).expect("occupy the only lane"); + + let (result, executed) = tokio::time::timeout( + Duration::from_secs(5), + run_recording(&walker, Some(Instant::now()), None), + ) + .await + .expect("an expired deadline must not wait in the queue"); + + assert!(result.is_err()); + assert!(executed.is_empty()); + assert_eq!(pools.occupancy()[0].queued, [0, 0, 0]); + } + + #[tokio::test] + async fn an_interactive_request_goes_ahead_of_queued_background_work() { + let mut snapshot = pooled_snapshot(3); + snapshot.models.truncate(1); + snapshot.assignments.truncate(1); + let pools = Arc::new(PoolRegistry::new()); + let walker = Arc::new(walker_with_pools(&snapshot, Arc::clone(&pools))); + let running = pools + .try_acquire(Some(1)) + .expect("a running job holds the lane"); + let order = Arc::new(std::sync::Mutex::new(Vec::new())); + + let mut runs = Vec::new(); + for (name, priority) in [ + ("queued draw", LanePriority::UserTool), + ("transcription", LanePriority::Interactive), + ] { + let walker = Arc::clone(&walker); + let order = Arc::clone(&order); + runs.push(tokio::spawn(async move { + let (result, _) = run_recording(&walker, None, Some(priority)).await; + order.lock().expect("order").push(name); + result.expect("served"); + })); + tokio::task::yield_now().await; + } + assert_eq!(pools.occupancy()[0].queued, [1, 1, 0]); + assert!( + order.lock().expect("order").is_empty(), + "the running job is never preempted" + ); + drop(running); + for run in runs { + run.await.expect("run"); + } + assert_eq!( - pools.occupancy(), - vec![openplotva_llm::router::PoolOccupancy { - id: 1, - max_concurrency: Some(1), - in_flight: 0, - }] + *order.lock().expect("order"), + vec!["transcription", "queued draw"] ); } } From 2471fac514efa34d69c350312304973a6a8cb8ec Mon Sep 17 00:00:00 2001 From: Valeriy Selitskiy <239034+iamwavecut@users.noreply.github.com> Date: Mon, 12 Oct 2026 21:47:58 +0200 Subject: [PATCH 07/18] Drop per-call slot waits now that lanes queue requests --- .env.example | 2 - crates/openplotva-app/src/dialog_runtime.rs | 41 +++++++------------- crates/openplotva-app/src/history_summary.rs | 4 -- crates/openplotva-app/src/lib.rs | 3 -- crates/openplotva-app/src/media.rs | 5 --- crates/openplotva-app/src/memory_runtime.rs | 1 - crates/openplotva-app/src/youtube.rs | 3 -- crates/openplotva-config/src/lib.rs | 20 ---------- docs/CODEBASE_MAP.md | 27 +++++++------ 9 files changed, 31 insertions(+), 75 deletions(-) diff --git a/.env.example b/.env.example index c80274d2..2dc43d64 100644 --- a/.env.example +++ b/.env.example @@ -192,8 +192,6 @@ # DIALOG_AIFARM_POOL_MODELS= # DIALOG_AIFARM_POOL_BASE_URLS= # DIALOG_AIFARM_POOL_API_KEY= -# How long the dialog router waits for a busy primary pool (Gemma) slot before falling back; ms. -# DIALOG_AIFARM_POOL_PRIMARY_CAPACITY_WAIT_MS=12000 # DIALOG_AIFARM_POOL_REASONING_MAX_TOKENS=8192 # DIALOG_VMLX_URL= # DIALOG_VMLX_API_KEY= diff --git a/crates/openplotva-app/src/dialog_runtime.rs b/crates/openplotva-app/src/dialog_runtime.rs index ca0de0fe..65339997 100644 --- a/crates/openplotva-app/src/dialog_runtime.rs +++ b/crates/openplotva-app/src/dialog_runtime.rs @@ -41,7 +41,6 @@ pub type DialogProviderHandle = Arc; /// Workflow key the dialog worker routes through. const DIALOG_WORKFLOW_KEY: &str = "dialog"; -const INTERACTIVE_CAPACITY_WAIT_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(1); /// Builds and caches per-provider transport clients from DB provider rows, so /// a provider added in the admin panel is immediately routable without a @@ -271,20 +270,12 @@ impl RouterChatProvider { factory: Arc, ) -> Self { Self { - walker: RoutedAttemptWalker::new(handle, breakers, triggers, pools) - .with_max_slot_wait(INTERACTIVE_CAPACITY_WAIT_TIMEOUT), + walker: RoutedAttemptWalker::new(handle, breakers, triggers, pools), factory, provider_name: "router".to_owned(), } } - /// Bounded wait for a busy primary pool before the fallback tail is used. - #[must_use] - pub fn with_primary_slot_wait(mut self, primary_slot_wait: std::time::Duration) -> Self { - self.walker = self.walker.with_primary_slot_wait(primary_slot_wait); - self - } - /// Extra samples from the same model after a validator rejects its output. #[must_use] pub fn with_model_output_retries(mut self, retries: usize) -> Self { @@ -505,16 +496,6 @@ pub fn router_dialog_provider( aifarm, prompt_store, )); - // Zero disables the wait on purpose; a negative value is a config slip and - // keeps the documented default instead of silently disabling it. - let configured_wait_ms = config.llm.dialog.aifarm_pool_primary_capacity_wait_ms; - let primary_wait_ms = if configured_wait_ms < 0 { - openplotva_config::DEFAULT_DIALOG_PRIMARY_CAPACITY_WAIT_MS - } else { - configured_wait_ms - }; - let primary_slot_wait = - std::time::Duration::from_millis(u64::try_from(primary_wait_ms).unwrap_or(0)); // A negative value is a config slip, not a way to disable the re-sampling; // zero disables it on purpose. let configured_retries = config.llm.dialog.model_output_retries; @@ -524,7 +505,6 @@ pub fn router_dialog_provider( configured_retries }; let provider = RouterChatProvider::new(handle, breakers, triggers, pools, factory) - .with_primary_slot_wait(primary_slot_wait) .with_model_output_retries(usize::try_from(model_output_retries).unwrap_or(0)); let provider = match routing_events { Some(reporter) => provider.with_routing_event_reporter(reporter), @@ -1018,7 +998,7 @@ mod tests { } #[tokio::test(start_paused = true)] - async fn interactive_router_stops_waiting_for_a_busy_gpu_after_one_virtual_second() { + async fn interactive_router_waits_for_a_busy_gpu_until_the_turn_deadline() { let mut snapshot = routed_dialog_snapshot(); snapshot.models[0].pool_id = Some(1); snapshot.pools.push(PoolRecord { @@ -1048,14 +1028,23 @@ mod tests { ) .with_routing_event_reporter(reporter.clone()); let started = tokio::time::Instant::now(); + let deadline = std::time::Instant::now() + std::time::Duration::from_millis(200); - let error = provider - .run_chat_step(default_step_request()) + let error = crate::dialog_turn::TURN_DEADLINE + .scope( + Some(deadline), + provider.run_chat_step(default_step_request()), + ) .await - .expect_err("a permanently busy interactive route must fail fast"); + .expect_err("a permanently busy interactive route fails at the turn deadline"); assert!(error.to_string().contains("capacity unavailable")); - assert_eq!(started.elapsed(), INTERACTIVE_CAPACITY_WAIT_TIMEOUT); + let waited = started.elapsed(); + assert!( + waited > std::time::Duration::from_millis(150) + && waited <= std::time::Duration::from_millis(200), + "{waited:?}" + ); assert!( reporter .buffer() diff --git a/crates/openplotva-app/src/history_summary.rs b/crates/openplotva-app/src/history_summary.rs index e69317a5..6ce51f1e 100644 --- a/crates/openplotva-app/src/history_summary.rs +++ b/crates/openplotva-app/src/history_summary.rs @@ -49,9 +49,6 @@ const OPENROUTER_MODEL_PREFIX: &str = "openrouter/"; const OPENROUTER_CHAT_COMPLETIONS_URL: &str = "https://openrouter.ai/api/v1/chat/completions"; /// Stage-one chunks summarized at the same time. const STAGE_ONE_CONCURRENCY: usize = 3; -/// Summaries queue for the GPU2 pool they share with memory work this long before -/// the chain falls back. -pub const HISTORY_SUMMARY_PRIMARY_SLOT_WAIT: Duration = Duration::from_secs(60); /// Concrete app history-summary service used by the dialog toolbox runtime. pub type AppHistorySummaryService = @@ -1824,7 +1821,6 @@ mod tests { dialog_aifarm_pool_models: Some("pool-a".to_owned()), dialog_aifarm_pool_base_urls: Some("http://pool-a.test".to_owned()), dialog_aifarm_pool_api_key: Some("pool-token".to_owned()), - dialog_aifarm_pool_primary_capacity_wait_ms: Some("250".to_owned()), ..openplotva_config::RawConfig::default() }) .expect("config"); diff --git a/crates/openplotva-app/src/lib.rs b/crates/openplotva-app/src/lib.rs index dc8522ca..28a36cbf 100644 --- a/crates/openplotva-app/src/lib.rs +++ b/crates/openplotva-app/src/lib.rs @@ -12579,7 +12579,6 @@ async fn start_runtime_workers( Arc::clone(&router_pools), ) .with_openrouter_free_gate(Arc::clone(&openrouter_free_gate)) - .with_primary_slot_wait(history_summary::HISTORY_SUMMARY_PRIMARY_SLOT_WAIT) .with_reporter(routing_event_reporter.clone()), ), ) as Arc); @@ -12596,7 +12595,6 @@ async fn start_runtime_workers( Arc::clone(&router_pools), ) .with_openrouter_free_gate(Arc::clone(&openrouter_free_gate)) - .with_primary_slot_wait(youtube::YOUTUBE_PRIMARY_SLOT_WAIT) .with_reporter(routing_event_reporter.clone()), ) { Ok(Some(summarizer)) => { @@ -13116,7 +13114,6 @@ async fn start_runtime_workers( Arc::clone(&router_pools), ) .with_openrouter_free_gate(Arc::clone(&openrouter_free_gate)) - .with_primary_slot_wait(media::MEDIA_OPTIMIZER_PRIMARY_SLOT_WAIT) .with_reporter(routing_event_reporter.clone()), config, Arc::clone(&prompt_store), diff --git a/crates/openplotva-app/src/media.rs b/crates/openplotva-app/src/media.rs index 76b22b68..f99625db 100644 --- a/crates/openplotva-app/src/media.rs +++ b/crates/openplotva-app/src/media.rs @@ -20,10 +20,6 @@ use crate::runtime_gemini_cache::resolve_google_ai_key; const OPENROUTER_MODEL_PREFIX: &str = "openrouter/"; const OPENROUTER_CHAT_COMPLETIONS_URL: &str = "https://openrouter.ai/api/v1/chat/completions"; -/// The farm model's two slots are shared with the dialog and are often both -/// busy for a few seconds; a prompt rewrite waits this long for one before it -/// takes a fallback route. -pub const MEDIA_OPTIMIZER_PRIMARY_SLOT_WAIT: Duration = Duration::from_secs(15); /// Boxed image-prompt optimizer future. pub type ImagePromptOptimizeFuture<'a> = @@ -996,7 +992,6 @@ mod tests { " http://secondary-a.test/v1, http://secondary-b.test ".to_owned(), ), dialog_aifarm_pool_api_key: Some("pool-key".to_owned()), - dialog_aifarm_pool_primary_capacity_wait_ms: Some("750".to_owned()), ..openplotva_config::RawConfig::default() }) .expect("config"); diff --git a/crates/openplotva-app/src/memory_runtime.rs b/crates/openplotva-app/src/memory_runtime.rs index 4952f2c2..18de9ffe 100644 --- a/crates/openplotva-app/src/memory_runtime.rs +++ b/crates/openplotva-app/src/memory_runtime.rs @@ -5602,7 +5602,6 @@ mod tests { "http://pool-a.test/v1, http://pool-b.test".to_owned(), ), dialog_aifarm_pool_api_key: Some("pool-token".to_owned()), - dialog_aifarm_pool_primary_capacity_wait_ms: Some("250".to_owned()), ..openplotva_config::RawConfig::default() }) .expect("config"); diff --git a/crates/openplotva-app/src/youtube.rs b/crates/openplotva-app/src/youtube.rs index 45f31da7..4927e592 100644 --- a/crates/openplotva-app/src/youtube.rs +++ b/crates/openplotva-app/src/youtube.rs @@ -41,9 +41,6 @@ const OPENROUTER_MODEL_PREFIX: &str = "openrouter/"; const OPENROUTER_CHAT_COMPLETIONS_URL: &str = "https://openrouter.ai/api/v1/chat/completions"; const YOUTUBE_SUMMARY_TEMPERATURE: f64 = 0.3; const YOUTUBE_TRANSCRIPT_TIMEOUT: Duration = Duration::from_secs(45); -/// The farm route shares its GPU pool with background memory work, so a summary -/// waits this long for a slot before it takes a fallback route. -pub const YOUTUBE_PRIMARY_SLOT_WAIT: Duration = Duration::from_secs(60); type BoxedError = Box; diff --git a/crates/openplotva-config/src/lib.rs b/crates/openplotva-config/src/lib.rs index f545fb48..64f0353f 100644 --- a/crates/openplotva-config/src/lib.rs +++ b/crates/openplotva-config/src/lib.rs @@ -76,9 +76,6 @@ pub const DEFAULT_SETTINGS_REQUIRE_INIT_DATA: bool = false; pub const DEFAULT_RUNTIME_API_ENABLED: bool = true; pub const DEFAULT_RUNTIME_API_HOST: &str = "127.0.0.1"; -/// Default `DIALOG_AIFARM_POOL_PRIMARY_CAPACITY_WAIT_MS`: how long the dialog -/// router waits for a busy primary pool before falling back. -pub const DEFAULT_DIALOG_PRIMARY_CAPACITY_WAIT_MS: i32 = 12_000; /// One corrected sample before the dialog router tries the next model. /// Repeated invalid samples must leave time for fallback and delivery. pub const DEFAULT_DIALOG_MODEL_OUTPUT_RETRIES: i32 = 1; @@ -901,7 +898,6 @@ pub struct DialogConfig { pub aifarm_pool_models: Vec, pub aifarm_pool_base_urls: Vec, pub aifarm_pool_api_key: String, - pub aifarm_pool_primary_capacity_wait_ms: i32, /// How many extra samples the same model gets after a validator rejects its /// output, before the router walks to another provider, from /// `DIALOG_MODEL_OUTPUT_RETRIES`. @@ -1398,10 +1394,6 @@ pub struct RawConfig { pub dialog_aifarm_pool_base_urls: Option, /// `DIALOG_AIFARM_POOL_API_KEY`. pub dialog_aifarm_pool_api_key: Option, - /// `DIALOG_AIFARM_POOL_PRIMARY_CAPACITY_WAIT_MS`: how long the dialog - /// router waits for a busy primary-role pool slot before it falls back to - /// the next candidate. Default 12000. - pub dialog_aifarm_pool_primary_capacity_wait_ms: Option, /// `DIALOG_MODEL_OUTPUT_RETRIES`: extra samples from the same model after a /// validator rejects its output, before the router walks the chain. Default 4. pub dialog_model_output_retries: Option, @@ -2645,11 +2637,6 @@ impl AppConfig { aifarm_pool_models: dialog_aifarm_pool_models, aifarm_pool_base_urls: dialog_aifarm_pool_base_urls, aifarm_pool_api_key: raw.dialog_aifarm_pool_api_key.unwrap_or_default(), - aifarm_pool_primary_capacity_wait_ms: parse_i32( - "DIALOG_AIFARM_POOL_PRIMARY_CAPACITY_WAIT_MS", - raw.dialog_aifarm_pool_primary_capacity_wait_ms, - DEFAULT_DIALOG_PRIMARY_CAPACITY_WAIT_MS, - )?, model_output_retries: parse_i32( "DIALOG_MODEL_OUTPUT_RETRIES", raw.dialog_model_output_retries, @@ -3312,9 +3299,6 @@ impl RawConfig { dialog_aifarm_pool_models: env("DIALOG_AIFARM_POOL_MODELS"), dialog_aifarm_pool_base_urls: env("DIALOG_AIFARM_POOL_BASE_URLS"), dialog_aifarm_pool_api_key: env("DIALOG_AIFARM_POOL_API_KEY"), - dialog_aifarm_pool_primary_capacity_wait_ms: env( - "DIALOG_AIFARM_POOL_PRIMARY_CAPACITY_WAIT_MS", - ), dialog_model_output_retries: env("DIALOG_MODEL_OUTPUT_RETRIES"), dialog_turn_budget_secs: env("DIALOG_TURN_BUDGET_SECS"), dialog_turn_max_queue_age_secs: env("DIALOG_TURN_MAX_QUEUE_AGE_SECS"), @@ -4121,10 +4105,6 @@ mod tests { assert_eq!(config.llm.dialog.task_timeout_seconds, 720); assert_eq!(config.llm.dialog.aifarm_capacity_wait_seconds, 60); assert_eq!(config.llm.dialog.aifarm_capacity_poll_seconds, 1); - assert_eq!( - config.llm.dialog.aifarm_pool_primary_capacity_wait_ms, - crate::DEFAULT_DIALOG_PRIMARY_CAPACITY_WAIT_MS - ); assert_eq!( config.llm.dialog.model_output_retries, crate::DEFAULT_DIALOG_MODEL_OUTPUT_RETRIES diff --git a/docs/CODEBASE_MAP.md b/docs/CODEBASE_MAP.md index 45cbe0d4..14787aea 100644 --- a/docs/CODEBASE_MAP.md +++ b/docs/CODEBASE_MAP.md @@ -391,11 +391,12 @@ classifier (typed-first, ASCII-fold message fallback; mirrors Go rules 1:1). **`trace.rs`**: `LlmCallObserver` + process singleton + task-local `LlmRunScope`. **`router/`**: `RoutingTable` (immutable snapshot) published atomically via `RouterHandle` (ArcSwap); `BreakerSet` (closed/open/half-open + -windowed error-rate, survives reloads); `PoolRegistry` (resize-tolerant -concurrency cells — deliberately not a Semaphore); `policy::select_chain` +windowed error-rate, survives reloads); `PoolRegistry` (lane pools: resize-tolerant +cells with a per-pool priority queue and direct lane handoff — deliberately not a +Semaphore); `policy::select_chain` (Efraimidis–Spirakis weighted permutation; dead-target share redistributes -proportionally; returns untruncated chain so executor can skip pool-busy -without hiding free candidates); `TriggerState` (engaged set + capacity-until). +proportionally; returns untruncated chain so executor can pass over a candidate +without hiding the rest); `TriggerState` (engaged set + capacity-until). No `sqlx`, no `tracing`, no `futures` — policy is pure/sync. **Reply finalization** *(2026-07-06, PR #17)*: `aifarm::extract_final_answer` and `gemini::gemini_final_answer` both delegate to `openplotva_dialog::finalize_dialog_reply`; suppression variants map to retryable errors (`AifarmDialogError::FinalAnswer{ContextLeak,Pathological,ProtocolOnly}` / gemini `ProviderProtocolError` with a granular reason string). aifarm also keeps a separate `ReasoningBudgetExhausted` (detected on `finish_reason=="length"`); gemini runs a second `sanitize_genkit_final_answer` after finalization that can re-suppress. #### `openplotva-storage` @@ -580,14 +581,19 @@ flowchart TB Walk --> Budget{budget + deadline ok?} Budget -- no --> Exhaust[all_attempts_exhausted
event + admin page] Budget -- yes --> Live{breaker live?} - Live -- open --> Skip[busy_skip / dead] - Live -- live --> Pool{pool try_acquire} - Pool -- full --> Wait[wait_for_release
excluded from wall_clock] + Live -- open --> Probe{lane free?} + Probe -- no --> Skip[pass over] + Probe -- yes --> Exec + Live -- live --> Pool{lane try_acquire} + Pool -- full --> Forecast{forecast within
deadline − reserve?} + Forecast -- no --> Skip + Forecast -- yes --> Wait[wait in lane queue
excluded from wall_clock] + Wait -- lane handed --> Exec + Wait -- budget ends --> Skip Pool -- acquired --> Exec[execute attempt
under remaining timeout] Exec -- Ok --> Succ[breaker.record_success
release permit → return] Exec -- retryable Err --> Fail[breaker.record_failure
release permit → loop] Exec -- terminal Err --> Surf[surface Attempt error] - Wait --> Walk Skip --> Walk ``` @@ -693,9 +699,8 @@ cascades, and the audit-preserving Maple cutover). Virtual-message subsystem - **`ledger.rs` VALUES bug**: `SQL_INSERT_TURN_OUTCOMES_PREFIX` stops at the column list — `push_values` emits `VALUES`. A previous trailing `VALUES` produced `VALUES VALUES` and silently broke every insert. Regression test pins exactly one `VALUES`. - **Provider canonicalization** (`runtime_llm.rs`): the observer overrides the stamped provider from the **model name** (`vram.cloud/*`→`vram-cloud`, `vibethinker-3b`→`aifarm-llamacpp-gpu2`, `maple-preview-2bit-mlx`→`aifarm-maple`) so historical backfill and forward classification cannot drift. Rollback-only legacy ids stay classifiable but are not active route targets. - **Maple request compatibility** (`aifarm.rs`): an `mlx` runtime hint strips unsupported DRY controls before Discovery serialization while retaining `repeat_penalty` and float `top_k`; migration 180 keeps Maple and Bonsai as distinct rows so routing-event history remains truthful. -- **Capacity pools are not semaphores** (`router/capacity.rs`): the custom `PoolCell` survives live resize without losing in-flight counts (tokio Semaphore shrink needs `forget_permits` debt). `apply()` rewrites `max` in place; outstanding permits hold their own `Arc`. -- **Capacity-slot waits are excluded from the retry wall**: pool-busy skips don't consume hops or trip breakers; `wait_for_release` is budgeted separately. Without this split, a busy pool would burn the entire retry budget on one candidate. -- **Busy primary pools are waited for, not skipped** *(2026-09-14)*: a `Role::Primary` candidate whose pool is full is waited for up to `DIALOG_AIFARM_POOL_PRIMARY_CAPACITY_WAIT_MS` (default 12 s; `RoutedAttemptWalker::with_primary_slot_wait`, capped by the caller deadline, charged to `total_slot_wait`, `primary_slot_wait` routing event) before the fallback tail runs; fallback/overflow candidates still skip instantly. Evidence: with the dialog router's old 1 s all-busy wait and instant primary skips, Gemma sat at 0.45–0.59 slot occupancy while ~750 turns/day were shed to the tool-less genkit path; the isolated 3090 replay showed the GPU is prefill-bound (~25 req/min at 11.5k tokens) and that raising vLLM `max-num-seqs` buys +5–7% throughput for ×4–×9 TPOT, so queueing on the primary is the lever, not more sequences. +- **Lane pools are not semaphores** (`router/capacity.rs`): the custom `PoolCell` survives live resize without losing in-flight counts (tokio Semaphore shrink needs `forget_permits` debt). `apply()` rewrites `max` in place and serves waiters of a grown pool; outstanding permits hold their own `Arc`. A released lane is handed to the head waiter (`workflows.lane_priority`, then arrival order) through a oneshot; a permit sent to a waiter that gave up releases itself on drop. +- **Fallback on failure or budget, not on load** *(2026-10-12, design `docs/superpowers/specs/2026-10-12-backend-lanes-design.md`)*: the walker waits in a busy candidate's lane queue until `deadline − workflows.fallback_reserve_ms` (no reserve for the last candidate; no deadline = no limit) and passes it over only when its breaker is open or the forecast wait (`waiters ahead / width × mean hold`) exceeds that budget (`lane_bypass` event). Lane waits (`lane_wait` event) don't consume hops, trip breakers or count against the retry wall. Evidence for queueing on the primary: with instant primary skips Gemma sat at 0.45–0.59 slot occupancy while ~750 turns/day were shed to the tool-less fallback; the 3090 replay showed the GPU is prefill-bound, so queueing on the primary is the lever, not more sequences. - **Breaker fairness ≥30s slice**: a deadline-cut charges the breaker only if `remaining ≥ 30s` — a fallback handed the last seconds of a turn must not open its circuit in lockstep with a degraded primary. - **WhiteCircle wrapper is a mandatory passthrough**: must implement `as_chat_step()→Some(self)` or the session engine silently falls back to the legacy loop. Audit fires once per session on `iteration ≤ 1`. - **Two distinct Telegram KDFs** (`openplotva-web`): Login Widget uses `secret = SHA256(bot_token)`; WebApp initData uses `secret = HMAC_SHA256("WebAppData", bot_token)`. Do not confuse. From 9fd25fa666abea556ae79ce50812fbb40cd3d5b2 Mon Sep 17 00:00:00 2001 From: Valeriy Selitskiy <239034+iamwavecut@users.noreply.github.com> Date: Mon, 12 Oct 2026 21:49:23 +0200 Subject: [PATCH 08/18] Queue memory embeddings and redactions at background priority --- crates/openplotva-app/src/embedder.rs | 26 ++++++- crates/openplotva-app/src/lib.rs | 78 +++++++++++---------- crates/openplotva-app/src/memory_runtime.rs | 1 + 3 files changed, 66 insertions(+), 39 deletions(-) diff --git a/crates/openplotva-app/src/embedder.rs b/crates/openplotva-app/src/embedder.rs index 800a5d4b..8790e459 100644 --- a/crates/openplotva-app/src/embedder.rs +++ b/crates/openplotva-app/src/embedder.rs @@ -18,6 +18,7 @@ use openplotva_llm::aifarm::{ DiscoveryJobResponse, }; use openplotva_llm::retry::FailureReason; +use openplotva_llm::router::LanePriority; use openplotva_storage::pg_embedding_vector; use crate::memory_runtime::{ @@ -497,6 +498,7 @@ impl EmbeddingProvider for DiscoveryEmbedderClient { pub struct RoutedDiscoveryEmbedder { walker: RoutedAttemptWalker, base_config: DiscoveryEmbedderConfig, + lane_priority: Option, } impl std::fmt::Debug for RoutedDiscoveryEmbedder { @@ -515,8 +517,16 @@ impl RoutedDiscoveryEmbedder { Self { walker, base_config, + lane_priority: None, } } + + /// Queue this embedder's requests at `priority` instead of the workflow default. + #[must_use] + pub fn with_lane_priority(mut self, priority: LanePriority) -> Self { + self.lane_priority = Some(priority); + self + } } impl EmbeddingProvider for RoutedDiscoveryEmbedder { @@ -536,7 +546,7 @@ impl EmbeddingProvider for RoutedDiscoveryEmbedder { let result = self .walker .run( - embedding_context(), + embedding_context(self.lane_priority), move |attempt| { let text = text.clone(); let task_description = task_description.clone(); @@ -575,7 +585,7 @@ impl EmbeddingProvider for RoutedDiscoveryEmbedder { let result = self .walker .run( - embedding_context(), + embedding_context(self.lane_priority), move |attempt| { let texts = texts.clone(); let task_description = task_description.clone(); @@ -601,9 +611,10 @@ impl EmbeddingProvider for RoutedDiscoveryEmbedder { } } -fn embedding_context() -> RoutedRequestContext { +fn embedding_context(lane_priority: Option) -> RoutedRequestContext { RoutedRequestContext { workflow_key: "embedding".to_owned(), + lane_priority, ..RoutedRequestContext::default() } } @@ -775,6 +786,15 @@ mod tests { use super::*; use openplotva_llm::aifarm::{DiscoveryJob, DiscoveryJobResult}; + #[test] + fn memory_writes_embed_at_background_priority() { + assert_eq!(embedding_context(None).lane_priority, None); + assert_eq!( + embedding_context(Some(LanePriority::Background)).lane_priority, + Some(LanePriority::Background) + ); + } + /// Live round-trip against a reachable Discovery + embedder. Ignored by /// default; run with `EMBEDDER_E2E_DISCOVERY_URL=http://127.0.0.1:50051 /// cargo test -p openplotva-app -- --ignored embed_roundtrip_through_discovery` diff --git a/crates/openplotva-app/src/lib.rs b/crates/openplotva-app/src/lib.rs index 28a36cbf..a0e4c86d 100644 --- a/crates/openplotva-app/src/lib.rs +++ b/crates/openplotva-app/src/lib.rs @@ -11328,25 +11328,28 @@ async fn start_runtime_workers( } if config.memory.enabled && config.bot.key.is_none() { - let memory_write_embedder = Some(embedder::RoutedDiscoveryEmbedder::new( - routed_attempts::RoutedAttemptWalker::new( - Arc::clone(&router_handle), - Arc::clone(&router_breakers), - Arc::clone(&router_triggers), - Arc::clone(&router_pools), + let memory_write_embedder = Some( + embedder::RoutedDiscoveryEmbedder::new( + routed_attempts::RoutedAttemptWalker::new( + Arc::clone(&router_handle), + Arc::clone(&router_breakers), + Arc::clone(&router_triggers), + Arc::clone(&router_pools), + ) + .with_openrouter_free_gate(Arc::clone(&openrouter_free_gate)) + .with_reporter(routing_event_reporter.clone()), + embedder::DiscoveryEmbedderConfig { + base_url: config.llm.discovery.base_url.clone(), + service_name: config.memory.embedder_service_name.clone(), + endpoint_name: config.memory.embedder_endpoint_name.clone(), + request_timeout: memory_runtime::EMBEDDER_DEFAULT_TIMEOUT, + task_timeout: memory_runtime::EMBEDDER_DEFAULT_TIMEOUT, + poll_interval: Duration::from_millis(100), + capacity_wait: Duration::from_secs(2), + }, ) - .with_openrouter_free_gate(Arc::clone(&openrouter_free_gate)) - .with_reporter(routing_event_reporter.clone()), - embedder::DiscoveryEmbedderConfig { - base_url: config.llm.discovery.base_url.clone(), - service_name: config.memory.embedder_service_name.clone(), - endpoint_name: config.memory.embedder_endpoint_name.clone(), - request_timeout: memory_runtime::EMBEDDER_DEFAULT_TIMEOUT, - task_timeout: memory_runtime::EMBEDDER_DEFAULT_TIMEOUT, - poll_interval: Duration::from_millis(100), - capacity_wait: Duration::from_secs(2), - }, - )); + .with_lane_priority(openplotva_llm::router::LanePriority::Background), + ); let memory_extractor = memory_runtime::routed_memory_extractor_from_app_config( config, routed_attempts::RoutedAttemptWalker::new( @@ -12123,25 +12126,28 @@ async fn start_runtime_workers( let mut shared_taskman_worker_counts = BTreeMap::new(); shared_taskman_worker_counts.insert(openplotva_taskman::CONTROL_QUEUE_NAME.to_owned(), 1); if config.memory.enabled { - let memory_write_embedder = Some(embedder::RoutedDiscoveryEmbedder::new( - routed_attempts::RoutedAttemptWalker::new( - Arc::clone(&router_handle), - Arc::clone(&router_breakers), - Arc::clone(&router_triggers), - Arc::clone(&router_pools), + let memory_write_embedder = Some( + embedder::RoutedDiscoveryEmbedder::new( + routed_attempts::RoutedAttemptWalker::new( + Arc::clone(&router_handle), + Arc::clone(&router_breakers), + Arc::clone(&router_triggers), + Arc::clone(&router_pools), + ) + .with_openrouter_free_gate(Arc::clone(&openrouter_free_gate)) + .with_reporter(routing_event_reporter.clone()), + embedder::DiscoveryEmbedderConfig { + base_url: config.llm.discovery.base_url.clone(), + service_name: config.memory.embedder_service_name.clone(), + endpoint_name: config.memory.embedder_endpoint_name.clone(), + request_timeout: memory_runtime::EMBEDDER_DEFAULT_TIMEOUT, + task_timeout: memory_runtime::EMBEDDER_DEFAULT_TIMEOUT, + poll_interval: Duration::from_millis(100), + capacity_wait: Duration::from_secs(2), + }, ) - .with_openrouter_free_gate(Arc::clone(&openrouter_free_gate)) - .with_reporter(routing_event_reporter.clone()), - embedder::DiscoveryEmbedderConfig { - base_url: config.llm.discovery.base_url.clone(), - service_name: config.memory.embedder_service_name.clone(), - endpoint_name: config.memory.embedder_endpoint_name.clone(), - request_timeout: memory_runtime::EMBEDDER_DEFAULT_TIMEOUT, - task_timeout: memory_runtime::EMBEDDER_DEFAULT_TIMEOUT, - poll_interval: Duration::from_millis(100), - capacity_wait: Duration::from_secs(2), - }, - )); + .with_lane_priority(openplotva_llm::router::LanePriority::Background), + ); let memory_scheduler_store = memory_store.clone(); let memory_scheduler_queue = Arc::clone(&task_queue_for_updates); let memory_scheduler_config = diff --git a/crates/openplotva-app/src/memory_runtime.rs b/crates/openplotva-app/src/memory_runtime.rs index 18de9ffe..da7dcf5a 100644 --- a/crates/openplotva-app/src/memory_runtime.rs +++ b/crates/openplotva-app/src/memory_runtime.rs @@ -4052,6 +4052,7 @@ impl TextRedactor for RoutedDiscoveryRedactor { RoutedRequestContext { workflow_key: "redaction".to_owned(), queue_name: Some(MEMORY_CONSOLIDATION_QUEUE_NAME.to_owned()), + lane_priority: Some(openplotva_llm::router::LanePriority::Background), suppress_all_attempts_exhausted_admin_report: true, suppressed_all_attempts_exhausted_reason: Some( "redaction_fail_open".to_owned(), From 73c9a1c716f4d1a58dab69e5427d9814cb458cd9 Mon Sep 17 00:00:00 2001 From: Valeriy Selitskiy <239034+iamwavecut@users.noreply.github.com> Date: Mon, 12 Oct 2026 21:51:51 +0200 Subject: [PATCH 09/18] Name lane pools by device and purpose and pool GPU1, embedder and privacy filter --- crates/openplotva-app/src/model_routing.rs | 6 +- crates/openplotva-storage/src/lib.rs | 29 ++++++++++ migrations/200_backend_lane_pools.down.sql | 15 +++++ migrations/200_backend_lane_pools.up.sql | 64 ++++++++++++++++++++++ 4 files changed, 111 insertions(+), 3 deletions(-) create mode 100644 migrations/200_backend_lane_pools.down.sql create mode 100644 migrations/200_backend_lane_pools.up.sql diff --git a/crates/openplotva-app/src/model_routing.rs b/crates/openplotva-app/src/model_routing.rs index 34e07fcf..7daadfba 100644 --- a/crates/openplotva-app/src/model_routing.rs +++ b/crates/openplotva-app/src/model_routing.rs @@ -1813,7 +1813,7 @@ const MEMORY_SUBJECT_MERGE_CASCADE_KEY: &str = "llm.routing.memory_subject_merge const DIALOG_FALLBACK_CASCADE_KEY: &str = "llm.routing.dialog_fallback_cascade_v1"; const MEMORY_EXTRACTION_WORKFLOW: &str = "memory_extraction"; const MEMORY_SUBJECT_MERGE_WORKFLOW: &str = "memory_subject_merge"; -const MEMORY_GPU2_POOL: &str = "aifarm-gpu2-qwen27b"; +const MEMORY_GPU2_POOL: &str = "aifarm-gpu2-utility"; #[derive(Clone, Copy, Debug, Eq, PartialEq)] struct MemoryRoutingCascadeTarget { @@ -3080,7 +3080,7 @@ mod tests { ('memory_subject_merge', 'chat', FALSE, 3, 60000, TRUE); \ INSERT INTO llm_capacity_pools \ (id, name, max_concurrency, description) VALUES \ - (7, 'aifarm-gpu2-qwen27b', 1, 'shared test pool'); \ + (7, 'aifarm-gpu2-utility', 1, 'shared test pool'); \ INSERT INTO llm_providers \ (id, name, kind, protocol, enabled) VALUES \ (2, 'aifarm-llamacpp-gpu2', 'chat', 'openai_compat', TRUE), \ @@ -3113,7 +3113,7 @@ mod tests { .fetch_one(&pool) .await?; let pool_count: i64 = sqlx::query_scalar( - "SELECT count(*) FROM llm_capacity_pools WHERE name = 'aifarm-gpu2-qwen27b'", + "SELECT count(*) FROM llm_capacity_pools WHERE name = 'aifarm-gpu2-utility'", ) .fetch_one(&pool) .await?; diff --git a/crates/openplotva-storage/src/lib.rs b/crates/openplotva-storage/src/lib.rs index 24fa2528..30ceb32c 100644 --- a/crates/openplotva-storage/src/lib.rs +++ b/crates/openplotva-storage/src/lib.rs @@ -10066,6 +10066,35 @@ mod tests { assert!(DOWN.contains("WHERE config ? 'textual_tool_calls_v1_previous'")); } + #[test] + fn backend_lane_pools_are_renamed_attached_and_reversible() { + const UP: &str = include_str!("../../../migrations/200_backend_lane_pools.up.sql"); + const DOWN: &str = include_str!("../../../migrations/200_backend_lane_pools.down.sql"); + + for value in [ + "('aifarm-vllm-gpu0', 'aifarm-gpu0-dialog'", + "('aifarm-gpu2-qwen27b', 'aifarm-gpu2-utility'", + "('aifarm-draw-gpu1', 'aifarm-gpu1-media'", + "('aifarm-embedder', 8,", + "('aifarm-gpu1-privacy', 4,", + "('aifarm-asr', 'aifarm-gpu1-media')", + "('aifarm-embed', 'aifarm-embedder')", + "('privacy-filter', 'aifarm-gpu1-privacy')", + "'backend_lanes_v1_previous_name', pool.name", + "'backend_lanes_v1_previous_pool_id', model.pool_id", + "created_by_backend_lanes_v1", + ] { + assert!(UP.contains(value), "missing lane pool contract {value}"); + } + assert!(!UP.contains("max_concurrency =")); + assert!(!UP.contains("workflow_assignments")); + assert!(DOWN.contains("WHERE config ? 'backend_lanes_v1_previous_pool_id'")); + assert!(DOWN.contains( + "DELETE FROM llm_capacity_pools WHERE config ? 'created_by_backend_lanes_v1'" + )); + assert!(DOWN.contains("WHERE config ? 'backend_lanes_v1_previous_name'")); + } + #[test] fn workflow_lane_settings_default_and_are_reversible() { const UP: &str = include_str!("../../../migrations/199_workflow_lane_settings.up.sql"); diff --git a/migrations/200_backend_lane_pools.down.sql b/migrations/200_backend_lane_pools.down.sql new file mode 100644 index 00000000..703a9fc4 --- /dev/null +++ b/migrations/200_backend_lane_pools.down.sql @@ -0,0 +1,15 @@ +LOCK TABLE llm_capacity_pools, provider_models IN SHARE ROW EXCLUSIVE MODE; + +UPDATE provider_models +SET pool_id = (config->>'backend_lanes_v1_previous_pool_id')::BIGINT, + config = config - 'backend_lanes_v1_previous_pool_id' +WHERE config ? 'backend_lanes_v1_previous_pool_id'; + +DELETE FROM llm_capacity_pools WHERE config ? 'created_by_backend_lanes_v1'; + +UPDATE llm_capacity_pools +SET name = config->>'backend_lanes_v1_previous_name', + description = config->>'backend_lanes_v1_previous_description', + config = config - 'backend_lanes_v1_previous_name' - 'backend_lanes_v1_previous_description', + updated_at = now() +WHERE config ? 'backend_lanes_v1_previous_name'; diff --git a/migrations/200_backend_lane_pools.up.sql b/migrations/200_backend_lane_pools.up.sql new file mode 100644 index 00000000..057ec774 --- /dev/null +++ b/migrations/200_backend_lane_pools.up.sql @@ -0,0 +1,64 @@ +-- Lane pools describe physical resources and are named aifarm--. +-- GPU1's draw, music and speech recognition share one sequential lane under the +-- VRAM arbiter; its privacy filter runs beside that lane. Renamed pools keep their +-- ids, so model rows keep their pool_id. Every change records what it replaced. + +LOCK TABLE llm_capacity_pools, llm_providers, provider_models IN SHARE ROW EXCLUSIVE MODE; + +DO $$ +BEGIN + IF EXISTS ( + SELECT 1 FROM llm_capacity_pools + WHERE config ?| ARRAY['backend_lanes_v1_previous_name', 'created_by_backend_lanes_v1'] + ) OR EXISTS ( + SELECT 1 FROM provider_models WHERE config ? 'backend_lanes_v1_previous_pool_id' + ) THEN + RAISE EXCEPTION 'backend lanes marker collision'; + END IF; +END $$; + +UPDATE llm_capacity_pools AS pool +SET config = pool.config || jsonb_build_object( + 'backend_lanes_v1_previous_name', pool.name, + 'backend_lanes_v1_previous_description', pool.description + ), + name = renamed.new_name, + description = renamed.description, + updated_at = now() +FROM ( + VALUES + ('aifarm-vllm-gpu0', 'aifarm-gpu0-dialog', 'GPU0 dialog model: eight lanes.'), + ('aifarm-gpu2-qwen27b', 'aifarm-gpu2-utility', 'GPU2 utility model: eight lanes.'), + ('aifarm-draw-gpu1', 'aifarm-gpu1-media', + 'GPU1 draw, music and speech recognition: one lane under the VRAM arbiter.') +) AS renamed (old_name, new_name, description) +WHERE pool.name = renamed.old_name + AND NOT EXISTS (SELECT 1 FROM llm_capacity_pools WHERE name = renamed.new_name); + +INSERT INTO llm_capacity_pools (name, max_concurrency, description, config) +SELECT new_pool.name, new_pool.width, new_pool.description, + '{"created_by_backend_lanes_v1": true}'::jsonb +FROM ( + VALUES + ('aifarm-embedder', 8, 'Embedder: eight lanes.'), + ('aifarm-gpu1-privacy', 4, 'GPU1 privacy filter: four lanes beside the media lane.') +) AS new_pool (name, width, description) +WHERE NOT EXISTS (SELECT 1 FROM llm_capacity_pools WHERE name = new_pool.name); + +UPDATE provider_models AS model +SET config = model.config || jsonb_build_object( + 'backend_lanes_v1_previous_pool_id', model.pool_id + ), + pool_id = target_pool.id +FROM llm_providers AS provider, + ( + VALUES + ('aifarm-asr', 'aifarm-gpu1-media'), + ('aifarm-embed', 'aifarm-embedder'), + ('privacy-filter', 'aifarm-gpu1-privacy') + ) AS mapping (provider_name, pool_name), + llm_capacity_pools AS target_pool +WHERE provider.id = model.provider_id + AND provider.name = mapping.provider_name + AND target_pool.name = mapping.pool_name + AND model.pool_id IS DISTINCT FROM target_pool.id; From e569b64eb8f69e0d678fbf0bb942bbd33cf2391d Mon Sep 17 00:00:00 2001 From: Valeriy Selitskiy <239034+iamwavecut@users.noreply.github.com> Date: Mon, 12 Oct 2026 21:57:06 +0200 Subject: [PATCH 10/18] Size dialog and memory workers from their primary lane pool --- .env.example | 10 +- crates/openplotva-app/src/dialog_workers.rs | 34 ++- crates/openplotva-app/src/lib.rs | 254 +++++++++---------- crates/openplotva-app/src/model_routing.rs | 116 +++++++-- crates/openplotva-app/src/task_queue.rs | 3 +- crates/openplotva-config/src/lib.rs | 53 ++-- crates/openplotva-llm/src/router/capacity.rs | 170 +++++++------ crates/openplotva-llm/src/router/mod.rs | 2 +- 8 files changed, 374 insertions(+), 268 deletions(-) diff --git a/.env.example b/.env.example index 2dc43d64..14278cbd 100644 --- a/.env.example +++ b/.env.example @@ -105,18 +105,22 @@ # PERSISTENT_QUEUE_JOB_MESSAGE_CLEANUP_MINUTES=30 # PERSISTENT_QUEUE_CONTROL_WORKERS=2 # PERSISTENT_QUEUE_TEXT_WORKERS=4 -# Deprecated: only the fallback worker count when the routing table has no -# dialog route; the live count derives from the capacity pools. +# Deprecated: only the fallback worker count when the dialog primary has no +# lane pool; the live count is the primary's lanes times the per-lane count. # PERSISTENT_QUEUE_DIALOG_AIFARM_WORKERS=2 # PERSISTENT_QUEUE_DIALOG_WORKERS_CAP=24 -# PERSISTENT_QUEUE_DIALOG_UNPOOLED_SHARE=2 +# PERSISTENT_QUEUE_DIALOG_WORKERS_PER_LANE=2 # PERSISTENT_QUEUE_DIALOG_AIFARM_FALLBACK_HIGH_WATERMARK=30 # PERSISTENT_QUEUE_DIALOG_AIFARM_FALLBACK_LOW_WATERMARK=20 # PERSISTENT_QUEUE_DIALOG_AIFARM_FALLBACK_POLL_INTERVAL_SECONDS=1 # PERSISTENT_QUEUE_IMAGE_REGULAR_WORKERS=1 # PERSISTENT_QUEUE_IMAGE_VIP_WORKERS=1 # PERSISTENT_QUEUE_MUSIC_VIP_WORKERS=1 +# Minimum memory workers; the live count is the memory primary's lanes times +# the per-lane count, up to the cap. 0 disables memory workers. # PERSISTENT_QUEUE_MEMORY_CONSOLIDATION_WORKERS=1 +# PERSISTENT_QUEUE_MEMORY_WORKERS_CAP=32 +# PERSISTENT_QUEUE_MEMORY_WORKERS_PER_LANE=2 # PERSISTENT_QUEUE_PLACEHOLDER_CLEANUP_INTERVAL_SECONDS=3600 # PERSISTENT_QUEUE_PLACEHOLDER_MAX_AGE_SECONDS=7200 # LLM_JOB_MAX_ATTEMPTS=5 diff --git a/crates/openplotva-app/src/dialog_workers.rs b/crates/openplotva-app/src/dialog_workers.rs index 34d43edf..8db6030a 100644 --- a/crates/openplotva-app/src/dialog_workers.rs +++ b/crates/openplotva-app/src/dialog_workers.rs @@ -1,9 +1,9 @@ -//! Dynamic dialog worker supervision. +//! Dynamic worker supervision for the dialog and memory-consolidation fleets. //! -//! The dialog worker count is derived from the routing table (the summed slot -//! budgets of the dialog workflow's capacity pools) and can change on any -//! admin routing edit. The supervisor watches the desired count and spawns or -//! retires workers to match: a retired worker receives a oneshot signal that +//! Each fleet's worker count is derived from the routing table (the lanes of +//! its workflow's primary lane pool) and can change on any admin routing edit. +//! The supervisor watches the desired count and spawns or retires workers to +//! match: a retired worker receives a oneshot signal that //! resolves its stop future, so it finishes its current job and exits at the //! next loop tick — no job is ever abandoned mid-turn. On global runtime stop //! the supervisor joins every worker it owns before exiting, preserving the @@ -20,14 +20,14 @@ use tokio::task::JoinHandle; /// the desired count even without a scale change. const SUPERVISOR_REAP_INTERVAL: Duration = Duration::from_secs(30); -/// Builds one dialog worker task. The worker must exit soon after `retire` +/// Builds one worker task. The worker must exit soon after `retire` /// resolves (fired or dropped) or the global runtime stop flips; the concrete /// spawner composes both into the worker loop's stop future. -pub trait DialogWorkerSpawner: Send + Sync + 'static { +pub trait WorkerSpawner: Send + Sync + 'static { fn spawn(&self, index: usize, retire: oneshot::Receiver<()>) -> JoinHandle<()>; } -impl DialogWorkerSpawner for F +impl WorkerSpawner for F where F: Fn(usize, oneshot::Receiver<()>) -> JoinHandle<()> + Send + Sync + 'static, { @@ -63,8 +63,8 @@ impl WorkerGauge { /// Run the supervisor until the global stop flips (or every signal source /// closes). Reconciles the worker set with `desired_rx` on every change and on /// a periodic reap tick, then joins all owned workers before returning. -pub fn spawn_dialog_worker_supervisor( - spawner: Arc, +pub fn spawn_worker_supervisor( + spawner: Arc, mut desired_rx: watch::Receiver, gauge: Arc, mut global_stop: watch::Receiver, @@ -141,7 +141,7 @@ mod tests { indices: Arc>>, } - impl DialogWorkerSpawner for FakeSpawner { + impl WorkerSpawner for FakeSpawner { fn spawn(&self, index: usize, retire: oneshot::Receiver<()>) -> JoinHandle<()> { self.spawned.fetch_add(1, Ordering::Relaxed); self.indices @@ -180,12 +180,8 @@ mod tests { indices: Arc::new(Mutex::new(Vec::new())), }); - let supervisor = spawn_dialog_worker_supervisor( - spawner, - desired_rx, - Arc::clone(&gauge), - stop_rx.clone(), - ); + let supervisor = + spawn_worker_supervisor(spawner, desired_rx, Arc::clone(&gauge), stop_rx.clone()); settle().await; assert_eq!(spawned.load(Ordering::Relaxed), 2); assert_eq!(gauge.desired(), 2); @@ -218,14 +214,14 @@ mod tests { struct CrashingSpawner { spawned: Arc, } - impl DialogWorkerSpawner for CrashingSpawner { + impl WorkerSpawner for CrashingSpawner { fn spawn(&self, _index: usize, _retire: oneshot::Receiver<()>) -> JoinHandle<()> { self.spawned.fetch_add(1, Ordering::Relaxed); tokio::spawn(async {}) } } - let supervisor = spawn_dialog_worker_supervisor( + let supervisor = spawn_worker_supervisor( Arc::new(CrashingSpawner { spawned: Arc::clone(&spawned), }), diff --git a/crates/openplotva-app/src/lib.rs b/crates/openplotva-app/src/lib.rs index a0e4c86d..12ab1135 100644 --- a/crates/openplotva-app/src/lib.rs +++ b/crates/openplotva-app/src/lib.rs @@ -9533,28 +9533,6 @@ fn dialog_memory_context_enabled(config: &AppConfig) -> bool { config.memory.enabled } -/// Memory-consolidation parallelism. Runs are claimed with -/// `FOR UPDATE SKIP LOCKED` under a lease (one run = one chat window), so -/// parallel workers safely chew different chats. The effective count follows -/// the workflow's capacity pools — a pooled route (e.g. the 16-slot -/// vram.cloud pool) invites parallel workers up to the cap, while an unpooled -/// route keeps the configured count (historically one). `0` disables workers. -fn effective_memory_consolidation_workers( - configured_workers: i32, - pool_derived: Option, - cap: i32, -) -> i32 { - if configured_workers <= 0 { - return 0; - } - let cap = cap.max(1); - let derived = pool_derived - .and_then(|derived| i32::try_from(derived).ok()) - .unwrap_or(1) - .clamp(1, cap); - configured_workers.max(derived) -} - fn admin_queue_config_from_app_config(config: &AppConfig) -> admin::AdminQueueCommandConfig { let queue = &config.persistent_queue; admin::AdminQueueCommandConfig { @@ -12125,6 +12103,7 @@ async fn start_runtime_workers( )); let mut shared_taskman_worker_counts = BTreeMap::new(); shared_taskman_worker_counts.insert(openplotva_taskman::CONTROL_QUEUE_NAME.to_owned(), 1); + let mut memory_worker_scale = None; if config.memory.enabled { let memory_write_embedder = Some( embedder::RoutedDiscoveryEmbedder::new( @@ -12173,76 +12152,106 @@ async fn start_runtime_workers( memory_scheduler, ); - let configured_memory_worker_count = config.persistent_queue.memory_consolidation_workers; - let memory_workers_cap = config.persistent_queue.memory_workers_cap.max(1); - let derived_memory_workers = openplotva_llm::router::derived_worker_count( - &router_handle.snapshot(), - memory_runtime::MEMORY_EXTRACTION_WORKFLOW_KEY, - 1, - u32::try_from(memory_workers_cap).unwrap_or(1), - ); - let memory_worker_count = effective_memory_consolidation_workers( - configured_memory_worker_count, - derived_memory_workers, - memory_workers_cap, - ); - if memory_worker_count != configured_memory_worker_count { + // Memory workers follow the memory primary's lanes; the configured + // count is the floor and `0` disables them. + let configured_memory_worker_count = + u32::try_from(config.persistent_queue.memory_consolidation_workers.max(0)).unwrap_or(0); + let memory_worker_count = if configured_memory_worker_count == 0 { + 0 + } else { + let (scale, desired) = tokio::sync::watch::channel(configured_memory_worker_count); + let lanes = model_routing::LaneWorkerScale { + workflow_key: memory_runtime::MEMORY_EXTRACTION_WORKFLOW_KEY, + per_lane: u32::try_from(config.persistent_queue.memory_workers_per_lane.max(1)) + .unwrap_or(1), + floor: configured_memory_worker_count, + cap: u32::try_from(config.persistent_queue.memory_workers_cap.max(1)).unwrap_or(1), + scale, + }; + let initial = lanes + .desired(&router_handle.snapshot()) + .unwrap_or(configured_memory_worker_count); + let _ = lanes.scale.send(initial); tracing::info!( configured_workers = configured_memory_worker_count, - effective_workers = memory_worker_count, - cap = memory_workers_cap, - "memory-consolidation worker count derived from capacity pools" - ); - } - for index in 0..memory_worker_count { - let memory_worker_extractor = memory_runtime::routed_memory_extractor_from_app_config( - config, - routed_attempts::RoutedAttemptWalker::new( - Arc::clone(&router_handle), - Arc::clone(&router_breakers), - Arc::clone(&router_triggers), - Arc::clone(&router_pools), - ) - .with_openrouter_free_gate(Arc::clone(&openrouter_free_gate)) - .with_reporter(routing_event_reporter.clone()), - ); - let memory_worker_store = memory_store.clone(); - let memory_worker_embedder = memory_write_embedder.clone(); - let memory_worker_queue = Arc::clone(&task_queue_for_updates); - let memory_worker_id = format!( - "{}-{index}", - memory_runtime::MEMORY_CONSOLIDATION_JOB_WORKER_PREFIX + initial_workers = initial, + cap = lanes.cap, + "memory-consolidation worker scale derived from lane pools" ); - let memory_worker_config = memory_runtime::MemoryConsolidationQueueWorkerConfig { - process: memory_runtime::memory_service_worker_config_from_memory_config( - &config.memory, - ) - .process, - worker_id: memory_worker_id.clone(), - interval: memory_runtime::MEMORY_CONSOLIDATION_JOB_POLL_INTERVAL, - pipeline_target: usize::try_from(memory_worker_count).unwrap_or(1), - }; - let memory_worker_stop = worker_stops.subscribe(RuntimeWorkerPhase::Processor); - let memory_worker_name = memory_worker_id.clone(); - let memory_worker = tokio::spawn(async move { - let report = memory_runtime::run_memory_consolidation_taskman_worker_until( - memory_worker_queue.as_ref(), - &memory_worker_extractor, - &memory_worker_store, - memory_worker_embedder.as_ref(), - memory_worker_config, - wait_for_runtime_stop(memory_worker_stop), - ) - .await; + memory_worker_scale = Some(lanes); - tracing::info!(?report, worker_id = %memory_worker_id, "memory-consolidation taskman worker stopped"); - }); + let memory_worker_spawner = { + let extractor = memory_runtime::routed_memory_extractor_from_app_config( + config, + routed_attempts::RoutedAttemptWalker::new( + Arc::clone(&router_handle), + Arc::clone(&router_breakers), + Arc::clone(&router_triggers), + Arc::clone(&router_pools), + ) + .with_openrouter_free_gate(Arc::clone(&openrouter_free_gate)) + .with_reporter(routing_event_reporter.clone()), + ); + let store = memory_store.clone(); + let embedder = memory_write_embedder.clone(); + let queue = Arc::clone(&task_queue_for_updates); + let process = + memory_runtime::memory_service_worker_config_from_memory_config(&config.memory) + .process; + let desired = desired.clone(); + let stop_rx = worker_stops.subscribe(RuntimeWorkerPhase::Processor); + move |index: usize, retire: tokio::sync::oneshot::Receiver<()>| { + let worker_extractor = extractor.clone(); + let worker_store = store.clone(); + let worker_embedder = embedder.clone(); + let worker_queue = Arc::clone(&queue); + let worker_id = format!( + "{}-{index}", + memory_runtime::MEMORY_CONSOLIDATION_JOB_WORKER_PREFIX + ); + let worker_config = memory_runtime::MemoryConsolidationQueueWorkerConfig { + process: process.clone(), + worker_id: worker_id.clone(), + interval: memory_runtime::MEMORY_CONSOLIDATION_JOB_POLL_INTERVAL, + pipeline_target: usize::try_from(*desired.borrow()).unwrap_or(1), + }; + let worker_stop = stop_rx.clone(); + tokio::spawn(async move { + // A retired worker finishes its current run and exits + // at the next loop tick, exactly like a runtime stop. + let stop_future = async move { + tokio::select! { + () = wait_for_runtime_stop(worker_stop) => {} + _ = retire => {} + } + }; + let report = memory_runtime::run_memory_consolidation_taskman_worker_until( + worker_queue.as_ref(), + &worker_extractor, + &worker_store, + worker_embedder.as_ref(), + worker_config, + stop_future, + ) + .await; + + tracing::info!(?report, %worker_id, "memory-consolidation taskman worker stopped"); + }) + } + }; + let memory_supervisor = dialog_workers::spawn_worker_supervisor( + Arc::new(memory_worker_spawner), + desired, + Arc::new(dialog_workers::WorkerGauge::new()), + worker_stops.subscribe(RuntimeWorkerPhase::Processor), + ); worker_registry.register( - memory_worker_name, + "memory-consolidation-worker-supervisor", RuntimeWorkerPhase::Processor, - memory_worker, + memory_supervisor, ); - } + initial + }; if config.memory.subject_merge_enabled { let merge_merger = memory_runtime::routed_subject_merger_from_app_config( config, @@ -12298,7 +12307,7 @@ async fn start_runtime_workers( workers.memory_restart_trigger = Some(Arc::clone(&memory_restart_trigger)); shared_taskman_worker_counts.insert( openplotva_taskman::MEMORY_CONSOLIDATION_QUEUE_NAME.to_owned(), - memory_worker_count, + i32::try_from(memory_worker_count).unwrap_or(i32::MAX), ); } let payment_store_for_updates = Arc::new(payment_store.clone()); @@ -12962,25 +12971,29 @@ async fn start_runtime_workers( let dialog_terminal_reaction_emoji = config.llm.dialog.terminal_reaction_emoji.clone(); let dialog_terminal_signal_max_age_secs = config.llm.dialog.terminal_signal_max_age_secs; - // The dialog worker count derives from the capacity pools of the - // dialog workflow's models; the semaphores in the walker are the - // real limit, so extra workers just wait for a slot. The env knob - // is only the fallback when no dialog route exists. + // The dialog worker count follows the dialog primary's lanes; the + // lane queues in the walker are the real limit, so extra workers + // wait in the primary's queue during their model rounds and run + // their non-model phases meanwhile. The env knob is only the + // fallback when the primary has no lane pool. let dialog_worker_fallback = u32::try_from(config.persistent_queue.dialog_aifarm_workers.max(0)).unwrap_or(0); let dialog_workers_cap = u32::try_from(config.persistent_queue.dialog_workers_cap.max(1)).unwrap_or(1); - let dialog_unpooled_share = - u32::try_from(config.persistent_queue.dialog_unpooled_share.max(0)).unwrap_or(0); - let initial_dialog_workers = openplotva_llm::router::derived_worker_count( - &router_handle.snapshot(), - "dialog", - dialog_unpooled_share, - dialog_workers_cap, - ) - .unwrap_or(dialog_worker_fallback); let (dialog_scale_tx, dialog_scale_rx) = - tokio::sync::watch::channel(initial_dialog_workers); + tokio::sync::watch::channel(dialog_worker_fallback); + let dialog_lanes = model_routing::LaneWorkerScale { + workflow_key: "dialog", + per_lane: u32::try_from(config.persistent_queue.dialog_workers_per_lane.max(1)) + .unwrap_or(1), + floor: 0, + cap: dialog_workers_cap, + scale: dialog_scale_tx, + }; + let initial_dialog_workers = dialog_lanes + .desired(&router_handle.snapshot()) + .unwrap_or(dialog_worker_fallback); + let _ = dialog_lanes.scale.send(initial_dialog_workers); let dialog_worker_gauge = Arc::new(dialog_workers::WorkerGauge::new()); let dialog_worker_spawner = { @@ -13056,7 +13069,7 @@ async fn start_runtime_workers( }) } }; - let dialog_supervisor = dialog_workers::spawn_dialog_worker_supervisor( + let dialog_supervisor = dialog_workers::spawn_worker_supervisor( Arc::new(dialog_worker_spawner), dialog_scale_rx, Arc::clone(&dialog_worker_gauge), @@ -13071,20 +13084,19 @@ async fn start_runtime_workers( workers.router_runtime = Some(Arc::new(model_routing::RouterRuntime { handle: Arc::clone(&router_handle), pools: Arc::clone(&router_pools), - dialog_scale: Some(dialog_scale_tx), - dialog_unpooled_share, - dialog_worker_cap: dialog_workers_cap, + dialog_workers: Some(dialog_lanes), + memory_workers: memory_worker_scale.take(), })); tracing::info!( initial_dialog_workers, cap = dialog_workers_cap, fallback = dialog_worker_fallback, - "dialog worker scale derived from capacity pools" + "dialog worker scale derived from lane pools" ); readiness_checks.push(ReadinessCheck::ok( "dialog_jobs", format!( - "Started dialog worker supervisor with {initial_dialog_workers} pool-derived workers (cap {dialog_workers_cap})" + "Started dialog worker supervisor with {initial_dialog_workers} lane-derived workers (cap {dialog_workers_cap})" ), )); if initial_dialog_workers > 0 { @@ -15248,15 +15260,14 @@ mod tests { apply_private_chat_response_defaults, authenticate_settings_init_data, build_deputy_display_name, chat_list_title, chat_member_can_manage_settings, chat_member_record_from_upsert, configure_telegram_bot_commands, current_unix_timestamp, - delete_webhook_on_shutdown_if_enabled, dialog_memory_context_enabled, - effective_memory_consolidation_workers, found, go_dispatcher_config, new_settings_response, - normalize_deputy_ids, parse_admin_non_negative_i32, parse_admin_optional_bool, - parse_admin_positive_i32, parse_deputy_candidates_limit, parse_deputy_update_request, - parse_memory_limit, parse_optional_i32, parse_settings_get_access, - parse_settings_memory_access, parse_settings_update_request, parse_shield_limit, - routing_json_config_patch, runtime_api_graphql_snapshot, - runtime_redis_prefix_groups_from_keys, runtime_redis_value_from_bytes, - send_dispatcher_work_item_with_transport, + delete_webhook_on_shutdown_if_enabled, dialog_memory_context_enabled, found, + go_dispatcher_config, new_settings_response, normalize_deputy_ids, + parse_admin_non_negative_i32, parse_admin_optional_bool, parse_admin_positive_i32, + parse_deputy_candidates_limit, parse_deputy_update_request, parse_memory_limit, + parse_optional_i32, parse_settings_get_access, parse_settings_memory_access, + parse_settings_update_request, parse_shield_limit, routing_json_config_patch, + runtime_api_graphql_snapshot, runtime_redis_prefix_groups_from_keys, + runtime_redis_value_from_bytes, send_dispatcher_work_item_with_transport, send_dispatcher_work_item_with_transport_and_history, settings_chat_display, settings_chat_full_info_type_name, settings_chat_state_from_full_info, settings_get_response, settings_reply_flags, shield_history_tail_messages_from_config, @@ -15377,19 +15388,6 @@ mod tests { Ok(()) } - #[test] - fn memory_consolidation_worker_count_follows_pools() { - // Explicit zero disables workers regardless of pools. - assert_eq!(effective_memory_consolidation_workers(0, Some(16), 8), 0); - // Unpooled route (no derivation) keeps the configured count. - assert_eq!(effective_memory_consolidation_workers(1, None, 8), 1); - // A pooled route scales up to the cap. - assert_eq!(effective_memory_consolidation_workers(1, Some(16), 8), 8); - assert_eq!(effective_memory_consolidation_workers(1, Some(3), 8), 3); - // An explicit operator count above the derivation wins. - assert_eq!(effective_memory_consolidation_workers(12, Some(16), 8), 12); - } - #[test] fn runtime_snapshot_and_shield_options_preserve_go_shield_config() -> Result<(), Box> { diff --git a/crates/openplotva-app/src/model_routing.rs b/crates/openplotva-app/src/model_routing.rs index 7daadfba..5023db95 100644 --- a/crates/openplotva-app/src/model_routing.rs +++ b/crates/openplotva-app/src/model_routing.rs @@ -2137,33 +2137,54 @@ async fn mark_app_setting_in_transaction( Ok(()) } +/// A worker fleet whose size follows its workflow's primary lane pool. +pub struct LaneWorkerScale { + pub workflow_key: &'static str, + pub per_lane: u32, + /// Configured minimum; the lane-derived count never drops below it. + pub floor: u32, + pub cap: u32, + pub scale: tokio::sync::watch::Sender, +} + +impl LaneWorkerScale { + /// Worker count for `table`; `None` when the primary has no lane pool, so + /// the running count stays. + #[must_use] + pub fn desired(&self, table: &RoutingTable) -> Option { + openplotva_llm::router::lane_worker_count(table, self.workflow_key, self.per_lane, self.cap) + .map(|workers| workers.max(self.floor)) + } + + fn apply(&self, table: &RoutingTable) { + if let Some(desired) = self.desired(table) { + let _ = self.scale.send(desired); + } + } +} + /// Everything the runtime rebuilds when the routing configuration changes: -/// the published table, the capacity-pool registry, and (once wired) the -/// derived dialog worker scale. Breakers/triggers are deliberately absent — -/// their state must survive a reload untouched. +/// the published table, the lane pool registry, and the dialog and memory +/// worker scales. Breakers/triggers are deliberately absent — their state must +/// survive a reload untouched. pub struct RouterRuntime { pub handle: Arc, pub pools: Arc, - pub dialog_scale: Option>, - pub dialog_unpooled_share: u32, - pub dialog_worker_cap: u32, + pub dialog_workers: Option, + pub memory_workers: Option, } impl RouterRuntime { - /// Reconcile the live pool registry and worker scale with the currently + /// Reconcile the live pool registry and worker scales with the currently /// published table. pub fn apply_published_table(&self) { let table = self.handle.snapshot(); self.pools.apply(&table.pool_specs()); - if let Some(scale) = &self.dialog_scale - && let Some(desired) = openplotva_llm::router::derived_worker_count( - &table, - "dialog", - self.dialog_unpooled_share, - self.dialog_worker_cap, - ) + for workers in [&self.dialog_workers, &self.memory_workers] + .into_iter() + .flatten() { - let _ = scale.send(desired); + workers.apply(&table); } } } @@ -2547,7 +2568,7 @@ fn build_route( mod tests { use openplotva_llm::router::{BreakerLiveness, BreakerSet, TriggerState, select}; use openplotva_storage::llm_routing::{ - AssignmentRecord, ModelRecord, ProviderRecord, RoutingSnapshot, TriggerRecord, + AssignmentRecord, ModelRecord, PoolRecord, ProviderRecord, RoutingSnapshot, TriggerRecord, WorkflowRecord, }; use rand::SeedableRng; @@ -2730,6 +2751,69 @@ mod tests { } } + fn dialog_snapshot_with_gpu0_lanes(width: i32) -> RoutingSnapshot { + let mut snapshot = dialog_snapshot(); + snapshot.models[0].pool_id = Some(1); + snapshot.pools.push(PoolRecord { + id: 1, + name: "aifarm-gpu0-dialog".to_owned(), + max_concurrency: Some(width), + description: None, + config: json!({}), + }); + snapshot + } + + fn worker_scale(floor: u32) -> (LaneWorkerScale, tokio::sync::watch::Receiver) { + let (scale, receiver) = tokio::sync::watch::channel(0); + ( + LaneWorkerScale { + workflow_key: "dialog", + per_lane: 2, + floor, + cap: 24, + scale, + }, + receiver, + ) + } + + #[test] + fn worker_scales_follow_the_primary_lane_pool_and_keep_their_floor() { + let table = build_routing_table(&dialog_snapshot_with_gpu0_lanes(8)); + assert_eq!(worker_scale(0).0.desired(&table), Some(16)); + assert_eq!(worker_scale(20).0.desired(&table), Some(20)); + + let unpooled = build_routing_table(&dialog_snapshot()); + assert_eq!( + worker_scale(3).0.desired(&unpooled), + None, + "a primary without lanes keeps the running count" + ); + } + + #[test] + fn a_routing_reload_resizes_dialog_and_memory_workers() { + let handle = RouterHandle::new(build_routing_table(&dialog_snapshot_with_gpu0_lanes(8))); + let (dialog, dialog_rx) = worker_scale(0); + let (memory, memory_rx) = worker_scale(3); + let runtime = RouterRuntime { + handle: Arc::clone(&handle), + pools: Arc::new(openplotva_llm::router::PoolRegistry::new()), + dialog_workers: Some(dialog), + memory_workers: Some(memory), + }; + + runtime.apply_published_table(); + assert_eq!(*dialog_rx.borrow(), 16); + assert_eq!(*memory_rx.borrow(), 16); + + handle.store(build_routing_table(&dialog_snapshot_with_gpu0_lanes(1))); + runtime.apply_published_table(); + assert_eq!(*dialog_rx.borrow(), 2); + assert_eq!(*memory_rx.borrow(), 3, "the configured floor holds"); + } + #[test] fn workflow_lane_settings_reach_the_route() { let mut snapshot = dialog_snapshot(); diff --git a/crates/openplotva-app/src/task_queue.rs b/crates/openplotva-app/src/task_queue.rs index 2cf99b5a..b8e9aeea 100644 --- a/crates/openplotva-app/src/task_queue.rs +++ b/crates/openplotva-app/src/task_queue.rs @@ -1353,7 +1353,7 @@ mod tests { text_workers: 4, dialog_aifarm_workers: 2, dialog_workers_cap: 24, - dialog_unpooled_share: 2, + dialog_workers_per_lane: 2, dialog_aifarm_fallback_high_watermark: 30, dialog_aifarm_fallback_low_watermark: 20, dialog_aifarm_fallback_poll_interval_seconds: 1, @@ -1362,6 +1362,7 @@ mod tests { music_vip_workers: 1, memory_consolidation_workers: 1, memory_workers_cap: 8, + memory_workers_per_lane: 2, placeholder_cleanup_interval_seconds: 3600, placeholder_max_age_seconds: 7200, llm_job_max_attempts: 5, diff --git a/crates/openplotva-config/src/lib.rs b/crates/openplotva-config/src/lib.rs index 64f0353f..8fa67769 100644 --- a/crates/openplotva-config/src/lib.rs +++ b/crates/openplotva-config/src/lib.rs @@ -219,9 +219,14 @@ pub const DEFAULT_PERSISTENT_QUEUE_DIALOG_AIFARM_WORKERS: i32 = 2; pub const DEFAULT_PERSISTENT_QUEUE_DIALOG_WORKERS_CAP: i32 = 24; -pub const DEFAULT_PERSISTENT_QUEUE_DIALOG_UNPOOLED_SHARE: i32 = 2; - -pub const DEFAULT_PERSISTENT_QUEUE_MEMORY_WORKERS_CAP: i32 = 4; +/// Dialog workers per lane of the dialog primary's lane pool: one runs its +/// model round while the other runs the turn's non-model phases. +pub const DEFAULT_PERSISTENT_QUEUE_DIALOG_WORKERS_PER_LANE: i32 = 2; + +pub const DEFAULT_PERSISTENT_QUEUE_MEMORY_WORKERS_CAP: i32 = 32; +/// Memory workers per lane of the memory primary's lane pool: a memory run +/// spends most of its time outside the model. +pub const DEFAULT_PERSISTENT_QUEUE_MEMORY_WORKERS_PER_LANE: i32 = 2; /// Matches the privacy-filter `max_concurrent_jobs` registered in Discovery. pub const DEFAULT_MEMORY_REDACTION_CONCURRENCY: i32 = 4; @@ -677,14 +682,13 @@ pub struct PersistentQueueConfig { pub control_workers: i32, /// Worker count for the `text` queue. pub text_workers: i32, - /// Worker count for the `dialog-aifarm` queue. With capacity pools active - /// this is only the fallback when the routing table has no dialog route. + /// Worker count for the `dialog-aifarm` queue. With lane pools active this + /// is only the fallback when the dialog primary has no lane pool. pub dialog_aifarm_workers: i32, - /// Upper bound for the pool-derived dialog worker count. + /// Upper bound for the lane-derived dialog worker count. pub dialog_workers_cap: i32, - /// Slot contribution of an unpooled/unlimited-pool dialog model when - /// deriving the worker count. - pub dialog_unpooled_share: i32, + /// Dialog workers per lane of the dialog primary's lane pool. + pub dialog_workers_per_lane: i32, pub dialog_aifarm_fallback_high_watermark: i32, pub dialog_aifarm_fallback_low_watermark: i32, pub dialog_aifarm_fallback_poll_interval_seconds: i32, @@ -696,8 +700,10 @@ pub struct PersistentQueueConfig { pub music_vip_workers: i32, /// Worker count for the `memory-consolidation` queue. pub memory_consolidation_workers: i32, - /// Upper bound for the pool-derived memory-consolidation worker count. + /// Upper bound for the lane-derived memory-consolidation worker count. pub memory_workers_cap: i32, + /// Memory-consolidation workers per lane of the memory primary's lane pool. + pub memory_workers_per_lane: i32, /// Placeholder cleanup interval in seconds, from `PERSISTENT_QUEUE_PLACEHOLDER_CLEANUP_INTERVAL_SECONDS`. pub placeholder_cleanup_interval_seconds: i32, /// Placeholder max age in seconds, from `PERSISTENT_QUEUE_PLACEHOLDER_MAX_AGE_SECONDS`. @@ -1264,7 +1270,8 @@ pub struct RawConfig { /// `PERSISTENT_QUEUE_DIALOG_AIFARM_WORKERS`. pub persistent_queue_dialog_aifarm_workers: Option, pub persistent_queue_dialog_workers_cap: Option, - pub persistent_queue_dialog_unpooled_share: Option, + /// `PERSISTENT_QUEUE_DIALOG_WORKERS_PER_LANE`. + pub persistent_queue_dialog_workers_per_lane: Option, /// `PERSISTENT_QUEUE_DIALOG_AIFARM_FALLBACK_HIGH_WATERMARK`. pub persistent_queue_dialog_aifarm_fallback_high_watermark: Option, /// `PERSISTENT_QUEUE_DIALOG_AIFARM_FALLBACK_LOW_WATERMARK`. @@ -1280,6 +1287,8 @@ pub struct RawConfig { /// `PERSISTENT_QUEUE_MEMORY_CONSOLIDATION_WORKERS`. pub persistent_queue_memory_consolidation_workers: Option, pub persistent_queue_memory_workers_cap: Option, + /// `PERSISTENT_QUEUE_MEMORY_WORKERS_PER_LANE`. + pub persistent_queue_memory_workers_per_lane: Option, /// `PERSISTENT_QUEUE_PLACEHOLDER_CLEANUP_INTERVAL_SECONDS`. pub persistent_queue_placeholder_cleanup_interval_seconds: Option, /// `PERSISTENT_QUEUE_PLACEHOLDER_MAX_AGE_SECONDS`. @@ -1946,10 +1955,10 @@ impl AppConfig { raw.persistent_queue_dialog_workers_cap, DEFAULT_PERSISTENT_QUEUE_DIALOG_WORKERS_CAP, )?, - dialog_unpooled_share: parse_i32( - "PERSISTENT_QUEUE_DIALOG_UNPOOLED_SHARE", - raw.persistent_queue_dialog_unpooled_share, - DEFAULT_PERSISTENT_QUEUE_DIALOG_UNPOOLED_SHARE, + dialog_workers_per_lane: parse_i32( + "PERSISTENT_QUEUE_DIALOG_WORKERS_PER_LANE", + raw.persistent_queue_dialog_workers_per_lane, + DEFAULT_PERSISTENT_QUEUE_DIALOG_WORKERS_PER_LANE, )?, dialog_aifarm_fallback_high_watermark: parse_i32( "PERSISTENT_QUEUE_DIALOG_AIFARM_FALLBACK_HIGH_WATERMARK", @@ -1991,6 +2000,11 @@ impl AppConfig { raw.persistent_queue_memory_workers_cap, DEFAULT_PERSISTENT_QUEUE_MEMORY_WORKERS_CAP, )?, + memory_workers_per_lane: parse_i32( + "PERSISTENT_QUEUE_MEMORY_WORKERS_PER_LANE", + raw.persistent_queue_memory_workers_per_lane, + DEFAULT_PERSISTENT_QUEUE_MEMORY_WORKERS_PER_LANE, + )?, placeholder_cleanup_interval_seconds: parse_i32( "PERSISTENT_QUEUE_PLACEHOLDER_CLEANUP_INTERVAL_SECONDS", raw.persistent_queue_placeholder_cleanup_interval_seconds, @@ -3221,7 +3235,9 @@ impl RawConfig { persistent_queue_text_workers: env("PERSISTENT_QUEUE_TEXT_WORKERS"), persistent_queue_dialog_aifarm_workers: env("PERSISTENT_QUEUE_DIALOG_AIFARM_WORKERS"), persistent_queue_dialog_workers_cap: env("PERSISTENT_QUEUE_DIALOG_WORKERS_CAP"), - persistent_queue_dialog_unpooled_share: env("PERSISTENT_QUEUE_DIALOG_UNPOOLED_SHARE"), + persistent_queue_dialog_workers_per_lane: env( + "PERSISTENT_QUEUE_DIALOG_WORKERS_PER_LANE", + ), persistent_queue_dialog_aifarm_fallback_high_watermark: env( "PERSISTENT_QUEUE_DIALOG_AIFARM_FALLBACK_HIGH_WATERMARK", ), @@ -3235,6 +3251,9 @@ impl RawConfig { persistent_queue_image_vip_workers: env("PERSISTENT_QUEUE_IMAGE_VIP_WORKERS"), persistent_queue_music_vip_workers: env("PERSISTENT_QUEUE_MUSIC_VIP_WORKERS"), persistent_queue_memory_workers_cap: env("PERSISTENT_QUEUE_MEMORY_WORKERS_CAP"), + persistent_queue_memory_workers_per_lane: env( + "PERSISTENT_QUEUE_MEMORY_WORKERS_PER_LANE", + ), persistent_queue_memory_consolidation_workers: env( "PERSISTENT_QUEUE_MEMORY_CONSOLIDATION_WORKERS", ), @@ -4052,6 +4071,8 @@ mod tests { config.persistent_queue.memory_workers_cap, DEFAULT_PERSISTENT_QUEUE_MEMORY_WORKERS_CAP ); + assert_eq!(config.persistent_queue.memory_workers_per_lane, 2); + assert_eq!(config.persistent_queue.dialog_workers_per_lane, 2); assert_eq!( config.persistent_queue.placeholder_cleanup_interval_seconds, DEFAULT_PERSISTENT_QUEUE_PLACEHOLDER_CLEANUP_INTERVAL_SECONDS diff --git a/crates/openplotva-llm/src/router/capacity.rs b/crates/openplotva-llm/src/router/capacity.rs index 5d6a08fd..29a381d1 100644 --- a/crates/openplotva-llm/src/router/capacity.rs +++ b/crates/openplotva-llm/src/router/capacity.rs @@ -351,61 +351,26 @@ impl PoolRegistry { } } -/// Derive a worker count for a workflow from the capacity of its candidates: -/// the slot budgets of the distinct pools its enabled models attach to, plus -/// `unpooled_share` for every distinct model that is unpooled or in an -/// unlimited pool. Clamped to `1..=cap`. Returns `None` when the workflow has -/// no route (callers keep their configured fallback count). +/// Workers for a workflow: the width of its heaviest primary's lane pool times +/// `per_lane` (at least one per lane), clamped to `1..=cap`. Fallback capacity +/// is not throughput, so the rest of the chain does not count. `None` when the +/// workflow has no route or its primary has no bounded lane pool (callers keep +/// their configured count). #[must_use] -pub fn derived_worker_count( +pub fn lane_worker_count( table: &RoutingTable, workflow_key: &str, - unpooled_share: u32, + per_lane: u32, cap: u32, ) -> Option { - let mut model_ids: Vec = Vec::new(); - let mut seen_route = false; - for vip in [false, true] { - let Some(route) = table.resolve(workflow_key, vip) else { - continue; - }; - seen_route = true; - let candidates = route - .primary - .iter() - .chain(route.fallback_tail.iter()) - .chain(route.canary.iter()) - .chain(route.shadow.iter()) - .chain(route.triggers.iter().map(|trigger| &trigger.engage)); - for candidate in candidates { - if !model_ids.contains(&candidate.model) { - model_ids.push(candidate.model); - } - } - } - if !seen_route { - return None; - } - - let mut counted_pools: Vec = Vec::new(); - let mut total: u32 = 0; - for model_id in model_ids { - let pool_slots = table - .model(model_id) - .and_then(|model| model.pool_id) - .and_then(|pool_id| table.pool(pool_id).map(|spec| (pool_id, spec))); - match pool_slots { - Some((pool_id, spec)) => { - if counted_pools.contains(&pool_id) { - continue; - } - counted_pools.push(pool_id); - total = total.saturating_add(spec.max_concurrency.unwrap_or(unpooled_share)); - } - None => total = total.saturating_add(unpooled_share), - } - } - Some(total.clamp(1, cap.max(1))) + let route = table.resolve(workflow_key, false)?; + let primary = route + .primary + .iter() + .max_by_key(|candidate| candidate.weight)?; + let pool = table.model(primary.model)?.pool_id?; + let width = table.pool(pool)?.max_concurrency?; + Some(width.saturating_mul(per_lane.max(1)).clamp(1, cap.max(1))) } #[cfg(test)] @@ -828,44 +793,87 @@ mod tests { table } + fn with_fallbacks(mut table: RoutingTable, fallbacks: Vec) -> RoutingTable { + let mut route = table.resolve("dialog", false).expect("route").clone(); + route.fallback_tail = fallbacks + .into_iter() + .map(|model| Candidate { + role: Role::Fallback, + ..candidate(model) + }) + .collect(); + table.set_route(Scope::Global, route); + table + } + + fn dialog_chain() -> RoutingTable { + with_fallbacks( + table_with( + vec![ + model_row(10, Some(1)), // GPU0: eight lanes + model_row(20, Some(2)), // GPU2: eight lanes + model_row(30, None), // hosted API + ], + vec![ + PoolSpec { + id: 1, + max_concurrency: Some(8), + }, + PoolSpec { + id: 2, + max_concurrency: Some(8), + }, + ], + vec![10], + ), + vec![20, 30], + ) + } + #[test] - fn derived_worker_count_sums_distinct_pools_and_unpooled_shares() { - let table = table_with( - vec![ - model_row(10, Some(1)), // aifarm pool of 2 - model_row(20, Some(2)), // vram pool of 16 - model_row(21, Some(2)), // same pool: counted once - model_row(30, None), // unpooled: unpooled_share - ], - vec![ - PoolSpec { - id: 1, - max_concurrency: Some(2), - }, - PoolSpec { - id: 2, - max_concurrency: Some(16), - }, - ], - vec![10, 20, 21, 30], + fn workers_follow_the_primary_lane_pool_only() { + let table = dialog_chain(); + assert_eq!(lane_worker_count(&table, "dialog", 2, 24), Some(16)); + assert_eq!( + lane_worker_count(&table, "dialog", 2, 10), + Some(10), + "cap clamps the count" ); - - assert_eq!(derived_worker_count(&table, "dialog", 2, 24), Some(20)); assert_eq!( - derived_worker_count(&table, "dialog", 2, 8), + lane_worker_count(&table, "dialog", 0, 24), Some(8), - "cap clamps the derived count" + "at least one worker per lane" ); assert_eq!( - derived_worker_count(&table, "missing", 2, 24), + lane_worker_count(&table, "missing", 2, 24), None, - "unknown workflow keeps the configured fallback" + "unknown workflow keeps the configured count" ); } #[test] - fn derived_worker_count_treats_unlimited_pool_as_unpooled_share() { - let table = table_with( + fn the_heaviest_primary_decides_the_worker_count() { + let mut table = dialog_chain(); + let mut route = table.resolve("dialog", false).expect("route").clone(); + route.primary = vec![ + Candidate { + weight: 10, + ..candidate(30) + }, + Candidate { + weight: 90, + ..candidate(10) + }, + ]; + table.set_route(Scope::Global, route); + assert_eq!(lane_worker_count(&table, "dialog", 2, 24), Some(16)); + } + + #[test] + fn an_unpooled_or_unlimited_primary_has_no_lane_count() { + let unpooled = table_with(vec![model_row(30, None)], vec![], vec![30]); + assert_eq!(lane_worker_count(&unpooled, "dialog", 2, 24), None); + let unlimited = table_with( vec![model_row(10, Some(1))], vec![PoolSpec { id: 1, @@ -873,12 +881,6 @@ mod tests { }], vec![10], ); - assert_eq!(derived_worker_count(&table, "dialog", 3, 24), Some(3)); - } - - #[test] - fn derived_worker_count_never_returns_zero() { - let table = table_with(vec![], vec![], vec![]); - assert_eq!(derived_worker_count(&table, "dialog", 0, 24), Some(1)); + assert_eq!(lane_worker_count(&unlimited, "dialog", 2, 24), None); } } diff --git a/crates/openplotva-llm/src/router/mod.rs b/crates/openplotva-llm/src/router/mod.rs index 7c92d0e9..d19f648f 100644 --- a/crates/openplotva-llm/src/router/mod.rs +++ b/crates/openplotva-llm/src/router/mod.rs @@ -14,7 +14,7 @@ pub mod triggers; pub use breaker::{BreakerConfig, BreakerLiveness, BreakerSet, BreakerStateView}; pub use capacity::{ - LanePriority, PoolOccupancy, PoolPermit, PoolRegistry, PoolSpec, derived_worker_count, + LanePriority, PoolOccupancy, PoolPermit, PoolRegistry, PoolSpec, lane_worker_count, }; pub use handle::RouterHandle; pub use policy::{Attempt, Liveness, RetryBudget, TriggerView, select, select_chain}; From 2ece9781be106417b794937bbef17a68aebfc6fc Mon Sep 17 00:00:00 2001 From: Valeriy Selitskiy <239034+iamwavecut@users.noreply.github.com> Date: Mon, 12 Oct 2026 22:03:10 +0200 Subject: [PATCH 11/18] Report lane pools that do not match the farm's Discovery limits --- crates/openplotva-app/src/lane_checks.rs | 481 ++++++++++++++++++ crates/openplotva-app/src/lib.rs | 10 + crates/openplotva-app/src/model_routing.rs | 14 +- .../src/runtime_llm_analytics.rs | 6 +- crates/openplotva-llm/src/router/table.rs | 23 + 5 files changed, 528 insertions(+), 6 deletions(-) create mode 100644 crates/openplotva-app/src/lane_checks.rs diff --git a/crates/openplotva-app/src/lane_checks.rs b/crates/openplotva-app/src/lane_checks.rs new file mode 100644 index 00000000..888f45f0 --- /dev/null +++ b/crates/openplotva-app/src/lane_checks.rs @@ -0,0 +1,481 @@ +//! Checks that lane pools match the farm: every routed farm model sits in a +//! lane pool, a Discovery service never splits across pools, and a pool is not +//! wider than the services it spans. Mismatches are reported, never enforced: +//! routing keeps the bot's configured widths. + +use std::collections::{BTreeMap, BTreeSet, HashMap}; + +use openplotva_llm::router::{PoolId, RoutingTable}; + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct LaneConfigIssue { + pub kind: &'static str, + pub pool_id: Option, + pub detail: String, +} + +/// Invariant violations of `table` given each Discovery service's +/// `max_concurrent_jobs`. Services missing from `discovery_widths` are not +/// width-checked. +#[must_use] +pub fn lane_config_issues( + table: &RoutingTable, + discovery_widths: &HashMap, +) -> Vec { + let mut issues = Vec::new(); + let mut pools_by_service: BTreeMap> = BTreeMap::new(); + let mut services_by_pool: BTreeMap> = BTreeMap::new(); + for model in table.routed_models().filter(|model| model.enabled) { + let Some(service) = farm_service(table, model.provider) else { + continue; + }; + match model.pool_id { + Some(pool) => { + pools_by_service + .entry(service.clone()) + .or_default() + .insert(pool); + services_by_pool.entry(pool).or_default().insert(service); + } + None => issues.push(LaneConfigIssue { + kind: "unpooled_farm_model", + pool_id: None, + detail: format!( + "model {} reaches {service} without a lane pool", + model.model_name + ), + }), + } + } + for (service, pools) in &pools_by_service { + if pools.len() > 1 { + issues.push(LaneConfigIssue { + kind: "split_service", + pool_id: None, + detail: format!("{service} is split across lane pools {pools:?}"), + }); + } + } + for (pool, services) in &services_by_pool { + let Some(width) = table.pool(*pool).and_then(|spec| spec.max_concurrency) else { + continue; + }; + let Some(narrowest) = services + .iter() + .filter_map(|service| discovery_widths.get(service).copied()) + .min() + else { + continue; + }; + let mismatch = if services.len() == 1 { + width != narrowest + } else { + width > narrowest + }; + if mismatch { + issues.push(LaneConfigIssue { + kind: "width_mismatch", + pool_id: Some(*pool), + detail: format!( + "lane pool {pool} has {width} lanes; Discovery allows {narrowest} for {services:?}" + ), + }); + } + } + issues +} + +/// Discovery services that enabled, routed models reach. +#[must_use] +pub fn routed_farm_services(table: &RoutingTable) -> Vec { + let services: BTreeSet = table + .routed_models() + .filter(|model| model.enabled) + .filter_map(|model| farm_service(table, model.provider)) + .collect(); + services.into_iter().collect() +} + +fn farm_service(table: &RoutingTable, provider: i64) -> Option { + table + .provider(provider) + .filter(|provider| provider.enabled) + .and_then(|provider| provider.discovery_service_name.as_deref()) + .map(str::trim) + .filter(|service| !service.is_empty()) + .map(str::to_owned) +} + +/// Runs the lane configuration check against the live Discovery widths and +/// records each mismatch as a `lane_config_mismatch` routing event. +#[derive(Clone)] +pub struct LaneCheck { + pub discovery_base_url: String, + pub reporter: Option, +} + +/// Result of one lane configuration check. +#[derive(Debug, Default)] +pub struct LaneCheckReport { + pub issues: Vec, + /// Services whose width could not be read from Discovery. + pub unreachable: Vec, +} + +impl LaneCheckReport { + /// One-line readiness detail. + #[must_use] + pub fn summary(&self) -> String { + let mut summary = if self.issues.is_empty() { + "lane pools match Discovery".to_owned() + } else { + let kinds: BTreeSet<&str> = self.issues.iter().map(|issue| issue.kind).collect(); + format!( + "{} lane config mismatches ({})", + self.issues.len(), + kinds.into_iter().collect::>().join(", ") + ) + }; + if !self.unreachable.is_empty() { + summary.push_str(&format!( + "; widths unchecked for {}", + self.unreachable.join(", ") + )); + } + summary + } +} + +impl LaneCheck { + pub async fn run(&self, table: &RoutingTable) -> LaneCheckReport { + let services = routed_farm_services(table); + let mut fetches = tokio::task::JoinSet::new(); + for service in services { + let client = crate::runtime_llm_analytics::DiscoveryCapacityClient::new( + self.discovery_base_url.clone(), + service.clone(), + ); + fetches.spawn(async move { (service, client.fetch().await) }); + } + let mut widths = HashMap::new(); + let mut unreachable = Vec::new(); + while let Some(fetched) = fetches.join_next().await { + let Ok((service, capacity)) = fetched else { + continue; + }; + match u32::try_from(capacity.max_concurrent_jobs) { + Ok(width) if capacity.error.is_none() && width > 0 => { + widths.insert(service, width); + } + _ => unreachable.push(service), + } + } + unreachable.sort(); + let issues = lane_config_issues(table, &widths); + for issue in &issues { + tracing::warn!(kind = issue.kind, pool_id = ?issue.pool_id, detail = %issue.detail, "lane pool configuration does not match the farm"); + if let Some(reporter) = &self.reporter { + reporter.record(lane_config_mismatch_event(issue)); + } + } + LaneCheckReport { + issues, + unreachable, + } + } +} + +fn lane_config_mismatch_event(issue: &LaneConfigIssue) -> crate::runtime_routing::RoutingEvent { + crate::runtime_routing::RoutingEvent { + severity: "warn".to_owned(), + event_type: "lane_config_mismatch".to_owned(), + workflow_key: "lanes".to_owned(), + provider_id: None, + model_id: None, + queue_name: None, + job_id: None, + chat_id: None, + user_id: None, + thread_id: None, + message_id: None, + dedupe_key: format!( + "lane_config_mismatch:{}:{}", + issue.kind, + issue.pool_id.unwrap_or_default() + ), + summary: issue.detail.clone(), + detail: serde_json::json!({ + "kind": issue.kind, + "pool_id": issue.pool_id, + "detail": issue.detail, + }), + } +} + +#[cfg(test)] +mod tests { + use openplotva_storage::llm_routing::{ + AssignmentRecord, ModelRecord, PoolRecord, ProviderRecord, RoutingSnapshot, WorkflowRecord, + }; + use serde_json::json; + + use super::*; + + fn provider(id: i64, name: &str, kind: &str, service: Option<&str>) -> ProviderRecord { + ProviderRecord { + id, + name: name.to_owned(), + kind: kind.to_owned(), + protocol: None, + runtime_hint: None, + endpoint: None, + discovery_service_name: service.map(str::to_owned), + discovery_endpoint_name: None, + api_key_ref: None, + api_key_encrypted: None, + enabled: true, + config: json!({}), + } + } + + fn model(id: i64, provider_id: i64, capability: &str, pool_id: Option) -> ModelRecord { + ModelRecord { + id, + provider_id, + model_name: format!("model-{id}"), + display_name: None, + base_url: None, + capabilities: vec![capability.to_owned()], + embedding_dim: None, + pool_id, + enabled: true, + config: json!({}), + } + } + + fn workflow(key: &str, kind: &str) -> WorkflowRecord { + WorkflowRecord { + key: key.to_owned(), + kind: kind.to_owned(), + full_routing: false, + retry_max_hops: 2, + retry_wall_ms: 60_000, + enabled: true, + lane_priority: 0, + fallback_reserve_ms: 30_000, + } + } + + fn assignment(id: i64, workflow: &str, role: &str, model_id: i64) -> AssignmentRecord { + AssignmentRecord { + id, + workflow_key: workflow.to_owned(), + scope: "global".to_owned(), + role: role.to_owned(), + provider_model_id: model_id, + weight: Some(100), + fallback_order: Some(0), + canary_percent: None, + enabled: true, + inference_overrides: json!({}), + cb_failure_threshold: 5, + cb_cooldown_ms: 30_000, + } + } + + fn pool(id: i64, name: &str, width: i32) -> PoolRecord { + PoolRecord { + id, + name: name.to_owned(), + max_concurrency: Some(width), + description: None, + config: json!({}), + } + } + + /// GPU0 dialog (llm-openai, 8 lanes) with a hosted fallback, GPU0 vision, + /// and GPU1 draw plus speech recognition on one sequential lane. + fn farm() -> RoutingSnapshot { + RoutingSnapshot { + providers: vec![ + provider(1, "aifarm-vllm-gpu0", "chat", Some("llm-openai")), + provider(2, "aifarm-vision", "vision", Some("llm-openai")), + provider(3, "openrouter", "chat", None), + provider(4, "aifarm-draw", "image", Some("draw-api")), + provider(5, "aifarm-asr", "asr", Some("asr-api")), + ], + models: vec![ + model(10, 1, "chat", Some(1)), + model(20, 2, "vision", Some(1)), + model(30, 3, "chat", None), + model(40, 4, "image", Some(3)), + model(50, 5, "asr", Some(3)), + ], + workflows: vec![ + workflow("dialog", "chat"), + workflow("vision", "vision"), + workflow("image_generation", "image"), + workflow("voice_transcription", "asr"), + ], + assignments: vec![ + assignment(100, "dialog", "primary", 10), + assignment(101, "dialog", "fallback", 30), + assignment(102, "vision", "primary", 20), + assignment(103, "image_generation", "primary", 40), + assignment(104, "voice_transcription", "primary", 50), + ], + triggers: vec![], + pools: vec![ + pool(1, "aifarm-gpu0-dialog", 8), + pool(3, "aifarm-gpu1-media", 1), + ], + } + } + + fn widths() -> HashMap { + HashMap::from([ + ("llm-openai".to_owned(), 8), + ("draw-api".to_owned(), 1), + ("asr-api".to_owned(), 1), + ]) + } + + fn issues(snapshot: &RoutingSnapshot, widths: &HashMap) -> Vec<&'static str> { + let table = crate::model_routing::build_routing_table(snapshot); + lane_config_issues(&table, widths) + .into_iter() + .map(|issue| issue.kind) + .collect() + } + + #[test] + fn matching_lanes_and_a_shared_sequential_lane_are_valid() { + assert!(issues(&farm(), &widths()).is_empty()); + } + + #[test] + fn a_routed_farm_model_without_a_lane_pool_is_reported() { + let mut snapshot = farm(); + snapshot.models[4].pool_id = None; + assert_eq!(issues(&snapshot, &widths()), vec!["unpooled_farm_model"]); + } + + #[test] + fn a_service_split_across_pools_is_reported() { + let mut snapshot = farm(); + snapshot.pools.push(pool(6, "aifarm-gpu2-utility", 8)); + snapshot.models[1].pool_id = Some(6); + assert!(issues(&snapshot, &widths()).contains(&"split_service")); + } + + #[test] + fn a_shared_lane_wider_than_any_of_its_services_is_reported() { + let mut snapshot = farm(); + snapshot.pools[1].max_concurrency = Some(2); + assert_eq!(issues(&snapshot, &widths()), vec!["width_mismatch"]); + } + + #[test] + fn a_single_service_pool_must_match_its_width_exactly() { + let mut snapshot = farm(); + snapshot.pools[0].max_concurrency = Some(4); + assert_eq!(issues(&snapshot, &widths()), vec!["width_mismatch"]); + } + + #[test] + fn unknown_discovery_widths_are_not_checked() { + let mut snapshot = farm(); + snapshot.pools[1].max_concurrency = Some(2); + assert!(issues(&snapshot, &HashMap::new()).is_empty()); + } + + async fn discovery_stub(widths: &'static [(&'static str, u32)]) -> String { + let app = axum::Router::new().route( + "/v1/services/{name}/capacity", + axum::routing::get( + move |axum::extract::Path(name): axum::extract::Path| async move { + match widths.iter().find(|(service, _)| *service == name) { + Some((_, width)) => ( + axum::http::StatusCode::OK, + axum::Json(json!({ + "capacity": {"service_name": name, "max_concurrent_jobs": width} + })), + ), + None => ( + axum::http::StatusCode::NOT_FOUND, + axum::Json(json!({"detail": "service not found"})), + ), + } + }, + ), + ); + let listener = tokio::net::TcpListener::bind("127.0.0.1:0") + .await + .expect("bind"); + let address = listener.local_addr().expect("address"); + tokio::spawn(async move { + let _ = axum::serve(listener, app).await; + }); + format!("http://{address}") + } + + #[tokio::test] + async fn the_check_reads_discovery_widths_and_records_each_mismatch() { + let base_url = discovery_stub(&[("llm-openai", 8), ("draw-api", 1), ("asr-api", 1)]).await; + let mut snapshot = farm(); + snapshot.pools[1].max_concurrency = Some(2); + let reporter = crate::runtime_routing::RoutingEventReporter::new( + crate::runtime_routing::RoutingEventBuffer::new(8), + None, + None, + ); + let check = LaneCheck { + discovery_base_url: base_url, + reporter: Some(reporter.clone()), + }; + + let report = check + .run(&crate::model_routing::build_routing_table(&snapshot)) + .await; + + assert_eq!(report.unreachable, Vec::::new()); + assert_eq!(report.issues.len(), 1); + assert_eq!( + report.summary(), + "1 lane config mismatches (width_mismatch)" + ); + let events = reporter.buffer().routing_events(8); + assert_eq!(events.len(), 1); + assert_eq!(events[0].event_type, "lane_config_mismatch"); + assert_eq!(events[0].severity, "warn"); + } + + #[tokio::test] + async fn services_missing_from_discovery_are_left_unchecked() { + let base_url = discovery_stub(&[("llm-openai", 8)]).await; + let check = LaneCheck { + discovery_base_url: base_url, + reporter: None, + }; + + let report = check + .run(&crate::model_routing::build_routing_table(&farm())) + .await; + + assert!(report.issues.is_empty()); + assert_eq!(report.unreachable, vec!["asr-api", "draw-api"]); + assert_eq!( + report.summary(), + "lane pools match Discovery; widths unchecked for asr-api, draw-api" + ); + } + + #[test] + fn unrouted_and_disabled_models_are_ignored() { + let mut snapshot = farm(); + snapshot.models.push(model(60, 5, "asr", None)); + snapshot.providers[4].enabled = false; + snapshot.models[4].pool_id = None; + assert!(issues(&snapshot, &widths()).is_empty()); + } +} diff --git a/crates/openplotva-app/src/lib.rs b/crates/openplotva-app/src/lib.rs index 12ab1135..033330e6 100644 --- a/crates/openplotva-app/src/lib.rs +++ b/crates/openplotva-app/src/lib.rs @@ -29,6 +29,7 @@ pub mod history_summary; pub mod image_jobs; pub mod ingestion_telemetry; pub mod inline; +mod lane_checks; mod maintenance_api; pub mod media; pub mod members; @@ -10761,6 +10762,14 @@ async fn start_runtime_workers( workers.router_pools = Some(Arc::clone(&router_pools)); workers.openrouter_free_gate = Some(Arc::clone(&openrouter_free_gate)); router_pools.apply(&router_handle.snapshot().pool_specs()); + let lane_check = lane_checks::LaneCheck { + discovery_base_url: config.llm.discovery.base_url.clone(), + reporter: Some(routing_event_reporter.clone()), + }; + readiness_checks.push(ReadinessCheck::ok( + "lane_config", + lane_check.run(&router_handle.snapshot()).await.summary(), + )); { let pool = service_clients.postgres.clone(); let handle = Arc::clone(&router_handle); @@ -13086,6 +13095,7 @@ async fn start_runtime_workers( pools: Arc::clone(&router_pools), dialog_workers: Some(dialog_lanes), memory_workers: memory_worker_scale.take(), + lane_check: Some(lane_check.clone()), })); tracing::info!( initial_dialog_workers, diff --git a/crates/openplotva-app/src/model_routing.rs b/crates/openplotva-app/src/model_routing.rs index 5023db95..c687032e 100644 --- a/crates/openplotva-app/src/model_routing.rs +++ b/crates/openplotva-app/src/model_routing.rs @@ -2164,14 +2164,15 @@ impl LaneWorkerScale { } /// Everything the runtime rebuilds when the routing configuration changes: -/// the published table, the lane pool registry, and the dialog and memory -/// worker scales. Breakers/triggers are deliberately absent — their state must -/// survive a reload untouched. +/// the published table, the lane pool registry, the dialog and memory worker +/// scales, and the lane check against Discovery. Breakers/triggers are +/// deliberately absent — their state must survive a reload untouched. pub struct RouterRuntime { pub handle: Arc, pub pools: Arc, pub dialog_workers: Option, pub memory_workers: Option, + pub(crate) lane_check: Option, } impl RouterRuntime { @@ -2198,6 +2199,12 @@ pub async fn reload_router_runtime( let table = load_routing_table(pool).await?; runtime.handle.store(table); runtime.apply_published_table(); + if let Some(check) = runtime.lane_check.clone() { + let table = runtime.handle.snapshot(); + tokio::spawn(async move { + check.run(&table).await; + }); + } Ok(()) } @@ -2802,6 +2809,7 @@ mod tests { pools: Arc::new(openplotva_llm::router::PoolRegistry::new()), dialog_workers: Some(dialog), memory_workers: Some(memory), + lane_check: None, }; runtime.apply_published_table(); diff --git a/crates/openplotva-app/src/runtime_llm_analytics.rs b/crates/openplotva-app/src/runtime_llm_analytics.rs index 5ad445ed..b672d767 100644 --- a/crates/openplotva-app/src/runtime_llm_analytics.rs +++ b/crates/openplotva-app/src/runtime_llm_analytics.rs @@ -541,14 +541,14 @@ impl PostgresRuntimeLlmAnalyticsReader { } #[derive(Clone)] -struct DiscoveryCapacityClient { +pub(crate) struct DiscoveryCapacityClient { base_url: String, service_name: String, http: reqwest::Client, } impl DiscoveryCapacityClient { - fn new(base_url: String, service_name: String) -> Self { + pub(crate) fn new(base_url: String, service_name: String) -> Self { Self { base_url, service_name, @@ -556,7 +556,7 @@ impl DiscoveryCapacityClient { } } - async fn fetch(&self) -> RuntimeAifarmCapacitySnapshotData { + pub(crate) async fn fetch(&self) -> RuntimeAifarmCapacitySnapshotData { let mut snapshot = RuntimeAifarmCapacitySnapshotData { service: self.service_name.trim().to_owned(), observed_at: format_rfc3339(OffsetDateTime::now_utc()), diff --git a/crates/openplotva-llm/src/router/table.rs b/crates/openplotva-llm/src/router/table.rs index bf3aff70..ee15a6f4 100644 --- a/crates/openplotva-llm/src/router/table.rs +++ b/crates/openplotva-llm/src/router/table.rs @@ -266,4 +266,27 @@ impl RoutingTable { pub fn workflow_keys(&self) -> impl Iterator { self.routes.keys().map(String::as_str) } + + /// Models any route can reach (primaries, fallbacks, canaries, shadows and + /// trigger targets of every workflow and scope), each once, by id. + pub fn routed_models(&self) -> impl Iterator { + let mut ids: Vec = self + .routes + .values() + .flat_map(|scoped| scoped.global.iter().chain(scoped.vip.iter())) + .flat_map(|route| { + route + .primary + .iter() + .chain(route.fallback_tail.iter()) + .chain(route.canary.iter()) + .chain(route.shadow.iter()) + .chain(route.triggers.iter().map(|trigger| &trigger.engage)) + .map(|candidate| candidate.model) + }) + .collect(); + ids.sort_unstable(); + ids.dedup(); + ids.into_iter().filter_map(|id| self.models.get(&id)) + } } From b7725b35ace497c18639eea29447cb3e83059363 Mon Sep 17 00:00:00 2001 From: Valeriy Selitskiy <239034+iamwavecut@users.noreply.github.com> Date: Mon, 12 Oct 2026 22:06:56 +0200 Subject: [PATCH 12/18] Expose lane occupancy, queues, hold times and wait percentiles --- crates/openplotva-app/src/lane_checks.rs | 64 ++++++++++++++ crates/openplotva-app/src/lib.rs | 17 ++++ crates/openplotva-app/src/model_routing.rs | 9 ++ crates/openplotva-llm/src/router/capacity.rs | 83 +++++++++++++++++-- crates/openplotva-llm/src/router/table.rs | 12 +++ crates/openplotva-server/src/lib.rs | 71 ++++++++++++++-- .../openplotva-server/src/runtime_graphql.rs | 36 ++++++++ 7 files changed, 282 insertions(+), 10 deletions(-) diff --git a/crates/openplotva-app/src/lane_checks.rs b/crates/openplotva-app/src/lane_checks.rs index 888f45f0..b57d42fd 100644 --- a/crates/openplotva-app/src/lane_checks.rs +++ b/crates/openplotva-app/src/lane_checks.rs @@ -185,6 +185,35 @@ impl LaneCheck { } } +/// Runtime API view of the live lane pools. +pub struct RuntimeLanes { + pub handle: std::sync::Arc, + pub pools: std::sync::Arc, +} + +impl openplotva_server::RuntimeLaneInspector for RuntimeLanes { + fn lanes(&self) -> Vec { + let table = self.handle.snapshot(); + let clamp = |value: u64| i32::try_from(value).unwrap_or(i32::MAX); + self.pools + .occupancy() + .into_iter() + .map(|lane| openplotva_server::RuntimeLaneData { + id: lane.id, + name: table.pool_name(lane.id).unwrap_or_default().to_owned(), + max_concurrency: lane.max_concurrency.map(|width| clamp(u64::from(width))), + in_flight: clamp(u64::from(lane.in_flight)), + queued_interactive: clamp(u64::from(lane.queued[0])), + queued_user_tool: clamp(u64::from(lane.queued[1])), + queued_background: clamp(u64::from(lane.queued[2])), + mean_hold_ms: lane.mean_hold_ms.map(clamp), + wait_p50_ms: lane.wait_p50_ms.map(clamp), + wait_p95_ms: lane.wait_p95_ms.map(clamp), + }) + .collect() + } +} + fn lane_config_mismatch_event(issue: &LaneConfigIssue) -> crate::runtime_routing::RoutingEvent { crate::runtime_routing::RoutingEvent { severity: "warn".to_owned(), @@ -470,6 +499,41 @@ mod tests { ); } + #[tokio::test] + async fn the_runtime_api_reports_lanes_by_name_with_their_queues() { + let table = crate::model_routing::build_routing_table(&farm()); + let pools = std::sync::Arc::new(openplotva_llm::router::PoolRegistry::new()); + pools.apply(&table.pool_specs()); + let handle = openplotva_llm::router::RouterHandle::new(table); + let _running = pools.try_acquire(Some(3)).expect("GPU1 lane"); + let queued = { + let pools = std::sync::Arc::clone(&pools); + tokio::spawn(async move { + pools + .acquire( + Some(3), + openplotva_llm::router::LanePriority::UserTool, + None, + ) + .await + }) + }; + tokio::task::yield_now().await; + + let lanes = openplotva_server::RuntimeLaneInspector::lanes(&RuntimeLanes { + handle, + pools: std::sync::Arc::clone(&pools), + }); + + assert_eq!(lanes.len(), 2); + assert_eq!(lanes[0].name, "aifarm-gpu0-dialog"); + assert_eq!(lanes[0].max_concurrency, Some(8)); + assert_eq!(lanes[1].name, "aifarm-gpu1-media"); + assert_eq!(lanes[1].in_flight, 1); + assert_eq!(lanes[1].queued_user_tool, 1); + queued.abort(); + } + #[test] fn unrouted_and_disabled_models_are_ignored() { let mut snapshot = farm(); diff --git a/crates/openplotva-app/src/lib.rs b/crates/openplotva-app/src/lib.rs index 033330e6..97783122 100644 --- a/crates/openplotva-app/src/lib.rs +++ b/crates/openplotva-app/src/lib.rs @@ -8838,6 +8838,10 @@ async fn admin_routing_status( return admin_auth_failure_response(error); } let now = std::time::Instant::now(); + let table = routes + .router_handle + .as_ref() + .map(|handle| handle.snapshot()); let pools: Vec = routes .router_pools .as_ref() @@ -8847,8 +8851,17 @@ async fn admin_routing_status( .map(|occupancy| { json!({ "id": occupancy.id, + "name": table.as_ref().and_then(|table| table.pool_name(occupancy.id)), "max_concurrency": occupancy.max_concurrency, "in_flight": occupancy.in_flight, + "queued": { + "interactive": occupancy.queued[0], + "user_tool": occupancy.queued[1], + "background": occupancy.queued[2], + }, + "mean_hold_ms": occupancy.mean_hold_ms, + "wait_p50_ms": occupancy.wait_p50_ms, + "wait_p95_ms": occupancy.wait_p95_ms, }) }) .collect(); @@ -11251,6 +11264,10 @@ async fn start_runtime_workers( llm_run_inspector: Some(Arc::new(llm_run_buffer.clone())), turn_outcome_inspector: Some(Arc::new(turn_outcome_buffer.clone())), routing_event_inspector: Some(Arc::new(routing_event_buffer.clone())), + lane_inspector: Some(Arc::new(lane_checks::RuntimeLanes { + handle: Arc::clone(&router_handle), + pools: Arc::clone(&router_pools), + })), llm_analytics_reader: Some(Arc::new( runtime_llm_analytics::PostgresRuntimeLlmAnalyticsReader::new( service_clients.postgres.clone(), diff --git a/crates/openplotva-app/src/model_routing.rs b/crates/openplotva-app/src/model_routing.rs index c687032e..7bb5a5d8 100644 --- a/crates/openplotva-app/src/model_routing.rs +++ b/crates/openplotva-app/src/model_routing.rs @@ -2469,6 +2469,13 @@ pub fn build_routing_table(snapshot: &RoutingSnapshot) -> RoutingTable { let mut table = RoutingTable::new(providers.clone(), model_rows); table.set_pools(pool_specs); + table.set_pool_names( + snapshot + .pools + .iter() + .map(|pool| (pool.id, pool.name.clone())) + .collect(), + ); for (&(workflow_key, scope_str), assignments) in &by_key_scope { let Some(workflow) = workflows.get(workflow_key) else { @@ -3070,6 +3077,8 @@ mod tests { let mut specs = table.pool_specs(); specs.sort_by_key(|spec| spec.id); assert_eq!(specs.len(), 2); + assert_eq!(table.pool_name(7), Some("aifarm-dialog")); + assert_eq!(table.pool_name(9), None); } #[test] diff --git a/crates/openplotva-llm/src/router/capacity.rs b/crates/openplotva-llm/src/router/capacity.rs index 29a381d1..e996d138 100644 --- a/crates/openplotva-llm/src/router/capacity.rs +++ b/crates/openplotva-llm/src/router/capacity.rs @@ -14,7 +14,7 @@ //! All locks are short synchronous sections; only the RAII permit, never a //! guard, crosses `.await`. -use std::collections::{BTreeMap, HashMap}; +use std::collections::{BTreeMap, HashMap, VecDeque}; use std::sync::{Arc, Mutex, MutexGuard}; use std::time::{Duration, Instant}; @@ -68,10 +68,16 @@ pub struct PoolOccupancy { /// Waiters per priority: interactive, user tool, background. pub queued: [u32; 3], pub mean_hold_ms: Option, + /// Median and 95th percentile wait of the recent admissions (zero for a + /// request that found a free lane). + pub wait_p50_ms: Option, + pub wait_p95_ms: Option, } /// Weight of the newest sample in the mean lane hold time. const HOLD_MEAN_WEIGHT: f64 = 0.2; +/// Admissions kept per pool for the wait percentiles. +const RECENT_WAITS: usize = 128; type WaiterKey = (LanePriority, u64); @@ -82,6 +88,7 @@ struct CellState { waiters: BTreeMap>, next_seq: u64, mean_hold: Option, + recent_waits: VecDeque, } impl CellState { @@ -92,9 +99,37 @@ impl CellState { waiters: BTreeMap::new(), next_seq: 0, mean_hold: None, + recent_waits: VecDeque::with_capacity(RECENT_WAITS), } } + fn admit(&mut self) { + self.in_flight = self.in_flight.saturating_add(1); + self.record_wait(Duration::ZERO); + } + + fn record_wait(&mut self, waited: Duration) { + if self.recent_waits.len() == RECENT_WAITS { + self.recent_waits.pop_front(); + } + self.recent_waits.push_back(waited); + } + + /// Nearest-rank percentiles (50th, 95th) of the recent waits, in ms. + fn wait_percentiles_ms(&self) -> (Option, Option) { + let mut waits: Vec = self.recent_waits.iter().copied().collect(); + if waits.is_empty() { + return (None, None); + } + waits.sort_unstable(); + let at = |quantile: f64| { + let rank = (waits.len() as f64 * quantile).ceil() as usize; + let wait = waits[rank.clamp(1, waits.len()) - 1]; + u64::try_from(wait.as_millis()).unwrap_or(u64::MAX) + }; + (Some(at(0.5)), Some(at(0.95))) + } + fn has_free_lane(&self) -> bool { self.max.is_none_or(|max| self.in_flight < max) } @@ -249,7 +284,7 @@ impl PoolRegistry { if !state.waiters.is_empty() || !state.has_free_lane() { return None; } - state.in_flight = state.in_flight.saturating_add(1); + state.admit(); } Some(PoolPermit::held(cell)) } @@ -277,7 +312,7 @@ impl PoolRegistry { let (receiver, _queued) = { let mut state = cell.lock(); if state.waiters.is_empty() && state.has_free_lane() { - state.in_flight = state.in_flight.saturating_add(1); + state.admit(); drop(state); return Some(PoolPermit::held(cell)); } @@ -293,10 +328,13 @@ impl PoolRegistry { }, ) }; - match budget { + let enqueued = Instant::now(); + let permit = match budget { Some(budget) => tokio::time::timeout(budget, receiver).await.ok()?.ok(), None => receiver.await.ok(), - } + }?; + cell.lock().record_wait(enqueued.elapsed()); + Some(permit) } /// Expected wait for a new request of `priority`: the waiters it would @@ -334,6 +372,7 @@ impl PoolRegistry { let slot = &mut queued[*priority as usize]; *slot = slot.saturating_add(1); } + let (wait_p50_ms, wait_p95_ms) = state.wait_percentiles_ms(); PoolOccupancy { id: *id, max_concurrency: state.max, @@ -342,6 +381,8 @@ impl PoolRegistry { mean_hold_ms: state .mean_hold .map(|mean| u64::try_from(mean.as_millis()).unwrap_or(u64::MAX)), + wait_p50_ms, + wait_p95_ms, } }) .collect(); @@ -710,6 +751,38 @@ mod tests { assert_eq!(pools.forecast_wait(None, LanePriority::Interactive), None); } + #[tokio::test] + async fn wait_percentiles_cover_recent_admissions() { + let pools = Arc::new(registry(1)); + assert_eq!(pools.occupancy()[0].wait_p50_ms, None); + for _ in 0..3 { + drop(pools.try_acquire(Some(1)).expect("free lane")); + } + let held = pools.try_acquire(Some(1)).expect("free lane"); + let waiter = { + let pools = Arc::clone(&pools); + tokio::spawn(async move { + pools + .acquire(Some(1), LanePriority::Interactive, None) + .await + }) + }; + tokio::time::sleep(Duration::from_millis(60)).await; + drop(held); + drop(waiter.await.expect("task").expect("lane")); + + let occupancy = pools.occupancy()[0]; + assert_eq!( + occupancy.wait_p50_ms, + Some(0), + "most admissions found a free lane" + ); + assert!( + occupancy.wait_p95_ms.is_some_and(|p95| p95 >= 50), + "{occupancy:?}" + ); + } + #[test] fn priorities_parse_from_the_database() { assert_eq!(LanePriority::from_db(0), LanePriority::Interactive); diff --git a/crates/openplotva-llm/src/router/table.rs b/crates/openplotva-llm/src/router/table.rs index ee15a6f4..7766c6df 100644 --- a/crates/openplotva-llm/src/router/table.rs +++ b/crates/openplotva-llm/src/router/table.rs @@ -200,6 +200,7 @@ pub struct RoutingTable { providers: HashMap, models: HashMap, pools: HashMap, + pool_names: HashMap, } impl RoutingTable { @@ -213,6 +214,7 @@ impl RoutingTable { providers, models, pools: HashMap::new(), + pool_names: HashMap::new(), } } @@ -226,6 +228,16 @@ impl RoutingTable { self.pools.get(&id) } + /// Replace the operator-facing pool names carried by this snapshot. + pub fn set_pool_names(&mut self, names: HashMap) { + self.pool_names = names; + } + + #[must_use] + pub fn pool_name(&self, id: PoolId) -> Option<&str> { + self.pool_names.get(&id).map(String::as_str) + } + /// The pool specs to reconcile the live registry with after a reload. #[must_use] pub fn pool_specs(&self) -> Vec { diff --git a/crates/openplotva-server/src/lib.rs b/crates/openplotva-server/src/lib.rs index f7878695..6b8ce88c 100644 --- a/crates/openplotva-server/src/lib.rs +++ b/crates/openplotva-server/src/lib.rs @@ -41,10 +41,10 @@ pub use runtime_graphql::{ RuntimeGeminiCachePurger, RuntimeGeminiCachePurgerFuture, RuntimeGradiusAuditReader, RuntimeGradiusAuditReaderFuture, RuntimeGradiusOpportunitiesFilter, RuntimeGradiusSummaryFilter, RuntimeIngestionGatesData, RuntimeJobAnalyticsStatData, - RuntimeLlmAnalyticsData, RuntimeLlmAnalyticsInferenceParamStatData, - RuntimeLlmAnalyticsModelSeriesPointData, RuntimeLlmAnalyticsModelStatData, - RuntimeLlmAnalyticsProviderStatData, RuntimeLlmAnalyticsReader, - RuntimeLlmAnalyticsReaderFuture, RuntimeLlmAnalyticsSeriesPointData, + RuntimeLaneData, RuntimeLaneInspector, RuntimeLlmAnalyticsData, + RuntimeLlmAnalyticsInferenceParamStatData, RuntimeLlmAnalyticsModelSeriesPointData, + RuntimeLlmAnalyticsModelStatData, RuntimeLlmAnalyticsProviderStatData, + RuntimeLlmAnalyticsReader, RuntimeLlmAnalyticsReaderFuture, RuntimeLlmAnalyticsSeriesPointData, RuntimeLlmAnalyticsStageMetricData, RuntimeLlmAnalyticsTopChatData, RuntimeLlmAnalyticsTotalsData, RuntimeLlmGenConfigData, RuntimeLlmRequestChatData, RuntimeLlmRequestData, RuntimeLlmRequestMessageData, RuntimeLlmRequestResultData, @@ -924,7 +924,8 @@ mod tests { RuntimeGeminiCachePurgeResultData, RuntimeGeminiCachePurger, RuntimeGeminiCachePurgerFuture, RuntimeGradiusAuditReader, RuntimeGradiusAuditReaderFuture, RuntimeGradiusOpportunitiesFilter, RuntimeGradiusSummaryFilter, - RuntimeJobAnalyticsStatData, RuntimeLlmAnalyticsData, RuntimeLlmAnalyticsModelStatData, + RuntimeJobAnalyticsStatData, RuntimeLaneData, RuntimeLaneInspector, + RuntimeLlmAnalyticsData, RuntimeLlmAnalyticsModelStatData, RuntimeLlmAnalyticsProviderStatData, RuntimeLlmAnalyticsReader, RuntimeLlmAnalyticsReaderFuture, RuntimeLlmAnalyticsSeriesPointData, RuntimeLlmAnalyticsStageMetricData, RuntimeLlmAnalyticsTopChatData, @@ -2703,6 +2704,66 @@ mod tests { assert_eq!(event["detail"]["admin_report"]["action"], "sent"); } + #[tokio::test] + async fn runtime_api_graphql_serves_llm_lanes() { + struct LaneInspector; + + impl RuntimeLaneInspector for LaneInspector { + fn lanes(&self) -> Vec { + vec![RuntimeLaneData { + id: 1, + name: "aifarm-gpu0-dialog".to_owned(), + max_concurrency: Some(8), + in_flight: 8, + queued_interactive: 2, + queued_user_tool: 1, + queued_background: 0, + mean_hold_ms: Some(3400), + wait_p50_ms: Some(0), + wait_p95_ms: Some(2900), + }] + } + } + + let schema = runtime_api_graphql_schema_with_live_diagnostics( + RuntimeApiGraphqlSnapshot::default(), + RuntimeApiLiveDiagnostics { + lane_inspector: Some(Arc::new(LaneInspector)), + ..RuntimeApiLiveDiagnostics::default() + }, + ); + let response = schema + .execute( + "query { llmLanes { id name maxConcurrency inFlight queuedInteractive queuedUserTool queuedBackground meanHoldMs waitP50Ms waitP95Ms } }", + ) + .await; + + assert!( + response.errors.is_empty(), + "runtime lanes GraphQL returned errors: {:?}", + response.errors + ); + let payload = serde_json::to_value(&response) + .unwrap_or_else(|error| panic!("serialize GraphQL response: {error}")); + let lane = &payload["data"]["llmLanes"][0]; + assert_eq!(lane["id"], "1"); + assert_eq!(lane["name"], "aifarm-gpu0-dialog"); + assert_eq!(lane["maxConcurrency"], 8); + assert_eq!(lane["inFlight"], 8); + assert_eq!(lane["queuedInteractive"], 2); + assert_eq!(lane["queuedUserTool"], 1); + assert_eq!(lane["meanHoldMs"], 3400); + assert_eq!(lane["waitP95Ms"], 2900); + + let unconfigured = runtime_api_graphql_schema_with_live_diagnostics( + RuntimeApiGraphqlSnapshot::default(), + RuntimeApiLiveDiagnostics::default(), + ) + .execute("query { llmLanes { name } }") + .await; + assert!(unconfigured.errors.is_empty()); + } + #[tokio::test] async fn runtime_api_graphql_serves_llm_analytics_like_go_shape() { struct LlmAnalyticsReader; diff --git a/crates/openplotva-server/src/runtime_graphql.rs b/crates/openplotva-server/src/runtime_graphql.rs index b0503da3..68ca05ce 100644 --- a/crates/openplotva-server/src/runtime_graphql.rs +++ b/crates/openplotva-server/src/runtime_graphql.rs @@ -168,6 +168,11 @@ pub trait RuntimeRoutingEventInspector: Send + Sync { ) -> Result, String>; } +/// Runtime API lane pool read boundary. +pub trait RuntimeLaneInspector: Send + Sync { + fn lanes(&self) -> Vec; +} + /// Runtime API LLM analytics read boundary. pub trait RuntimeLlmAnalyticsReader: Send + Sync { fn llm_analytics<'a>(&'a self, range: &'a str) -> RuntimeLlmAnalyticsReaderFuture<'a>; @@ -366,6 +371,7 @@ pub struct RuntimeApiLiveDiagnostics { pub llm_run_inspector: Option>, pub turn_outcome_inspector: Option>, pub routing_event_inspector: Option>, + pub lane_inspector: Option>, pub llm_analytics_reader: Option>, pub gradius_audit_reader: Option>, pub log_inspector: Option>, @@ -1382,6 +1388,25 @@ pub struct RuntimeRoutingEventsFilter { pub limit: i32, } +/// Runtime API lane pool state. +#[derive(Clone, Debug, Default, PartialEq, SimpleObject)] +#[graphql(name = "Lane")] +pub struct RuntimeLaneData { + #[graphql(derived(owned, name = "id", into = "ID", with = "graphql_id"), skip)] + pub id: i64, + pub name: String, + pub max_concurrency: Option, + pub in_flight: i32, + pub queued_interactive: i32, + pub queued_user_tool: i32, + pub queued_background: i32, + pub mean_hold_ms: Option, + #[graphql(name = "waitP50Ms")] + pub wait_p50_ms: Option, + #[graphql(name = "waitP95Ms")] + pub wait_p95_ms: Option, +} + // Runtime API LLM routing event. #[derive(Clone, Debug, Default, PartialEq, SimpleObject)] #[graphql(name = "RoutingEvent", complex)] @@ -1967,6 +1992,7 @@ pub fn runtime_api_graphql_schema_with_live_diagnostics( llm_run_inspector: diagnostics.llm_run_inspector, turn_outcome_inspector: diagnostics.turn_outcome_inspector, routing_event_inspector: diagnostics.routing_event_inspector, + lane_inspector: diagnostics.lane_inspector, llm_analytics_reader: diagnostics.llm_analytics_reader, gradius_audit_reader: diagnostics.gradius_audit_reader, log_inspector: diagnostics.log_inspector, @@ -1999,6 +2025,7 @@ pub struct RuntimeQuery { llm_run_inspector: Option>, turn_outcome_inspector: Option>, routing_event_inspector: Option>, + lane_inspector: Option>, llm_analytics_reader: Option>, gradius_audit_reader: Option>, log_inspector: Option>, @@ -2282,6 +2309,15 @@ impl RuntimeQuery { .map_err(async_graphql::Error::new) } + /// Lane pools: width, in-flight requests, queue depth per priority, mean + /// lane hold time and recent wait percentiles. + async fn llm_lanes(&self) -> Vec { + self.lane_inspector + .as_deref() + .map(RuntimeLaneInspector::lanes) + .unwrap_or_default() + } + async fn llm_analytics( &self, range: Option, From 4227852e1a34970bfc8b69b16521757d5a200381 Mon Sep 17 00:00:00 2001 From: Valeriy Selitskiy <239034+iamwavecut@users.noreply.github.com> Date: Mon, 12 Oct 2026 22:08:36 +0200 Subject: [PATCH 13/18] Record where the backend lanes code differs from its plan --- docs/superpowers/plans/2026-10-12-backend-lanes.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/docs/superpowers/plans/2026-10-12-backend-lanes.md b/docs/superpowers/plans/2026-10-12-backend-lanes.md index 59182bfb..26de5a6e 100644 --- a/docs/superpowers/plans/2026-10-12-backend-lanes.md +++ b/docs/superpowers/plans/2026-10-12-backend-lanes.md @@ -1774,3 +1774,15 @@ After `deploy-production.yml` succeeds: - `_sqlx_migrations` has 199 and 200; `llm_capacity_pools` has the five lane names; the ASR, embedder and privacy filter models point at their pools. - The startup log shows dialog workers 16 and memory workers 16; readiness detail for `lane_config` says the pools match Discovery. - Over the first hour: dialog requests served by GPU2 and OpenRouter are only failures and budget bypasses (`lane_bypass`), `lane_wait` events show queueing on GPU0, `llmLanes` shows queues per priority. + +--- + +## Implementation Notes + +Where the delivered code differs from the task text above: + +- Task 1: a `QueuedWaiter` guard removes a waiter from the queue on any exit (budget end or cancellation), so no closed senders linger. The pools also keep the last 128 admission waits for p50/p95 (`PoolOccupancy::wait_p50_ms`, `wait_p95_ms`). +- Task 3: the previous step's preferred target is moved to the front only while its lane is free. Free-lane reordering touches only live primary and overflow candidates. A request that ran and failed re-selects once more, so a candidate passed over for its budget gets the time that is left. A failed attempt is reported as such, not as a capacity timeout. +- Task 7: `RouterRuntime` holds two `LaneWorkerScale`s (dialog and memory). Memory workers now run under the same supervisor as dialog workers (`dialog_workers::spawn_worker_supervisor`), so both fleets resize on a routing reload. `PERSISTENT_QUEUE_MEMORY_CONSOLIDATION_WORKERS` is the floor, and `0` disables memory workers. +- Task 8: `lane_checks::LaneCheck` reuses the analytics `DiscoveryCapacityClient` and fetches all services at once. Readiness reports the summary, including services whose width Discovery did not return. +- Task 9: the runtime API type is `Lane` (`llmLanes`), with `waitP50Ms` and `waitP95Ms`. Pool names come from `RoutingTable::pool_name`. From 5df67b3b62f7a79c130af54e15dbc1ff3ed2c2d1 Mon Sep 17 00:00:00 2001 From: Valeriy Selitskiy <239034+iamwavecut@users.noreply.github.com> Date: Mon, 12 Oct 2026 22:11:21 +0200 Subject: [PATCH 14/18] Add llmLanes to the runtime GraphQL schema contract --- .../fixtures/runtime_graphql_schema.graphql | 21 +++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/crates/openplotva-server/tests/fixtures/runtime_graphql_schema.graphql b/crates/openplotva-server/tests/fixtures/runtime_graphql_schema.graphql index 39d3ed3a..86e23e24 100644 --- a/crates/openplotva-server/tests/fixtures/runtime_graphql_schema.graphql +++ b/crates/openplotva-server/tests/fixtures/runtime_graphql_schema.graphql @@ -224,6 +224,22 @@ A scalar that can represent any JSON value. """ scalar JSON +""" +Runtime API lane pool state. +""" +type Lane { + id: ID! + name: String! + maxConcurrency: Int + inFlight: Int! + queuedInteractive: Int! + queuedUserTool: Int! + queuedBackground: Int! + meanHoldMs: Int + waitP50Ms: Int + waitP95Ms: Int +} + type LlmAnalytics { range: String! bucket: String! @@ -463,6 +479,11 @@ type RuntimeQuery { dialogTurnOutcomes(filter: DialogTurnOutcomesFilterInput): [DialogTurnOutcome!]! dispatcherSendFailures(limit: Int): [DispatcherSendFailure!]! llmRoutingEvents(filter: RoutingEventsFilterInput): [RoutingEvent!]! + """ + Lane pools: width, in-flight requests, queue depth per priority, mean + lane hold time and recent wait percentiles. + """ + llmLanes: [Lane!]! llmAnalytics(range: String): LlmAnalytics! virtualDialog(sessionID: String!): VirtualDialog updatesRuntime: UpdatesRuntime! From eddf53b17c7aafe0ef09fb4ecb9e53253d4e71d1 Mon Sep 17 00:00:00 2001 From: Valeriy Selitskiy <239034+iamwavecut@users.noreply.github.com> Date: Mon, 12 Oct 2026 22:14:02 +0200 Subject: [PATCH 15/18] Bump quinn-proto to 0.11.19 for RUSTSEC-2026-0338..0342 --- Cargo.lock | 84 ++++++++++++------------------------------------------ 1 file changed, 18 insertions(+), 66 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index 402a050d..3057ad9f 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1307,11 +1307,9 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "899def5c37c4fd7b2664648c28120ecec138e4d395b459e5ca34f9cce2dd77fd" dependencies = [ "cfg-if", - "js-sys", "libc", "r-efi 5.3.0", "wasip2", - "wasm-bindgen", ] [[package]] @@ -1321,11 +1319,13 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0de51e6874e94e7bf76d726fc5d13ba782deca734ff60d5bb2fb2607c7406555" dependencies = [ "cfg-if", + "js-sys", "libc", "r-efi 6.0.0", "rand_core 0.10.1", "wasip2", "wasip3", + "wasm-bindgen", ] [[package]] @@ -2128,7 +2128,7 @@ dependencies = [ "openplotva-updates", "openplotva-web", "quick-xml", - "rand 0.10.1", + "rand", "redis", "reqwest", "serde", @@ -2198,7 +2198,7 @@ dependencies = [ "openplotva-media", "openplotva-memory", "openplotva-prompts", - "rand 0.10.1", + "rand", "reqwest", "serde", "serde_json", @@ -2583,15 +2583,6 @@ version = "0.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "439ee305def115ba05938db6eb1644ff94165c5ab5e9420d1c1bcedbba909391" -[[package]] -name = "ppv-lite86" -version = "0.2.21" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "85eae3c4ed2f50dcfe72643da4befc30deadb458a9b590d720cde2f2b1e97da9" -dependencies = [ - "zerocopy", -] - [[package]] name = "precomputed-hash" version = "0.1.1" @@ -2690,15 +2681,16 @@ dependencies = [ [[package]] name = "quinn-proto" -version = "0.11.15" +version = "0.11.19" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4fcb935c5bec503c2f0e306bdd3e58bb9029dcb14fa8d9ac76e3a5256ac0763e" +checksum = "0e750cca55fe4f0439a15d0bb529da9651e79993e8e72c61a899a36d462befbe" dependencies = [ "aws-lc-rs", "bytes", - "getrandom 0.3.4", + "getrandom 0.4.2", "lru-slab", - "rand 0.9.4", + "rand", + "rand_pcg", "ring", "rustc-hash", "rustls", @@ -2745,16 +2737,6 @@ version = "6.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf" -[[package]] -name = "rand" -version = "0.9.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "44c5af06bb1b7d3216d91932aed5265164bf384dc89cd6ba05cf59a35f5f76ea" -dependencies = [ - "rand_chacha", - "rand_core 0.9.5", -] - [[package]] name = "rand" version = "0.10.1" @@ -2766,16 +2748,6 @@ dependencies = [ "rand_core 0.10.1", ] -[[package]] -name = "rand_chacha" -version = "0.9.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d3022b5f1df60f26e1ffddd6c66e8aa15de382ae63b3a0c1bfc0e4d3e3f325cb" -dependencies = [ - "ppv-lite86", - "rand_core 0.9.5", -] - [[package]] name = "rand_core" version = "0.6.4" @@ -2787,18 +2759,18 @@ dependencies = [ [[package]] name = "rand_core" -version = "0.9.5" +version = "0.10.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "76afc826de14238e6e8c374ddcc1fa19e374fd8dd986b0d2af0d02377261d83c" -dependencies = [ - "getrandom 0.3.4", -] +checksum = "63b8176103e19a2643978565ca18b50549f6101881c443590420e4dc998a3c69" [[package]] -name = "rand_core" -version = "0.10.1" +name = "rand_pcg" +version = "0.10.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "63b8176103e19a2643978565ca18b50549f6101881c443590420e4dc998a3c69" +checksum = "caa0f4137e1c0a72f4c651489402276c8e8e1cf081f3b0ba156d2cbeef09e86a" +dependencies = [ + "rand_core 0.10.1", +] [[package]] name = "rcgen" @@ -3485,7 +3457,7 @@ dependencies = [ "log", "md-5", "memchr", - "rand 0.10.1", + "rand", "serde", "serde_json", "sha2 0.11.0", @@ -4649,26 +4621,6 @@ dependencies = [ "synstructure", ] -[[package]] -name = "zerocopy" -version = "0.8.48" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "eed437bf9d6692032087e337407a86f04cd8d6a16a37199ed57949d415bd68e9" -dependencies = [ - "zerocopy-derive", -] - -[[package]] -name = "zerocopy-derive" -version = "0.8.48" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "70e3cd084b1788766f53af483dd21f93881ff30d7320490ec3ef7526d203bad4" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.117", -] - [[package]] name = "zerofrom" version = "0.1.8" From 290ffb512c074ae16c73bd67f5a30d96e64bc189 Mon Sep 17 00:00:00 2001 From: Valeriy Selitskiy <239034+iamwavecut@users.noreply.github.com> Date: Mon, 12 Oct 2026 22:24:46 +0200 Subject: [PATCH 16/18] Enqueue memory runs hourly as chat hours complete --- .env.example | 4 +- crates/openplotva-app/src/lib.rs | 6 +- crates/openplotva-app/src/memory_runtime.rs | 281 ++++++++++-------- crates/openplotva-app/src/runtime_llm.rs | 2 +- crates/openplotva-config/src/lib.rs | 22 -- crates/openplotva-storage/src/lib.rs | 165 ++++++++-- docs/CODEBASE_MAP.md | 4 +- .../specs/2026-10-12-backend-lanes-design.md | 11 +- 8 files changed, 319 insertions(+), 176 deletions(-) diff --git a/.env.example b/.env.example index 14278cbd..5f367221 100644 --- a/.env.example +++ b/.env.example @@ -290,8 +290,8 @@ # MEMORY_CONSOLIDATION_PROVIDER=aifarm # MEMORY_CONSOLIDATION_MODEL=qwen3.8-27b # MEMORY_CONSOLIDATION_TIMEOUT_SECONDS=600 -# MEMORY_DAILY_SCHEDULE=03:17 -# MEMORY_DAILY_WINDOW_HOURS=24 +# Each chat hour with enough messages becomes a memory run at HH:02 UTC, after +# the hour completes; the daily cap counts runs created since midnight UTC. # MEMORY_WORKER_CONCURRENCY=1 # MEMORY_MAX_MESSAGES_PER_RUN=200 # MEMORY_MIN_MESSAGES_PER_RUN=20 diff --git a/crates/openplotva-app/src/lib.rs b/crates/openplotva-app/src/lib.rs index 97783122..d18a2e94 100644 --- a/crates/openplotva-app/src/lib.rs +++ b/crates/openplotva-app/src/lib.rs @@ -4448,9 +4448,10 @@ async fn admin_memory_restart_execute_response( let mut queued_runs = 0_i64; let result = if run_now { store - .ensure_daily_runs( + .ensure_hourly_runs( OffsetDateTime::now_utc(), routes.memory_retention, + routes.memory_retention, routes.memory_enqueue_policy, ) .await @@ -4566,9 +4567,10 @@ async fn admin_memory_restart_override_execute_response( let result = if run_now { runtime .store - .ensure_daily_runs( + .ensure_hourly_runs( OffsetDateTime::now_utc(), routes.memory_retention, + routes.memory_retention, routes.memory_enqueue_policy, ) .await diff --git a/crates/openplotva-app/src/memory_runtime.rs b/crates/openplotva-app/src/memory_runtime.rs index da7dcf5a..7dc5c016 100644 --- a/crates/openplotva-app/src/memory_runtime.rs +++ b/crates/openplotva-app/src/memory_runtime.rs @@ -31,7 +31,7 @@ use openplotva_taskman::{ use serde::{Deserialize, Serialize}; use serde_json::Value; use thiserror::Error; -use time::{OffsetDateTime, Time}; +use time::OffsetDateTime; use crate::embedder::DiscoveryEmbedderClient; use crate::routed_attempts::{ @@ -207,11 +207,13 @@ pub trait MemoryWriteStore: std::fmt::Debug + Send + Sync { /// Storage boundary needed to claim and finalize memory runs. pub trait MemoryRunStore: MemoryWriteStore { - /// Ensure daily memory runs exist for the configured retention window. - fn ensure_daily_runs<'a>( + /// Ensure a memory run exists for every completed chat hour that started + /// within the last `lookback` of the retention window. + fn ensure_hourly_runs<'a>( &'a self, now: OffsetDateTime, retention: Duration, + lookback: Duration, policy: openplotva_memory::MemoryRunEnqueuePolicy, ) -> MemoryWriteStoreFuture<'a, u64, Self::Error>; @@ -400,14 +402,15 @@ impl MemoryWriteStore for PostgresMemoryStore { } impl MemoryRunStore for PostgresMemoryStore { - fn ensure_daily_runs<'a>( + fn ensure_hourly_runs<'a>( &'a self, now: OffsetDateTime, retention: Duration, + lookback: Duration, policy: openplotva_memory::MemoryRunEnqueuePolicy, ) -> MemoryWriteStoreFuture<'a, u64, Self::Error> { Box::pin(async move { - PostgresMemoryStore::ensure_daily_runs(self, now, retention, policy).await + PostgresMemoryStore::ensure_hourly_runs(self, now, retention, lookback, policy).await }) } @@ -564,10 +567,8 @@ pub struct MemoryServiceTickReport { pub enabled: bool, /// Whether this tick is the startup tick. pub startup: bool, - /// Whether the configured daily window was active. - pub daily_window_active: bool, - /// Newly queued daily runs reported by storage. - pub queued_daily_runs: u64, + /// Newly queued hourly runs reported by storage. + pub queued_runs: u64, /// Runs completed or skipped without a process error. pub processed_runs: i64, /// Claimed runs that ended in a process failure and were marked failed. @@ -585,8 +586,8 @@ pub struct MemoryServiceWorkerReport { pub ticks: i64, /// Number of runtime API triggered queue-drain ticks. pub triggered_ticks: i64, - /// Newly queued daily runs across startup and scheduled ticks. - pub queued_daily_runs: u64, + /// Newly queued hourly runs across startup and scheduled ticks. + pub queued_runs: u64, /// Runs completed or skipped without a process error. pub processed_runs: i64, /// Claimed runs that ended in a process failure and were marked failed. @@ -602,13 +603,11 @@ pub struct MemoryConsolidationQueueScheduleReport { pub enabled: bool, /// Whether this tick is the startup tick. pub startup: bool, - /// Whether the configured daily window was active. - pub daily_window_active: bool, - /// Newly queued daily runs reported by storage. - pub queued_daily_runs: u64, + /// Newly queued hourly runs reported by storage. + pub queued_runs: u64, /// Assigned taskman job ID. pub assigned_job_id: Option, - /// Daily-run enqueue error, if any. + /// Run enqueue error, if any. pub store_error: Option, } @@ -621,11 +620,11 @@ pub struct MemoryConsolidationQueueSchedulerReport { pub ticks: i64, /// Number of runtime API trigger ticks. pub triggered_ticks: i64, - /// Newly queued daily runs across startup and scheduled ticks. - pub queued_daily_runs: u64, + /// Newly queued hourly runs across startup and scheduled ticks. + pub queued_runs: u64, /// Number of taskman jobs assigned. pub assigned_jobs: i64, - /// Daily-run enqueue errors. + /// Run enqueue errors. pub store_errors: i64, } @@ -697,11 +696,8 @@ pub struct MemoryRunProcessConfig { pub struct MemoryServiceWorkerConfig { /// Whether memory consolidation is enabled. pub enabled: bool, - /// Retention window used when ensuring daily runs. + /// Retention window used when ensuring hourly runs. pub retention: Duration, - pub daily_schedule: String, - pub daily_window: Duration, - pub tick_interval: Duration, pub enqueue_policy: openplotva_memory::MemoryRunEnqueuePolicy, /// Per-run processing config. pub process: MemoryRunProcessConfig, @@ -786,9 +782,6 @@ impl Default for MemoryServiceWorkerConfig { Self { enabled: true, retention: Duration::from_secs(7 * 24 * 60 * 60), - daily_schedule: "03:17".to_owned(), - daily_window: Duration::from_secs(24 * 60 * 60), - tick_interval: Duration::from_secs(60 * 60), enqueue_policy: openplotva_memory::MemoryRunEnqueuePolicy::default(), process: MemoryRunProcessConfig::default(), } @@ -816,21 +809,6 @@ impl MemoryServiceWorkerConfig { } else { self.retention }, - daily_schedule: if self.daily_schedule.trim().is_empty() { - fallback.daily_schedule - } else { - self.daily_schedule.trim().to_owned() - }, - daily_window: if self.daily_window.is_zero() { - fallback.daily_window - } else { - self.daily_window - }, - tick_interval: if self.tick_interval.is_zero() { - fallback.tick_interval - } else { - self.tick_interval - }, enqueue_policy: self.enqueue_policy.normalized(), process: self.process.normalized(), } @@ -1359,7 +1337,7 @@ where let stop = stop; tokio::pin!(stop); loop { - let sleep = tokio::time::sleep(cfg.tick_interval); + let sleep = tokio::time::sleep(next_memory_enqueue_delay(OffsetDateTime::now_utc())); tokio::pin!(sleep); let trigger_notified = trigger.notified(); tokio::pin!(trigger_notified); @@ -1379,7 +1357,7 @@ where report.add_tick(&tick); } () = &mut trigger_notified => { - let tick = run_memory_service_once_at_with_daily( + let tick = run_memory_service_once_at_with_enqueue( extractor, store, embedder, @@ -1412,17 +1390,18 @@ where Store: MemoryRunStore, Embedder: EmbeddingProvider, { - run_memory_service_once_at_with_daily(extractor, store, embedder, cfg, now, startup, true).await + run_memory_service_once_at_with_enqueue(extractor, store, embedder, cfg, now, startup, true) + .await } -async fn run_memory_service_once_at_with_daily( +async fn run_memory_service_once_at_with_enqueue( extractor: &Extractor, store: &Store, embedder: Option<&Embedder>, cfg: &MemoryServiceWorkerConfig, now: OffsetDateTime, startup: bool, - ensure_daily: bool, + enqueue_runs: bool, ) -> MemoryServiceTickReport where Extractor: MemoryExtractor + Send + Sync, @@ -1434,19 +1413,23 @@ where let mut report = MemoryServiceTickReport { enabled: cfg.enabled, startup, - daily_window_active: memory_schedule_active_at(now, &cfg.daily_schedule, cfg.daily_window), ..MemoryServiceTickReport::default() }; if !cfg.enabled { return report; } - if ensure_daily && (startup || report.daily_window_active) { + if enqueue_runs { match store - .ensure_daily_runs(now, cfg.retention, cfg.enqueue_policy) + .ensure_hourly_runs( + now, + cfg.retention, + memory_enqueue_lookback(now, startup, cfg.retention), + cfg.enqueue_policy, + ) .await { - Ok(queued) => report.queued_daily_runs = queued, + Ok(queued) => report.queued_runs = queued, Err(error) => { report.store_errors += 1; tracing::warn!(%error, "memory run enqueue failed"); @@ -1482,7 +1465,7 @@ pub async fn schedule_memory_consolidation_taskman_once_at( cfg: &MemoryServiceWorkerConfig, now: OffsetDateTime, startup: bool, - ensure_daily: bool, + enqueue_runs: bool, ) -> MemoryConsolidationQueueScheduleReport where Store: MemoryRunStore, @@ -1491,19 +1474,23 @@ where let mut report = MemoryConsolidationQueueScheduleReport { enabled: cfg.enabled, startup, - daily_window_active: memory_schedule_active_at(now, &cfg.daily_schedule, cfg.daily_window), ..MemoryConsolidationQueueScheduleReport::default() }; if !cfg.enabled { return report; } - if ensure_daily && (startup || report.daily_window_active) { + if enqueue_runs { match store - .ensure_daily_runs(now, cfg.retention, cfg.enqueue_policy) + .ensure_hourly_runs( + now, + cfg.retention, + memory_enqueue_lookback(now, startup, cfg.retention), + cfg.enqueue_policy, + ) .await { - Ok(queued) => report.queued_daily_runs = queued, + Ok(queued) => report.queued_runs = queued, Err(error) => { report.store_error = Some(error.to_string()); tracing::warn!(%error, "memory run enqueue failed"); @@ -1561,7 +1548,7 @@ where let stop = stop; tokio::pin!(stop); loop { - let sleep = tokio::time::sleep(cfg.tick_interval); + let sleep = tokio::time::sleep(next_memory_enqueue_delay(OffsetDateTime::now_utc())); tokio::pin!(sleep); let trigger_notified = trigger.notified(); tokio::pin!(trigger_notified); @@ -1746,7 +1733,7 @@ where impl MemoryServiceWorkerReport { fn add_tick(&mut self, tick: &MemoryServiceTickReport) { - self.queued_daily_runs += tick.queued_daily_runs; + self.queued_runs += tick.queued_runs; self.processed_runs += tick.processed_runs; self.failed_runs += tick.failed_runs; self.store_errors += tick.store_errors; @@ -1755,7 +1742,7 @@ impl MemoryServiceWorkerReport { impl MemoryConsolidationQueueSchedulerReport { fn add_schedule_tick(&mut self, tick: &MemoryConsolidationQueueScheduleReport) { - self.queued_daily_runs += tick.queued_daily_runs; + self.queued_runs += tick.queued_runs; if tick.assigned_job_id.is_some() { self.assigned_jobs += 1; } @@ -2531,44 +2518,42 @@ fn estimate_extraction_tokens( ) } -#[must_use] -pub fn memory_schedule_due_at(now: OffsetDateTime, schedule: &str) -> bool { - let (hour, minute) = parse_daily_schedule(schedule); - now.hour() > hour || (now.hour() == hour && now.minute() >= minute) -} +/// Minutes past the hour at which the scheduler enqueues the hour that just +/// completed, so history writes that trail the hour boundary are included. +const MEMORY_ENQUEUE_MINUTE: u8 = 2; +/// Completed hours an hourly tick looks back over; a tick that ran late or a +/// short outage still lands its hours. +pub const MEMORY_RECENT_LOOKBACK: Duration = Duration::from_secs(2 * 60 * 60); +/// UTC hour whose tick rescans the whole retention window for hours a longer +/// outage or the daily cap left without a run. +const MEMORY_FULL_SCAN_HOUR: u8 = 3; +/// History span one enqueue tick scans: the whole retention window on startup +/// and once a day, otherwise the recent hours. #[must_use] -pub fn memory_schedule_active_at(now: OffsetDateTime, schedule: &str, window: Duration) -> bool { - if window.is_zero() || window >= Duration::from_secs(24 * 60 * 60) { - return memory_schedule_due_at(now, schedule); - } - let (hour, minute) = parse_daily_schedule(schedule); - let start_time = Time::from_hms(hour, minute, 0).unwrap_or(Time::MIDNIGHT); - let mut start = now.replace_time(start_time); - if now < start { - start -= time::Duration::days(1); +pub fn memory_enqueue_lookback( + now: OffsetDateTime, + startup: bool, + retention: Duration, +) -> Duration { + if startup || now.to_offset(time::UtcOffset::UTC).hour() == MEMORY_FULL_SCAN_HOUR { + retention + } else { + MEMORY_RECENT_LOOKBACK.min(retention) } - let window = time::Duration::try_from(window).unwrap_or(time::Duration::MAX); - now >= start && now < start + window } -fn parse_daily_schedule(schedule: &str) -> (u8, u8) { - let Some((hour, minute)) = schedule.trim().split_once(':') else { - return (3, 17); - }; - if minute.contains(':') { - return (3, 17); - } - let Ok(hour) = hour.parse::() else { - return (3, 17); - }; - let Ok(minute) = minute.parse::() else { - return (3, 17); - }; - if hour > 23 || minute > 59 { - return (3, 17); +/// Time from `now` until the next enqueue tick at `HH:02` UTC. +#[must_use] +pub fn next_memory_enqueue_delay(now: OffsetDateTime) -> Duration { + let now = now.to_offset(time::UtcOffset::UTC); + let hour_start = + now.date().midnight().assume_utc() + time::Duration::hours(i64::from(now.hour())); + let mut next = hour_start + time::Duration::minutes(i64::from(MEMORY_ENQUEUE_MINUTE)); + if next <= now { + next += time::Duration::HOUR; } - (hour, minute) + Duration::try_from(next - now).unwrap_or(Duration::from_secs(60 * 60)) } /// Config for one extracted memory batch write. @@ -4687,9 +4672,6 @@ pub fn memory_service_worker_config_from_memory_config( MemoryServiceWorkerConfig { enabled: config.enabled, retention: positive_hours(config.retention_hours, 7 * 24), - daily_schedule: config.daily_schedule.clone(), - daily_window: positive_hours(config.daily_window_hours, 24), - tick_interval: Duration::from_secs(60 * 60), enqueue_policy: openplotva_memory::MemoryRunEnqueuePolicy { min_messages_per_run: config.min_messages_per_run, max_queued_runs: config.max_queued_runs, @@ -4975,11 +4957,12 @@ mod tests { #[derive(Debug, Default)] struct FakeMemoryWriteStore { ids: Vec, - ensure_daily_count: u64, + ensure_hourly_count: u64, ensured: Mutex< Vec<( OffsetDateTime, Duration, + Duration, openplotva_memory::MemoryRunEnqueuePolicy, )>, >, @@ -5177,18 +5160,19 @@ mod tests { } impl MemoryRunStore for FakeMemoryWriteStore { - fn ensure_daily_runs<'a>( + fn ensure_hourly_runs<'a>( &'a self, now: OffsetDateTime, retention: Duration, + lookback: Duration, policy: openplotva_memory::MemoryRunEnqueuePolicy, ) -> MemoryWriteStoreFuture<'a, u64, Self::Error> { Box::pin(async move { self.ensured .lock() .expect("ensured") - .push((now, retention, policy)); - Ok(self.ensure_daily_count) + .push((now, retention, lookback, policy)); + Ok(self.ensure_hourly_count) }) } @@ -6603,29 +6587,79 @@ mod tests { } #[test] - fn memory_schedule_matches_go_daily_window_rules() { - let before = utc_test_time(3, 16); - let after = utc_test_time(3, 17); - assert!(!memory_schedule_due_at(before, "03:17")); - assert!(memory_schedule_due_at(after, "03:17")); - assert!(memory_schedule_due_at(after, "03:17:00")); - - let inside = utc_test_time(5, 0); - let outside = utc_test_time(8, 0); - assert!(memory_schedule_active_at( - inside, - "03:17", - Duration::from_secs(4 * 60 * 60) - )); - assert!(!memory_schedule_active_at( - outside, - "03:17", - Duration::from_secs(4 * 60 * 60) - )); + fn memory_enqueue_ticks_land_shortly_after_each_hour() { + assert_eq!( + next_memory_enqueue_delay(utc_test_time(3, 1)), + Duration::from_secs(60) + ); + assert_eq!( + next_memory_enqueue_delay(utc_test_time(3, 2)), + Duration::from_secs(60 * 60) + ); + assert_eq!( + next_memory_enqueue_delay(utc_test_time(3, 17)), + Duration::from_secs(45 * 60) + ); + assert_eq!( + next_memory_enqueue_delay(utc_test_time(23, 30)), + Duration::from_secs(32 * 60) + ); + } + + #[tokio::test] + async fn every_scheduler_tick_enqueues_completed_hours() { + let store = FakeMemoryWriteStore { + ensure_hourly_count: 2, + ..FakeMemoryWriteStore::default() + }; + let queue = openplotva_taskman::InMemoryTaskQueue::new(); + let cfg = MemoryServiceWorkerConfig::default(); + + let report = schedule_memory_consolidation_taskman_once_at( + &store, + &queue, + &cfg, + utc_test_time(14, 2), + false, + true, + ) + .await; + schedule_memory_consolidation_taskman_once_at( + &store, + &queue, + &cfg, + utc_test_time(3, 2), + false, + true, + ) + .await; + schedule_memory_consolidation_taskman_once_at( + &store, + &queue, + &cfg, + utc_test_time(14, 2), + true, + true, + ) + .await; + + assert_eq!(report.queued_runs, 2); + let lookbacks: Vec = store + .ensured + .lock() + .expect("ensured") + .iter() + .map(|(_, _, lookback, _)| *lookback) + .collect(); + assert_eq!( + lookbacks, + vec![MEMORY_RECENT_LOOKBACK, cfg.retention, cfg.retention], + "hourly ticks scan recent hours; the daily tick and startup scan the retention window" + ); } #[tokio::test] - async fn memory_service_startup_ensures_daily_runs_and_drains_queue() { + async fn memory_service_startup_ensures_hourly_runs_and_drains_queue() { let now = utc_test_time(3, 17); let run = openplotva_memory::Run { id: 90, @@ -6635,7 +6669,7 @@ mod tests { ..openplotva_memory::Run::default() }; let store = FakeMemoryWriteStore { - ensure_daily_count: 4, + ensure_hourly_count: 4, ids: vec![901], run: Mutex::new(Some(run)), messages: Mutex::new(vec![openplotva_memory::Message { @@ -6671,7 +6705,6 @@ mod tests { Option::<&FakeEmbedder>::None, &MemoryServiceWorkerConfig { retention: Duration::from_secs(12 * 60 * 60), - daily_window: Duration::from_secs(60), ..MemoryServiceWorkerConfig::default() }, now, @@ -6681,7 +6714,7 @@ mod tests { assert!(report.enabled); assert!(report.startup); - assert_eq!(report.queued_daily_runs, 4); + assert_eq!(report.queued_runs, 4); assert_eq!(report.processed_runs, 1); assert_eq!(report.failed_runs, 0); assert_eq!(report.store_errors, 0); @@ -6691,10 +6724,10 @@ mod tests { } #[tokio::test] - async fn memory_taskman_scheduler_ensures_daily_runs_and_assigns_go_job() { + async fn memory_taskman_scheduler_ensures_hourly_runs_and_assigns_go_job() { let now = utc_test_time(3, 17); let store = FakeMemoryWriteStore { - ensure_daily_count: 3, + ensure_hourly_count: 3, ..FakeMemoryWriteStore::default() }; let queue = openplotva_taskman::InMemoryTaskQueue::new(); @@ -6704,7 +6737,6 @@ mod tests { &queue, &MemoryServiceWorkerConfig { retention: Duration::from_secs(12 * 60 * 60), - daily_window: Duration::from_secs(60), ..MemoryServiceWorkerConfig::default() }, now, @@ -6718,8 +6750,7 @@ mod tests { MemoryConsolidationQueueScheduleReport { enabled: true, startup: true, - daily_window_active: true, - queued_daily_runs: 3, + queued_runs: 3, assigned_job_id: Some(1), store_error: None, } diff --git a/crates/openplotva-app/src/runtime_llm.rs b/crates/openplotva-app/src/runtime_llm.rs index 72f9dd45..6e2b707b 100644 --- a/crates/openplotva-app/src/runtime_llm.rs +++ b/crates/openplotva-app/src/runtime_llm.rs @@ -271,7 +271,7 @@ ON CONFLICT (source, kind, granularity, bucket_start, dimensions_hash) DO UPDATE metrics = EXCLUDED.metrics, updated_at = CURRENT_TIMESTAMP"#; // Purge terminal memory_runs after they are rolled up. Cutoff = retention_days, -// which (default 14) MUST stay >= the memory pipeline's ensure_daily window +// which (default 14) MUST stay >= the memory pipeline's enqueue window // (MEMORY_RETENTION_HOURS=168=7d) so re-creation never violates the runs' UNIQUE // idempotency constraint. Never touches queued/processing runs. const SQL_DELETE_OLD_MEMORY_RUNS: &str = r#" diff --git a/crates/openplotva-config/src/lib.rs b/crates/openplotva-config/src/lib.rs index 8fa67769..d5ce7f68 100644 --- a/crates/openplotva-config/src/lib.rs +++ b/crates/openplotva-config/src/lib.rs @@ -1043,8 +1043,6 @@ pub struct MemoryConfig { pub consolidation_provider: String, pub consolidation_model: String, pub consolidation_timeout_seconds: i32, - pub daily_schedule: String, - pub daily_window_hours: i32, pub worker_concurrency: i32, pub max_messages_per_run: i32, pub min_messages_per_run: i32, @@ -1568,10 +1566,6 @@ pub struct RawConfig { pub memory_consolidation_model: Option, /// `MEMORY_CONSOLIDATION_TIMEOUT_SECONDS`. pub memory_consolidation_timeout_seconds: Option, - /// `MEMORY_DAILY_SCHEDULE`. - pub memory_daily_schedule: Option, - /// `MEMORY_DAILY_WINDOW_HOURS`. - pub memory_daily_window_hours: Option, /// `MEMORY_WORKER_CONCURRENCY`. pub memory_worker_concurrency: Option, /// `MEMORY_MAX_MESSAGES_PER_RUN`. @@ -2880,14 +2874,6 @@ impl AppConfig { raw.memory_consolidation_timeout_seconds, 600, )?, - daily_schedule: raw - .memory_daily_schedule - .unwrap_or_else(|| "03:17".to_owned()), - daily_window_hours: parse_i32( - "MEMORY_DAILY_WINDOW_HOURS", - raw.memory_daily_window_hours, - 24, - )?, worker_concurrency: parse_i32( "MEMORY_WORKER_CONCURRENCY", raw.memory_worker_concurrency, @@ -3402,8 +3388,6 @@ impl RawConfig { memory_consolidation_provider: env("MEMORY_CONSOLIDATION_PROVIDER"), memory_consolidation_model: env("MEMORY_CONSOLIDATION_MODEL"), memory_consolidation_timeout_seconds: env("MEMORY_CONSOLIDATION_TIMEOUT_SECONDS"), - memory_daily_schedule: env("MEMORY_DAILY_SCHEDULE"), - memory_daily_window_hours: env("MEMORY_DAILY_WINDOW_HOURS"), memory_worker_concurrency: env("MEMORY_WORKER_CONCURRENCY"), memory_max_messages_per_run: env("MEMORY_MAX_MESSAGES_PER_RUN"), memory_min_messages_per_run: env("MEMORY_MIN_MESSAGES_PER_RUN"), @@ -4216,8 +4200,6 @@ mod tests { DEFAULT_MEMORY_CONSOLIDATION_MODEL ); assert_eq!(config.memory.consolidation_timeout_seconds, 600); - assert_eq!(config.memory.daily_schedule, "03:17"); - assert_eq!(config.memory.daily_window_hours, 24); assert_eq!(config.memory.worker_concurrency, 1); assert_eq!(config.memory.max_messages_per_run, 200); assert_eq!( @@ -4937,8 +4919,6 @@ mod tests { memory_consolidation_provider: Some("genkit".to_owned()), memory_consolidation_model: Some("memory-model".to_owned()), memory_consolidation_timeout_seconds: Some("777".to_owned()), - memory_daily_schedule: Some("04:05".to_owned()), - memory_daily_window_hours: Some("12".to_owned()), memory_worker_concurrency: Some("2".to_owned()), memory_max_messages_per_run: Some("55".to_owned()), memory_min_messages_per_run: Some("20".to_owned()), @@ -4977,8 +4957,6 @@ mod tests { assert_eq!(config.memory.consolidation_provider, "genkit"); assert_eq!(config.memory.consolidation_model, "memory-model"); assert_eq!(config.memory.consolidation_timeout_seconds, 777); - assert_eq!(config.memory.daily_schedule, "04:05"); - assert_eq!(config.memory.daily_window_hours, 12); assert_eq!(config.memory.worker_concurrency, 2); assert_eq!(config.memory.max_messages_per_run, 55); assert_eq!(config.memory.min_messages_per_run, 20); diff --git a/crates/openplotva-storage/src/lib.rs b/crates/openplotva-storage/src/lib.rs index 30ceb32c..f62b61a1 100644 --- a/crates/openplotva-storage/src/lib.rs +++ b/crates/openplotva-storage/src/lib.rs @@ -1273,7 +1273,7 @@ pub const SQL_RECORD_MEMORY_ENQUEUE_ROLLUPS: &str = r#"WITH grouped_windows AS ( FROM chat_history_entries h LEFT JOIN telegram_chats_effective c ON c.id = h.chat_id WHERE h.bucket_day >= GREATEST($2::timestamptz, $4::timestamptz)::date - AND h.bucket_day < $3::timestamptz::date + AND h.bucket_day <= $3::timestamptz::date AND h.occurred_at >= GREATEST($2::timestamptz, $4::timestamptz) AND h.occurred_at < $3::timestamptz AND h.kind = 'text' @@ -1375,7 +1375,7 @@ ON CONFLICT (source, kind, granularity, bucket_start, dimensions_hash) DO UPDATE metrics = EXCLUDED.metrics, updated_at = CURRENT_TIMESTAMP"#; -pub const SQL_ENSURE_DAILY_MEMORY_RUNS: &str = r#"WITH grouped_windows AS ( +pub const SQL_ENSURE_HOURLY_MEMORY_RUNS: &str = r#"WITH grouped_windows AS ( SELECT h.chat_id, CASE @@ -1387,7 +1387,7 @@ pub const SQL_ENSURE_DAILY_MEMORY_RUNS: &str = r#"WITH grouped_windows AS ( FROM chat_history_entries h LEFT JOIN telegram_chats_effective c ON c.id = h.chat_id WHERE h.bucket_day >= GREATEST($2::timestamptz, $4::timestamptz)::date - AND h.bucket_day < $3::timestamptz::date + AND h.bucket_day <= $3::timestamptz::date AND h.occurred_at >= GREATEST($2::timestamptz, $4::timestamptz) AND h.occurred_at < $3::timestamptz AND h.kind = 'text' @@ -5818,10 +5818,15 @@ impl PostgresMemoryStore { .rows_affected()) } - pub async fn ensure_daily_runs( + /// Enqueue a run for every chat hour that completed before `now`, started + /// within the last `lookback` (at most `retention`), and has enough + /// messages: newest first, within the queue and daily caps. The enqueue + /// rollups still describe whole UTC days of the scanned range. + pub async fn ensure_hourly_runs( &self, now: OffsetDateTime, retention: Duration, + lookback: Duration, policy: openplotva_memory::MemoryRunEnqueuePolicy, ) -> Result { let retention = if retention.is_zero() { @@ -5831,19 +5836,21 @@ impl PostgresMemoryStore { }; let policy = policy.normalized(); let now = now.to_offset(time::UtcOffset::UTC); - let day_end = now.date().midnight().assume_utc(); + let day_start = now.date().midnight().assume_utc(); + let completed_hours_end = day_start + time::Duration::hours(i64::from(now.hour())); let earliest = now - duration_to_time(retention); - let earliest_day_start = earliest.date().midnight().assume_utc(); - let mut window_end = day_end; + let scan_from = earliest.max(completed_hours_end - duration_to_time(lookback)); + let earliest_day_start = scan_from.date().midnight().assume_utc(); + let mut window_end = completed_hours_end; let mut total = 0; let active_queue_depth = self.active_memory_run_count().await?; - let base_runs_enqueued_today = self.base_memory_run_count_created_since(day_end).await?; + let base_runs_enqueued_today = self.base_memory_run_count_created_since(day_start).await?; if memory_enqueue_remaining_capacity(policy, active_queue_depth, base_runs_enqueued_today) .is_none() { self.record_memory_enqueue_backpressure( earliest_day_start, - day_end, + completed_hours_end, active_queue_depth, policy, ) @@ -5852,7 +5859,12 @@ impl PostgresMemoryStore { return Ok(total); } while window_end > earliest_day_start { - let window_start = window_end - time::Duration::days(1); + // One rollup bucket per UTC day: today's runs to the last completed + // hour, then whole earlier days. + let window_start = (window_end - time::Duration::NANOSECOND) + .date() + .midnight() + .assume_utc(); let Some(limit) = memory_enqueue_remaining_capacity( policy, active_queue_depth.saturating_add(i64::try_from(total).unwrap_or(i64::MAX)), @@ -5870,7 +5882,7 @@ impl PostgresMemoryStore { ) .await?; total += self - .ensure_daily_run_window(window_start, window_end, earliest, policy, limit) + .ensure_run_window(window_start, window_end, scan_from, policy, limit) .await?; window_end = window_start; } @@ -5942,7 +5954,7 @@ impl PostgresMemoryStore { Ok(()) } - async fn ensure_daily_run_window( + async fn ensure_run_window( &self, window_start: OffsetDateTime, window_end: OffsetDateTime, @@ -5950,7 +5962,7 @@ impl PostgresMemoryStore { policy: openplotva_memory::MemoryRunEnqueuePolicy, daily_limit: i32, ) -> Result { - Ok(sqlx::query(SQL_ENSURE_DAILY_MEMORY_RUNS) + Ok(sqlx::query(SQL_ENSURE_HOURLY_MEMORY_RUNS) .bind(openplotva_memory::PROMPT_VERSION) .bind(window_start) .bind(window_end) @@ -11616,18 +11628,18 @@ mod tests { assert!( super::SQL_LIST_MEMORY_CARDS.contains("($8::timestamptz IS NULL OR (recorded_at <= $8") ); - assert!(super::SQL_ENSURE_DAILY_MEMORY_RUNS.contains("date_trunc('hour', h.occurred_at)")); - assert!(super::SQL_ENSURE_DAILY_MEMORY_RUNS.contains("COALESCE(c.is_forum, false)")); - assert!(super::SQL_ENSURE_DAILY_MEMORY_RUNS.contains("canonical_thread_id")); - assert!(super::SQL_ENSURE_DAILY_MEMORY_RUNS.contains("HAVING count(*) >= $5")); + assert!(super::SQL_ENSURE_HOURLY_MEMORY_RUNS.contains("date_trunc('hour', h.occurred_at)")); + assert!(super::SQL_ENSURE_HOURLY_MEMORY_RUNS.contains("COALESCE(c.is_forum, false)")); + assert!(super::SQL_ENSURE_HOURLY_MEMORY_RUNS.contains("canonical_thread_id")); + assert!(super::SQL_ENSURE_HOURLY_MEMORY_RUNS.contains("HAVING count(*) >= $5")); assert!( - super::SQL_ENSURE_DAILY_MEMORY_RUNS + super::SQL_ENSURE_HOURLY_MEMORY_RUNS .contains("ORDER BY range_start_at DESC, message_count DESC") ); assert!( - super::SQL_ENSURE_DAILY_MEMORY_RUNS.contains("WHERE NOT EXISTS (\n SELECT 1") + super::SQL_ENSURE_HOURLY_MEMORY_RUNS.contains("WHERE NOT EXISTS (\n SELECT 1") ); - assert!(super::SQL_ENSURE_DAILY_MEMORY_RUNS.contains("LIMIT $6")); + assert!(super::SQL_ENSURE_HOURLY_MEMORY_RUNS.contains("LIMIT $6")); assert!( super::SQL_SELECT_MEMORY_RUN_MESSAGES .contains("WHEN COALESCE(c.is_forum, false) AND e.thread_id <> 0") @@ -11650,7 +11662,7 @@ mod tests { assert!(super::SQL_RECORD_MEMORY_ENQUEUE_BACKPRESSURE.contains("'backpressure'")); assert!(super::SQL_RECORD_MEMORY_ENQUEUE_BACKPRESSURE.contains("'min_range_start'")); assert!(super::SQL_LIST_MEMORY_ENQUEUE_ROLLUPS.contains("kind = 'consolidation_enqueue'")); - assert!(super::SQL_ENSURE_DAILY_MEMORY_RUNS.contains("h.bucket_day >=")); + assert!(super::SQL_ENSURE_HOURLY_MEMORY_RUNS.contains("h.bucket_day >=")); assert!(super::SQL_SKIP_SUPERSEDED_MEMORY_RUNS.contains("prompt_version <> $1")); assert!( super::SQL_SELECT_MEMORY_RUN_MESSAGES @@ -12012,6 +12024,117 @@ mod tests { assert_eq!(r#ref.message_id, 20); } + #[tokio::test] + async fn live_memory_runs_are_enqueued_through_the_last_complete_hour() + -> Result<(), Box> { + let Ok(dsn) = env::var("OPENPLOTVA_TEST_POSTGRES_DSN") else { + return Ok(()); + }; + let pool = PgPoolOptions::new() + .max_connections(2) + .connect(&dsn) + .await?; + super::run_migrations_on(&pool).await?; + let store = super::PostgresMemoryStore::new(pool.clone()); + let suffix = SystemTime::now().duration_since(UNIX_EPOCH)?.as_nanos() as i64; + let chat_id = -940_000_000 - (suffix % 1_000_000); + // 2030-03-04 10:05 UTC: 09:00-10:00 is complete, 10:00-11:00 is not. + let now = time::OffsetDateTime::from_unix_timestamp(1_898_849_100)?; + let hour = time::Duration::HOUR; + let messages = [ + now - hour + time::Duration::minutes(5), + now - time::Duration::minutes(4), + now - 12 * hour + time::Duration::minutes(25), + ]; + + let result: Result<(), Box> = async { + let mut message_id = 0; + for window in messages { + sqlx::query("SELECT ensure_chat_history_partition($1::date)") + .bind(window.date()) + .execute(&pool) + .await?; + for offset in 0..3 { + message_id += 1; + sqlx::query( + "INSERT INTO chat_history_entries \ + (bucket_day, chat_id, thread_id, message_id, entry_id, kind, role, \ + occurred_at, sender_id, payload) \ + VALUES ($1::date, $2, 0, $3, $4, 'text', 'user', $5, 1, \ + '{\"text\":\"hourly memory\"}'::jsonb)", + ) + .bind(window.date()) + .bind(chat_id) + .bind(message_id) + .bind(format!("hourly-{message_id}")) + .bind(window + time::Duration::seconds(offset)) + .execute(&pool) + .await?; + } + } + + let policy = openplotva_memory::MemoryRunEnqueuePolicy { + min_messages_per_run: 3, + max_queued_runs: 1_000_000, + max_daily_enqueued_runs: 1_000_000, + }; + let retention = Duration::from_secs(7 * 24 * 60 * 60); + let windows = || async { + sqlx::query_as::<_, (time::OffsetDateTime, time::OffsetDateTime)>( + "SELECT range_start_at, range_end_at FROM memory_runs \ + WHERE chat_id = $1 ORDER BY range_start_at", + ) + .bind(chat_id) + .fetch_all(&pool) + .await + }; + let yesterday_evening = now - 12 * hour - time::Duration::minutes(5); + let this_morning = now - hour - time::Duration::minutes(5); + + let recent = store + .ensure_hourly_runs(now, retention, Duration::from_secs(2 * 60 * 60), policy) + .await?; + assert!(recent >= 1, "{recent}"); + assert_eq!( + windows().await?, + vec![(this_morning, this_morning + hour)], + "an hourly tick looks back only over the recent hours" + ); + + let full = store + .ensure_hourly_runs(now, retention, retention, policy) + .await?; + assert!(full >= 1, "{full}"); + assert_eq!( + windows().await?, + vec![ + (yesterday_evening, yesterday_evening + hour), + (this_morning, this_morning + hour), + ], + "a full scan fills the retention window; the current hour waits until it completes" + ); + Ok(()) + } + .await; + + sqlx::query("DELETE FROM memory_runs WHERE chat_id = $1") + .bind(chat_id) + .execute(&pool) + .await?; + sqlx::query("DELETE FROM chat_history_entries WHERE chat_id = $1") + .bind(chat_id) + .execute(&pool) + .await?; + sqlx::query( + "DELETE FROM telemetry_rollups \ + WHERE source = 'memory' AND kind = 'consolidation_enqueue' AND bucket_start >= $1", + ) + .bind(now - time::Duration::days(8)) + .execute(&pool) + .await?; + result + } + #[tokio::test] async fn live_memory_store_round_trips_lexical_cards_links_and_episodes_when_postgres_dsn_is_set() -> Result<(), Box> { diff --git a/docs/CODEBASE_MAP.md b/docs/CODEBASE_MAP.md index 14787aea..1a489d38 100644 --- a/docs/CODEBASE_MAP.md +++ b/docs/CODEBASE_MAP.md @@ -332,7 +332,7 @@ queueing glue). | `runtime_llm.rs` | `RuntimeLlmObserver` (single chokepoint → ring + Postgres recorder + run buffer); provider canonicalization from model name; cleanup/scrub workers | | `runtime_llm_runs.rs` | 512-entry run ring for admin "LLM Dialogs" view; 30min stale-run watchdog | | `runtime_llm_analytics.rs` | Postgres analytics reader (raw + hourly rollups, percentile reconstruction) | -| `memory_runtime.rs` (~5.4k) | Memory consolidation wiring: extractor composition, daily-run scheduling, claim/process pipeline, 6-op resolution resolver (`write_memory_extraction_cards`), +2 workers (archival 1h, dup-collapse 6h) | +| `memory_runtime.rs` (~5.4k) | Memory consolidation wiring: extractor composition, hourly run scheduling, claim/process pipeline, 6-op resolution resolver (`write_memory_extraction_cards`), +2 workers (archival 1h, dup-collapse 6h) | | `agent_runtime.rs` (~1.5k) | `Reasoner`/`AgentTools` adapters; image agent optimizer provider; `SongContextGatherer` (history + memory context for the song director) | | `history_summary.rs` (~1.7k) | Chat-history summary service (edge pre-summary cascade, AIFarm↔GenKit fallback) | | `embedder.rs` | Discovery-routed embedder + process-wide cooling circuit breaker (shared by retrieval/shield/consolidation) | @@ -633,7 +633,7 @@ sequenceDiagram ```mermaid flowchart LR - Sched[Daily scheduler / taskman
MemoryConsolidation job] --> Claim[claim_run
lease 15min] + Sched[Hourly scheduler HH:02 / taskman
MemoryConsolidation job] --> Claim[claim_run
lease 15min] Claim --> Gate{embedder available?
shared breaker} Gate -- no --> Release[release run
no attempt burned] Gate -- yes --> Load[load messages
binary-search fit to token budget] diff --git a/docs/superpowers/specs/2026-10-12-backend-lanes-design.md b/docs/superpowers/specs/2026-10-12-backend-lanes-design.md index 44b5de33..c4ce00a3 100644 --- a/docs/superpowers/specs/2026-10-12-backend-lanes-design.md +++ b/docs/superpowers/specs/2026-10-12-backend-lanes-design.md @@ -142,6 +142,16 @@ percentiles, in the admin routing status and the runtime GraphQL API. Routing ev `lane_bypass` records a budget skip with the forecast; `lane_config_mismatch` records invariant violations. +### Hourly memory runs + +Memory runs already cover one chat hour each, but they were enqueued once a day for the previous +UTC day, which put a whole day of memory work into a morning burst. The scheduler now ticks at +`HH:02` UTC and enqueues every chat hour that completed since the last tick. An hourly tick scans +the last two completed hours. The startup tick and the `03:02` tick scan the whole retention +window, for hours that a longer outage or the daily cap left without a run. The daily cap still +counts runs created since midnight UTC, and the enqueue rollups still describe whole UTC days. +`MEMORY_DAILY_SCHEDULE` and `MEMORY_DAILY_WINDOW_HOURS` are removed. + ## Migrations 1. Rename the pools to `aifarm-gpu0-dialog`, `aifarm-gpu2-utility` and `aifarm-gpu1-media`, @@ -180,6 +190,5 @@ violations. ## Out of Scope - The GPU1 arbiter's internal scheduling. -- Hourly memory windows (the next change, after this one). - Discovery server changes beyond the read-only width check. - Startup catch-up and restart cost. From c693e2c4e92ae56373691fc913f625327fd13787 Mon Sep 17 00:00:00 2001 From: Valeriy Selitskiy <239034+iamwavecut@users.noreply.github.com> Date: Mon, 12 Oct 2026 22:28:47 +0200 Subject: [PATCH 17/18] Point the memory backfill live tests at the current GPU2 reasoner Their fixtures still used the llama.cpp reasoner that 2730ef8 replaced, so the backfill resolved no subject-merge target and both tests failed whenever a test database was configured. The fresh-migration test drops its migration 180 round trip: later migrations rename and widen the pool that the 180 down migration requires, and the storage live tests cover 180 directly. --- crates/openplotva-app/src/model_routing.rs | 205 ++++++++++----------- 1 file changed, 96 insertions(+), 109 deletions(-) diff --git a/crates/openplotva-app/src/model_routing.rs b/crates/openplotva-app/src/model_routing.rs index 7bb5a5d8..ae48a224 100644 --- a/crates/openplotva-app/src/model_routing.rs +++ b/crates/openplotva-app/src/model_routing.rs @@ -3170,66 +3170,77 @@ mod tests { role TEXT NOT NULL, provider_model_id BIGINT NOT NULL, weight INTEGER, \ fallback_order INTEGER, canary_percent INTEGER, enabled BOOLEAN NOT NULL, \ inference_overrides JSONB NOT NULL DEFAULT '{}'::jsonb, \ - cb_failure_threshold INTEGER NOT NULL, cb_cooldown_ms INTEGER NOT NULL, \ - CONSTRAINT reject_maple CHECK (provider_model_id <> 21)\ + cb_failure_threshold INTEGER NOT NULL, cb_cooldown_ms INTEGER NOT NULL\ )", ) .execute(&pool) .await?; - sqlx::raw_sql( + // The reasoner (model 20) is the subject-merge target; Maple (model 21) + // serves the route before the backfill and shares the GPU2 pool. + sqlx::raw_sql(sqlx::AssertSqlSafe(format!( "INSERT INTO workflows VALUES \ ('memory_subject_merge', 'chat', FALSE, 3, 60000, TRUE); \ INSERT INTO llm_capacity_pools \ (id, name, max_concurrency, description) VALUES \ - (7, 'aifarm-gpu2-utility', 1, 'shared test pool'); \ + (7, '{MEMORY_GPU2_POOL}', 1, 'shared test pool'); \ INSERT INTO llm_providers \ (id, name, kind, protocol, enabled) VALUES \ - (2, 'aifarm-llamacpp-gpu2', 'chat', 'openai_compat', TRUE), \ + (2, '{reasoner_provider}', 'chat', 'openai_compat', TRUE), \ (3, 'aifarm-maple', 'chat', 'openai_compat', TRUE); \ INSERT INTO provider_models \ (id, provider_id, model_name, capabilities, pool_id, enabled, config) VALUES \ - (20, 2, 'vibethinker-3b', ARRAY['chat', 'tools'], NULL, TRUE, '{}'), \ + (20, 2, '{reasoner_model}', ARRAY['chat', 'tools'], NULL, TRUE, '{{}}'), \ (21, 3, 'maple-preview-2bit-mlx', ARRAY['chat', 'tools'], 7, TRUE, \ jsonb_build_object('managed_by', 'maple_gpu2_cutover_v1', 'origin', 'fresh')); \ INSERT INTO workflow_assignments \ (id, workflow_key, scope, role, provider_model_id, weight, fallback_order, \ enabled, cb_failure_threshold, cb_cooldown_ms) VALUES \ - (99, 'memory_subject_merge', 'global', 'primary', 20, 100, NULL, \ - TRUE, 5, 30000)", - ) + (99, 'memory_subject_merge', 'global', 'primary', 21, 100, NULL, \ + TRUE, 5, 30000); \ + ALTER TABLE workflow_assignments \ + ADD CONSTRAINT reject_reasoner CHECK (provider_model_id <> 20)", + reasoner_provider = crate::agent_runtime::LOCAL_REASONER_PROVIDER_NAME, + reasoner_model = crate::agent_runtime::DEFAULT_LOCAL_REASONER_MODEL, + ))) .execute(&pool) .await?; assert!(backfill_memory_routing_cascades(&pool).await.is_err()); - let surviving_assignment: i64 = sqlx::query_scalar( - "SELECT id FROM workflow_assignments WHERE workflow_key = 'memory_subject_merge'", + let surviving_assignment: (i64, i64) = sqlx::query_as( + "SELECT id, provider_model_id FROM workflow_assignments \ + WHERE workflow_key = 'memory_subject_merge'", ) .fetch_one(&pool) .await?; - assert_eq!(surviving_assignment, 99); + assert_eq!(surviving_assignment, (99, 21)); let guard_count: i64 = sqlx::query_scalar( "SELECT count(*) FROM app_settings \ WHERE key = 'llm.routing.memory_subject_merge_cascade_v1'", ) .fetch_one(&pool) .await?; - let pool_count: i64 = sqlx::query_scalar( - "SELECT count(*) FROM llm_capacity_pools WHERE name = 'aifarm-gpu2-utility'", - ) - .fetch_one(&pool) - .await?; + let pool_count: i64 = + sqlx::query_scalar("SELECT count(*) FROM llm_capacity_pools WHERE name = $1") + .bind(MEMORY_GPU2_POOL) + .fetch_one(&pool) + .await?; + let reasoner_pool: Option = + sqlx::query_scalar("SELECT pool_id FROM provider_models WHERE id = 20") + .fetch_one(&pool) + .await?; assert_eq!(guard_count, 0); assert_eq!(pool_count, 1); + assert_eq!(reasoner_pool, None, "the failed backfill rolled back"); - sqlx::query("ALTER TABLE workflow_assignments DROP CONSTRAINT reject_maple") + sqlx::query("ALTER TABLE workflow_assignments DROP CONSTRAINT reject_reasoner") .execute(&pool) .await?; assert!(backfill_memory_routing_cascades(&pool).await?); - let subject_count: i64 = sqlx::query_scalar( - "SELECT count(*) FROM workflow_assignments \ - WHERE workflow_key = 'memory_subject_merge'", + let subject_targets: Vec<(String, i64)> = sqlx::query_as( + "SELECT role, provider_model_id FROM workflow_assignments \ + WHERE workflow_key = 'memory_subject_merge' ORDER BY id", ) - .fetch_one(&pool) + .fetch_all(&pool) .await?; let extraction_count: i64 = sqlx::query_scalar( "SELECT count(*) FROM workflow_assignments \ @@ -3244,8 +3255,11 @@ mod tests { sqlx::query_scalar("SELECT pool_id FROM provider_models ORDER BY id") .fetch_all(&pool) .await?; - assert_eq!(subject_count, 2); - assert_eq!(extraction_count, 0); + assert_eq!(subject_targets, vec![("primary".to_owned(), 20)]); + assert_eq!( + extraction_count, 0, + "extraction does not wait for the hosted cascade models" + ); assert_eq!(guards, vec![MEMORY_SUBJECT_MERGE_CASCADE_KEY.to_owned()]); assert_eq!(model_pools, vec![Some(7), Some(7)]); let maple_config: Value = @@ -3260,7 +3274,7 @@ mod tests { } #[tokio::test] - async fn live_fresh_migrations_memory_backfill_and_maple_rollback_round_trip() + async fn live_fresh_migrations_then_backfill_route_subject_merge_to_the_gpu2_reasoner() -> Result<(), Box> { let Ok(dsn) = std::env::var("OPENPLOTVA_TEST_POSTGRES_DSN") else { return Ok(()); @@ -3269,7 +3283,7 @@ mod tests { .max_connections(1) .connect(&dsn) .await?; - let schema = format!("maple_fresh_runtime_{}", std::process::id()); + let schema = format!("memory_backfill_fresh_{}", std::process::id()); sqlx::raw_sql(sqlx::AssertSqlSafe(format!( "DROP SCHEMA IF EXISTS {schema} CASCADE; CREATE SCHEMA {schema}" ))) @@ -3284,97 +3298,70 @@ mod tests { .max_connections(1) .connect_with(options) .await?; - openplotva_storage::run_migrations_on(&pool).await?; - - let maple: (i64, i64, i64, Value) = sqlx::query_as( - "SELECT provider.id, model.id, model.pool_id, model.config FROM llm_providers AS provider JOIN provider_models AS model ON model.provider_id = provider.id WHERE provider.name = 'aifarm-maple' AND model.model_name = 'maple-preview-2bit-mlx'", - ) - .fetch_one(&pool) - .await?; - assert_eq!(maple.3["origin"], json!("fresh")); - - let vibe_provider_id: i64 = sqlx::query_scalar( - "INSERT INTO llm_providers (name, kind, protocol, runtime_hint, discovery_service_name, discovery_endpoint_name, api_key_ref, enabled, config) VALUES ('aifarm-llamacpp-gpu2', 'chat', 'openai_compat', 'llama_cpp', 'llm-openai-qwen27b-gguf', 'chat_completions', 'DIALOG_API_KEY', TRUE, '{}'::jsonb) RETURNING id", - ) - .fetch_one(&pool) - .await?; - let vibe_model_id: i64 = sqlx::query_scalar( - "INSERT INTO provider_models (provider_id, model_name, display_name, capabilities, enabled, config) VALUES ($1, 'vibethinker-3b', 'VibeThinker 3B', ARRAY['chat', 'tools'], TRUE, '{}'::jsonb) RETURNING id", - ) - .bind(vibe_provider_id) - .fetch_one(&pool) - .await?; - assert!(backfill_memory_routing_cascades(&pool).await?); - let maple_config_after_backfill: Value = - sqlx::query_scalar("SELECT config FROM provider_models WHERE id = $1") - .bind(maple.1) - .fetch_one(&pool) - .await?; - assert_eq!(maple_config_after_backfill, maple.3); - let vibe_pool_id: i64 = - sqlx::query_scalar("SELECT pool_id FROM provider_models WHERE id = $1") - .bind(vibe_model_id) - .fetch_one(&pool) - .await?; - assert_eq!(vibe_pool_id, maple.2); - - let maple_assignment_id: i64 = sqlx::query_scalar( - "SELECT assignment.id FROM workflow_assignments AS assignment WHERE assignment.workflow_key = 'memory_subject_merge' AND assignment.provider_model_id = $1", - ) - .bind(maple.1) - .fetch_one(&pool) - .await?; - let maple_event_id: i64 = sqlx::query_scalar( - "INSERT INTO llm_routing_events (severity, event_type, workflow_key, provider_id, model_id, dedupe_key, summary, detail) VALUES ('info', 'maple_fresh_test', 'memory_subject_merge', $1, $2, 'maple-fresh-runtime-round-trip', 'Maple fresh runtime test', jsonb_build_object('preserve', TRUE)) RETURNING id", - ) - .bind(maple.0) - .bind(maple.1) - .fetch_one(&pool) - .await?; - - sqlx::raw_sql(include_str!( - "../../../migrations/180_maple_gpu2_cutover.down.sql" - )) - .execute(&pool) - .await?; - let rolled_back_assignment: (i64, bool, bool) = sqlx::query_as( - "SELECT provider_model_id, enabled, inference_overrides ? 'disabled_by_maple_gpu2_cutover_v1_rollback' FROM workflow_assignments WHERE id = $1", - ) - .bind(maple_assignment_id) - .fetch_one(&pool) - .await?; - assert_eq!(rolled_back_assignment, (maple.1, false, true)); - let preserved_event: (i64, Option, Option) = sqlx::query_as( - "SELECT id, provider_id, model_id FROM llm_routing_events WHERE id = $1", - ) - .bind(maple_event_id) - .fetch_one(&pool) - .await?; - assert_eq!( - preserved_event, - (maple_event_id, Some(maple.0), Some(maple.1)) - ); + let result: Result<(), Box> = async { + openplotva_storage::run_migrations_on(&pool).await?; + let reasoner_model_id: i64 = sqlx::query_scalar( + "SELECT model.id FROM llm_providers AS provider \ + JOIN provider_models AS model ON model.provider_id = provider.id \ + WHERE provider.name = $1 AND model.model_name = $2", + ) + .bind(crate::agent_runtime::LOCAL_REASONER_PROVIDER_NAME) + .bind(crate::agent_runtime::DEFAULT_LOCAL_REASONER_MODEL) + .fetch_one(&pool) + .await?; + let gpu2_pool_id: i64 = + sqlx::query_scalar("SELECT id FROM llm_capacity_pools WHERE name = $1") + .bind(MEMORY_GPU2_POOL) + .fetch_one(&pool) + .await?; + let maple_before: (i64, Value) = sqlx::query_as( + "SELECT model.id, model.config FROM llm_providers AS provider \ + JOIN provider_models AS model ON model.provider_id = provider.id \ + WHERE provider.name = 'aifarm-maple' \ + AND model.model_name = 'maple-preview-2bit-mlx'", + ) + .fetch_one(&pool) + .await?; - sqlx::raw_sql(include_str!( - "../../../migrations/180_maple_gpu2_cutover.up.sql" - )) - .execute(&pool) - .await?; - let reapplied_assignment: (i64, bool, bool) = sqlx::query_as( - "SELECT provider_model_id, enabled, inference_overrides ? 'disabled_by_maple_gpu2_cutover_v1_rollback' FROM workflow_assignments WHERE id = $1", - ) - .bind(maple_assignment_id) - .fetch_one(&pool) - .await?; - assert_eq!(reapplied_assignment, (maple.1, true, false)); + assert!(backfill_memory_routing_cascades(&pool).await?); + let subject_targets: Vec<(String, i64)> = sqlx::query_as( + "SELECT role, provider_model_id FROM workflow_assignments \ + WHERE workflow_key = 'memory_subject_merge' AND scope = 'global' \ + ORDER BY id", + ) + .fetch_all(&pool) + .await?; + assert_eq!( + subject_targets, + vec![("primary".to_owned(), reasoner_model_id)] + ); + let reasoner_pool: Option = + sqlx::query_scalar("SELECT pool_id FROM provider_models WHERE id = $1") + .bind(reasoner_model_id) + .fetch_one(&pool) + .await?; + assert_eq!(reasoner_pool, Some(gpu2_pool_id)); + let maple_config_after: Value = + sqlx::query_scalar("SELECT config FROM provider_models WHERE id = $1") + .bind(maple_before.0) + .fetch_one(&pool) + .await?; + assert_eq!(maple_config_after, maple_before.1); + assert!( + !backfill_memory_routing_cascades(&pool).await?, + "the guard makes the backfill run once" + ); + Ok(()) + } + .await; pool.close().await; sqlx::raw_sql(sqlx::AssertSqlSafe(format!("DROP SCHEMA {schema} CASCADE"))) .execute(&admin) .await?; admin.close().await; - Ok(()) + result } #[test] From 4a543d54fae3f0efc1b0df531b0fb312ae58f74b Mon Sep 17 00:00:00 2001 From: Valeriy Selitskiy <239034+iamwavecut@users.noreply.github.com> Date: Mon, 12 Oct 2026 23:21:05 +0200 Subject: [PATCH 18/18] Cap lane waits of interactive and user-tool requests without a deadline Only the dialog sets a deadline, so history summary, YouTube, the media optimizer, vision, image and music requests would otherwise wait in a busy lane without limit. They now get an admission deadline 300 s after the run starts, the old all-busy cap; background work still waits for its lane. --- crates/openplotva-app/src/routed_attempts.rs | 69 +++++++++++++++++-- docs/CODEBASE_MAP.md | 2 +- .../specs/2026-10-12-backend-lanes-design.md | 6 +- 3 files changed, 69 insertions(+), 8 deletions(-) diff --git a/crates/openplotva-app/src/routed_attempts.rs b/crates/openplotva-app/src/routed_attempts.rs index 2695fff6..0eafaebb 100644 --- a/crates/openplotva-app/src/routed_attempts.rs +++ b/crates/openplotva-app/src/routed_attempts.rs @@ -10,6 +10,12 @@ use openplotva_llm::router::{ }; use serde_json::{Value, json}; +/// Admission deadline, from the start of a run, for an interactive or +/// user-tool request whose caller set no deadline: a lane that stays busy this +/// long hands the request to the next candidate or to the caller's retry. +/// Background work without a deadline keeps its place in the queue. +const NO_DEADLINE_LANE_WAIT: Duration = Duration::from_secs(300); + /// Minimum share of the caller's budget an attempt must have received for a /// deadline cut to count against the provider's circuit breaker. const ATTEMPT_BREAKER_MIN_SLICE: Duration = Duration::from_secs(30); @@ -106,16 +112,17 @@ impl RoutedAttemptWalker { /// Admit one candidate through its lane queue. A free lane is taken at once. /// A candidate with an open breaker is never waited for. Otherwise the - /// request waits in the lane queue until `deadline - reserve`, keeping the - /// workflow's fallback reserve only while another candidate follows, and - /// moves on at once when the forecast wait already exceeds that budget. - /// Lane time is charged to the caller's deadline, not to the retry wall. + /// request waits in the lane queue until `admission_deadline - reserve`, + /// keeping the workflow's fallback reserve only while another candidate + /// follows, and moves on at once when the forecast wait already exceeds that + /// budget. Lane time is charged to the deadline, not to the retry wall. #[allow(clippy::too_many_arguments)] async fn admit_lane( &self, pool: Option, live: bool, has_next_candidate: bool, + admission_deadline: Option, route: &WorkflowRoute, context: &RoutedRequestContext, attempt: &Attempt, @@ -133,7 +140,7 @@ impl RoutedAttemptWalker { } else { Duration::ZERO }; - let budget = context.deadline.map(|deadline| { + let budget = admission_deadline.map(|deadline| { deadline .saturating_duration_since(Instant::now()) .saturating_sub(reserve) @@ -245,6 +252,10 @@ impl RoutedAttemptWalker { }; let started = Instant::now(); + let admission_deadline = context.deadline.or_else(|| { + (context.lane_priority.unwrap_or(route.lane_priority) != LanePriority::Background) + .then(|| started + NO_DEADLINE_LANE_WAIT) + }); let max_hops = route.retry.max_hops.max(1) as usize; let mut started_attempts = 0usize; let mut total_slot_wait = Duration::ZERO; @@ -377,6 +388,7 @@ impl RoutedAttemptWalker { pool_id, live, has_next_candidate, + admission_deadline, route, &context, attempt, @@ -2127,6 +2139,53 @@ mod tests { assert_eq!(pools.occupancy()[0].queued, [0, 0, 0]); } + #[tokio::test(start_paused = true)] + async fn a_request_without_a_deadline_stops_waiting_at_the_lane_cap() { + let mut snapshot = pooled_snapshot(3); + snapshot.models.truncate(1); + snapshot.assignments.truncate(1); + let pools = Arc::new(PoolRegistry::new()); + let walker = walker_with_pools(&snapshot, Arc::clone(&pools)); + let _busy = pools.try_acquire(Some(1)).expect("occupy the only lane"); + let started = tokio::time::Instant::now(); + + let (result, executed) = run_recording(&walker, None, None).await; + + assert!(executed.is_empty()); + assert!(matches!( + result, + Err(RoutedAttemptRunError::Routing(error)) + if error.kind() == RoutedRoutingErrorKind::CapacityWaitTimeout + )); + let waited = started.elapsed(); + assert!( + waited > NO_DEADLINE_LANE_WAIT - Duration::from_secs(1) + && waited <= NO_DEADLINE_LANE_WAIT, + "{waited:?}" + ); + } + + #[tokio::test(start_paused = true)] + async fn background_work_without_a_deadline_waits_past_the_lane_cap() { + let mut snapshot = pooled_snapshot(3); + snapshot.models.truncate(1); + snapshot.assignments.truncate(1); + let pools = Arc::new(PoolRegistry::new()); + let walker = walker_with_pools(&snapshot, Arc::clone(&pools)); + let _busy = pools.try_acquire(Some(1)).expect("occupy the only lane"); + + let waiting = tokio::time::timeout( + NO_DEADLINE_LANE_WAIT * 2, + run_recording(&walker, None, Some(LanePriority::Background)), + ) + .await; + + assert!( + waiting.is_err(), + "background work keeps its place in the queue" + ); + } + #[tokio::test] async fn an_interactive_request_goes_ahead_of_queued_background_work() { let mut snapshot = pooled_snapshot(3); diff --git a/docs/CODEBASE_MAP.md b/docs/CODEBASE_MAP.md index 1a489d38..5308864f 100644 --- a/docs/CODEBASE_MAP.md +++ b/docs/CODEBASE_MAP.md @@ -700,7 +700,7 @@ cascades, and the audit-preserving Maple cutover). Virtual-message subsystem - **Provider canonicalization** (`runtime_llm.rs`): the observer overrides the stamped provider from the **model name** (`vram.cloud/*`→`vram-cloud`, `vibethinker-3b`→`aifarm-llamacpp-gpu2`, `maple-preview-2bit-mlx`→`aifarm-maple`) so historical backfill and forward classification cannot drift. Rollback-only legacy ids stay classifiable but are not active route targets. - **Maple request compatibility** (`aifarm.rs`): an `mlx` runtime hint strips unsupported DRY controls before Discovery serialization while retaining `repeat_penalty` and float `top_k`; migration 180 keeps Maple and Bonsai as distinct rows so routing-event history remains truthful. - **Lane pools are not semaphores** (`router/capacity.rs`): the custom `PoolCell` survives live resize without losing in-flight counts (tokio Semaphore shrink needs `forget_permits` debt). `apply()` rewrites `max` in place and serves waiters of a grown pool; outstanding permits hold their own `Arc`. A released lane is handed to the head waiter (`workflows.lane_priority`, then arrival order) through a oneshot; a permit sent to a waiter that gave up releases itself on drop. -- **Fallback on failure or budget, not on load** *(2026-10-12, design `docs/superpowers/specs/2026-10-12-backend-lanes-design.md`)*: the walker waits in a busy candidate's lane queue until `deadline − workflows.fallback_reserve_ms` (no reserve for the last candidate; no deadline = no limit) and passes it over only when its breaker is open or the forecast wait (`waiters ahead / width × mean hold`) exceeds that budget (`lane_bypass` event). Lane waits (`lane_wait` event) don't consume hops, trip breakers or count against the retry wall. Evidence for queueing on the primary: with instant primary skips Gemma sat at 0.45–0.59 slot occupancy while ~750 turns/day were shed to the tool-less fallback; the 3090 replay showed the GPU is prefill-bound, so queueing on the primary is the lever, not more sequences. +- **Fallback on failure or budget, not on load** *(2026-10-12, design `docs/superpowers/specs/2026-10-12-backend-lanes-design.md`)*: the walker waits in a busy candidate's lane queue until `deadline − workflows.fallback_reserve_ms` (no reserve for the last candidate; without a caller deadline, background work waits without limit and other priorities get an admission deadline 300 s after the run starts) and passes it over only when its breaker is open or the forecast wait (`waiters ahead / width × mean hold`) exceeds that budget (`lane_bypass` event). Lane waits (`lane_wait` event) don't consume hops, trip breakers or count against the retry wall. Evidence for queueing on the primary: with instant primary skips Gemma sat at 0.45–0.59 slot occupancy while ~750 turns/day were shed to the tool-less fallback; the 3090 replay showed the GPU is prefill-bound, so queueing on the primary is the lever, not more sequences. - **Breaker fairness ≥30s slice**: a deadline-cut charges the breaker only if `remaining ≥ 30s` — a fallback handed the last seconds of a turn must not open its circuit in lockstep with a degraded primary. - **WhiteCircle wrapper is a mandatory passthrough**: must implement `as_chat_step()→Some(self)` or the session engine silently falls back to the legacy loop. Audit fires once per session on `iteration ≤ 1`. - **Two distinct Telegram KDFs** (`openplotva-web`): Login Widget uses `secret = SHA256(bot_token)`; WebApp initData uses `secret = HMAC_SHA256("WebAppData", bot_token)`. Do not confuse. diff --git a/docs/superpowers/specs/2026-10-12-backend-lanes-design.md b/docs/superpowers/specs/2026-10-12-backend-lanes-design.md index c4ce00a3..a5003b26 100644 --- a/docs/superpowers/specs/2026-10-12-backend-lanes-design.md +++ b/docs/superpowers/specs/2026-10-12-backend-lanes-design.md @@ -115,8 +115,10 @@ For each candidate in chain order, the walker: The admission budget is the time left until the request deadline minus a fallback reserve: the time the next candidate needs to answer. The reserve is a per-workflow value (`workflows.fallback_reserve_ms`): 30 s for interactive and user-tool workflows, 0 for -background ones. A request without a deadline (background work) waits in its queue without a -limit and never falls back because of load. +background ones. Background work without a deadline waits in its queue without a limit and never +falls back because of load. An interactive or user-tool request whose caller sets no deadline gets +an admission deadline 300 s after its run starts (the old all-busy cap), so a lane that stays busy +hands it to the next candidate or to the caller's retry instead of holding it forever. `primary_slot_wait`, the busy-skip pass and the per-construction `max_slot_wait` disappear; the policy is the same for all 25 walker constructions. Unpooled candidates are admitted at once.