Backend lanes and hourly memory runs

[iamwavecut]

Summary

Each remote inference backend is now loaded at its physical width. A request waits in its own backend's queue. It moves to a fallback model only when the primary fails or when waiting would use up the request's time budget, not when the primary is busy.

Design: docs/superpowers/specs/2026-10-12-backend-lanes-design.md. Plan: docs/superpowers/plans/2026-10-12-backend-lanes.md.

Changes

• Lane queue (router/capacity.rs):
  • Each lane pool has a priority queue: interactive, then user tool, then background, and arrival order within a priority.
  • A released lane goes directly to the head waiter. The registry-wide Notify wake-all race is gone.
  • Cancellation is safe both before and after the handoff.
  • Resizing keeps in-flight accounting.
  • Each pool keeps an EWMA of lane hold time and p50/p95 of recent admission waits.
• Walker (routed_attempts.rs):
  • A busy candidate is waited for until deadline − workflows.fallback_reserve_ms. The last candidate keeps no reserve.
  • Background work without a deadline waits without a limit. An interactive or user-tool request whose caller sets no deadline gets an admission deadline 300 s after its run starts, the old all-busy cap.
  • The walker passes a candidate over only when its breaker is open or when the forecast wait (waiters ahead / width × mean hold) exceeds the budget. Such a skip records a lane_bypass event.
  • Lane waits record lane_wait events.
  • The per-call slot waits and their config are removed: primary_slot_wait, max_slot_wait and DIALOG_AIFARM_POOL_PRIMARY_CAPACITY_WAIT_MS.
• Priorities:
  • Migration 199 adds workflows.lane_priority and workflows.fallback_reserve_ms.
  • History summary, YouTube, media optimizer, image and music workflows are user tools.
  • Memory extraction and subject merge are background work with no fallback reserve.
  • Memory-write embeddings and redactions pass background priority from the caller.
• Lane pools (migration 200):
  • Pools are named aifarm-<device>-<purpose>: aifarm-gpu0-dialog, aifarm-gpu2-utility, aifarm-gpu1-media, aifarm-embedder (8) and aifarm-gpu1-privacy (4).
  • GPU1 draw, music and speech recognition share one sequential lane.
  • Pool ids do not change. Previous names and pool assignments are recorded for the down migration.
• Workers:
  • Dialog workers = the dialog primary's lanes × PERSISTENT_QUEUE_DIALOG_WORKERS_PER_LANE (default 2).
  • Memory workers = the memory primary's lanes × PERSISTENT_QUEUE_MEMORY_WORKERS_PER_LANE (default 2). PERSISTENT_QUEUE_MEMORY_CONSOLIDATION_WORKERS is the floor, and the default cap is now 32.
  • Both fleets run under the worker supervisor and resize on a routing reload.
  • PERSISTENT_QUEUE_DIALOG_UNPOOLED_SHARE is removed. Fallback capacity no longer counts as throughput.
• Lane check (lane_checks.rs):
  • At startup and on every reload, the bot compares lane pools with the farm job service's per-service limits.
  • It reports a routed farm model without a pool, a service split across pools, and a pool wider than its services.
  • The result goes to lane_config_mismatch events and the lane_config readiness detail. It never fails readiness.
• Hourly memory runs:
  • Memory runs already cover one chat hour each, but they were enqueued once a day for the previous UTC day.
  • The scheduler now ticks at HH:02 UTC and enqueues every chat hour that completed. An hourly tick scans the last two completed hours. Startup and the 03:02 tick scan the whole retention window.
  • The daily cap still counts runs created since midnight UTC, and the enqueue rollups still describe whole UTC days.
  • The history scan now includes the partition of its end day, so today's completed hours are found.
  • MEMORY_DAILY_SCHEDULE and MEMORY_DAILY_WINDOW_HOURS are removed.
• Tests: the memory backfill live tests use the current GPU2 reasoner fixtures.
• Dependencies: quinn-proto 0.11.15 → 0.11.19 (lockfile only) for RUSTSEC-2026-0338 to RUSTSEC-2026-0342, which the dependency gate reported on this branch.
• Observability:
  • The admin routing status pools carry name, queued per priority, mean_hold_ms and wait percentiles.
  • The runtime GraphQL API adds llmLanes.

Compatibility

• Routing events: primary_slot_wait and capacity_pool_wait are replaced by lane_wait and lane_bypass. capacity_wait_timeout and its retryable message prefix stay.
• Env: DIALOG_AIFARM_POOL_PRIMARY_CAPACITY_WAIT_MS, PERSISTENT_QUEUE_DIALOG_UNPOOLED_SHARE, MEMORY_DAILY_SCHEDULE and MEMORY_DAILY_WINDOW_HOURS are ignored if they are still set.
• Migration 200 renames pools in place. The memory routing backfill constant follows the new GPU2 name.

Verification

• cargo fmt --all -- --check: clean.
• cargo clippy --workspace --all-targets -- -D warnings: clean.
• cargo test --workspace: all pass. One run hit a flaky temp-directory failure in openplotva-prompts prompt_store_resolves_custom_root_and_partials, a crate this branch does not touch; it passed on rerun.
• cargo audit --ignore RUSTSEC-2023-0071, cargo deny check and cargo machete: clean.
• Live Postgres tests on a fresh pgvector/pgvector:pg17, run serially:
  • openplotva-storage outbox_roundtrip_is_idempotent_ordered_and_fenced live_: 30 passed. The new live_memory_runs_are_enqueued_through_the_last_complete_hour covers the recent and full scans and the current-hour boundary.
  • openplotva-app model_routing: 28 passed. Two backfill live tests were already broken on main: their fixtures used the GPU2 reasoner that 2730ef8 replaced, and CI does not run them. This branch points them at the current reasoner. The fresh-migration test drops its migration 180 round trip, because later migrations rename and widen the pool that the 180 down migration requires. The storage live tests cover migration 180 directly.
• Migrations 199 and 200: up, then down, inside one rolled-back transaction on a copy of the production routing configuration. After the up migrations, the five lane pools, their widths, the priorities and the model assignments were as intended. After the down migrations, 0 rows differed in pools, models or workflows.

Deployment risk

• Dialog turns that used to spill to the GPU2 fallback or the hosted API on load now wait in the GPU0 queue, up to the turn deadline minus 30 s. Hosted API traffic should drop. Turn latency at peak load may rise by the queue wait.
• A voice message's transcription now waits for a running draw or song on GPU1. It goes ahead of queued draws and songs.
• Memory now runs about 16 workers at background priority on GPU2, behind dialog fallbacks and user tools. Its load is spread over the day instead of a morning burst.
• Each hourly tick scans today's history partition for the rollups and the last two hours for runs. The daily full scan costs what one daily pass cost before.

[github-actions]

	Warnings
⚠️	Large PR (5867 changed lines). Consider splitting unless this is generated/mechanical.

	Messages
📖	Danger checked 33 changed file(s).

Generated by 🚫 dangerJS against 4a543d5

[github-actions]

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 5 🔵🔵🔵🔵🔵

🔒 No security concerns identified

⚡ Recommended focus areas for review Unbounded lane waits without a deadline admit_lane derives its wait budget solely from context.deadline; with no deadline it passes budget = None to PoolRegistry::acquire, which waits without limit. The PR simultaneously removes every per-workflow bound (primary_slot_wait 60 s/15 s for history_summary, youtube, media optimizer; max_slot_wait). Any caller of these workflows whose RoutedRequestContext does not set a deadline (not visible in the diff for the history-summary/YouTube/media-optimizer/embedding/redaction call sites) now queues behind a busy lane forever instead of falling back after the old bounded wait, turning a slow backend into a hung request. Uncertainty: if all of these callers set deadlines outside the shown hunks, the risk does not materialize — worth verifying before merge. async fn admit_lane( &self, pool: Option<PoolId>, live: bool, has_next_candidate: bool, route: &WorkflowRoute, context: &RoutedRequestContext, attempt: &Attempt, total_slot_wait: &mut Duration, ) -> Option<PoolPermit> { if let Some(permit) = self.pools.try_acquire(pool) { return Some(permit); } if !live { return None; } let priority = context.lane_priority.unwrap_or(route.lane_priority); let reserve = if has_next_candidate { route.fallback_reserve } else { Duration::ZERO }; let budget = context.deadline.map(|deadline| { deadline .saturating_duration_since(Instant::now()) .saturating_sub(reserve) });

[iamwavecut]

Unbounded lane waits without a deadline. Valid. Only the dialog passes a deadline. History summary, YouTube, the media optimizer, vision, image and music requests would have waited in a busy lane without limit. Fixed in 4a543d5:

• An interactive or user-tool request with no caller deadline gets an admission deadline 300 s after its run starts. This is the old all-busy cap (DEFAULT_MAX_SLOT_WAIT).
• When another candidate follows, the request moves to it after 300 s − fallback_reserve.
• The last candidate returns the retryable CapacityWaitTimeout, so taskman jobs requeue as before.
• Background work (memory) still waits for its own lane by design.

Tests: a_request_without_a_deadline_stops_waiting_at_the_lane_cap and background_work_without_a_deadline_waits_past_the_lane_cap.

[iamwavecut]

Danger: large PR. Kept as one change on purpose. About 1,900 of the changed lines are the design spec and the implementation plan. The lane queue, walker, worker sizing, pool migrations and hourly memory runs have to deploy together, because the walker relies on the new pool names, the workflow columns and the worker counts.