#178 models in subprocesses

.github/workflows/windows-worker-tests.yml

@@ -0,0 +1,139 @@
+# Windows worker tests -- the "how do we test Windows from a Linux box" answer.
+#
+# This project has no local Windows environment, but the worker's Windows
+# behaviour is split in src/engine/worker/platform_support.py into a host-
+# independent DECISION and one host-specific EXECUTION:
+#
+#   * The DECISION (which platform, hide the per-worker console, ";"-joined
+#     PYTHONPATH, TerminateProcess) is pure and is asserted for BOTH platforms on
+#     any machine by tests/unit/test_worker_platform_unit.py -- which already runs
+#     in the normal ubuntu `Run Tests` job (it is under tests/unit). So the Windows
+#     *decision* is already covered on Linux.
+#
+#   * The one thing Linux genuinely cannot do is the EXECUTION: resolve the
+#     description into a real subprocess STARTUPINFO / STARTF_USESHOWWINDOW /
+#     CREATE_NO_WINDOW and actually spawn the child with it. That runs HERE, on a
+#     real windows-latest host.
+#
+# SCOPE -- why this job does NOT run the whole tests/unit suite
+# ----------------------------------------------------------------
+# A previous version ran `pytest tests/unit` in its entirety, which pulled in the
+# ~30 model/engine tests that need real downloaded model files, network, GPUs, or
+# Linux-only behaviour -- they fail on a fresh windows-latest runner for reasons
+# that have nothing to do with the worker, so that (correctly) painted the job red.
+# Instead this job runs ONLY the worker, and in two steps for maximum signal from a
+# single (manual, inconvenient) run:
+#
+#   step 1  the PURE worker tests (platform-decision + protocol + log-file) --
+#           no model / OpenVINO / GPU / network. These validate the Windows
+#           *decisions* and are guaranteed to run on any host, so a green here
+#           is unambiguous proof the seam is right.
+#   step 2  the REAL-SPAWN end to end (the stub-lifecycle tests spawn an actual
+#           python.exe child) + the win32-gated integration test -- run with
+#           `if: always()`, so it executes even if step 1 tripped, and any failure
+#           is the meaningful one: does the worker actually work on Windows?
+#           (Needs the OpenVINO/torch deps only where a test imports them; the
+#           stub lifecycle itself sets OPENARC_WORKER_STUB=1, so no model files.)
+#
+# It is `workflow_dispatch`-only on purpose (NOT on push/pull_request): uv sync
+# --frozen installs the full project (it is proven to succeed on windows-latest --
+# step "Sync dependencies" passed) but the broader suite is not yet Windows-ready,
+# so this opt-in job must not gate normal CI.
+name: Windows Worker Tests
+
+on:
+  workflow_dispatch: {}
+
+jobs:
+  windows-worker:
+    name: Worker Windows path (manual)
+    runs-on: windows-latest
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Install uv and Python
+        uses: astral-sh/setup-uv@v5
+        with:
+          # Automatically installs the Python version specified in your pyproject.toml
+          enable-cache: true
+
+      - name: Sync dependencies
+        run: |
+          uv sync --frozen
+          uv sync --group test --frozen
+
+      - name: Worker platform / protocol tests (pure, cross-platform)
+        # No model / OpenVINO / GPU / network: these assert the Windows
+        # *decisions* (hidden console, ';' PYTHONPATH, TerminateProcess) and the
+        # host-independent protocol/log-file behaviour. Guaranteed to run on any
+        # host, so a pass here is a clean signal the seam is correct on Windows.
+        run: |
+          uv run pytest -W ignore -rs -v `
+            tests/unit/test_worker_platform_unit.py `
+            tests/unit/test_worker_protocol_unit.py `
+            tests/unit/test_worker_plain_protocol_unit.py `
+            tests/unit/test_worker_log_file_unit.py
+
+      - name: Worker spawn end-to-end + Windows-only assertions
+        # `if: always()` so this runs even if the previous step tripped. This is
+        # the part that truly needs Windows: the stub-lifecycle tests spawn a real
+        # python.exe child (auto-covering the Windows spawn/terminate), and the
+        # integration test asserts the spawn asked for a hidden console. -rs
+        # shows the skip reasons; -v so any real failure is easy to read.
+        #
+        # A worker redirects its FULL logging + native C-level stderr into its OWN
+        # per-model log file -- the supervisor drains the worker's stderr and
+        # forwards nothing to the main log (see test_worker_log_file_unit.py). So
+        # on a fresh runner the worker's real output is invisible unless we point
+        # that file at the checkout. Set OPENARC_LOG_FILE (a relative path -> the
+        # worker file lands beside it in <workspace>/openarc-logs/) so the next
+        # step can lift it out; pre-create that dir so the best-effort redirect
+        # (which makedirs's it) definitely succeeds.
+        env:
+          OPENARC_LOG_FILE: openarc-logs/openarc.log
+        if: always()
+        run: |
+          echo "Worker logs will land in: $env:OPENARC_LOG_FILE"
+          New-Item -ItemType Directory -Force -Path openarc-logs | Out-Null
+          uv run pytest -W ignore -rs -v `
+            tests/unit/test_remote_vlm_worker_unit.py `
+            tests/unit/test_worker_plain_unit.py `
+            tests/integration/test_worker_windows_integration.py
+
+      - name: Collect per-worker logs into an artifact
+        # The whole point of OPENARC_LOG_FILE above: surface what the worker
+        # ACTUALLY wrote -- its full logging and any native C-level stderr -- so a
+        # 30s load timeout or a startup crash on Windows is no longer invisible.
+        # (The supervisor drains the worker's stderr and forwards nothing, so this
+        # per-worker file is the only place its real output lands.) If no worker
+        # file was produced at all -- e.g. the spawn never even got far enough to
+        # log -- there is nothing to upload and the upload is a harmless no-op.
+        if: always()
+        shell: pwsh
+        run: |
+          echo "Capturing per-worker log files (the worker's real output, not the main log) ..."
+          New-Item -ItemType Directory -Force -Path "$env:GITHUB_WORKSPACE/_worker_logs" | Out-Null
+          Get-ChildItem -Recurse -Force -Path $env:GITHUB_WORKSPACE `
+            -Filter *.log -ErrorAction SilentlyContinue | ForEach-Object {
+              if ($_.FullName -notlike '*_worker_logs*') {
+                Copy-Item -LiteralPath $_.FullName `
+                  -Destination "$env:GITHUB_WORKSPACE/_worker_logs/" -Force -ErrorAction SilentlyContinue
+              }
+            }
+          $found = @(Get-ChildItem -Path "$env:GITHUB_WORKSPACE/_worker_logs" `
+            -Filter *.log -ErrorAction SilentlyContinue)
+          echo "Captured $($found.Count) worker log file(s):"
+          $found | ForEach-Object { "  {0}  ({1} bytes)" -f $_.Name, $_.Length }
+
+      - name: Upload per-worker logs to a build artifact
+        # Downloadable from the run's Artifacts tab: the next time a worker fails
+        # to load on Windows, the real cause (an import/redirect failure, or an
+        # IPC/loop problem) is right here instead of being discarded.
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: openarc-worker-logs
+          path: ${{ github.workspace }}/_worker_logs/*.log
+          if-no-files-found: ignore
+          retention-days: 3

docs/commands.md

@@ -7,7 +7,7 @@ icon: lucide/terminal
 
 After installation run ```openarc --help``` to see focused usage documentation inside the openarc command line tool.
 
-This page contains example commands to help you choose models and configure OpenArc. 
+This page contains example commands to help you choose models and configure OpenArc.
 
 === "add"
 

docs/index.md

@@ -61,6 +61,7 @@ Request-time parameters passed via `extra_body` on the OpenAI-compatible API.
 ## Concepts
 
 - [Tool and Reasoning Parsing](tool_use.md#tool-and-reasoning-parsing)
+- [Out-of-Process Inference Workers](worker-processes.md) — VLM/LLM/Whisper run in supervised worker subprocesses so unload/load always get a clean OpenVINO Core
 
 
 ## Models

docs/worker-processes.md

@@ -0,0 +1,158 @@
+# Out-of-Process Inference Workers
+
+OpenVINO models are not loaded in the server process. Each one runs in a
+**dedicated worker subprocess** owned by a supervisor, and the server talks
+to it over the process's stdin/stdout (one JSON object per line). The worker
+builds and runs the pipeline; the server process never touches OpenVINO for
+that model.
+
+Two model families use two protocol dialects over the same byte-identical
+framing (`src/engine/worker/protocol.py`, extended by
+`src/engine/worker/plain/protocol.py`):
+
+- **OpenVINO GenAI** (VLM, LLM, Whisper) — `OP_GENERATE` / `OP_TRANSCRIBE`
+  stream tokens and segments. The process boundary exists because a wedged
+  GPU plugin poisons the process-wide `ov::Core` and only a new process
+  recovers (see below).
+- **Plain OpenVINO** (Kokoro TTS, Qwen3-ASR, Qwen3-TTS) — `OP_RUN` (one
+  result) and `OP_RUN_STREAM` (audio chunks, base64 float32 samples). The
+  process boundary exists for **segfault isolation**: these engines crash
+  natively, and a crash must take down only the worker, which the supervisor
+  respawns — not the server.
+
+A request line can be very large — a VLM chat request carries the whole
+conversation (base64 images included) inside its JSON, so lines of tens of
+MB are normal. Both sides therefore read the pipe with a line limit of
+`PROTOCOL_LINE_LIMIT` (256 MiB, `protocol.py`), not asyncio's 64 KiB
+`readline()` default; a line beyond the limit is reported as a clear `FATAL`
+rather than killing the reader silently. The limit is per-model and can be
+overridden with `worker_line_limit` in the model config (`openarc add
+--worker-line-limit 256M`), in bytes or with a K/M/G suffix, minimum 64 KiB.
+It bounds the IPC memory a single request can use; requests with larger
+payloads fail with a clear error (the worker respawns). Because it is an IPC
+setting rather than a compilation setting, changing it never invalidates the
+compiled-model cache.
+
+## Why a process boundary
+
+openvino_genai pipelines share a **process-wide singleton `ov::Core`**. When
+the GPU plugin wedges — `CL_OUT_OF_RESOURCES`, device loss, "could not execute
+a primitive", `ProgramBuilder build failed`, ... — no in-process model unload
+or recompile can fix it: the poisoned Core outlives the model. The only way to
+get a clean Core is a new process. That is why:
+
+| event | what happens |
+| --- | --- |
+| **unload** (`openarc unload`, API, or error-triggered) | the worker process is terminated. No wedged state can survive. |
+| **load** (`openarc load`, startup) | a fresh process is spawned and the pipeline is built inside it (fresh Core). |
+| **recoverable inference error** (the worker stays up) | the model is unloaded as before — now guaranteed clean, because unload is a process kill. |
+| **non-recoverable error** (`CL_*` / driver failure) | the worker reports `FATAL` and exits; the supervisor **respawns a fresh process and re-runs the same load**, transparently. |
+| **native crash / OOM-kill** (no `FATAL` message) | the supervisor detects the dead process and respawns it the same way. |
+| **respawn budget exhausted** | the worker keeps crashing past its budget, so the model is unloaded from the registry: readiness drops, and an operator reloads it with a fresh budget (see below). |
+
+The **respawn budget** is per load episode: after a crash the supervisor runs a
+fresh process (re-runs the same load) at most `worker_max_respawns` times before
+it gives up, at which point the model is unloaded from the registry and
+readiness drops. The default is **2** — the worker is reloaded twice, and on the
+**3rd** failure it is quarantined (the historical "does not recover after a few
+retries, so we unload it" behaviour). Lower it for a model you suspect is
+permanently wedged, or set **`0` or a negative number for no limit**, in which
+case the worker is reloaded on **every** crash and is **never** automatically
+unloaded (useful for a model that crashes transiently but would otherwise be
+permanently quarantined). It is per-model config, not an environment variable:
+
+    openarc add --model-name foo --model-path /models/foo --en ovgenai --mt llm --d GPU --worker-max-respawns 0
+
+A PING watchdog (every 30 s, 5 s timeout) kills a worker that stops
+responding, so a wedged-but-alive process is recovered the same way.
+
+The plain-openvino engines (Kokoro, Qwen3-ASR, Qwen3-TTS) do not use
+openvino_genai at all; they compile `ov::Model`s directly with `ov.Core`.
+They do not wedge the Core — they **segfault** (native crashes inside the
+OpenVINO/torch inference paths). A process boundary turns such a crash from
+a server death into a worker death: the supervisor sees the dead process,
+respawns, and the model comes back; the server process itself never crashes.
+Audio crosses the pipe as base64 float32 sample arrays; WAV encoding stays
+in the server, so the public response shapes are unchanged.
+
+## What you will see in `openarc.log`
+
+The worker's own logging is **no longer here** — it goes to the worker's own
+file (below). This file carries only the **supervisor's** view of the worker's
+lifecycle:
+
+- `spawning inference worker: ...`, then `inference worker started (pid=...)` when the process is up (once per start/respawn), then `model loaded (pid=...)` when its model has loaded
+- `inference worker exited unexpectedly (code=...)` + `respawning inference worker (n/<budget>)` on recovery — or `respawning inference worker (respawn #n; no restart limit)` when `worker_max_respawns` is `0`/negative (no limit)
+- `respawn budget exhausted; worker is dead` + the resulting unload — only when `worker_max_respawns` is a positive number; with `0`/negative it never fires
+
+## What you will see in the worker's own log file
+
+Each worker's **full** logging goes to a file beside the main log, named after
+it with `-worker-<model>` spliced in before the `.log` — so `openarc.log`
+becomes `openarc-worker-<model>.log` (in the same directory; the model name is
+used verbatim, only path separators reduced to `-`). Its location follows the
+main log: set `OPENARC_LOG_FILE=/some/dir/openarc.log` and the worker file is
+`/some/dir/openarc-worker-<model>.log`; with it unset, the log defaults to the
+project-root `openarc.log`. The supervisor hands each worker its model name in
+`OPENARC_WORKER_MODEL` and its log-file path in the per-model
+`OPENARC_WORKER_LOGFILE_<model>` (dashes in the model name become underscores in
+both), so the worker reads its own key exactly -- a sibling model's key in the
+server's environment can't be mistaken for its own; a value already set there
+pins a worker's file instead, which the tests use to keep it in a temp dir.
+
+**Only the worker writes to this file** — never the server, never the supervisor
+— so it holds the worker's side alone and the main log keeps only the
+supervisor's. It carries the full log at the worker's normal verbosity: the
+pipeline build/load, every OpenVINO/OpenCL diagnostic (written straight to the
+process's stderr, which the worker redirects into this file at startup rather
+than onto the pipe, which the supervisor only drains, forwarding nothing), and
+the full traceback of any load or inference failure. That is the point: the
+per-request tracebacks that would previously flood `openarc.log` (once per
+concurrent in-flight request, for a worker that is only restarting within its
+respawn budget) now land here instead, leaving `openarc.log` quiet. Workers
+append across respawns, so a file holds the full history of every episode of a
+model's life; rotate it the same way you already rotate `openarc.log`.
+
+## Operator knobs
+
+| environment variable | effect |
+| --- | --- |
+| `OPENARC_OVGENAI_WORKER=0` | master switch: disable worker processes for all OpenVINO GenAI models (VLM/LLM/Whisper) and restore the historical in-process behaviour |
+| `OPENARC_VLM_WORKER=0` | additionally disable the worker process for VLMs only (stage-1 escape hatch) |
+| `OPENARC_OPENVINO_WORKER=0` | master switch: disable worker processes for the plain-openvino engines (Kokoro, Qwen3-ASR, Qwen3-TTS) and restore the historical in-process behaviour |
+| `OPENARC_LOG_FILE` | set the main log file; each worker's own log then lands beside it, named `<base>-worker-<model>.log` |
+| `OPENARC_WORKER_MODEL` | not an operator knob: the supervisor always sets it to the model name, so the child can name its own log-file key exactly |
+| `OPENARC_WORKER_LOGFILE_<model>` | pin one worker's own file instead (model name with `-` written as `_`); it is the value the child reads, and an already-set value wins over the derived path |
+
+The respawn budget has config-file support: `worker_max_respawns` in the model
+config, set via `openarc add --worker-max-respawns` (positive `N` = the number
+of reloads per load episode before the model is quarantined; `0` or negative
+= no limit, the worker is always reloaded and is never automatically unloaded).
+The remaining per-load settings (watchdog timings, unload timeouts) are still
+only `WorkerSupervisor` parameters and will get config-file support in a later
+stage.
+
+## Implementation map
+
+| file | role |
+| --- | --- |
+| `src/engine/worker/protocol.py` | wire protocol, error types, non-recoverable-error classification |
+| `src/engine/worker/supervisor.py` | process lifecycle, protocol session, respawn + watchdog (inherited by both supervisors) |
+| `src/engine/worker/worker_process.py` | GenAI child entry point; builds the pipeline inside the worker (base class for the plain worker) |
+| `src/engine/worker/worker_client.py` | `RemoteEngine` base + `RemoteOVGenAI_*` facades (same surface as the engines) |
+| `src/engine/worker/plain/protocol.py` | plain-OpenVINO dialect: re-exports the shared framing byte-identically, adds `OP_RUN` / `OP_RUN_STREAM` |
+| `src/engine/worker/plain/supervisor.py` | `PlainWorkerSupervisor` — the same lifecycle, plain child entry point |
+| `src/engine/worker/plain/worker_process.py` | plain child entry point; builds Kokoro / Qwen3-ASR / Qwen3-TTS in the worker |
+| `src/engine/worker/plain/worker_client.py` | `RemoteOV_Kokoro` / `RemoteOVQwen3ASR` / `RemoteOVQwen3TTS` facades (same surface as the engines) |
+| `src/server/model_registry.py` | the factory picks the facade by `(engine, model_type)` *before* instantiating (engine constructors have side effects) and wraps GenAI and plain-openvino engines |
+| `src/server/worker_registry.py` | dispatches VLM/LLM/Whisper/Kokoro/ASR/TTS packets to the facades; a *worker death* does not trigger a registry unload (the supervisor owns recovery) |
+
+## Scope (stages 1–3 done)
+
+Stages 1–3 are complete: OpenVINO GenAI **VLM, LLM, and Whisper** and the
+plain-openvino engines **Kokoro TTS, Qwen3-ASR, and Qwen3-TTS** (all three
+voice modes) run out-of-process. The optimum engines (embedding, rerank)
+still load in-process and join the worker pool in stage 4 — their call shape
+returns tuples rather than streaming generators, which the single-result
+`OP_RUN` op already supports. `openarc bench` also still builds its pipeline
+in-process.