Repository navigation
Conversation
ecky-l
force-pushed
the
178-models-in-subprocesses
branch
from
October 7, 2026 18:59
a21d26b to
1ecfff9
Compare
…r using max_position_embedding And warning, when configured --context-window is larger than max_position_embeddings.
…ocesses. New package src/engine/worker/ (4 modules + __init__), plus tests: protocol.py: JSON-lines wire protocol (LOAD/GENERATE/CANCEL/PING/UNLOAD ↔ READY/LOAD_OK/ITEM/DONE/ERROR/FATAL/BYE/CANCEL_ACK), error types, and the non-recoverable-error classifier (CL_OUT_OF_RESOURCES, CL_DEVICE_LOST, CL_DRIVER_ERROR, CL_INVALID_EVENT, "could not execute a primitive", "ProgramBuilder build failed") supervisor.py: WorkerSupervisor, spawns the child with the parent's interpreter/sys.path/cwd, reads stdout, forwards the child's stderr to openarc.log, PING watchdog (30 s / 5 s), death detection, and respawn with a per-load budget (2) worker_process.py: Child entry point, builds the real OVGenAI_VLM inside the worker process and runs its actual generate_type (streamers, tool-call parsing, metrics all unchanged). OpenVINO is imported lazily there; a _StubVLM (env OPENARC_WORKER_STUB=1) lets tests exercise the full pipe without GPU/OpenVINO worker_client.py: RemoteOVGenAI_VLM facade — same surface WorkerRegistry calls (generate_type/cancel/unload_model/async load_model)
…rvised worker subprocesses. + Worker machinery generalized: protocol.py: new TRANSCRIBE op (reuses the ITEM/DONE/ERROR/FATAL flow — Whisper's transcribe is an async generator just like generate_type) worker_process.py: model construction now dispatches by model_type (VLM/LLM/Whisper, each lazy-imported in the child); gen-config validation per engine (OVGenAI_GenConfig for VLM/LLM with the messages/input_ids normalization, OVGenAI_WhisperGenConfig for Whisper); run dispatch is method-name driven (generate_type / transcribe); the stub now implements transcribe (control keywords via the decoded audio payload) supervisor.py: begin_generate → begin_run(op, ...) — fully engine-agnostic worker_client.py: shared RemoteOVGenAI base (supervisor plumbing, generate_type, transcribe, cancel, unload, respawn callback) + thin RemoteOVGenAI_VLM / RemoteOVGenAI_LLM / RemoteOVGenAI_Whisper subclasses for dispatch.
…Increased default to 256MiB.
…add --worker-line-limit,--wll)
…orker subprocesses. Stage 3 implementation: openvino only models on worker subprocess. New package src/engine/worker/plain/ — built on inheritance from the stage 1–2 machinery: protocol.py: re-exports the shared framing byte-identically; adds OP_RUN (→ one MSG_RESULT dict) and OP_RUN_STREAM (→ MSG_ITEM per audio chunk, then MSG_DONE). Audio crosses the pipe as base64 float32 samples supervisor.py: PlainWorkerSupervisor(WorkerSupervisor) — 3 lines; only the child entry point differs, the whole lifecycle/watchdog/respawn ladder is inherited worker_process.py: _PlainWorker(_Worker) inherits the stdin loop, load flow, gen-config validation, FATAL/ERROR ladder, cancel, shutdown; overrides only model construction, the gen-config contracts, and the RUN/RUN_STREAM ops (Qwen3-TTS's blocking sync generate_stream is advanced in a worker thread so the event loop stays free) worker_client.py: RemoteOV_Kokoro / RemoteOVQwen3ASR / RemoteOVQwen3TTS inherit a new RemoteEngine base extracted from RemoteOVGenAI — same public surface as the real engines (chunk_forward_pass yields torch-tensor StreamChunks, transcribe → (text, metrics, segments), generate → (wav, sr)) model_registry.create_model_instance now picks the facade by (engine, model_type) before instantiating — important because e.g. OVQwen3ASR.__init__ reads config.json and creates an ov.Core, which must not run in the server process. worker_registry dispatches Kokoro/ASR/TTS packets to the facades; infer_qwen3_tts_stream handles both the sync (in-process) and async (worker) generator shapes, so stream=True on /audio/speech is unchanged.
…t of worker respawn. If a worker process for a model crashes, the supervisor tries to respawn the process and keeps track of the number of respawn attempts. The parameter --worker-max-respawns specifies the max limit of these respawn attempts. If <= 0, the worker is respawned endlessly.
… failure. Including a tiny model (4.07Mb).
…out of the main log
The worker's full logging -- a load/inference failure's traceback, and any
OpenVINO/OpenCL C-level stderr -- now goes to its own file next to the main log,
named after it with -worker-<model> spliced in before the .log
(openarc.log -> openarc-worker-<model>.log):
* WorkerSupervisor._worker_log_file() derives that path from OPENARC_LOG_FILE
(+ the per-model name, used verbatim -- only /, \ and NUL reduced to - so
the file cannot leave the log directory) and passes it to the child as
OPENARC_WORKER_LOG_FILE; launch_server publishes the resolved main-log path
to that env var at startup, so the path is known even when the shell never
set OPENARC_LOG_FILE.
* The worker redirects its logging + native stderr onto that file at startup
(_redirect_stderr_to, via fd-level dup2).
* Only the worker writes to its own file -- never the server or superclass --
and the supervisor's stderr pump no longer forwards the worker's output into
the main log, so the main log keeps just the lifecycle lines.
* The traceback demotion from the prior spawn-logging commit is undone: the
worker logs the FULL trace at ERROR again; it now lands in its own file
instead of flooding the main log.
…ive worker restarts. The user.total_tokens usage per session is returned by the model worker from GenAI. When the worker crashed and was respawn, the token usage was reset to 0 from a clients viewpoint, which accounts for wrong request sizing and auto.compaction behavior on client side. To solve this (optionally), the server must keep track of this value in a session object per requests from a client that belong to a session in the clients sense of sessions. Clients may add a special HTTP header to indicate the session by id, to which the sent request belongs. Openarc is now able to accept this header key with a new parameter --session-id-header,sih to the "serve start" command, and if set, creates a session to keep token counters for that session. This survives a worker restart and then the session's token counters is reported to the client instead of the new model worker's 0 - until the client resends the whole conversation with the next request, and the model's token context is updated.
…kflow Powershell scripts to runtests via ssh on a windows host (or VM). The ssh connection must be set up so that a prepared venv is activated with the OpenArc dependencies and the work folder is mapped into the VM or setup via samba if working with a real remote host - and the CWD must be set to this mapped directory when ssh is executed.
… subprocess was not properly terminated.
ecky-l
force-pushed
the
178-models-in-subprocesses
branch
from
October 7, 2026 21:05
1ecfff9 to
4675e47
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Missing parts: