Out of process workers for model execution on openvino/openvino.genai

I recognized it by getting errors CL_OUT_OF_RESOURCES from openvino or similar: A model unload happens. I found this a little strange, because you would not expect from a server that your model is suddenly unloaded and tried to change this behavior to a model reload. There I recognized that the model cannot be reloaded because the OpenVINO model context was corrupt and cannot be restored. So the only possibility was to restart the server.
This made me think that it would be a better approach to execute the loaded models in a subprocess instead of the server process itself, so that in case of an unrecoverable model failure, i.e. the out of resource error above, the subprocess can be killed and restarted by the server - instead of crashing the whole server. This would solve quite a few issues that I had, and others apparently as well (#153).

The idea is that for every model a subprocess is started with a model worker, which does the heavy inference lifting. Communication works via pipes and an a very simple text protocol, which enables the client to call openvino or genai functions, transfer data, etc. via a few commands. Pipes are an established mechanism and work on *nix and windows likewise. When the client receives an error from the model worker which he classifies as unrecoverable (i.e. CL_OUT_OF_RESOURCES), he kills the subprocess and creates a new one, then loads the same model configuration in the new worker.

So the feature request is to implement subprocess workers for the openvino/genai inference work.

Note:

I am already working on this in branch separate-processes in my fork. Implemented it for all supported model types and *nix pipes first. So far I can confirm that it works quite well on Linux and the VLM pipeline (using Qwen-3.8), including the reload on CL_OUT_OF_RESOURCES errors. It needs more testing and some hardening/bugfixing though, and windows support is missing.

[SearchSavior#178] Run OpenVINO GenAI VLMs in supervised worker subpr…

[SearchSavior#178] Run OpenVINO GenAI LLM- and Whisper models in supe…

[SearchSavior#178] Fixed schema flaw in OVGenAI_GenConfig (default pr…

[SearchSavior#178] Transfer limit to/from worker exceeded bug fixed. …

[SearchSavior#178] Add per-model worker_line_limit override (openarc …

[SearchSavior#178] Run OpenVINO Kokoro/ASR/TTS models in supervised w…

I like this, please rebase seperate-processes so I can review. This has potential to be a great addition, thank you for investigating.

Here it is :). I added an option --worker-line-limit, which is rather important because the default 64k (?) limit for data transfer to subprocesses turned out to be not enough. With the new model config.yaml approach there might be a little more work necessary to integrate it in the config.yaml file. Pls feel free to add whatever is necessary.