Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 20 additions & 1 deletion docs/configure.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ OpenArc uses a YAML based configuration system! Models are added to `config.yaml
There are a few different blocks in a models config that accept different parameters. Some will be common across implementations while others are specific. `openarc add` writes to these blocks; `openarc add --help` shows one help panel per key.

### load_config
- engine: ovgenai, openvino, optimum
- engine: ovgenai, openvino, optimum, strata
- model_type: llm, vlm, whisper, kokoro, emb, rerank, qwen3_tts_*
- device: device for this model
- tool_call_parser: gemma4, qwen35, hermes
Expand Down Expand Up @@ -163,6 +163,25 @@ openarc add \
--runtime-config '{"PERFORMANCE_HINT": "LATENCY"}'
```

### Strata (remote proxy)

`engine=strata` does not load a local model: it proxies requests to a running [Strata](https://github.com/Niko1221/Strata) server's OpenAI-compatible API, which lets OpenArc front models OpenVINO cannot run (GGUF MoE models, for example). `--model-path` carries the endpoint base URL instead of an IR directory, and `--device` is a placeholder. The served model id is discovered from the server's `GET /v1/models` at load time. Alternatively, set the URL as `endpoint` in `--runtime-config` and use any placeholder for `--model-path`.

The proxy deliberately speaks only the stable OpenAI wire protocol (`GET /v1/models`, `POST /v1/chat/completions`, SSE deltas, `usage` events) and nothing Strata-specific, so it is decoupled from Strata's release cadence — and works with any other OpenAI-compatible server that supports those endpoints. All proxy logic lives in the single module `src/engine/strata_proxy.py`.

Metrics for strata models (`ttft`, throughputs, token counts) are measured client-side from the upstream usage event; they carry a `proxy: true` marker.

```
openarc add \
--model-name flash-next \
--model-path http://localhost:8080 \
--engine strata \
--model-type llm \
--device strata \
--temperature 0.7 \
--max-tokens 4096
```

### Qwen3-TTS

All 3 flavors of qwen3-tts are supported, with a rich set of configuration options. PRs to improve performance or other enhancements are welcome
Expand Down
18 changes: 12 additions & 6 deletions src/cli/groups/add.py
Original file line number Diff line number Diff line change
Expand Up @@ -64,11 +64,11 @@ def _fail(extra: str = "") -> NoReturn:
help='Public facing name of the model.')
@click.option('--model-path', '--m',
required=True,
help='Path to OpenVINO IR converted model.')
help="Path to OpenVINO IR converted model. For engine=strata this is the endpoint base URL instead (e.g. http://localhost:8080).")
@click.option('--engine', '--en',
type=click.Choice(['ovgenai', 'openvino', 'optimum']),
type=click.Choice(['ovgenai', 'openvino', 'optimum', 'strata']),
required=True,
help='Engine used to load the model (ovgenai, openvino, optimum)')
help='Engine used to load the model (ovgenai, openvino, optimum, strata)')
@click.option('--model-type', '--mt',
type=click.Choice([
'llm', 'vlm', 'whisper', 'qwen3_asr', 'kokoro',
Expand Down Expand Up @@ -135,8 +135,13 @@ def add(ctx, model_path, model_name, engine, model_type, device, runtime_config,
keeps using the contract's own default.
"""

# Validate model path
if not validate_model_path(model_path):
# Validate model path. engine=strata carries an endpoint URL in model_path,
# not an on-disk IR directory, so the file check does not apply.
if engine == "strata":
if "://" not in model_path:
console.print(f"[red]engine=strata expects an endpoint URL in --model-path (e.g. http://localhost:8080), got: {model_path}[/red]")
ctx.exit(1)
elif not validate_model_path(model_path):
console.print(f"[red]Model file check failed! {model_path} does not contain openvino model files OR your chosen path is malformed. Verify chosen path is correct and acquired model files match source on the hub, or the destination of converted model.[/red]")
ctx.exit(1)

Expand All @@ -157,7 +162,8 @@ def add(ctx, model_path, model_name, engine, model_type, device, runtime_config,
# An explicit CACHE_DIR in --runtime-config wins. Resolved to an absolute
# path here because runtime_config values are passed to OpenVINO verbatim,
# unlike model_path they are not resolved against the config file.
if model_type in ("llm", "vlm"):
# engine=strata has no IR (model_path is a URL), so no cache applies.
if model_type in ("llm", "vlm") and engine != "strata":
parsed_runtime_config.setdefault(
"CACHE_DIR", str((Path(model_path) / "model_cache").resolve())
)
Expand Down
6 changes: 4 additions & 2 deletions src/cli/modules/server_config.py
Original file line number Diff line number Diff line change
Expand Up @@ -263,11 +263,13 @@ def get_all_models(self) -> Dict[str, Dict[str, Any]]:

def _resolve_model_paths(self, model_config: Dict[str, Any]) -> Dict[str, Any]:
"""Return a copy of model_config with relative model_path, draft_model_path,
and cache_dir made absolute by joining them onto the config file's directory."""
and cache_dir made absolute by joining them onto the config file's directory.

URL values (e.g. a strata engine's endpoint in model_path) are left alone."""
resolved = dict(model_config)

path = resolved.get("model_path")
if path and not Path(path).is_absolute():
if path and "://" not in path and not Path(path).is_absolute():
resolved["model_path"] = str((self.config_file.parent / path).resolve())

draft_model_path = resolved.get("draft_model_path")
Expand Down
Loading