feat(libsy): core interfaces and examples#17
Conversation
|
messiaen
changed the title
messiaen
force-pushed
the
grclark/feat/libsy-routers-poc
branch
from
81c5d5c to
9261a09
Compare
WalkthroughAdds a new Changeslibsy orchestration crate and proxy demo
Estimated code review effort: 4 (Complex) | ~75 minutes Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (5)
demo/libsy-proxy/src/main.rs (2)
184-193: 🎯 Functional Correctness | 🔵 Trivial | 💤 Low value
completion_textdoesn't handle array-typecontent.
Value::as_strreturnsNonewhencontentis an array of content blocks (e.g.[{"type":"text","text":"..."}]), yielding an empty completion viaunwrap_or_default(). For a demo this is likely fine, but if the upstream ever returns array-type content the proxy will silently produce blank responses. Consider reusingcontent_to_textas a fallback.Proposed fix
fn completion_text(body: &Value) -> Option<String> { body.get("choices") .and_then(Value::as_array) .and_then(|choices| choices.first()) .and_then(|choice| choice.get("message")) .and_then(|message| message.get("content")) - .and_then(Value::as_str) - .map(str::to_string) + .and_then(|content| { + content.as_str().map(str::to_string).or_else(|| content_to_text(content)) + }) }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@demo/libsy-proxy/src/main.rs` around lines 184 - 193, The completion_text helper only reads string content, so it returns nothing when the assistant message content is an array of blocks. Update completion_text to fall back to content_to_text when message.content is not a string, using the existing content_to_text helper and keeping the current choices/message traversal intact so array-based responses still produce text.
150-166: 🎯 Functional Correctness | 🔵 Trivial | 💤 Low value
extract_promptignores system prompts and prior conversation turns.Only the last
role: "user"message is extracted;role: "system"messages and earlier user turns are discarded. For a demo this is acceptable, but routing quality may degrade for multi-turn conversations where the system prompt or earlier context determines complexity. Consider concatenating system + user messages into the prompt passed to the classifier, or document this as a known demo limitation.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@demo/libsy-proxy/src/main.rs` around lines 150 - 166, `extract_prompt` currently only returns the last user message and drops system or earlier conversation context, which can reduce classifier quality for multi-turn chats. Update the prompt extraction logic in `extract_prompt` to either combine relevant `system` plus prior `user` turns into the returned prompt, or explicitly preserve the current behavior but document it as a known demo limitation near `extract_prompt` and the inbound body handling.crates/libsy/Cargo.toml (2)
20-24: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueRedundant
tokiodev-dependency entry.The main dependency already enables
features = ["full"], which is a superset of["macros", "rt"]. Cargo unifies features across dependencies and dev-dependencies, so this entry adds nothing.♻️ Proposed fix
[dev-dependencies] -tokio = { version = "1", features = ["macros", "rt"] }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/libsy/Cargo.toml` around lines 20 - 24, Remove the redundant tokio entry from the dev-dependencies section in Cargo.toml because the main tokio dependency already enables a superset of the needed features. Keep the existing tokio and tokio-stream setup, but delete the extra dev-dependency specification so Cargo only resolves tokio once.
14-21: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick winNarrow the Tokio feature set. The main
tokiodependency can dropfullhere; the library/examples only needmacros,rt,sync, andtime, and the#[tokio::test(flavor = "multi_thread")]cases can keeprt-multi-threadin dev-dependencies.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/libsy/Cargo.toml` around lines 14 - 21, The tokio dependency is using an overly broad feature set, so narrow it in the Cargo.toml dependency list for libsy. Update the main tokio entry to use only the features needed by the library and examples (macros, rt, sync, time), and keep rt-multi-thread only where the #[tokio::test(flavor = "multi_thread")] cases need it under dev-dependencies.crates/libsy/src/lib.rs (1)
485-490: 🚀 Performance & Scalability | 🔵 Trivial | 💤 Low valueConsider making channel capacity configurable.
Both the stream channel (line 485) and promise channel (line 490) use a hardcoded capacity of 10. For algorithms that fan out to many models concurrently (e.g., ensemble with >10 candidates), this creates backpressure. While not a deadlock (the consumer drains
CallLlmsteps), a configurable capacity or a builder pattern would give hosts control over buffering.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/libsy/src/lib.rs` around lines 485 - 490, The `orchestrate` channel setup currently hardcodes capacity 10 for both `stream_tx/stream_rx` and `promise_tx/promise_rx`, which limits buffering under high fan-out workloads. Update the `orchestrate` flow in `libsy::lib` to take a configurable channel capacity (or expose it through the existing builder/config path), then use that value when creating both `tokio::sync::mpsc::channel` instances so hosts can tune backpressure behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@demo/libsy-proxy/src/main.rs`:
- Around line 210-217: `build_orchestrator` currently uses
`std::env::var("ANTHROPIC_API_KEY").ok()`, which hides a missing key and lets
the proxy start in a broken state. Update `build_orchestrator` to validate
`ANTHROPIC_API_KEY` at startup by requiring the env var before constructing
`EndpointConfig`, and return an explicit error with a clear message if it is
absent or empty. Keep the fix localized around `build_orchestrator` and the
`EndpointConfig` initialization so startup fails fast instead of deferring the
auth failure to request time.
---
Nitpick comments:
In `@crates/libsy/Cargo.toml`:
- Around line 20-24: Remove the redundant tokio entry from the dev-dependencies
section in Cargo.toml because the main tokio dependency already enables a
superset of the needed features. Keep the existing tokio and tokio-stream setup,
but delete the extra dev-dependency specification so Cargo only resolves tokio
once.
- Around line 14-21: The tokio dependency is using an overly broad feature set,
so narrow it in the Cargo.toml dependency list for libsy. Update the main tokio
entry to use only the features needed by the library and examples (macros, rt,
sync, time), and keep rt-multi-thread only where the #[tokio::test(flavor =
"multi_thread")] cases need it under dev-dependencies.
In `@crates/libsy/src/lib.rs`:
- Around line 485-490: The `orchestrate` channel setup currently hardcodes
capacity 10 for both `stream_tx/stream_rx` and `promise_tx/promise_rx`, which
limits buffering under high fan-out workloads. Update the `orchestrate` flow in
`libsy::lib` to take a configurable channel capacity (or expose it through the
existing builder/config path), then use that value when creating both
`tokio::sync::mpsc::channel` instances so hosts can tune backpressure behavior.
In `@demo/libsy-proxy/src/main.rs`:
- Around line 184-193: The completion_text helper only reads string content, so
it returns nothing when the assistant message content is an array of blocks.
Update completion_text to fall back to content_to_text when message.content is
not a string, using the existing content_to_text helper and keeping the current
choices/message traversal intact so array-based responses still produce text.
- Around line 150-166: `extract_prompt` currently only returns the last user
message and drops system or earlier conversation context, which can reduce
classifier quality for multi-turn chats. Update the prompt extraction logic in
`extract_prompt` to either combine relevant `system` plus prior `user` turns
into the returned prompt, or explicitly preserve the current behavior but
document it as a known demo limitation near `extract_prompt` and the inbound
body handling.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: eb1842e9-fae5-42b4-81b4-8b9d0ea849d5
⛔ Files ignored due to path filters (1)
Cargo.lockis excluded by!**/*.lock,!Cargo.lock
📒 Files selected for processing (12)
Cargo.tomlcrates/libsy/Cargo.tomlcrates/libsy/DESIGN.mdcrates/libsy/README.mdcrates/libsy/examples/research_agent.rscrates/libsy/examples/research_agent_core.rscrates/libsy/src/ensemble.rscrates/libsy/src/lib.rscrates/libsy/src/llm_class.rscrates/libsy/src/rand.rsdemo/libsy-proxy/Cargo.tomldemo/libsy-proxy/src/main.rs
messiaen
assigned
messiaen,
grahamking and
nachiketb-nvidia and unassigned
grahamking,
sabhatinas,
nachiketb-nvidia and
messiaen
messiaen
force-pushed
the
grclark/feat/libsy-routers-poc
branch
from
1df67dc to
aa72aa0
Compare
Signed-off-by: Greg Clark <grclark@nvidia.com>
messiaen
force-pushed
the
grclark/feat/libsy-routers-poc
branch
from
fcdc0ad to
eab29f6
Compare
messiaen
changed the title
4 participants
Summary
Adds
libsy— a lightweight, embeddable Rust SDK for multi-LLM agentoptimization, with routing as the first use case. An algorithm decides —
statefully, using more than the request — which model(s) to call and how, and
never performs the call itself ("ask, don't call"), keeping the core provider-
and transport-agnostic. Ships with reference routers, runnable examples, and a
demo HTTP proxy that wires libsy routing into Switchyard's existing server /
translation crates.
What's included
crates/libsy/src/lib.rs)OrchAlgo(the algorithm) +MultiLlmOrchestrator(the driver).LlmTargettargets that either serve a call (carry anLlmClient) or offload it as a promise the host fulfills.DecisionTracetrait-object decisions (uniform to read, downcast when thealgorithm is known).
orchestrate_direct(all targets self-serve →(trace, response)) and streamingorchestrate(offloaded calls surface asCallLlmpromises;set_response(Ok/Err)propagates results and errors).Arc<dyn OrchAlgo>with no lock;process_request(&self)runs requests in parallel; each
orchestraterun gets its own promise channelvia a task-local, so concurrent requests' offloaded calls never cross.
RandomOrchAlgo(rand.rs) — uniform random, single call.LlmClassifierOrchAlgo(llm_class.rs) — classify → strong/weak, fail-open.EnsembleOrchAlgo(ensemble.rs) — stateful fan-out → judge → commit.research_agent.rs(self-serving /orchestrate_direct) andresearch_agent_core.rs(offloaded / streaming).README.md(code-first scope) andDESIGN.md(narrative design).demo/libsy-proxy) — a real HTTP proxy: Switchyard's server +translation crates handle the OpenAI / Anthropic / Responses APIs and format
translation, while all routing is libsy's classifier algorithm. Shows
libsy embedded in a production-shaped I/O stack without depending on
Switchyard's own routing.
Architecture
request → OrchAlgo.process_request (N × target.call) → (DecisionTrace, response)
│
serve (has client)│ or offload → CallLlm promise → host fulfills
The core owns no HTTP; the single piece of I/O it doesn't own is the
LlmClienttrait the host implements. The demo proxy supplies one backed by Switchyard's
OpenAI-compatible backend.
Testing
cargo test -p libsy— unit tests covering the reference algorithms, parallelrequest processing (multi-thread barrier), and offload error propagation.
demo/libsy-proxyverified end-to-end:/health,/v1/models, and routedPOST /v1/chat/completions+/v1/messages.Scope note
This branch is far ahead of the current public
main(05a4e979), so the rawdiff also includes a large amount of already-landed internal work unrelated to
libsy (stage_router rename, skill-distillation, latency-service, release infra,
etc.). Recommend retargeting the PR base to an up-to-date
main(or splittingthe libsy POC into its own PR) so review focuses on the ~10 new libsy/demo files.
Closes #
How tested
uv run ruff check .cleanuv run mypy switchyardcleanuv run pytest tests/greenChecklist
snake_caseof the primary class.switchyard/__init__.py.__all__if intended for downstream use.--helpupdated if customer-facing surface changed.Signed-off-by: Your Name <email>) per the DCO.Notes for reviewers
Anything reviewers should pay extra attention to — risky paths, follow-up tickets, intentional trade-offs.
Summary by CodeRabbit
New Features
Documentation