feat(translation): harden cross-protocol translation#46
Merged
bbednarski9 merged 2 commits intoJul 9, 2026
Conversation
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
bbednarski9
changed the title
bbednarski9
merged commit 8f9db9a
into
NVIDIA-NeMo:topic/nemo-relay-integration
16 checks passed
This was referenced Jul 9, 2026
rapids-bot Bot
pushed a commit
to NVIDIA/NeMo-Relay
that referenced
this pull request
Jul 13, 2026
> [!WARNING] > **BREAKING CHANGE:** **[Rust source compatibility]** This PR adds `FlowError::Upstream` and new fields to the public `AtofEndpointConfig`, `AtofEndpointSectionConfig`, and `EmitMarkEventParams` types. Downstream exhaustive `FlowError` matches must handle the new variant, and direct struct literals must initialize the new fields or migrate to the provided constructor/builder APIs. Builder-based mark emission and default Relay configurations remain compatible. #### Overview This adds Relay's first-party integration with the [Switchyard](https://github.com/NVIDIA-NeMo/Switchyard) Decision API. The plugin routes buffered and streaming LLM requests through an external Switchyard service, executes the selected Relay-owned backend, and records causal model-routing optimization evidence. Switchyard owns provider-protocol translation through its `switchyard-translation` Rust library. Relay consumes that library only when the native Switchyard plugin is enabled; default Relay builds do not include either the plugin or the translation dependency. - [x] I confirm this contribution is my own work, or I have the right to submit it under this project's license. - [x] I searched existing issues and open pull requests, and this does not duplicate existing work. #### Details ##### Plugin and execution behavior - Add the standalone `nemo-relay-switchyard` first-party plugin. - Normalize stable Relay identity and build `switchyard.routing_request.v1` requests. - Validate selected routes and optional counterfactual baselines against exact Relay-owned backend bindings. - Support enforce and observe-only modes, six request-materialization modes, bounded retry-aware dispatch, protocol-specific trusted fallbacks, and no retry after the first upstream stream item. - Record one model-routing optimization contribution only after the selected route commits. Failed decisions, failed provider attempts, and fallback dispatches do not claim routing savings. - Preserve Switchyard routing metadata, including snapshot freshness, decision source, event count, and turn depth. - Continue using Relay's existing ATOF exporter and lifecycle; the plugin does not install local routers, feature accumulators, or a duplicate event subscriber. ##### Translation ownership and optional dependency Request, buffered-response, and SSE translation across OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages use Switchyard's public `switchyard-translation` library instead of a Relay-local translation engine. The dependency is pinned to the immutable topic-branch commit [`8f9db9a6a47f848cdff1d262276ba25a8ae9cbc8`](NVIDIA-NeMo/Switchyard@8f9db9a), which includes the Decision API series, StageRouter baseline/freshness behavior, and [Switchyard #46](NVIDIA-NeMo/Switchyard#46). The Relay CLI exposes this through `--features switchyard`: - Without that feature, `nemo-relay-cli` contains neither `nemo-relay-switchyard` nor `switchyard-translation`. - With that feature, the plugin and its pinned Switchyard translation library are compiled and registered. - Relay continues to own target bindings, credentials, dispatch, retries, fallbacks, lifecycle, and observability. - Relay retains narrow fail-open policy checks for provider extensions that cannot be translated without data loss. ##### Runtime requirements The current experimental deployment requires a separately running Switchyard service from the pinned topic commit. The plugin does not start or supervise that service. Switchyard availability is not a Relay startup requirement. It is a request-time dependency: connection failures, timeouts, malformed decisions, selected-route drift, unsupported extensions, and missing required identity fail open to the configured same-protocol default. ATOF-backed profiles additionally require Relay's ATOF HTTP exporter to target Switchyard's `/v1/atof/events` endpoint with preserved field names and environment-referenced authentication. The public examples document this service boundary, verify the exact Switchyard revision before startup, and intentionally remain manual compatibility workflows rather than production orchestration. ##### Optimization accounting and observability This PR builds on the plugin-neutral accounting contract merged in [NeMo Relay #389](#389). Applied routing contributions carry the selected and baseline model transition, decision identity, backend and tier, attempt, rollout mode, reason, and router metadata. Observe-only decisions remain visible as non-applied evidence. Relay core combines routing and other plugin contributions at LLM close while retaining explicit token evidence for downstream repricing. Existing requested, decision, retry, fallback, and error routing marks remain available alongside the canonical optimization contribution. The generic ATIF change that attributes `step.model_name` to the effective response model is intentionally separated into [NeMo Relay #399](#399). This PR still ensures translated provider responses carry the actual selected model; #399 independently controls how ATIF presents it. ##### Testing summary - `cargo fmt --all -- --check` - `cargo clippy -p nemo-relay-switchyard --all-targets -- -D warnings` - `cargo clippy -p nemo-relay-cli --features switchyard --all-targets -- -D warnings` - `cargo check -p nemo-relay-cli` — proves the default CLI builds without Switchyard. - `cargo check -p nemo-relay-cli --features switchyard` — proves the opt-in dependency path. - `cargo test -p nemo-relay-switchyard` — 31 passed. - `cargo test -p nemo-relay-cli gateway::tests` — 33 passed without the feature. - `cargo test -p nemo-relay-cli --features switchyard gateway::tests` — 33 passed with the feature. - `cargo test -p nemo-relay-cli --features switchyard --test switchyard_process_e2e` — buffered, streaming, retry, and fail-open process boundary passed after the split. - `examples/switchyard/run-real-e2e.sh` against the exact pinned Switchyard service — passed with the expected StageRouter route sequence `provider/weak`, `provider/strong`, `provider/strong`. - The pinned Switchyard translation crate separately passes 61 request, response, streaming, extension, and round-trip tests under Rust 1.93. ##### Previous feedback addressed - Split the implementation at repository ownership boundaries. - Replaced Relay's custom translation engine with the Switchyard-owned reusable library. - Kept the dependency compile-time optional for users who do not enable the Relay-native Switchyard plugin. - Removed in-process Random, LLM, and Cascade routers, Relay-side feature accumulation, duplicate Switchyard subscriptions, and the global fallback target. - Replaced literal credentials with environment-referenced headers and activation-time validation. - Hardened internal dispatch headers and structured upstream-failure metadata. - Standardized routing terminology on StageRouter. - Standardized model-routing evidence on Relay's shared optimization contract instead of adding a second post-call savings mark. - Moved generic ATIF effective-model attribution into focused PR #399. - Moved Fern documentation into focused experimental docs PR #400 so this functional PR remains code-focused. - Kept public examples free of internal-only providers, credentials, Visor source references, and private infrastructure assumptions. #### Where should the reviewer start? 1. `crates/switchyard/src/component.rs` — Decision API lifecycle, exact target and baseline validation, retries, fallback, and terminal route commitment. 2. `crates/switchyard/src/translation.rs` and `stream_translation.rs` — thin policy adapters over the Switchyard-owned translation library. 3. `crates/cli/Cargo.toml` and the `cfg(feature = "switchyard")` registration paths — compile-time optionality. 4. `crates/cli/src/gateway.rs` — reserved dispatch headers and structured retry-aware upstream failures. 5. `crates/cli/tests/switchyard_process_e2e.rs` — process-boundary buffered, streaming, and fail-open coverage. 6. `examples/switchyard/README.md` and `run-real-e2e.sh` — pinned experimental service workflow. #### Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to) - Relates to [NVIDIA/NeMo-Relay #389](#389). - Relates to [NVIDIA/NeMo-Relay #399](#399). - Relates to [NVIDIA/NeMo-Relay #400](#400). - Relates to [NVIDIA-NeMo/Switchyard #46](NVIDIA-NeMo/Switchyard#46). - Paired with the Switchyard Decision API topic series: - [NVIDIA-NeMo/Switchyard #8](NVIDIA-NeMo/Switchyard#8) - [NVIDIA-NeMo/Switchyard #9](NVIDIA-NeMo/Switchyard#9) - [NVIDIA-NeMo/Switchyard #11](NVIDIA-NeMo/Switchyard#11) - [NVIDIA-NeMo/Switchyard #35](NVIDIA-NeMo/Switchyard#35) Authors: - Bryan Bednarski (https://github.com/bbednarski9) Approvers: - Will Killian (https://github.com/willkill07) - Maryam Najafian (https://github.com/mnajafian-nv) - https://github.com/Salonijain27 URL: #369
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Harden the public
switchyard-translationcrate for downstream Rust consumers that need to make Switchyard routing decisions actionable across provider protocols.response.incompletefor Responses clients and map its reason to valid Chat or Anthropic terminal semantics.Why
Switchyard should own the provider-protocol translation engine needed to execute its routing decisions. A reusable library lets infrastructure consumers use the same n-by-n request, buffered-response, and streaming behavior instead of maintaining downstream translation forks.
This is the Switchyard-side companion to NVIDIA/NeMo-Relay #369. Relay consumes this crate only when its native Switchyard plugin feature is enabled; routing-service lifecycle, credentials, dispatch, retries, and observability remain Relay-owned.
The PR is based on
topic/nemo-relay-integrationafter #35, keeping StageRouter baseline/freshness policy separate from translation behavior.How tested
cargo fmt --all -- --checkcargo test -p switchyard-translation— 61 passedcargo clippy -p switchyard-translation --all-targets -- -D warningscargo test -p switchyard-translationinrust:1.93.0— 61 passedswitchyardfeaturecargo test -p nemo-relay-switchyard— 31 passedChecklist
Notes for reviewers
Suggested review order:
The change deliberately remains protocol-neutral. Relay-specific target binding, fallback policy, and unsupported-extension fail-open rules stay outside this crate.