feat(router): add escalation_router profile with judge-latched strong/weak routing#62
Open
linj-glitch wants to merge 3 commits into
Open
feat(router): add escalation_router profile with judge-latched strong/weak routing#62linj-glitch wants to merge 3 commits into
linj-glitch wants to merge 3 commits into
Conversation
|
WalkthroughAdds an escalation-router profile that starts requests on a weak tier, uses a trajectory judge for one-way escalation to a strong tier, persists decisions with session affinity, integrates YAML route bundles, exports configuration types, documents the behavior, and updates model pricing aliases. ChangesEscalation Router
Estimated code review effort: 4 (Complex) | ~60 minutes Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Comment |
linj-glitch
force-pushed
the
linj/escalation-router
branch
from
4a16e3c to
c3ee920
Compare
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
tests/test_escalation_router_profile.py (1)
105-124: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winCoverage gap:
judge_timeout_snot asserted.This test threads several
judge_*settings but omitsjudge_timeout_s. Adding an override + assertion (e.g._config(judge_timeout_s=12.0)→judge._config.timeout_s == 12.0) would have surfaced the wiring gap flagged inescalation_router_profile_config.py.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/test_escalation_router_profile.py` around lines 105 - 124, Extend test_build_threads_judge_settings to pass a non-default judge_timeout_s value through _config, then assert the resulting judge._config.timeout_s matches it. Keep the existing judge settings and assertions unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/routing_algorithms/escalation_router_routing.md`:
- Line 73: Update the session_key_depth link and the corresponding heading
reference in the routing documentation so the link target matches the generated
Markdown slug for “Repeated-trial benchmarking (k>1)”. Ensure all references use
the same valid anchor and satisfy markdownlint MD051.
In `@docs/routing_algorithms/overview.md`:
- Around line 156-160: Update the escalation-router description to clarify that
the judge runs on every turn from EscalationJudgeConfig.min_judge_turn onward
until the latch fires, while earlier turns route weak without a judge call.
Preserve the existing affinity and strong-tier pinning behavior.
---
Nitpick comments:
In `@tests/test_escalation_router_profile.py`:
- Around line 105-124: Extend test_build_threads_judge_settings to pass a
non-default judge_timeout_s value through _config, then assert the resulting
judge._config.timeout_s matches it. Keep the existing judge settings and
assertions unchanged.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: ae4d07e2-3bd2-4af8-b27e-66c159c6b73a
📒 Files selected for processing (15)
.agents/skills/switchyard-lib-core/SKILL.mdcrates/switchyard-components/src/stats/cost.rsdocs/routing_algorithms/escalation_router_routing.mddocs/routing_algorithms/overview.mdmkdocs.ymlswitchyard/__init__.pyswitchyard/cli/route_bundle.pyswitchyard/lib/cost_estimator.pyswitchyard/lib/processors/escalation_judge_request_processor.pyswitchyard/lib/profiles/__init__.pyswitchyard/lib/profiles/escalation_router_config.pyswitchyard/lib/profiles/escalation_router_profile_config.pytests/test_escalation_judge_request_processor.pytests/test_escalation_router_profile.pytests/test_route_bundle.py
| | `judge.prompt` | built-in | Judge system-prompt override. | | ||
| | `judge.max_request_chars` | `12000` | Cap on the judge transcript; oldest window messages are dropped first. | | ||
| | `fallback_target_on_evict` | required | `strong` or `weak`; context-window-eviction reroute target. | | ||
| | `session_key_depth` | `0` | See [Repeated-trial benchmarking](#repeated-trial-benchmarking) below. | |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Broken anchor link (matches markdownlint MD051).
#repeated-trial-benchmarking doesn't match the heading ## Repeated-trial benchmarking (k>1) (line 87); the generated slug would be repeated-trial-benchmarking-k1 (or similar, depending on the slugifier's handling of >).
📝 Proposed fix
-| `session_key_depth` | `0` | See [Repeated-trial benchmarking](`#repeated-trial-benchmarking`) below. |
+| `session_key_depth` | `0` | See [Repeated-trial benchmarking](`#repeated-trial-benchmarking-k1`) below. |Also applies to: 87-87
🧰 Tools
🪛 markdownlint-cli2 (0.23.0)
[warning] 73-73: Link fragments should be valid
(MD051, link-fragments)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/routing_algorithms/escalation_router_routing.md` at line 73, Update the
session_key_depth link and the corresponding heading reference in the routing
documentation so the link target matches the generated Markdown slug for
“Repeated-trial benchmarking (k>1)”. Ensure all references use the same valid
anchor and satisfy markdownlint MD051.
Source: Linters/SAST tools
Comment on lines
+156
to
+160
| The escalation router uses the same affinity store with the opposite policy: | ||
| affinity is always on (the pin *is* the escalation latch), only the strong | ||
| tier is ever pinned, and the judge keeps running every turn until the latch | ||
| fires. See | ||
| [Escalation-Router Routing](escalation_router_routing.md). |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
"Every turn" overstates when the judge actually runs.
Per escalation_router_routing.md (and EscalationJudgeConfig.min_judge_turn), the judge only starts running from min_turn (default 3) onward — earlier turns route weak with no judge call. "keeps running every turn until the latch fires" reads as running from turn 1, contradicting the dedicated page's own description.
📝 Proposed fix
-The escalation router uses the same affinity store with the opposite policy:
-affinity is always on (the pin *is* the escalation latch), only the strong
-tier is ever pinned, and the judge keeps running every turn until the latch
-fires. See
+The escalation router uses the same affinity store with the opposite policy:
+affinity is always on (the pin *is* the escalation latch), only the strong
+tier is ever pinned, and once the judge starts (a configurable minimum turn),
+it keeps running each judged turn until the latch fires. See
[Escalation-Router Routing](escalation_router_routing.md).📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| The escalation router uses the same affinity store with the opposite policy: | |
| affinity is always on (the pin *is* the escalation latch), only the strong | |
| tier is ever pinned, and the judge keeps running every turn until the latch | |
| fires. See | |
| [Escalation-Router Routing](escalation_router_routing.md). | |
| The escalation router uses the same affinity store with the opposite policy: | |
| affinity is always on (the pin *is* the escalation latch), only the strong | |
| tier is ever pinned, and once the judge starts (a configurable minimum turn), | |
| it keeps running each judged turn until the latch fires. See | |
| [Escalation-Router Routing](escalation_router_routing.md). |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/routing_algorithms/overview.md` around lines 156 - 160, Update the
escalation-router description to clarify that the judge runs on every turn from
EscalationJudgeConfig.min_judge_turn onward until the latch fires, while earlier
turns route weak without a judge call. Preserve the existing affinity and
strong-tier pinning behavior.
…/weak routing Signed-off-by: Lin Jia <linj@nvidia.com>
…neralization audit Signed-off-by: Lin Jia <linj@nvidia.com>
Signed-off-by: Lin Jia <linj@nvidia.com>
linj-glitch
force-pushed
the
linj/escalation-router
branch
from
9417102 to
5831550
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Ports the escalation-router work from the private fork (branch
linj_router_exp) to the public repo.escalation_routerprofile with judge-latched strong/weak routing, including the escalation-judge request processor, profile configs, androute_bundleCLI support.switchyard/lib/cost_estimator.py,crates/switchyard-components/src/stats/cost.rs), and expanded tests.docs/routing_algorithms/escalation_router_routing.md) and registers them inmkdocs.yml.Note: changes under the
benchmark/folder from the original branch are intentionally excluded from this PR.Test plan
tests/test_escalation_judge_request_processor.pytests/test_escalation_router_profile.pytests/test_route_bundle.py🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Documentation