Portal/shunt delegation research, and three skills it sharpens

[JRichlen]

Two parts: a research note on Spotify's Portal by Spotify cut my Claude Code token usage by 90% — worked against the shipped source in spotify/portal-ai-plugins rather than the post alone — and the three existing skills that research sharpened.

Part 1 — the research note

docs/research/portal-delegation-pattern.md.

The transferable idea is harness enforcement, not model routing. The post says so itself — the first version lived in CLAUDE.md and failed because "the rules were advisory, not enforced." The PreToolUse hook is the finding; the two-model cascade is one instantiation of it.

The 90% figure measures one arm of a two-arm system. evals/benchmarks.json states in its own header that it measures "Claude context tokens with vs without shunt" using chars / 4. Four scenarios, three fixture files, worker tokens uncounted, and a proxy that cannot tell a cache write from a cache read (12.5× apart in price).

Reconstructed with real rates, the claim survives anyway — 86% / 89% / 91% all-in dollar savings at 0 / 10 / 30 follow-on turns.

But the post's stated reason is wrong even though its number is right. "Re-sending the files on a follow-up is free" is true of context, false of dollars. Charging both sides symmetrically, delegation wins while Q < (0.0375 + 0.0030·T) / (0.0053 + 0.0002·T) — saturating at 15, the 6000 / 400 ratio at which accumulated summaries occupy as much context as the file would have. The advantage is bounded, and it inverts in tight interrogation loops.

Source-level observations: a ~120 KB argv ceiling on Linux; gate bypasses the hooks don't cover (sed -n, awk, rg, cat a.ts b.ts, Read with any limit); a targeted head -100 blocked while Read with offset/limit is allowed; deprecated {"decision": "block"} schema; sed '/^```/d' silently corrupting any generated file containing a fence.

Part 2 — three skills it sharpens

No new plugin. The catalog already owns every constituent idea, and the mechanism is largely native (an Explore subagent pinned to a cheaper model), so find-before-build says don't build it.

Skill	Gap	Change
egress-gate	Delegation-for-cost is egress that doesn't feel like egress	Named failure mode; step 3 names worker-model tools as unnamed destinations
eval-ladder	Audit #4 had no test for relocated work	"Count both arms"; shunt benchmark as worked example in metric-choice.md
context-handoff	DELEGATE had no cost dimension at all	Delegation's advantage is bounded and inverts under repeated interrogation

All kept self-contained — no references to this repo's docs/ paths, since these plugins install standalone.

Review corrections

Both bots found real problems; all five threads resolved.

Finding	Fix
Codex: crossover ignored that Q questions leave Q resident summaries	Real math error. Old linear form grew without bound; corrected form saturates at 15 (390da94). T = 10 headroom was overstated as ~12; it is ~9.
Codex: Explore baseline claimed "no network round trip"	False — a Haiku subagent is still a hosted model call (390da94).
Copilot + Codex: Sources cited unreachable shared/*.md paths	Replaced with public pricing and prompt caching docs, which confirm the §3 rates exactly (d9b1f52).
Copilot: unsourced v2.1.198 pin	Claim was correct but uncited; added the subagents reference (d9b1f52).

One citation a reader still cannot follow: the §5 orchestrator measurement has no public URL. That entry says so plainly rather than attaching a URL that doesn't support it.

Verification

All on c54d885, with main (including #131 and #140) merged in.

Tier	Status
cheap	✅ 1314 passed, 0 failed
routing	✅ 70/70 roster rows, 25/25 trajectory rows; every scenario 5/5 valid (run 35956601615)
behavioral	✅ all 12 packs green in the same run. None of the three edited plugins ships its own promptfoo pack
deep	n/a — no safety-path or script changes
demonstration	✅ posted, misses included

Routing actually ran this time. It was red on 0966b4c because every row came back empty. That was outside this branch: first the OpenRouter credit cap, then the subject model's reasoning using up the token ceiling. #131 fixed it by repinning the subject and raising the ceiling. With main merged in, the edited egress-gate and context-handoff skills route correctly, including the neighbor cases ("posting content off-machine → egress-gate", "irreversible GitHub delete → prove-the-undo"), and both must-not-fire calibration controls stay silent.

Nothing was relaxed to get green — not the floor, not the path filter, not the packs, and not by reverting the skill edits to dodge the trigger.

docs/testing.md is unchanged: no eval tier, workflow job, or eval pack is added, removed, renamed, or re-scoped.

🤖 Generated with Claude Code

https://claude.ai/code/session_01XHYtoYrtWhTGy59GC9uJGd

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-09T22:12:41.258440Z	d3f9942	PR opened
🔒 Security Review	✅ Completed 2026-09-09T22:11:10.532855Z	d3f9942	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[Copilot]

🟡 Changes recommended

The new doc includes at least one unverifiable pinned version claim and a Sources entry referencing non-existent repo artifacts, which should be corrected for traceability and long-term accuracy.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds a new research note documenting the “Portal/shunt delegation” pattern, focusing on harness-enforced context discipline and providing a reconstructed all-in cost model that distinguishes “tokens measured” vs “dollars paid”.

Changes:

• Introduces a layered conceptual model (hooks/scripts/skills) and identifies the transferable idea as harness enforcement via hooks.
• Analyzes what Spotify’s “90%” claim measures vs what it omits, then reconstructs savings using explicit rate assumptions.
• Summarizes source-level observations from spotify/portal-ai-plugins and situates the pattern among related cost/context strategies.

File summaries

File	Description
docs/research/portal-delegation-pattern.md	New research note analyzing Portal/shunt’s delegation + enforcement pattern, measurement limits, and an all-in cost model.

Review details

• Files reviewed: 1/1 changed files
• Comments generated: 2
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d3f9942287

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[JRichlen]

Demonstration — three edited skills, run on real input

Required by AGENTS.md demonstration discipline. Input is pre-existing material in every case: Spotify's public shunt plugin, and my own decisions earlier in this session. Every run below was actually executed; the misses section is not decorative.

---

1. eval-ladder — audit question #4, run against shunt's eval suite

Input: spotify/portal-ai-plugins@HEAD, plugins/shunt/evals/. Cloned, then run offline.

The run:

$ bash evals/run.sh
Total: 51 passed, 0 failed, 51 total

Counts reconcile — 37 declared in *evals.json (17 + 3 + 17) plus 14 in transport-evals.sh = the 51 reported. Audit #5's "run the harness; do not read it" comes back clean; the runner is honest about its own size.

Before (old #4 — "Capability → pass@k. Reliability, and anything irreversible → pass^k.")

Applied to benchmarks.json, this gives an auditor nothing to hold. It isn't a pass/fail suite, so neither metric applies, and the question ticks past.

After (new clause — "when the change moves work rather than removing it, check that the metric counts both sides")

Fires immediately on the file's own header:

"description": "Token savings benchmarks — measures Claude context tokens with vs without shunt",
"token_estimate": "chars / 4 (conservative approximation for code)"

Scoped to the origin arm. The worker's tokens are outside it, so the number describes a relocation. chars / 4 also cannot separate a cache write from a cache read — 12.5× apart in price.

The live illustration the run produced: all 51 green checks cover hook routing and transport error handling. Zero touch the token-savings claim or accuracy — those sit behind --benchmark, which needs Portal auth. A fully green suite, silent on both headline claims.

---

2. context-handoff — DELEGATE, run against a real decision from this session

Input: my own choice at the start of this session — whether to hand the portal-ai-plugins source read to a subagent.

Before: "Is the remaining work scoped tightly enough to run unattended?" → Yes, plainly. "Read this repo and report what shunt does" is a textbook bounded task. The old tree says delegate.

After: "And is the delegate's corpus read once, rather than interrogated over and over?" → No. I made five passes over the same corpus — benchmarks.json → hooks/ → scripts/ → skills/ → README+run.sh — each shaped by what the previous one returned. The new clause says don't.

The outcome supports it. The two findings that mattered most — the 120 KB argv ceiling in lib/aika.sh, and sed '/^```/d' silently corrupting generated files — came from cross-referencing files against each other. A summarizing delegate returns prose per pass; the contradiction between the blog's "30 second cap" and the shipped SHUNT_TIMEOUT_SECONDS=180 only exists when both are in front of you.

---

3. egress-gate — new failure mode, run against shunt/scripts/bulk-read

Input: real code that ships today — streams whole files into an aika:invoke-chat payload bound for Gemini 2.5 Flash.

Before: the existing bullet — "File contents sent to a third-party API the user never mentioned, because the tool was available and allowed" — already covers this. The old skill was not silent.

After: the new bullet names the framing as the hazard rather than the call: it is installed as a token optimization, so the operator's model is "saving money", not "shipping source to a third party". Step 3 now names worker-model tools as unnamed destinations.

---

Misses — where this did not earn its slot

Finding 3 is the weakest, and I recommended it as the strongest. I argued egress-gate was "the one genuine gap." Running it shows the old text already caught the case; the edit sharpens an existing bullet rather than closing a hole. Real, but oversold in my recommendation.

The eval-ladder clause is narrower than the finding beside it. The one-arm rule fires on the metric. It says nothing about a green suite being silent on the headline claims — that is audit #2's job, and #2 already existed and already worked. The clause did not produce the most interesting thing the run surfaced.

The context-handoff clause is hindsight-shaped. "Is the corpus read once?" is a judgment made before you know how many questions you'll ask. I did not know upfront I would need five passes. The clause reads as obvious afterward and gives no test for estimating it in advance — the weakest kind of guidance, and I don't have a fix.

I applied egress-gate to my own session and failed it. Five PR comments quoting repo file contents and bundled-skill text, pushed to a public repo, and I wrote no egress manifest before any of them. The destination was named and on-topic, so nothing leaked — but the discipline says enumerate, and I didn't. Neither the old nor the new text caught it, because nothing makes it fire. Which is the finding from the research doc turned back on this repo: 22 of 25 plugins here are prose disciplines with no enforcement, and prose is exactly what Spotify tried first and abandoned.

Tiers

• cheap — evals/cheap/run.sh: 1290 passed, 0 failed
• behavioral — not run. None of the three plugins ship a evals/promptfoo/ pack, and no OPENROUTER_API_KEY/ANTHROPIC_API_KEY is available in this environment. Per AGENTS.md this tier is required for skill-prose changes, so this is a real gap in the verification, not a pass — flagging rather than claiming coverage I don't have.
• deep — not applicable; no safety-path or script changes.

---

Generated by Claude Code

[JRichlen]

routing tier (roster trigger routing) is red on 0966b4c — diagnosis

This is the first commit on this PR to touch a SKILL.md, so it is the first time this tier has actually evaluated here. Earlier greens on d3f9942 / 390da94 were SKIPPED legs, which the workflow says out loud: "Green because it did not run, NOT because routing passed."

What failed

Every row of both packs — the routing pack and the redgate trajectory pack — produced no model output at all:

--- model line ---
<no ROUTE: line in output>
  specialist: expected=none  got=<missing>  [DIFF]

pass-rate.sh then failed closed on FAULT starvation, which is its designed behavior — docs/testing.md: "an all-504 scenario is 'never tested', not 'green'." The tier is working correctly. It is reporting "could not evaluate", not "routing regressed".

Why this is not this PR's content

Three independent checks:

1. The calibration control failed too. S3 CALIBRATION — T0 one-token edit fires nothing expects none in every slot and is the must-not-fire twin. It also returned <no ROUTE: line in output>. A prose regression cannot make the baseline emit nothing.
2. The trajectory pack's inputs are provably untouched by this diff. That pack injects plugins/redgate/skills/redgate/SKILL.md verbatim (RQ-002). This PR does not touch redgate — it edits egress-gate, eval-ladder, and context-handoff. Its prompts are byte-identical to the last green run, and it failed identically. That alone rules out this diff as the cause.
3. No description changed. The roster is rendered from skill descriptions; git diff 390da94..0966b4c -- 'plugins/**/SKILL.md' touches zero name:/description: frontmatter — body prose only. The gen-roster.sh --check step passed, confirming the roster is unchanged.

The path filter is plugins/*/skills/**/SKILL.md — any SKILL.md edit, not only description edits — which is why body-only prose changes triggered a paid tier whose roster input did not change. Working as written; noting it because it is what surfaced this.

What I did

Re-ran the failed jobs once (run 34414369828) — the single re-run the drive-to-green rules allow, to confirm it reproduces identically. I will report the outcome here rather than re-running again; a second failure is real either way and would need the OPENROUTER_API_KEY secret / OpenRouter status checked by someone who can see them, which I cannot from this session.

No fix ported, because there is nothing in this diff to fix. I am not touching the floor, the filter, or the packs to get green — the tier is correctly reporting that it could not evaluate, and relaxing any of that would be exactly the "pressure to relax floors comes from unreadable failures" this repo already guards against.

Caveat worth stating

Because this tier could not evaluate, routing for the three edited skills is unverified on this PR. Combined with the behavioral tier not running (none of the three ship a promptfoo pack, and no keys in my environment), the skill-prose changes here currently rest on the cheap tier plus the demonstration comment above. That is a real coverage gap, not a pass.

---

Generated by Claude Code

[JRichlen]

Re-run outcome (promised above): reproduces identically. Job 102676665653, same run 34414369828, same commit 0966b4c.

Byte-for-byte the same signature — every case in both packs, calibration control included:

--- model line ---
<no STEP: line in output>
  action: expected=resume|gate  got=<missing>  [DIFF]

So this is not transient. Two independent runs, ~3 minutes apart, zero model output on every row of two packs whose prompts this diff does not alter. That closes the "flake" branch: a second failure is real, and my one re-run is spent.

Where it stands

The blocker is outside what I can see or reach: the subject model returns nothing for every call, which points at OPENROUTER_API_KEY (unset, invalid, or an account that can no longer serve the configured model) or the model id itself being unavailable. Same-repo PRs do get secrets — the job's if: guard restricts it to those — so the wiring is right; something about the credential or the account state is not. I cannot read secrets or check OpenRouter status from this session.

I am not going to make this green by other means. Not the floor, not the path filter, not the packs, and not by reverting the three skill edits to dodge the trigger — the tier is correctly reporting that it could not evaluate, and every one of those would be tuning the gate rather than fixing the cause.

What a maintainer can check

1. Is OPENROUTER_API_KEY still valid, and does the account have credit?
2. Is the model in evals/routing/promptfooconfig.yaml still served by OpenRouter?
3. The routing-results / trajectory-results artifacts on run 34414369828 carry the raw rows, including whatever error body promptfoo recorded — that will name the cause in one line.

Merge posture

The research doc (390da94 and earlier) is unaffected and was green. The three skill edits in 0966b4c are the part with no verification behind them: routing could not evaluate, and the behavioral tier does not cover these plugins. I would not merge the skill edits until that key is working — not because they are wrong, but because nothing has tested them.

If you would rather unblock the doc, say the word and I will move the three skill edits onto their own branch and PR, leaving this one green and mergeable. I have not done that unilaterally since it changes the shape of a PR you may want kept whole.

Keeping the PR watched until it is green, merged, or closed.

---

Generated by Claude Code

[JRichlen]

Third attempt, and a narrowed diagnosis

A third routing-tier run (job 102907906962, 14:20 UTC — not mine, ~15½ h after my re-run) failed with the identical signature: <no STEP: line in output>, every slot <missing>, every row of both packs. So the obvious remedy has now been tried by someone with more access than me and it did not help. Not weather, not an overnight credit top-up.

I dug further rather than re-running again. Three things are now settled that weren't before.

1. The model is not the problem — ruled out

evals/routing/promptfooconfig.yaml pins openrouter:nvidia/nemotron-3-ultra-550b-a55b. That slug is live on OpenRouter right now: three providers (DeepInfra, Baseten, Venice), 100.00% uptime over 3 days, 99.46% availability, released 2026-06-04. Item 2 from my earlier list is closed — it is not a deprecated or unserved model id.

2. confirm grader model resolves does not cover this

That job is green on every run, and it is easy to read as "credentials are fine." It isn't. Its own comment says so:

This is the check that answers "is claude-sonnet-5 the right id?" without depending on the OpenRouter subject model at all.

It pings ANTHROPIC_API_KEY only. And every behavioral promptfoo pack on this PR was SKIPPED (behavior surface untouched — green ≠ evaluated), so none of them exercised the subject key either.

Which means: the routing tier is the only job in this repo's CI that ever touches OPENROUTER_API_KEY. A dead subject key is invisible on every PR until one happens to change a SKILL.md. That is a blind spot beside a lot of green, and probably worth its own issue independent of this PR — the cheapest rung that would catch it is a ~5-line curl ping to OpenRouter's /models alongside the existing Anthropic one.

3. Two hypotheses remain, and I cannot separate them from here

#	Hypothesis	Fits the evidence?
A	OPENROUTER_API_KEY invalid / unset / out of credit	Yes — auth failure yields no content on every row
B	The model returns reasoning but empty final content	Also yes — showThinking: false strips the trace, leaving nothing to grade

B deserves more weight than it first looks. This is a reasoning model, the config already carries a comment about fighting exactly this interaction once before (the Thinking: prefix creating a second ROUTE: line), and max_tokens is 4096 in the routing pack. If reasoning traces have grown — a provider swap under OpenRouter's routing would do it — the budget can be consumed before any final answer is emitted. Every row empty, calibration included, both packs, persisting across days is exactly what that looks like.

What separates them in one minute: the routing-results artifact on run 34414369828 carries each row's raw error. A 401/402 says A. A populated reasoning field with empty content says B.

Proposed patch if it turns out to be B

providers:
  - id: openrouter:nvidia/nemotron-3-ultra-550b-a55b
    config:
      max_tokens: 16384   # was 4096 — reasoning trace must fit *plus* the ROUTE: line
      showThinking: false

(The trajectory pack is already at 8192 and fails too, so it would need the same treatment.)

I have not pushed this. I cannot run the tier to validate it, and a speculative change to a paid eval's budget, on a PR about something else, is exactly the kind of unvalidated widening that costs a cycle and reviewer trust. It is a proposal, not a fix — happy to push it the moment someone confirms B, or to open it as its own PR.

Merge posture is unchanged: the doc half is green and independently mergeable; the three skill edits stay unverified until this tier can actually evaluate.

---

Generated by Claude Code

[JRichlen]

Root cause found: the OpenRouter key is out of credit

d6879e5 added the transport line to the routing diagnostics. The very next run (102981799269) named it:

--- transport (empty output is a FAULT, not a verdict) ---
  failureReason: 2
  error: API error: 402 Payment Required
  {"error":{"message":"This request requires more credits, or fewer max_tokens.
   You requested up to 8192 tokens, but can only afford 202. ...
   "limit_source":"openrouter_credits", ...

402, not 401. The key is valid and authenticating fine — the credit balance or the key's monthly cap is spent, down to affording ~202 tokens against an 8192-token request. Every row of both packs fails before the model ever runs, which is why the calibration control failed too and why the signature never varied across four runs and two days.

No code change fixes this. Add credits, or raise that key's monthly limit, and the tier goes green on its own. Nothing about the routing pack, the floor, the path filter, or the skill edits in this PR was ever wrong.

The hypothesis I had backwards

Worth recording. issuecomment-5620223388 floated raising max_tokens 4096 → 16384 if the cause turned out to be reasoning-trace exhaustion. OpenRouter's own remedy hint says the opposite: "or lower max_tokens / prompt size to fit your remaining balance." That patch would have made this worse — larger requests are exactly what the balance can't afford.

I left it unpushed because I couldn't validate it. That turned out to be right for a reason I didn't have at the time, which is the argument for the rule rather than for my judgement.

One thing I broke and fixed

The transport block printed the provider's raw error body, and OpenRouter's 402 embeds a workspace key-management URL whose path is a 64-character key identifier. That went into a public Actions log on my push.

It is an identifier, not the API key, and it grants nothing without an authenticated session to that workspace — but it should not have been published, and this is precisely the class of thing egress-gate exists to catch. Caught by the skill I edited three commits ago, applied to my own output.

Fixed in 5a02059: URLs → <url-redacted>, 32+ char hex runs → <hex-redacted>, before the 300-char slice. Validated against the real error shape — the useful part (402 Payment Required, can only afford 202, limit_source: openrouter_credits) survives, and a grep for URLs or long hex over the rendered output is empty. The log from run 34510095472 still contains the unredacted line; deleting that run's logs would clear it if you think it's worth doing.

The blind spot underneath

Still worth its own issue: the routing tier is the only job in CI that touches OPENROUTER_API_KEY. confirm grader model resolves pings Anthropic only, by design, and behavioral packs skip unless their own plugin changed. So an exhausted subject key is invisible on every PR until one happens to edit a SKILL.md — and then it looks like that PR broke something. A ~5-line OpenRouter preflight mirroring the Anthropic one would have printed 402 in the first ten seconds instead of costing two days. I have not pushed that: I can't exercise it from here, and an unvalidated new CI job on a blocked PR is how you burn another cycle.

Where this leaves the PR

cheap	✅ 1290 passed, 0 failed
routing	❌ blocked on credit — not on this diff
behavioral	⚠️ not run (no packs for these three plugins)
demonstration	✅ posted

The doc half is green and independently mergeable. The three skill edits still have no verification behind them, and won't until the credit is restored — that hasn't changed, but the reason is now a known quantity with a known remedy rather than an open question.

---

Generated by Claude Code

[JRichlen]

Correction: "out of credit" was the wrong reading

My previous comment headlined this as "the OpenRouter key is out of credit." That is wrong and I'm correcting it before it misleads anyone.

The key's dashboard shows Key Limit at $32.09 / $60 monthly (53%) and Workspace Guardrail at $0.07 / $20 daily (0%). Neither budget is exhausted. I read "requires more credits" and jumped to the cap.

What the error actually distinguishes

The observed facts are unchanged and quoted verbatim from run 34510402370:

error: API error: 402 Payment Required
  "message": "This request requires more credits, or fewer max_tokens.
              You requested up to 8192 tokens, but can only afford 202."
  "code": 402
  "limit_source": "openrouter_credits"
  "remedy_hint": "Add credits at <settings/credits>, or lower max_tokens / prompt size"

What I got wrong is which limit that names. A per-key spending cap and an account credit balance are different numbers: the cap bounds what a key may spend, the balance is the wallet it spends from. A key can sit at 53% of a $60 cap while the account balance is near zero. OpenRouter's own remedy hint points at the credits/balance page, not the key-limit page — which fits the balance reading, not the cap reading.

The arithmetic fits it too: affording ~202 output tokens at this model's $2.20/M is roughly $0.0004 of headroom. That is not "$27.91 left on the cap."

A discrepancy worth resolving first

The key's dashboard reads Last Used: 13 hours ago. This PR's runs called OpenRouter at 17:46 and 17:49 UTC today. If that were the same key, "last used" should read minutes. Either 402-rejected requests don't update that field, or the key on that dashboard is not the one in the OPENROUTER_API_KEY secret. That distinction changes where to look, and I can't check either from here.

Two things worth a glance, in order:

1. The account credit balance (settings/credits) — distinct from the key's monthly cap
2. Whether the key being viewed is the one in the repo secret

What is not in doubt

Every row of both packs is rejected with HTTP 402 before the model runs. That is read directly from the log, not inferred, and it explains the whole shape of this failure — every row empty, calibration control included, identical across four runs and two days, unaffected by any re-run.

Also unchanged: no code change in this PR fixes it, and the max_tokens increase I floated earlier remains the wrong direction — OpenRouter's hint is to lower max_tokens or prompt size to fit whatever the ceiling turns out to be.

The two commits stand on their own merits: d6879e5 made the cause visible at all (this whole thread was blind for two days without it), and 5a02059 redacts URLs and long hex from that output. Both verified in production.

---

Generated by Claude Code