tests: track busbar 1.5.3's per-stage notify contract - #8
Open
MattJackson wants to merge 3 commits into
Open
Conversation
added 3 commits
August 8, 2026 11:22
…r publish an assetless release The v1.0.4 release run failed on every target with: error: cannot update the lock file .../Cargo.lock because --locked was passed Cargo.toml's busbar-* dependencies are path deps into a sibling ../busbarAI checkout. release-on-upstream re-pins that checkout to a new commit in .busbar-ref but never regenerated Cargo.lock, so when busbar 1.5.3 pulled `valuable` in behind `tracing`, the committed lock described a graph that no longer resolved and release.yml's `--locked` build correctly refused to proceed. `--locked` is kept deliberately: it is what makes a release build exactly the graph that was resolved at pin time. Instead the lock is now refreshed in the same step that re-pins, against the exact commit being pinned, and committed alongside .busbar-ref so the two can never drift apart. Plain cargo (no --locked) is used for that refresh so the resolution is minimal, and the result is proven to satisfy --locked before anything is committed or tagged. Cargo.lock here is that refresh for the busbar 1.5.3 ref already recorded on main, which is otherwise unbuildable. Separately, create-release published the Release before the build matrix ran, so a total build failure left a tag whose releases/latest carried zero assets. verify-assets already detected that, but only after the empty release was public. The Release is now created as a draft, which releases/latest and releases/tags/<tag> do not resolve, and verify-assets promotes it to published only once assets are provably attached. A release is now either complete or absent. Also brings two test fixtures up to busbar 1.5.3: RoutingRequest/Candidate gained request_id and signals, and inline module entries under auth.admin_auth were retired in favour of a named identity-providers definition.
full_stack_e2e asserted that one real request produces exactly one webhook
call. Against busbar 1.5.3 it produces four. That is a documented breaking
change, not a regression:
busbarAI CHANGELOG.md, "[1.5.3] Breaking changes":
"A hand-written hook with no stage list now fires at all four stages
rather than once per request; set `phase: [request]` for the old
behaviour."
The test registered its tap over the admin API with no stage scoping, so it
was getting the new all-stages fan-out: the stage-less request envelope plus
`stage.at` of candidate, routing and response. The assertion was encoding the
pre-1.5.3 default, not a property of this plugin.
Three changes.
1. Pin the tap to the request stage with `at: "request"`. `phase:` is the
config-file spelling of that opt-out; `at:` is the admin API's, and busbar
resolves stages as: a non-empty `phase:` list wins, else the single `at:`,
else all four core stages. The pin restores "one call per request" as a
correct expectation rather than a stale one, and it keeps this test aimed
at what it exists to prove: that the real engine's wire projection, with
real prompt content under the `prompt: ro` grant, reaches a real webhook
target. Only the request-stage envelope carries content, so only it can
carry that proof.
2. Settle the capture instead of breaking on the first sighting. A tap is
fire-and-forget: busbar spawns the POST detached, so the deliveries for one
request land over a window. The old loop returned the instant the sink went
non-empty, which sampled the middle of that window. That is why the count
assertion could pass by accident on slow delivery: it was guarding "at
least one call arrived and we looked early", not the count. The new helper
requires the count to hold steady across consecutive polls, so a late extra
envelope fails the assertion rather than being raced past.
3. Add tests/stage_fanout_e2e.rs, which covers the new contract positively.
It registers a tap with NO stage scoping (the configuration an operator
writes when they have not thought about stages) and asserts what such a
hook really receives: all four core stages for one request; prompt content
on the request envelope and on none of the others; the stage envelopes
shape-only (no candidates, no messages, no user); the documented per-stage
fields (remaining_candidates, attempt_number + model, outcome + status);
and one shared request_id join key across all four, which is the only thing
that lets a sidecar correlate four POSTs back into one request.
Without 3, the repo would merely tolerate the change: full_stack_e2e would go
green because it opted out, and nothing would assert the fan-out happens or
has the shape sidecar authors are told to expect.
Red before green, both captured: full_stack_e2e failed on the count assertion
before the pin and passes after; stage_fanout_e2e fails on the stage-set
assertion when its hook is scoped to the request stage and passes unscoped.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
A stale-test fix.
full_stack_e2easserted that one real request produces exactly one webhook call. Against busbar 1.5.3 it produces four. That is a documented breaking change in busbar, not a regression in this plugin, and not a bug in the engine:The test registered its tap over the admin API with no stage scoping, so it was receiving the new fan-out: the stage-less request envelope, plus
stage.atofcandidate,routingandresponse. Thelen() == 1assertion was encoding the pre-1.5.3 default, not a property of this plugin. The plugin itself needs no change:notifyis a blind forwarder and has no opinion about stages.Changes
1. Pin the tap to the request stage (
at: "request").phase:is the config-file spelling of the opt-out the CHANGELOG names;at:is the admin API's. Busbar resolves a hook's stages as: a non-emptyphase:list wins, else the singleat:, else all four core stages (HookCfg::fires_at_stage). The pin makes "one call per request" a correct expectation rather than a stale one, and keeps the test aimed at what it exists to prove: that the real engine's own wire projection, carrying real prompt content under theprompt: rogrant, reaches a real webhook target through a real admin-API plugin install. Only the request-stage envelope carries content, so only it can carry that proof.2. Settle the capture instead of breaking on the first sighting.
A tap is fire-and-forget: busbar spawns the POST detached and never awaits it, so the deliveries for one request land over a short window. The old loop returned the instant the sink turned non-empty, sampling the middle of that window. That is why the count assertion could pass by accident on slow delivery: it was guarding "at least one call arrived and we looked before the rest showed up", not the count. The new helper requires the observed count to hold steady across consecutive polls, so a late extra envelope fails the assertion rather than being raced past. Worth having on its own merits, independent of the stage change.
3. New test:
tests/stage_fanout_e2e.rs.This is the part that turns "the plugin tolerates the new contract" into "the plugin has coverage of it". Without it,
full_stack_e2ewould go green because it opted out, and nothing anywhere would assert that the fan-out happens or that its envelopes have the shape a sidecar author is told to expect.It registers a tap with no stage scoping, the exact configuration an operator writes when they have not thought about stages, and asserts what such a hook really receives:
stage.atcoveringcandidate,routing,response. A sidecar sized for one call per request is sized wrong by a factor of four.system/messages/useras absent on stage taps regardless of grant, so aprompt: rohook does not receive content four times over. A sidecar that assumed every notify carries content would silently screen nothing on most of its traffic.candidates, nomessages, nouser.remaining_candidatesoncandidate,attempt_number+modelonrouting,outcome+statusonresponse, and noprevious_failureon a first attempt.request.request_id. That is the documented join key and the only thing that lets a sidecar correlate four separate POSTs back into one request. If it ever drifted per stage, stage taps would be unusable for audit and nothing else would notice.Following this repo's convention, the new test is its own
tests/*.rsbinary carrying its own harness helpers rather than sharing atests/commonmodule, the same calltests/e2e.rsalready makes forplugin_path(). Only the hook registration and the assertions differ fromfull_stack_e2e, and both differences are the point of the file.Red before green
Both captured against a real busbar 1.5.3 build, not asserted from reasoning.
full_stack_e2e, before the pin:Note that the old loop caught only three of the four envelopes: the response-stage delivery had not landed yet. That is change 2's race, visible in the failure itself.
After the pin:
stage_fanout_e2e, deliberately broken by scoping its hook toat: "request"so the extra envelopes never arrive:Restored, unscoped:
cargo fmt --checkandcargo clippy --all-targets -- -D warningsare both clean.Note on the base
This branch is cut from
fix/relock-and-draft-releaserather thanmain, because that branch'sCargo.lockrelock is what makes the path-dependency build against busbar 1.5.3 resolve at all. It therefore carries that commit too. Rebase ontomainonce the relock lands, or merge it first.