Skip to content

tests: track busbar 1.5.3's per-stage notify contract - #8

Open
MattJackson wants to merge 3 commits into
mainfrom
fix/hook-stage-contract-1.5.3
Open

tests: track busbar 1.5.3's per-stage notify contract#8
MattJackson wants to merge 3 commits into
mainfrom
fix/hook-stage-contract-1.5.3

Conversation

@MattJackson

Copy link
Copy Markdown
Contributor

What this is

A stale-test fix. full_stack_e2e asserted that one real request produces exactly one webhook call. Against busbar 1.5.3 it produces four. That is a documented breaking change in busbar, not a regression in this plugin, and not a bug in the engine:

busbarAI CHANGELOG.md, "[1.5.3] Breaking changes":
"Hooks are defined once by name and attached by name. [...] A hand-written hook with no stage list now fires at all four stages rather than once per request; set phase: [request] for the old behaviour."

The test registered its tap over the admin API with no stage scoping, so it was receiving the new fan-out: the stage-less request envelope, plus stage.at of candidate, routing and response. The len() == 1 assertion was encoding the pre-1.5.3 default, not a property of this plugin. The plugin itself needs no change: notify is a blind forwarder and has no opinion about stages.

Changes

1. Pin the tap to the request stage (at: "request").

phase: is the config-file spelling of the opt-out the CHANGELOG names; at: is the admin API's. Busbar resolves a hook's stages as: a non-empty phase: list wins, else the single at:, else all four core stages (HookCfg::fires_at_stage). The pin makes "one call per request" a correct expectation rather than a stale one, and keeps the test aimed at what it exists to prove: that the real engine's own wire projection, carrying real prompt content under the prompt: ro grant, reaches a real webhook target through a real admin-API plugin install. Only the request-stage envelope carries content, so only it can carry that proof.

2. Settle the capture instead of breaking on the first sighting.

A tap is fire-and-forget: busbar spawns the POST detached and never awaits it, so the deliveries for one request land over a short window. The old loop returned the instant the sink turned non-empty, sampling the middle of that window. That is why the count assertion could pass by accident on slow delivery: it was guarding "at least one call arrived and we looked before the rest showed up", not the count. The new helper requires the observed count to hold steady across consecutive polls, so a late extra envelope fails the assertion rather than being raced past. Worth having on its own merits, independent of the stage change.

3. New test: tests/stage_fanout_e2e.rs.

This is the part that turns "the plugin tolerates the new contract" into "the plugin has coverage of it". Without it, full_stack_e2e would go green because it opted out, and nothing anywhere would assert that the fan-out happens or that its envelopes have the shape a sidecar author is told to expect.

It registers a tap with no stage scoping, the exact configuration an operator writes when they have not thought about stages, and asserts what such a hook really receives:

  • All four core stages arrive for one request: the stage-less request envelope plus stage.at covering candidate, routing, response. A sidecar sized for one call per request is sized wrong by a factor of four.
  • Exactly one envelope carries prompt text, and it is the stage-less one. The engine sends system/messages/user as absent on stage taps regardless of grant, so a prompt: ro hook does not receive content four times over. A sidecar that assumed every notify carries content would silently screen nothing on most of its traffic.
  • The stage envelopes are shape-only in the other direction too: no candidates, no messages, no user.
  • The documented per-stage fields are present: remaining_candidates on candidate, attempt_number + model on routing, outcome + status on response, and no previous_failure on a first attempt.
  • Every envelope carries the same request.request_id. That is the documented join key and the only thing that lets a sidecar correlate four separate POSTs back into one request. If it ever drifted per stage, stage taps would be unusable for audit and nothing else would notice.

Following this repo's convention, the new test is its own tests/*.rs binary carrying its own harness helpers rather than sharing a tests/common module, the same call tests/e2e.rs already makes for plugin_path(). Only the hook registration and the assertions differ from full_stack_e2e, and both differences are the point of the file.

Red before green

Both captured against a real busbar 1.5.3 build, not asserted from reasoning.

full_stack_e2e, before the pin:

thread 'install_over_admin_api_and_drive_a_real_hook_invocation' panicked at tests/full_stack_e2e.rs:539:5:
assertion `left == right` failed: exactly one real webhook call expected for one real request:
  [ {"op":"notify","request":{...,"messages":[{"role":"user","text":"hello from full_stack_e2e ..."}]}},
    {"op":"notify","request":{...},"stage":{"at":"candidate","remaining_candidates":1}},
    {"op":"notify","request":{...},"stage":{"at":"routing","attempt_number":1,"model":"m","remaining_candidates":0}} ]
test result: FAILED. 0 passed; 1 failed

Note that the old loop caught only three of the four envelopes: the response-stage delivery had not landed yet. That is change 2's race, visible in the failure itself.

After the pin:

test install_over_admin_api_and_drive_a_real_hook_invocation ... ok
test result: ok. 1 passed; 0 failed

stage_fanout_e2e, deliberately broken by scoping its hook to at: "request" so the extra envelopes never arrive:

thread 'unscoped_hook_observes_every_core_stage' panicked at tests/stage_fanout_e2e.rs:581:5:
assertion `left == right` failed: an unscoped hook must observe the candidate, routing and response
stages (plus the stage-less request envelope asserted above):
  [ {"op":"notify","request":{...,"messages":[{"role":"user","text":"hello from stage_fanout_e2e ..."}]}} ]
test result: FAILED. 0 passed; 1 failed

Restored, unscoped:

test unscoped_hook_observes_every_core_stage ... ok
test result: ok. 1 passed; 0 failed

cargo fmt --check and cargo clippy --all-targets -- -D warnings are both clean.

Note on the base

This branch is cut from fix/relock-and-draft-release rather than main, because that branch's Cargo.lock relock is what makes the path-dependency build against busbar 1.5.3 resolve at all. It therefore carries that commit too. Rebase onto main once the relock lands, or merge it first.

Matthew Jackson added 3 commits August 8, 2026 11:22
…r publish an assetless release

The v1.0.4 release run failed on every target with:

  error: cannot update the lock file .../Cargo.lock because --locked was passed

Cargo.toml's busbar-* dependencies are path deps into a sibling ../busbarAI
checkout. release-on-upstream re-pins that checkout to a new commit in
.busbar-ref but never regenerated Cargo.lock, so when busbar 1.5.3 pulled
`valuable` in behind `tracing`, the committed lock described a graph that no
longer resolved and release.yml's `--locked` build correctly refused to
proceed.

`--locked` is kept deliberately: it is what makes a release build exactly the
graph that was resolved at pin time. Instead the lock is now refreshed in the
same step that re-pins, against the exact commit being pinned, and committed
alongside .busbar-ref so the two can never drift apart. Plain cargo (no
--locked) is used for that refresh so the resolution is minimal, and the result
is proven to satisfy --locked before anything is committed or tagged.

Cargo.lock here is that refresh for the busbar 1.5.3 ref already recorded on
main, which is otherwise unbuildable.

Separately, create-release published the Release before the build matrix ran,
so a total build failure left a tag whose releases/latest carried zero assets.
verify-assets already detected that, but only after the empty release was
public. The Release is now created as a draft, which releases/latest and
releases/tags/<tag> do not resolve, and verify-assets promotes it to published
only once assets are provably attached. A release is now either complete or
absent.

Also brings two test fixtures up to busbar 1.5.3: RoutingRequest/Candidate
gained request_id and signals, and inline module entries under auth.admin_auth
were retired in favour of a named identity-providers definition.
full_stack_e2e asserted that one real request produces exactly one webhook
call. Against busbar 1.5.3 it produces four. That is a documented breaking
change, not a regression:

  busbarAI CHANGELOG.md, "[1.5.3] Breaking changes":
    "A hand-written hook with no stage list now fires at all four stages
     rather than once per request; set `phase: [request]` for the old
     behaviour."

The test registered its tap over the admin API with no stage scoping, so it
was getting the new all-stages fan-out: the stage-less request envelope plus
`stage.at` of candidate, routing and response. The assertion was encoding the
pre-1.5.3 default, not a property of this plugin.

Three changes.

1. Pin the tap to the request stage with `at: "request"`. `phase:` is the
   config-file spelling of that opt-out; `at:` is the admin API's, and busbar
   resolves stages as: a non-empty `phase:` list wins, else the single `at:`,
   else all four core stages. The pin restores "one call per request" as a
   correct expectation rather than a stale one, and it keeps this test aimed
   at what it exists to prove: that the real engine's wire projection, with
   real prompt content under the `prompt: ro` grant, reaches a real webhook
   target. Only the request-stage envelope carries content, so only it can
   carry that proof.

2. Settle the capture instead of breaking on the first sighting. A tap is
   fire-and-forget: busbar spawns the POST detached, so the deliveries for one
   request land over a window. The old loop returned the instant the sink went
   non-empty, which sampled the middle of that window. That is why the count
   assertion could pass by accident on slow delivery: it was guarding "at
   least one call arrived and we looked early", not the count. The new helper
   requires the count to hold steady across consecutive polls, so a late extra
   envelope fails the assertion rather than being raced past.

3. Add tests/stage_fanout_e2e.rs, which covers the new contract positively.
   It registers a tap with NO stage scoping (the configuration an operator
   writes when they have not thought about stages) and asserts what such a
   hook really receives: all four core stages for one request; prompt content
   on the request envelope and on none of the others; the stage envelopes
   shape-only (no candidates, no messages, no user); the documented per-stage
   fields (remaining_candidates, attempt_number + model, outcome + status);
   and one shared request_id join key across all four, which is the only thing
   that lets a sidecar correlate four POSTs back into one request.

Without 3, the repo would merely tolerate the change: full_stack_e2e would go
green because it opted out, and nothing would assert the fan-out happens or
has the shape sidecar authors are told to expect.

Red before green, both captured: full_stack_e2e failed on the count assertion
before the pin and passes after; stage_fanout_e2e fails on the stage-set
assertion when its hook is scoped to the request stage and passes unscoped.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant