Repository navigation
feat(agents): the Run Analyst says whether a finished run regressed - #554
Merged
Merged
Conversation
PR 1 of plans/autonomous-agents.md §2.6, which this commit adds: an agent at step 0 of the autonomy ladder that reads Kates with an agent key that can only read, and posts to Slack or GitHub only when a run is not normal. - receiver.py: the test.completed webhook. Kates signs no body, so the body only names a run id; the worker reads the run back and drops it quietly when it does not exist, has not finished, or is over 48 hours old. 202 at once, dedupe on testId:status for 7 days, a budget of analyses per hour, one GATE_ERROR notice an hour while Kates cannot be read. - poll.py: the same with no webhook, for a laptop and a kind lab. On an empty store the first poll marks existing runs as seen. - analyst.py: one Claude Agent SDK session whose only tools are kates mcp's read tools (tools=[], dontAsk, no setting sources), a cost cap, structured output, and one retry with the reasons an answer was refused. - verdict.py: the answer's schema; the verdict re-derived from the answer's own fields by the plan's rules; every number in the headline and the next command held by a field; the next command read-only. - sinks.py: log and a JSON file always; Slack and one GitHub issue per run for verdicts other than NORMAL, with Slack controls and @ escaped. The unit tests need no model, Kates or SDK; CI · Lint runs them.
…n's cost The first lab run (panda, 2026-10-29) wrote sentences with figures into caveats, such as "p99LatencyMs +66.09%". verdict.py counted every number in a field as grounding, so a figure written there could back a headline number no tool gave. - caveats must match ^[a-z0-9]+(-[a-z0-9]+)*$, in the schema the SDK enforces and in verdict.py, and are left out of the grounding numbers. - analyst.py logs each session's cost, turns and outcome as the result arrives, before the SDK can raise after a failed session. - The session's working directory is created if missing: the SDK refuses to start without it. Re-analysing run 4d786f7d with this passed first time with caveats ["summary-averages-started-tasks"], at $0.89 for 5 turns.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Kates already grades a finished run (
assess_run, the noise band, the advisor), but only when someone asks. The Run Analyst asks for them: when a run finishes it says whether the run is normal, regressed, improved, or can't be judged, and why, and posts only when it is not normal. It is step 0 of the ladder inplans/autonomous-agents.md(added here), so it needs nothing new from the backend. Its key istype: agentwithscopes: [read], which Kates already enforces (#349, #350).The most consequential choice: the model does not decide the verdict alone.
verdict.pyworks the verdict out again from the answer's own evidence fields, using the plan's rules, and refuses any answer that disagrees. It also refuses a headline number that isn't held in a structured field, and anextcommand that would change anything. A refused answer gets one retry with the reasons, then becomes a one-lineANALYST_ERRORnotice.What
agents/run-analyst/:receiver.py: thetest.completedwebhook. Kates signs no body, so the body is only a hint naming a run id. The receiver answers 202 at once and reads the run back with the agent key. It drops the event quietly when the run does not exist, has not finished, or was created more than 48 h ago. It dedupes ontestId:statusfor 7 days and allows 20 analyses an hour. While Kates cannot be read, it posts oneGATE_ERRORnotice an hour.poll.py: the same analyses with no webhook, for a laptop or a kind lab. Kates refuses to deliver webhooks to loopback. On an empty store, the first poll marks the runs already there as seen.analyst.py: one Claude Agent SDK session. Its only tools are sixkates mcpread tools (tools=[],permission_mode="dontAsk", no setting sources), with a cost cap and structured output. The key goes tokates mcpin its environment, never on its command line.sinks.py: every answer goes to the log and to a JSON file. Verdicts other than NORMAL also go to Slack and to one GitHub issue per run. Slack control characters and@are escaped, so text from Kates data can't ping anyone.lint.yml: a newagentsfilter and job that runs the unit tests, added to the gate.Deliberately not done
analyst-*eval tasks and the rules-only arm (plan §2.5): PR 2.eventsfield, which is stored today but never used.Lab run (panda, 2026-10-29)
Poll mode with a
type: agent,scopes: [read]key, against two LOAD runs of 50,000 records:114b1df1: NOT_COMPARABLE. No LOAD baseline is set, and only two earlier runs share its spec.4d786f7d: IMPROVED. p99 82.367 ms against a 3-run band of 97.919–369.919 ms. Checked against the runs with a human key.Both answers passed
verdict.pyfirst time, but the model wrote sentences with figures intocaveats, which counted as grounding for the headline. The second commit makes caveats ids only and leaves them out of grounding, and it logs each session's cost. Re-analysing4d786f7dwith it passed first time withcaveats: ["summary-averages-started-tasks"], at $0.89 for 5 turns.Tests
Run locally:
python3 -m unittest discover -p 'test_*.py'inagents/run-analyst: 40 tests on Python 3.15 and 3.14.actionlint,scripts/check-versions.sh --workflows, shellcheck,scripts/check-install-kates.sh, hadolint, yamllint (only warnings that were already there),make check-help, the chart-versions checks, andspotless:check(all jobs inCI · Lint, which runs in full becauselint.ymlchanged).go vetpasses. golangci-lint 2.15.0 flags two errcheck issues incli/cmd/mcp_test.go, which this PR doesn't touch. CI pins v2.14.0.