Repository navigation
Hosts, receipts and reproduction (adapters, receipts renderer, REPRODUCE.md, agentdojo-tables CI) - #56
Merged
Conversation
…d, agentdojo-tables CI
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hosts, receipts and reproduction. No engine change: every document byte for byte as 0.12.0.
Adapters (docs/adapters.md). The AgentDojo run's host logic as one module,
python/probbit_host.py(StanceHost: one individual per conversation, oneprobbit liveevent per model call, thesecuritysense from tool results, the stance block for the system prompt), and the same host in Node (examples/node/stance-host.mjs). Bindings: Claude Code hooks (examples/hosts/claude-code/,UserPromptSubmit+PostToolUse→additionalContext), the OpenAI Agents SDK (dynamic instructions +RunHooks.on_tool_end), LangChain / LangGraph (callback + Runnable), the Vercel AI SDK (prepareStep+onToolExecutionEnd). The table indocs/adapters.mdsays which are tested here (Python host, Node host, the Claude Code hook contract: real binary, no model) and which are written to the framework's contract only ("not run here").Receipts (docs/receipts.md).
python3 python/probbit_host.py receipts STRAND: a strand replayed from its header alone, one record per decision (inputs as used, stance, moods, habits, the engine'swhy, the replayed state digest against the logged one), thenprobbit live verifyfor the verdict. Worked example from the run (bench/agentdojo/receipts-example.txt: banking user task 13 / injection task 1, arm A stopped the attack with the task completed, arms B and C fell for it). What the receipt proves (the engine's decision and inputs) and what it does not (the model's obedience, the words).Reproduction (REPRODUCE.md). Five minutes (prove a rule, verify a recorded strand), thirty seconds (recompute the §14 tables, sha256s listed), an afternoon (the benchmark). A reproduction-report issue template. CI job
agentdojo-tables: the tables recomputed on Ubuntu and macOS runners on every push and PR, byte-identical or the job fails. BENCHMARKS.md §14 gets a "Reproductions" line (none by outside people yet; said so).Tests:
python -m unittest discover -s python60/60 (11 new inpython/test_host.py),node --test examples/node/stance-host.test.mjs3/3, both against the 0.12.0 binary; the two offline adapter smokes exit 0. Sweeps on every added line: law 0, private 0, keys 0.