Log full agent invocation trace with intermediate tool calls - #30
Open
mborne wants to merge 3 commits into
Open
Conversation
…calls in reports/{model_name}/{test_name}.txt (refs #28)
Contributor
Author
|
Here is an example of the |
There was a problem hiding this comment.
Pull request overview
This PR adds per-test conversation tracing for the MCP agent used in the test suite, writing serialized agent interactions to reports/<model>/<test>.txt to improve debuggability (including tool-call visibility via LangChain message rendering).
Changes:
- Add
slugify()andwrite_agent_trace()helpers to safely generate trace paths and write conversation logs. - Wrap the session-scoped agent with a per-test
RecordingAgentand write traces during fixture teardown, annotated with test outcome. - Document the new
reports/traces and add unit tests for the helper behavior.
Reviewed changes
Copilot reviewed 5 out of 6 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
helpers.py |
Adds filename sanitization and trace serialization/writing helper. |
conftest.py |
Updates MCP agent fixture to record per-test invocations and emit trace files on teardown. |
test_helpers.py |
Adds unit tests for slugify() and write_agent_trace(). |
README.md |
Documents that tests produce per-test conversation trace files. |
CODING.md |
Adds usage notes and an example for inspecting trace output. |
.gitignore |
Ensures generated reports/<model>/... traces are not versioned. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Contributor
Author
|
@tgerbeau you may use the following command to test : uv run scripts/run_tests.py .\config\models-anthropic.yaml --model=claude-haiku-4-5...and then have a look to |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces conversation tracing for agent-based tests, allowing each test's agent interactions to be recorded and saved under
reports/<model>/<test>.txt. This makes it easier to debug and understand model behavior during test runs. The implementation includes new helper functions for safe file naming and trace writing, updates to the test agent fixture, and comprehensive tests for the new functionality.Agent conversation tracing and reporting:
RecordingAgentproxy and updated themcp_agentfixture to record all agent invocations during a test, writing the conversation trace toreports/<model>/<test>.txton teardown, regardless of test success or failure. This includes storing the test status for trace annotation. (conftest.py, conftest.pyL100-R147)write_agent_tracehelper function to serialize agent conversations, ensure safe file paths, and create report files only when there is content to write. (helpers.py, helpers.pyR11-R61)slugifyhelper to generate safe directory and file names from model and test names, removing provider prefixes and illegal characters. (helpers.py, helpers.pyR11-R61)Documentation updates:
CODING.mdandREADME.mdto document the new tracing feature, including usage instructions and the purpose of the generated reports. (CODING.md, [1];README.md, [2]Testing improvements:
slugifyandwrite_agent_traceto ensure correct file naming and trace writing, including edge cases where no results are present. (test_helpers.py, test_helpers.pyR436-R474)These changes provide a robust and testable mechanism for inspecting agent/model behavior during test runs, improving debuggability and transparency.
(closes #28)