AgentMedic

AgentMedic is an engineering-grade diagnostic framework for subjecting autonomous AI agents to synthetic validation scenarios. It detects structural output failures, timeout regressions, token/cost inflation, and tool-use instability inside isolated subprocess runs.

Architectural Overview

• Isolation engine: Executes target agents as independent subprocesses.
• Telemetry hooks: Captures duration, token counters, tool interactions, and scenario lifecycle events.
• Delta analysis: Compares current runs against local historical baselines.
• Structural enforcement: Validates agent output against expected JSON or Markdown schemas.

30-Second Quickstart

pip install pyyaml pytest
python src/profiler_engine.py

Sample synthetic run output:

Running profiling matrix for: SCEN_001_DIRTY_INPUT
Running profiling matrix for: SCEN_002_TOOL_FAILURE

[SUCCESS] Baseline profile matrix exported to results/matrix.md

The generated report looks like:

| Scenario ID | Status | Duration | Token Volume | Cost (USD) | Tool Interactions | Observations / Failures |
|---|---|---|---|---|---|---|
| **SCEN_001_DIRTY_INPUT** | `PASS` | 1.4s | 1800 | $0.009960 | 1 | N/A |
| **SCEN_002_TOOL_FAILURE** | `FAIL` | 2.1s | 1220 | $0.008700 | 2 | Runtime Exit Code 2 |

Project Structure

agentmedic/
├── config/                  # Scenario matrix configuration
├── src/                     # Profiler engine and telemetry helpers
├── tests/                   # Pytest coverage for validation behavior
└── results/                 # Generated local reports, ignored in normal use

Run Tests

pip install pyyaml pytest
pytest tests/ -v

License

MIT. See LICENSE for details.