diff --git a/README.md b/README.md index e0328fa6..e5c3d430 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ [![Code style: ruff](https://img.shields.io/badge/style-ruff-261230?logo=ruff&logoColor=white)](https://github.com/astral-sh/ruff) [![Type-checked: mypy](https://img.shields.io/badge/types-mypy-2a6db2?logo=python&logoColor=white)](https://mypy-lang.org/) -Python SDK for **AI Agent Assembly** — a governance-native runtime for AI agents. One `init_assembly()` call wires your agent into the policy gateway and applies pre-execution allow/deny on tool calls, without changing how the agent itself is written. +Python SDK for **AI Agent Assembly** — a governance-native runtime for AI agents. `init_assembly()` registers your agent with the policy gateway and installs the framework adapters; the adapters apply pre-execution allow/deny on tool calls. **For LangChain, `init_assembly()` alone does not govern tool calls: you must pass the SDK's callback handler** (see [Quick Start](#quick-start)). For other frameworks, see [Framework support](docs/examples/framework-support.md) and that framework's example page for what it requires. > **The SDK hands records to the runtime; it does not give you an audit trail.** The framework adapters offer a governed call's outcome to an audit hook on the governance interceptor, and over a connected runtime that hook writes it to the native event channel — the same one agent registration uses. That is a handoff, **not** evidence: the send is unacknowledged, so this SDK cannot tell you the record arrived, and does not claim it did. Downstream, [AAASM-5783](https://lightning-dust-mite.atlassian.net/browse/AAASM-5783) is open on `report_event` payloads reaching neither the live stream nor the durable entry — until it lands, no SDK can claim ADR 0033 §6 *Observed*. Without a reachable runtime there is no channel at all and nothing is emitted. > @@ -162,7 +162,13 @@ with init_assembly( api_key="dev-key", agent_id="quickstart-agent", mode="sdk-only", -): +) as ctx: + # init_assembly() does NOT hook LangChain by itself. LangChain only calls a + # callback handler it is given, so pass the SDK's handler on every run. + adapter = next((a for a in ctx.adapters if a.get_framework_name() == "langchain"), None) + if adapter is None: + raise RuntimeError("LangChain adapter is not active: is `langchain` importable?") + handler = adapter.get_callback_handler() llm = FakeListLLM(responses=[ "Thought: I should look up the user.\nAction: whoami\nAction Input: alice\n", "Thought: I have the answer.\nFinal Answer: alice is in engineering\n", @@ -172,14 +178,21 @@ with init_assembly( "Use the tools.\n{tools}\nTool names: {tool_names}\nQ: {input}\n{agent_scratchpad}" ) executor = AgentExecutor(agent=create_react_agent(llm, tools, prompt), tools=tools, max_iterations=2) - print(executor.invoke({"input": "Which team is alice on?"})["output"]) + result = executor.invoke( + {"input": "Which team is alice on?"}, + config={"callbacks": [handler]}, # <- this is what puts the policy gate in front of each tool + ) + print(result["output"]) ``` What this does: -1. `init_assembly()` registers the agent with the gateway and auto-loads the LangChain adapter — each tool call the wrapped agent makes from now on goes through the policy gate. -2. The `FakeListLLM` replays canned responses so the example runs **offline** with no real LLM. -3. The `with` block tears down the gateway connection and unwinds adapter hooks on exit. +1. `init_assembly()` registers the agent with the gateway and activates the LangChain adapter, which creates one `AssemblyCallbackHandler` (`adapter.get_callback_handler()`). It installs **no** hook into LangChain itself. +2. `config={"callbacks": [handler]}` is what routes each tool call through the policy gate. The handler runs before the tool and raises `ToolExecutionBlockedError` on a deny, so a denied tool's body does not run. **Without this argument a LangChain tool runs regardless of policy**, even though `init_assembly()` succeeded and reports the adapter as active. +3. The `FakeListLLM` replays canned responses so the example runs **offline** with no real LLM. +4. The `with` block tears down the gateway connection and unwinds adapter hooks on exit. + +> **What `init_assembly()` alone does not enforce.** It does not govern LangChain tool calls (pass the handler as above), and an offline `sdk-only` run has no gateway policy authority to ask unless a native runtime is reachable. `test/unit/test_quickstart_langchain_denied_tool_wiring.py` pins the wiring: a denied tool run with the handler leaves no side effect, and the same tool run without it executes. ## Public API @@ -190,8 +203,10 @@ What this does: Agent **registration** and the **pre-execution policy check** are not REST calls: the SDK goes through the native `aa-sdk-client` shim to the core over gRPC/UDS (ADR 0004) — it never calls a core HTTP endpoint directly for those. -`init_assembly` registers the agent on startup, and a tool call is checked via -the native `query_policy` so a `deny` blocks it before the tool runs. +`init_assembly` registers the agent on startup, and a tool call that reaches an +adapter hook (for LangChain, only when the callback handler is passed in +`callbacks`) is checked via the native `query_policy` so a `deny` blocks it +before the tool runs. ### Control-plane routing diff --git a/docs/concepts/architecture.md b/docs/concepts/architecture.md index fa5fb2ac..dd6684ed 100644 --- a/docs/concepts/architecture.md +++ b/docs/concepts/architecture.md @@ -24,7 +24,7 @@ The SDK governs *third-party* agent frameworks (LangChain, LangGraph, CrewAI, Op - `get_framework_name() -> str` — the framework's import name (e.g. `"langchain"`). - `get_supported_versions() -> list[str]` — PEP 440 version specifiers. -- `register_hooks(interceptor) -> None` — install monkey-patches that route framework calls through the governance interceptor. +- `register_hooks(interceptor) -> None` — install monkey-patches that route framework calls through the governance interceptor (except LangChain, which only gets a callback handler; see below). - `unregister_hooks() -> None` — revert all patches in reverse install order. Must be idempotent. Adapter authors target this contract and nothing else. The public boundary is intentionally narrow so that adding a new framework does not require changes to the gateway client, the policy interceptor, or the registry's selection logic. @@ -37,9 +37,11 @@ Priority matters because two frameworks can coexist in the same process — e.g. ### Per-framework patches — the actual monkey-patching +(LangChain is the exception: its adapter patches nothing; see the first bullet below.) + Each adapter's `register_hooks()` delegates to one or more `RuntimePatch` objects (a `Protocol` defined in `agent_assembly.core.assembly`). A `RuntimePatch` knows how to apply and revert a single monkey-patch on a specific framework class or function. Examples: -- `agent_assembly.adapters.langchain.patch.LangChainPatch` — patches `BaseTool._run` and `BaseTool._arun` so every tool invocation passes through the governance gate. +- `agent_assembly.adapters.langchain.patch.LangChainPatch` — does **not** patch `BaseTool`. It only creates and stores the `AssemblyCallbackHandler` (`runtime.auto_inject_callback_handler`); a LangChain tool is governed only when that handler is passed as `callbacks=[...]`. - `agent_assembly.adapters.langchain.langgraph_patch.patch_stategraph_compile` — wraps `StateGraph.compile()` so the resulting graph's nodes are wrapped before any invocation. - `agent_assembly.adapters.crewai.patch.CrewAIPatch` — analogous wrappers for CrewAI's tool invocation entry points. @@ -54,7 +56,7 @@ flowchart LR Registry["AdapterRegistry
get_available_adapters_by_priority()"] Adapter["FrameworkAdapter
(LangChainAdapter, CrewAIAdapter, …)"] Patch["RuntimePatch
(LangChainPatch, CrewAIPatch, …)"] - Framework["Third-party framework
(BaseTool._run, StateGraph.compile, …)"] + Framework["Third-party framework
(e.g. StateGraph.compile; LangChain is not patched)"] Interceptor["GovernanceInterceptor"] Gateway["Gateway / policy engine"] diff --git a/docs/concepts/index.md b/docs/concepts/index.md index e3d3d023..8747373a 100644 --- a/docs/concepts/index.md +++ b/docs/concepts/index.md @@ -36,8 +36,9 @@ Assembly**. It does this with a three-layer pattern: each one's `is_available()` to see if its framework is importable in this process, and returns the available ones in priority order. 3. **Per-framework patches** — each adapter's `register_hooks()` installs one or more - `RuntimePatch` objects that monkey-patch the framework's actual tool-call entry points - (e.g. LangChain's `BaseTool._run`). + `RuntimePatch` objects. Some adapters patch framework entry points (for example the Agno adapter patches + `FunctionCall.execute` and the LangGraph adapter wraps `StateGraph.compile`); the LangChain + adapter does not patch `BaseTool` — it only creates a callback handler that you pass as `callbacks`. `init_assembly()` calls `AdapterRegistry.get_available_adapters_by_priority()` exactly once at startup — this is the **single detection path** (see [ADR-0001](../development/adr/0001-hook-architecture.md)). diff --git a/docs/examples/framework-support.md b/docs/examples/framework-support.md index 9473ad7f..5d498b09 100644 --- a/docs/examples/framework-support.md +++ b/docs/examples/framework-support.md @@ -30,20 +30,22 @@ with init_assembly( ... ``` -Inside the `with` block, the relevant adapter has monkey-patched the framework's tool-call -entry points. On exit, all hooks are removed in reverse order. +Inside the `with` block, each importable framework's adapter has registered its hooks. What a +hook governs depends on the framework: LangChain gets only a callback handler that you must pass +as `callbacks` (see the LangChain quick start below), so run your agent with that wiring. On exit, +all hooks are removed in reverse order. ## Supported frameworks Each framework below ships an adapter under `agent_assembly/adapters/`. The -**Runnable example** column reflects whether a complete, validated example exists today — +**Runnable example** column reflects whether a complete, validated example exists today (the LangChain row states its own, narrower status) — either inline in this guide or as a curated script in the central [`examples`](https://github.com/ai-agent-assembly/examples/tree/master/python) repository. | Framework | Adapter | Runnable example | | --- | --- | --- | -| LangChain | `agent_assembly.adapters.langchain` | ✅ Validated — see [Quick start](#langchain-quick-start) below (runs offline against a mock LLM), plus the [LangChain basic agent](langchain-basic-agent.md) and [LangChain research agent](langchain-research-agent.md) pages. | +| LangChain | `agent_assembly.adapters.langchain` | Example pages present; the callback wiring is pinned by a unit test on a bare tool, but the `AgentExecutor` quick-start snippet has not been run — see [Quick start](#langchain-quick-start) below, plus the [LangChain basic agent](langchain-basic-agent.md) and [LangChain research agent](langchain-research-agent.md) pages. | | LangGraph | `agent_assembly.adapters.langgraph` | ✅ Validated — see [LangGraph node-level governance](langgraph.md) (state-graph governance; wraps the compiled `StateGraph`). | | CrewAI | `agent_assembly.adapters.crewai` | ✅ Validated — see [CrewAI research crew](crewai-research-crew.md) (multi-agent crew). | | OpenAI Agents | `agent_assembly.adapters.openai_agents` | ✅ Validated — see [OpenAI Agents SDK](openai-agents-sdk.md). | @@ -58,7 +60,9 @@ repository. !!! note "Adapter present vs. example present" Every framework above has an **adapter that is implemented and registered** — `init_assembly()` - will detect and hook the framework whenever it's installed, regardless of the example status. + activates it whenever the framework is importable, regardless of the example status. What an + adapter then governs differs per framework: for LangChain, `init_assembly()` only creates the + callback handler and you must pass it as `callbacks` (see the quick start below). A ✅ row additionally has a curated, end-to-end runnable example you can clone and run. A ⏳ row (currently only MCP servers) means the adapter ships but a runnable example is not yet vendored. Contributions adding one are welcome; see @@ -86,7 +90,13 @@ with init_assembly( api_key="dev-key", agent_id="quickstart-agent", mode="sdk-only", -): +) as ctx: + # init_assembly() does NOT hook LangChain by itself. LangChain only calls a + # callback handler it is given, so pass the SDK's handler on every run. + adapter = next((a for a in ctx.adapters if a.get_framework_name() == "langchain"), None) + if adapter is None: + raise RuntimeError("LangChain adapter is not active: is `langchain` importable?") + handler = adapter.get_callback_handler() llm = FakeListLLM(responses=[ "Thought: I should look up the user.\nAction: whoami\nAction Input: alice\n", "Thought: I have the answer.\nFinal Answer: alice is in engineering\n", @@ -96,11 +106,17 @@ with init_assembly( "Use the tools.\n{tools}\nTool names: {tool_names}\nQ: {input}\n{agent_scratchpad}" ) executor = AgentExecutor(agent=create_react_agent(llm, tools, prompt), tools=tools, max_iterations=2) - print(executor.invoke({"input": "Which team is alice on?"})["output"]) + result = executor.invoke( + {"input": "Which team is alice on?"}, + config={"callbacks": [handler]}, # <- this is what puts the policy gate in front of each tool + ) + print(result["output"]) ``` -The same snippet lives in the repository [README](https://github.com/ai-agent-assembly/python-sdk#quick-start) -and is the canonical, validated example. See the [`examples/`](https://github.com/ai-agent-assembly/python-sdk/tree/master/examples) +Without `config={"callbacks": [handler]}` a LangChain tool runs regardless of policy. The wiring +mechanism is pinned by `test/unit/test_quickstart_langchain_denied_tool_wiring.py` (a bare +`langchain_core` tool; this exact `AgentExecutor` snippet has not been run). The same snippet +lives in the repository [README](https://github.com/ai-agent-assembly/python-sdk#quick-start). See the [`examples/`](https://github.com/ai-agent-assembly/python-sdk/tree/master/examples) directory for additional in-repo runnable scripts and their status. ## More runnable examples diff --git a/docs/examples/langchain-basic-agent.md b/docs/examples/langchain-basic-agent.md index 9af89fab..08e54e60 100644 --- a/docs/examples/langchain-basic-agent.md +++ b/docs/examples/langchain-basic-agent.md @@ -4,7 +4,7 @@ Integrate Agent Assembly with LangChain to enforce governance policy on tool cal ## What this example demonstrates -This example wires Agent Assembly governance into a LangChain agent so that each tool call the agent makes is checked against policy *before* the tool runs. It covers: +This example wires Agent Assembly governance into a LangChain agent so that each tool call the agent makes is checked against policy *before* the tool runs. The example calls the `AssemblyCallbackHandler` explicitly; LangChain does not govern a tool unless the handler is passed in `callbacks`. It covers: - Initializing Agent Assembly with `init_assembly()`. - Wrapping LangChain tools with `AssemblyCallbackHandler` + a governance interceptor. diff --git a/docs/index.md b/docs/index.md index 2e667b5f..5153e2d5 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,8 +1,8 @@ # Agent Assembly Python SDK **In plain terms:** this SDK is how a Python agent asks for permission before it acts. -You wrap your existing agent in one `init_assembly()` call, and from that point on each -tool call your agent makes is checked against a governance policy — allowed or denied — +You wrap your existing agent in one `init_assembly()` call, and each +tool call that reaches an installed adapter hook is checked against a governance policy — allowed or denied — without you rewriting a single line of the agent itself. Over a connected runtime the outcome of each call is also handed to the runtime's event channel (see below). @@ -24,8 +24,9 @@ You keep writing agents the way you always have. The SDK is the seatbelt you cli The SDK does not replace your agent framework — it **intercepts** it. When you call `init_assembly()`, the SDK detects which supported framework is importable in your process -and monkey-patches that framework's tool-call entry points so each call passes through a -policy gate first. Today it ships adapters for **LangChain, LangGraph, CrewAI, OpenAI Agents, +and installs its adapter. **For LangChain,** `init_assembly()` only creates the callback handler, +and LangChain runs a tool ungoverned unless you pass it as `callbacks=[...]` (see the README +Quick Start). For other frameworks, see [Framework support](examples/framework-support.md) and that framework's example page for what it requires. Today it ships adapters for **LangChain, LangGraph, CrewAI, OpenAI Agents, Pydantic AI, Google ADK, and MCP servers**. ```mermaid diff --git a/test/unit/test_quickstart_langchain_denied_tool_wiring.py b/test/unit/test_quickstart_langchain_denied_tool_wiring.py new file mode 100644 index 00000000..c95eaf70 --- /dev/null +++ b/test/unit/test_quickstart_langchain_denied_tool_wiring.py @@ -0,0 +1,142 @@ +"""A denied LangChain tool has no effect only when the callback is wired (AAASM-6415). + +The README Quick Start used to say that ``init_assembly()`` alone sends "each tool +call the wrapped agent makes from now on" through the policy gate. For LangChain +that is false: ``init_assembly()`` registers the adapter and stores one +``AssemblyCallbackHandler`` (``get_callback_handler()``), but it installs no hook +into LangChain itself. LangChain only calls a handler it is handed through +``callbacks=[...]``, so an un-wired tool runs whatever the policy says. The +corrected Quick Start therefore passes the handler explicitly. + +What runs here: the real ``init_assembly`` (real ``RuntimeQueryInterceptor``, fake +native runtime as the decision source, as in ``test_quickstart_negative_control.py``), +the handler taken from the langchain adapter on the returned context, and a bare +``langchain_core`` ``StructuredTool`` invoked through LangChain's own callback +manager. The tool body writes a file, so a deny is asserted as the absence of that +file. This proves the handler-wiring mechanism the Quick Start relies on; it does +NOT run ``AgentExecutor``/``create_react_agent``, so propagation of +``config={"callbacks": [...]}`` through an agent executor is not verified here. + +* ``test_denied_tool_without_wiring_still_executes`` is the negative control and + documents the gap: same init, same deny, handler not passed, file written. +* ``test_denied_tool_with_wiring_has_no_side_effect`` is the claim. +* ``test_allowed_tool_with_wiring_executes`` is the positive control; without it + "no file" is equally explained by the tool never running at all. + +``langchain`` itself is not a dependency of the test environment, and the adapter +is only selected when ``import langchain`` succeeds (``FrameworkAdapter.is_available``), +so a bare module is placed in ``sys.modules`` to make that probe pass. Only the +probe is satisfied; the adapter, handler and tool-invocation path are the real ones. +""" + +from __future__ import annotations + +import sys +import types +from collections.abc import Iterator +from pathlib import Path +from typing import Any + +import pytest + +pytest.importorskip("langchain_core") +from langchain_core.tools import StructuredTool # noqa: E402 + +from agent_assembly import init_assembly # noqa: E402 +from agent_assembly.adapters.langchain import LangChainAdapter # noqa: E402 +from agent_assembly.core import assembly as core_assembly # noqa: E402 +from agent_assembly.exceptions import ToolExecutionBlockedError # noqa: E402 + +from .core._fake_core import FakeRuntimeClient, install_fake_core # noqa: E402 +from .negative_control import FileSideEffect # noqa: E402 + +_AGENT_ID = "quickstart-langchain-wiring-agent" +_DENY_REASON = "policy forbids disk writes" + + +@pytest.fixture(autouse=True) +def _cleanup_active_context() -> None: + active = core_assembly._ACTIVE_CONTEXT + if active is not None and not active.is_shutdown: + active.shutdown() + core_assembly._ACTIVE_CONTEXT = None + + +@pytest.fixture(autouse=True) +def _langchain_importable(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setitem(sys.modules, "langchain", types.ModuleType("langchain")) + + +@pytest.fixture +def file_effect(tmp_path: Path) -> FileSideEffect: + return FileSideEffect(path=tmp_path / "langchain-tool-write.txt") + + +def _quickstart(monkeypatch: pytest.MonkeyPatch, *, decision: str) -> Iterator[tuple[Any, Any]]: + install_fake_core(monkeypatch, FakeRuntimeClient(decision=decision, reason=_DENY_REASON)) + monkeypatch.setattr( + core_assembly, + "_start_network_layer", + lambda **_kwargs: ("sdk-only", core_assembly._noop_shutdown), + ) + context = init_assembly( + gateway_url="https://gateway.test", + api_key="test-key", + agent_id=_AGENT_ID, + mode="sdk-only", + enforcement_mode="enforce", + ) + try: + adapter = next(a for a in context.adapters if isinstance(a, LangChainAdapter)) + yield context, adapter.get_callback_handler() + finally: + context.shutdown() + + +@pytest.fixture +def denied(monkeypatch: pytest.MonkeyPatch) -> Iterator[tuple[Any, Any]]: + yield from _quickstart(monkeypatch, decision="deny") + + +@pytest.fixture +def allowed(monkeypatch: pytest.MonkeyPatch) -> Iterator[tuple[Any, Any]]: + yield from _quickstart(monkeypatch, decision="allow") + + +def _write_tool(effect: FileSideEffect) -> StructuredTool: + def write_to_disk(content: str) -> str: + """Write ``content`` to disk.""" + return effect.write(content) + + return StructuredTool.from_function(write_to_disk, name="write_to_disk", description="Write to disk.") + + +def test_denied_tool_without_wiring_still_executes(denied: tuple[Any, Any], file_effect: FileSideEffect) -> None: + """init_assembly() alone does not gate a LangChain tool: the gap the old README hid.""" + _, handler = denied + assert handler is not None + + result = _write_tool(file_effect).invoke({"content": "ungated"}) + + assert file_effect.occurred() is True + assert file_effect.content() == "ungated" + assert result == str(file_effect.path) + + +def test_denied_tool_with_wiring_has_no_side_effect(denied: tuple[Any, Any], file_effect: FileSideEffect) -> None: + _, handler = denied + + with pytest.raises(ToolExecutionBlockedError, match=_DENY_REASON): + _write_tool(file_effect).invoke({"content": "denied"}, config={"callbacks": [handler]}) + + assert file_effect.occurred() is False + assert file_effect.content() is None + + +def test_allowed_tool_with_wiring_executes(allowed: tuple[Any, Any], file_effect: FileSideEffect) -> None: + _, handler = allowed + + result = _write_tool(file_effect).invoke({"content": "allowed"}, config={"callbacks": [handler]}) + + assert result == str(file_effect.path) + assert file_effect.content() == "allowed"