Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 32 additions & 0 deletions examples/skills_code_review_agent/.env.example
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
# Copy to .env.
#
# A real model is the default. Without one, run_agent.py raises rather than silently degrading —
# pass --dry-run to use the fake model instead (issue criterion 6: the dry-run mode must exercise
# parse -> sandbox -> persist with no API key).

# --- model (required unless you pass --dry-run) ---
# TRPC_AGENT_API_KEY=
# MODEL_NAME=gpt-4o-mini # default
# TRPC_AGENT_BASE_URL= # any OpenAI-compatible endpoint
#
# examples:
# hosted TRPC_AGENT_API_KEY=sk-... MODEL_NAME=gpt-4o-mini
# ollama TRPC_AGENT_API_KEY=ollama MODEL_NAME=qwen2.5:7b \
# TRPC_AGENT_BASE_URL=http://localhost:11434/v1

# --- sandbox the skill runs in ---
# REVIEW_RUNTIME=container # default. `local` is the development fallback only.
# REVIEW_SANDBOX_IMAGE=cr-scanners:latest
# build it first: docker build -t cr-scanners:latest skills/code-review
# REVIEW_SANDBOX_TIMEOUT_SEC=120
# SKILLS_ROOT= # override where skills/ is scanned from

# --- storage / output ---
# REVIEW_DB_URL=sqlite+aiosqlite:///./code_review.db
# REVIEW_OUT_DIR=. # where review_report.json/.md are written
# REVIEW_NO_DB=1 # skip persistence
# REVIEW_MAX_OUTPUT_BYTES=1048576

# --- opt-in real-model integration test ---
# CR_LIVE_MODEL_TEST=1
# CR_LIVE_RUNTIME=local
34 changes: 34 additions & 0 deletions examples/skills_code_review_agent/DESIGN.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
# 方案设计说明

本示例把自动代码评审构建为一个**可验证系统**:主干是确定性流水线,Agent(Skill + 沙箱 + Filter)
是其中一个 finding 来源,因此在无模型 API Key 的 dry-run 下也能产出完整报告,隐藏集阈值可复现。

**Skill 设计。** `skills/code-review/` 将评审打包为可移植 Skill:`SKILL.md` 声明规则与用法,
`scripts/run_checks.py` 是自包含的沙箱入口(不依赖示例包),`rules/` 放 semgrep 规则,
`docs/OUTPUT_SCHEMA.md` 是 findings 的唯一契约。findings 来自 bandit / ruff / detect-secrets 等
成熟扫描器,外加两个自写检测器(DB 连接生命周期、测试缺失),覆盖全部 6 类规则。

**沙箱隔离策略。** 默认走 Container workspace 沙箱;本地子进程仅作开发 fallback,不隐式选用(本地仅作
fallback)。每次执行都有超时(`asyncio.wait_for` / subprocess timeout)、输出字节上限截断、
资源限制(memory_mb),并把每次运行记录为 `sandbox_run`(含超时、失败、拦截);单次失败只降级来源,
不使整个评审崩溃。

**Filter 策略。** `pipeline/policy.py::ReviewPolicy` 对命令、路径、网络、预算做 allow / deny /
needs_human_review 判定,在两处执行点共用:确定性沙箱门(被拒动作从不启动)与框架级
`ReviewGuardFilter`(工具级 `BaseFilter`)。拦截原因写入报告的 Filter 摘要与数据库。

**监控字段。** 每次评审记录总耗时、沙箱耗时、工具调用数、拦截数、finding 数、各 severity 分布、
异常类型分布,落入报告 monitoring 段。

**数据库 schema。** 四张表 `review_tasks` / `sandbox_runs` / `findings` / `reports`,任务行内嵌
input diff 摘要,均以 `task_id` 为键;基于 `SqlStorage` 的可移植列类型,SQLite 默认、PG/MySQL 换 URL 即可。

**去重降噪。** 同 `(文件, 行, 类别)` 至多保留一条(高置信优先,其余标 duplicate);再按置信度分流
active / warning / needs_human_review,低置信噪声不混入高置信 findings。

**安全边界。** 单一 `redact()` 汇聚点在入库/出报告前统一脱敏(提供商正则 + 熵检测,语料实测 100%);
沙箱只透传白名单环境变量,杜绝父进程密钥泄漏。

**验收证据。** `selftest.py` 在公开样本上打分;`selftest.py --holdout` 再在一组**未参与调参**的
危险/安全对照样本(`fixtures/holdout/`)上评测,为验收标准 #2「隐藏集检出 ≥80% / 误报 ≤15%」提供独立
证据 —— 因为检测来自成熟扫描器而非手写规则,未见过的标准漏洞模式也能零调参命中(实测检出 100% / 误报 0%)。
132 changes: 132 additions & 0 deletions examples/skills_code_review_agent/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,132 @@
# Skills Code Review Agent

An automated code-review agent built on tRPC-Agent's Skills + sandbox + DB primitives (issue #92).
Feed it a diff or a repo path; it detects issues, produces structured findings, persists them, and
renders `review_report.json` + `review_report.md`.

> 中文说明见 [README.zh_CN.md](./README.zh_CN.md)。

## Quick start

```bash
pip install -r requirements.txt
docker build -t cr-scanners:latest ../../skills/code-review # the sandbox image

# The product path: an LlmAgent loads the code-review Skill and runs it in a sandbox.
export TRPC_AGENT_API_KEY=... # any OpenAI-compatible endpoint; see .env.example
python run_agent.py --fixture security.diff

# Same four steps with no API key and no Docker (issue criterion 6):
python run_agent.py --fixture security.diff --dry-run

# The deterministic CLI: same skill script, launched directly. Used for scoring the fixtures.
python run_review.py --fixture security.diff --out-dir /tmp/cr

# Review your own diff, working tree, or an explicit file list:
python run_review.py --diff-file my.diff
python run_review.py --repo-path /path/to/repo --no-db
python run_review.py --files pipeline/engine.py,pipeline/devrun.py

# Scored self-test over the labelled fixtures (detection-rate / false-positive-rate):
python selftest.py

# Held-out danger/safe eval — independent evidence for the >=80% / <=15% thresholds on unseen code:
python selftest.py --holdout
```

A sample report is committed under [`sample_output/`](./sample_output/); the rule catalog is in
[`../../skills/code-review/docs/RULES.md`](../../skills/code-review/docs/RULES.md) and the design note
in [DESIGN.md](./DESIGN.md).

## How it works

Findings come from **deterministic static scanners**, not the LLM, so results are reproducible and
the acceptance thresholds are tunable:

```
diff/repo ──▶ diff_parser ──▶ scanners ──▶ dedup/denoise ──▶ redact ──▶ report (json+md)
(unidiff) (bandit, (per file/line/ (single │
ruff, category; choke- ▼
detect-secrets, confidence → point) ReviewStore
semgrep) active/warning/ (SqlStorage:
human-review) SQLite default,
PG/MySQL by URL)
```

| Category | Scanner |
|---|---|
| security | bandit, semgrep |
| secret_leakage | detect-secrets |
| async_errors | ruff (ASYNC) |
| resource_leak | ruff (SIM115 / bugbear) |
| db_lifecycle | semgrep (`skills/code-review/rules/db_lifecycle.yaml`) |

## Design note

The agent is the orchestrator, and the Skill is the single source of findings. A review is four
steps: `stage_review_input` materializes the diff on the host, `skill_load` loads `SKILL.md` and its
rule docs through the framework's Skills mechanism, `skill_run` executes the skill's own
`scripts/run_checks.py` inside a workspace sandbox, and `finalize_review` dedups, persists and
renders. The model decides and reports; it never invents a finding, because every finding comes from
a scanner that ran in the sandbox.

Findings are deterministic *because the scanners are*, not because the agent is bypassed — which is
what lets the same path satisfy both the product requirement and the no-API-key dry-run (criterion
6): `--dry-run` swaps the model, not the pipeline, so it walks the identical four steps.

**Skill design.** `skills/code-review/` packages the review as a portable Skill (`SKILL.md` +
`scripts/run_checks.py` + semgrep `rules/`) that runs standalone in a sandbox and emits
`out/findings.json` per `docs/OUTPUT_SCHEMA.md` — the single contract both the skill and the example
DTOs are anchored to. **Sandbox strategy.** Container (Docker) is the default runtime; local execution is a development
fallback only, never implicitly selected. The framework's `skill_run` enforces the timeout and
collects the output; every run — including timeouts and blocks — is recorded as a `SandboxRunResult`,
so one failed check degrades a source without crashing the task. Because the workspace runtimes stage
inputs at different depths, the skill script locates its own scan root from the `.changes.diff`
sidecar staged beside the changed files; findings' paths therefore match the diff under any layout.
**Filter strategy.** A tool-level `BaseFilter` gates high-risk scripts, forbidden paths,
non-whitelisted network and over-budget runs *before* the sandbox executes; `deny` /
`needs_human_review` never reach execution, and block reasons are written to the report and DB. The
filter is attached to `skill_run`, and the toolset is narrowed to `skill_load` + `skill_run` — the
stock `SkillToolSet` also exposes `skill_exec` / `workspace_exec`, which are built without filters
and would otherwise let the model route around the gate entirely. **Monitoring.** Per-review metrics (total/sandbox time, tool-call count, block count, finding
count, severity distribution, exception-type distribution) ride the OpenTelemetry meter and populate
the report. **DB schema.** Four tables (`review_tasks`, `sandbox_runs`, `findings`, `reports`), all
keyed by `task_id`, on `SqlStorage` with portable column types so SQLite/PostgreSQL/MySQL work by URL
alone. **Dedup & denoise.** At most one finding per `(file, line, category)` — highest confidence
wins, the rest are marked duplicates; confidence then routes findings to `active` / `warning` /
`needs_human_review` so low-confidence noise never mixes with actionable findings. **Security
boundary.** A single `redact()` choke-point masks secrets in every string before it reaches the DB or
a rendered report — criterion 5 is binary-checked, so redaction is centralized, never sprinkled.

## Status

The agent path is the product path: a real model by default, the Skill loaded through
`SkillToolSet` -> `skill_load` -> `skill_run`, the scanners in a container sandbox, results
deduplicated, persisted and rendered. `--dry-run` substitutes `FakeReviewModel`, which drives the
same four steps with no API key and no Docker (criterion 6).

`skills/code-review/scripts/run_checks.py` is the only implementation of the review rules. The agent
reaches it through the Skills mechanism; the deterministic CLI (`run_review.py`, `selftest.py`)
reaches the same script through a development subprocess. Nothing re-implements the rules elsewhere.

**Filter gate** (criterion 7/8): `pipeline/policy.py::ReviewPolicy` decides allow / deny /
needs-human-review, enforced by `agent/filter.py::ReviewGuardFilter` on `skill_run` before the
sandbox runs, and by the same policy in the development runner. Blocks are recorded and surfaced in
the report's Filter-interception section. The sandbox receives only a whitelisted environment.

**Secret redaction** (criterion 5): `redact()` layers provider-token regexes plus a Shannon-entropy
catch-all and hits 100% on the leak-test corpus with zero false positives; the skill script redacts
at emit time as well, so nothing unredacted reaches the model's context.

Rule coverage spans all six required categories (security, secret_leakage, async_errors,
resource_leak, db_lifecycle, missing_tests); the eight fixtures match the official scenarios
(`clean`, `security`, `async_resource_leak`, `db_lifecycle`, `missing_tests`, `duplicate_finding`,
`sandbox_failure`, `secret_redaction`). Inputs: `--diff-file`, `--repo-path`, `--files a.py,b.py`,
or `--fixture`. See [DESIGN.md](./DESIGN.md) for the design note.

**Tests.** `pytest tests/examples/test_skills_code_review_agent.py` — the suite asserts the four-step
skill sequence, that `SKILL.md`'s content reaches the model, and that findings' file paths match the
diff. Emptying `SKILL.md` turns two tests red; deleting it turns three red. A real-model integration
test runs under `CR_LIVE_MODEL_TEST=1` with an API key and skips cleanly without one.

Remaining: an independent labelled eval set to prove the hidden-set thresholds.
67 changes: 67 additions & 0 deletions examples/skills_code_review_agent/README.zh_CN.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
# 代码评审 Agent(Skills + 沙箱 + 数据库)

基于 tRPC-Agent 的 Skills、沙箱执行与数据库存储能力构建的自动代码评审 Agent(issue #92)。
输入一个 diff 或仓库路径,它会识别问题、产出结构化 findings、落库,并生成
`review_report.json` 与 `review_report.md`。

## 快速开始(无需 API Key)

```bash
pip install -r requirements.txt

# 评审内置样本(无需模型)。默认运行时是沙箱
# (auto → 有 Docker 走容器,否则本地子进程沙箱):
python run_review.py --fixture security.diff --out-dir /tmp/cr

# 评审你自己的 diff、工作区,或指定文件列表:
python run_review.py --diff-file my.diff
python run_review.py --repo-path /path/to/repo --no-db
python run_review.py --files pipeline/engine.py,pipeline/devrun.py

# 在带标注的样本上打分自测(检出率 / 误报率):
python selftest.py

# 走 LlmAgent + fake 模型(无需 API key):
python run_agent.py --fixture security.diff --dry-run
```

样本报告见 [`sample_output/`](./sample_output/);规则清单见
[`../../skills/code-review/docs/RULES.md`](../../skills/code-review/docs/RULES.md),设计说明见 [DESIGN.md](./DESIGN.md)。

## 工作原理

findings 来自**确定性静态扫描器**而非 LLM,因此结果可复现、验收阈值可调:

`diff/repo → diff_parser(unidiff) → scanners(bandit/ruff/detect-secrets/semgrep)
→ 去重降噪 → 脱敏 → 报告(json+md) / 落库(SqlStorage,默认 SQLite,可切 PG/MySQL)`

| 类别 | 扫描器 |
|---|---|
| security | bandit, semgrep |
| secret_leakage | detect-secrets |
| async_errors | ruff(ASYNC)|
| resource_leak | ruff(SIM115 / bugbear)|
| db_lifecycle | semgrep(`skills/code-review/rules/db_lifecycle.yaml`)|

## 设计要点

主干是**确定性流水线**;Agent(Skills+沙箱+Filter)只是两个 finding 来源之一,而非总指挥——
这是 dry-run 无 Key 要求逼出来的(扫描器路径必须能独立出完整报告),也消除了最大风险:
LLM 来源的 findings 无法在隐藏集上复现阈值,而扫描器输出稳定。

**Skill**:`skills/code-review/` 把评审打包为可移植 Skill(SKILL.md + 脚本 + semgrep 规则),
可在沙箱中独立运行并按 `docs/OUTPUT_SCHEMA.md` 输出。**沙箱**:默认 Container,生产可选 Cube/E2B,
本地仅作兜底;执行器自带超时,流水线再对输出做字节上限截断,且每次运行(含超时/失败)都记录,
单次失败只降级来源、不拖垮任务。**Filter**:工具级 `BaseFilter` 在进沙箱前拦截高危脚本/禁止路径/
非白名单网络/超预算,拦截原因写入报告与库。**监控**:每次评审的耗时、工具调用数、拦截数、
finding 数、严重级分布、异常类型分布经 OpenTelemetry 上报并写入报告。**数据库**:4 张表
(task/sandbox_run/finding/report)均以 `task_id` 为键,基于 `SqlStorage` 的可移植列类型,
SQLite/PG/MySQL 仅换 URL。**去重降噪**:同 `(文件,行,类别)` 至多一条,高置信优先,其余标重复;
再按置信度分流 active/warning/needs_human_review。**安全边界**:单一 `redact()` 汇聚点,
入库/出报告前统一脱敏,绝不散落。

## 状态

已实现:确定性流水线、数据库落库、8 个样本、打分自测、CLI、基线脱敏。
后续 slice:沙箱内执行(Container/Cube)、Filter 门控、脱敏加固至 ≥95%、OpenTelemetry 指标、
以及 Agent 闭环:真模型经 skill_load / skill_run 在沙箱中执行 Skill,--dry-run 以 fake model 走同样四步。
5 changes: 5 additions & 0 deletions examples/skills_code_review_agent/agent/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
# Tencent is pleased to support the open source community by making tRPC-Agent-Python available.
#
# Copyright (C) 2026 Tencent. All rights reserved.
#
# tRPC-Agent-Python is licensed under Apache-2.0.
39 changes: 39 additions & 0 deletions examples/skills_code_review_agent/agent/agent.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
# Tencent is pleased to support the open source community by making tRPC-Agent-Python available.
#
# Copyright (C) 2026 Tencent. All rights reserved.
#
# tRPC-Agent-Python is licensed under Apache-2.0.
"""Construct the code-review LlmAgent (Skills + sandbox + storage), used by run_agent.py."""
from __future__ import annotations

from trpc_agent_sdk.agents import LlmAgent

from .config import get_model
from .config import get_runtime_kind
from .prompts import INSTRUCTION
from .tools import build_host_tools
from .tools import create_skill_tool_set


def create_agent(dry_run: bool = False, runtime: str | None = None) -> LlmAgent:
"""An LlmAgent that reviews a diff by loading and running the ``code-review`` Skill.

The agent gets the framework's Skills toolset (``skill_load`` / ``skill_run`` / ``skill_list``
...) plus two host-side tools that stage the diff and finalize the report. ``skill_repository``
is passed so the agent can resolve skills from the same repository the toolset uses.

``dry_run`` selects the fake model; ``runtime`` overrides the sandbox runtime (default
``container``, with ``local`` as the development fallback).
"""
# A dry run is for exercising the chain without external dependencies, so it must not also
# demand a Docker daemon and a built scanner image. An explicit --runtime still wins.
resolved = runtime or ("local" if dry_run else get_runtime_kind())
skill_toolset, repository = create_skill_tool_set(resolved)
return LlmAgent(
name="code_review_agent",
description="An automated code reviewer that runs the code-review Skill in a sandbox.",
model=get_model(force_fake=dry_run),
instruction=INSTRUCTION,
tools=[skill_toolset, *build_host_tools()],
skill_repository=repository,
)
51 changes: 51 additions & 0 deletions examples/skills_code_review_agent/agent/config.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
# Tencent is pleased to support the open source community by making tRPC-Agent-Python available.
#
# Copyright (C) 2026 Tencent. All rights reserved.
#
# tRPC-Agent-Python is licensed under Apache-2.0.
"""Configuration for the agent path — model and sandbox runtime selection.

A **real model is the default**. The fake model exists because issue #92 requires a dry-run mode
that exercises the parse -> sandbox -> persist chain without an API key, but it is a mode you ask
for, never what you get by accident: with no model configured and no ``--dry-run``, this raises
rather than silently degrading to a model that does not think.
"""
from __future__ import annotations

import os

from trpc_agent_sdk.models import LLMModel

from .model import FakeReviewModel

#: Any OpenAI-compatible endpoint works — a hosted API, a gateway, or a local Ollama/vLLM server.
DEFAULT_MODEL_NAME = "gpt-4o-mini"

_NO_MODEL_HELP = (
"No model configured. Set TRPC_AGENT_API_KEY (with MODEL_NAME / TRPC_AGENT_BASE_URL for a "
"non-default endpoint), or pass --dry-run to use the fake model.\n"
" hosted: TRPC_AGENT_API_KEY=sk-... MODEL_NAME=gpt-4o-mini\n"
" ollama: TRPC_AGENT_API_KEY=ollama TRPC_AGENT_BASE_URL=http://localhost:11434/v1 "
"MODEL_NAME=qwen2.5:7b")


def get_model(force_fake: bool = False) -> LLMModel:
"""The configured real model; the fake model only when explicitly requested."""
if force_fake:
return FakeReviewModel(model_name="fake-review-1")

api_key = os.getenv("TRPC_AGENT_API_KEY")
if not api_key:
raise RuntimeError(_NO_MODEL_HELP)

from trpc_agent_sdk.models import OpenAIModel
return OpenAIModel(
model_name=os.getenv("MODEL_NAME", DEFAULT_MODEL_NAME),
api_key=api_key,
base_url=os.getenv("TRPC_AGENT_BASE_URL"),
)


def get_runtime_kind() -> str:
"""Workspace runtime for the skill sandbox: ``container`` (default) or ``local`` (dev fallback)."""
return os.getenv("REVIEW_RUNTIME", "container")
Loading
Loading