feat(hunt): add trace-aware hunting (grounding, propagation, get_trace enrichment)

[jithin23-kv]

Problem

opfor hunt planned and judged attacks from the visible target reply alone. When a target is wired to an observability backend (Netra / Langfuse), two signals were going unused:

• Real production traces that show the agent's actual tools, data, and user flows — high-value reconnaissance the commander was ignoring in favor of generic guesses.
• Silent leaks — data that leaks into a tool call or an unauthorized record the target fetches but renders as a clean answer. Invisible in the reply, so the judge scored those turns "defended."

Solution

Optional, strictly opt-in trace-aware hunting, reusing the exact telemetry config shape opfor run already accepts. It unlocks three independently-gated capabilities (single source of truth in telemetryCapabilities()):

1. Grounded planning (provider + adapter) — curates historic traces into a summary the commander plans against. Works on any instrumented backend.
2. Propagation (propagation block) — mints an OTEL trace id per thread, injects it (header/body) on every send, and gives operators a get_trace tool to inspect the tool calls / retrieval behind a reply.
3. Finding enrichment (propagation + enrichJudgeFromTrace) — confirmed findings carry the recorded trace excerpt and the independent verifier judges against it too.

Because propagation/enrichment only work if the target echoes the injected id back into its own telemetry, a recon-time round-trip preflight verifies that assumption once and reports trace round-trip: OK / NOT DETECTED. On a miss the hunt continues (ingestion lag can cause a false negative) but warns, records it, and tells operators an empty trace is not proof the target is clean. Grounded planning is unaffected either way. Every leg degrades gracefully — a curation/fetch failure never aborts the hunt.

Two correctness fixes are included after a self-review of the feature:

• Trace cache was served stale. A per-attack/per-run trace grows as turns are added, but the memo was keyed by trace id alone — so a second get_trace (or a later-turn finding) got the first fetch back, missing later spans. Now keyed by trace id + the turn's response anchor, so each turn fetches fresh while a finding and its self_check still share a cache hit.
• Propagation-only findings self-rejected. Operators were told (propagation gate) to cite get_trace content in record_finding, but the validation fetch was gated on enrichment — so those citations were structurally rejected whenever enrichJudgeFromTrace was off, breaking the documented silent-leak-detection mode. record_finding now fetches the trace to validate a citation whenever the evidence isn't in a visible reply and propagation sent an id; feeding the trace to the verifier stays gated on the enrichment opt-in.

Changes

core (engine)

• autonomous/lib/telemetry.ts (new) — capability gating, per-thread trace-id minting/reuse (fresh id per fork), per-send propagation, curator-model auth gating, round-trip preflight, cache-aware trace fetch (traceCacheKey).
• autonomous/lib/models.ts (new) — shared resolveModelId alias resolution (extracted from selfCheck).
• autonomous/tools/getTrace.ts (new) — get_trace tool (output wrapped as untrusted).
• autonomous/tools/{recordFinding,selfCheck,sendToTarget,server}.ts — trace-aware validation/enrichment + propagation on send + tool registration.
• autonomous/{orchestrator/run.ts,orchestrator/context.ts,prompts/commander.ts,prompts/operator.ts,report/*,state/runLog.ts,target/http.ts,lib/types.ts} — wiring, prompt doctrine gated on capabilities, report/runlog fields.
• config/schema.ts — parseTelemetry() + Zod schema (bare block or { telemetry } wrapper).
• targets/httpClient.ts — inject trace id into a configured top-level body field.

Runners / docs

• runners/cli/src/commands/hunt.ts — --telemetry-config flag (or a telemetry block in --target-config) + capability-accurate banner.
• runners/sdk/src/{hunt,types}.ts — telemetry option on hunt().
• docs/hunt.md, docs/sdk.md, AGENTS.md — trace-aware hunting section + field reference.

Tests — core/tests/huntTelemetry.test.ts (new): capability gating, per-thread/per-run/per-fork id semantics, propagation header/body expansion, preflight, cache keying, and an end-to-end propagation check over the HTTP send path.

Issue

Closes #147

How to test

npm run typecheck        # clean across core + all runners
npm test                 # core suite — 183 pass / 3 skipped

End-to-end against an instrumented target (Netra example):

# telemetry.json
# { "provider": "netra",
#   "netra": { "baseUrl": "http://localhost:3000", "traceSelection": { "lookbackHours": 24 } },
#   "propagation": { "headers": { "x-trace-id": "{{traceId}}" }, "traceIdStrategy": "per-attack" },
#   "enrichJudgeFromTrace": true }

opfor hunt --endpoint https://your-target/chat --objective "Find data leaks and authz flaws" \
  --telemetry-config telemetry.json

Expect a telemetry : netra (grounding · propagation · enrichment) banner line and a trace round-trip: OK / NOT DETECTED progress line during recon. With no telemetry config the zero-config quick-start is unchanged.

Screenshots

N/A

Summary by CodeRabbit

• New Features

  • Added optional trace-aware telemetry for autonomous hunts: grounded planning, trace-id propagation to targets, preflight “OK vs not detected” checking, and finding enrichment.
  • Introduced a get_trace tool for retrieving recorded target trace excerpts.
  • Added --telemetry-config support in the hunt CLI and telemetry support in the SDK, enabling capability status reporting.

• Documentation

  • Updated hunt and SDK docs with trace-aware setup, warnings, and examples.

• Tests

  • Added extensive telemetry and end-to-end trace propagation test coverage.

• Refactor

  • Centralized shared model-alias resolution.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 44dde3a5-9900-4e96-8059-49fb53f1d25b

📥 Commits

Reviewing files that changed from the base of the PR and between 9e57e99 and 935705c.

📒 Files selected for processing (3)

• core/src/config/schema.ts
• core/tests/huntTelemetry.test.ts
• runners/cli/src/commands/hunt.ts

🚧 Files skipped from review as they are similar to previous changes (2)

• core/src/config/schema.ts
• runners/cli/src/commands/hunt.ts

---

Walkthrough

Optional telemetry support flows from hunt configuration through trace grounding, trace-ID propagation, retrieval, finding validation, report enrichment, CLI/SDK integration, and automated coverage.

Changes

Trace-aware hunt telemetry

Layer / File(s)	Summary
Telemetry configuration and entrypoints core/src/config/schema.ts, core/src/autonomous/lib/{models,types,telemetry}.ts, runners/cli/src/commands/hunt.ts, runners/sdk/src/*	Telemetry configuration is parsed and exposed through CLI and SDK hunt options, with curator model resolution and capability-aware status reporting.
Trace state and telemetry services core/src/autonomous/state/runLog.ts, core/src/autonomous/lib/telemetry.ts, core/src/autonomous/orchestrator/context.ts	Run, thread, and turn state track trace identifiers while telemetry services mint IDs, probe round trips, cache excerpts, and retrieve selected traces.
Propagation and run orchestration core/src/autonomous/target/http.ts, core/src/targets/httpClient.ts, core/src/autonomous/tools/sendToTarget.ts, core/src/autonomous/orchestrator/run.ts, core/src/autonomous/prompts/*	Trace metadata is injected into target requests, propagated per thread or run, and supplied to orchestration prompts and conditionally enabled tools.
Trace tools and finding reporting core/src/autonomous/tools/{getTrace,recordFinding,selfCheck,server}.ts, core/src/autonomous/report/*	The get_trace tool retrieves wrapped trace evidence; finding validation and self-check use trace excerpts, and reports preserve trace and telemetry fields.
Telemetry validation and documentation core/tests/huntTelemetry.test.ts, docs/{hunt,sdk}.md, AGENTS.md	Tests cover configuration, propagation, caching, evidence matching, and HTTP integration; documentation describes CLI and SDK usage.

Estimated code review effort: 4 (Complex) | ~60 minutes

Suggested reviewers: achuvyas-kv

Sequence Diagram(s)

sequenceDiagram
  participant Operator
  participant HuntCLI
  participant runAutonomous
  participant TargetClient
  participant TelemetryAdapter
  Operator->>HuntCLI: provide telemetry configuration
  HuntCLI->>runAutonomous: start hunt with telemetry
  runAutonomous->>TargetClient: probe trace propagation
  TargetClient-->>runAutonomous: return trace round-trip status
  runAutonomous->>TargetClient: send attack with trace metadata
  TargetClient-->>runAutonomous: return target response
  runAutonomous->>TelemetryAdapter: fetch recorded trace
  TelemetryAdapter-->>runAutonomous: return trace excerpt

Loading

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 34.78% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title is concise and accurately describes the main trace-aware hunt changes.
Description check	✅ Passed	The description follows the required template and includes all requested sections with concrete content.
Linked Issues check	✅ Passed	The changes satisfy the hunt telemetry objectives: config acceptance, grounded recon, trace propagation, and verifier trace enrichment.
Out of Scope Changes check	✅ Passed	The added docs, tests, and helper refactors all support the trace-aware hunt feature, with no clear unrelated changes.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 9

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

core/src/autonomous/tools/selfCheck.ts (1)

74-93: 🔒 Security & Privacy | 🟠 Major | ⚡ Quick win

Wrap the verifier’s trace data as untrusted output.

selfCheck.ts inserts traceJson directly into the verifier userPrompt, while other target-originated tool outputs are wrapped by wrapUntrustedOutput(...) with <untrusted_target_output>...</untrusted_target_output> and reinforced by the shared untrusted-output defense. Since this trace comes from target/telemetry metadata, treat it as evidence, not instructions, and reuse the shared wrapper or an equivalent explicit untrusted-data delimiter here.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@core/src/autonomous/tools/selfCheck.ts` around lines 74 - 93, Wrap traceJson
with the shared wrapUntrustedOutput(...) helper before interpolating it into
traceBlock in the verifier prompt. Preserve the existing enrichment gate and
trace content, while clearly delimiting the target/telemetry trace as untrusted
evidence rather than instructions.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@core/src/autonomous/lib/telemetry.ts`:
- Around line 222-224: Update traceCacheKey to include a unique per-turn anchor,
specifically threadId and turnIndex or an equivalent monotonic identifier,
alongside traceId and expectedResponse. Propagate the new inputs through callers
so cached traces and validation lookups cannot collide across turns.
- Around line 321-325: Update the turn selection logic around turnIndex and
traceId so numeric indexes must be positive integers within thread.turns.length;
reject 0, negative, fractional, and out-of-range values with an actionable range
error instead of falling back to thread.traceId. Preserve the existing fallback
only when turnIndex is not provided.
- Around line 312-319: Update the unavailable-trace reason messages in the
trace-availability check, including the branches around telemetry provider
validation and telemetry.propagation, to state actionable recovery steps. Tell
operators to configure telemetry.propagation when absent, send a turn before
retrying where applicable, and retry after ingestion delay while verifying
target instrumentation when traces are not yet available.

In `@core/src/autonomous/orchestrator/run.ts`:
- Around line 103-126: Update the preflight flow in the run function to check
the cancellation signal before and after curateHuntTracesIfConfigured, and
before calling probeTraceRoundTrip, so an aborted run skips further telemetry
work and agent-query creation. Pass the signal into those telemetry operations
when their APIs support it, then finalize immediately with the partial report
and stopReason "user-interrupted" after an aborted preflight.

In `@core/src/autonomous/prompts/commander.ts`:
- Around line 21-27: Update the traceSummary prompt section in the commander
prompt construction to clearly delimit the telemetry as untrusted reference
data, since it may contain attacker-controlled prompts or outputs. Explicitly
instruct the commander not to follow or execute instructions embedded in the
summary, while preserving its use for corroborating observed tools, flows, and
sensitive fields; apply the same treatment to the corresponding trace-summary
occurrence near the referenced line.

In `@core/src/autonomous/state/runLog.ts`:
- Around line 57-58: Update the recordFindingTool context so the fetched trace
remains transient when enrichJudgeFromTrace is disabled, and assign
Finding.traceJson only when ctx.telemetryCaps.enrichment is enabled. Preserve
trace use for invisible-evidence validation without persisting propagation-only
data in run logs or reports.

In `@core/src/autonomous/tools/getTrace.ts`:
- Around line 26-29: Update the missing-thread error returned in getTrace around
the thread lookup to include an actionable next step, matching the guidance and
tone of record_finding’s equivalent rejection. Keep the existing thread ID in
the message and tell the calling agent what operation must be performed before
retrying.

In `@core/src/config/schema.ts`:
- Around line 190-204: The parseTelemetry function currently treats malformed
supplied telemetry values as disabled. Distinguish an absent telemetry property
from a supplied one, route every supplied telemetry block—including non-object
values and empty objects—through TelemetryConfigSchema.safeParse so invalid
configurations throw the existing descriptive error, and return undefined only
when telemetry is absent or the parsed provider is "none".

In `@runners/cli/src/commands/hunt.ts`:
- Around line 412-420: Update the catch block around resolveTelemetry in the
hunt command to choose the error prefix based on the telemetry source: use
--telemetry-config for explicit telemetry configuration and --target-config when
telemetryFromTargetConfig was used. Preserve the existing error detail,
process.exitCode assignment, and return behavior.

---

Outside diff comments:
In `@core/src/autonomous/tools/selfCheck.ts`:
- Around line 74-93: Wrap traceJson with the shared wrapUntrustedOutput(...)
helper before interpolating it into traceBlock in the verifier prompt. Preserve
the existing enrichment gate and trace content, while clearly delimiting the
target/telemetry trace as untrusted evidence rather than instructions.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 294a90b4-bc34-4a38-b6ab-8a33e3da243c

📥 Commits

Reviewing files that changed from the base of the PR and between e45a62c and 9b3ee9e.

📒 Files selected for processing (25)

• AGENTS.md
• core/src/autonomous/lib/models.ts
• core/src/autonomous/lib/telemetry.ts
• core/src/autonomous/lib/types.ts
• core/src/autonomous/orchestrator/context.ts
• core/src/autonomous/orchestrator/run.ts
• core/src/autonomous/prompts/commander.ts
• core/src/autonomous/prompts/operator.ts
• core/src/autonomous/report/mapRunLog.ts
• core/src/autonomous/report/types.ts
• core/src/autonomous/state/runLog.ts
• core/src/autonomous/target/http.ts
• core/src/autonomous/tools/getTrace.ts
• core/src/autonomous/tools/recordFinding.ts
• core/src/autonomous/tools/selfCheck.ts
• core/src/autonomous/tools/sendToTarget.ts
• core/src/autonomous/tools/server.ts
• core/src/config/schema.ts
• core/src/targets/httpClient.ts
• core/tests/huntTelemetry.test.ts
• docs/hunt.md
• docs/sdk.md
• runners/cli/src/commands/hunt.ts
• runners/sdk/src/hunt.ts
• runners/sdk/src/types.ts

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@core/src/config/schema.ts`:
- Around line 195-199: Update unwrapTelemetryBlock and the telemetry validation
flow around TelemetryConfigSchema.safeParse so an explicitly supplied {
telemetry: null } or { telemetry: undefined } is passed to validation and fails
with “Invalid telemetry config” rather than disabling telemetry. Preserve
disabling only when the telemetry value is absent or the parsed configuration
explicitly uses provider: "none".
- Around line 198-199: Update the configuration key checks in the raw-object
parsing logic around the telemetry selection and related validation checks to
use Object.prototype.hasOwnProperty.call(...) instead of the in operator. Ensure
only own properties such as telemetry, constructor, toString, or __proto__ are
treated as present, preserving existing parsing and typo-warning behavior
otherwise.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 1090ecb9-60d8-4ba8-996d-dd00783dc16e

📥 Commits

Reviewing files that changed from the base of the PR and between 6be453c and 9e57e99.

📒 Files selected for processing (3)

• core/src/config/schema.ts
• core/tests/huntTelemetry.test.ts
• runners/cli/src/commands/hunt.ts

🚧 Files skipped from review as they are similar to previous changes (2)

• runners/cli/src/commands/hunt.ts
• core/tests/huntTelemetry.test.ts