Skip to content

Extend v1 findings output with provenance, corroboration, and run metadata - #460

Open
CalebKAston wants to merge 2 commits into
getsentry:mainfrom
babylist:output-schema-extend-v1-upstream
Open

Extend v1 findings output with provenance, corroboration, and run metadata#460
CalebKAston wants to merge 2 commits into
getsentry:mainfrom
babylist:output-schema-extend-v1-upstream

Conversation

@CalebKAston

@CalebKAston CalebKAston commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Summary

The GitHub Action's warden-findings.json export (v1) is the durable artifact external consumers build on (dashboards, telemetry pipelines, alerting). Right now it's missing data those consumers need to reason about why a finding is (or isn't) in the file, and which run/build produced it.

This adds that data additively — version stays '1', every new field is optional, so existing consumers see zero difference unless they read the new fields:

  • harness (name/version/actionRef) — which build of Warden produced this run.
  • resolvedDefaults — the action-level failOn/reportOn/etc. this run resolved to.
  • skippedTriggers[] — configured triggers that didn't fire this run, with a structured reason (path_filter, draft_state, label_mismatch, no_event_match, error, pending).
  • Per-skill skillExecutionId/triggerId/triggerName — a stable join key across live snapshots, the final write, and (in split analyze/report mode) the replay artifact.
  • Per-finding contentHash, reportedId (set once cross-run dedupe recenters a finding onto an already-posted comment's id), reportedBy[] (which skills independently flagged this finding), and provenance (verification/merge history), plus a top-level discardedFindings[] for candidates that were rejected, merged away, or deduped before ever reaching findings[].
  • Live incremental writes — the findings file is rewritten after each trigger settles (no .done marker, no findings-file action output — those stay reserved for the run's one true final write), so an external follower can observe progress instead of only the terminal state.

New fields at a glance

All fields below are new and optional. Grouped by where they live in the JSON structure.

Top level

Field Type Description
runAttempt string GitHub Actions run attempt number.
harness { name, version, actionRef? } Which build of Warden produced this run.
resolvedDefaults { failOn?, reportOn?, failCheck?, requestChanges?, maxFindings? } The action-level config this run resolved to.
skippedTriggers[] { skillName, triggerId?, triggerName?, reason }[] Configured triggers that never fired this run, with why (path_filter, draft_state, label_mismatch, no_event_match, error, pending).
discardedFindings[] { stage, severity, title, location?, survivorFindingId?, ... }[] Candidates that never reached findings[] — rejected by verification, absorbed by a merge, or dropped as a duplicate.
summary.totalSkillExecutions number Same as totalSkills today; kept distinct in case grouping ever changes.
summary.byOutcome { posted, deduped, skipped, resolved, failed } Tally of findingObservations by outcome.

Per skill (skills[])

Field Type Description
skillExecutionId string Stable join key for this skill×trigger execution — ties a skill row to its triggerResults[]/replay entry.
triggerId / triggerName string Which configured trigger produced this run.
auxiliaryModel / synthesisModel string Models used for auxiliary/synthesis calls, when applicable.
findingsBySeverity { high, medium, low } Same shape as the top-level summary, scoped to this skill.
checkRunUrl / checkRunId string / number The GitHub Check run this skill posted to.
reviewEvent 'APPROVE' | 'REQUEST_CHANGES' | 'COMMENT' The review event actually posted to GitHub (absent from analyze-mode replay and live writes, since nothing's posted yet).
checkConclusion 'success' | 'failure' | 'neutral' | 'cancelled' Mirrors the conclusion posted to the check run.
issueNumber / issueUrl number / string Schedule-mode only — the tracking issue this run created/updated.

Per finding (skills[].findings[])

Field Type Description
contentHash string Stable cross-run key — same hash output/dedup.ts already uses for hash-based dedupe.
reportedId string Set to the same value as id once cross-run dedupe recenters this finding onto an already-posted comment's id.
verification string The verifier's evidence trace, when a verification pass ran.
reportedBy[] { skillExecutionId?, skillName, role, matchType? }[] Which skills independently flagged this finding — self as role: 'primary', others matched via dedupe as 'corroborating'.
provenance { originSkillExecutionId?, originModel?, verification?, merge? } This finding's history — was it revised by verification, or merged from sibling findings at other locations?

Implementation notes

  • skillExecutionId is a short hash of the existing per-trigger replay identity (config/loader.ts), reused everywhere a stable join key is needed.
  • provenance.ts turns captured FindingProcessingEvents (verification/merge/dedupe stages the SDK already emits via onFindingProcessing) into per-finding provenance and the discardedFindings list; this now round-trips through the analyze/report replay artifact so report mode's export doesn't lose it.
  • The report-mode replay's legacy name+skill fallback join (for artifacts predating triggerId) now fails loudly instead of silently guessing when 2+ current triggers share a name and skill.
  • getVersion() now prefers GITHUB_ACTION_PATH to locate package.json, correct regardless of whether the running code is TypeScript source or ncc's flattened dist/action bundle — the same relative-path assumption that works from source lands on the wrong package.json once bundled, which would otherwise break every real Action run given harness.version is a required field. Verified this specific failure mode live (see below).

Test plan

  • pnpm lint && pnpm build && pnpm test — all green (92 test files, 1801 passing, 4 pre-existing skips, 0 failures)
  • New/extended tests across provenance.test.ts, utils/fs.test.ts, utils/version.test.ts, output.test.ts, pr-workflow.test.ts, schedule.test.ts, poster.test.ts, base.test.ts, dedup.test.ts, extract.test.ts, executor.test.ts, matcher.test.ts covering the new fields, live-write cadence, skip-reason derivation, chained-merge attribution, and the report-mode ambiguous-fallback fix
  • Byte-for-byte regression test proving the exact pre-existing output shape is unchanged when none of the new inputs are available
  • Live-validated end-to-end against a real consuming repo (two rounds: run/skill-level fields, then a deliberately-planted bug to exercise the per-finding fields) — this is what caught the getVersion()/ncc-bundling bug above, which no unit test could reach since it only manifests once code is actually running from the compiled dist/action bundle rather than TypeScript source

🤖 Generated with Claude Code

…adata

Extends the GitHub Action's v1 warden-findings.json export with data an
external consumer needs to build real observability on top of Warden runs,
without breaking existing consumers: version stays '1', every new field is
additive and optional.

- harness (name/version/actionRef) — which build of Warden produced this run.
- resolvedDefaults — the action-level fail-on/report-on/etc. this run resolved to.
- skippedTriggers[] — configured triggers that didn't fire this run, with a
  structured reason (path_filter, draft_state, label_mismatch, no_event_match,
  error, pending).
- Per-skill skillExecutionId/triggerId/triggerName — a stable join key across
  live snapshots, the final write, and (in split analyze/report mode) the
  replay artifact.
- Per-finding contentHash, reportedId (set once cross-run dedupe recenters a
  finding onto an already-posted comment's id), reportedBy[] (which skills
  independently flagged this finding), and provenance (verification/merge
  history) plus a top-level discardedFindings[] for candidates that were
  rejected, merged away, or deduped before ever reaching findings[].
- Live incremental writes: the findings file is rewritten after each trigger
  settles (no .done marker, no findings-file action output — those stay
  reserved for the run's one true final write), so an external follower can
  observe progress instead of only the terminal state.

Implementation notes:
- skillExecutionId is a short hash of the existing per-trigger replay
  identity (config/loader.ts), reused everywhere a stable join key is needed.
- provenance.ts turns captured FindingProcessingEvents (verification/merge/
  dedupe stages the SDK already emits via onFindingProcessing) into
  per-finding provenance and the discardedFindings list; this now round-trips
  through the analyze/report replay artifact so report mode's export doesn't
  lose it.
- The report-mode replay's legacy name+skill fallback join (for artifacts
  predating triggerId) now fails loudly instead of silently guessing when 2+
  current triggers share a name and skill.
- getVersion() now prefers GITHUB_ACTION_PATH to locate package.json,
  correct regardless of whether the running code is TypeScript source or
  ncc's flattened dist/action bundle — the same relative-path assumption
  that works from source lands on the wrong package.json once bundled,
  which would otherwise break every real Action run given harness.version
  is a required field.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 2 potential issues.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Want reviews to match your repository better? Bugbot Learning can learn team-specific rules from PR activity. A team admin can enable Learning in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 02256da. Configure here.

Comment thread packages/warden/src/action/reporting/output.ts Outdated
Comment thread packages/warden/src/action/workflow/pr-workflow.ts
runAttempt never fell back to GITHUB_RUN_ATTEMPT (unlike runId), so
production findings exports never carried it. The PR path also never
threaded a trigger's auxiliaryModel/synthesisModel onto TriggerResult,
so those fields stayed unset outside schedule mode.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant