Extend v1 findings output with provenance, corroboration, and run metadata - #460
Open
CalebKAston wants to merge 2 commits into
Open
Extend v1 findings output with provenance, corroboration, and run metadata#460CalebKAston wants to merge 2 commits into
CalebKAston wants to merge 2 commits into
Conversation
…adata Extends the GitHub Action's v1 warden-findings.json export with data an external consumer needs to build real observability on top of Warden runs, without breaking existing consumers: version stays '1', every new field is additive and optional. - harness (name/version/actionRef) — which build of Warden produced this run. - resolvedDefaults — the action-level fail-on/report-on/etc. this run resolved to. - skippedTriggers[] — configured triggers that didn't fire this run, with a structured reason (path_filter, draft_state, label_mismatch, no_event_match, error, pending). - Per-skill skillExecutionId/triggerId/triggerName — a stable join key across live snapshots, the final write, and (in split analyze/report mode) the replay artifact. - Per-finding contentHash, reportedId (set once cross-run dedupe recenters a finding onto an already-posted comment's id), reportedBy[] (which skills independently flagged this finding), and provenance (verification/merge history) plus a top-level discardedFindings[] for candidates that were rejected, merged away, or deduped before ever reaching findings[]. - Live incremental writes: the findings file is rewritten after each trigger settles (no .done marker, no findings-file action output — those stay reserved for the run's one true final write), so an external follower can observe progress instead of only the terminal state. Implementation notes: - skillExecutionId is a short hash of the existing per-trigger replay identity (config/loader.ts), reused everywhere a stable join key is needed. - provenance.ts turns captured FindingProcessingEvents (verification/merge/ dedupe stages the SDK already emits via onFindingProcessing) into per-finding provenance and the discardedFindings list; this now round-trips through the analyze/report replay artifact so report mode's export doesn't lose it. - The report-mode replay's legacy name+skill fallback join (for artifacts predating triggerId) now fails loudly instead of silently guessing when 2+ current triggers share a name and skill. - getVersion() now prefers GITHUB_ACTION_PATH to locate package.json, correct regardless of whether the running code is TypeScript source or ncc's flattened dist/action bundle — the same relative-path assumption that works from source lands on the wrong package.json once bundled, which would otherwise break every real Action run given harness.version is a required field. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
4 tasks
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Want reviews to match your repository better? Bugbot Learning can learn team-specific rules from PR activity. A team admin can enable Learning in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 02256da. Configure here.
runAttempt never fell back to GITHUB_RUN_ATTEMPT (unlike runId), so production findings exports never carried it. The PR path also never threaded a trigger's auxiliaryModel/synthesisModel onto TriggerResult, so those fields stayed unset outside schedule mode. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Summary
The GitHub Action's
warden-findings.jsonexport (v1) is the durable artifact external consumers build on (dashboards, telemetry pipelines, alerting). Right now it's missing data those consumers need to reason about why a finding is (or isn't) in the file, and which run/build produced it.This adds that data additively —
versionstays'1', every new field is optional, so existing consumers see zero difference unless they read the new fields:harness(name/version/actionRef) — which build of Warden produced this run.resolvedDefaults— the action-levelfailOn/reportOn/etc. this run resolved to.skippedTriggers[]— configured triggers that didn't fire this run, with a structured reason (path_filter,draft_state,label_mismatch,no_event_match,error,pending).skillExecutionId/triggerId/triggerName— a stable join key across live snapshots, the final write, and (in splitanalyze/reportmode) the replay artifact.contentHash,reportedId(set once cross-run dedupe recenters a finding onto an already-posted comment's id),reportedBy[](which skills independently flagged this finding), andprovenance(verification/merge history), plus a top-leveldiscardedFindings[]for candidates that were rejected, merged away, or deduped before ever reachingfindings[]..donemarker, nofindings-fileaction output — those stay reserved for the run's one true final write), so an external follower can observe progress instead of only the terminal state.New fields at a glance
All fields below are new and optional. Grouped by where they live in the JSON structure.
Top level
runAttemptstringharness{ name, version, actionRef? }resolvedDefaults{ failOn?, reportOn?, failCheck?, requestChanges?, maxFindings? }skippedTriggers[]{ skillName, triggerId?, triggerName?, reason }[]path_filter,draft_state,label_mismatch,no_event_match,error,pending).discardedFindings[]{ stage, severity, title, location?, survivorFindingId?, ... }[]findings[]— rejected by verification, absorbed by a merge, or dropped as a duplicate.summary.totalSkillExecutionsnumbertotalSkillstoday; kept distinct in case grouping ever changes.summary.byOutcome{ posted, deduped, skipped, resolved, failed }findingObservationsby outcome.Per skill (
skills[])skillExecutionIdstringtriggerResults[]/replay entry.triggerId/triggerNamestringauxiliaryModel/synthesisModelstringfindingsBySeverity{ high, medium, low }checkRunUrl/checkRunIdstring/numberreviewEvent'APPROVE' | 'REQUEST_CHANGES' | 'COMMENT'checkConclusion'success' | 'failure' | 'neutral' | 'cancelled'issueNumber/issueUrlnumber/stringPer finding (
skills[].findings[])contentHashstringoutput/dedup.tsalready uses for hash-based dedupe.reportedIdstringidonce cross-run dedupe recenters this finding onto an already-posted comment's id.verificationstringreportedBy[]{ skillExecutionId?, skillName, role, matchType? }[]role: 'primary', others matched via dedupe as'corroborating'.provenance{ originSkillExecutionId?, originModel?, verification?, merge? }Implementation notes
skillExecutionIdis a short hash of the existing per-trigger replay identity (config/loader.ts), reused everywhere a stable join key is needed.provenance.tsturns capturedFindingProcessingEvents (verification/merge/dedupe stages the SDK already emits viaonFindingProcessing) into per-finding provenance and thediscardedFindingslist; this now round-trips through the analyze/report replay artifact so report mode's export doesn't lose it.triggerId) now fails loudly instead of silently guessing when 2+ current triggers share a name and skill.getVersion()now prefersGITHUB_ACTION_PATHto locatepackage.json, correct regardless of whether the running code is TypeScript source or ncc's flatteneddist/actionbundle — the same relative-path assumption that works from source lands on the wrongpackage.jsononce bundled, which would otherwise break every real Action run givenharness.versionis a required field. Verified this specific failure mode live (see below).Test plan
pnpm lint && pnpm build && pnpm test— all green (92 test files, 1801 passing, 4 pre-existing skips, 0 failures)provenance.test.ts,utils/fs.test.ts,utils/version.test.ts,output.test.ts,pr-workflow.test.ts,schedule.test.ts,poster.test.ts,base.test.ts,dedup.test.ts,extract.test.ts,executor.test.ts,matcher.test.tscovering the new fields, live-write cadence, skip-reason derivation, chained-merge attribution, and the report-mode ambiguous-fallback fixgetVersion()/ncc-bundling bug above, which no unit test could reach since it only manifests once code is actually running from the compileddist/actionbundle rather than TypeScript source🤖 Generated with Claude Code