feat(reporting): add configuredSkills roster to findings output - #436
feat(reporting): add configuredSkills roster to findings output#436CalebKAston wants to merge 5 commits into
Conversation
|
one concern i have here is this bloats the jsonl since skills arent really related to each individual finding i had been thinking about this because the reporting format, due to it only being a log, ends up somewhat limited. my thinking was to bundle a metadata file in addition to the log. i dont think this exists yet (tho i might have started it). I assume that'd help the situation you have? e.g. foobar.jsonl next to foobar-meta.json, and the meta file would be like the total configuration and maybe aggregate statistics |
I think it's interesting that you say the skills aren't related to the individual findings. Why do you say that? I have this mental model that the skills are the source of the findings, meaning there's a one-to-many relationship between skills and findings. Maybe there's some verifier-level abstraction that "disconnects" the findings from the skills when aggregating and presenting them? And the metadata file would be a solid option for helping my situation. My goal is to get as much "relevant" information about the runs as possible. If the best way to do that is to break out a separate file, then I'm on board with that. What's your reasoning for being concerned about jsonl bloat? I'd be happy to take a crack at building the metadata file version of this, but I want to understand your mental model of what each file represents so I can get it right. |
…portWorkflow options
0d884e2 to
297f1de
Compare
|
One of my other open PRs (#440) might fit better in the metadata file, if that's the pattern you'd like to take here. Calling that out so we don't have the same conversation twice. |
|
@CalebKAston more what i mean is: each finding is found by a skill (or multiple) - to me that makes sense in each log entry, which one(s) found it - but the broader "which skills were configured in total" is more meta level, that i think should just be a non-log-based format fwiw theres probably a handful of jsonl things that might better fit as aggregate concerns. i had this issue in the past when i was tyring to move to a concrete jsonl thing, because i wanted to be able to fully repro the CLI output just from the log files, but doing that is a little odd unless you support random types of log entries, and i didnt love that as it seemed overkill |
|
@dcramer interesting... I'll give the output file format some deep thought today and put together a new PR that uses a different file output structure (and captures the three additions from my open PRs) and we can chat there about how you'd like to move forward. |
Summary
configuredSkillsarray to the findings-file JSON: one{name, triggered}row per configured skill, so downstream consumers can see skills that never triggered (e.g. schedule-only skills on apull_requestrun), not just the ones that ran.buildConfiguredSkillsList()inaction/reporting/output.ts, wired throughwriteFindingsOutputand everypr-workflow.tscall site that writes the findings artifact.pr-workflow.ts). The schedule workflow'swriteFindingsOutputcall sites are untouched, so scheduled-run findings files do not carry aconfiguredSkillsroster — the field is optional everywhere, so this is not a breaking change for existing consumers.Real-world verification
This feature (plus the two hardening fixes folded into this PR: documenting the dedup behavior and closing a silent-empty-roster gap in
finalizeReportWorkflow) has been running in production on ourbabylist/wardenfork since 2026-07-15, against our main application monorepo. TheWarden / reviewcheck has run successfully across 275+ merged pull requests in that window — this isn't a new, unexercised code path.Test plan
pnpm test -- output.test.ts— schema/helper unit tests (roster present/absent, dedup across multiple trigger blocks for one skill, empty-config case).pnpm test -- pr-workflow.test.ts— integration coverage including a schedule-only-skill fixture proving a configured-but-never-matched skill still appears withtriggered: false.pnpm lint && pnpm build && pnpm test— full repo-wide run, clean (1774 passed, 4 skipped).