diff --git a/research/__init__.py b/research/__init__.py new file mode 100644 index 0000000..4606669 --- /dev/null +++ b/research/__init__.py @@ -0,0 +1 @@ +"""Research experiments built on CodeGuard (not part of the released package).""" diff --git a/research/finding_identity/.gitignore b/research/finding_identity/.gitignore new file mode 100644 index 0000000..351a656 --- /dev/null +++ b/research/finding_identity/.gitignore @@ -0,0 +1,2 @@ +.repos/ +results/ diff --git a/research/finding_identity/README.md b/research/finding_identity/README.md new file mode 100644 index 0000000..6c915c5 --- /dev/null +++ b/research/finding_identity/README.md @@ -0,0 +1,91 @@ +# Finding identity: replay harness + +Research code for the study *How well do static-analysis findings keep their identity as +code changes?* It is not part of the released `codeguard` package. + +Baselines, suppressions, "fail only on new findings" CI checks and alert de-duplication all +rely on recognising that a finding in version *n+1* of a file is the same finding as one in +version *n*. This harness replays real git histories, scans every changed file before and +after each commit with CodeGuard, and scores several **identity schemes** against a +**ground truth** built from the diff. + +## Schemes + +Every key includes the rule ID and the file path. + +| Scheme | Identity is built from | +|---|---| +| `line` | the line number | +| `line_text` | the exact text of the flagged line | +| `github` | GitHub code scanning's `primaryLocationLineHash` (exact port of `fingerprints.ts` from github/codeql-action, verified against the original) | +| `codeguard` | CodeGuard's `codeguard/v1` fingerprint: enclosing function/class plus the normalized statement (literals masked, comments and whitespace removed) | +| `cg_no_scope` | ablation: `codeguard` without the enclosing scope | +| `cg_no_mask` | ablation: `codeguard` without masking literals | +| `cg_raw` | ablation: `codeguard` without any normalization (scope plus raw line) | + +## Ground truth + +For each changed file, old lines are mapped to new lines with a line diff +(`linemap.py`). Unchanged lines map exactly ("moved"). Inside a replaced block, lines are +paired when their similarity is at least 0.6 ("edited"). An old and a new finding are the +same finding when they have the same rule and the old line maps to the new line. Each new +finding is then truly **known** (it existed before) or truly **new**. + +## Outcomes and metrics + +For each new finding, a scheme either matches it to an old finding or not: + +| Outcome | Truth | Scheme | Meaning | +|---|---|---|---| +| `known_ok` | known | matched to the right finding | correct | +| `mispaired` | known | matched to a different old finding | wrong history carried over | +| `false_new` | known | not matched | old problem reported as new: noise, broken baselines | +| `false_known` | new | matched to an old finding | **new problem hidden** | +| `new_ok` | new | not matched | correct | + +The summary reports, per scheme: + +- **false-new rate** = `false_new / known`, also split by moved and edited findings +- **hidden rate** = `false_known / introduced` +- **mispair rate** = `mispaired / known` +- **collision rate** = findings sharing a key with another finding in the same file version, + over all findings in all scanned versions + +Only files that changed in a commit are compared. Every scheme trivially keeps the findings +in untouched files, so including them would only inflate the counts. + +## Usage + +From the repository root, with CodeGuard installed (`pip install -e .`): + +```bash +# Replay the last 200 first-parent commits of a repository (clones shallowly if a URL) +python -m research.finding_identity replay https://github.com/pallets/flask --commits 200 + +# Branch, label, rule categories, and one record per finding for manual labelling +python -m research.finding_identity replay https://github.com/DefectDojo/django-DefectDojo \ + --ref dev --commits 200 --label defectdojo --categories security,performance,quality --details + +# Rate tables for one or more runs; pools all runs into an "ALL" table +python -m research.finding_identity summarize research/finding_identity/results/* --csv rates.csv +``` + +Clones go to `.repos/` and results to `results/