Finding identity: 35-repository corpus, batch runner and statistics

research/finding_identity/.gitignore

@@ -1,2 +1,3 @@
 .repos/
 results/
+tables/

research/finding_identity/README.md

@@ -118,7 +118,8 @@ python -m research.finding_identity drift-summary research/finding_identity/resu
 ```
 
 Clones go to `.repos/` and results to `results/<label>/` (drift: `results/drift/<label>/`);
-both are git-ignored. Each result
+both are git-ignored, as is `tables/` (see [Statistics](#statistics)); the aggregated
+tables of the corpus run are committed under `paper/`. Each result
 directory holds `summary.json`, `pairs.jsonl` (one line per commit pair) and, with
 `--details`, `findings.jsonl` (one line per compared finding with its true status and each
 scheme's outcome, for sampling and manual checks).
@@ -129,6 +130,182 @@ Tests:
 python -m pytest research/finding_identity/tests
 ```
 
+`pip install -r research/finding_identity/requirements.txt` adds the optional plotting
+dependency.
+
+## Corpus
+
+`corpus.csv` lists the repositories of the study, each pinned to one commit. Columns:
+`label`, `url`, `branch`, `sha` (the pinned commit), `pinned` (date of pinning),
+`language` (backend language: `python`, or `javascript` for JavaScript/TypeScript),
+`framework`, `license` (SPDX), `rationale` (why it qualifies and where its
+security-relevant code is), and four columns measured at the pinned commit by
+`corpus check`: `commit_date`, `commits_2y`, `py_files`, `js_files`.
+
+### Selection criteria
+
+A repository is in the corpus when all of these hold:
+
+1. **Kind.** A deployable web *application* (or an application server distributed as a
+   package), not a library, framework, SDK, tutorial, template, sample or intentionally
+   vulnerable training project; and not a fork. Judged by hand from the project's own
+   description; the `rationale` column records it.
+2. **Language.** A Python back end (Flask, Django, FastAPI, or another Python web
+   framework: Pyramid, web.py, Frappe) or a JavaScript/TypeScript back end (Express,
+   Koa, Fastify, NestJS, Next.js API routes). The back-end language must have at least
+   100 source files that the harness analyses (vendored, generated and minified files
+   excluded) and more of them than the other language, so that every repository counts
+   cleanly as a Python or a JavaScript/TypeScript project.
+3. **Security-relevant code.** Handles HTTP requests from users and does at least one of:
+   builds SQL or other database queries, renders templates, runs subprocesses, accepts
+   file uploads, fetches user-supplied URLs, or implements authentication. Recorded in
+   `rationale`.
+4. **History.** At least 200 first-parent commits on the pinned branch in the 730 days
+   before the pinned commit (the replay needs 200 commit pairs; this also shows
+   sustained activity).
+5. **Activity.** The pinned commit is at most 90 days older than the pinning date.
+6. **Licence.** The root licence file (`LICENSE`, `LICENCE` or `COPYING`) is one
+   OSI-approved licence (MIT, BSD-2/3-Clause, Apache-2.0, MPL-2.0, GPL-2.0/3.0,
+   AGPL-3.0, LGPL, PostgreSQL) for the whole repository. Source-available licences
+   (Business Source, Sustainable Use, Functional Source, Elastic, Commons Clause) and
+   root licences that put some directories under other terms (typically an `ee/`
+   enterprise edition) are excluded, because the replay scans every directory.
+7. **No committed build output.** The repository does not track bundler output (minified,
+   content-hashed JavaScript) outside the paths the harness already skips (`dist/`,
+   `build/`, `node_modules/`, `*.min.js`, ...). Such files are rewritten wholesale by every
+   build and would swamp the real code's findings.
+8. **Branch.** The default branch, unless the project develops on another branch
+   (DefectDojo's `dev`, IntelOwl's `develop`: their default branches only receive
+   releases); the branch is recorded.
+
+Criteria 1, 3 and 7 are judged by hand. The file counts of criterion 2 and criteria 4, 5
+and 6 are measured from git alone at the pinned commit, so anyone can re-check them later
+with the same result:
+
+```bash
+python -m research.finding_identity corpus check research/finding_identity/corpus.csv
+```
+
+`corpus pin` re-pins every row (or `--only a,b`) to the current head of its branch; that
+is how the corpus was built, from a candidate list of 60 repositories on 2026-10-14. The
+35 that met every criterion (25 Python, 10 JavaScript/TypeScript) form the corpus; the 25
+that failed a criterion were dropped (reasons as `corpus check` reported them; "mixed"
+means the root licence puts some directories under other terms). Criterion 7 was added
+during the first batch run, when CTFd turned out to commit its Vite build output: its
+replay compared 12,944 findings in its first 75 commit pairs, against at most about 4,000
+over 200 pairs for any other repository. No other repository was affected. Its finished
+replay (200 pairs, 28 minutes) has 22,122 persisting findings, as many as the other 35
+repositories together (23,237); including it would raise the pooled `codeguard`
+false-new rate from 1.8% to 48.7% while the per-repository median stays at 0.5%.
+
+| Candidate | Reason |
+|---|---|
+| ctfd (CTFd/CTFd) | commits Vite build output (`CTFd/themes/*/static/assets/*-<hash>.js`) that the harness does not skip (criterion 7) |
+| paperless-ngx (paperless-ngx/paperless-ngx) | Python is not the majority language (410 py, 435 js) |
+| mathesar (mathesar-foundation/mathesar) | pinned commit is 93 days old; Python is not the majority language (235 py, 492 js) |
+| redash (getredash/redash) | only 194 first-parent commits in 730 days; Python is not the majority language (291 py, 577 js) |
+| superset (apache/superset) | Python is not the majority language (2988 py, 4187 js) |
+| dispatch (Netflix/dispatch) | pinned commit is 407 days old |
+| polar (polarsource/polar) | Python is not the majority language (2167 py, 2473 js) |
+| archery (hhyo/Archery) | only 102 first-parent commits in 730 days |
+| tandoor (TandoorRecipes/recipes) | licence file reads as non-osi (commons clause), declared AGPL-3.0 |
+| label-studio (HumanSignal/label-studio) | Python is not the majority language (746 py, 1951 js) |
+| plane (makeplane/plane) | Python is not the majority language (671 py, 3031 js) |
+| privacyidea (privacyidea/privacyidea) | Python is not the majority language (685 py, 989 js) |
+| authentik (goauthentik/authentik) | licence file reads as mixed (MIT), declared MIT |
+| posthog (PostHog/posthog) | licence file reads as mixed (MIT), declared MIT |
+| seahub (haiwen/seahub) | Python is not the majority language (1069 py, 1592 js) |
+| twenty (twentyhq/twenty) | licence file reads as mixed (AGPL-3.0), declared AGPL-3.0 |
+| strapi (strapi/strapi) | licence file reads as mixed (MIT), declared MIT |
+| medusa (medusajs/medusa) | licence file reads as mixed (MIT), declared MIT |
+| parse-server (parse-community/parse-server) | only 83 first-parent commits in 730 days |
+| umami (umami-software/umami) | only 87 first-parent commits in 730 days |
+| budibase (Budibase/budibase) | licence file reads as non-osi (business source license), declared GPL-3.0 |
+| rocketchat (RocketChat/Rocket.Chat) | licence file reads as mixed (MIT), declared MIT |
+| infisical (Infisical/infisical) | licence file reads as mixed (MIT), declared MIT |
+| dub (dubinc/dub) | licence file reads as mixed (AGPL-3.0), declared AGPL-3.0 |
+| novu (novuhq/novu) | licence file reads as missing, declared MIT |
+
+## Batch runs
+
+```bash
+# Replay every corpus repository at its pinned SHA, 4 repositories at a time
+python -m research.finding_identity batch research/finding_identity/corpus.csv \
+    --commits 200 --jobs 4
+```
+
+Each repository is fetched shallowly at its pinned SHA (no checkout) into `.repos/<label>`
+and replayed in its own `replay` process, so a crash or a `--timeout` only affects that
+repository. The batch is **resumable**: a repository whose `results/<label>/summary.json`
+was produced from the pinned SHA with at least `--commits` commit pairs is skipped, so
+rerunning the same command only retries failures and repositories that were not reached
+(`--force` reruns everything; a changed SHA or a larger `--commits` reruns that
+repository). Failures do not stop the batch: each attempt is appended to
+`results/batch.jsonl` (label, status, error, seconds) and each replay's output goes to
+`results/_logs/<label>.log`; the command exits with status 1 if any repository failed.
+`--only a,b` restricts the run, `--delete-clones` removes each clone after its replay.
+
+Every replay also records its totals **per rule category** (`by_category` in
+`summary.json`, with the same `totals` and `collisions` blocks as the summary itself).
+Scheme keys include the rule ID and the ground truth pairs findings of the same rule
+only, so each category is scored on its own findings and the categories add up exactly to
+the totals.
+
+## Statistics
+
+```bash
+python -m research.finding_identity stats research/finding_identity/results \
+    --corpus research/finding_identity/corpus.csv --out research/finding_identity/tables --plot
+```
+
+For each scope (`all` findings and `security` findings by default; `--scopes` takes any
+rule categories), each scheme and each metric (`false_new`, `false_new_moved`,
+`false_new_edited`, `hidden`, `mispair`, `collision`, defined as above), `stats` reports:
+
+- **pooled (micro) rate**: total numerator over total denominator across repositories,
+  with a **Wilson 95% interval**. Wilson assumes independent findings, so it is too narrow
+  for a corpus where findings in one repository are correlated;
+- **cluster bootstrap 95% interval** of the pooled rate: repositories are resampled with
+  replacement (10,000 resamples, fixed seed, the same resamples for every table), the
+  pooled rate is recomputed, and the 2.5th and 97.5th percentiles are taken. This is the
+  interval to quote;
+- **macro rate**: the mean of per-repository rates, with its own cluster bootstrap
+  interval. Pooled rates are dominated by the repositories with the most findings; macro
+  rates weigh every repository equally;
+- **median and interquartile range** of the per-repository rates (quantiles with linear
+  interpolation, numpy's default).
+
+A repository with a zero denominator for a metric (for example no persisting security
+findings) has no rate: it adds nothing to the pooled sums and is left out of the macro
+rate, median and quartiles; `repos_with_rate` counts the rest.
+
+Outputs in `--out`:
+
+| File | Contents |
+|---|---|
+| `pooled.csv` | one row per scope, scheme and metric: counts, pooled rate, Wilson and bootstrap intervals, macro rate and interval, median, quartiles |
+| `per_repo.csv` | one row per scope, repository, scheme and metric: numerator, denominator, rate |
+| `repos.csv` | one row per repository: pinned head, commit pairs, versions scanned, runtime, findings per scope |
+| `tables.md` | the pooled tables in Markdown |
+| `per_repo.md` | per-repository false-new rates of every scheme, per scope, in Markdown |
+| `false_new_by_repo.png` | with `--plot`: box plot of per-repository false-new rates per scheme, one panel per scope (needs `matplotlib`, see `requirements.txt`) |
+
+### Committed results
+
+`paper/` holds the aggregated outputs of the corpus run of 2026-10-14 (CodeGuard 2.2.0,
+200 commit pairs per repository, all rule categories), produced with:
+
+```bash
+python -m research.finding_identity batch research/finding_identity/corpus.csv --commits 200 --jobs 4
+python -m research.finding_identity stats research/finding_identity/results \
+    --corpus research/finding_identity/corpus.csv --out research/finding_identity/paper --plot
+```
+
+The batch took 14 min 44 s of wall-clock time for the 35 repositories on 4 cores
+(2,781 s summed over repositories, fetching included): 7,000 commit pairs, 34,275 file
+versions scanned, 36,232 findings, 8,703 of them security findings. Clones and per-commit
+results are not committed.
+
 ## Known limitations
 
 - The ground truth is a line diff. A finding whose anchor line is rewritten beyond the
@@ -137,6 +314,10 @@ python -m pytest research/finding_identity/tests
   form feeds and other rare separators, so in files containing them the `codeguard` key can
   hash a neighbouring line. The harness measures the tool as it is.
 - Taint findings depend on several lines; the ground truth tracks only the reported line.
+- Every scheme's key includes the file path, so a renamed or moved file makes every
+  scheme report its findings as new, even on unchanged lines. In the corpus run this is
+  most of the residual false-new rate of the text-based schemes (for example Ghost, which
+  renamed many `.js` files to `.ts`).
 
 Drift mode adds its own limits:
 

research/finding_identity/batch.py

@@ -0,0 +1,177 @@
+"""Replay every repository of the corpus at its pinned commit, several at a time.
+
+Each repository is fetched shallowly at its pinned SHA (no checkout) and replayed in its
+own ``python -m research.finding_identity replay`` process, so one crash or timeout
+affects only that repository. The batch is resumable: a repository whose
+``summary.json`` already covers the pinned SHA and the requested number of commit pairs
+is skipped. Every attempt is appended to ``<out>/batch.jsonl`` and each replay's output
+goes to ``<out>/_logs/<label>.log``.
+"""
+
+from __future__ import annotations
+
+import json
+import shutil
+import subprocess
+import sys
+import time
+import traceback
+from collections.abc import Callable, Sequence
+from concurrent.futures import ThreadPoolExecutor, as_completed
+from dataclasses import dataclass
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Any
+
+from .corpus import CorpusEntry, fetch_pinned, first_parent_count, has_commit
+
+#: The directory ``python -m research.finding_identity`` must run from.
+PROJECT_ROOT = Path(__file__).resolve().parents[2]
+
+
+@dataclass(frozen=True)
+class BatchOptions:
+    commits: int
+    out: Path
+    workdir: Path
+    categories: str = "security,performance,quality"
+    details: bool = False
+    timeout: float | None = None
+    delete_clones: bool = False
+
+
+def summary_path(opts: BatchOptions, label: str) -> Path:
+    return opts.out / label / "summary.json"
+
+
+def is_done(entry: CorpusEntry, opts: BatchOptions) -> bool:
+    """True if a finished replay of *entry* at its pinned SHA covers ``opts.commits``."""
+    path = summary_path(opts, entry.label)
+    if not path.exists():
+        return False
+    try:
+        summary = json.loads(path.read_text(encoding="utf-8"))
+    except (OSError, ValueError):
+        return False
+    return bool(summary.get("head") == entry.sha and summary.get("commit_pairs", 0) >= opts.commits)
+
+
+def prepare(entry: CorpusEntry, opts: BatchOptions) -> Path:
+    """A local repository holding the pinned commit and enough first-parent history."""
+    target = opts.workdir / entry.label
+    if not has_commit(target, entry.sha) or first_parent_count(target, entry.sha) <= opts.commits:
+        fetch_pinned(entry.url, entry.sha, target, depth=opts.commits + 2)
+    return target
+
+
+def run_one(entry: CorpusEntry, opts: BatchOptions) -> None:
+    """Fetch and replay one repository; raise if the replay does not finish."""
+    log_path = opts.out / "_logs" / f"{entry.label}.log"
+    log_path.parent.mkdir(parents=True, exist_ok=True)
+    target = prepare(entry, opts)
+    cmd = [
+        sys.executable,
+        "-m",
+        "research.finding_identity",
+        "replay",
+        str(target.resolve()),
+        "--ref",
+        entry.sha,
+        "--commits",
+        str(opts.commits),
+        "--label",
+        entry.label,
+        "--out",
+        str((opts.out / entry.label).resolve()),
+        "--categories",
+        opts.categories,
+    ]
+    if opts.details:
+        cmd.append("--details")
+    with log_path.open("w", encoding="utf-8") as log:
+        log.write(" ".join(cmd) + "\n")
+        log.flush()
+        proc = subprocess.run(
+            cmd,
+            cwd=PROJECT_ROOT,
+            stdout=log,
+            stderr=subprocess.STDOUT,
+            timeout=opts.timeout,
+            check=False,
+        )
+    if proc.returncode != 0:
+        raise RuntimeError(f"replay exited with status {proc.returncode}, see {log_path}")
+    if not is_done(entry, opts):
+        raise RuntimeError(f"replay finished without a complete summary.json, see {log_path}")
+    if opts.delete_clones:
+        shutil.rmtree(target, ignore_errors=True)
+
+
+Runner = Callable[[CorpusEntry, BatchOptions], None]
+
+
+def _attempt(entry: CorpusEntry, opts: BatchOptions, runner: Runner) -> dict[str, Any]:
+    started = time.monotonic()
+    status: dict[str, Any] = {"label": entry.label, "sha": entry.sha, "status": "ok"}
+    try:
+        runner(entry, opts)
+    except subprocess.TimeoutExpired:
+        status.update(status="failed", error=f"timed out after {opts.timeout}s")
+    except Exception as exc:  # one repository must not stop the batch
+        status.update(status="failed", error=f"{type(exc).__name__}: {exc}")
+        log_path = opts.out / "_logs" / f"{entry.label}.log"
+        log_path.parent.mkdir(parents=True, exist_ok=True)
+        with log_path.open("a", encoding="utf-8") as log:
+            log.write(traceback.format_exc())
+    status["seconds"] = round(time.monotonic() - started, 1)
+    status["finished"] = datetime.now(timezone.utc).isoformat(timespec="seconds")
+    return status
+
+
+def run_batch(
+    entries: Sequence[CorpusEntry],
+    opts: BatchOptions,
+    *,
+    jobs: int = 1,
+    force: bool = False,
+    runner: Runner = run_one,
+) -> list[dict[str, Any]]:
+    """Replay every entry not already done; return one status record per entry."""
+    opts.out.mkdir(parents=True, exist_ok=True)
+    statuses: list[dict[str, Any]] = []
+    pending: list[CorpusEntry] = []
+    for entry in entries:
+        if not force and is_done(entry, opts):
+            statuses.append({"label": entry.label, "sha": entry.sha, "status": "skipped"})
+        else:
+            pending.append(entry)
+    print(
+        f"{len(entries)} repositories: {len(statuses)} already done, {len(pending)} to run "
+        f"with {jobs} job(s)",
+        file=sys.stderr,
+    )
+    started = time.monotonic()
+    with (
+        ThreadPoolExecutor(max_workers=max(jobs, 1)) as pool,
+        (opts.out / "batch.jsonl").open("a", encoding="utf-8") as journal,
+    ):
+        futures = {pool.submit(_attempt, e, opts, runner): e for e in pending}
+        for done, future in enumerate(as_completed(futures), start=1):
+            status = future.result()
+            statuses.append(status)
+            journal.write(json.dumps(status) + "\n")
+            journal.flush()
+            note = f" ({status['error']})" if status["status"] == "failed" else ""
+            print(
+                f"[{done}/{len(pending)}] {status['label']}: {status['status']} "
+                f"in {status['seconds']:.0f}s{note}",
+                file=sys.stderr,
+            )
+    failed = [s["label"] for s in statuses if s["status"] == "failed"]
+    print(
+        f"batch finished in {time.monotonic() - started:.0f}s; "
+        f"{len(failed)} failed{': ' + ', '.join(failed) if failed else ''}",
+        file=sys.stderr,
+    )
+    order = {e.label: i for i, e in enumerate(entries)}
+    return sorted(statuses, key=lambda s: order[s["label"]])