diff --git a/research/finding_identity/.gitignore b/research/finding_identity/.gitignore index 351a656..ec761da 100644 --- a/research/finding_identity/.gitignore +++ b/research/finding_identity/.gitignore @@ -1,2 +1,3 @@ .repos/ results/ +.cache/ diff --git a/research/finding_identity/README.md b/research/finding_identity/README.md index 6c915c5..89c86b1 100644 --- a/research/finding_identity/README.md +++ b/research/finding_identity/README.md @@ -81,6 +81,104 @@ Tests: python -m pytest research/finding_identity/tests ``` +## Controlled transformation experiment (RQ2) + +Real histories rarely edit the flagged line itself and introduce few new findings, so +they cannot say *which kinds of change break which schemes*. The `transform` subcommand +answers that with a controlled experiment: it samples Python files with findings, +rewrites each one with a set of transformations whose effect on line identity is known +by construction, scans before and after, and scores every scheme. + +| Transformation | What it does | Behaviour | +|---|---|---| +| `reformat` | `black` at line length 79 (`--reformat-line-length`; `ruff format` if black is missing) | kept | +| `insert_top` | a comment and an unused import below the docstring and `__future__` imports | kept | +| `insert_below` | a comment line right below the flagged statement (GitHub's hash reads 100 non-whitespace characters forward) | kept | +| `rename_local` | renames a local variable (not a parameter) used on the flagged line, in its whole function | kept | +| `rename_function` | renames the enclosing function or method and its references in the file (`self.m`, `cls.m`, `Class.m` for methods) | kept in the file | +| `move_function` | swaps the enclosing function with the adjacent function in the same body | kept | +| `change_literal` | appends `_v2` to a string literal on the flagged line, or adds one to a number | changed | +| `extract_helper` | moves the flagged statement into a new module-level function; reads become parameters, assignments are returned | kept | +| `duplicate_above` / `duplicate_below` | copies the flagged statement directly above / below itself: a genuinely new finding | changed | +| `ai_refactor` | asks a language model to refactor the enclosing function without changing its behaviour (optional) | checked by hand | + +Each transformation returns the new text and an exact old-line to new-line map, which +is the ground truth. `reformat` pairs the syntax trees before and after (black keeps the +tree), so when it splits a line the finding is followed to the line its node starts on. +`ai_refactor` is the exception: its ground truth is the line diff of `linemap.py`, so +its cases are flagged `needs_review` and written to `review.jsonl` with the old and new +function for a manual check. + +Renames, extraction and moves are rewritten with `libcst` (source preserving) and only +where they are safe within the file: names bound once, no `global`/`nonlocal`/`locals()`, +no special methods, no overridden method names, no statements that `yield`, `await` or +use `super()`. When a transformation does not fit a target finding it tries the next +finding in the file (in a seeded random order); if none fits, the file counts as "n/a" +for that transformation, with the reason. Other modules are not updated by renames. + +### Running it + +```bash +pip install -r research/finding_identity/requirements.txt + +python -m research.finding_identity transform https://github.com/pallets/flask --samples 200 --heatmap +python -m research.finding_identity transform https://github.com/DefectDojo/django-DefectDojo \ + --ref dev --label defectdojo --samples 200 --heatmap + +# A subset of transformations, on a local directory +python -m research.finding_identity transform path/to/src --transforms rename_local,extract_helper +``` + +`--samples` is the number of files with findings to use; `--seed` fixes the sample and +the target findings. Results go to `results/transform-