diff --git a/CLAUDE.md b/CLAUDE.md index 128eae6..cb26d09 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -2,13 +2,17 @@ Build and test: `pip install -e . && pytest` -## Changing the `--ui live` dashboard +## Changing the `--ui live` TUI -Unit tests check strings in the frame. They can't tell you whether the layout reads well, so -look at the dashboard before you commit any change to `strain/dashboard.py`, `strain/console.py` -or `strain/keys.py`: +The TUI is a Textual app in `strain/tui.py`, drawn from the `LiveState` in `strain/live_state.py` +that the runner's console hooks update from the runner's thread. Textual is the optional `ui` extra: +without it, `--ui live` prints plain output. Keep it that way; don't add a second, hand-drawn TUI. -1. `pip install -e '.[ui-snapshot]'` (adds `pyte` and `pillow`). +The Pilot tests in `tests/test_tui.py` press keys and click headlessly. They can't tell you whether the +layout reads well, so look at the TUI before you commit any change to `strain/tui.py`, +`strain/live_state.py` or `strain/console.py`: + +1. `pip install -e '.[ui-snapshot]'` (adds `textual`, `pyte` and `pillow`). 2. Configure a copy of the fixture as a git repository with a coverage build (strain refuses a folder without `CMakeCache.txt`), using the flags from `configure_project` in `tests/test_end_to_end.py`: ```sh @@ -18,20 +22,23 @@ or `strain/keys.py`: "-DCMAKE_CXX_FLAGS=--coverage -fsanitize=address,undefined -fno-omit-frame-pointer -D_GLIBCXX_ASSERTIONS -ftrivial-auto-var-init=pattern" \ "-DCMAKE_EXE_LINKER_FLAGS=--coverage -fsanitize=address,undefined" cmake --build $demo/build - python tools/ui_snapshot.py $demo/project --size 120x36 --keys down,tab,q --out ui-snapshots \ + python tools/ui_snapshot.py $demo/project --size 120x36 --keys 2,down,3,4,q --out ui-snapshots \ -- --build-dir $PWD/$demo/build --report-dir $PWD/$demo/report ``` - This needs the fixture's toolchain (CMake, a C++ compiler, gcov) and takes about a minute. - Arguments after `--` go to `strain run`. - Also try a small size such as `--size 80x24` to check truncation and panel sizing. + - Keys can be mouse events, such as `click:12:5`; `--delay 9` waits for the toasts to fade first. 3. Read the `ui-snapshots/*.txt` grids to check alignment, and view the `*.png` frames to check - colours and reverse video. `pyte` drops SGR 2 (faint), so `DIM` text looks like normal text in the PNGs. + colours and reverse video. `pyte` drops SGR 2 (faint), so `dim` text looks like normal text in the PNGs, + and the PNG font has no braille, so the spinner shows as a box. 4. Check these points, which have regressed before: - The header totals match the survivors view, so each mutant is counted once across tests. - - The progress bar and the title name the same test as the queue. + - The header names the same test as the queue. - Each survivor detail row is padded to `DETAIL_LABEL_WIDTH`. - No panel leaves large blank areas while another panel is truncated. + - At 120x36, every table fits its panel without a horizontal scrollbar. 5. Attach before and after frames, or the relevant `.txt` excerpt, to the pull request. Don't commit `ui-snapshots/`. -Keep iterations short: render the dashboard once, fix what you see, and render it once more to confirm. +Keep iterations short: render the TUI once, fix what you see, and render it once more to confirm. diff --git a/README.md b/README.md index 44b9ca9..545cdef 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ strain changes your code in small ways (`<` becomes `<=`, `&&` becomes `||`) and checks whether your tests notice. Every mutant that survives points at a test that runs the code but doesn't really check it. -![The strain run --ui live dashboard in an 80x24 terminal, listing three survivors and showing the selected one as a diff with a hint](docs/images/dashboard-80x24.png) +![The strain run --ui live TUI in a 120x36 terminal: the Survivors tab lists three survivors and shows the selected one with its coverage, a hint and a highlighted diff](docs/images/tui-120x36.png) *`strain run --ui live` on `tests/fixtures/sample_project` after the run, with the second survivor selected.* @@ -37,14 +37,14 @@ Notes for whoever fills this in (human or Claude Code): - Optional: - `cppcheck`: the best mutation engine - `tree-sitter`: the fallback engine (pip) - - `rich`: the live dashboard (pip) + - `textual`: the live TUI, `--ui live` (pip) ## Install ```sh # TODO(claude): replace once packaging exists pip install strain # core -pip install "strain[ui]" # with the live dashboard +pip install "strain[ui]" # with the live TUI ``` ## Quick start @@ -72,6 +72,8 @@ strain run --build-dir build-strain -R parser # only tests matching "parser" strain run --build-dir build-strain --since main # only lines changed since main ``` +Add `--ui live` (with `strain[ui]` installed) to watch the run in a full-screen TUI, described [below](#live-tui). + ### 3. Read the results @@ -85,6 +87,36 @@ Action items 3. parser.cpp:97 `+` → `-` survived (not_detected): assert the computed offset ``` +## Live TUI + +`strain run --ui live` shows the run in a full-screen TUI built on [Textual](https://textual.textualize.io/). Without +Textual installed, it warns and prints plain output instead. Use the keyboard or the mouse. + +| View | Shows | +|------|-------| +| `1` Run | The mutated code as a highlighted diff, the mutant queue with each result and time, findings and the build log | +| `2` Survivors | Each survivor with its coverage, the tests it survived, a hint and its source, and the score of each file | +| `3` Mutants | Every mutant so far, filtered by state, with the same details | +| `4` Stats | The [safety nets](#safety-nets) the configuration let fail, why each matters and how to fix it; scores by file, operator and test; the slowest mutants | +| `5` Log | The full console output | + +Above the views, the header shows the score with its history, the totals, the elapsed time and an estimate of the time +left. When a safety net fails but its level is `warn` or `ignore`, a `! N safety nets off` badge appears there; click it +for the details. It is red when one of those nets errors by default, because then a mutant that breaks the code can +survive, and each survivor's details carry the same caution. Under it, a bar has one cell per mutant, colored by its result: hover to name a mutant, click to show it in the +queue. + +| Key | Does | +|-----|------| +| `1`–`5`, `tab`, `shift+tab` | Switch views; or click a tab | +| `↑` `↓`, `PgUp` `PgDn` | Move through a list; in Run, look at another mutant while the run goes on | +| `enter` | In Run, open the selected mutant in Mutants | +| `f` | In Run, follow the running mutant again; in Mutants, show the next state filter | +| `p` | Pause before the next mutant starts, with the sources unmutated; press again to resume | +| `ctrl+c` | Stop the run: strain restores the sources and writes the reports, as with `--resume` | +| `q` | Leave once the run has finished, then print the plain summary | +| `?`, `ctrl+p` | Show every key; open the command palette, which also changes the theme | + ## How it works For each selected test: diff --git a/docs/images/dashboard-80x24.png b/docs/images/dashboard-80x24.png deleted file mode 100644 index 68bb7b5..0000000 Binary files a/docs/images/dashboard-80x24.png and /dev/null differ diff --git a/docs/images/tui-120x36.png b/docs/images/tui-120x36.png new file mode 100644 index 0000000..a6794a0 Binary files /dev/null and b/docs/images/tui-120x36.png differ diff --git a/pyproject.toml b/pyproject.toml index 0aad37b..2f3d102 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -12,7 +12,8 @@ dependencies = [] [project.optional-dependencies] tree-sitter = ["tree-sitter>=0.26.0", "tree-sitter-cpp>=0.23.4"] -ui-snapshot = ["pyte>=0.8.2", "pillow>=13.0.0"] +ui = ["textual>=8.2.8"] +ui-snapshot = ["textual>=8.2.8", "pyte>=0.8.2", "pillow>=13.0.0"] [project.scripts] strain = "strain.cli:main" diff --git a/requirements-ci.in b/requirements-ci.in index c7147f6..8ecbeeb 100644 --- a/requirements-ci.in +++ b/requirements-ci.in @@ -5,3 +5,5 @@ lizard pytest ruff>=0.17.4 setuptools>=64 +# The live TUI, so its tests run rather than skip. +textual>=8.2.8 diff --git a/requirements-ci.txt b/requirements-ci.txt index fe3f0bb..90db772 100644 --- a/requirements-ci.txt +++ b/requirements-ci.txt @@ -129,10 +129,29 @@ iniconfig==2.3.1 \ --hash=sha256:67f4b9c50da0dedf52af349e7749a80a9057a5031199791b906c3bb3ae878960 \ --hash=sha256:9121e2c1fdb355232495be3194c8dfe87ccc2d5dee45947b78e68f499790d7a7 # via pytest +linkify-it-py==2.2.0 \ + --hash=sha256:3adc40eb5af300b2605fcfdb968c24e1d780a90f1f2221af7c15e5111e94d443 \ + --hash=sha256:907acd2d17ac1fbb9ddb62c8957ccbd6158cac602231a15c3b0cd1e215f03cee + # via markdown-it-py lizard==1.25.1 \ --hash=sha256:3513ce263d17ed580f8efa202658b02c684d5da2b39ef40c7663971be822a9d6 \ --hash=sha256:789e7103875cb15aac273e1701cd72d4b9d5fa521790e258fe071dbf79dbc4ee # via -r requirements-ci.in +markdown-it-py==4.3.0 \ + --hash=sha256:7aa1511a46c39138de78d9eba92ce47e63124d8003fc98b3a351909e04239efa \ + --hash=sha256:91a58fa5b72c71a40cf7177250fd767480d7a2bb3ac426ccfb76fb90305fdaaa + # via + # mdit-py-plugins + # rich + # textual +mdit-py-plugins==0.7.0 \ + --hash=sha256:26d2179227aaf4d1dc7acffe7a738cbd15cade6b5a61cff0a2cbf9fed93efede \ + --hash=sha256:7f8e30403f3dd8bfe3afcdcbc6e61066d3f2e714167507b9c26eb385ec399353 + # via textual +mdurl==0.1.2 \ + --hash=sha256:84008a41e51615a49fc9966191ff91509e3c40b939176e643fd50a5c2196b8f8 \ + --hash=sha256:bb413d29f5eea38f31dd4754dd7377d4465116fb207585f97bf925588687c1ba + # via markdown-it-py packaging==26.3 \ --hash=sha256:94edc256424af38762eb31306eed28beb9f0efc50a8837492c9d6fd6004aed79 \ --hash=sha256:d7193f7c8e4e93f444fde0262bf90af30e16fa0ad0ad44cb553c87339b23cd1c @@ -141,6 +160,10 @@ pathspec==1.1.1 \ --hash=sha256:17db5ecd524104a120e173814c90367a96a98d07c45b2e10c2f3919fff91bf5a \ --hash=sha256:a00ce642f577bf7f473932318056212bc4f8bfdf53128c78bbd5af0b9b20b189 # via lizard +platformdirs==4.14.5 \ + --hash=sha256:32e9921db7385aa8169bee4fe9501923e5715ba9fb7ea3b9c69b0d8e0639f7a1 \ + --hash=sha256:8e778c6fa421516f4d3369449f8df3b0e1b72c0e0005fe88e3ec772d0a14e750 + # via textual pluggy==1.7.0 \ --hash=sha256:7dd7b0d8832ba3cb632c306926ded123429211b83641b35dc5c41ad2d34f9bec \ --hash=sha256:d1eaa46ebb595891b860ab086b4d09c8588af65ebd4361b8e8f4bb8920b90ba8 @@ -151,10 +174,16 @@ pygments==2.21.0 \ # via # lizard # pytest + # rich + # textual pytest==9.2.0 \ --hash=sha256:07acdcf88ec174f907dc01af70bf52d49b014bbab03faeeade31c96b18c2e588 \ --hash=sha256:97c4f3b1f3c7414aa36465e09b8220f1d03627993bd51b1282e193753c80557b # via -r requirements-ci.in +rich==15.0.0 \ + --hash=sha256:33bd4ef74232fb73fe9279a257718407f169c09b78a87ad3d296f548e27de0bb \ + --hash=sha256:edd07a4824c6b40189fb7ac9bc4c52536e9780fbbfbddf6f1e2502c31b068c36 + # via textual ruff==0.17.4 \ --hash=sha256:0d1f4848fd2ec0499be4046db9b7dfcc952666885f00233cc7033515282556a8 \ --hash=sha256:13e781c3c5f663e7fba737eaaeb70dd943dd55171daf316afb25cd5268cfe0c7 \ @@ -175,6 +204,14 @@ ruff==0.17.4 \ --hash=sha256:f32d178b50e826e00f6c6c1be3f56333068acb9b67fe167143527fa9b81e890d \ --hash=sha256:f99e95a1515161138be3cbd1b06c06e739d8feb38ade35564556999fffc3686e # via -r requirements-ci.in +textual==8.2.8 \ + --hash=sha256:267375fd402dc8d981457212efa71f0e3365fd17bba144ba9bb3ed7563cb374a \ + --hash=sha256:3f106a9fbc73e39dd266c9712432087de78a6d644084c7c241d6a25c3169115b + # via -r requirements-ci.in +typing-extensions==4.16.0 \ + --hash=sha256:481caa481374e813c1b176ada14e97f1f67a4539ce9cfeb3f350d78d6370c2e8 \ + --hash=sha256:dc983d19a509c94dba722ee6abd33940f7c05a89e243c47e907eb4db6f1a43e5 + # via textual # The following packages are considered to be unsafe in a requirements file: setuptools==84.2.1 \ diff --git a/strain/cli.py b/strain/cli.py index 2b5c38e..72cc13b 100644 --- a/strain/cli.py +++ b/strain/cli.py @@ -57,7 +57,7 @@ def _add_backend_arguments(parser: argparse.ArgumentParser) -> None: choices=list(UiMode), default=UiMode.AUTO, help=( - "console output: auto (live progress and color on a terminal), live (full-screen dashboard), " + "console output: auto (live progress and color on a terminal), live (full-screen TUI; needs strain[ui]), " "log (plain lines) or json (events)" ), ) diff --git a/strain/console.py b/strain/console.py index 43f0002..54c58a4 100644 --- a/strain/console.py +++ b/strain/console.py @@ -1,13 +1,15 @@ """Human-readable console output: phases, steps, a live status line and colored mutant states.""" import enum +import importlib.util import os -from collections.abc import Mapping, Sequence +from collections.abc import Callable, Mapping, Sequence from typing import TextIO from .coverage import LineCoverage from .mutators import Mutant from .results import MutantResult, MutantState +from .safety_nets import RelaxedSafetyNet, SafetyNet RESET = "\033[0m" BOLD = "\033[1m" @@ -127,7 +129,11 @@ def warn(self, message: str) -> None: self.finish() print(f"strain: {paint('warning:', BOLD + YELLOW, self.err_color)} {message}", file=self.err, flush=True) - # Progress hooks for the live dashboard; plain console output shows the same progress as lines. + def run(self, work: Callable[[], int]) -> int: + """Runs the whole of strain's `work`; the live TUI overrides this to run it beside the TUI.""" + return work() + + # Progress hooks for the live TUI; plain console output shows the same progress as lines. def test_started(self, name: str, number: int, total: int) -> None: pass @@ -146,8 +152,24 @@ def mutant_evaluated(self, result: MutantResult, source: str, line: LineCoverage def build_started(self, command: str) -> None: pass + def safety_nets_checked(self, relaxed: Sequence[RelaxedSafetyNet], holding: Sequence[SafetyNet]) -> None: + """The safety nets that fail but did not stop the run, and those that hold; the lines show them already.""" + def build_output(self, line: str) -> None: pass def run_finished(self) -> None: pass + + +def create_console(out: TextIO, err: TextIO, mode: UiMode) -> Console: + """The Textual TUI for `--ui live` on a terminal; plain console output everywhere else, or without Textual.""" + if mode is not UiMode.LIVE or not is_terminal(out): + return Console(out, err, mode) + if importlib.util.find_spec("textual") is None: + console = Console(out, err, UiMode.AUTO) + console.warn("--ui live needs Textual: pip install 'strain[ui]'; showing plain output instead") + return console + from .tui import TuiConsole + + return TuiConsole(out, err) diff --git a/strain/dashboard.py b/strain/dashboard.py deleted file mode 100644 index ea2b54e..0000000 --- a/strain/dashboard.py +++ /dev/null @@ -1,634 +0,0 @@ -"""The `--ui live` dashboard: the mutated code, the mutant queue, live findings, the build log and a survivor browser.""" - -import collections -import os -import re -import shutil -import threading -import time -from collections.abc import Callable, Sequence -from dataclasses import dataclass, field -from typing import TextIO - -from .config import CONFIG_FILE -from .console import ( - BOLD, - CYAN, - DIM, - GREEN, - RED, - RESET, - REVERSE, - STATE_STYLES, - STATE_WIDTH, - YELLOW, - Console, - UiMode, - is_terminal, - paint, - plural, - score_style, -) -from .coverage import LineCoverage -from .keys import KeyHandler, KeyReader, stdin_key_reader -from .mutators import Mutant, apply_mutant, describe_mutant -from .report import FileScore, action_items, file_scores, format_score, hint -from .results import CombinedResult, MutantResult, MutantState, combine, mutation_score - -ALTERNATE_SCREEN = "\033[?1049h\033[?25l" -MAIN_SCREEN = "\033[?25h\033[?1049l" -HOME = "\033[H" -CLEAR_TO_END_OF_SCREEN = "\033[J" -ESCAPE_SEQUENCE = re.compile(r"(\033\[[0-9;?]*[A-Za-z])") -ELLIPSIS = "…" -TAB_WIDTH = 4 -SNIPPET_CONTEXT = 3 -SNIPPET_SEPARATOR = "│ " -CHANGE_STYLES = {"-": RED, "+": GREEN} -PROGRESS_BAR_WIDTH = 20 -# The per-file scores take at most this share of the survivors view. -FILES_PANEL_SHARE = 4 -BUILD_LOG_LINES = 200 -# Builds can print thousands of lines; redrawing for each one would cost more than the build. -BUILD_OUTPUT_REFRESH_SECONDS = 0.05 -# The queue needs this many rows before it can spare two for the "earlier" and "more" markers. -QUEUE_MARKER_ROWS = 3 -RUNNING = "running" -PENDING = "pending" -RUN_VIEW = "run" -SURVIVORS_VIEW = "survivors" -PAGE_ROWS = 10 -DETAIL_LABEL_WIDTH = 10 -# How often the finished run checks whether the user pressed q. -QUIT_POLL_SECONDS = 0.1 -EQUIVALENT_HINT = f"equivalent? list it under [ignore] mutants in {CONFIG_FILE}" - -TerminalSize = Callable[[], os.terminal_size] -KeyReaderFactory = Callable[[KeyHandler], KeyReader | None] - - -def visible_length(text: str) -> int: - return len(ESCAPE_SEQUENCE.sub("", text)) - - -def fit(text: str, width: int) -> str: - """`text` padded or truncated to exactly `width` visible characters, keeping its escape sequences.""" - if width <= 0: - return "" - excess = visible_length(text) - width - if excess <= 0: - return text + " " * -excess - remaining = width - len(ELLIPSIS) - kept: list[str] = [] - for index, piece in enumerate(ESCAPE_SEQUENCE.split(text)): - is_escape = index % 2 == 1 - kept.append(piece if is_escape else piece[:remaining]) - remaining -= 0 if is_escape else len(kept[-1]) - reset = RESET if ESCAPE_SEQUENCE.search(text) else "" - return f"{''.join(kept)}{ELLIPSIS}{reset}" - - -def fit_keeping_end(text: str, end: str, width: int) -> str: - """`text` then `end` in exactly `width` visible characters, truncating `text` before `end` loses anything.""" - if not end: - return fit(text, width) - end_width = visible_length(end) - if end_width >= width: - return fit(end, width) - return fit(text, width - end_width) + end - - -def code_snippet(source: str, mutant: Mutant, context: int = SNIPPET_CONTEXT) -> list[str]: - """The lines around `mutant`, with its line shown before (`-`) and after (`+`) the mutation.""" - lines = source.splitlines() - mutated = apply_mutant(source, mutant).splitlines()[mutant.line - 1] - first = max(1, mutant.line - context) - last = min(len(lines), mutant.line + context) - width = len(str(last)) - - def numbered(marker: str, number: int, text: str) -> str: - return f"{marker} {number:>{width}} │ {text.expandtabs(TAB_WIDTH)}" - - snippet: list[str] = [] - for number in range(first, last + 1): - if number == mutant.line: - snippet += [numbered("-", number, lines[number - 1]), numbered("+", number, mutated)] - else: - snippet.append(numbered(" ", number, lines[number - 1])) - return snippet - - -def token_offset(source: str, mutant: Mutant) -> int: - """Where the mutated token starts in the tab-expanded text of its line.""" - text = source.splitlines()[mutant.line - 1] - return len(text[: mutant.column - 1].expandtabs(TAB_WIDTH)) - - -def progress_bar(done: int, total: int, width: int = PROGRESS_BAR_WIDTH) -> str: - filled = 0 if total <= 0 else min(width, done * width // total) - return "█" * filled + "░" * (width - filled) - - -def window_start(focus: int, count: int, rows: int) -> int: - """The first of `rows` consecutive items out of `count` that keeps `focus` in the middle.""" - return max(0, min(focus - rows // 2, count - rows)) - - -def execution_summary(line: int, executions: dict[str, LineCoverage]) -> str: - """How often each test ran `line`, and how many of its branches they took.""" - if not executions: - return f"no coverage data for line {line}" - total = sum(coverage.count for coverage in executions.values()) - if len(executions) == 1: - summary = f"line {line} executed {plural(total, 'time')} by {next(iter(executions))}" - else: - per_test = ", ".join(f"{test} {coverage.count}" for test, coverage in executions.items()) - summary = f"line {line} executed {plural(total, 'time')}: {per_test}" - branches = max(coverage.branches for coverage in executions.values()) - if branches == 0: - return summary - taken = max(coverage.branches_taken for coverage in executions.values()) - return f"{summary}; {taken}/{branches} branches taken" - - -@dataclass(slots=True) -class QueueEntry: - description: str - state: str = PENDING - - -@dataclass(slots=True) -class MutantContext: - """What the survivor browser shows beyond the result: the source and each test's coverage of the line.""" - - source: str - executions: dict[str, LineCoverage] = field(default_factory=dict) - - -@dataclass(frozen=True, slots=True) -class Layout: - snippet: int - queue: int - findings: int - build_log: int - - -def layout(body_rows: int, snippet_rows: int) -> Layout: - """Rows for each panel, title included: the snippet first, then the rest split over queue, findings and build log.""" - snippet = min(body_rows, snippet_rows + 1) - rest = body_rows - snippet - queue = (rest + 2) // 3 - findings = (rest - queue + 1) // 2 - return Layout(snippet, queue, findings, rest - queue - findings) - - -@dataclass(frozen=True, slots=True) -class SurvivorLayout: - survivor_list: int - details: int - files: int - - -def survivor_layout(body_rows: int, survivors: int, files: int) -> SurvivorLayout: - """Rows for the survivor list, the details of the selected survivor and the per-file scores, titles included.""" - file_rows = 0 if files == 0 else min(files + 1, body_rows // FILES_PANEL_SHARE) - rest = body_rows - file_rows - survivor_list = min(rest, max(1, survivors) + 1, max(2, rest // 3)) - return SurvivorLayout(survivor_list, rest - survivor_list, file_rows) - - -class Dashboard: - def __init__(self, color: bool, interactive: bool = False) -> None: - self.color = color - self.interactive = interactive - self.view = RUN_VIEW - self.run_done = False - self.selected = 0 - self.results: list[MutantResult] = [] - self.contexts: dict[str, MutantContext] = {} - self._combined: list[CombinedResult] | None = None - self.test = "" - self.test_name = "" - self.queue: list[QueueEntry] = [] - self.current: int | None = None - self.mutant_title = "waiting for the first mutant" - self.mutant: Mutant | None = None - self.mutant_offset = 0 - self.snippet: list[str] = [] - self.build_command = "" - self.build_log: collections.deque[str] = collections.deque(maxlen=BUILD_LOG_LINES) - self.status = "" - - def paint(self, text: str, style: str) -> str: - return paint(text, style, self.color) - - def set_test(self, name: str, number: int, total: int) -> None: - """Starts a test with an empty queue, so the title never pairs it with the previous test's mutants.""" - self.test = f"Test {number}/{total}: {name}" - self.test_name = name - self.set_plan([]) - - def set_plan(self, descriptions: Sequence[str]) -> None: - self.queue = [QueueEntry(description) for description in descriptions] - self.current = None - - def start_mutant(self, index: int, source: str, mutant: Mutant) -> None: - self.current = index - self.queue[index].state = RUNNING - self.mutant_title = describe_mutant(mutant) - self.mutant = mutant - self.mutant_offset = token_offset(source, mutant) - self.snippet = code_snippet(source, mutant) - - def finish_mutant(self, index: int, state: str) -> None: - self.queue[index].state = state - - def record_result(self, result: MutantResult, source: str, line: LineCoverage | None) -> None: - self.results.append(result) - context = self.contexts.setdefault(result.mutant.id, MutantContext(source)) - if line is not None: - context.executions[result.test] = line - self._combined = None - - def combined(self) -> list[CombinedResult]: - if self._combined is None: - self._combined = combine(self.results) - return self._combined - - def survivors(self) -> list[CombinedResult]: - return [entry for entry in self.combined() if entry.state is MutantState.SURVIVED] - - def selected_survivor(self, survivors: Sequence[CombinedResult]) -> int: - return max(0, min(self.selected, len(survivors) - 1)) - - def press(self, key: str) -> None: - if key == "tab": - self.view = SURVIVORS_VIEW if self.view == RUN_VIEW else RUN_VIEW - return - steps = {"up": -1, "down": 1, "pageup": -PAGE_ROWS, "pagedown": PAGE_ROWS} - if self.view == SURVIVORS_VIEW and key in steps: - survivors = self.survivors() - self.selected = max(0, min(self.selected_survivor(survivors) + steps[key], len(survivors) - 1)) - - def finish_run(self) -> None: - """Ends the run with every mutant in the queue, in its state combined over all tests.""" - self.run_done = True - self.test_name = "all tests" - self.queue = [QueueEntry(describe_mutant(entry.mutant), entry.state) for entry in self.combined()] - self.current = None - if self.survivors(): - self.view = SURVIVORS_VIEW - - def start_build(self, command: str) -> None: - self.build_command = command - self.build_log.clear() - - def build_output(self, line: str) -> None: - self.build_log.append(line.rstrip().expandtabs(TAB_WIDTH)) - - def set_status(self, text: str) -> None: - self.status = text.strip() - - def key_hints(self) -> str: - if not self.interactive: - return "" - hints = ["tab survivors"] if self.view == RUN_VIEW else ["tab run", "↑↓ select"] - if self.run_done: - hints.append("q quit") - return self.paint(" ".join(hints), DIM) - - def title(self, width: int) -> str: - """The run's position on the left and the key hints on the right, so a narrow terminal never hides `q quit`.""" - parts = [self.paint("strain live", BOLD + CYAN), self.test] - if self.run_done: - parts.append("finished") - elif self.current is not None: - parts.append(f"mutant {self.current + 1}/{len(self.queue)}") - return fit_keeping_end(" · ".join(part for part in parts if part) + " ", self.key_hints(), width) - - def progress(self) -> str: - done = sum(entry.state not in (PENDING, RUNNING) for entry in self.queue) - label = " ".join(part for part in (self.test_name, f"{done}/{len(self.queue)}") if part) - return f"{self.paint(progress_bar(done, len(self.queue)), CYAN)} {label}" - - def score(self, score: float | None) -> str: - percent = None if score is None else score * 100 - return self.paint(format_score(score), score_style(percent)) - - def tally(self) -> str: - """The progress of the current test, the score, then the totals over every test, counting each mutant once. - - The score comes before the counts, so a narrow terminal cuts the counts first. - """ - states = [entry.state for entry in self.combined()] - counts = [ - self.paint(f"{state} {states.count(state)}", STATE_STYLES[state]) - for state in MutantState - if state in states - ] - return " ".join([self.progress(), f"score {self.score(mutation_score(states))}", *counts]) - - def rule(self, title: str, width: int) -> str: - filler = "─" * max(0, width - len(f"─ {title} ")) - return fit(f"{self.paint('─', DIM)} {self.paint(title, BOLD + CYAN)} {self.paint(filler, DIM)}", width) - - def snippet_line(self, line: str, mutant: Mutant | None, offset: int) -> str: - """`line` with a dim gutter and, on a changed line, the mutated token highlighted.""" - gutter, separator, text = line.partition(SNIPPET_SEPARATOR) - style = CHANGE_STYLES.get(line[:1], "") - if not style: - return self.paint(gutter + separator, DIM) + text - prefix = self.paint(gutter + separator, style) - if mutant is None: - return prefix + self.paint(text, style) - end = offset + len(mutant.original if line.startswith("-") else mutant.replacement) - return ( - prefix - + self.paint(text[:offset], style) - + self.paint(text[offset:end], BOLD + REVERSE + style) - + self.paint(text[end:], style) - ) - - def snippet_lines(self, snippet: Sequence[str], mutant: Mutant | None, offset: int) -> list[str]: - return [self.snippet_line(line, mutant, offset) for line in snippet] - - def queue_line(self, index: int, entry: QueueEntry) -> str: - marker = "▶" if index == self.current else " " - state = self.paint(f"{entry.state:<{STATE_WIDTH}}", STATE_STYLES.get(entry.state, DIM)) - return f"{marker} {state} {entry.description}" - - def queue_lines(self, rows: int) -> list[str]: - """A window over the queue that keeps the current mutant in the middle.""" - if rows <= 0: - return [] - if not self.queue: - return [self.paint(" no mutants planned yet", DIM)] - start = window_start(self.current or 0, len(self.queue), rows) - lines = [ - self.queue_line(index, self.queue[index]) for index in range(start, min(len(self.queue), start + rows)) - ] - if rows < QUEUE_MARKER_ROWS: - return lines - hidden_after = len(self.queue) - (start + rows) - if start > 0: - lines[0] = self.paint(f" ↑ {start + 1} earlier", DIM) - if hidden_after > 0: - lines[-1] = self.paint(f" ↓ {hidden_after + 1} more", DIM) - return lines - - def findings_lines(self, rows: int) -> list[str]: - if rows <= 0: - return [] - items = action_items(self.combined(), limit=None) - if not items: - return [self.paint(" no findings yet", DIM)] - lines = [f"{self.paint(' •', YELLOW)} {item}" for item in items] - if len(lines) > rows: - return [*lines[: rows - 1], self.paint(f" … {len(lines) - rows + 1} more; tab shows the survivors", DIM)] - return lines - - def survivor_list_lines(self, rows: int) -> list[str]: - survivors = self.survivors() - if rows <= 0: - return [] - if not survivors: - return [self.paint(" no surviving mutants", DIM)] - selected = self.selected_survivor(survivors) - start = window_start(selected, len(survivors), rows) - return [ - self.survivor_list_line(survivors[index], index == selected) - for index in range(start, min(len(survivors), start + rows)) - ] - - def survivor_list_line(self, entry: CombinedResult, selected: bool) -> str: - kind = "" if entry.survivor is None else f" {self.paint(str(entry.survivor), DIM)}" - description = describe_mutant(entry.mutant) - if selected: - return f"{self.paint('▶', BOLD + CYAN)} {self.paint(description, BOLD)}{kind}" - return f" {description}{kind}" - - def file_line(self, row: FileScore, width: int) -> str: - percent = None if row.score is None else row.score * 100 - survived_style = STATE_STYLES[MutantState.SURVIVED] if row.survived else DIM - return ( - f" {row.file:<{width}} {self.paint(f'{format_score(row.score):>6}', score_style(percent))}" - f" {self.paint(progress_bar(row.detected, row.detected + row.survived), CYAN)}" - f" {plural(row.mutants, 'mutant')} {self.paint(f'{row.survived} survived', survived_style)}" - ) - - def file_lines(self, rows: int) -> list[str]: - scores = file_scores(self.combined()) - if rows <= 0 or not scores: - return [] - width = max(len(row.file) for row in scores) - return [self.file_line(row, width) for row in scores[:rows]] - - def detail(self, label: str, value: str) -> str: - return f"{self.paint(f'{label:<{DETAIL_LABEL_WIDTH}}', DIM)} {value}" - - def survivor_facts(self, entry: CombinedResult) -> list[str]: - mutant = entry.mutant - context = self.contexts[mutant.id] - kind = "" if entry.survivor is None else f"{entry.survivor} " - return [ - self.detail("id", f"{mutant.id} {self.paint(EQUIVALENT_HINT, DIM)}"), - self.detail("function", entry.function or "unknown"), - self.detail("coverage", execution_summary(mutant.line, context.executions)), - self.detail(str(MutantState.SURVIVED), f"{kind}under {', '.join(entry.covered_by)}"), - self.detail("hint", hint(entry)), - "", - ] - - def survivor_details(self, entry: CombinedResult, rows: int) -> list[str]: - """The facts about `entry`, then as much of its source as fits in `rows`.""" - mutant = entry.mutant - source = self.contexts[mutant.id].source - facts = self.survivor_facts(entry) - context = max(SNIPPET_CONTEXT, (rows - len(facts) - len(CHANGE_STYLES)) // 2) - snippet = self.snippet_lines(code_snippet(source, mutant, context), mutant, token_offset(source, mutant)) - return [*facts, *snippet][:rows] - - def survivor_detail_lines(self, rows: int) -> list[str]: - survivors = self.survivors() - if rows <= 0 or not survivors: - return [] - return self.survivor_details(survivors[self.selected_survivor(survivors)], rows) - - def build_log_lines(self, rows: int) -> list[str]: - if rows <= 0: - return [] - return list(self.build_log)[-rows:] - - def panel(self, title: str, rows: int, content: Callable[[int], list[str]], width: int) -> list[str]: - if rows <= 0: - return [] - body = content(rows - 1) - return [self.rule(title, width), *body, *[""] * (rows - 1 - len(body))] - - def run_panels(self, body_rows: int, width: int) -> list[str]: - rows = layout(body_rows, len(self.snippet)) - build_title = f"Build log: {self.build_command}" if self.build_command else "Build log" - return [ - *self.panel( - f"Mutant: {self.mutant_title}", - rows.snippet, - lambda count: self.snippet_lines(self.snippet, self.mutant, self.mutant_offset)[:count], - width, - ), - *self.panel("Mutants", rows.queue, self.queue_lines, width), - *self.panel("Findings", rows.findings, self.findings_lines, width), - *self.panel(build_title, rows.build_log, self.build_log_lines, width), - ] - - def survivor_panels(self, body_rows: int, width: int) -> list[str]: - survivors = self.survivors() - rows = survivor_layout(body_rows, len(survivors), len(file_scores(self.combined()))) - detail_title = "Survivor" - if survivors: - detail_title = f"Survivor: {describe_mutant(survivors[self.selected_survivor(survivors)].mutant)}" - return [ - *self.panel(f"Survivors ({len(survivors)})", rows.survivor_list, self.survivor_list_lines, width), - *self.panel(detail_title, rows.details, self.survivor_detail_lines, width), - *self.panel("Files", rows.files, self.file_lines, width), - ] - - def render(self, width: int, height: int) -> list[str]: - header = [self.title(width), self.tally()] - footer = [self.paint(self.status, DIM)] - body_rows = max(0, height - len(header) - len(footer)) - panels = self.run_panels if self.view == RUN_VIEW else self.survivor_panels - lines = [*header, *panels(body_rows, width), *footer][:height] - return [fit(line, width) for line in lines] - - -class LiveConsole(Console): - """Draws the dashboard on the alternate screen, then prints the plain transcript once the run is over.""" - - def __init__( - self, - out: TextIO, - err: TextIO, - terminal_size: TerminalSize = shutil.get_terminal_size, - key_reader: KeyReaderFactory = stdin_key_reader, - ) -> None: - super().__init__(out, err, UiMode.LIVE) - # Keys arrive on their own thread; the lock keeps them from redrawing in the middle of an update. - self.lock = threading.RLock() - self.quit = threading.Event() - self.dashboard = Dashboard(self.color) - self.terminal_size = terminal_size - self.transcript: list[str] = [] - self.warnings: list[str] = [] - self.active = True - self.last_refresh = 0.0 - out.write(ALTERNATE_SCREEN) - self.refresh() - self.keys = key_reader(self.key_pressed) - self.dashboard.interactive = self.keys is not None - - def refresh(self) -> None: - with self.lock: - size = self.terminal_size() - # One column short of the edge, so a full line never wraps and scrolls the screen. - frame = "\n".join(self.dashboard.render(max(0, size.columns - 1), size.lines)) - self.out.write(f"{HOME}{frame}{CLEAR_TO_END_OF_SCREEN}") - self.out.flush() - self.last_refresh = time.monotonic() - - def key_pressed(self, key: str) -> None: - with self.lock: - if not self.active: - return - if key == "q" and self.dashboard.run_done: - self.quit.set() - return - self.dashboard.press(key) - self.refresh() - - def line(self, text: str = "") -> None: - if not self.active: - super().line(text) - return - self.transcript.append(text) - if text: - self.status(text) - - def status(self, text: str) -> None: - with self.lock: - if self.active: - self.dashboard.set_status(text) - self.refresh() - - def warn(self, message: str) -> None: - if not self.active: - super().warn(message) - return - self.warnings.append(message) - self.status(f"warning: {message}") - - def finish(self) -> None: - # Outside the lock: the key thread may be waiting for it, and closing waits for that thread. - if self.keys is not None: - self.keys.close() - with self.lock: - if self.active: - self.active = False - self.out.write(MAIN_SCREEN) - for text in self.transcript: - super().line(text) - for message in self.warnings: - super().warn(message) - super().finish() - - def run_finished(self) -> None: - """Leaves the results on screen to browse until the user presses q.""" - with self.lock: - self.dashboard.finish_run() - self.dashboard.set_status("finished; browse the survivors, then press q for the full transcript") - self.refresh() - if self.keys is not None: - while not self.quit.wait(QUIT_POLL_SECONDS): - pass - - def test_started(self, name: str, number: int, total: int) -> None: - with self.lock: - self.dashboard.set_test(name, number, total) - self.refresh() - - def plan(self, mutants: Sequence[Mutant]) -> None: - with self.lock: - self.dashboard.set_plan([describe_mutant(mutant) for mutant in mutants]) - self.refresh() - - def mutant_started(self, index: int, source: str, mutant: Mutant) -> None: - with self.lock: - self.dashboard.start_mutant(index, source, mutant) - self.refresh() - - def mutant_finished(self, index: int, state: str) -> None: - with self.lock: - self.dashboard.finish_mutant(index, state) - self.refresh() - - def mutant_evaluated(self, result: MutantResult, source: str, line: LineCoverage | None) -> None: - with self.lock: - self.dashboard.record_result(result, source, line) - self.refresh() - - def build_started(self, command: str) -> None: - with self.lock: - self.dashboard.start_build(command) - self.refresh() - - def build_output(self, line: str) -> None: - with self.lock: - self.dashboard.build_output(line) - if time.monotonic() - self.last_refresh >= BUILD_OUTPUT_REFRESH_SECONDS: - self.refresh() - - -def create_console(out: TextIO, err: TextIO, mode: UiMode) -> Console: - """The dashboard for `--ui live` on a terminal; plain console output everywhere else.""" - if mode is UiMode.LIVE and is_terminal(out): - return LiveConsole(out, err) - return Console(out, err, mode) diff --git a/strain/keys.py b/strain/keys.py deleted file mode 100644 index 8e12e79..0000000 --- a/strain/keys.py +++ /dev/null @@ -1,115 +0,0 @@ -"""Key presses on the controlling terminal, read on a background thread so the dashboard reacts during builds.""" - -import os -import re -import sys -import threading -import time -from collections.abc import Callable, Mapping -from typing import TextIO - -from .console import is_terminal - -KeyHandler = Callable[[str], None] - -# How long one read waits for a key, which bounds how long closing the reader takes. -POLL_SECONDS = 0.1 -READ_SIZE = 64 -ESCAPE_SEQUENCES: Mapping[str, str] = { - "\033[A": "up", - "\033OA": "up", - "\033[B": "down", - "\033OB": "down", - "\033[5~": "pageup", - "\033[6~": "pagedown", -} -ALIASES: Mapping[str, str] = {"j": "down", "k": "up", "\t": "tab"} -# A CSI sequence (parameters, then a final byte), an SS3 sequence, or a single character. -KEY_TOKEN = re.compile(r"\033\[[0-9;?]*[@-~]|\033O.|.", re.DOTALL) -# Windows reports special keys as one of these prefixes followed by a scan code. -WINDOWS_PREFIXES = frozenset({"\x00", "\xe0"}) -WINDOWS_SCAN_CODES: Mapping[str, str] = {"H": "\033[A", "P": "\033[B", "I": "\033[5~", "Q": "\033[6~"} - - -def decode_key(token: str) -> str | None: - if token.startswith("\033"): - return ESCAPE_SEQUENCES.get(token) - return ALIASES.get(token, token) - - -def decode_keys(text: str) -> list[str]: - keys = (decode_key(match.group()) for match in KEY_TOKEN.finditer(text)) - return [key for key in keys if key is not None] - - -class KeyReader: - """Calls `on_key` from a background thread for every key read, until closed.""" - - def __init__(self, read: Callable[[], str], restore: Callable[[], None], on_key: KeyHandler) -> None: - self._read = read - self._restore = restore - self._on_key = on_key - self._stopped = threading.Event() - self._thread = threading.Thread(target=self._loop, name="strain-keys", daemon=True) - self._thread.start() - - def _loop(self) -> None: - while not self._stopped.is_set(): - for key in decode_keys(self._read()): - self._on_key(key) - - def close(self) -> None: - if not self._stopped.is_set(): - self._stopped.set() - self._thread.join() - self._restore() - - -def _windows_reader(on_key: KeyHandler) -> KeyReader | None: - if sys.platform != "win32": - return None - import msvcrt - - def read() -> str: - deadline = time.monotonic() + POLL_SECONDS - while not msvcrt.kbhit(): - if time.monotonic() >= deadline: - return "" - time.sleep(POLL_SECONDS / 10) - key = msvcrt.getwch() - return WINDOWS_SCAN_CODES.get(msvcrt.getwch(), "") if key in WINDOWS_PREFIXES else key - - return KeyReader(read, lambda: None, on_key) - - -def _posix_reader(stream: TextIO, on_key: KeyHandler) -> KeyReader | None: - if sys.platform == "win32": - return None - import select - import termios - import tty - - descriptor = stream.fileno() - try: - saved = termios.tcgetattr(descriptor) - # cbreak keeps signals, so Ctrl-C still interrupts the run and restores the sources. - tty.setcbreak(descriptor) - except termios.error: - return None - - def read() -> str: - ready, _, _ = select.select([descriptor], [], [], POLL_SECONDS) - return os.read(descriptor, READ_SIZE).decode(errors="replace") if ready else "" - - return KeyReader(read, lambda: termios.tcsetattr(descriptor, termios.TCSADRAIN, saved), on_key) - - -def open_key_reader(stream: TextIO, on_key: KeyHandler) -> KeyReader | None: - """A reader of the keys typed on `stream`, or None when `stream` is not an interactive terminal.""" - if not is_terminal(stream): - return None - return _windows_reader(on_key) if sys.platform == "win32" else _posix_reader(stream, on_key) - - -def stdin_key_reader(on_key: KeyHandler) -> KeyReader | None: - return open_key_reader(sys.stdin, on_key) diff --git a/strain/live_state.py b/strain/live_state.py new file mode 100644 index 0000000..f070a92 --- /dev/null +++ b/strain/live_state.py @@ -0,0 +1,345 @@ +"""What the live TUI shows: the state of the run, updated by the runner's console hooks from the runner's thread. + +The TUI reads it under `lock` on its own thread, and redraws whatever `version` says has changed. +""" + +import threading +import time +from collections import deque +from collections.abc import Callable, Sequence +from dataclasses import dataclass, field +from typing import TypeVar + +from .coverage import LineCoverage +from .mutators import Mutant +from .report import FileScore, action_items, file_scores +from .results import DETECTED_STATES, CombinedResult, MutantResult, MutantState, combine, mutation_score +from .safety_nets import RelaxedSafetyNet, SafetyNet + +RUNNING = "running" +PENDING = "pending" +SKIPPED = "skipped" +ELLIPSIS = "…" +TAB_WIDTH = 4 +BUILD_LOG_LINES = 2000 +SLOWEST_MUTANTS = 10 + +Clock = Callable[[], float] +T = TypeVar("T") + + +def plural(count: int, noun: str) -> str: + return f"{count} {noun}{'' if count == 1 else 's'}" + + +def token_offset(source: str, mutant: Mutant) -> int: + """Where the mutated token starts in the tab-expanded text of its line.""" + text = source.splitlines()[mutant.line - 1] + return len(text[: mutant.column - 1].expandtabs(TAB_WIDTH)) + + +def execution_summary(line: int, executions: dict[str, LineCoverage]) -> str: + """How often each test ran `line`, and how many of its branches they took.""" + if not executions: + return f"no coverage data for line {line}" + total = sum(coverage.count for coverage in executions.values()) + if len(executions) == 1: + summary = f"line {line} executed {plural(total, 'time')} by {next(iter(executions))}" + else: + per_test = ", ".join(f"{test} {coverage.count}" for test, coverage in executions.items()) + summary = f"line {line} executed {plural(total, 'time')}: {per_test}" + branches = max(coverage.branches for coverage in executions.values()) + if branches == 0: + return summary + taken = max(coverage.branches_taken for coverage in executions.values()) + return f"{summary}; {taken}/{branches} branches taken" + + +def status_phase(status: str) -> str: + """What the run is doing, from a status line such as `[3/6] src/a.cpp:7:13 [ror] `<` → `>=` building…`.""" + if not status.endswith(ELLIPSIS): + return "" + return status.rpartition(" ")[2].removesuffix(ELLIPSIS) + + +def clock_time(seconds: float) -> str: + minutes, seconds = divmod(int(seconds), 60) + hours, minutes = divmod(minutes, 60) + return f"{hours}:{minutes:02}:{seconds:02}" if hours else f"{minutes}:{seconds:02}" + + +@dataclass(slots=True) +class QueueEntry: + mutant: Mutant + state: str = PENDING + duration: float | None = None + + +@dataclass(slots=True) +class MutantContext: + """What the browsers show beyond the result: the source and each test's coverage of the line.""" + + source: str + executions: dict[str, LineCoverage] = field(default_factory=dict) + + +@dataclass(frozen=True, slots=True) +class Tally: + """How many mutants a group has and how they fared, for the statistics tables.""" + + name: str + mutants: int + detected: int + survived: int + + @property + def score(self) -> float | None: + scored = self.detected + self.survived + return None if scored == 0 else self.detected / scored + + +def tallies(groups: dict[str, list[MutantState]]) -> list[Tally]: + return [ + Tally(name, len(states), sum(state in DETECTED_STATES for state in states), states.count(MutantState.SURVIVED)) + for name, states in sorted(groups.items()) + ] + + +class LiveState: + """The run as the TUI shows it. Every change bumps `version`; `plan_version` and `build_version` mark resets.""" + + def __init__(self, clock: Clock = time.monotonic) -> None: + self.lock = threading.RLock() + self.clock = clock + self.started = clock() + self.finished_at: float | None = None + self.version = 0 + self.results_version = 0 + self.plan_version = 0 + self.build_version = 0 + self.status = "" + self.paused = False + self.waiting = False + self.stopping = False + self.done = False + # None until the safety nets are checked; then the failing nets the run went on without, and the rest. + self.relaxed_nets: list[RelaxedSafetyNet] | None = None + self.holding_nets: list[SafetyNet] = [] + self._init_progress() + + def _init_progress(self) -> None: + self.test = "" + self.test_name = "" + self.queue: list[QueueEntry] = [] + self.current: int | None = None + self.mutant_started_at: float | None = None + self.results: list[MutantResult] = [] + self.contexts: dict[str, MutantContext] = {} + self.sources: dict[str, str] = {} + self.durations: dict[str, float] = {} + self.score_history: list[float] = [] + self.build_command = "" + self.build_log: deque[str] = deque(maxlen=BUILD_LOG_LINES) + # How many build lines arrived in all, so a reader can tell which ones it has not shown yet. + self.build_lines_seen = 0 + # What the Log view shows: the console lines and the warnings, in the order they came. + self.log: list[str] = [] + self._combined: list[CombinedResult] | None = None + self._derived: dict[str, object] = {} + + def _changed(self) -> None: + self.version += 1 + + # Updates from the run. + + def set_test(self, name: str, number: int, total: int) -> None: + """Starts a test with an empty queue, so the title never pairs it with the previous test's mutants.""" + with self.lock: + self.test = f"Test {number}/{total}: {name}" + self.test_name = name + self._set_queue([]) + + def plan(self, mutants: Sequence[Mutant]) -> None: + with self.lock: + self._set_queue([QueueEntry(mutant) for mutant in mutants]) + + def _set_queue(self, queue: list[QueueEntry]) -> None: + self.queue = queue + self.current = None + self.plan_version += 1 + self._changed() + + def start_mutant(self, index: int, source: str, mutant: Mutant) -> None: + with self.lock: + self.current = index + self.queue[index].state = RUNNING + self.sources[mutant.file] = source + self.mutant_started_at = self.clock() + self._changed() + + def finish_mutant(self, index: int, state: str) -> None: + with self.lock: + entry = self.queue[index] + entry.state = state + if self.mutant_started_at is not None and index == self.current: + entry.duration = self.clock() - self.mutant_started_at + self.durations[entry.mutant.id] = self.durations.get(entry.mutant.id, 0.0) + entry.duration + self.mutant_started_at = None + self._changed() + + def record_result(self, result: MutantResult, source: str, line: LineCoverage | None) -> None: + with self.lock: + self.results.append(result) + self.sources.setdefault(result.mutant.file, source) + context = self.contexts.setdefault(result.mutant.id, MutantContext(source)) + if line is not None: + context.executions[result.test] = line + self._combined = None + self._derived.clear() + score = mutation_score(entry.state for entry in self.combined()) + if score is not None: + self.score_history.append(score) + self.results_version += 1 + self._changed() + + def finish_run(self) -> None: + """Ends the run with every mutant in the queue, in its state combined over all tests.""" + with self.lock: + self.done = True + self.paused = False + self.finished_at = self.clock() + self.test_name = "all tests" + self._set_queue( + [ + QueueEntry(entry.mutant, entry.state, self.durations.get(entry.mutant.id)) + for entry in self.combined() + ] + ) + + def start_build(self, command: str) -> None: + with self.lock: + self.build_command = command + self.build_log.clear() + self.build_version += 1 + self._changed() + + def build_output(self, line: str) -> None: + with self.lock: + self.build_log.append(line.rstrip().expandtabs(TAB_WIDTH)) + self.build_lines_seen += 1 + self._changed() + + def add_line(self, text: str) -> None: + with self.lock: + self.log.append(text) + if text.strip(): + self.status = text.strip() + self._changed() + + def set_status(self, text: str) -> None: + with self.lock: + self.status = text.strip() + self._changed() + + def set_safety_nets(self, relaxed: Sequence[RelaxedSafetyNet], holding: Sequence[SafetyNet]) -> None: + with self.lock: + self.relaxed_nets = list(relaxed) + self.holding_nets = list(holding) + self._changed() + + def bug_hiding_nets(self) -> list[RelaxedSafetyNet]: + """The relaxed nets without which a mutant that breaks the code can survive.""" + return [failure for failure in self.relaxed_nets or [] if failure.hides_bugs] + + def set_flag(self, name: str, value: bool) -> None: + with self.lock: + setattr(self, name, value) + self._changed() + + # What the results add up to; callers hold `lock`. + + def combined(self) -> list[CombinedResult]: + if self._combined is None: + self._combined = combine(self.results) + return self._combined + + def _remember(self, name: str, compute: Callable[[], T]) -> T: + """`compute()`, kept until the next result arrives; the TUI redraws far more often than results come.""" + if name not in self._derived: + self._derived[name] = compute() + return self._derived[name] # type: ignore[return-value] + + def survivors(self) -> list[CombinedResult]: + return self._remember( + "survivors", lambda: [entry for entry in self.combined() if entry.state is MutantState.SURVIVED] + ) + + def findings(self) -> list[str]: + return self._remember("findings", lambda: action_items(self.combined(), limit=None)) + + def file_scores(self) -> list[FileScore]: + return self._remember("files", lambda: file_scores(self.combined())) + + def operator_tallies(self) -> list[Tally]: + def compute() -> list[Tally]: + groups: dict[str, list[MutantState]] = {} + for entry in self.combined(): + groups.setdefault(entry.mutant.operator, []).append(entry.state) + return tallies(groups) + + return self._remember("operators", compute) + + def test_tallies(self) -> list[Tally]: + """How each test fared against the mutants it ran on; a mutant counts once for every test that ran it.""" + + def compute() -> list[Tally]: + groups: dict[str, list[MutantState]] = {} + for result in self.results: + groups.setdefault(result.test, []).append(result.state) + return tallies(groups) + + return self._remember("tests", compute) + + def slowest(self) -> list[tuple[float, CombinedResult]]: + by_id = {entry.mutant.id: entry for entry in self.combined()} + timed = sorted( + ((duration, by_id[mutant_id]) for mutant_id, duration in self.durations.items() if mutant_id in by_id), + key=lambda pair: -pair[0], + ) + return timed[:SLOWEST_MUTANTS] + + def counts(self) -> dict[MutantState, int]: + states = [entry.state for entry in self.combined()] + return {state: states.count(state) for state in MutantState if state in states} + + def score(self) -> float | None: + return mutation_score(entry.state for entry in self.combined()) + + def elapsed(self) -> float: + return (self.finished_at if self.finished_at is not None else self.clock()) - self.started + + def done_count(self) -> int: + return sum(entry.state not in (PENDING, RUNNING) for entry in self.queue) + + def mean_duration(self) -> float | None: + timed = [entry.duration for entry in self.queue if entry.duration is not None] + return sum(timed) / len(timed) if timed else None + + def remaining_seconds(self) -> float | None: + """How long the queue of the current test will take, from the average time per mutant so far.""" + mean = self.mean_duration() + if mean is None or self.done: + return None + return mean * (len(self.queue) - self.done_count()) + + def phase(self) -> str: + return status_phase(self.status) + + def current_elapsed(self) -> float | None: + return None if self.mutant_started_at is None else self.clock() - self.mutant_started_at + + def mutant_index(self, mutant_id: str) -> int | None: + for index, entry in enumerate(self.queue): + if entry.mutant.id == mutant_id: + return index + return None diff --git a/strain/runner.py b/strain/runner.py index bc2d6a9..c4f7afd 100644 --- a/strain/runner.py +++ b/strain/runner.py @@ -27,9 +27,8 @@ ) from .build_cache import BuildFailureCache, cache_path from .config import CONFIG_FILE, Config, load_config -from .console import BOLD, DIM, RED, STATE_STYLES, UiMode, plural, score_style +from .console import BOLD, DIM, RED, STATE_STYLES, UiMode, create_console, plural, score_style from .coverage import FileCoverage, GcovCoverage -from .dashboard import create_console from .journal import Journal from .mutators import ( Engine, @@ -72,8 +71,9 @@ Level, SafetyNet, build_wide_problems, - describe_build_wide, - describe_missing, + classify_failures, + where_build_wide, + where_missing, missing_safety_nets, resolve_levels, ) @@ -328,28 +328,25 @@ def check_prerequisites(self, commands: Sequence[CompileCommand]) -> None: self.console.step("coverage instrumentation is enabled and gcov matches the compiler") def safety_net_failures(self, commands: Sequence[CompileCommand]) -> dict[SafetyNet, str]: + """Each failing safety net and where it fails.""" build_wide = build_wide_problems(read_cmake_cache(self.build_dir), self.database.commands) - failures = {net: describe_build_wide(net, evidence) for net, evidence in build_wide.items()} - failures.update((net, describe_missing(net, files)) for net, files in missing_safety_nets(commands).items()) + failures = {net: where_build_wide(evidence) for net, evidence in build_wide.items()} + failures.update((net, where_missing(files)) for net, files in missing_safety_nets(commands).items()) return failures def check_safety_nets(self, commands: Sequence[CompileCommand]) -> None: levels = resolve_levels(self.config.safety_nets, self.options.safety_nets) missing = self.safety_net_failures(commands) - errors: list[str] = [] - for net in SafetyNet: - if net not in missing: - continue - message = missing[net] - if levels[net] is Level.ERROR: - errors.append(message) - elif levels[net] is Level.WARN: - self.console.warn(message) + errors, relaxed = classify_failures(missing, levels) + for failure in relaxed: + if failure.level is Level.WARN: + self.console.warn(failure.message) else: - self.console.note(f"safety net {net} is missing but ignored by configuration") + self.console.note(f"safety net {failure.net} is missing but ignored by configuration") if errors: raise StrainError("\n".join([*errors, "rebuild with the flags from the README or pass --safety-nets=warn"])) present = [net for net in SafetyNet if net not in missing] + self.console.safety_nets_checked(relaxed, present) if present: self.console.step(f"safety nets that hold for the whole build: {', '.join(present)}") @@ -831,6 +828,6 @@ def run(self) -> int: def run(options: RunOptions, out: TextIO, err: TextIO) -> int: runner = Runner(options, out, err) try: - return runner.run() + return runner.console.run(runner.run) finally: runner.console.finish() diff --git a/strain/safety_nets.py b/strain/safety_nets.py index 21d99d5..86082ba 100644 --- a/strain/safety_nets.py +++ b/strain/safety_nets.py @@ -2,6 +2,7 @@ import enum from collections.abc import Callable, Iterable, Iterator, Mapping, Sequence +from dataclasses import dataclass from pathlib import Path from .build import CompileCommand, cmake_is_true @@ -63,12 +64,34 @@ class SafetyNet(enum.StrEnum): } +def fix(net: SafetyNet) -> str: + """How to put `net` back, the part of its explanation before the colon.""" + return EXPLANATIONS[net].partition(": ")[0] + + +def why(net: SafetyNet) -> str: + """What goes wrong without `net`, the part of its explanation after the colon.""" + return EXPLANATIONS[net].partition(": ")[2] + + +def where_missing(files: Sequence[Path]) -> str: + return f"fails in {len(files)} translation unit(s), e.g. {files[0]}" + + +def where_build_wide(evidence: str) -> str: + return f"fails for the whole build ({evidence})" + + +def describe(net: SafetyNet, where: str) -> str: + return f"safety net {net} {where}; {EXPLANATIONS[net]}" + + def describe_missing(net: SafetyNet, files: Sequence[Path]) -> str: - return f"safety net {net} fails in {len(files)} translation unit(s), e.g. {files[0]}; {EXPLANATIONS[net]}" + return describe(net, where_missing(files)) def describe_build_wide(net: SafetyNet, evidence: str) -> str: - return f"safety net {net} fails for the whole build ({evidence}); {EXPLANATIONS[net]}" + return describe(net, where_build_wide(evidence)) class Level(enum.StrEnum): @@ -91,6 +114,55 @@ class Level(enum.StrEnum): } +@dataclass(frozen=True, slots=True) +class RelaxedSafetyNet: + """A safety net that fails, but whose configured level let the run go on: what that costs and how to fix it.""" + + net: SafetyNet + level: Level + where: str + + @property + def default(self) -> Level: + return DEFAULT_LEVELS[self.net] + + @property + def hides_bugs(self) -> bool: + """Whether a mutant that breaks the code can survive without this net, which is why it errors by default.""" + return self.default is Level.ERROR + + @property + def fix(self) -> str: + return fix(self.net) + + @property + def why(self) -> str: + return why(self.net) + + @property + def message(self) -> str: + return describe(self.net, self.where) + + +def classify_failures( + failures: Mapping[SafetyNet, str], levels: Mapping[SafetyNet, Level] +) -> tuple[list[str], list[RelaxedSafetyNet]]: + """The failing nets (net to where it fails) that stop the run, as messages, and those the levels let pass. + + Nets come in declaration order, so the ones that hide bugs come before the ones that cost precision or speed. + """ + errors: list[str] = [] + relaxed: list[RelaxedSafetyNet] = [] + for net in SafetyNet: + if net not in failures: + continue + if levels[net] is Level.ERROR: + errors.append(describe(net, failures[net])) + else: + relaxed.append(RelaxedSafetyNet(net, levels[net], failures[net])) + return errors, relaxed + + def _sanitizers(arguments: Sequence[str]) -> set[str]: return { sanitizer diff --git a/strain/tui.py b/strain/tui.py new file mode 100644 index 0000000..f30ba34 --- /dev/null +++ b/strain/tui.py @@ -0,0 +1,1205 @@ +"""The `--ui live` TUI, built on Textual: the run as it happens, the survivors, every mutant, statistics and the log. + +Textual must own the main thread, so `TuiConsole.run` starts the runner on a worker thread. The runner's console +hooks update a `LiveState`, which the app redraws from several times a second. Stopping the run (Ctrl-C, or SIGTERM) +raises KeyboardInterrupt from the runner's next console hook, so the runner restores the sources and writes its +reports as it does for a plain Ctrl-C. +""" + +import functools +import signal +import threading +from collections.abc import Callable, Sequence +from types import FrameType +from typing import ClassVar, TextIO + +from rich.console import Group, RenderableType +from rich.syntax import Syntax +from rich.table import Table +from rich.text import Text +from textual import events +from textual.app import App, ComposeResult +from textual.binding import Binding, BindingType +from textual.css.query import NoMatches +from textual.containers import Horizontal, Vertical, VerticalScroll +from textual.widget import Widget +from textual.widgets import DataTable, Footer, RichLog, Static, Tab, TabbedContent, TabPane, Tabs + +from .config import CONFIG_FILE +from .console import Console, UiMode +from .coverage import LineCoverage +from .live_state import ( + PENDING, + RUNNING, + TAB_WIDTH, + LiveState, + QueueEntry, + Tally, + clock_time, + execution_summary, + plural, + token_offset, +) +from .mutators import Mutant, apply_mutant, describe_mutant +from .report import format_score, hint +from .results import CombinedResult, MutantResult, MutantState +from .safety_nets import Level, RelaxedSafetyNet, SafetyNet + +REFRESH_SECONDS = 0.1 +SPINNER = "⠋⠙⠹⠸⠼⠴⠦⠧⠇⠏" +SPINNER_FRAMES_PER_SECOND = 10 +SPARKS = "▁▂▃▄▅▆▇█" +SPARKLINE_WIDTH = 24 +SYNTAX_THEME = "one-dark" +REMOVED_LINE = "on #4b1d22" +ADDED_LINE = "on #1b4227" +DETAIL_LABEL_WIDTH = 10 +DETAIL_CONTEXT = 12 +SNIPPET_CONTEXT = 3 +# Survivors are worth a toast, but a burst of them should not bury the screen. +TOAST_SECONDS = 2.0 +HIGH_SCORE = 80.0 +LOW_SCORE = 60.0 +WORKER_POLL_SECONDS = 0.1 +EQUIVALENT_HINT = f"equivalent? list it under [ignore] mutants in {CONFIG_FILE}" +STATE_GLYPHS = { + MutantState.KILLED: "✔", + MutantState.KILLED_BY_SANITIZER: "✔", + MutantState.TIMEOUT: "✔", + MutantState.SURVIVED: "✗", + MutantState.SKIPPED_UNEVALUATED: "·", + MutantState.COMPILE_ERROR: "⊘", + MutantState.IGNORED: "○", + RUNNING: "▶", + PENDING: "·", +} +STATE_STYLES = { + MutantState.KILLED: "green", + MutantState.KILLED_BY_SANITIZER: "green", + MutantState.TIMEOUT: "green", + MutantState.SURVIVED: "bold red", + MutantState.SKIPPED_UNEVALUATED: "dim", + MutantState.COMPILE_ERROR: "dim", + MutantState.IGNORED: "dim", + RUNNING: "bold cyan", + PENDING: "dim", +} +# Narrow lists use these labels; the details give the full state. +SHORT_STATES = { + MutantState.KILLED_BY_SANITIZER: "sanitizer", + MutantState.COMPILE_ERROR: "no build", + MutantState.SKIPPED_UNEVALUATED: "skipped", +} +STATE_COLUMN_WIDTH = 2 + max(len(SHORT_STATES.get(state, state)) for state in [*MutantState, RUNNING, PENDING]) +MAP_COLORS = {"survived": "red", "detected": "green", "running": "bold cyan", "other": "grey50", "pending": "grey23"} +VIEWS = ("run", "survivors", "mutants", "stats", "log") +# The widget each view focuses, so the arrow keys work in it straight away. +VIEW_FOCUS = { + "run": "#queue", + "survivors": "#survivor-table", + "mutants": "#mutant-table", + "stats": "#stats-files", + "log": "#transcript", +} +MUTANT_COLUMNS = (("state", "state"), ("where", "location"), ("op", "op"), ("change", "change"), ("time", "time")) +TALLY_COLUMNS = (("name", "file"), ("score", "score"), ("bar", ""), ("mutants", "mutants"), ("survived", "survived")) +# Each table's columns, as (key, label), and its title. +TABLES = { + "queue": ("Mutants", MUTANT_COLUMNS), + "survivor-table": ("Survivors", (("where", "location"), ("op", "op"), ("change", "change"), ("kind", "survivor"))), + "survivor-files": ("Files", (("name", "file"), ("score", "score"), ("bar", ""), ("survived", "survived"))), + "mutant-table": ("Mutants", MUTANT_COLUMNS), + "stats-files": ("Files", TALLY_COLUMNS), + "stats-operators": ("Operators", (("name", "operator"), *TALLY_COLUMNS[1:])), + "stats-tests": ( + "Tests", + ( + ("name", "test"), + *TALLY_COLUMNS[1:3], + ("mutants", "runs"), + ("detected", "detected"), + ("survived", "survived"), + ), + ), + "stats-slowest": ( + "Slowest mutants", + (("time", "time"), ("state", "state"), ("where", "location"), ("change", "change")), + ), +} +PANEL_TITLES = { + "#safety-box": "Safety nets", + "#findings-box": "Findings", + "#build": "Build log", + "#transcript": "Log", + "#code": "Mutant", +} + + +def score_style(score: float | None) -> str: + if score is None: + return "dim" + percent = score * 100 + if percent >= HIGH_SCORE: + return "bold green" + return "bold yellow" if percent >= LOW_SCORE else "bold red" + + +def score_text(score: float | None) -> Text: + return Text(format_score(score), style=score_style(score)) + + +def state_text(state: str, glyph_only: bool = False, short: bool = False) -> Text: + glyph = STATE_GLYPHS.get(state, "·") + label = SHORT_STATES.get(state, state) if short else state + return Text(glyph if glyph_only else f"{glyph} {label}", style=STATE_STYLES.get(state, "dim")) + + +def sparkline(values: list[float], width: int = SPARKLINE_WIDTH) -> str: + """The last `width` values between 0 and 1, one block each, taller for higher values.""" + return "".join(SPARKS[min(len(SPARKS) - 1, int(value * len(SPARKS)))] for value in values[-width:]) + + +def bar(done: int, total: int, width: int = 12) -> Text: + filled = 0 if total <= 0 else min(width, round(done * width / total)) + return Text.assemble(("█" * filled, "green"), ("█" * (width - filled), "red" if total else "grey23")) + + +def location(mutant: Mutant, short: bool = False) -> str: + """Where `mutant` is; `short` keeps only the file name, for lists whose details give the whole path.""" + file = mutant.file.rpartition("/")[2] if short else mutant.file + return f"{file}:{mutant.line}:{mutant.column}" + + +def table(table_id: str, classes: str = "panel") -> DataTable: + """A row-selecting table whose cells keep their own colors on the cursor row.""" + return DataTable( + id=table_id, classes=classes, cursor_type="row", zebra_stripes=True, cursor_foreground_priority="renderable" + ) + + +def change(mutant: Mutant) -> Text: + return Text.assemble((mutant.original, "red"), " → ", (mutant.replacement, "green")) + + +def map_category(state: str) -> str: + if state == RUNNING: + return "running" + if state == PENDING: + return "pending" + if state == MutantState.SURVIVED: + return "survived" + if state in (MutantState.KILLED, MutantState.KILLED_BY_SANITIZER, MutantState.TIMEOUT): + return "detected" + return "other" + + +@functools.lru_cache(maxsize=32) +def highlighted_lines(source: str) -> list[Text]: + """`source` with C++ syntax highlighting, one Text per line, tabs expanded.""" + syntax = Syntax("", "cpp", theme=SYNTAX_THEME, background_color="default") + return syntax.highlight(source.expandtabs(TAB_WIDTH)).split("\n", allow_blank=True) + + +def diff_view(source: str, mutant: Mutant, context: int) -> Text: + """The lines around `mutant` with syntax highlighting, its line shown before (`-`) and after (`+`) the mutation.""" + lines = highlighted_lines(source) + mutated = highlighted_lines(apply_mutant(source, mutant))[mutant.line - 1] + first = max(1, mutant.line - context) + # The highlighter ends with an empty line of its own, so count the source's lines. + last = min(len(source.splitlines()), mutant.line + context) + width = len(str(last)) + offset = token_offset(source, mutant) + rows: list[Text] = [] + for number in range(first, last + 1): + if number != mutant.line: + rows.append(Text.assemble((f" {number:>{width}} │ ", "grey42"), lines[number - 1])) + continue + for marker, text, token, style in ( + ("-", lines[number - 1], mutant.original, REMOVED_LINE), + ("+", mutated, mutant.replacement, ADDED_LINE), + ): + line = text.copy() + line.stylize(style) + line.stylize("bold reverse", offset, offset + len(token)) + color = "red" if marker == "-" else "green" + rows.append(Text.assemble((f"{marker} {number:>{width}} │ ", f"bold {color} {style}"), line)) + view = Text("\n").join(rows) + view.no_wrap = True + view.overflow = "ellipsis" + return view + + +def detail_row(table: Table, label: str, value: str | Text) -> None: + table.add_row(Text(f"{label:<{DETAIL_LABEL_WIDTH}}", style="dim"), value) + + +def mutant_details(state: LiveState, entry: CombinedResult) -> RenderableType: + """Everything known about `entry`: its id, function, coverage, which tests it survived or died to, and its source.""" + mutant = entry.mutant + context = state.contexts[mutant.id] + facts = Table.grid(padding=(0, 1)) + facts.add_column(min_width=DETAIL_LABEL_WIDTH, no_wrap=True) + facts.add_column() + survived = entry.state is MutantState.SURVIVED + mutant_id = Text.assemble((mutant.id, "bold"), (" " + EQUIVALENT_HINT, "dim")) if survived else mutant.id + detail_row(facts, "id", mutant_id) + detail_row(facts, "state", state_text(entry.state)) + detail_row(facts, "operator", Text.assemble(f"{mutant.operator} ", change(mutant))) + detail_row(facts, "function", entry.function or "unknown") + detail_row(facts, "coverage", execution_summary(mutant.line, context.executions)) + hiding = state.bug_hiding_nets() + if survived and hiding: + names = ", ".join(failure.net for failure in hiding) + caution = f"{names} off: this mutant may survive only because of that; 4 Stats explains" + detail_row(facts, "caution", Text(caution, style="bold red")) + if survived: + kind = "" if entry.survivor is None else f"{entry.survivor} " + detail_row(facts, str(MutantState.SURVIVED), f"{kind}under {', '.join(entry.covered_by)}") + detail_row(facts, "hint", Text(hint(entry), style="yellow")) + elif entry.killed_by: + detail_row(facts, "killed by", ", ".join(entry.killed_by)) + duration = state.durations.get(mutant.id) + if duration is not None: + detail_row(facts, "time", f"{duration:.2f}s") + return Group(facts, Text(""), diff_view(context.source, mutant, DETAIL_CONTEXT)) + + +def safety_badge(state: LiveState) -> Text: + """How many failing safety nets the run went on without; red when one of them lets broken mutants survive.""" + relaxed = state.relaxed_nets or [] + if not relaxed: + return Text("") + style = "bold white on red" if state.bug_hiding_nets() else "bold black on yellow" + return Text(f" ! {plural(len(relaxed), 'safety net')} off ", style=style) + + +def relaxed_net_view(failure: RelaxedSafetyNet) -> RenderableType: + """One relaxed net: how the configuration let it pass, where it fails, why that matters and how to fix it.""" + went_on = "with a warning" if failure.level is Level.WARN else "without a warning" + heading = Text.assemble( + ("✗ ", "bold red" if failure.hides_bugs else "bold yellow"), + (str(failure.net), "bold"), + f" set to {failure.level}, so the run went on {went_on}", + ) + if failure.hides_bugs: + heading.append("; by default it stops the run", style="red") + facts = Table.grid(padding=(0, 1)) + facts.add_column(min_width=DETAIL_LABEL_WIDTH, no_wrap=True) + facts.add_column() + detail_row(facts, "fails", failure.where.removeprefix("fails ")) + detail_row(facts, "why", Text(failure.why, style="yellow" if failure.hides_bugs else "")) + detail_row(facts, "fix", Text(failure.fix, style="cyan")) + return Group(heading, facts) + + +def safety_report(state: LiveState) -> RenderableType: + """The safety nets the configuration let fail, what that means for the results, and the nets that hold.""" + if state.relaxed_nets is None: + return Text("strain checks the safety nets while it prepares the run", style="dim") + holding = Text.assemble(("✔ hold: ", "green"), ", ".join(state.holding_nets) or "none") + if not state.relaxed_nets: + return Text.assemble(("✔ every safety net holds: ", "bold green"), ", ".join(state.holding_nets)) + hiding = state.bug_hiding_nets() + if hiding: + names = ", ".join(failure.net for failure in hiding) + summary = Text( + f"{names} {'is' if len(hiding) == 1 else 'are'} off, so a mutant that breaks the code can still pass the " + "tests: some survivors may point at the build rather than at a weak test. Fix the build before writing " + "tests for them.", + style="bold red", + ) + else: + summary = Text( + "These nets cost coverage precision or build speed, not correctness: survivors still point at weak tests.", + style="yellow", + ) + views: list[RenderableType] = [summary] + for failure in state.relaxed_nets: + views += [Text(""), relaxed_net_view(failure)] + return Group(*views, Text(""), holding) + + +def run_badge(state: LiveState, paused: bool) -> Text: + """What the run is doing, as a colored badge: its phase with a spinner, paused, stopping or finished.""" + if state.done: + return Text(" ✔ finished ", style="bold black on green") + if state.stopping: + return Text(" ■ stopping… ", style="bold white on red") + if paused: + return Text(" ‖ paused " if state.waiting else " ‖ pausing after this mutant ", style="bold black on yellow") + spinner = SPINNER[int(state.clock() * SPINNER_FRAMES_PER_SECOND) % len(SPINNER)] + return Text(f" {spinner} {state.phase() or 'preparing'} ", style="bold black on cyan") + + +def run_position(state: LiveState) -> Text: + where = Text.assemble( + (" ◆ strain ", "bold black on dark_cyan"), " ", (state.test_name if state.done else state.test, "bold") + ) + if state.current is not None and not state.done: + where.append(f" · mutant {state.current + 1}/{len(state.queue)}") + return where + + +def run_timing(state: LiveState) -> Text: + timing = Text(f"elapsed {clock_time(state.elapsed())}", style="grey62") + remaining = state.remaining_seconds() + if remaining is not None: + timing.append(f" · eta {clock_time(remaining)}") + mean = state.mean_duration() + if mean is not None: + timing.append(f" · {mean:.1f}s/mutant") + return timing + + +def run_scores(state: LiveState) -> Text: + """The score and its history, then the totals over every test, counting each mutant once. + + The score comes first, so a narrow terminal cuts the counts before it. + """ + scores = Text.assemble("score ", score_text(state.score())) + if len(state.score_history) > 1: + scores.append(f" {sparkline(state.score_history)}", style="cyan") + for kind, count in state.counts().items(): + scores.append(" ") + scores.append_text(state_text(kind, glyph_only=True)) + scores.append(f" {kind} {count}", style=STATE_STYLES[kind]) + return scores + + +class RunHeader(Static): + """The run's position, its state, the score and its history, and how long the run has taken and will take. + + Clicking the safety net badge shows why the nets it counts matter. + """ + + def __init__(self) -> None: + super().__init__() + self.safety_columns = range(0) + + def render_state(self, state: LiveState, paused: bool) -> RenderableType: + safety = safety_badge(state) + badge = run_badge(state, paused) + # The badges sit at the right edge, inside one column of padding. + end = self.size.width - 1 - len(badge) - (1 if safety else 0) + self.safety_columns = range(end - len(safety), end) + grid = Table.grid(expand=True) + grid.add_column(ratio=1, no_wrap=True, overflow="ellipsis") + grid.add_column(justify="right", no_wrap=True) + grid.add_row(run_position(state), Text.assemble(safety, " " if safety else "", badge)) + grid.add_row(run_scores(state), run_timing(state)) + return grid + + def on_click(self, event: events.Click) -> None: + if event.y == 0 and event.x in self.safety_columns: + self.app.show_safety_nets() # type: ignore[attr-defined] + + +class MutationMap(Widget): + """One cell per mutant (or per group of them on a long queue), colored by its result: the run at a glance. + + Hovering names the mutant under the pointer; clicking shows it in the queue. + """ + + DEFAULT_CSS = "MutationMap { height: 1; }" + + def __init__(self, state: LiveState, **kwargs: object) -> None: + super().__init__(**kwargs) # type: ignore[arg-type] + self.state = state + + def label_width(self) -> int: + return len(f" {len(self.state.queue)}/{len(self.state.queue)}") + 1 + + def cells(self) -> int: + return max(1, self.size.width - self.label_width()) + + def index_at(self, x: int) -> int | None: + count = len(self.state.queue) + if count == 0 or x >= self.cells(): + return None + return min(count - 1, x * count // self.cells()) + + def render(self) -> RenderableType: + with self.state.lock: + queue = self.state.queue + width = self.cells() + text = Text() + if not queue: + text.append("░" * width, style=MAP_COLORS["pending"]) + for cell in range(width if queue else 0): + first = cell * len(queue) // width + last = max(first + 1, (cell + 1) * len(queue) // width) + categories = {map_category(entry.state) for entry in queue[first:last]} + worst = next( + name for name in ("running", "survived", "detected", "other", "pending") if name in categories + ) + text.append("░" if worst == "pending" else "█", style=MAP_COLORS[worst]) + text.append(f" {self.state.done_count()}/{len(queue)}", style="bold") + return text + + def on_mouse_move(self, event: events.MouseMove) -> None: + index = self.index_at(event.x) + with self.state.lock: + if index is None: + self.tooltip = None + else: + entry = self.state.queue[index] + self.tooltip = f"{index + 1}. {entry.state}: {describe_mutant(entry.mutant)}" + + def on_click(self, event: events.Click) -> None: + index = self.index_at(event.x) + if index is not None: + self.app.show_queue_entry(index) # type: ignore[attr-defined] + + +class CodeView(Static): + """The mutant under the queue cursor, as a highlighted diff that grows with the panel.""" + + def __init__(self, **kwargs: object) -> None: + super().__init__(**kwargs) # type: ignore[arg-type] + self.shown: tuple[str, Mutant] | None = None + + def show(self, source: str | None, mutant: Mutant | None) -> None: + shown = None if source is None or mutant is None else (source, mutant) + if shown == self.shown: + return + self.shown = shown + self.redraw() + + def redraw(self) -> None: + if self.shown is None: + self.update(Text("waiting for the first mutant", style="dim")) + return + source, mutant = self.shown + context = max(SNIPPET_CONTEXT, (self.content_size.height - 2) // 2) + self.update(diff_view(source, mutant, context)) + + def on_resize(self, event: events.Resize) -> None: + self.redraw() + + +class StrainApp(App[None]): + TITLE = "strain" + CSS = """ + Screen { layout: vertical; } + RunHeader { height: 2; padding: 0 1; } + MutationMap { margin: 0 1; } + TabbedContent { height: 1fr; } + TabPane { padding: 0; } + .panel { border: round $panel-lighten-2; border-title-color: $accent; border-title-style: bold; } + .panel:focus-within { border: round $accent; } + #run-left { width: 5fr; } + #run-right { width: 4fr; } + #code { height: 3fr; padding: 0 1; text-wrap: nowrap; text-overflow: ellipsis; } + #build { height: 2fr; } + /* The time column comes last, so a narrow terminal loses it rather than scrolling sideways. */ + #queue { height: 3fr; overflow-x: hidden; } + #findings-box { height: 2fr; } + #findings { padding: 0 1; } + .list { width: 2fr; } + .list DataTable { height: 1fr; } + .list #survivor-files { height: auto; max-height: 40%; } + .details { width: 3fr; padding: 0 1; } + #filters { height: 2; } + #safety-box { height: auto; max-height: 60%; padding: 0 1; } + #stats-tables { height: 1fr; } + .stats-column { width: 1fr; } + .stats-column DataTable { height: auto; max-height: 50%; } + DataTable { scrollbar-size-vertical: 1; } + """ + BINDINGS: ClassVar[list[BindingType]] = [ + Binding("ctrl+c", "stop", "Stop / quit", priority=True), + Binding("q", "quit_when_done", "Quit"), + Binding("p", "pause", "Pause"), + Binding("f", "follow", "Follow"), + Binding("?", "show_help_panel", "Keys"), + *( + Binding(str(number), f"show_view('{view}')", view.capitalize(), show=False) + for number, view in enumerate(VIEWS, start=1) + ), + Binding("tab", "next_view", "Next view", show=False, priority=True), + Binding("shift+tab", "previous_view", "Previous view", show=False, priority=True), + ] + + def __init__(self, console: "TuiConsole") -> None: + super().__init__() + self.runner_console = console + self.state = console.live + self.following = True + self.seen_version = -1 + self.seen_results = -1 + self.seen_plan = -1 + self.seen_build = -1 + self.build_lines_shown = 0 + self.transcript_shown = 0 + self.queue_rows: list[tuple[str, float | None]] = [] + self.survivors_seen: set[str] = set() + self.pending_toasts: list[str] = [] + self.last_toast = 0.0 + self.announced_finish = False + # The safety net report last shown: None before the check, then the list the console reported. + self.shown_safety: list[RelaxedSafetyNet] | None = None + self.state_filter: str | None = None + self.dirty_tables: set[str] = set() + + def compose(self) -> ComposeResult: + yield RunHeader() + yield MutationMap(self.state) + with TabbedContent(initial="run"): + yield from self.compose_run() + yield from self.compose_browsers() + yield from self.compose_stats() + with TabPane("5 Log", id="log"): + yield RichLog(id="transcript", classes="panel", wrap=True, markup=False) + yield Footer() + + def compose_run(self) -> ComposeResult: + with TabPane("1 Run", id="run"), Horizontal(): + with Vertical(id="run-left"): + yield CodeView(id="code", classes="panel") + yield RichLog(id="build", classes="panel", max_lines=2000, wrap=True, markup=False, min_width=20) + with Vertical(id="run-right"): + yield table("queue") + with VerticalScroll(id="findings-box", classes="panel"): + yield Static(id="findings") + + def compose_browsers(self) -> ComposeResult: + with TabPane("2 Survivors", id="survivors"), Horizontal(): + with Vertical(classes="list"): + yield table("survivor-table") + yield table("survivor-files") + with VerticalScroll(id="survivor-box", classes="panel details"): + yield Static(id="survivor-detail") + with TabPane("3 Mutants", id="mutants"), Horizontal(): + with Vertical(classes="list"): + yield Tabs(Tab("all", id="filter-all"), id="filters") + yield table("mutant-table") + with VerticalScroll(id="mutant-box", classes="panel details"): + yield Static(id="mutant-detail") + + def compose_stats(self) -> ComposeResult: + with TabPane("4 Stats", id="stats"), Vertical(): + with VerticalScroll(id="safety-box", classes="panel"): + yield Static(id="safety") + with Horizontal(id="stats-tables"): + with Vertical(classes="stats-column"): + yield table("stats-files") + yield table("stats-operators") + with Vertical(classes="stats-column"): + yield table("stats-tests") + yield table("stats-slowest") + + def on_mount(self) -> None: + for table_id, (title, columns) in TABLES.items(): + data_table = self.query_one(f"#{table_id}", DataTable) + data_table.border_title = title + for key, label in columns: + # A fixed width, so a longer state arriving later never shifts the other columns. + data_table.add_column(label, key=key, width=STATE_COLUMN_WIDTH if key == "state" else None) + for selector, title in PANEL_TITLES.items(): + self.query_one(selector).border_title = title + self.set_interval(REFRESH_SECONDS, self.refresh_from_state) + self.refresh_from_state() + # Only now can the runner's thread rely on the app to be running. + self.runner_console.start_worker() + + # Keeping the widgets up to date. + + def refresh_from_state(self) -> None: + try: + self.redraw() + except NoMatches: + # The app is closing and its widgets are already gone; there is nothing left to draw. + return + # The run failed, or the user asked to stop it, and the runner has wound down: nothing left to show. + if self.runner_console.worker_failed() or (self.state.stopping and self.runner_console.worker_finished()): + self.exit() + + def redraw(self) -> None: + with self.state.lock: + self.query_one(RunHeader).update( + self.query_one(RunHeader).render_state(self.state, self.runner_console.paused) + ) + self.query_one(MutationMap).refresh() + self.refresh_code_title() + if self.state.version != self.seen_version: + self.seen_version = self.state.version + self.refresh_queue() + self.refresh_build_log() + self.refresh_transcript() + self.refresh_safety_nets() + if self.state.results_version != self.seen_results: + self.seen_results = self.state.results_version + self.refresh_results() + self.refresh_code() + self.flush_toasts() + + def refresh_queue(self) -> None: + table = self.query_one("#queue", DataTable) + queue = self.state.queue + if self.state.plan_version != self.seen_plan: + self.seen_plan = self.state.plan_version + table.clear() + self.queue_rows = [] + for index, entry in enumerate(queue): + table.add_row(*self.queue_cells(entry), key=str(index)) + self.queue_rows.append((entry.state, entry.duration)) + self.following = True + for index, entry in enumerate(queue): + if self.queue_rows[index] != (entry.state, entry.duration): + self.queue_rows[index] = (entry.state, entry.duration) + cells = self.queue_cells(entry) + table.update_cell(str(index), "state", cells[0]) + table.update_cell(str(index), "time", cells[-1], update_width=True) + done = self.state.done_count() + table.border_title = f"Mutants {done}/{len(queue)}" if queue else "Mutants" + table.border_subtitle = "" if self.following else "browsing · f follows the run" + if self.following and self.state.current is not None and table.cursor_row != self.state.current: + table.move_cursor(row=self.state.current) + + def queue_cells(self, entry: QueueEntry) -> tuple[Text, ...]: + time = Text("" if entry.duration is None else f"{entry.duration:.1f}s", style="grey62", justify="right") + return ( + state_text(entry.state, short=True), + Text(location(entry.mutant, short=True)), + Text(entry.mutant.operator, style="magenta"), + change(entry.mutant), + time, + ) + + def cursor_entry(self) -> tuple[int, QueueEntry] | None: + table = self.query_one("#queue", DataTable) + queue = self.state.queue + if not queue: + return None + index = self.state.current if self.following and self.state.current is not None else table.cursor_row + index = max(0, min(index, len(queue) - 1)) + return index, queue[index] + + def refresh_code(self) -> None: + shown = self.cursor_entry() + code = self.query_one(CodeView) + if shown is None: + code.show(None, None) + return + _, entry = shown + code.show(self.state.sources.get(entry.mutant.file), entry.mutant) + + def refresh_code_title(self) -> None: + code = self.query_one(CodeView) + shown = self.cursor_entry() + if shown is None: + return + index, entry = shown + code.border_title = f"Mutant {index + 1}: {describe_mutant(entry.mutant)}" + elapsed = self.state.current_elapsed() + if index == self.state.current and elapsed is not None: + code.border_subtitle = f"{self.state.phase() or RUNNING} {elapsed:.1f}s" + else: + code.border_subtitle = str(entry.state) + + def refresh_build_log(self) -> None: + log = self.query_one("#build", RichLog) + if self.state.build_version != self.seen_build: + self.seen_build = self.state.build_version + log.clear() + log.border_title = f"Build log: {self.state.build_command}" + self.build_lines_shown = self.state.build_lines_seen - len(self.state.build_log) + new = self.state.build_lines_seen - self.build_lines_shown + if new > 0: + for line in list(self.state.build_log)[-min(new, len(self.state.build_log)) :]: + log.write(Text.from_ansi(line, style="grey70")) + self.build_lines_shown = self.state.build_lines_seen + + def refresh_transcript(self) -> None: + log = self.query_one("#transcript", RichLog) + for line in self.state.log[self.transcript_shown :]: + log.write(Text.from_ansi(line)) + self.transcript_shown = len(self.state.log) + + def refresh_results(self) -> None: + findings = self.state.findings() + box = self.query_one("#findings-box") + box.border_title = f"Findings {len(findings)}" + bullets = [Text.assemble((" • ", "yellow"), item) for item in findings] + self.query_one("#findings", Static).update( + Text("\n").join(bullets) if bullets else Text("no findings yet", style="dim") + ) + survivors = self.state.survivors() + tabs = self.query_one(TabbedContent) + tabs.get_tab("survivors").label = f"2 Survivors {len(survivors)}" + tabs.get_tab("mutants").label = f"3 Mutants {len(self.state.combined())}" + for entry in survivors: + if entry.mutant.id not in self.survivors_seen: + self.survivors_seen.add(entry.mutant.id) + self.pending_toasts.append(describe_mutant(entry.mutant)) + self.refresh_filters() + self.dirty_tables = {"survivors", "mutants", "stats"} + self.refresh_active_tables() + + def refresh_filters(self) -> None: + filters = self.query_one("#filters", Tabs) + counts = self.state.counts() + filters.query_one("#filter-all", Tab).label = f"all {sum(counts.values())}" + for state, count in counts.items(): + tab_id = f"filter-{state}" + existing = filters.query(f"#{tab_id}") + label = f"{state} {count}" + if existing: + existing.first(Tab).label = label + else: + filters.add_tab(Tab(label, id=tab_id)) + + def refresh_active_tables(self) -> None: + active = self.query_one(TabbedContent).active + if active in self.dirty_tables: + self.dirty_tables.discard(active) + {"survivors": self.refresh_survivors, "mutants": self.refresh_mutants, "stats": self.refresh_stats}[ + active + ]() + + def fill_browser( + self, + table: DataTable, + entries: list[CombinedResult], + cells: Callable[[CombinedResult], tuple[RenderableType, ...]], + ) -> None: + """Refills `table` with `entries`, keeping the cursor on the same mutant.""" + selected = None + if table.row_count and 0 <= table.cursor_row < table.row_count: + selected = table.coordinate_to_cell_key((table.cursor_row, 0)).row_key.value + table.clear() + for entry in entries: + table.add_row(*cells(entry), key=entry.mutant.id) + keys = [entry.mutant.id for entry in entries] + if selected in keys: + table.move_cursor(row=keys.index(selected), animate=False) + + def refresh_survivors(self) -> None: + table = self.query_one("#survivor-table", DataTable) + survivors = self.state.survivors() + table.border_title = f"Survivors {len(survivors)}" + self.fill_browser( + table, + survivors, + lambda entry: ( + Text(location(entry.mutant, short=True)), + Text(entry.mutant.operator, style="magenta"), + change(entry.mutant), + Text(str(entry.survivor or ""), style="yellow"), + ), + ) + self.show_selected("survivor-table", "survivor-detail", survivors) + files = self.query_one("#survivor-files", DataTable) + files.clear() + for row in self.state.file_scores(): + files.add_row( + Text(row.file.rpartition("/")[2]), + score_text(row.score), + bar(row.detected, row.detected + row.survived, 8), + Text(str(row.survived), style="bold red" if row.survived else "dim", justify="right"), + ) + + def fill_file_scores(self, files: DataTable) -> None: + files.clear() + for row in self.state.file_scores(): + files.add_row(*self.tally_row(Tally(row.file, row.mutants, row.detected, row.survived))) + + def filtered_mutants(self) -> list[CombinedResult]: + combined = self.state.combined() + return ( + combined if self.state_filter is None else [entry for entry in combined if entry.state == self.state_filter] + ) + + def refresh_mutants(self) -> None: + table = self.query_one("#mutant-table", DataTable) + entries = self.filtered_mutants() + table.border_title = f"Mutants {len(entries)}" + ( + "" if self.state_filter is None else f" · {self.state_filter}" + ) + self.fill_browser( + table, + entries, + lambda entry: ( + state_text(entry.state, short=True), + Text(location(entry.mutant, short=True)), + Text(entry.mutant.operator, style="magenta"), + change(entry.mutant), + Text( + f"{self.state.durations[entry.mutant.id]:.1f}s" if entry.mutant.id in self.state.durations else "", + style="grey62", + justify="right", + ), + ), + ) + self.show_selected("mutant-table", "mutant-detail", entries) + + def tally_row(self, tally: Tally, *counts: Text) -> tuple[RenderableType, ...]: + return ( + Text(tally.name), + score_text(tally.score), + bar(tally.detected, tally.detected + tally.survived, 8), + Text(str(tally.mutants), justify="right"), + *counts, + Text(str(tally.survived), style="bold red" if tally.survived else "dim", justify="right"), + ) + + def refresh_stats(self) -> None: + self.fill_file_scores(self.query_one("#stats-files", DataTable)) + operators = self.query_one("#stats-operators", DataTable) + operators.clear() + for tally in self.state.operator_tallies(): + operators.add_row(*self.tally_row(tally)) + tests = self.query_one("#stats-tests", DataTable) + tests.clear() + for tally in self.state.test_tallies(): + tests.add_row(*self.tally_row(tally, Text(str(tally.detected), style="green", justify="right"))) + slowest = self.query_one("#stats-slowest", DataTable) + slowest.clear() + for duration, entry in self.state.slowest(): + slowest.add_row( + Text(f"{duration:.2f}s", style="yellow", justify="right"), + state_text(entry.state, short=True), + location(entry.mutant, short=True), + change(entry.mutant), + ) + + def show_selected(self, table_id: str, detail_id: str, entries: list[CombinedResult]) -> None: + table = self.query_one(f"#{table_id}", DataTable) + detail = self.query_one(f"#{detail_id}", Static) + box = detail.parent + if not entries: + detail.update( + Text("no surviving mutants" if table_id == "survivor-table" else "no mutants yet", style="dim") + ) + if isinstance(box, Widget): + box.border_title = "Details" + return + entry = entries[max(0, min(table.cursor_row, len(entries) - 1))] + detail.update(mutant_details(self.state, entry)) + if isinstance(box, Widget): + box.border_title = describe_mutant(entry.mutant) + + def flush_toasts(self) -> None: + if self.state.done and not self.announced_finish: + self.announced_finish = True + self.pending_toasts.clear() + survivors = len(self.state.survivors()) + self.notify( + f"score {format_score(self.state.score())}, {plural(survivors, 'survivor')} to look at", + title="Run finished", + severity="information" if survivors == 0 else "warning", + timeout=8, + ) + return + if not self.pending_toasts or self.state.clock() - self.last_toast < TOAST_SECONDS: + return + self.last_toast = self.state.clock() + if len(self.pending_toasts) == 1: + self.notify(self.pending_toasts[0], title="✗ Survivor", severity="warning", timeout=4) + else: + self.notify(f"{len(self.pending_toasts)} new survivors", title="✗ Survivors", severity="warning", timeout=4) + self.pending_toasts.clear() + + # Events. + + def on_data_table_row_highlighted(self, event: DataTable.RowHighlighted) -> None: + table_id = event.data_table.id + with self.state.lock: + if table_id == "queue": + if event.cursor_row != self.state.current: + self.following = False + event.data_table.border_subtitle = "browsing · f follows the run" + self.refresh_code() + self.refresh_code_title() + elif table_id == "survivor-table": + self.show_selected("survivor-table", "survivor-detail", self.state.survivors()) + elif table_id == "mutant-table": + self.show_selected("mutant-table", "mutant-detail", self.filtered_mutants()) + + def on_data_table_row_selected(self, event: DataTable.RowSelected) -> None: + """Enter, or a click on the highlighted row, opens a queue entry in the mutant browser.""" + if event.data_table.id != "queue": + return + with self.state.lock: + if event.cursor_row >= len(self.state.queue): + return + mutant_id = self.state.queue[event.cursor_row].mutant.id + if mutant_id not in self.state.contexts: + self.notify("this mutant has no result yet", timeout=2) + return + self.open_mutant(mutant_id) + + def open_mutant(self, mutant_id: str) -> None: + self.state_filter = None + self.query_one("#filters", Tabs).active = "filter-all" + self.show_view("mutants") + with self.state.lock: + self.refresh_mutants() + keys = [entry.mutant.id for entry in self.filtered_mutants()] + if mutant_id in keys: + self.query_one("#mutant-table", DataTable).move_cursor(row=keys.index(mutant_id)) + + def on_tabs_tab_activated(self, event: Tabs.TabActivated) -> None: + if event.tabs.id != "filters" or event.tab is None: + return + name = (event.tab.id or "").removeprefix("filter-") + self.state_filter = None if name == "all" else name + with self.state.lock: + self.refresh_mutants() + + def on_tabbed_content_tab_activated(self, event: TabbedContent.TabActivated) -> None: + """Fills the newly shown view and, after a click on its tab, focuses it so the arrow keys work there.""" + with self.state.lock: + self.refresh_active_tables() + pane = event.pane + # The message may arrive after the user has moved on; focusing into that pane would switch back to it. + if pane.id != self.query_one(TabbedContent).active: + return + if self.focused is None or pane not in self.focused.ancestors: + self.screen.set_focus(self.query_one(VIEW_FOCUS[pane.id or "run"])) + + def show_view(self, view: str) -> None: + """Shows `view` and focuses it in the same step. + + Focus must not stay on a widget in the old view: when that view hides, Textual hands focus to a neighbouring + widget there, and a focused widget makes its view the active one again. + """ + self.query_one(TabbedContent).active = view + self.screen.set_focus(self.query_one(VIEW_FOCUS[view])) + + def refresh_safety_nets(self) -> None: + relaxed = self.state.relaxed_nets + if relaxed is self.shown_safety: + return + self.shown_safety = relaxed + self.query_one("#safety", Static).update(safety_report(self.state)) + box = self.query_one("#safety-box") + box.border_title = "Safety nets" if not relaxed else f"Safety nets: {len(relaxed)} off" + if relaxed: + hiding = self.state.bug_hiding_nets() + names = ", ".join(failure.net for failure in relaxed) + self.notify( + f"{names} off; 4 Stats explains why it matters", + title="! Safety nets", + severity="error" if hiding else "warning", + timeout=6, + ) + + def show_safety_nets(self) -> None: + self.show_view("stats") + self.query_one("#safety-box").scroll_home(animate=False) + + def show_queue_entry(self, index: int) -> None: + self.show_view("run") + self.following = False + self.query_one("#queue", DataTable).move_cursor(row=index) + + # Actions. + + def action_show_view(self, view: str) -> None: + self.show_view(view) + + def action_next_view(self) -> None: + self.cycle_view(1) + + def action_previous_view(self) -> None: + self.cycle_view(-1) + + def cycle_view(self, step: int) -> None: + self.show_view(VIEWS[(VIEWS.index(self.query_one(TabbedContent).active) + step) % len(VIEWS)]) + + def action_follow(self) -> None: + if self.query_one(TabbedContent).active == "mutants": + filters = self.query_one("#filters", Tabs) + filters.action_next_tab() + return + self.following = True + with self.state.lock: + self.refresh_queue() + self.refresh_code() + + def action_pause(self) -> None: + if self.state.done: + return + self.runner_console.set_paused(not self.runner_console.paused) + self.notify("pausing after this mutant" if self.runner_console.paused else "resumed", timeout=2) + + def action_quit_when_done(self) -> None: + if self.state.done: + self.exit() + else: + self.notify("the run is still going: Ctrl-C stops it and restores the sources", timeout=3) + + def action_stop(self) -> None: + if self.state.done or self.runner_console.worker_finished(): + self.exit() + return + if self.state.stopping: + self.notify("still stopping: waiting for the current build or test to end", timeout=3) + return + self.runner_console.request_stop() + self.notify("stopping after the current step; the sources will be restored", title="Stopping", severity="error") + + +class TuiConsole(Console): + """Collects the run's progress for the TUI, then prints the plain transcript once the TUI has closed.""" + + def __init__(self, out: TextIO, err: TextIO, headless: bool = False) -> None: + super().__init__(out, err, UiMode.LIVE) + self.live = LiveState() + self.headless = headless + self.active = True + self.paused = False + self.transcript: list[str] = [] + self.warnings: list[str] = [] + self.resume = threading.Event() + self.resume.set() + self.stop_requested = threading.Event() + self.stop_raised = False + self.worker: threading.Thread | None = None + self.worker_started = False + self.worker_error: BaseException | None = None + self.app: StrainApp | None = None + + # Running the TUI. + + def run(self, work: Callable[[], int]) -> int: + """Runs `work` on a worker thread while the TUI owns the terminal, and returns what `work` returns.""" + result: list[int] = [] + + def target() -> None: + try: + result.append(work()) + except BaseException as error: # noqa: BLE001 - re-raised on the main thread below + self.worker_error = error + + self.app = StrainApp(self) + self.worker = threading.Thread(target=target, name="strain-runner", daemon=True) + previous = signal.signal(signal.SIGTERM, self._stop_on_signal) + try: + self.app.run(headless=self.headless) + finally: + signal.signal(signal.SIGTERM, previous) + self.close() + if self.worker_error is not None: + raise self.worker_error + if not result: + raise RuntimeError("the live TUI closed before the run started; try --ui auto") + return result[0] + + def start_worker(self) -> None: + """Starts the run, once the app is up; called from the app's first mount.""" + if self.worker is not None and not self.worker_started: + self.worker_started = True + self.worker.start() + + def _stop_on_signal(self, signal_number: int, frame: FrameType | None) -> None: + self.request_stop() + + def close(self) -> None: + """Stops a run the user left early and waits for it, so the sources are restored before strain exits.""" + self.active = False + if self.worker is not None and self.worker.is_alive(): + self.request_stop() + self.err.write("strain: stopping the run and restoring the sources…\n") + self.err.flush() + while self.worker.is_alive(): + self.worker.join(WORKER_POLL_SECONDS) + + def worker_finished(self) -> bool: + return self.worker is not None and self.worker_started and not self.worker.is_alive() + + def worker_failed(self) -> bool: + return self.worker_finished() and self.worker_error is not None + + def request_stop(self) -> None: + self.live.set_flag("stopping", True) + self.stop_requested.set() + self.resume.set() + + def set_paused(self, paused: bool) -> None: + self.paused = paused + self.live.set_flag("paused", paused) + if paused: + self.resume.clear() + else: + self.resume.set() + + def check_stop(self) -> None: + """Raises KeyboardInterrupt in the runner, once, after the user asked to stop.""" + if self.stop_requested.is_set() and not self.stop_raised: + self.stop_raised = True + raise KeyboardInterrupt + + # Console output, kept for the log view and the transcript. + + def line(self, text: str = "") -> None: + if not self.active: + super().line(text) + return + self.check_stop() + self.transcript.append(text) + self.live.add_line(text) + + def status(self, text: str) -> None: + if self.active: + self.check_stop() + self.live.set_status(text) + + def warn(self, message: str) -> None: + if not self.active: + super().warn(message) + return + self.warnings.append(message) + self.live.add_line(f"warning: {message}") + + def finish(self) -> None: + # Taken before printing: Console.warn calls finish() again. + transcript, self.transcript = self.transcript, [] + warnings, self.warnings = self.warnings, [] + self.active = False + for text in transcript: + super().line(text) + for message in warnings: + super().warn(message) + super().finish() + + # Progress hooks. + + def test_started(self, name: str, number: int, total: int) -> None: + self.check_stop() + self.live.set_test(name, number, total) + + def plan(self, mutants: Sequence[Mutant]) -> None: + self.live.plan(mutants) + + def mutant_started(self, index: int, source: str, mutant: Mutant) -> None: + self.check_stop() + if not self.resume.is_set(): + self.live.set_flag("waiting", True) + while not self.resume.wait(WORKER_POLL_SECONDS): + pass + self.live.set_flag("waiting", False) + self.check_stop() + self.live.start_mutant(index, source, mutant) + + def mutant_finished(self, index: int, state: str) -> None: + self.live.finish_mutant(index, state) + + def mutant_evaluated(self, result: MutantResult, source: str, line: LineCoverage | None) -> None: + self.live.record_result(result, source, line) + + def safety_nets_checked(self, relaxed: Sequence[RelaxedSafetyNet], holding: Sequence[SafetyNet]) -> None: + self.live.set_safety_nets(relaxed, holding) + + def build_started(self, command: str) -> None: + self.check_stop() + self.live.start_build(command) + + def build_output(self, line: str) -> None: + self.check_stop() + self.live.build_output(line) + + def run_finished(self) -> None: + self.set_paused(False) + self.live.finish_run() diff --git a/tests/test_console.py b/tests/test_console.py index a535fca..89b532a 100644 --- a/tests/test_console.py +++ b/tests/test_console.py @@ -1,9 +1,21 @@ +import importlib.util import io import os import unittest from unittest import mock -from strain.console import CLEAR_LINE, RESET, STATE_WIDTH, Console, UiMode, paint, plural, score_style, wants_color +from strain.console import ( + CLEAR_LINE, + RESET, + STATE_WIDTH, + Console, + UiMode, + create_console, + paint, + plural, + score_style, + wants_color, +) from strain.results import MutantState @@ -84,5 +96,37 @@ def test_json_mode_prints_only_warnings(self) -> None: self.assertEqual(err.getvalue(), "strain: warning: test t fails without mutation\n") +class CreateConsoleTest(unittest.TestCase): + def setUp(self) -> None: + environment = mock.patch.dict(os.environ, {"NO_COLOR": "1"}, clear=True) + environment.start() + self.addCleanup(environment.stop) + + def test_live_mode_off_a_terminal_falls_back_to_plain_lines(self) -> None: + out = io.StringIO() + console = create_console(out, io.StringIO(), UiMode.LIVE) + self.assertIs(type(console), Console) + console.step("working tree is clean") + self.assertEqual(out.getvalue(), " ✓ working tree is clean\n") + + def test_live_mode_without_textual_warns_and_shows_plain_output(self) -> None: + err = io.StringIO() + with mock.patch("importlib.util.find_spec", return_value=None): + console = create_console(TerminalBuffer(), err, UiMode.LIVE) + self.assertIs(type(console), Console) + self.assertIn("--ui live needs Textual: pip install 'strain[ui]'", err.getvalue()) + + @unittest.skipUnless(importlib.util.find_spec("textual"), "the TUI needs Textual") + def test_live_mode_on_a_terminal_uses_the_tui(self) -> None: + from strain.tui import TuiConsole + + self.assertIsInstance(create_console(TerminalBuffer(), io.StringIO(), UiMode.LIVE), TuiConsole) + + def test_other_modes_never_use_the_tui(self) -> None: + for mode in (UiMode.AUTO, UiMode.LOG, UiMode.JSON): + with self.subTest(mode=mode): + self.assertIs(type(create_console(TerminalBuffer(), io.StringIO(), mode)), Console) + + if __name__ == "__main__": unittest.main() diff --git a/tests/test_dashboard.py b/tests/test_dashboard.py deleted file mode 100644 index ebfd8b7..0000000 --- a/tests/test_dashboard.py +++ /dev/null @@ -1,411 +0,0 @@ -import io -import os -import re -import threading -import unittest -from unittest import mock - -from strain.console import BOLD, GREEN, RED, RESET, REVERSE, UiMode, score_style -from strain.dashboard import ( - ALTERNATE_SCREEN, - MAIN_SCREEN, - Dashboard, - LiveConsole, - RUN_VIEW, - SURVIVORS_VIEW, - code_snippet, - create_console, - fit, - fit_keeping_end, - progress_bar, -) -from strain.coverage import LineCoverage -from strain.mutators import Mutant -from strain.results import MutantResult, MutantState, SurvivorKind - -SOURCE = "int clamp(int a, int b)\n{\n if (a < b)\n return a;\n return b;\n}\n" -MUTANT = Mutant("src/clamp.cpp", 3, 11, SOURCE.index("<"), "ror", "<", "<=") -SECOND_MUTANT = Mutant("src/clamp.cpp", 3, 11, SOURCE.index("<"), "ror", "<", ">=") -LINE_COVERAGE = LineCoverage(count=12, branches=2, branches_taken=2) -ESCAPE = re.compile(r"\033\[[0-9;?]*[A-Za-z]") - - -class TerminalBuffer(io.StringIO): - def isatty(self) -> bool: - return True - - -def terminal(columns: int, lines: int): - return lambda: os.terminal_size((columns, lines)) - - -def no_keys(on_key): - return None - - -class FakeKeys: - def __init__(self) -> None: - self.closed = False - - def close(self) -> None: - self.closed = True - - -def plain(text: str) -> str: - return ESCAPE.sub("", text) - - -class FitTest(unittest.TestCase): - def test_pads_and_truncates_by_visible_width(self) -> None: - self.assertEqual(fit("abc", 5), "abc ") - self.assertEqual(fit("abcdef", 4), "abc…") - self.assertEqual(plain(fit("\033[1mabcdef\033[0m", 4)), "abc…") - self.assertEqual(plain(fit("\033[1mab\033[0m", 4)), "ab ") - - -class FitKeepingEndTest(unittest.TestCase): - def test_truncates_the_text_before_the_end(self) -> None: - self.assertEqual(fit_keeping_end("abcdef", "XY", 6), "abc…XY") - self.assertEqual(fit_keeping_end("ab", "XY", 6), "ab XY") - - def test_an_end_wider_than_the_line_is_truncated_itself(self) -> None: - self.assertEqual(fit_keeping_end("ab", "WXYZ", 3), "WX…") - - -class CodeSnippetTest(unittest.TestCase): - def test_shows_the_original_and_mutated_line_in_context(self) -> None: - self.assertEqual( - code_snippet(SOURCE, MUTANT, context=1), - [ - " 2 │ {", - "- 3 │ if (a < b)", - "+ 3 │ if (a <= b)", - " 4 │ return a;", - ], - ) - - def test_context_is_clipped_at_the_file_edges(self) -> None: - first = Mutant("f.cpp", 1, 1, 0, "aor", "+", "-") - self.assertEqual(code_snippet("+x\ny\n", first, context=2), ["- 1 │ +x", "+ 1 │ -x", " 2 │ y"]) - - -class ProgressBarTest(unittest.TestCase): - def test_fills_in_proportion_to_the_finished_work(self) -> None: - self.assertEqual(progress_bar(0, 4, width=8), "░░░░░░░░") - self.assertEqual(progress_bar(1, 4, width=8), "██░░░░░░") - self.assertEqual(progress_bar(4, 4, width=8), "████████") - - def test_an_empty_plan_shows_an_empty_bar(self) -> None: - self.assertEqual(progress_bar(0, 0, width=4), "░░░░") - - -class DashboardTest(unittest.TestCase): - def make_dashboard(self, columns: int = 60, lines: int = 30) -> Dashboard: - dashboard = Dashboard(color=False) - dashboard.set_test("sample_tests", 1, 2) - dashboard.set_plan([f"src/clamp.cpp:{line}:1 [ror] `<` → `<=`" for line in range(1, 21)]) - return dashboard - - def test_frame_has_every_panel_and_fits_the_terminal(self) -> None: - dashboard = self.make_dashboard() - dashboard.start_mutant(0, SOURCE, MUTANT) - dashboard.build_output("[1/2] Building CXX object clamp.cpp.o") - dashboard.set_status("building…") - frame = dashboard.render(60, 30) - self.assertEqual(len(frame), 30) - self.assertTrue(all(len(line) == 60 for line in frame)) - text = "\n".join(frame) - self.assertIn("Test 1/2: sample_tests", text) - self.assertIn("mutant 1/20", text) - self.assertIn("- 3 │ if (a < b)", text) - self.assertIn("+ 3 │ if (a <= b)", text) - self.assertIn("[1/2] Building CXX object clamp.cpp.o", text) - self.assertIn("building…", frame[-1]) - - def test_queue_shows_previous_results_and_upcoming_mutants(self) -> None: - dashboard = self.make_dashboard() - dashboard.start_mutant(0, SOURCE, MUTANT) - dashboard.record_result(killed(MUTANT), SOURCE, LINE_COVERAGE) - dashboard.finish_mutant(0, MutantState.KILLED) - dashboard.start_mutant(1, SOURCE, SECOND_MUTANT) - dashboard.record_result(survived(SECOND_MUTANT), SOURCE, LINE_COVERAGE) - dashboard.finish_mutant(1, MutantState.SURVIVED) - dashboard.start_mutant(2, SOURCE, MUTANT) - text = "\n".join(dashboard.render(80, 30)) - self.assertRegex(text, r"killed +src/clamp.cpp:1:1") - self.assertRegex(text, r"survived +src/clamp.cpp:2:1") - self.assertRegex(text, r"▶ running +src/clamp.cpp:3:1") - self.assertRegex(text, r"pending +src/clamp.cpp:4:1") - self.assertIn("score 50.0% killed 1 survived 1", text) - self.assertIn("██░░░░░░░░░░░░░░░░░░ sample_tests 2/20", text) - - def test_colors_highlight_the_mutated_token_and_the_score(self) -> None: - dashboard = Dashboard(color=True) - dashboard.set_plan(["first"]) - dashboard.start_mutant(0, SOURCE, MUTANT) - dashboard.record_result(killed(MUTANT), SOURCE, LINE_COVERAGE) - dashboard.finish_mutant(0, MutantState.KILLED) - frame = dashboard.render(60, 30) - text = "\n".join(frame) - self.assertIn(f"{BOLD}{REVERSE}{RED}<{RESET}", text) - self.assertIn(f"{BOLD}{REVERSE}{GREEN}<={RESET}", text) - self.assertIn(f"score {score_style(100.0)}100.0%{RESET}", text) - self.assertTrue(all(len(plain(line)) == 60 for line in frame)) - - def test_queue_scrolls_to_keep_the_current_mutant_visible(self) -> None: - dashboard = self.make_dashboard() - for index in range(15): - dashboard.start_mutant(index, SOURCE, MUTANT) - dashboard.finish_mutant(index, MutantState.KILLED) - dashboard.start_mutant(15, SOURCE, MUTANT) - text = "\n".join(dashboard.render(60, 30)) - self.assertIn("src/clamp.cpp:16:1", text) - self.assertNotIn("src/clamp.cpp:1:1 ", text) - self.assertRegex(text, r"↑ \d+ earlier") - - def test_build_log_keeps_the_latest_lines_of_the_current_build(self) -> None: - dashboard = self.make_dashboard() - dashboard.start_build("cmake --build build") - for number in range(100): - dashboard.build_output(f"line {number}") - text = "\n".join(dashboard.render(60, 30)) - self.assertIn("line 99", text) - self.assertNotRegex(text, r"\bline 0\b") - dashboard.start_build("cmake --build build") - self.assertNotIn("line 99", "\n".join(dashboard.render(60, 30))) - - def test_tiny_terminals_still_render(self) -> None: - dashboard = self.make_dashboard() - dashboard.start_mutant(0, SOURCE, MUTANT) - for columns, lines in ((10, 3), (1, 1), (80, 8)): - with self.subTest(size=(columns, lines)): - frame = dashboard.render(columns, lines) - self.assertEqual(len(frame), lines) - self.assertTrue(all(len(line) == columns for line in frame)) - - -def survived(mutant: Mutant, test: str = "sample_tests") -> MutantResult: - return MutantResult(mutant, test, MutantState.SURVIVED, SurvivorKind.NOT_DETECTED, "clamp(int, int)") - - -def killed(mutant: Mutant, test: str = "other_tests") -> MutantResult: - return MutantResult(mutant, test, MutantState.KILLED, None, "clamp(int, int)") - - -class SurvivorBrowserTest(unittest.TestCase): - def make_dashboard(self) -> Dashboard: - dashboard = Dashboard(color=False, interactive=True) - dashboard.set_test("sample_tests", 1, 2) - dashboard.set_plan(["first", "second"]) - return dashboard - - def test_findings_panel_reports_survivors_while_the_run_goes_on(self) -> None: - dashboard = self.make_dashboard() - dashboard.record_result(survived(MUTANT), SOURCE, LINE_COVERAGE) - text = "\n".join(dashboard.render(120, 40)) - self.assertIn("Findings", text) - self.assertIn("clamp() is pseudo-tested", text) - self.assertIn("src/clamp.cpp:3 `<` → `<=` survived (not_detected)", text) - - def test_survivors_view_shows_the_selected_survivor_with_its_context(self) -> None: - dashboard = self.make_dashboard() - dashboard.record_result(survived(MUTANT), SOURCE, LINE_COVERAGE) - dashboard.press("tab") - self.assertEqual(dashboard.view, SURVIVORS_VIEW) - text = "\n".join(dashboard.render(120, 40)) - self.assertIn("Survivors (1)", text) - self.assertRegex(text, r"▶ src/clamp.cpp:3:11 \[ror\] `<` → `<=`") - self.assertIn(f"id {MUTANT.id}", text) - self.assertIn("function clamp(int, int)", text) - self.assertIn("line 3 executed 12 times by sample_tests; 2/2 branches taken", text) - self.assertIn("survived not_detected under sample_tests", text) - self.assertIn("add a test where `<` and `<=` give different results", text) - self.assertIn("+ 3 │ if (a <= b)", text) - self.assertRegex(text, r"`<` → `<=` +not_detected") - - def test_survivors_view_shows_the_score_of_each_file(self) -> None: - dashboard = self.make_dashboard() - dashboard.record_result(survived(MUTANT), SOURCE, LINE_COVERAGE) - dashboard.record_result(killed(SECOND_MUTANT), SOURCE, LINE_COVERAGE) - dashboard.press("tab") - text = "\n".join(dashboard.render(120, 40)) - self.assertIn("─ Files ─", text) - self.assertRegex(text, r"src/clamp.cpp +50.0% +█{10}░{10} +2 mutants +1 survived") - - def test_arrow_keys_move_the_selection_and_stay_in_range(self) -> None: - dashboard = self.make_dashboard() - dashboard.record_result(survived(MUTANT), SOURCE, LINE_COVERAGE) - dashboard.record_result(survived(SECOND_MUTANT), SOURCE, LINE_COVERAGE) - dashboard.press("tab") - for key in ("down", "down", "down"): - dashboard.press(key) - text = "\n".join(dashboard.render(120, 40)) - self.assertRegex(text, r"▶ src/clamp.cpp:3:11 \[ror\] `<` → `>=`") - self.assertIn(f"id {SECOND_MUTANT.id}", text) - for key in ("up", "up", "up"): - dashboard.press(key) - self.assertIn(f"id {MUTANT.id}", "\n".join(dashboard.render(120, 40))) - - def test_execution_counts_add_up_over_every_test_that_ran_the_line(self) -> None: - dashboard = self.make_dashboard() - dashboard.record_result(survived(MUTANT), SOURCE, LINE_COVERAGE) - dashboard.record_result(survived(MUTANT, "other_tests"), SOURCE, LineCoverage(3, 2, 1)) - dashboard.press("tab") - text = "\n".join(dashboard.render(120, 40)) - self.assertIn("line 3 executed 15 times: sample_tests 12, other_tests 3; 2/2 branches taken", text) - - def test_a_mutant_killed_by_a_later_test_leaves_the_survivors(self) -> None: - dashboard = self.make_dashboard() - dashboard.record_result(survived(MUTANT), SOURCE, LINE_COVERAGE) - dashboard.record_result(killed(MUTANT), SOURCE, LINE_COVERAGE) - dashboard.press("tab") - text = "\n".join(dashboard.render(120, 40)) - self.assertIn("Survivors (0)", text) - self.assertIn("no surviving mutants", text) - - def test_totals_count_a_mutant_once_over_every_test(self) -> None: - dashboard = self.make_dashboard() - dashboard.record_result(killed(MUTANT, "sample_tests"), SOURCE, LINE_COVERAGE) - dashboard.record_result(survived(MUTANT, "other_tests"), SOURCE, LINE_COVERAGE) - dashboard.record_result(survived(SECOND_MUTANT), SOURCE, LINE_COVERAGE) - tally = dashboard.render(120, 40)[1] - self.assertIn("score 50.0% killed 1 survived 1", tally) - - def test_starting_a_test_drops_the_previous_tests_queue(self) -> None: - dashboard = self.make_dashboard() - dashboard.start_mutant(1, SOURCE, MUTANT) - dashboard.set_test("other_tests", 2, 2) - title = dashboard.render(120, 40)[0] - self.assertIn("Test 2/2: other_tests", title) - self.assertNotIn("mutant 2/2", title) - - def test_survivor_details_are_aligned_and_the_source_fills_the_panel(self) -> None: - source = "".join(f"int line{number} = {number};\n" for number in range(1, 40)) - mutant = Mutant("src/many.cpp", 20, 14, source.index("20;"), "crp", "20", "21") - dashboard = self.make_dashboard() - dashboard.record_result(survived(mutant), source, LINE_COVERAGE) - dashboard.press("tab") - text = "\n".join(dashboard.render(120, 40)) - self.assertIn(f"id {mutant.id}", text) - self.assertIn(" 10 │ int line10 = 10;", text) - self.assertIn(" 30 │ int line30 = 30;", text) - - def test_tab_returns_to_the_run_view(self) -> None: - dashboard = self.make_dashboard() - dashboard.press("tab") - dashboard.press("tab") - self.assertEqual(dashboard.view, RUN_VIEW) - - def test_finishing_the_run_opens_the_survivors_and_tells_how_to_quit(self) -> None: - dashboard = self.make_dashboard() - dashboard.record_result(survived(MUTANT), SOURCE, LINE_COVERAGE) - dashboard.finish_run() - frame = dashboard.render(120, 40) - self.assertEqual(dashboard.view, SURVIVORS_VIEW) - self.assertIn("q quit", frame[0]) - - def test_a_narrow_title_keeps_the_quit_key(self) -> None: - dashboard = self.make_dashboard() - dashboard.finish_run() - title = dashboard.render(40, 40)[0] - self.assertTrue(title.endswith("q quit"), title) - self.assertIn("…", title) - - def test_a_narrow_tally_keeps_the_score_and_cuts_the_counts(self) -> None: - dashboard = self.make_dashboard() - dashboard.record_result(killed(MUTANT, "sample_tests"), SOURCE, LINE_COVERAGE) - dashboard.record_result(survived(SECOND_MUTANT), SOURCE, LINE_COVERAGE) - tally = dashboard.render(56, 40)[1] - self.assertIn("score 50.0%", tally) - self.assertNotIn("survived 1", tally) - - def test_the_finished_run_view_lists_every_mutant_once(self) -> None: - dashboard = self.make_dashboard() - dashboard.record_result(killed(MUTANT, "sample_tests"), SOURCE, LINE_COVERAGE) - dashboard.set_test("other_tests", 2, 2) - dashboard.set_plan(["only the last test's mutant"]) - dashboard.record_result(survived(MUTANT, "other_tests"), SOURCE, LINE_COVERAGE) - dashboard.record_result(survived(SECOND_MUTANT, "other_tests"), SOURCE, LINE_COVERAGE) - dashboard.finish_run() - dashboard.press("tab") - text = "\n".join(dashboard.render(120, 40)) - self.assertNotIn("only the last test's mutant", text) - self.assertRegex(text, r"killed +src/clamp.cpp:3:11 \[ror\] `<` → `<=`") - self.assertRegex(text, r"survived +src/clamp.cpp:3:11 \[ror\] `<` → `>=`") - self.assertIn("all tests 2/2", text) - - def test_key_hints_are_only_shown_when_keys_are_read(self) -> None: - self.assertNotIn("tab", Dashboard(color=False).render(120, 40)[0]) - self.assertIn("tab survivors", self.make_dashboard().render(120, 40)[0]) - - def test_survivors_view_fits_tiny_terminals(self) -> None: - dashboard = self.make_dashboard() - dashboard.record_result(survived(MUTANT), SOURCE, LINE_COVERAGE) - dashboard.press("tab") - for columns, lines in ((10, 3), (1, 1), (80, 8)): - with self.subTest(size=(columns, lines)): - frame = dashboard.render(columns, lines) - self.assertEqual(len(frame), lines) - self.assertTrue(all(len(line) == columns for line in frame)) - - -class LiveConsoleTest(unittest.TestCase): - def setUp(self) -> None: - environment = mock.patch.dict(os.environ, {"NO_COLOR": "1"}, clear=True) - environment.start() - self.addCleanup(environment.stop) - - def test_live_mode_on_a_terminal_uses_the_dashboard(self) -> None: - self.assertIsInstance(create_console(TerminalBuffer(), io.StringIO(), UiMode.LIVE), LiveConsole) - - def test_live_mode_off_a_terminal_falls_back_to_plain_lines(self) -> None: - out = io.StringIO() - console = create_console(out, io.StringIO(), UiMode.LIVE) - self.assertNotIsInstance(console, LiveConsole) - console.step("working tree is clean") - self.assertEqual(out.getvalue(), " ✓ working tree is clean\n") - - def test_draws_on_the_alternate_screen_and_replays_the_transcript_when_finished(self) -> None: - out = TerminalBuffer() - err = io.StringIO() - console = LiveConsole(out, err, terminal(60, 20), no_keys) - console.heading("Preparing") - console.plan([MUTANT]) - console.mutant_started(0, SOURCE, MUTANT) - console.build_started("cmake --build build") - console.build_output("[1/1] Linking CXX executable sample_tests") - console.warn("test t fails without mutation") - self.assertTrue(out.getvalue().startswith(ALTERNATE_SCREEN)) - self.assertIn("Linking CXX executable sample_tests", out.getvalue()) - self.assertEqual(err.getvalue(), "") - console.finish() - console.finish() - screen, _, transcript = out.getvalue().rpartition(MAIN_SCREEN) - self.assertEqual(out.getvalue().count(MAIN_SCREEN), 1) - self.assertEqual(transcript, "\n▸ Preparing\n") - self.assertEqual(err.getvalue(), "strain: warning: test t fails without mutation\n") - - def test_run_finished_keeps_the_browser_open_until_q_is_pressed(self) -> None: - keys = FakeKeys() - console = LiveConsole(TerminalBuffer(), io.StringIO(), terminal(60, 20), lambda on_key: keys) - console.mutant_evaluated(survived(MUTANT), SOURCE, LINE_COVERAGE) - waiter = threading.Thread(target=console.run_finished) - waiter.start() - console.key_pressed("tab") - waiter.join(0.2) - self.assertTrue(waiter.is_alive()) - self.assertIn("Survivors (1)", console.out.getvalue()) - console.key_pressed("q") - waiter.join(2) - self.assertFalse(waiter.is_alive()) - console.finish() - self.assertTrue(keys.closed) - - def test_run_finished_returns_at_once_without_a_keyboard(self) -> None: - console = LiveConsole(TerminalBuffer(), io.StringIO(), terminal(60, 20), no_keys) - console.run_finished() - console.finish() - - -if __name__ == "__main__": - unittest.main() diff --git a/tests/test_keys.py b/tests/test_keys.py deleted file mode 100644 index 60eb00c..0000000 --- a/tests/test_keys.py +++ /dev/null @@ -1,24 +0,0 @@ -import io -import unittest - -from strain.keys import decode_keys, open_key_reader - - -class DecodeKeysTest(unittest.TestCase): - def test_arrow_keys_and_letters(self) -> None: - self.assertEqual(decode_keys("\033[Aj\033[Bk\tq"), ["up", "down", "down", "up", "tab", "q"]) - - def test_application_mode_arrows_and_paging(self) -> None: - self.assertEqual(decode_keys("\033OA\033OB\033[5~\033[6~"), ["up", "down", "pageup", "pagedown"]) - - def test_unknown_sequences_are_dropped(self) -> None: - self.assertEqual(decode_keys("\033[1;5Cx"), ["x"]) - - -class OpenKeyReaderTest(unittest.TestCase): - def test_no_reader_without_a_terminal(self) -> None: - self.assertIsNone(open_key_reader(io.StringIO(), lambda key: None)) - - -if __name__ == "__main__": - unittest.main() diff --git a/tests/test_live_state.py b/tests/test_live_state.py new file mode 100644 index 0000000..cc8da63 --- /dev/null +++ b/tests/test_live_state.py @@ -0,0 +1,179 @@ +import unittest + +from strain.coverage import LineCoverage +from strain.live_state import ( + PENDING, + RUNNING, + LiveState, + clock_time, + execution_summary, + status_phase, +) +from strain.mutators import Mutant +from strain.results import MutantResult, MutantState, SurvivorKind +from strain.safety_nets import Level, RelaxedSafetyNet, SafetyNet + +SOURCE = "int clamp(int a, int b)\n{\n if (a < b)\n return a;\n return b;\n}\n" +MUTANT = Mutant("src/clamp.cpp", 3, 11, SOURCE.index("<"), "ror", "<", "<=") +SECOND_MUTANT = Mutant("src/clamp.cpp", 3, 11, SOURCE.index("<"), "ror", "<", ">=") +LINE_COVERAGE = LineCoverage(count=12, branches=2, branches_taken=2) + + +def survived(mutant: Mutant, test: str = "sample_tests") -> MutantResult: + return MutantResult(mutant, test, MutantState.SURVIVED, SurvivorKind.NOT_DETECTED, "clamp(int, int)") + + +def killed(mutant: Mutant, test: str = "other_tests") -> MutantResult: + return MutantResult(mutant, test, MutantState.KILLED, None, "clamp(int, int)") + + +class FakeClock: + def __init__(self) -> None: + self.now = 100.0 + + def __call__(self) -> float: + return self.now + + +class HelpersTest(unittest.TestCase): + def test_status_phase_is_the_activity_after_the_mutant(self) -> None: + self.assertEqual(status_phase("[3/6] src/a.cpp:7:13 [ror] `<` → `>=` building…"), "building") + self.assertEqual( + status_phase("[3/6] src/a.cpp:7:13 [ror] `<` → `>=` running sample_tests…"), "running sample_tests" + ) + self.assertEqual( + status_phase("running sample_tests without mutation…"), "running sample_tests without mutation" + ) + self.assertEqual(status_phase("✓ working tree is clean"), "") + + def test_clock_time(self) -> None: + self.assertEqual(clock_time(5), "0:05") + self.assertEqual(clock_time(125), "2:05") + self.assertEqual(clock_time(3725), "1:02:05") + + def test_execution_counts_add_up_over_every_test_that_ran_the_line(self) -> None: + executions = {"sample_tests": LINE_COVERAGE, "other_tests": LineCoverage(3, 2, 1)} + self.assertEqual( + execution_summary(3, executions), + "line 3 executed 15 times: sample_tests 12, other_tests 3; 2/2 branches taken", + ) + self.assertEqual(execution_summary(3, {}), "no coverage data for line 3") + + +class LiveStateTest(unittest.TestCase): + def setUp(self) -> None: + self.clock = FakeClock() + self.state = LiveState(self.clock) + self.state.set_test("sample_tests", 1, 2) + self.state.plan([MUTANT, SECOND_MUTANT]) + + def test_queue_tracks_each_mutant_and_how_long_it_took(self) -> None: + self.state.start_mutant(0, SOURCE, MUTANT) + self.assertEqual([entry.state for entry in self.state.queue], [RUNNING, PENDING]) + self.clock.now += 2.5 + self.state.finish_mutant(0, MutantState.KILLED) + self.assertEqual(self.state.queue[0].duration, 2.5) + self.assertEqual(self.state.durations[MUTANT.id], 2.5) + self.assertEqual(self.state.done_count(), 1) + self.assertEqual(self.state.remaining_seconds(), 2.5) + self.assertEqual(self.state.sources["src/clamp.cpp"], SOURCE) + + def test_totals_count_a_mutant_once_over_every_test(self) -> None: + self.state.record_result(killed(MUTANT, "sample_tests"), SOURCE, LINE_COVERAGE) + self.state.record_result(survived(MUTANT, "other_tests"), SOURCE, LINE_COVERAGE) + self.state.record_result(survived(SECOND_MUTANT), SOURCE, LINE_COVERAGE) + self.assertEqual(self.state.counts(), {MutantState.KILLED: 1, MutantState.SURVIVED: 1}) + self.assertEqual(self.state.score(), 0.5) + self.assertEqual([entry.mutant for entry in self.state.survivors()], [SECOND_MUTANT]) + + def test_a_mutant_killed_by_a_later_test_leaves_the_survivors(self) -> None: + self.state.record_result(survived(MUTANT), SOURCE, LINE_COVERAGE) + self.assertEqual(len(self.state.survivors()), 1) + self.state.record_result(killed(MUTANT), SOURCE, LINE_COVERAGE) + self.assertEqual(self.state.survivors(), []) + self.assertEqual(self.state.score_history, [0.0, 1.0]) + + def test_findings_and_statistics(self) -> None: + self.state.record_result(survived(MUTANT), SOURCE, LINE_COVERAGE) + self.state.record_result(killed(SECOND_MUTANT), SOURCE, LINE_COVERAGE) + self.assertIn("src/clamp.cpp:3 `<` → `<=` survived (not_detected)", "\n".join(self.state.findings())) + [operator] = self.state.operator_tallies() + self.assertEqual((operator.name, operator.mutants, operator.detected, operator.survived), ("ror", 2, 1, 1)) + tests = {tally.name: tally for tally in self.state.test_tallies()} + self.assertEqual(tests["sample_tests"].survived, 1) + self.assertEqual(tests["other_tests"].detected, 1) + [file] = self.state.file_scores() + self.assertEqual(file.score, 0.5) + + def test_contexts_keep_each_tests_coverage_of_the_line(self) -> None: + self.state.record_result(survived(MUTANT), SOURCE, LINE_COVERAGE) + self.state.record_result(survived(MUTANT, "other_tests"), SOURCE, None) + self.assertEqual(self.state.contexts[MUTANT.id].executions, {"sample_tests": LINE_COVERAGE}) + + def test_starting_a_test_drops_the_previous_tests_queue(self) -> None: + self.state.start_mutant(1, SOURCE, SECOND_MUTANT) + plan = self.state.plan_version + self.state.set_test("other_tests", 2, 2) + self.assertEqual(self.state.queue, []) + self.assertIsNone(self.state.current) + self.assertEqual(self.state.test, "Test 2/2: other_tests") + self.assertGreater(self.state.plan_version, plan) + + def test_finishing_the_run_lists_every_mutant_once_in_its_combined_state(self) -> None: + self.state.record_result(killed(MUTANT, "sample_tests"), SOURCE, LINE_COVERAGE) + self.state.set_test("other_tests", 2, 2) + self.state.plan([SECOND_MUTANT]) + self.state.record_result(survived(MUTANT, "other_tests"), SOURCE, LINE_COVERAGE) + self.state.record_result(survived(SECOND_MUTANT, "other_tests"), SOURCE, LINE_COVERAGE) + self.clock.now += 9 + self.state.finish_run() + self.assertTrue(self.state.done) + self.assertEqual(self.state.test_name, "all tests") + self.assertEqual( + [(entry.mutant, entry.state) for entry in self.state.queue], + [(MUTANT, MutantState.KILLED), (SECOND_MUTANT, MutantState.SURVIVED)], + ) + self.assertEqual(self.state.elapsed(), 9) + self.clock.now += 50 + self.assertEqual(self.state.elapsed(), 9) + self.assertIsNone(self.state.remaining_seconds()) + + def test_build_log_counts_every_line_and_keeps_the_latest(self) -> None: + self.state.start_build("cmake --build build") + for number in range(3000): + self.state.build_output(f"line {number}\twith a tab") + self.assertEqual(self.state.build_lines_seen, 3000) + self.assertEqual(self.state.build_log[-1], "line 2999 with a tab") + self.assertEqual(len(self.state.build_log), 2000) + build = self.state.build_version + self.state.start_build("cmake --build build") + self.assertEqual(len(self.state.build_log), 0) + self.assertGreater(self.state.build_version, build) + + def test_every_change_bumps_the_version(self) -> None: + version = self.state.version + self.state.set_status("[1/2] x building…") + self.state.add_line("✓ step") + self.assertEqual(self.state.version, version + 2) + self.assertEqual(self.state.phase(), "") + self.assertEqual(self.state.log, ["✓ step"]) + + +class SafetyNetStateTest(unittest.TestCase): + def test_the_relaxed_nets_that_hide_bugs_are_told_apart(self) -> None: + state = LiveState(FakeClock()) + self.assertIsNone(state.relaxed_nets) + self.assertEqual(state.bug_hiding_nets(), []) + asan = RelaxedSafetyNet(SafetyNet.ASAN, Level.WARN, "fails in 1 translation unit(s), e.g. a.cpp") + lto = RelaxedSafetyNet( + SafetyNet.LTO, Level.IGNORE, "fails for the whole build (CMAKE_INTERPROCEDURAL_OPTIMIZATION=ON)" + ) + version = state.version + state.set_safety_nets([asan, lto], [SafetyNet.UBSAN]) + self.assertEqual(state.bug_hiding_nets(), [asan]) + self.assertEqual(state.holding_nets, [SafetyNet.UBSAN]) + self.assertGreater(state.version, version) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_safety_nets.py b/tests/test_safety_nets.py index 1eaea7d..cce00f1 100644 --- a/tests/test_safety_nets.py +++ b/tests/test_safety_nets.py @@ -8,10 +8,13 @@ Level, SafetyNet, build_wide_problems, + classify_failures, describe_build_wide, describe_missing, + fix, missing_safety_nets, resolve_levels, + why, ) FULL_FLAGS = ( @@ -164,5 +167,39 @@ def test_unknown_config_key_is_rejected(self) -> None: resolve_levels(config={"msan": "error"}, cli=None) +class ClassifyFailuresTest(unittest.TestCase): + def test_error_levels_stop_the_run_and_the_rest_are_kept_with_why_they_matter(self) -> None: + failures = { + SafetyNet.OPTIMIZATION: "fails in 1 translation unit(s), e.g. /s/a.cpp", + SafetyNet.ASAN: "fails in 2 translation unit(s), e.g. /s/b.cpp", + SafetyNet.NDEBUG: "fails in 1 translation unit(s), e.g. /s/c.cpp", + } + levels = {**DEFAULT_LEVELS, SafetyNet.ASAN: Level.WARN, SafetyNet.OPTIMIZATION: Level.IGNORE} + errors, relaxed = classify_failures(failures, levels) + self.assertEqual(len(errors), 1) + self.assertIn("safety net ndebug fails in 1 translation unit(s), e.g. /s/c.cpp; remove -DNDEBUG", errors[0]) + self.assertEqual( + [(failure.net, failure.level) for failure in relaxed], + [ + (SafetyNet.ASAN, Level.WARN), + (SafetyNet.OPTIMIZATION, Level.IGNORE), + ], + ) + asan, optimization = relaxed + self.assertTrue(asan.hides_bugs) + self.assertIs(asan.default, Level.ERROR) + self.assertFalse(optimization.hides_bugs) + self.assertEqual(asan.fix, "add -fsanitize=address") + self.assertIn("out-of-bounds", asan.why) + self.assertEqual(asan.message, describe_missing(SafetyNet.ASAN, [Path("/s/b.cpp"), Path("/s/x.cpp")])) + + def test_every_explanation_splits_into_a_fix_and_a_reason(self) -> None: + for net in SafetyNet: + with self.subTest(net=net): + self.assertTrue(fix(net)) + self.assertIn("mutant", why(net)) + self.assertEqual(f"{fix(net)}: {why(net)}", EXPLANATIONS[net]) + + if __name__ == "__main__": unittest.main() diff --git a/tests/test_tui.py b/tests/test_tui.py new file mode 100644 index 0000000..d8ad879 --- /dev/null +++ b/tests/test_tui.py @@ -0,0 +1,402 @@ +import importlib.util +import io +import threading +import time +import unittest +from unittest import mock + +from rich.console import Console as RichConsole + +from strain.coverage import LineCoverage +from strain.mutators import Mutant +from strain.results import MutantResult, MutantState, SurvivorKind + +HAS_TEXTUAL = importlib.util.find_spec("textual") is not None +if HAS_TEXTUAL: + from textual.widgets import DataTable, Static, TabbedContent + + from strain.safety_nets import Level, RelaxedSafetyNet, SafetyNet + from strain.tui import CodeView, MutationMap, RunHeader, StrainApp, TuiConsole, diff_view + +SOURCE = "int clamp(int a, int b)\n{\n if (a < b)\n return a;\n return b;\n}\n" +MUTANT = Mutant("src/clamp.cpp", 3, 11, SOURCE.index("<"), "ror", "<", "<=") +SECOND_MUTANT = Mutant("src/clamp.cpp", 3, 11, SOURCE.index("<"), "ror", "<", ">=") +LINE_COVERAGE = LineCoverage(count=12, branches=2, branches_taken=2) +SIZE = (120, 36) + + +def survived(mutant: Mutant, test: str = "sample_tests") -> MutantResult: + return MutantResult(mutant, test, MutantState.SURVIVED, SurvivorKind.NOT_DETECTED, "clamp(int, int)") + + +def killed(mutant: Mutant, test: str = "other_tests") -> MutantResult: + return MutantResult(mutant, test, MutantState.KILLED, None, "clamp(int, int)") + + +def text_of(renderable: object, width: int = 120) -> str: + console = RichConsole(width=width, file=io.StringIO(), color_system=None) + console.print(renderable) + return console.file.getvalue() # type: ignore[attr-defined] + + +def make_console() -> "TuiConsole": + return TuiConsole(io.StringIO(), io.StringIO(), headless=True) + + +def run_two_mutants(console: "TuiConsole") -> None: + """The console hooks a per-test run makes for one killed and one surviving mutant.""" + console.test_started("sample_tests", 1, 2) + console.plan([MUTANT, SECOND_MUTANT]) + console.mutant_started(0, SOURCE, MUTANT) + console.mutant_evaluated(killed(MUTANT, "sample_tests"), SOURCE, LINE_COVERAGE) + console.mutant_finished(0, MutantState.KILLED) + console.mutant_started(1, SOURCE, SECOND_MUTANT) + console.mutant_evaluated(survived(SECOND_MUTANT), SOURCE, LINE_COVERAGE) + console.mutant_finished(1, MutantState.SURVIVED) + + +@unittest.skipUnless(HAS_TEXTUAL, "the TUI needs Textual") +class DiffViewTest(unittest.TestCase): + def test_shows_the_original_and_mutated_line_in_context(self) -> None: + self.assertEqual( + text_of(diff_view(SOURCE, MUTANT, context=1)).splitlines(), + [" 2 │ {", "- 3 │ if (a < b)", "+ 3 │ if (a <= b)", " 4 │ return a;"], + ) + + def test_context_is_clipped_at_the_file_edges(self) -> None: + first = Mutant("f.cpp", 1, 1, 0, "aor", "+", "-") + self.assertEqual( + text_of(diff_view("+x\ny\n", first, context=2)).splitlines(), ["- 1 │ +x", "+ 1 │ -x", " 2 │ y"] + ) + + +@unittest.skipUnless(HAS_TEXTUAL, "the TUI needs Textual") +class StrainAppTest(unittest.IsolatedAsyncioTestCase): + async def asyncSetUp(self) -> None: + self.console = make_console() + self.app = StrainApp(self.console) + + async def settle(self, pilot: object) -> None: + self.app.refresh_from_state() + await pilot.pause() # type: ignore[attr-defined] + + async def test_run_view_shows_the_test_the_queue_and_the_running_mutant(self) -> None: + async with self.app.run_test(size=SIZE) as pilot: + self.console.test_started("sample_tests", 1, 2) + self.console.plan([MUTANT, SECOND_MUTANT]) + self.console.mutant_started(0, SOURCE, MUTANT) + self.console.status("[1/2] src/clamp.cpp:3:11 [ror] `<` → `<=` building…") + self.console.build_started("cmake --build build") + self.console.build_output("[1/2] Building CXX object clamp.cpp.o") + await self.settle(pilot) + header = text_of(self.app.query_one(RunHeader).content) + self.assertIn("Test 1/2: sample_tests", header) + self.assertIn("mutant 1/2", header) + self.assertIn("building", header) + queue = self.app.query_one("#queue", DataTable) + self.assertEqual(queue.row_count, 2) + self.assertIn("running", text_of(queue.get_row_at(0)[0])) + self.assertIn("pending", text_of(queue.get_row_at(1)[0])) + self.assertIn("+ 3 │ if (a <= b)", text_of(self.app.query_one(CodeView).content)) + + async def test_header_totals_count_each_mutant_once(self) -> None: + async with self.app.run_test(size=SIZE) as pilot: + run_two_mutants(self.console) + self.console.mutant_evaluated(survived(MUTANT, "other_tests"), SOURCE, LINE_COVERAGE) + await self.settle(pilot) + header = text_of(self.app.query_one(RunHeader).content) + self.assertIn("score 50.0%", header) + self.assertIn("killed 1", header) + self.assertIn("survived 1", header) + + async def test_survivors_view_shows_the_selected_survivor_with_its_context(self) -> None: + async with self.app.run_test(size=SIZE) as pilot: + run_two_mutants(self.console) + await self.settle(pilot) + await pilot.press("2") + await pilot.pause() + self.assertEqual(self.app.query_one(TabbedContent).active, "survivors") + self.assertEqual(self.app.query_one("#survivor-table", DataTable).row_count, 1) + detail = text_of(self.app.query_one("#survivor-detail", Static).content) + self.assertIn(f"id {SECOND_MUTANT.id}", detail) + self.assertIn("function clamp(int, int)", detail) + self.assertIn("line 3 executed 12 times by sample_tests; 2/2 branches taken", detail) + self.assertIn("survived not_detected under sample_tests", detail) + self.assertIn("add a test where `<` and `>=` give different results", detail) + self.assertIn("+ 3 │ if (a >= b)", detail) + + async def test_tabs_switch_with_numbers_tab_and_the_mouse(self) -> None: + async with self.app.run_test(size=SIZE) as pilot: + tabs = self.app.query_one(TabbedContent) + await pilot.press("4") + await pilot.pause() + self.assertEqual(tabs.active, "stats") + await pilot.press("tab") + await pilot.pause() + self.assertEqual(tabs.active, "log") + await pilot.press("shift+tab") + await pilot.pause() + await pilot.press("shift+tab") + await pilot.pause() + self.assertEqual(tabs.active, "mutants") + await pilot.click("#--content-tab-run") + await pilot.pause() + self.assertEqual(tabs.active, "run") + + async def test_switching_views_moves_focus_into_the_new_view(self) -> None: + """Focus left in a hidden view would move to a neighbour there, which switches back to that view.""" + async with self.app.run_test(size=SIZE) as pilot: + tabs = self.app.query_one(TabbedContent) + self.assertIs(self.app.focused, self.app.query_one("#queue")) + for key, view, widget in ( + ("5", "log", "#transcript"), + ("4", "stats", "#stats-files"), + ("tab", "log", "#transcript"), + ("1", "run", "#queue"), + ): + await pilot.press(key) + await pilot.pause() + self.assertEqual(tabs.active, view, key) + self.assertIs(self.app.focused, self.app.query_one(widget), key) + + async def test_a_late_activation_for_a_view_already_left_is_ignored(self) -> None: + """A view's activation message can arrive after the user has moved on, as when tab follows 4 quickly. + + Focusing into that view would switch back to it. + """ + async with self.app.run_test(size=SIZE) as pilot: + tabs = self.app.query_one(TabbedContent) + await pilot.press("5") + await pilot.pause() + tabs.post_message(TabbedContent.TabActivated(tabs, tabs.get_tab("stats"))) + await pilot.pause() + self.assertEqual(tabs.active, "log") + self.assertIs(self.app.focused, self.app.query_one("#transcript")) + + async def test_mutants_view_filters_by_state(self) -> None: + async with self.app.run_test(size=SIZE) as pilot: + run_two_mutants(self.console) + await self.settle(pilot) + await pilot.press("3") + await pilot.pause() + table = self.app.query_one("#mutant-table", DataTable) + self.assertEqual(table.row_count, 2) + await pilot.click("#filter-survived") + await pilot.pause() + self.assertEqual(table.row_count, 1) + self.assertIn(SECOND_MUTANT.id, text_of(self.app.query_one("#mutant-detail", Static).content)) + await pilot.press("f") + await pilot.pause() + self.assertEqual(self.app.state_filter, None) + + async def test_selecting_a_queue_entry_opens_it_in_the_mutant_browser(self) -> None: + async with self.app.run_test(size=SIZE) as pilot: + run_two_mutants(self.console) + await self.settle(pilot) + await pilot.press("up") + await pilot.pause() + self.assertFalse(self.app.following) + await pilot.press("enter") + await pilot.pause() + self.assertEqual(self.app.query_one(TabbedContent).active, "mutants") + self.assertIn(f"id {MUTANT.id}", text_of(self.app.query_one("#mutant-detail", Static).content)) + + async def test_clicking_the_mutation_map_shows_that_mutant_in_the_queue(self) -> None: + async with self.app.run_test(size=SIZE) as pilot: + run_two_mutants(self.console) + await self.settle(pilot) + await pilot.press("4") + await pilot.click(MutationMap, offset=(0, 0)) + await pilot.pause() + self.assertEqual(self.app.query_one(TabbedContent).active, "run") + self.assertEqual(self.app.query_one("#queue", DataTable).cursor_row, 0) + self.assertFalse(self.app.following) + await pilot.press("f") + self.assertTrue(self.app.following) + + async def test_p_pauses_the_run_before_the_next_mutant(self) -> None: + async with self.app.run_test(size=SIZE) as pilot: + await pilot.press("p") + self.assertTrue(self.console.paused) + self.assertFalse(self.console.resume.is_set()) + await pilot.press("p") + self.assertTrue(self.console.resume.is_set()) + + async def test_q_quits_only_once_the_run_has_finished(self) -> None: + async with self.app.run_test(size=SIZE) as pilot: + with mock.patch.object(self.app, "exit") as exit_app: + await pilot.press("q") + exit_app.assert_not_called() + self.console.run_finished() + await pilot.press("q") + exit_app.assert_called_once() + + async def test_ctrl_c_asks_the_runner_to_stop(self) -> None: + async with self.app.run_test(size=SIZE) as pilot: + self.console.worker = threading.Thread(target=time.sleep, args=(5,), daemon=True) + self.console.start_worker() + await pilot.press("ctrl+c") + self.assertTrue(self.console.stop_requested.is_set()) + self.assertTrue(self.console.live.stopping) + + async def test_finishing_the_run_opens_with_every_mutant_once(self) -> None: + async with self.app.run_test(size=SIZE) as pilot: + run_two_mutants(self.console) + self.console.test_started("other_tests", 2, 2) + self.console.plan([MUTANT]) + self.console.run_finished() + await self.settle(pilot) + self.assertEqual(self.app.query_one("#queue", DataTable).row_count, 2) + header = text_of(self.app.query_one(RunHeader).content) + self.assertIn("all tests", header) + self.assertIn("finished", header) + + async def test_small_terminals_still_render(self) -> None: + async with self.app.run_test(size=(40, 12)) as pilot: + run_two_mutants(self.console) + await self.settle(pilot) + tabs = self.app.query_one(TabbedContent) + for key, view in zip("12345", ("run", "survivors", "mutants", "stats", "log"), strict=True): + await pilot.press(key) + await pilot.pause() + self.assertEqual(tabs.active, view) + + +@unittest.skipUnless(HAS_TEXTUAL, "the TUI needs Textual") +class TuiConsoleTest(unittest.TestCase): + def test_stopping_raises_keyboard_interrupt_in_the_runner_once(self) -> None: + console = make_console() + console.status("working") + console.request_stop() + with self.assertRaises(KeyboardInterrupt): + console.build_output("[1/2] Building") + console.line("the report after the interruption") + self.assertIn("the report after the interruption", console.transcript) + + def test_a_paused_run_waits_before_the_next_mutant(self) -> None: + console = make_console() + console.plan([MUTANT]) + console.set_paused(True) + runner = threading.Thread(target=console.mutant_started, args=(0, SOURCE, MUTANT)) + runner.start() + runner.join(0.3) + self.assertTrue(runner.is_alive()) + self.assertTrue(console.live.waiting) + console.set_paused(False) + runner.join(2) + self.assertFalse(runner.is_alive()) + self.assertEqual(console.live.current, 0) + + def test_run_returns_what_the_work_returns_and_replays_the_transcript(self) -> None: + out, err = io.StringIO(), io.StringIO() + console = TuiConsole(out, err, headless=True) + + def work() -> int: + console.heading("Preparing") + console.warn("test t fails without mutation") + console.run_finished() + console.app.call_from_thread(console.app.exit) # type: ignore[union-attr] + return 3 + + self.assertEqual(console.run(work), 3) + console.finish() + console.finish() + self.assertEqual(out.getvalue(), "\n▸ Preparing\n") + self.assertEqual(err.getvalue(), "strain: warning: test t fails without mutation\n") + + def test_an_error_in_the_run_closes_the_tui_and_is_raised(self) -> None: + console = make_console() + + def work() -> int: + raise ValueError("no tests selected") + + with self.assertRaisesRegex(ValueError, "no tests selected"): + console.run(work) + + def test_stopping_the_run_ends_it_with_keyboard_interrupt(self) -> None: + console = make_console() + + def work() -> int: + while True: + console.status("running…") + time.sleep(0.01) + + threading.Timer(0.5, console.request_stop).start() + with self.assertRaises(KeyboardInterrupt): + console.run(work) + + +ASAN_OFF = "fails in 2 translation unit(s), e.g. src/clamp.cpp" +LTO_ON = "fails for the whole build (CMAKE_INTERPROCEDURAL_OPTIMIZATION=ON)" + + +@unittest.skipUnless(HAS_TEXTUAL, "the TUI needs Textual") +class SafetyNetViewTest(unittest.IsolatedAsyncioTestCase): + async def asyncSetUp(self) -> None: + self.console = make_console() + self.app = StrainApp(self.console) + + def relax(self, *failures: "RelaxedSafetyNet") -> None: + relaxed = {failure.net for failure in failures} + self.console.safety_nets_checked(list(failures), [net for net in SafetyNet if net not in relaxed]) + + async def settle(self, pilot: object) -> None: + self.app.refresh_from_state() + await pilot.pause() # type: ignore[attr-defined] + + async def test_stats_explain_each_relaxed_net_and_why_it_mattered(self) -> None: + async with self.app.run_test(size=SIZE) as pilot: + self.relax( + RelaxedSafetyNet(SafetyNet.ASAN, Level.WARN, ASAN_OFF), + RelaxedSafetyNet(SafetyNet.LTO, Level.IGNORE, LTO_ON), + ) + await self.settle(pilot) + report = text_of(self.app.query_one("#safety", Static).content) + self.assertIn("asan is off, so a mutant that breaks the code can still pass the tests", report) + self.assertIn("asan set to warn, so the run went on with a warning; by default it stops the run", report) + self.assertIn("lto set to ignore, so the run went on without a warning", report) + self.assertIn("fails in 2 translation unit(s), e.g. src/clamp.cpp", report) + self.assertIn("why without AddressSanitizer an out-of-bounds", report) + self.assertIn("fix add -fsanitize=address", report) + self.assertIn("✔ hold: ubsan", report) + self.assertEqual(self.app.query_one("#safety-box").border_title, "Safety nets: 2 off") + + async def test_nets_that_only_cost_speed_say_survivors_still_count(self) -> None: + async with self.app.run_test(size=SIZE) as pilot: + self.relax(RelaxedSafetyNet(SafetyNet.LTO, Level.WARN, LTO_ON)) + await self.settle(pilot) + report = text_of(self.app.query_one("#safety", Static).content) + self.assertIn("survivors still point at weak tests", report) + + async def test_every_net_holding_is_one_line(self) -> None: + async with self.app.run_test(size=SIZE) as pilot: + self.relax() + await self.settle(pilot) + self.assertIn( + "✔ every safety net holds: asan, ubsan", text_of(self.app.query_one("#safety", Static).content) + ) + self.assertNotIn("safety net", text_of(self.app.query_one(RunHeader).content)) + + async def test_the_header_badge_counts_relaxed_nets_and_opens_their_report(self) -> None: + async with self.app.run_test(size=SIZE) as pilot: + self.relax(RelaxedSafetyNet(SafetyNet.ASAN, Level.IGNORE, ASAN_OFF)) + await self.settle(pilot) + header = self.app.query_one(RunHeader) + self.assertIn("! 1 safety net off", text_of(header.content)) + await pilot.click(RunHeader, offset=(header.safety_columns.start, 0)) + await pilot.pause() + self.assertEqual(self.app.query_one(TabbedContent).active, "stats") + + async def test_survivors_carry_a_caution_while_a_bug_hiding_net_is_off(self) -> None: + async with self.app.run_test(size=SIZE) as pilot: + self.relax(RelaxedSafetyNet(SafetyNet.ASAN, Level.WARN, ASAN_OFF)) + run_two_mutants(self.console) + await self.settle(pilot) + await pilot.press("2") + await pilot.pause() + detail = text_of(self.app.query_one("#survivor-detail", Static).content) + self.assertIn("caution asan off: this mutant may survive only because of that", detail) + + +if __name__ == "__main__": + unittest.main() diff --git a/tools/ui_snapshot.py b/tools/ui_snapshot.py index 3bb905f..aea5e6a 100644 --- a/tools/ui_snapshot.py +++ b/tools/ui_snapshot.py @@ -1,11 +1,13 @@ """Runs `strain run --ui live` in a pseudo-terminal and saves what the screen shows as text and PNG frames. pip install -e '.[ui-snapshot]' - python tools/ui_snapshot.py tests/fixtures/sample_project --size 120x36 --keys down,tab,q + python tools/ui_snapshot.py tests/fixtures/sample_project --size 120x36 --keys 2,down,q Frames go to `ui-snapshots/` by default: `frame-NNN.txt` every `--interval` seconds, and `key-NN-.txt` after each key, each with a matching `.png` when Pillow is installed. -The keys are sent once the run has finished, which is when the title shows `finished`. +The keys are sent once the run has finished, which is when the header shows `✔ finished`, and `--delay` +seconds later. A key is a name such as `down`, `tab` or `enter`, a character such as `2` or `q`, or a mouse +event on a cell, counting from 1: `click:COLUMN:ROW`, `scrollup:COLUMN:ROW` or `scrolldown:COLUMN:ROW`. """ import argparse @@ -23,8 +25,24 @@ import pyte -KEYS = {"up": "\033[A", "down": "\033[B", "pageup": "\033[5~", "pagedown": "\033[6~", "tab": "\t"} -FINISHED_MARKER = "· finished" +KEYS = { + "up": "\033[A", + "down": "\033[B", + "left": "\033[D", + "right": "\033[C", + "pageup": "\033[5~", + "pagedown": "\033[6~", + "home": "\033[H", + "end": "\033[F", + "tab": "\t", + "shift+tab": "\033[Z", + "backtab": "\033[Z", + "enter": "\r", + "escape": "\033", +} +# Mouse keys name the cell, counting from 1: `click:COLUMN:ROW`, `scrollup:COLUMN:ROW`, `scrolldown:COLUMN:ROW`. +MOUSE_BUTTONS = {"click": 0, "scrollup": 64, "scrolldown": 65} +FINISHED_MARKER = "✔ finished" READ_SIZE = 65536 SETTLE_SECONDS = 0.5 CELL_WIDTH = 9 @@ -54,6 +72,16 @@ } +def key_sequence(key: str) -> str: + """What the terminal sends for `key`: an escape sequence, an SGR mouse report, or the key itself.""" + kind, _, cell = key.partition(":") + if kind in MOUSE_BUTTONS and cell: + column, _, row = cell.partition(":") + press = f"\033[<{MOUSE_BUTTONS[kind]};{column};{row}M" + return press + (f"\033[<0;{column};{row}m" if kind == "click" else "") + return KEYS.get(key, key) + + def parse_size(text: str) -> tuple[int, int]: columns, _, lines = text.partition("x") return int(columns), int(lines) @@ -137,7 +165,8 @@ def main(argv: Sequence[str] | None = None) -> int: parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) parser.add_argument("project", type=Path) parser.add_argument("--size", type=parse_size, default=(120, 36), help="COLUMNSxLINES (default 120x36)") - parser.add_argument("--keys", default="down,tab,q", help="keys to send once the run has finished") + parser.add_argument("--keys", default="2,down,q", help="keys to send once the run has finished") + parser.add_argument("--delay", type=float, default=0.0, help="seconds to wait after the run before the keys") parser.add_argument("--interval", type=float, default=1.0, help="seconds between frames during the run") parser.add_argument("--out", type=Path, default=Path("ui-snapshots")) parser.add_argument("--timeout", type=float, default=600.0) @@ -154,14 +183,16 @@ def main(argv: Sequence[str] | None = None) -> int: running = pump(descriptor, stream, options.interval) save_frame(screen, options.out, f"frame-{frame:03}") frame += 1 + if running and options.delay > 0: + running = pump(descriptor, stream, options.delay) keys = [key for key in options.keys.split(",") if key] sent = 0 for number, key in enumerate(keys, start=1): if not running: break - os.write(descriptor, KEYS.get(key, key).encode()) + os.write(descriptor, key_sequence(key).encode()) running = pump(descriptor, stream, SETTLE_SECONDS) - save_frame(screen, options.out, f"key-{number:02}-{key}") + save_frame(screen, options.out, f"key-{number:02}-{key.replace(':', '-')}") sent = number if running: pump(descriptor, stream, SETTLE_SECONDS)