Fix #1036: count transcript lines with C tools, never in a shell variable

[fdaviddpt]

Closes #1036

Reported and measured by @KRR-66 -- the step-by-step timing table in the issue is what pointed straight at the offset-0 path. Thank you.

What was wrong

_pt_scan_lines (scripts/post-tool-hook.sh) read everything after its resume offset into a bash variable (_chunk=$(tail -c +OFFSET file; printf X)), cut the trailing partial line off it with ## / % expansions, and counted newlines with a wc -l herestring. At offset 0 -- no sidecar yet, which is the first call of every new session and of every existing session after an upgrade -- that is the whole transcript held in bash. On the reporter's Windows bash 5.2.37: 5 MB 1.5 s, 20 MB 13.5 s, 50 MB 92 s. 247 MB and 1.33 GB transcripts hit the 600 s PostToolUse timeout, the hook died before writing the sidecar, so every later call started from 0 again.

A second, worse case turned up while writing the test: a multi-MB trailing partial line (a record still being written). ${_chunk%"$_partial"} with a 3 MB pattern did not finish in 120 s on macOS bash 5.3.

The fix

The transcript never enters a shell variable now, on either path. Only one byte at a time does:

1. warm only: tail -c +OFFSET on fd 9, one byte read with read -r -d '' -n 1 (bash reads a pipe byte by byte, so the rest stays in the pipe). It must be the newline OFFSET claims to follow, else full recount from 0 -- the same check and fallback as before.
2. wc -lc over the rest (warm: the rest of fd 9; cold: the file itself) gives N newlines and B bytes.
3. a one-byte read of the last of those B bytes (tail -c +END). Appends never change a written byte, so this is race-free against a growing file. Newline there: OFFSET = OFFSET + B, LINES += N. This is the common case.
4. only when a partial line is pending: head -n N | wc -c gives the length through the Nth newline. Cold it reads the file directly, because macOS tail -c +N streams at ~30 MB/s (27 MB in 0.9 s even to /dev/null) where head -n N file took 0.014 s.

Contract unchanged: PT_SCAN_OFFSET = byte offset just past the last \n, PT_SCAN_LINES = newline count, a partial line is never counted and is re-read next call, local LC_ALL=C keeps read -n 1 at one byte (#695). The long comment inside the function now describes this algorithm; the one above it no longer claims length/substring ops on the chunk.

Lint fixes the repo's own scanners asked for: no case statement (directory scanner shape, #898) and 10# on every count in $(( )) (#332). tests/test_locale_ranges_695.py allowlist lines shifted +68 (1011->1079, 1129->1197); the new code adds no letter range.

Forks

PID-delta measurement, macOS arm64, /bin/bash 3.2.57 and Homebrew bash 5.3.15, min of 10 (OBSERVED on macOS only):

form	forks
old: $(tail ...; printf X) + $(wc -l <<< ...)	4
<(exec tail ...)	1
$(exec wc -lc < file) / $(exec wc -lc <&9)	1
$(a | b)	3

So per scan: cold 2 (was 4), warm with new bytes 3 (was 4), warm with nothing new 2 (was 2). A pending partial line adds 3 cold / 4 warm (the 4 is the same per-stage rule, REASONED, not measured). The explicit exec inside each substitution is what saves bash 3.2 the second fork a redirect otherwise costs. The Linux strace totals come from CI's test_post_tool_total_processes_1011.py (its fixture is the cold path) -- see the CI comment below once it has run.

Tests (written first, watched fail)

tests/test_pt_scan_cold_large_transcript_1036.py runs the real hook through resolve_bash(), with the #1034 harness, so it is meant to run on windows-latest too:

• timing: 24 MB of 4 MB lines, cold, with and without a 3 MB partial record at the end; the hook's own fixed cost is measured on a 10-line transcript and subtracted. Bound 1.0 s on POSIX, 5.0 s on Windows.
• correctness vs Python ground truth: trailing newline, partial trailing line, no newline at all, one 3 MB line, a 3 MB partial line, lines around a 2 MB line; warm resume after appends (completing a partial, adding a 2 MB record, ending on a newline, and a no-op call).
• must take the warm path: a valid OFFSET with LINES inflated by 1000 survives the next call. Paired with must recount: a rewritten file and a shrunk file both discard the inflated count.

Before / after, macOS arm64 (OBSERVED), seconds over the 10-line baseline:

case	old, bash 5.3.15	old, /bin/bash 3.2.57	new, both
24 MB, ends on a newline	1.48	1.71	0.02-0.04
27 MB, ends in a 3 MB partial record	did not finish in 120 s	(not run)	0.04
correctness case with a 3 MB partial line	did not finish in 120 s		passes

Locally green (macOS, bash 5.3 and with /bin/bash 3.2 first on PATH for the new module and #1034): the new module, #913, #1017, #1034, #695, #330, #1021, the spawn tests, the #898/#332 scanners, and every test file that names post-tool-hook (1645 passed, 55 skipped -- Windows-only and strace-only cases). The #1011 strace tests skip here (no strace on macOS).

Per platform: observed vs reasoned

• macOS (bash 3.2.57, 5.3.15): OBSERVED -- correctness, timing and fork counts above.
• Linux: REASONED until CI runs -- GNU coreutils wc/tail/head, same bash code; the strace fork test will measure it.
• Windows (MSYS2 bash): REASONED until windows-latest runs the new module. Process substitution and byte-wise read -n 1 on a pipe are standard bash behaviour there too, but this is exactly the claim CI has to confirm. The reporter's real 247 MB / 1.33 GB transcripts are not reproduced in CI; at wc speed they should take seconds, not minutes -- reasoned, not measured.

Not done: a PostToolUse timeout in hooks/hooks.json (the issue's suggestion 2)

Decided against. A timeout that kills the hook mid-scan never writes the sidecar either, so it would only turn a 600 s stall on every call into an N s stall on every call -- and if N is below what a legitimate cold scan of a huge transcript on a slow disk needs, it turns a one-time cost into a permanent per-call cost that would otherwise heal after one slow call. With the scan now linear in C, the default 600 s is far from anything this code path reaches; picking a smaller number would trade a hypothetical future regression for a real risk of killing slow-but-finishing calls. Not clearly right, so not done.

🤖 Generated with Claude Code

[AI-generated]