[clemlesne]

Fixes #27.

The scanner read each ctags reply line into a 4096-byte buffer and silently skipped lines that did not fit (from zoekt 5851b63). Parse then dropped the tag with no error and no log. Because scope repeats the parent's full name, the children of a long-named type were dropped too. A real case is corretto-8 LongName.java (141,446 B): today Parse returns only the class, not its 140,000-character field.

Two commits:

1. keep ctags tags whose reply line is 4 KiB or longer
   • Lines are still read with ReadSlice into the 4096-byte buffer. Only lines that do not fit are copied into a reused buffer, so short lines cost nothing extra.
   • Lines over maxLineSize (1 MiB) are skipped, as long lines were before.
   • A long line that is not a tag, or that does not decode, is skipped as before instead of failing Parse or New. So they return no error that main would not.
   • Parse still returns a nil error. If anything was skipped, it logs one line per file through Options.Info: ctags skipping N long output lines for <file>.
   • Two smaller changes: an unmarshal error now quotes at most 200 bytes of the line, and the Debug logger now sees long lines in full (up to 1 MiB).
2. bound the memory kept from long ctags reply lines
   • Parent strings of 256 bytes or more are shared through a per-Parse map, so N children of one long parent keep one copy of its name, not N. Shorter parents are not worth a map entry.
   • Bytes kept from long lines (4096 B or more) count against a per-Parse budget of max(1 MiB, 100 × len(content)), a shared parent counted once. Once a tag does not fit, the remaining long lines of that Parse are skipped without being decoded. Short lines are never counted, so Parse returns every tag that main returns, plus the long ones.

No API change. The limits are unexported constants.

Why not ReadBytes with no limit, as the issue suggested? The 4 KiB skip was also the only bound on memory. A 1 MB Python file with one class whose name is 1 MB long and 200 inner classes makes ctags print 401 MB of reply. Every scope is a different string, so sharing Parent cannot help, and Zoekt would hold those strings until the whole shard is written. With the budget (about 100.6 MB here), Parse keeps 100.9 MB of heap. The bound counts reply bytes, so heap can be somewhat higher: about 1.2× on the JS file below.

Retained heap after Parse on recorded ctags replies, main → this PR:

• s5.go, a 232 KB file with one struct whose name has about 3.5 KB and 20,000 fields (all lines under 4 KiB): 85.5 MB → 3.5 MB, thanks to the shared Parent.
• enum_ceiling.c, a 63 KB C file with one enum whose name has about 3.3 KB and 20,000 members (all reply lines under 4 KiB): 72.2 MB → 3.1 MB.
• A typical 10k-tag Go file: 1.79 MB on both.
• The largest increase in this set, a JS file with 6,000 nested scopes: 1.79 MB → 7.38 MB (budget 4.69 MB).

On ordinary files (all lines under 4 KiB) the Go side of Parse shows no measurable difference in time or allocations from main (replayed ctags replies, benchstat, n=10).

Two things are left out on purpose. Happy to change either:

1. Main already lets short lines amplify output: single-parent cascades reach ~300–1,165×, and dense one-line generated files 150–166×, with every line under 4 KiB. Counting all lines against the budget would close that, but it would also drop tags that main keeps today: for example a synthetic one-line C enum is 166×, over the 100× ratio. I can add it as a follow-up if you want that trade-off.
2. The constants could become Options fields if Zoekt wants to tune them.

Test Plan:

• TestScanner now sets a small line cap: lines longer than the buffer are kept, lines over the cap are skipped. New TestScannerBufferBoundary pins 4095/4096 B, and FuzzScanner checks that lines come back in order and intact, every line under the cap comes back, and none over it (30 s locally, no failure).
• New Parse tests use assertions, not goldens, so they do not depend on the ctags version: the issue's Go func and C function, a long Python class and Go struct with their children (checking Parent), 32 KiB and 100 KiB names, the 1 MiB cap, a shared Parent on long and on short lines, a lowered budget that drops long lines but keeps every short one, and the stop once a long tag does not fit. Others use a fake ctags (fake_test.go) or a small script: long lines that are not tags are skipped (in Parse and in New), and Parse returns an error instead of hanging when ctags dies in the middle of a long line.
• All new tests fail on main, except the two that guard behavior main already has: an error instead of a hang when ctags dies mid-line, and long lines while New starts.
• Each commit passes go vet, go test -race ./... on ctags 6.2.1 and go test ./... on 6.0.0.2783f009 (the CI build, darwin). ./gen.sh leaves no diff.
• zoekt c0b97d2 with a replace to this branch: go test ./internal/ctags/... ./index/... ./cmd/zoekt-index/... passes. zoekt-index -require_ctags on LongName.java and the issue's sample.go: sym: finds both long symbols (0 before), with no increase in max RSS or wall time.

🤖 Generated with Claude Code

[keegancsmith]

LGTM. I was at first concerned a bit about returning larger symbols and the impact of pathological files. However, the impact isn't that bad + I really like the fix around sharing parent strings.

I did push a commit to lower the heuristic limit though from 100 * len to 10 * len just to be on the safe side (eg think about 1mb files).