Git hooks fire for every worktree with no concurrency cap — 36 detached indexers, ~66GB RAM, host unusable

[itsandyking]

Summary

install_hooks writes the auto-reindex hook into the main repository's .git/hooks/. Git worktrees share that directory through the common dir, so every git worktree add, commit and merge inside any worktree fires the hook and detaches another unbounded cce index.

For anyone running AI agents in git worktrees this scales with agent parallelism. On my machine it reached 36 concurrent indexers at ~1.85 GB RSS each (~66 GB on a 96 GB box), load average 426, and took down an unrelated production service on the same host — its origin stopped answering and Cloudflare served a connection error to users. The web app itself was healthy; the machine was simply out of memory.

This is distinct from #139 (68 cce serve processes). Same class of failure, different process and different trigger — and, as noted below, the 0.4.26 resource governor does not cover this path.

Root cause

src/context_engine/indexer/git_hooks.py:

HOOK_NAMES = ["post-commit", "post-checkout", "post-merge"]

def _hook_script() -> str:
    bin_path = shlex.quote(_resolve_cce_binary())
    return f"""{HOOK_MARKER}
{bin_path} index >/dev/null 2>&1 &
"""

def install_hooks(project_dir: str) -> list[str]:
    hooks_dir = Path(project_dir) / ".git" / "hooks"

Two properties combine badly:

1. .git/hooks is shared with every worktree. In a worktree .git is a file pointing at <main>/.git/worktrees/<name>/, and hook lookup resolves to the common dir — so a hook installed once in the main checkout runs for all N worktrees. Improve git worktree support: hook installer + storage path collision #48 observed the inverse (hooks don't install from inside a worktree); the consequence that main-repo hooks fire for every worktree seems not to have been considered.
2. The spawn is unbounded. & detaches it, so the process is orphaned to PID 1, git returns instantly, and nothing serialises, nices, or caps the result. There is no lock and no awareness of sibling indexers.

Creating a worktree fires post-checkout; the agent's first commit fires post-commit; a merge fires post-merge. I observed three separate indexers against a single worktree from one agent's activity.

Observed

• 35–36 concurrent cce index, all PPID 1, oldest ~3h49m
• ~1.85 GB RSS each, ≈66 GB of 96 GB
• Load average 426 (15-min average sustained >200 for hours)
• 8 concurrent agents, each cutting its own worktree under /private/tmp/w-*
• Host: macOS (Apple silicon), CCE 0.4.25, embedding backend ollama
• Recovery required killing all of them by hand; memory returned to 83% free immediately

Why the 0.4.26 resource governor (#142) does not cover this

I upgraded to 0.4.26 expecting it to help, and on reading it, it does not address this path:

Mechanism	Why it misses
Per-project index lock (fcntl.flock)	Keyed per project, and every worktree resolves to a distinct project slug — so the lock never engages between worktrees. It dedupes same-project duplicates only.
Linux PSI memory-pressure deferral	Reads /proc/pressure/memory; explicitly a no-op off Linux. Nothing on macOS, which is where the agent-worktree workflow is common.
Idle auto-shutdown, ORT thread caps	Both apply to cce serve. This failure is standalone cce index invoked from a git hook.

Suggested fixes

Any one of these would have prevented it:

1. A global (not per-project) concurrency cap for hook-triggered indexing — the lock in fix(serve): resource governor for multi-instance system freezes #142 is the right mechanism at the wrong scope. Hook-spawned indexers should contend machine-wide.
2. Detect worktrees in install_hooks / the hook itself and skip, or coalesce to the main checkout. git rev-parse --git-common-dir distinguishes a worktree from a main checkout in one call.
3. nice the hook-spawned process. It's background maintenance; it should never compete with the user's foreground work.
4. Skip ephemeral worktrees. Agent harnesses create throwaway trees (/private/tmp/w-*, <repo>/.claude/worktrees/*) that are deleted minutes later — indexing them is pure cost, since the index dies with the tree.

Related: #48 (hooks and worktrees), #52 (slug collision), #139/#142 (multi-instance resource exhaustion).

Workaround for others hitting this

Repoint the hooks at a wrapper that holds an fcntl semaphore (macOS has no flock(1)), caps concurrency machine-wide, dedupes per directory, runs at nice 10, and skips ephemeral worktree paths. After that change the same workload produces at most 2 concurrent indexers and the machine stays responsive.

Worth noting for anyone applying a local fix: cce init rewrites the hook files, so re-running it silently discards the workaround.

[itsandyking]

Following up with a source-level check, since the obvious first response to this is "isn't that fixed by the 0.4.26 resource governor (#142)?"

It isn't, and the code is explicit about why. resource_governor is imported exactly once in the whole package, inside cce serve:

# Force single-process embedding inside `cce serve` unless the user
# explicitly overrode it. ...
#   bulk `cce index` run from a separate shell still gets the multiprocess path.
os.environ.setdefault("CCE_EMBED_PARALLEL", "0")

# Cap ONNX Runtime threads per process (#139). With N concurrent
# cce-serve processes, uncapped threads (default = cpu_count) ...
from context_engine.resource_governor import (
    cap_ort_threads, IdleTracker, ProjectIndexLock, is_memory_pressured,
)
cap_ort_threads(max_threads=getattr(config, "serve_max_ort_threads", None))

The index command contains no references to cap_ort_threads, ProjectIndexLock, or is_memory_pressured. So the hook-spawned indexer this issue is about gets no thread cap, no lock, no memory-pressure check — and, by that first comment, deliberately the multiprocess embedding path, i.e. the heavier one.

Two different failure modes, easy to conflate because both end in an unusable machine:

	#139	this issue
Process	cce serve	cce index, from the git hooks cce init installs
Bounded by	ORT caps, per-project lock, idle shutdown	nothing
Embedding	forced single-process	multiprocess
Scales with	projects × sessions	worktree events — every worktree add, commit, merge

The comment "bulk cce index run from a separate shell" is the crux: it assumes a human typing in a terminal. The hooks make it fire automatically and unattended, and because worktrees share .git/hooks through the common dir, one hook install fans out across every worktree.

ProjectIndexLock also wouldn't help if it were simply extended to cce index, because it keys on storage_base and every worktree resolves to its own slug — N worktrees means N locks that never contend. That's the part that makes this orthogonal to #142 rather than a subset of it.

For anyone hitting this before an upstream fix, a wrapper holding an fcntl semaphore across all projects (not per-project), plus nice, plus skipping ephemeral worktree paths, took the same workload from 36 concurrent indexers to a cap of 2. macOS has no flock(1), so the lock has to come from fcntl in Python or similar.

[rajkumarsakthivel]

Good analysis throughout, especially the distinction from #139/#142 and explaining exactly why the per-project lock doesn't help here.

You're right that the resource governor doesn't cover this path. The ProjectIndexLock is keyed per storage slug, and every worktree resolves to its own slug, so N worktrees means N locks that never contend. The hook script itself had no concurrency awareness at all.

Fix is up in #160. Three changes:

1. Global machine-wide lock via mkdir (POSIX atomic, no flock dependency, works on macOS). One hook-triggered indexer at a time across all projects and worktrees. Stale locks are reclaimed when the owner PID is dead.

2. Skip ephemeral worktree paths (/tmp/*, .claude/worktrees/*). Indexing throwaway trees is pure cost.

3. nice -n 10 on the spawned process.

Also fixed the install side: install_hooks() now resolves through git rev-parse --git-common-dir so it works from inside a worktree, and cce init replaces old hook blocks instead of silently skipping when the marker is already present (so existing installs pick up the cap on upgrade).

Re: your point about cce init overwriting custom wrappers. The new _install_single_hook only replaces content between the CCE markers. User lines outside the block are preserved.