Keep leaf nodes spread across files when pruning the candidate set

[StefanZoerner]

Refs #75

Problem

With functions included as leaf candidates (840e1f4), umoria produces 770
candidates, which crosses LEAF_REDUCTION_THRESHOLD (400). The pruning in
topo_sort.py then discards everything that is a dependency of anything
else, leaving only components nothing calls — 27 of 770, covering 11 of 46
source files. The node set is then small enough to skip LLM clustering
entirely and fall back to whole-repository mode.

The two mechanisms work against each other: the better function inclusion
gets, the more reliably the threshold is crossed and the harder the pruning
cuts.

Change

Compare file coverage before and after the pruning. If it drops below half,
discard the pruned set and cap the full candidate list instead, picking
entries round-robin across files so no file is dropped while another
contributes dozens.

Also raises the pruning log from DEBUG to INFO — previously there was no
indication at normal verbosity that a reduction had happened — and stops
rejecting identifiers that merely contain a word like invalid.

Effect on umoria

	before	after
leaf nodes	27	400
files covered by entry points	11	46
modules created	0 (fallback)	46
source files mentioned in the generated docs	12 of 77	31 of 77

Tests

pytest tests/ --ignore=tests/smoke_test_mcp.py: 48 passed, 2 failed. Both
failures are in test_gitignore_filtering.py and occur on main as well.

[anhnh2002]

Thanks for looking into this, and sorry for the slow review. I went through the change carefully and want to explain why I'm hesitant about the fallback before we go further.

The leaf nodes from get_leaf_nodes are meant to be the in-degree-0 roots of the dependency graph, i.e. the entry points nothing else depends on. They're not supposed to cover every file. The idea is that they seed the clustering step and each module agent, and the agent then explores downstream from those roots (it gets the full file for each core component plus read_code_components to pull in dependencies). So a root set that only touches 20% of files isn't a bug by itself; the other 80% is supposed to be reached through exploration.

The new fallback works against that. When coverage drops below 0.5 it stops using roots and instead ranks each file by descending in-degree, so it picks the most-depended-upon nodes first. Those are hubs and shared helpers, which is pretty much the opposite of an entry point. On a CLI-shaped repo with a few main functions as the true roots, we'd end up handing the clustering step utility code as "core components" and dropping the actual entry points. That flips which branch runs on a 50% threshold, so the same repo could get very different module trees depending on which side it lands on.

That said, I suspect you hit a real problem to motivate this. If the docs were missing large parts of a repo, my guess is the exploration side is the weak link rather than the root selection: the agent has to guess component IDs from file content, and there's no tool that lists a component's dependencies. Could you share which repo you were testing on? If that's the issue, I'd rather fix it in a way that keeps roots as the seeds, e.g.:

• Give the module agent a tool that returns the direct dependencies of a component, so exploration is reliable instead of inferred.

Happy to discuss either direction. Thanks again for digging into this.