Skip to content

Incremental merge: stray output misattributed to non-dispatched files silently deletes their prior graph contributions #2926

Description

@pureoption

Summary

During incremental extraction, LLM output whose source_file is misattributed to a file that was not dispatched silently deletes that file's entire prior contribution from graph.json. The semantic cache is protected against this; the graph build is not.

Environment

graphifyy 0.9.16, Python 3.12 (WSL2 Ubuntu 24.04), gemini backend, markdown-only corpus (26 docs).

Mechanism

  1. Extracting file A, the model also emits a few stray nodes/edges attributed to other files (observed: one file's extraction produced strays attributed to 7 other files, plus nonexistent forward-reference paths like src/foo.ts).
  2. _save_semantic_cache(..., allowed_source_files=uncached_paths) correctly discards them from the cache (semantic cache skipped out-of-scope source_file warning) — the cache stays clean.
  3. But in cli.py, the same unfiltered fresh result is extended into sem_result and reaches build_merge().
  4. build_merge() derives its replace-set from the source_files present in the new chunks — so a 2-node stray fragment for file B replaces B's entire prior contribution (observed: a file whose cache entry held 20 nodes was reduced to 1 node in graph.json).
  5. B's manifest entry still says unchanged, so B is never re-dispatched and its clean cache entry is never re-applied. The loss is permanent until a full rebuild.
  6. Strays attributed to nonexistent files become phantom nodes carried forward forever — never in any changed set, never matched by the delete pass (observed: 24 phantoms accumulated).

Reproduction sketch

Corpus of N cross-referencing markdown docs. Force one file to re-extract (touch content). If the model attributes any output to a second, unchanged file, that file's graph contribution is replaced by the fragment. Diff unique node ids in cache/semantic/*.json against graph.json to observe the divergence.

Suggested fix

Apply the same allowed_source_files scope filter to fresh before extending sem_result (mirror of the cache-write guard). We run 0.9.16 with exactly that local patch: a run that previously clobbered 7 files holds every node, and the filter logs 9–33 dropped strays per routine run on our corpus — misattribution is common enough that the graph-side guard matters.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions