Add opt-in Japanese query tokenization - #2958
Conversation
![graphify-labs[bot]](https://avatars.githubusercontent.com/in/4356270?s=60&v=4){kind=small}
There was a problem hiding this comment.
Graphify reviewed this change.
Looks safe to merge — no coupling regressions and no blocking issues, checked against the code graph (not a self-assessment).
Formal verification. 3 change(s) tested, no difference found (not proven).
Graphify review — findings
Adds an opt-in janome_content query tokenizer for Japanese content-word segmentation via the optional japanese/Janome extra, wiring it through _query_terms/_query_graph_text, the graphify query --tokenizer CLI flag, and the MCP query_graph tool schema. Raises clear ImportError/ValueError when Janome is missing or an unsupported tokenizer name is passed, with the CLI reporting these as errors and exiting non-zero. Defaults everywhere remain baseline, preserving existing behavior.
No blocking issues surfaced. 4 lower-confidence candidates did not survive cross-model review.
Analysis details — impact, health, verification
Impact & health
Graphify review
Impact — 755 functions depend on the 497 functions this change touches.
Health — this change adds coupling hotspots:
- new:
dispatch_command()— 2 callers, 119 callees - new:
_make_graph()— 34 callers, 6 callees - new:
_query_graph_text()— 20 callers, 9 callees - new:
_query_terms()— 24 callers, 4 callees - new:
_score_query()— 15 callers, 5 callees - new:
run_benchmark()— 16 callers, 3 callees - new:
_stale_graph_sources()— 7 callers, 6 callees - new:
_build_server()— 2 callers, 16 callees - …and 12 more — each is listed as a finding
Verification — 755 functions in the blast radius were not formally verified this run (proofs are advisory here).
Gate & verification
graphify gate
PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.
Advisory (not blocking):
- verification_scope: 755 function(s) in the blast radius were not formally verified this run
Formal verification
Could not verify: Could not verify dispatch\_command.
The verifier did not have enough to check dispatch\_command, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: not verifiable: all 23 sampled inputs raised on both versions — the function never executed, so 'no divergence' would be vacuous (mostly SystemExit — names the real obstacle, not a sampling gap)
No difference found (not proven): No behavior difference found in \_run\_cli (not a proof).
The verifier ran both versions of \_run\_cli on many inputs and saw identical behavior every time. Strong evidence the change is safe, but evidence, not a proof.
Guarantee: Empirical: differential testing (both versions run on many generated inputs). A divergence on an untested input remains possible, so this is 'no counterexample found', not 'proven equivalent'.
Note: An input the sampler did not try could still differ.
Could not verify: Could not verify \_build\_server.
The verifier did not have enough to check \_build\_server, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: not verifiable: all 23 sampled inputs raised on both versions — the function never executed, so 'no divergence' would be vacuous
No difference found (not proven): No behavior difference found in \_query\_graph\_text (not a proof).
The verifier ran both versions of \_query\_graph\_text on many inputs and saw identical behavior every time. Strong evidence the change is safe, but evidence, not a proof.
Guarantee: Empirical: differential testing (both versions run on many generated inputs). A divergence on an untested input remains possible, so this is 'no counterexample found', not 'proven equivalent'.
Note: An input the sampler did not try could still differ.
No difference found (not proven): No behavior difference found in \_query\_terms (not a proof).
The verifier ran both versions of \_query\_terms on many inputs and saw identical behavior every time. Strong evidence the change is safe, but evidence, not a proof.
Guarantee: Empirical: differential testing (both versions run on many generated inputs). A divergence on an untested input remains possible, so this is 'no counterexample found', not 'proven equivalent'.
Note: An input the sampler did not try could still differ.
· 2 grounded finding(s) anchored inline below; 18 more finding(s) on lines outside this diff (see the check run).
| _JANOME_TOKENIZER_LOCK = threading.Lock() | ||
|
|
||
|
|
||
| def _load_graph(graph_path: str) -> nx.Graph: |
There was a problem hiding this comment.
_load_graph()
9 callers depend on it (afferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| if _is_searchable(tok): | ||
| terms.append(tok) | ||
| content = [t for t in terms if t not in _QUERY_STOPWORDS] | ||
| def _query_terms(question: str, *, tokenizer: str = "baseline") -> list[str]: |
There was a problem hiding this comment.
_query_terms()
24 callers depend on it (afferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
1 participant
Summary
Japanese text commonly omits whitespace between words, which makes the existing query tokenizer less effective for Japanese-language questions. This change adds an explicit, opt-in Japanese tokenizer powered by Janome.
japaneseoptional dependency extra and include it in theallextra.--tokenizer janome_contentto the CLI.tokenizeroption to the MCPquery_graphtool.baselinetokenizer as the default, so existing installations and retrieval behavior remain unchanged unless the new option is selected.For Japanese-speaking users, this improves term-level matching for queries without spaces and can reduce missed relevant graph nodes in Japanese code and documentation. The opt-in design keeps retrieval behavior stable for existing corpora because tokenization can affect search results.
Validation
uv run --frozen --group dev --extra japanese pytest -q tests/test_serve.py tests/test_cli_export.py— 198 passeduv run --frozen --group dev --extra japanese ruff check graphify/serve.py graphify/cli.py graphify/__main__.py tests/test_serve.py tests/test_cli_export.py— passeduv sync --all-extras --frozen && uv run --frozen pytest tests/ -q --tb=short— 4883 passed, 11 skipped, and 4 environment-dependent failures in this workspace (test_detect,test_export,test_extract, andtest_hooks); none involve the changed tokenizer paths