fix(vocabulary): no definitions of function words (M19c)

PLAN.md

@@ -5347,8 +5347,10 @@ not spell, and the deterministic class is deliberately withheld so the
 judgement is the model's — the gap between the two is layer 3's first
 report. Words of class `short` and `keyword` are not asked (`i`, `x`,
 `type`); `technical` ones are, since a technical word in a domain position
-is the whole point (`service`, `manager`, `handler`). Locals and lambda
-parameters are counted (weight 1) and not defined (§20.5). The block gains:
+is the whole point (`service`, `manager`, `handler`). *Nor, since, are
+function words (`or`, `of`, `all`): "As built — function words".* Locals and
+lambda parameters are counted (weight 1) and not defined (§20.5). The block
+gains:
 
 ```ts
 words: [{
@@ -5649,6 +5651,38 @@ Vocabulary tab's entries and the Navigate panel's, over `answerGlossary` in
   Screenshots reviewed at 1600×1000 on the Java fixture (a hand-worded store
   over its ids) and on Broadleaf.
 
+**As built — function words (2026-11-03).** The PetClinic run defined `by`,
+`with`, `is`, `of`, `all` and `or` (10 of its 301 word entries), and the
+`owner` module reconciled `or` into a glossary entry flagged `ambiguous`.
+Their senses were grammar ("lookup criterion", "boolean predicate",
+"identifier connector"), and `--polysemy` listed `or`. The decisions:
+
+- **A list in `define.ts`, not a fifth class.** `definitionSpeller` and
+  `rankedWords` leave out `FUNCTION_WORDS` (60 words: articles and
+  determiners, pronouns, conjunctions, prepositions, auxiliaries and modals,
+  `not`) whatever class the word has: `or` is on the technical list, `all`
+  is domain. The cloud, the classes and drift are unchanged, every word still
+  counted and tagged (§20.1). A class would have moved the artifact, the
+  tab's toggles and `--class` for a question only layer 2 asks.
+- **Not on the list:** a word that names a thing as often as it joins two —
+  `no` (a number), `may`, `will`, the particles of phrasal verbs (`out` of
+  `checkOut`, `up` of `signUp`), the prepositions of time and order
+  (`before`, `after`, `between`). And only English: `du` of `getNomDuClient`
+  is still asked.
+- **No option.** `explain` takes no technical list either (layer 2's as-built
+  note); a correction to the list is a code change, and the fingerprint
+  shows what it touches.
+- **What it costs a store already paid for.** The asked words are in the
+  fingerprint, so a unit whose names spell a function word goes stale, and
+  through the Merkle chain so does every unit that depends on it. On
+  PetClinic, `--dry-run` goes from 106 reused calls to 53 to redo (~121 k
+  prompt, ~56 k completion tokens, about $0.08 at the run's prices): 9 units
+  directly (7 operations, 2 controllers with an `…_OR_…` view field), 44
+  through their dependencies. Nothing is lost before a run: the old records
+  stay, and the readers show their entries until they are replaced.
+- **The Java fixture's plan** is now 114 474 prompt and 52 415 completion
+  tokens (was 114 483 and 52 900).
+
 ### 20.4 Milestone split
 
 **M19a — the lexicon, the split, the artifact** (closes the profile half of #19)

docs/insights.md

@@ -107,9 +107,10 @@ than `--max-scc` is chunked: each chunk sees the others' signatures only.
 what the words of its names mean in it: a section *Words to define* lists, per
 identifier, the words of its own name and of an operation's parameters or a
 type's fields, split by `@codegraph/vocabulary` (injected as a `Speller`: the
-insights package runs no splitter). Short words and keywords are not asked; the
-deterministic class is withheld, so the judgement — `domain`, `technical` or
-`ambiguous` — is the model's. The answer is one `words` entry per word, with
+insights package runs no splitter). Short words, keywords and function words
+(`or`, `of`, `all`: `FUNCTION_WORDS`) are not asked; the deterministic class
+is withheld, so the judgement — `domain`, `technical` or `ambiguous` — is the
+model's. The answer is one `words` entry per word, with
 its `sense` and its `meaning` in this unit. The words are given, not chosen: on
 receipt an entry for a word not listed is dropped, a repeat keeps the first,
 and `in` is restated from the split. A module is also shown what its types

packages/insights/src/fingerprint.ts

@@ -34,7 +34,8 @@ export interface FingerprintInputs {
   /**
    * Per member, the words its prompt asks to define and a module's glossary
    * words: a function of the names and the lexicon, hashed so a lexicon
-   * correction re-explains exactly the units whose words it changed.
+   * correction re-explains the units whose words it changed — and, through
+   * `dependencyFingerprints`, everything that depends on them.
    */
   readonly words: readonly { readonly names: readonly { readonly identifier: string; readonly words: readonly string[] }[]; readonly glossary: readonly string[] }[];
 }

packages/vocabulary/src/define.ts

@@ -10,13 +10,32 @@ import { splitIdentifier } from "./split.js";
  * per unit, for the words of its names; these are the words, so that the
  * prompt asks for exactly what the cloud counts. A qualifier's words are its
  * container's, asked where they are the own name. `short` and `keyword` words
- * are not asked (`i`, `type`); `technical` ones are, since a technical word in
- * a domain position is the point. The class itself is withheld from the
- * caller: the judgement is the model's.
+ * are not asked (`i`, `type`), nor function words (`or`, `of`); `technical`
+ * ones are, since a technical word in a domain position is the point. The
+ * class itself is withheld from the caller: the judgement is the model's.
  */
 
 const ASKED: ReadonlySet<TermClass> = new Set(["domain", "technical"]);
 
+/**
+ * Grammar, not vocabulary: words no glossary defines, whatever class they get
+ * (`or` is on the technical list, `all` is domain). Counted in the cloud,
+ * never asked. Contestable like the technical list; a word that names a thing
+ * as often as it joins two is left out (`no`, `may`, `will`, `out`, `before`).
+ */
+export const FUNCTION_WORDS: readonly string[] = Object.freeze([
+  "all", "an", "and", "any", "are", "as", "at", "be", "been", "but",
+  "by", "can", "could", "did", "do", "does", "each", "else", "every", "for",
+  "from", "had", "has", "have", "if", "in", "into", "is", "it", "its",
+  "must", "my", "nor", "not", "of", "on", "onto", "or", "our", "per",
+  "shall", "should", "some", "than", "that", "the", "their", "then", "these", "this",
+  "those", "to", "upon", "via", "was", "were", "with", "without", "would", "your",
+]);
+
+const GRAMMAR: ReadonlySet<string> = new Set(FUNCTION_WORDS);
+
+const isAsked = (word: string, termClass: TermClass): boolean => ASKED.has(termClass) && !GRAMMAR.has(word);
+
 /**
  * An entity's own-name words to define, each once, in the order the name
  * spells them; none for an unnamed or conventional name. No technical list:
@@ -37,7 +56,7 @@ export function definitionSpeller(graph: CodeGraph): (id: EntityId) => readonly
       classifiers.set(lang, classify);
     }
     const split = splitIdentifier(name, entity.kind, lexicon);
-    const asked = split.tokens.slice(split.qualifier).filter((token) => ASKED.has(classify(token)));
+    const asked = split.tokens.slice(split.qualifier).filter((token) => isAsked(token, classify(token)));
     return [...new Set(asked)];
   };
 }
@@ -48,7 +67,7 @@ export function rankedWords(artifact: VocabularyArtifact, scopeId: EntityId | un
   const words = new Set<string>();
   for (const [index] of scope?.terms ?? []) {
     const term = artifact.terms[index];
-    if (term !== undefined && ASKED.has(term.class)) words.add(term.text);
+    if (term !== undefined && isAsked(term.text, term.class)) words.add(term.text);
   }
   return [...words];
 }

packages/vocabulary/test/define.test.ts

@@ -1,7 +1,7 @@
 import type { Entity } from "@codegraph/core";
 import { describe, expect, it } from "vitest";
 import { buildVocabulary } from "../src/aggregate.js";
-import { definitionSpeller, rankedWords } from "../src/define.js";
+import { FUNCTION_WORDS, definitionSpeller, rankedWords } from "../src/define.js";
 import { cls, graphOf, method, orderCorpus, pkg, structural } from "./toy.js";
 
 /**
@@ -24,6 +24,11 @@ describe("definitionSpeller", () => {
         method("java:com.acme.order/OrderService.toString()", "toString", "java:com.acme.order/OrderService"),
         method("java:com.acme.order/OrderService.<init>()", undefined, "java:com.acme.order/OrderService"),
         structural("java:com.acme.order/OrderService.MAX_LINES", "attribute", "MAX_LINES", "java:com.acme.order/OrderService"),
+        method("java:com.acme.order/OrderService.findByLastNameOrFirstName()", "findByLastNameOrFirstName", "java:com.acme.order/OrderService"),
+        method("java:com.acme.order/OrderService.findAll()", "findAll", "java:com.acme.order/OrderService"),
+        method("java:com.acme.order/OrderService.isDuplicateOf()", "isDuplicateOf", "java:com.acme.order/OrderService"),
+        method("java:com.acme.order/OrderService.andThen()", "andThen", "java:com.acme.order/OrderService"),
+        structural("java:com.acme.order/OrderService.VIEWS_CREATE_OR_UPDATE_FORM", "attribute", "VIEWS_CREATE_OR_UPDATE_FORM", "java:com.acme.order/OrderService"),
       ],
     },
     {
@@ -53,6 +58,20 @@ describe("definitionSpeller", () => {
     expect(words("go:shop/Type")).toEqual(["order"]);
   });
 
+  it("leaves out function words, whatever class the lists give them", () => {
+    // `by` and `or` are on the technical list, `all` is domain: grammar either way.
+    expect(words("java:com.acme.order/OrderService.findByLastNameOrFirstName()")).toEqual(["find", "last", "name", "first"]);
+    expect(words("java:com.acme.order/OrderService.findAll()")).toEqual(["find"]);
+    expect(words("java:com.acme.order/OrderService.isDuplicateOf()")).toEqual(["duplicate"]);
+    expect(words("java:com.acme.order/OrderService.VIEWS_CREATE_OR_UPDATE_FORM")).toEqual(["views", "create", "update", "form"]);
+    expect(words("java:com.acme.order/OrderService.andThen()")).toEqual([]);
+  });
+
+  it("keeps every function word lowercase and one word, as the splitter spells a token", () => {
+    for (const word of FUNCTION_WORDS) expect(word).toMatch(/^[a-z]+$/);
+    expect(new Set(FUNCTION_WORDS).size).toBe(FUNCTION_WORDS.length);
+  });
+
   it("asks for the own name only: a qualifier's words are its container's", () => {
     expect(words("java:com.acme.order")).toEqual(["order"]);
     expect(words("rb:lib/acme_order/order.rb/AcmeOrder::OrderLine")).toEqual(["order", "line"]);
@@ -92,6 +111,11 @@ describe("rankedWords", () => {
     expect(rankedWords(mixed, undefined)).toEqual(["shop", "type"]);
   });
 
+  it("lists no function word, so a glossary is not cut on one", () => {
+    const shop = buildVocabulary(graphOf([{ lang: "java", entities: [pkg("java:p", "shop"), cls("java:p/ShopOfTheDay", "ShopOfTheDay", "java:p")] }]));
+    expect(rankedWords(shop, "java:p")).toEqual(["shop", "day"]);
+  });
+
   it("knows no word of a scope the artifact does not hold", () => {
     expect(rankedWords(artifact, "java:nowhere")).toEqual([]);
   });

website/content/docs/discover/meaning.md

@@ -46,11 +46,11 @@ codegraph explain app.jsonl --src ~/src/app/src/main/java --dry-run
 
 ```text
 explain plan: 77 units in 8 layers
-  operation     57 units     48 calls ~78148 prompt tokens
-  type          17 units     17 calls ~31001 prompt tokens
-  module         3 units      3 calls ~5334 prompt tokens
+  operation     57 units     48 calls ~78146 prompt tokens
+  type          17 units     17 calls ~31000 prompt tokens
+  module         3 units      3 calls ~5328 prompt tokens
   statuses: llm 68, template 9, reuse 0, skip-scope 0, skip-budget 0
-  total: 68 calls, ~114483 prompt tokens in, ~52900 completion tokens out
+  total: 68 calls, ~114474 prompt tokens in, ~52415 completion tokens out
 ```
 
 ```bash
@@ -59,7 +59,7 @@ codegraph explain app.jsonl --src ~/src/app/src/main/java \
 ```
 
 ```text
-  cost at $0.1/M in, $0.6/M out: $0.0432
+  cost at $0.1/M in, $0.6/M out: $0.0429
 ```
 
 Prices are dollars per million tokens for the model you intend to use; the

website/content/docs/how-to/explain-cost.md

@@ -19,12 +19,12 @@ codegraph explain model.jsonl --src fixtures/java/src --dry-run
 
 ```text
 explain plan: 77 units in 8 layers
-  operation     57 units     48 calls ~78148 prompt tokens
-  type          17 units     17 calls ~31001 prompt tokens
-  module         3 units      3 calls ~5334 prompt tokens
+  operation     57 units     48 calls ~78146 prompt tokens
+  type          17 units     17 calls ~31000 prompt tokens
+  module         3 units      3 calls ~5328 prompt tokens
   statuses: llm 68, template 9, reuse 0, skip-scope 0, skip-budget 0
   models: openai/gpt-5.6-luna (operations), openai/gpt-5.6-luna (types, modules); depth 1
-  total: 68 calls, ~114483 prompt tokens in, ~41670 completion tokens out
+  total: 68 calls, ~114474 prompt tokens in, ~52415 completion tokens out
 
 llm         L0 operation java:com.acme.order.adapter/LedgerAdapter.<init>() calls=1 ~1461tok
 template    L0 operation java:com.acme.order.legacy/List.head()
@@ -43,12 +43,12 @@ codegraph explain model.jsonl --src fixtures/java/src \
 
 ```text
   level        calls   input tokens  output tokens
-  operation       48         78 148         30 360
-  type            17         31 001         15 590
-  module           3          5 334          6 950
-  total           68        114 483         52 900
+  operation       48         78 146         30 120
+  type            17         31 000         15 470
+  module           3          5 328          6 825
+  total           68        114 474         52 415
 
-  cost at $0.1/M in, $0.6/M out: $0.0432
+  cost at $0.1/M in, $0.6/M out: $0.0429
   input ≈ rendered prompts at 4 characters per token; output ≈ measured on a live run: operation 470, type 530, module 530 tokens per block asked for, plus 120 per word and 125 per glossary entry asked to define (gpt-5.6-luna, reasoning included).
   not included: repair re-asks, rate-limit retries. Records already in the side-car are reused, not re-sent.
 ```

website/content/docs/reference/artifacts/insights-jsonl.md

@@ -149,7 +149,7 @@ Every block carries a `name` — the ubiquitous-language name the model proposes
 
 ### `WordEntry` and `GlossaryEntry`
 
-`explain` asks each unit to define the words its names spell, split as [`codegraph vocabulary`](/docs/reference/cli/vocabulary/) splits them; short words and keywords are not asked. The words are given, never chosen: on receipt an entry for a word the prompt did not list is dropped, a word given twice keeps its first entry, and `in` is restated from the split. A record explained under prompt 1, and a templated one, has no `words`.
+`explain` asks each unit to define the words its names spell, split as [`codegraph vocabulary`](/docs/reference/cli/vocabulary/) splits them; short words, keywords and function words (`or`, `of`, `all`) are not asked. The words are given, never chosen: on receipt an entry for a word the prompt did not list is dropped, a word given twice keeps its first entry, and `in` is restated from the split. A record explained under prompt 1, and a templated one, has no `words`.
 
 | `WordEntry` field | Type |
 |---|---|

website/content/docs/reference/cli/explain.md

@@ -61,12 +61,12 @@ A `401`, `402` or `403` is about the account, not the unit: the run **aborts** 
 $ codegraph explain fixtures/java/expected/model.jsonl --src fixtures/java/src --dry-run
 cache: fixtures/java/expected/model.db
 explain plan: 77 units in 8 layers
-  operation     57 units     48 calls ~78148 prompt tokens
-  type          17 units     17 calls ~31001 prompt tokens
-  module         3 units      3 calls ~5334 prompt tokens
+  operation     57 units     48 calls ~78146 prompt tokens
+  type          17 units     17 calls ~31000 prompt tokens
+  module         3 units      3 calls ~5328 prompt tokens
   statuses: llm 68, template 9, reuse 0, skip-scope 0, skip-budget 0
   models: openai/gpt-5.6-luna (operations), openai/gpt-5.6-luna (types, modules); depth 1
-  total: 68 calls, ~114483 prompt tokens in, ~41670 completion tokens out
+  total: 68 calls, ~114474 prompt tokens in, ~52415 completion tokens out
 ```
 
 Trimmed after the plan summary; one line per unit follows. `--dry-run` needs no API key.

website/content/docs/tutorials/explain.md

@@ -65,12 +65,12 @@ codegraph explain order.jsonl --src "$SRC" --dry-run
 
 ```text
 explain plan: 77 units in 8 layers
-  operation     57 units     48 calls ~78148 prompt tokens
-  type          17 units     17 calls ~31001 prompt tokens
-  module         3 units      3 calls ~5334 prompt tokens
+  operation     57 units     48 calls ~78146 prompt tokens
+  type          17 units     17 calls ~31000 prompt tokens
+  module         3 units      3 calls ~5328 prompt tokens
   statuses: llm 68, template 9, reuse 0, skip-scope 0, skip-budget 0
   models: openai/gpt-5.6-luna (operations), openai/gpt-5.6-luna (types, modules); depth 1
-  total: 68 calls, ~114483 prompt tokens in, ~41670 completion tokens out
+  total: 68 calls, ~114474 prompt tokens in, ~52415 completion tokens out
 
 llm         L0 operation java:com.acme.order.adapter/LedgerAdapter.<init>() calls=1 ~1461tok
 llm         L0 operation java:com.acme.order.adapter/LedgerAdapter.post(java.lang.Object) calls=1 ~1468tok
@@ -122,12 +122,12 @@ explain estimate: order.jsonl
   models: openai/gpt-5.6-luna (operations), openai/gpt-5.6-luna (types, modules); depth 1
 
   level        calls   input tokens  output tokens
-  operation       48         78 148         30 360
-  type            17         31 001         15 590
-  module           3          5 334          6 950
-  total           68        114 483         52 900
+  operation       48         78 146         30 120
+  type            17         31 000         15 470
+  module           3          5 328          6 825
+  total           68        114 474         52 415
 
-  cost at $0.1/M in, $0.6/M out: $0.0432
+  cost at $0.1/M in, $0.6/M out: $0.0429
   input ≈ rendered prompts at 4 characters per token; output ≈ measured on a live run: operation 470, type 530, module 530 tokens per block asked for, plus 120 per word and 125 per glossary entry asked to define (gpt-5.6-luna, reasoning included).
   not included: repair re-asks, rate-limit retries. Records already in the side-car are reused, not re-sent.
 ```