Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 32 additions & 1 deletion PLAN.md
Original file line number Diff line number Diff line change
Expand Up @@ -5451,7 +5451,8 @@ with their `polysemous` flag: the signifier/signified glossary of the issue,
at every scope, from one index lookup. `--glossary MODULE_ID` — a module's
reconciled glossary whole. `--polysemy` — the words a module flagged, and
the words whose module-level senses differ across the corpus, side by side
for a human to judge (the corpus has no call to judge for it). `--concerns`
for a human to judge (the corpus has no call to judge for it). *A flag
counts when its senses are two: "As built — two senses to a polysemy".* `--concerns`
— per module, how many of its words the model called domain, technical,
ambiguous: the cognitive-load proxy split by kind. `codegraph vocabulary
--drift-insights` — the deterministic classes against the model's concerns
Expand Down Expand Up @@ -5710,6 +5711,36 @@ words the PetClinic modules were asked to reconcile, 2 no type had defined:
module prompt now prints no glossary lines at all, so the plan's input is a
little lower (the Java fixture: 114 136 prompt tokens, was 114 474).

**As built — two senses to a polysemy (2026-11-03).** PetClinic's `owner`
module flagged `owners` polysemous and listed its three types, each giving
the one sense "owner repository dependency", and `or`, two types and one
sense: 2 of the run's 16 flags, over 52 glossary entries. `--polysemy` listed
both, and the page said "its types disagree" with nothing to disagree on.
The decisions:

- **A flag counts when its senses bear it out.** `typesDisagree(entry)` in
`glossary.ts`: `polysemous`, and two distinct `senses` at least, case and
spacing aside — the comparison `--polysemy` already made across modules.
The flag is the model's judgement of meaning, the senses its evidence; both
are needed, so two wordings the model did not flag stay one sense.
- **Read, not rewritten.** The record keeps what the model wrote: the store
is not a cache, and the side-car's decode → import → export stays the
identity. `--polysemy`,
`--word`, `--glossary`, the Vocabulary tab and the Navigate panel ask the
rule; the page restates it (it imports nothing of `@codegraph/insights`),
pinned to the package's by a test over the same cases.
- **A flag that does not hold adds no sense** to a word's count in
`--polysemy`: its one sense rewords the entry's own.
- **Lexical, not semantic.** `pet`'s four wordings of "animal under care" in
the same module stay four senses: judging meaning is a model call, and the
readers make none.
- **The prompt is unchanged.** It already asks for a flag only where the
types disagree; rewording it is a new `PROMPT_VERSION`, which every
fingerprint includes — 106 calls again on PetClinic — for what a reader
can check. No fingerprint input moved: no record goes stale.
- **On PetClinic** `--polysemy` lists 14 words, was 16 (`owners` and `or`
gone; `or` is no longer asked at all since the function words).

### 20.4 Milestone split

**M19a — the lexicon, the split, the artifact** (closes the profile half of #19)
Expand Down
11 changes: 6 additions & 5 deletions docs/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -362,14 +362,15 @@ invocation:
tab-separated line per entry — the module, `glossary` or the entity's level,
the id, the concern, the confidence, the sense, and the identifiers the word
is in. Each module's own glossary entry comes first (its last cell lists the
types' senses when the module flagged the word polysemous), then the module,
its types, its operations. A record is placed by its natural key; one
without a key is under `-`. The meaning is not shown: `--glossary` and `--id`
show it;
types' senses when the module flagged the word polysemous and they are two
senses at least, case and spacing aside), then the module, its types, its
operations. A record is placed by its natural key; one without a key is
under `-`. The meaning is not shown: `--glossary` and `--id` show it;
- `--glossary MODULE_ID` — a module's glossary whole, one line per word: the
word, the concern, sense and meaning the module reconciled from its types,
and the types' senses where they disagree;
- `--polysemy` — the words a module flagged, and the words two modules'
- `--polysemy` — the words a module flagged, its types giving two senses at
least (a flag over one sense is not counted), and the words two modules'
glossaries define differently (in concern or sense, case and spacing aside):
every module's entry for each, side by side, the most senses first. No
model call judges them across the corpus: a human does;
Expand Down
4 changes: 3 additions & 1 deletion docs/insights.md
Original file line number Diff line number Diff line change
Expand Up @@ -119,7 +119,9 @@ projection, as summaries are, and only when its prompt renders — heaviest
first by the vocabulary's module weights, cut at `--terms N`, and answers a
`glossary` that reconciles those a type defined (a word its types all left
undefined is not asked), flagging a word its types disagree on as
`polysemous`. Templated records carry no `words`.
`polysemous` with each type's sense. A reader counts the flag only when those
senses are two at least, case and spacing aside (`typesDisagree`): the record
keeps what the model wrote. Templated records carry no `words`.

`PROMPT_VERSION` is `"2"`. A record explained under prompt 1 keeps decoding and
re-encoding to its own bytes (`words` and `glossary` are optional in the record
Expand Down
7 changes: 4 additions & 3 deletions packages/cli/src/commands/insights.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ import {
answerInsight,
concernsReport,
polysemyReport,
typesDisagree,
wordReport,
wordsAbsence,
type FailureRecord,
Expand Down Expand Up @@ -198,9 +199,9 @@ export function noteWithoutWords(count: number, io: IoSink): void {
/** A cell for a placed-nowhere module: a record without a natural key. */
const moduleCell = (module: string | undefined): string => module ?? "-";

/** The types' senses of a word their module flagged; an empty cell when they agree. */
/** The types' senses of a word their module flagged, when two differ (`typesDisagree`); an empty cell when they agree. */
function polysemyCell(entry: GlossaryEntry): string {
return entry.polysemous ? `polysemous: ${entry.senses.map((s) => `${s.in} "${s.sense}"`).join(" · ")}` : "";
return typesDisagree(entry) ? `polysemous: ${entry.senses.map((s) => `${s.in} "${s.sense}"`).join(" · ")}` : "";
}

/** One tab-separated line per entry, the module first so the grouping survives `sort`: module, what, id, concern, confidence, sense, and the identifiers (or the flag). */
Expand Down Expand Up @@ -251,7 +252,7 @@ function polysemy(store: InsightsStore, options: InsightsOptions, io: IoSink): E
outLine(io, JSON.stringify({ kind: "codegraph.insightsPolysemy/1", ...report }));
return EXIT.OK;
}
if (report.words.length === 0) errLine(io, "no word has two senses: no module flagged one, and no two modules' glossaries define one differently.");
if (report.words.length === 0) errLine(io, "no word has two senses: no module's types give one two, and no two modules' glossaries define one differently.");
else outLine(io, report.words.flatMap((w) => w.entries.map((g) => [w.word, g.id, g.entry.concern, g.entry.sense, polysemyCell(g.entry)].join("\t"))).join("\n"));
noteWithoutWords(report.withoutWords, io);
return EXIT.OK;
Expand Down
3 changes: 2 additions & 1 deletion packages/cli/test/insights.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -257,7 +257,8 @@ describe("codegraph insights: the words of the names, as the model judged them (
const WORDED = [
module_(ORDERING, key("com.acme.order"), [word("order", "domain", "the ordering business", ["com.acme.order"], 0.8)], [
gloss("order", "domain", "a customer's request", [{ in: "Order", sense: "a customer's request" }, { in: "SortSpec", sense: "a sequence" }]),
gloss("lines", "domain", "the items ordered"),
// Flagged, its two types giving one sense: no disagreement to show.
gloss("lines", "domain", "the items ordered", [{ in: "Order", sense: "the items ordered" }, { in: "OrderLine", sense: "The items ordered" }]),
]),
worded(type(ORDER, "aggregate", 0.9), key("com.acme.order", "Order"), [word("order", "domain", "a customer's request", ["Order"], 0.9), word("lines", "domain", "the items ordered", ["lines", "addLine"], 0.85)]),
worded(type("java:com.acme.order/SortSpec", "valueType", 0.7), key("com.acme.order", "SortSpec"), [word("order", "technical", "a sequence", ["sortOrder"], 0.6)]),
Expand Down
22 changes: 17 additions & 5 deletions packages/insights/src/glossary.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
import { naturalKeyIndex, naturalKeyIssues, renderId, type NaturalKey } from "@codegraph/core";
import { WORD_CONCERNS, type InsightRecord, type Level, type RecordKey, type WordConcern } from "./schema.js";
import { WORD_CONCERNS, type GlossaryEntry, type InsightRecord, type Level, type RecordKey, type WordConcern } from "./schema.js";
import type { GlossaryHit, InsightsStore, WordHit } from "./store.js";

/**
Expand Down Expand Up @@ -97,11 +97,11 @@ export function wordReport(store: InsightsStore, word: string): WordReport {

export interface PolysemousWord {
readonly word: string;
/** A module's glossary flags it: that module's types disagree on it. */
/** A module's types disagree on it (`typesDisagree`). */
readonly flagged: boolean;
/** The modules' entries give it more than one sense. */
readonly acrossModules: boolean;
/** Distinct senses over the entries and the senses a flag lists. */
/** Distinct senses over the entries and the senses of each flag that holds. */
readonly senses: number;
/** Every module's entry for the word, by module id: side by side, for a human to judge. */
readonly entries: readonly GlossaryHit[];
Expand All @@ -116,6 +116,16 @@ export interface PolysemyReport {
/** Two senses are one when they differ only in case and spacing: the model's wording varies, its meaning is what is compared. */
const sense = (text: string): string => text.trim().replace(/\s+/gu, " ").toLowerCase();

/**
* A module's `polysemous` flag, as far as the senses it lists bear it out: two
* at least. The flag is the model's judgement of meaning, the senses its
* evidence (PetClinic's `owner` module flagged `owners`, its three types giving
* one sense). The record keeps what the model wrote; every reader asks this.
*/
export function typesDisagree(entry: GlossaryEntry): boolean {
return entry.polysemous && new Set(entry.senses.map((s) => sense(s.sense))).size > 1;
}

/** `insights --polysemy`: the corpus has no call to judge a word across modules, so this lays the modules' senses side by side. */
export function polysemyReport(store: InsightsStore): PolysemyReport {
const byWord = new Map<string, GlossaryHit[]>();
Expand All @@ -124,11 +134,13 @@ export function polysemyReport(store: InsightsStore): PolysemyReport {
hits.push(hit);
byWord.set(hit.entry.word, hits);
}
// A flag its senses do not bear out lists one sense: a rewording of the entry's own, not another.
const listed = (entry: GlossaryEntry): readonly string[] => (typesDisagree(entry) ? entry.senses.map((s) => s.sense) : []);
const words: PolysemousWord[] = [];
for (const [word, entries] of byWord) {
const flagged = entries.some((hit) => hit.entry.polysemous);
const flagged = entries.some((hit) => typesDisagree(hit.entry));
const judged = new Set(entries.map((hit) => `${hit.entry.concern}\u0000${sense(hit.entry.sense)}`));
const senses = new Set(entries.flatMap((hit) => [hit.entry.sense, ...hit.entry.senses.map((s) => s.sense)].map(sense)));
const senses = new Set(entries.flatMap((hit) => [hit.entry.sense, ...listed(hit.entry)].map(sense)));
if (flagged || judged.size > 1) words.push({ word, flagged, acrossModules: judged.size > 1, senses: senses.size, entries });
}
words.sort((a, b) => b.senses - a.senses || byCodeUnits(a.word, b.word));
Expand Down
29 changes: 26 additions & 3 deletions packages/insights/test/glossary.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ import {
concernsReport,
moduleOf,
polysemyReport,
typesDisagree,
wordReport,
wordsAbsence,
type GlossaryAnswer,
Expand Down Expand Up @@ -74,8 +75,10 @@ const RECORDS: InsightRecord[] = [
glossary: [
gloss("order", "domain", "a customer's request", [{ in: "Order", sense: "a customer's request" }, { in: "SortSpec", sense: "a sequence" }]),
gloss("line", "domain", "an order line", [{ in: "Order", sense: "an order line" }, { in: "Text", sense: "a line of text" }]),
gloss("total", "domain", "the sum"),
gloss("price", "domain", "a unit price"),
// Flagged, but its one type gives it one sense.
gloss("total", "domain", "the sum", [{ in: "Order", sense: "the sum" }]),
// Flagged, but its two types give it one sense, case and spacing aside (PetClinic's `owners`).
gloss("price", "domain", "a unit price", [{ in: "Order", sense: "the price of one unit" }, { in: "LineItem", sense: "The price of one unit" }]),
gloss("service", "technical", "a layer of the code"),
],
}),
Expand Down Expand Up @@ -159,15 +162,35 @@ describe("polysemyReport: the words a module flagged, and the words its modules'
{ word: "order", flagged: true, acrossModules: true, senses: 3, entries: [`${BILLING}: a purchase order`, `${ORDERING}: a customer's request`] },
// Flagged by its one module, whose types disagree.
{ word: "line", flagged: true, acrossModules: false, senses: 2, entries: [`${ORDERING}: an order line`] },
// Its flag lists one sense: not flagged, and that sense is not a third.
{ word: "price", flagged: false, acrossModules: true, senses: 2, entries: [`${BILLING}: an amount due`, `${ORDERING}: a unit price`] },
// One sense, judged domain here and technical there: a disagreement all the same.
{ word: "service", flagged: false, acrossModules: true, senses: 1, entries: [`${BILLING}: a layer of the code`, `${ORDERING}: a layer of the code`] },
]);
// "the sum" and "The Sum " are one sense: case and spacing aside. A word one module defines once is not listed.
// "the sum" and "The Sum " are one sense: case and spacing aside. A word one module defines once is not listed, flagged or not.
expect(withoutWords).toBe(1);
});
});

describe("typesDisagree: a module's flag, as far as the senses it lists bear it out", () => {
const flag = (polysemous: boolean, ...senses: string[]): GlossaryEntry => ({
word: "order", concern: "domain", sense: "a request", meaning: "What a customer asks for.", polysemous, senses: senses.map((sense, i) => ({ in: `T${i}`, sense })),
});

it("holds for a flag that lists two senses, from two types or from one", () => {
expect(typesDisagree(flag(true, "a request", "a sequence"))).toBe(true);
expect(typesDisagree({ ...flag(true, "a request", "a sequence"), senses: [{ in: "Order", sense: "a request" }, { in: "Order", sense: "a sequence" }] })).toBe(true);
});

it("holds for no flag of one sense, case and spacing aside, nor for senses the module did not flag", () => {
expect(typesDisagree(flag(true, "an owner repository", "An owner repository ", "an owner repository"))).toBe(false);
expect(typesDisagree(flag(true, "a request"))).toBe(false);
expect(typesDisagree(flag(true))).toBe(false);
// The flag is the model's judgement of meaning: two wordings it calls one sense are one.
expect(typesDisagree(flag(false, "a request", "a customer's request"))).toBe(false);
});
});

describe("concernsReport: per module, how many of its words the model called domain, technical, ambiguous", () => {
it("counts each word once per concern it was given, the most-worded module first, the records without a key last", () => {
expect(concernsReport(seeded())).toEqual({
Expand Down
4 changes: 2 additions & 2 deletions packages/navigator-ui/src/components/VocabularyView.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ import type { TermClass } from "@codegraph/vocabulary";
import { CLOUD_DEFAULTS, MONOSPACE_LINE, layoutCloud, monospaceMeasure, type Measure } from "@codegraph/vocabulary/cloud";
import type { ModelIndexes } from "../model/indexes.js";
import { carrierNode, cloudWords, indexVocabulary, scopeChain, scopeForNode, wordRows, type WordRow } from "../model/words.js";
import { createGlossaryLoader, type Glossary } from "../glossary.js";
import { createGlossaryLoader, typesDisagree, type Glossary } from "../glossary.js";
import { TERM_CLASSES, createCarriersLoader, loadVocabulary, type Carriers, type VocabularyLoad } from "../vocabulary.js";

/**
Expand Down Expand Up @@ -451,7 +451,7 @@ function GlossaryList({
</button>
<p className="gloss-sense">{gloss.entry.sense}</p>
{gloss.entry.meaning !== "" && <p className="gloss-meaning">{gloss.entry.meaning}</p>}
{gloss.entry.polysemous && gloss.entry.senses.length > 0 && (
{typesDisagree(gloss.entry) && (
<p className="gloss-meaning">
its types disagree: {gloss.entry.senses.map((sense) => `${sense.in} — ${sense.sense}`).join("; ")}
</p>
Expand Down
10 changes: 10 additions & 0 deletions packages/navigator-ui/src/glossary.ts
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,16 @@ function isWord(value: unknown): value is GlossaryWord {
return isString(entry["word"]) && Array.isArray(entry["in"]) && isString(entry["concern"]) && isString(entry["sense"]) && isNumber(entry["confidence"]);
}

/**
* A module's flag, as far as the senses it lists bear it out: two at least,
* case and spacing aside. Restated from `@codegraph/insights`' `typesDisagree`
* and pinned to it by `glossary.test.ts`; a sense of another shape is none.
*/
export function typesDisagree(entry: { readonly polysemous?: unknown; readonly senses: readonly unknown[] }): boolean {
const senses = entry.senses.flatMap((s) => (isObject(s) && isString(s["sense"]) ? [s["sense"].trim().replace(/\s+/gu, " ").toLowerCase()] : []));
return entry.polysemous === true && new Set(senses).size > 1;
}

function isGloss(value: unknown): value is ModuleGloss {
if (!isObject(value) || !isString(value["id"]) || !isObject(value["entry"])) return false;
const entry = value["entry"];
Expand Down
3 changes: 2 additions & 1 deletion packages/navigator-ui/src/insight.ts
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
import { typesDisagree } from "./glossary.js";
import { noStore, rememberedLookup, type Fetcher } from "./lookup.js";

export type { Fetcher } from "./lookup.js";
Expand Down Expand Up @@ -174,7 +175,7 @@ const DEFINED: Readonly<Record<string, (item: Record<string, unknown>) => string
const senses = item["senses"];
if (head === undefined || !Array.isArray(senses)) return undefined;
const disagree = senses.flatMap((s) => (isObject(s) && typeof s["in"] === "string" && typeof s["sense"] === "string" ? [`${s["in"]} — ${s["sense"]}`] : []));
return item["polysemous"] === true && disagree.length > 0 ? `${head} Its types disagree: ${disagree.join("; ")}` : head;
return typesDisagree({ polysemous: item["polysemous"], senses }) ? `${head} Its types disagree: ${disagree.join("; ")}` : head;
},
};

Expand Down
19 changes: 17 additions & 2 deletions packages/navigator-ui/test/glossary.test.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import { describe, expect, it } from "vitest";
import { GLOSSARY_ANSWER_KIND as CANONICAL_KIND } from "@codegraph/insights";
import { GLOSSARY_ANSWER_KIND, createGlossaryLoader, glossaryUrl, parseGlossaryAnswer, type Fetcher } from "../src/glossary.js";
import { GLOSSARY_ANSWER_KIND as CANONICAL_KIND, typesDisagree as canonicalTypesDisagree, type GlossaryEntry } from "@codegraph/insights";
import { GLOSSARY_ANSWER_KIND, createGlossaryLoader, glossaryUrl, parseGlossaryAnswer, typesDisagree, type Fetcher } from "../src/glossary.js";

const MODULE = "java:com.acme.order";
const SERVICE = "java:com.acme.order/OrderService";
Expand All @@ -23,6 +23,21 @@ describe("the glossary's literal", () => {
it("restates the answer's kind exactly", () => {
expect(GLOSSARY_ANSWER_KIND).toBe(CANONICAL_KIND);
});

it("restates when a module's types disagree exactly as the package judges it", () => {
const flag = (polysemous: boolean, ...senses: string[]): GlossaryEntry => ({
word: "order", concern: "domain", sense: "a request", meaning: "", polysemous, senses: senses.map((sense, i) => ({ in: `T${i}`, sense })),
});
const cases = [
flag(true, "a request", "a sequence"),
flag(true, "an owner repository", "An owner repository "),
flag(true, "a request"),
flag(true),
flag(false, "a request", "a sequence"),
];
expect(cases.map(typesDisagree)).toEqual(cases.map(canonicalTypesDisagree));
expect(cases.map(typesDisagree)).toEqual([true, false, false, false, false]);
});
});

describe("parseGlossaryAnswer", () => {
Expand Down
Loading
Loading