Skip to content

feat(cli): codegraph vocabulary and drift (M19a) - #29

Open
jgrodziski wants to merge 7 commits into
feat/m19a-vocabulary-aggregatefrom
feat/m19a-vocabulary-command
Open

jgrodziski wants to merge 7 commits into
feat/m19a-vocabulary-aggregatefrom
feat/m19a-vocabulary-command

Conversation

@jgrodziski

Copy link
Copy Markdown
Contributor

Fourth step of the word cloud / glossary (#6), milestone M19a in PLAN.md §20. Stacked on #28 (itself on #27 and #26): review those first; this PR's diff is the seven commits after #28.

It adds the codegraph vocabulary command over the artifact #28 builds, and the drift report against a list of business terms.

The command

codegraph vocabulary MODEL… [--level corpus|module|type | --scope ID] [--top N]
                            [--class domain|technical|keyword|short|all]
                            [--weights KEY=N,…] [--technical FILE] [--drift FILE]
                            [--out FILE] [--json] [--internal-only] [--declared-only] [--no-cache]
  • No flag: the corpus's 40 heaviest domain words, each with its weight, occurrences, carriers, and how many modules and types say it.
  • --level module|type: one block per module or type, the scopes with the most distinct words first (a rough cognitive-load ranking), 10 words per block by default.
  • --scope ID: one module or type. An operation's id is counted on demand (operationScope), since the artifact stores no operations. An unknown id, a stub or a kind that is none of these is refused with a message saying what the id is and, for a field, which scope counts its words.
  • --class, --top, --weights, --technical FILE choose what is shown and how it is counted. --weights type=0 re-weighs; an unknown role, a repeated key or a malformed entry is a usage error.
  • --out FILE writes vocabulary.json and still prints the report.
  • Every report opens with the models, the level or scope, the view, the weights and the technical list (size, sha256, and its file when given), since a count means nothing without them.

On the Java fixture:

  TERM      CLASS   WEIGHT  OCC  CARRIERS  MODULES  TYPES
  order     domain      40   12        12        3      4
  price     domain      18    6         6        1      6
  basket    domain      16    2         2        1      2

Drift

--drift FILE reads one business term per line (Price list, Invoice) and reports, at the chosen level:

  • never said: terms whose words no scope says, each with the missing words;
  • said: terms some scope says in full, with the scopes;
  • code only: the code's domain words that no business term contains, by weight.

Words match exactly, with no stemming: lines does not match line. A term repeating an earlier term's words is merged; # lines and blank lines are skipped; a line with no word (2024) is warned about with its line number.

Decisions worth a look

  • --drift exits 3 when a business term is said nowhere, like analyze --report cycles on a cycle, so CI can gate on the glossary.
  • --out keeps the report on stdout. city and navigator replace stdout with the file instead.
  • A flag the report would not read is a usage error rather than ignored: --level or --drift with --scope, --class with --drift.

Commits

  • feat(vocabulary): drift against business terms: parseBusinessTerms and vocabularyDrift.
  • feat(vocabulary): an operation's words on demand: operationScope, with the counting rule shared with the aggregator. A lambda counts with its operation, a local class does not, and a C# delegate is a type.
  • feat(vocabulary): read a technical list from text: parseTechnicalWords, one word per line.
  • refactor(cli): share the view label: moved out of analyze.
  • feat(cli): codegraph vocabulary: the command, its option spec and the JSON report codegraph.vocabularyReport/1.
  • Docs: docs/cli.md; website pages for the command and for vocabulary.json; the CLI, artifact and exit-code index pages; PLAN §20.

Tests

  • Drift: a hand-counted toy at three levels, and properties over the Java and Ruby fixtures.
  • Operation scope: toy cases, a property that an operation's rows never exceed its parent scope's, and the C# delegate.
  • CLI: 29 command tests, the option parser, and the vocabulary command added to the determinism, exit-code, stream-purity and JSON-parity suites. Reading through the cached model.db gives the same output as reading the file on the Java, Ruby and Elixir fixtures.
  • 21 mutations across drift, the operation scope and the command, one at a time: each caught.

Verification

  • pnpm -r build and pnpm -r typecheck green; eslint clean on every changed file; pnpm run gen:schemas and the website profiles page leave no diff; the Hugo site builds and the new pages' links resolve.
  • vocabulary 284 tests, core 544, cli 973; every other package passes except the Node 26 node:sqlite warning test noted on feat(core): language lexicons on profiles (M19a) #26.
  • The single-executable bundle runs vocabulary on the fixtures.

Part of #6.

🤖 Generated with Claude Code

Jérémie Grodziski and others added 7 commits October 26, 2026 08:12
`parseBusinessTerms(text)` reads a plain glossary, one term per line,
each split with no lexicon so `Order Line` meets `order` and `line`.
Blank lines and `#` comments are skipped; a line naming no word (`2024`)
is reported by its number; a term whose words repeat an earlier one's
is that term.

`vocabularyDrift(artifact, terms, {level})` says, per business term,
which scopes of the level speak every one of its words (whatever class
the code gives them) and which words no scope of the level says. The
other half is the code-only words: `domain` terms some scope of the
level speaks that no business term contains, summed over those scopes,
by weight then term. Nothing is stemmed: `lines` is not `line`.

Tested on the hand-counted corpus at the three levels, and by
properties over the java and ruby fixtures.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`operationScope(graph, artifact, id, options)` counts one operation's
words as the artifact counts a scope: the operation and every kept
entity written inside it, lambdas and their parameters included, short
of a type or module, which are scopes of their own. Its rows index the
artifact's terms and its parent is the nearest stored scope, so a
reader shows it beside the stored ones. PLAN.md §20.5 keeps these out
of the artifact; `codegraph vocabulary --scope ID` will ask for one.

The counting rule moves into one function both paths call, so the
fixture snapshots are unchanged. An artifact built under other options
is refused rather than mis-indexed. A C# delegate, invocable but a
type, stays a type scope.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`parseTechnicalWords(text)` reads the list `--technical FILE` will
replace the shared one with: one word per line, lowercased as the
splitter lowers a token, blank lines and `#` comments skipped. A line
that is not one word (`HTTPServer`, `value object`, `2024`) could never
match a token, so it is refused by its line number instead of kept.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`analyze` printed its view as `name (filters)`, or `all (nothing
filtered …)`, through a private helper. It moves into view.ts beside
`resolveView`, so the next report that states its view prints it the
same way. analyze's output is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`codegraph vocabulary [model.jsonl...]` reports the words the model's
names are made of, as `@codegraph/vocabulary` counts them (PLAN.md
§20.1). Every report states its view, weights and technical list.

- No flag: the corpus's 40 heaviest domain words. Each row gives
  weight, occurrences, carriers, and how many modules and types say it.
- `--level module|type`: one block per scope, ranked by distinct terms
  (the cognitive-load report), 10 words each by default.
- `--scope ID`: one stored module or type, or an operation counted on
  demand. An unknown id, a stub, or a member that is no operation is a
  usage error that says what the id is.
- `--class domain|technical|keyword|short|all` and `--top N` pick the
  rows.
- `--weights KEY=N,…` overrides weights over the defaults. An unknown
  key, a repeated key or a negative weight is a usage error.
- `--technical FILE` replaces the shared list whole; a line that is not
  one word is refused by its number.
- `--drift FILE` compares with a glossary at `--level`. It lists the
  business terms no scope says, with their missing words, the ones it
  says, and the domain words no term contains. It exits 3 when a term
  goes unsaid, as `analyze --report cycles` does for a cycle.
- `--out FILE` writes vocabulary.json beside the report.
- `--json`, `--internal-only`, `--declared-only` and `--no-cache` work
  as on the other commands. The cache path counts the same words.

Tests cover each flag, text/JSON parity, determinism, stream purity and
the exit codes end to end.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The command gets its section and synopsis in docs/cli.md and a website
page: the flags, what a scope is, how drift matches a glossary, the
exit codes, and two examples taken from the java fixture's real
output. The artifact gets a website page: every field of
`codegraph.vocabulary/1` and the weights table. The CLI index, the
artifacts index and the exit-codes table list both, and `vocabulary`
joins the commands that build and reuse a sibling model.db.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
§20.1's command paragraph now describes what was built:
- the header's view, weights and technical list;
- the 10-word blocks under --level;
- an operation's scope on demand, and the usage errors that say what an
  id is;
- the --weights and --technical rules, and --out beside the report;
- the glossary's comments, merges and wordless lines;
- drift's two halves and its exit 3;
- the flags a report would not read, refused;
- the JSON kind.

§20.4 checks the drift and command item. Left in M19a: the coverage
table on the real corpora.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
codegraph 294e5cc Commit Preview URL

Branch Preview URL
Oct 26 2026, 09:00 AM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant