diff --git a/PLAN.md b/PLAN.md index ca710ad..2709b78 100644 --- a/PLAN.md +++ b/PLAN.md @@ -5094,7 +5094,8 @@ scope per line, the snapshots' form. Size, measured on BroadleafCommerce [--scope ID] [--top N] [--class domain|technical|keyword|short|all] [--weights …] [--technical FILE] [--drift FILE] [--out vocabulary.json] [--svg FILE] [--json] [--internal-only] [--declared-only] [--no-cache]` -(`--svg` arrives with M19b's layout), a shell over the package like +(`--svg`, M19b's, draws the one scope reported — the corpus or `--scope`'s — +so it is a usage error with `--level module|type` or `--drift`), a shell over the package like `analyze` over the analyzer. Every report's header states the view, the weights and the technical list (its size, the first 12 hex of its digest, and the file it came from). No flag: the corpus @@ -5201,7 +5202,14 @@ it. The text measurer is INJECTED — `measure(text, size) → { width, height } is exact for the font the SVG declares, and the browser passes `canvas.measureText` for its proportional face. Properties: no two placed boxes intersect; every box lies inside the reported viewBox; the same input -gives the same positions; placement order is weight order. +gives the same positions; placement order is weight order. **As built** +(`cloud.ts`, type imports only, so a page runs it without the pipeline): +12 to 56 px rounded to the pixel, 4 px kept clear on each side (2 px let +neighbours touch on screen), the spiral stretched 2:1 for a landscape image, +ties by text then class so ANY order of the same words lays out the same; +the monospace measurer counts a combining mark as no cell and a Han, kana, +Hangul or fullwidth character as two, as a terminal draws them; the spiral's +points are whole pixels and the SVG rounds to 1/100 px. **SVG** (`svg.ts`): size by weight, colour by class and nothing else (`domain` in ink, `technical` muted, `keyword` and `short` faint — meaning @@ -5210,6 +5218,14 @@ class filter and the counts, a monospace font stack. Deterministic bytes: `fixtures/java/expected/vocabulary.svg` is a snapshot, and the demo image is this command's output on a public corpus, committed under `docs/`. `codegraph vocabulary --svg FILE [--scope ID] [--top N] [--class …]`. +**As built** (`vocabularyCloudSvg`): the palette is a stylesheet keyed by +class, light and dark (`prefers-color-scheme`), so a word carries one class +attribute and no fill; a key line appears when more than one class is drawn; +the legend's three lines are the scope and its counts, the weights, and the +view with the technical list's size and digest. `docs/vocabulary-broadleaf.svg` +is BroadleafCommerce's corpus at `--top 80` (6.7 KB; 1.9 s end to end, the +load dominating) — `order item customer product offer fulfillment payment sku +cart catalog…`, the commerce vocabulary §20.1 asked for. **`serve`** builds `vocabulary.json` from the same graph and view as the two others and serves it at `/vocabulary.json`; evidence is a LOOKUP, @@ -5416,8 +5432,8 @@ never looks like a fact. **M19b — the cloud, the tab, the city channel** -- [ ] `layout.ts` and `svg.ts` with their properties and the SVG snapshot; - `--svg`. +- [x] `layout.ts` and `svg.ts` with their properties and the SVG snapshot; + `--svg` (the layout is `cloud.ts`). - [ ] `serve`: `/vocabulary.json` built with the two others, `/carriers.json` as a lookup; both on the app daemon's page for free (the artifact rides the same build), the lookups following the @@ -5428,9 +5444,9 @@ never looks like a fact. - [ ] `vocabulary:` metric source through the `metric(name)` hook; `--color vocabulary:order` on Broadleaf reviewed as a screenshot. - [ ] The demo images: the corpus cloud of a public repository as SVG - (`docs/vocabulary-.svg`) and the city coloured by one term, - with the README section that shows them — the issue's fourth - acceptance criterion. + (`docs/vocabulary-.svg` — Broadleaf's, committed with `--svg`) + and the city coloured by one term, with the README section that shows + them — the issue's fourth acceptance criterion. **M19c — the glossary: words in the explanation** diff --git a/docs/cli.md b/docs/cli.md index 5594913..0c237dc 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -69,7 +69,8 @@ codegraph navigator [model.jsonl] [--out FILE] codegraph vocabulary [model.jsonl] [--level corpus|module|type] [--scope ID] [--top N] [--class domain|technical|keyword|short|all] [--weights KEY=N,…] [--technical FILE] [--drift FILE] - [--out FILE] [--json] [--internal-only] [--declared-only] + [--out FILE] [--svg FILE] [--json] [--internal-only] + [--declared-only] codegraph domain-facts [model.jsonl] [--framework spring] [--out FILE] [--internal-only] [--declared-only] @@ -233,6 +234,12 @@ which one counted. `--drift FILE` compares with a glossary, one business term per line: the terms no scope of `--level` says (exit `3`), the ones it says and where, and the domain words no term contains. `--out FILE` writes the `vocabulary.json` artifact (`codegraph.vocabulary/1`) beside the report. +`--svg FILE` draws the cloud of the one scope reported — the corpus, or +`--scope`'s — from the words `--class` and `--top` pick: size on the square +root of the weight, colour by class and nothing else, a legend stating the +scope, weights, view and technical list, the same bytes on every run. +[`vocabulary-broadleaf.svg`](vocabulary-broadleaf.svg) is BroadleafCommerce's, +drawn with `--top 80`. ### `domain-facts` One JSON artifact (`codegraph.domainFacts/1`) with one dossier per corpus diff --git a/docs/vocabulary-broadleaf.svg b/docs/vocabulary-broadleaf.svg new file mode 100644 index 0000000..e6af019 --- /dev/null +++ b/docs/vocabulary-broadleaf.svg @@ -0,0 +1,92 @@ + +codegraph vocabulary — corpus BroadleafCommerce + + + +order +item +customer +offer +product +admin +group +fulfillment +category +payment +price +cache +site +persistence +sku +date +address +broadleaf +code +all +option +filter +override +log +metadata +page +form +message +rule +search +processor +criteria +active +password +friendly +email +list +cart +properties +items +target +file +section +path +template +content +query +tab +user +status +state +resource +sand +total +instance +phone +parent +collection +fields +locale +catalog +quantity +view +credit +currency +sale +additional +attribute +display +custom +validate +asset +box +retrieve +module +current +bundle +security +transaction +execute + + +corpus BroadleafCommerce · domain: 80 of 3191 terms by weight, size on √weight +weights module 8 · type 8 · operation 3 · attribute 4 · parameter 1 · local 1 · other 1 · qualifier 1 +view all · technical: 138 shared words (sha256 4f34e69f52f5) + each language's own + + diff --git a/fixtures/java/expected/vocabulary.svg b/fixtures/java/expected/vocabulary.svg new file mode 100644 index 0000000..797cd49 --- /dev/null +++ b/fixtures/java/expected/vocabulary.svg @@ -0,0 +1,52 @@ + +codegraph vocabulary — corpus src + + + +order +price +basket +channel +discount +ledger +cursor +lines +total +reference +amount +line +audited +batch +cents +discountable +empty +guard +head +legacy +list +money +notifications +priceable +reporting +stock +trail +max +running +shipped +archive +counts +prices +tail +first +phone +store +web +acme +allocator + + +corpus src · domain: 40 of 73 terms by weight, size on √weight +weights module 8 · type 8 · operation 3 · attribute 4 · parameter 1 · local 1 · other 1 · qualifier 1 +view all · technical: 138 shared words (sha256 4f34e69f52f5) + each language's own + + diff --git a/packages/cli/src/args.ts b/packages/cli/src/args.ts index bb55e7b..86da1d5 100644 --- a/packages/cli/src/args.ts +++ b/packages/cli/src/args.ts @@ -440,6 +440,12 @@ export const VOCABULARY_SPEC: CommandSpec = { describe: "Also write the vocabulary.json artifact to this file.", placeholder: "FILE", }, + { + name: "svg", + type: "string", + describe: "Also draw one scope's cloud to this SVG file: the corpus, or the --scope one; --class and --top pick its words.", + placeholder: "FILE", + }, JSON_OPTION, ...VIEW_OPTIONS, NO_CACHE_OPTION, @@ -1236,6 +1242,8 @@ export interface VocabularyOptions extends ModelInputOptions, ViewOptions, Cache readonly drift: string | undefined; /** `--out FILE`: the artifact, beside the report. */ readonly out: string | undefined; + /** `--svg FILE`: the cloud of the one scope reported, beside the report. */ + readonly svg: string | undefined; readonly json: boolean; } @@ -2136,6 +2144,12 @@ function vocabularyOf(values: ParsedValues): Omit { : "--drift compares the business terms with every word, and lists the domain words no term contains.", ); } + const svg = stringOf(values, "svg"); + const level = stringOf(values, "level"); + const many = drift !== undefined ? "--drift" : level !== undefined && level !== "corpus" ? `--level ${level}` : undefined; + if (svg !== undefined && many !== undefined) { + throw new UsageError(`--svg does not apply with ${many}`, "--svg draws one scope's cloud: the corpus, or the one --scope names."); + } return { level: (stringOf(values, "level") ?? "corpus") as ScopeLevel, scope, @@ -2145,6 +2159,7 @@ function vocabularyOf(values: ParsedValues): Omit { technical: stringOf(values, "technical"), drift, out: stringOf(values, "out"), + svg, json: flagOf(values, "json"), ...viewOf(values), noCache: flagOf(values, "no-cache"), diff --git a/packages/cli/src/commands/vocabulary.ts b/packages/cli/src/commands/vocabulary.ts index 9df86e4..4440828 100644 --- a/packages/cli/src/commands/vocabulary.ts +++ b/packages/cli/src/commands/vocabulary.ts @@ -8,6 +8,7 @@ import { operationScope, parseBusinessTerms, parseTechnicalWords, + vocabularyCloudSvg, vocabularyDrift, vocabularyToJsonString, type BusinessTerms, @@ -32,7 +33,8 @@ import { resolveView, viewLabel } from "../view.js"; * A SHELL over `@codegraph/vocabulary`, like `analyze` over the analyzer: * every count printed here is the artifact's, or an operation's counted by * the package on demand. This command picks the scope, the class and the - * report, and writes the artifact on `--out` beside the report. + * report, and writes the artifact on `--out` and the cloud on `--svg` beside + * the report. * * The header IS part of the report: a count without its view, its weights and * its technical list is not a fact, so all three are stated on stdout. @@ -82,6 +84,9 @@ export function vocabularyCommand(options: VocabularyOptions, io: IoSink): ExitC : [scopeOf(graph, artifact, options.scope, counting)]; const top = options.top ?? (options.scope === undefined && options.level !== "corpus" ? 10 : 40); outLines(io, wordsReport(artifact, scopes, header, options, top)); + // The parser let --svg through with one scope only: the corpus, or --scope's. + const [one] = scopes; + if (options.svg !== undefined && one !== undefined) writeCloud(io, options.svg, artifact, one, options.termClass, top); } if (options.out !== undefined) { @@ -359,6 +364,23 @@ function scopeJson( return { ...counts, ...(around === undefined ? {} : { parent: around }), shown: rows.length, total, terms: rows }; } +function writeCloud( + io: IoSink, + path: string, + artifact: VocabularyArtifact, + scope: AnyScope, + termClass: VocabularyOptions["termClass"], + top: number, +): void { + const svg = vocabularyCloudSvg(artifact, scope, { termClass, top }); + io.writeFile(path, svg); + const { rows, total } = rowsOfClass(artifact, scope, termClass, top); + const what = termClass === "all" ? "" : `${termClass} `; + const count = rows.length === total ? plural(total, `${what}term`) : `${rows.length} of ${total} ${what}terms`; + const label = scope.level === "corpus" ? `corpus ${scope.name}` : `${scope.level} ${scope.name}`; + errLine(io, `wrote ${plural(Buffer.byteLength(svg, "utf8"), "byte")} to ${path} (cloud of ${label}: ${count}).`); +} + /* ------------------------------------------------------------ drift report */ function neverSays(level: ScopeLevel): string { diff --git a/packages/cli/test/args.test.ts b/packages/cli/test/args.test.ts index 35c508a..d569656 100644 --- a/packages/cli/test/args.test.ts +++ b/packages/cli/test/args.test.ts @@ -312,6 +312,7 @@ describe("vocabulary", () => { technical: undefined, drift: undefined, out: undefined, + svg: undefined, json: false, internalOnly: false, declaredOnly: false, @@ -342,6 +343,9 @@ describe("vocabulary", () => { expect(parse(["vocabulary", "a.json", "--scope", "java:com.acme/Order", "--class", "all"])).toMatchObject({ options: { scope: "java:com.acme/Order", termClass: "all" }, }); + expect(parse(["vocabulary", "a.json", "--svg", "cloud.svg", "--level", "corpus"])).toMatchObject({ + options: { svg: "cloud.svg", level: "corpus" }, + }); }); it("offers the package's levels and classes, not a copy", () => { @@ -386,6 +390,19 @@ describe("vocabulary", () => { expect(error.message, argv.join(" ")).toContain(flag); } }); + + // A cloud is one scope's: the corpus, or the one --scope names. + it("refuses --svg with a report of many scopes", () => { + for (const [argv, message] of [ + [["--level", "module"], "--svg does not apply with --level module"], + [["--level", "type"], "--svg does not apply with --level type"], + [["--drift", "g.txt"], "--svg does not apply with --drift"], + ] as const) { + const error = usageErrorFor(["vocabulary", "a.json", "--svg", "c.svg", ...argv]); + expect(error.message).toBe(message); + expect(error.hint ?? "").toContain("--scope"); + } + }); }); describe("parseArgs errors are re-said in codegraph's terms", () => { diff --git a/packages/cli/test/cache.test.ts b/packages/cli/test/cache.test.ts index 9e53223..f7c3dfd 100644 --- a/packages/cli/test/cache.test.ts +++ b/packages/cli/test/cache.test.ts @@ -264,6 +264,7 @@ describe("vocabulary through the store", () => { technical: undefined, drift: undefined, out: undefined, + svg: undefined, json: true, internalOnly: false, declaredOnly: false, diff --git a/packages/cli/test/vocabulary.test.ts b/packages/cli/test/vocabulary.test.ts index f617e10..75cddec 100644 --- a/packages/cli/test/vocabulary.test.ts +++ b/packages/cli/test/vocabulary.test.ts @@ -4,7 +4,7 @@ import { join } from "node:path"; import { fileURLToPath } from "node:url"; import { readModelFileSync } from "@codegraph/core"; import { buildGraph, loadModels } from "@codegraph/analyzer"; -import { buildVocabulary, vocabularyToJsonString } from "@codegraph/vocabulary"; +import { buildVocabulary, operationScope, vocabularyCloudSvg, vocabularyToJsonString } from "@codegraph/vocabulary"; import { describe, expect, it } from "vitest"; import type { VocabularyOptions } from "../src/args.js"; import { vocabularyCommand } from "../src/commands/vocabulary.js"; @@ -32,6 +32,7 @@ function options(overrides: Partial = {}): VocabularyOptions technical: undefined, drift: undefined, out: undefined, + svg: undefined, json: false, internalOnly: false, declaredOnly: false, @@ -384,6 +385,37 @@ describe("vocabulary: --out", () => { }); }); +describe("vocabulary: --svg", () => { + const graph = buildGraph(loadModels(readModelFileSync(FIXTURE), { sources: [FIXTURE] }).union); + const artifact = buildVocabulary(graph); + const svgPath = (): string => join(mkdtempSync(join(tmpdir(), "codegraph-cli-vocabulary-")), "cloud.svg"); + + it("draws the corpus cloud beside the report, and confirms on stderr", () => { + const path = svgPath(); + const { io, code } = vocabularyTo({ svg: path }); + expect(code).toBe(EXIT.OK); + expect(rows(io.stdout()).length).toBe(40); + expect(io.files().get(path)).toBe(vocabularyCloudSvg(artifact, artifact.scopes[0]!)); + expect(io.stderr()).toMatch(/wrote \d+ bytes to .*cloud\.svg \(cloud of corpus src: 40 of 73 domain terms\)\./u); + }); + + it("draws the scope --scope names, with the class and the top it is given", () => { + const path = svgPath(); + const { io } = vocabularyTo({ svg: path, scope: ABSTRACT_ORDER, termClass: "all", top: 3 }); + const scope = artifact.scopes.find((candidate) => candidate.id === ABSTRACT_ORDER)!; + expect(io.files().get(path)).toBe(vocabularyCloudSvg(artifact, scope, { termClass: "all", top: 3 })); + expect(io.stderr()).toContain("(cloud of type AbstractOrder: 3 of 5 terms)."); + }); + + it("draws an operation counted on demand", () => { + const path = svgPath(); + const { io } = vocabularyTo({ svg: path, scope: `${ABSTRACT_ORDER}.price()` }); + const scope = operationScope(graph, artifact, `${ABSTRACT_ORDER}.price()`, {})!; + expect(io.files().get(path)).toBe(vocabularyCloudSvg(artifact, scope)); + expect(io.stderr()).toContain("(cloud of operation price: 1 domain term)."); + }); +}); + describe("vocabulary: through the real dispatcher", () => { it("runs from argv", () => { const io = captureIo(); diff --git a/packages/vocabulary/src/cloud.ts b/packages/vocabulary/src/cloud.ts new file mode 100644 index 0000000..ac75c9a --- /dev/null +++ b/packages/vocabulary/src/cloud.ts @@ -0,0 +1,154 @@ +import type { TermClass } from "./classify.js"; + +/** + * THE CLOUD'S GEOMETRY (PLAN.md §20.2): pure, no randomness, so a cloud is a + * function of its words — the CLI's SVG and the tab's page run this one + * function and differ only by the measurer they hand it. + * + * - WEIGHT ORDER. Words by weight descending, ties by text then class (code + * units): the heaviest word takes the centre, and the same words lay out + * the same whatever order they arrive in. + * - AREA READS AS MAGNITUDE. The font size is interpolated on the square + * root of the weight, between `minSize` and `maxSize`, rounded to a pixel. + * - FIRST FIT ON A SPIRAL. Each box walks an Archimedean spiral out from the + * centre, stretched by `aspect`, and stops at the first point where its + * padded box overlaps no box placed before it. There is no fixed canvas: + * the spiral leaves every placed box behind eventually, and the canvas is + * whatever the words cover. + * + * This module imports types only: a page can run it without the Node + * pipeline behind `aggregate.ts`. + */ + +export interface CloudWord { + readonly text: string; + readonly weight: number; + /** Carried for the renderer, which colours by it; the geometry never reads it. */ + readonly class: TermClass; +} + +/** The extent of `text` drawn at `size` px: the CLI's monospace default, or a page's `measureText`. */ +export type Measure = (text: string, size: number) => { readonly width: number; readonly height: number }; + +export interface CloudSettings { + readonly minSize: number; + readonly maxSize: number; + /** Kept clear around every word, on each side. */ + readonly padding: number; + /** The spiral's horizontal stretch: 2 lays a cloud out twice as wide as tall. */ + readonly aspect: number; +} + +export interface CloudOptions extends Partial { + readonly measure?: Measure; +} + +/** A placed word; `(x, y)` is the CENTRE of its box, `width` × `height` its measured extent. */ +export interface PlacedWord extends CloudWord { + readonly size: number; + readonly x: number; + readonly y: number; + readonly width: number; + readonly height: number; +} + +/** The canvas spans `[0, width] × [0, height]`; every padded box lies within. */ +export interface CloudLayout { + readonly words: readonly PlacedWord[]; + readonly width: number; + readonly height: number; + readonly options: CloudSettings; +} + +export const CLOUD_DEFAULTS: CloudSettings = Object.freeze({ minSize: 12, maxSize: 56, padding: 4, aspect: 2 }); + +/** The em fractions `monospaceMeasure` assumes: exact for the monospace faces the SVG declares, to a percent. */ +export const MONOSPACE_ADVANCE = 0.6; +export const MONOSPACE_LINE = 1.2; + +const MARK = /\p{M}/u; +const WIDE = /[\p{Script=Han}\p{Script=Hiragana}\p{Script=Katakana}\p{Script=Hangul}\u{FF01}-\u{FF60}\u{FFE0}-\u{FFE6}]/u; + +/** One cell per code point, none for a combining mark, two for a wide character — as a terminal draws them. */ +export const monospaceMeasure: Measure = (text, size) => { + let cells = 0; + for (const char of text) cells += MARK.test(char) ? 0 : WIDE.test(char) ? 2 : 1; + return { width: MONOSPACE_ADVANCE * size * cells, height: MONOSPACE_LINE * size }; +}; + +/** Radians between two points tried on the spiral; its radius grows one pixel per radian. */ +const SPIRAL_STEP = 0.1; +/** A finite box always finds room this side of it; past it, something is not a box. */ +const MAX_STEPS = 10_000_000; + +interface Box { + readonly x0: number; + readonly y0: number; + readonly x1: number; + readonly y1: number; +} + +export function layoutCloud(words: readonly CloudWord[], options: CloudOptions = {}): CloudLayout { + const { measure = monospaceMeasure, ...given } = options; + const settings: CloudSettings = { ...CLOUD_DEFAULTS, ...given }; + for (const word of words) { + if (!Number.isFinite(word.weight) || word.weight < 0) { + throw new RangeError(`a cloud word's weight is a finite number >= 0; ${word.text} has ${word.weight}`); + } + } + const ordered = [...words].sort( + (a, b) => b.weight - a.weight || compareUnits(a.text, b.text) || compareUnits(a.class, b.class), + ); + const size = sizer(ordered, settings); + + const boxes: Box[] = []; + const centred = ordered.map((word) => { + const px = size(word.weight); + const { width, height } = measure(word.text, px); + if (!Number.isFinite(width) || !Number.isFinite(height) || width < 0 || height < 0) { + throw new RangeError(`the measure gave ${width} × ${height} for ${word.text} at ${px} px; a box is finite and >= 0`); + } + const halfW = width / 2 + settings.padding; + const halfH = height / 2 + settings.padding; + for (let step = 0; step < MAX_STEPS; step++) { + const theta = step * SPIRAL_STEP; + const x = Math.round(settings.aspect * theta * Math.cos(theta)); + const y = Math.round(theta * Math.sin(theta)); + const candidate = { x0: x - halfW, y0: y - halfH, x1: x + halfW, y1: y + halfH }; + if (boxes.some((placed) => intersects(placed, candidate))) continue; + boxes.push(candidate); + return { ...word, size: px, x, y, width, height }; + } + throw new RangeError(`no room found for ${word.text} in ${MAX_STEPS} steps of the spiral`); + }); + + if (boxes.length === 0) return { words: [], width: 0, height: 0, options: settings }; + const x0 = Math.min(...boxes.map((b) => b.x0)); + const y0 = Math.min(...boxes.map((b) => b.y0)); + const x1 = Math.max(...boxes.map((b) => b.x1)); + const y1 = Math.max(...boxes.map((b) => b.y1)); + return { + words: centred.map((word) => ({ ...word, x: word.x - x0, y: word.y - y0 })), + width: x1 - x0, + height: y1 - y0, + options: settings, + }; +} + +/** Weight → font size on the square root, rounded to a pixel; one weight throughout is drawn at the maximum. */ +function sizer(ordered: readonly CloudWord[], settings: CloudSettings): (weight: number) => number { + const high = Math.sqrt(ordered[0]?.weight ?? 0); + const low = Math.sqrt(ordered[ordered.length - 1]?.weight ?? 0); + const span = settings.maxSize - settings.minSize; + if (high === low) return () => settings.maxSize; + return (weight) => Math.round(settings.minSize + (span * (Math.sqrt(weight) - low)) / (high - low)); +} + +/** Touching is not overlapping. */ +function intersects(a: Box, b: Box): boolean { + return a.x0 < b.x1 && b.x0 < a.x1 && a.y0 < b.y1 && b.y0 < a.y1; +} + +function compareUnits(a: string, b: string): number { + return a < b ? -1 : a > b ? 1 : 0; +} diff --git a/packages/vocabulary/src/index.ts b/packages/vocabulary/src/index.ts index a919578..6ca4f66 100644 --- a/packages/vocabulary/src/index.ts +++ b/packages/vocabulary/src/index.ts @@ -12,3 +12,5 @@ export * from "./weight.js"; export * from "./artifact.js"; export * from "./aggregate.js"; export * from "./drift.js"; +export * from "./cloud.js"; +export * from "./svg.js"; diff --git a/packages/vocabulary/src/svg.ts b/packages/vocabulary/src/svg.ts new file mode 100644 index 0000000..0839343 --- /dev/null +++ b/packages/vocabulary/src/svg.ts @@ -0,0 +1,116 @@ +import type { OperationScope, VocabularyArtifact, VocabularyScope } from "./artifact.js"; +import { TERM_CLASSES, type TermClass } from "./classify.js"; +import { layoutCloud, monospaceMeasure, type CloudOptions, type CloudWord } from "./cloud.js"; +import { WEIGHT_KEYS } from "./weight.js"; + +/** + * A scope's cloud as an SVG document (PLAN.md §20.2), for the CLI's `--svg` + * and the committed demo image. + * + * MEANING CONTROLS APPEARANCE. Size is the weight (through the layout) and + * colour is the class, through one class attribute per word and a stylesheet: + * `domain` in ink, `technical` muted, `keyword` and `short` faint, in a light + * palette and a dark one. Nothing else varies. The legend says what was + * counted — scope, class filter, counts, weights, view, technical list — since + * a cloud without them is a picture, not a fact. + * + * DETERMINISTIC BYTES: the font is monospace, so the monospace measurer is + * the one the layout runs with, and every coordinate is rounded to 1/100 px. + */ + +export interface CloudSvgOptions extends Omit { + /** Which words to draw: one class, or `all` coloured by class. */ + readonly termClass?: TermClass | "all"; + /** The N heaviest of that class. */ + readonly top?: number; +} + +export const CLOUD_TOP = 40; + +const FONT = "ui-monospace, SFMono-Regular, Menlo, Consolas, 'Liberation Mono', monospace"; +const STYLE = + ".bg{fill:#ffffff}.domain{fill:#1f2328}.technical{fill:#59636e}.keyword,.short{fill:#9198a1}.legend{fill:#59636e}" + + ".cloud text{text-anchor:middle;dominant-baseline:central}" + + "@media (prefers-color-scheme:dark){.bg{fill:#0d1117}.domain{fill:#f0f6fc}.technical{fill:#9198a1}" + + ".keyword,.short{fill:#59636e}.legend{fill:#9198a1}}"; +const MARGIN = 16; +const GAP = 12; +const LEGEND_SIZE = 12; +const LEGEND_LINE = 18; + +export function vocabularyCloudSvg( + artifact: VocabularyArtifact, + scope: VocabularyScope | OperationScope, + options: CloudSvgOptions = {}, +): string { + const { termClass = "domain", top = CLOUD_TOP, ...cloud } = options; + const matching: CloudWord[] = []; + for (const [term, weight] of scope.terms) { + const entry = artifact.terms[term]; + if (entry !== undefined && (termClass === "all" || entry.class === termClass)) { + matching.push({ text: entry.text, weight, class: entry.class }); + } + } + const layout = layoutCloud(matching.slice(0, top), cloud); + const label = scope.level === "corpus" ? `corpus ${scope.name}` : `${scope.level} ${scope.name}`; + + const what = termClass === "all" ? "all classes" : termClass; + const shown = layout.words.length; + const counts = + matching.length === 0 + ? "no term" + : `${shown === matching.length ? plural(shown, "term") : `${shown} of ${matching.length} terms`} by weight, size on √weight`; + const classes = TERM_CLASSES.filter((cls) => layout.words.some((word) => word.class === cls)); + const legend: { readonly plain: string; readonly markup: string }[] = [ + `${label} · ${what}: ${counts}`, + `weights ${WEIGHT_KEYS.map((key) => `${key} ${artifact.weights[key]}`).join(" · ")}`, + `view ${artifact.view.name} · technical: ${plural(artifact.technical.words, "shared word")} ` + + `(sha256 ${artifact.technical.sha256.slice(0, 12)}) + each language's own`, + ].map((line) => ({ plain: line, markup: escape(line) })); + if (classes.length > 1) { + legend.push({ + plain: classes.join(" · "), + markup: classes.map((cls) => `${cls}`).join(" · "), + }); + } + + const legendWidth = Math.max(...legend.map((line) => monospaceMeasure(line.plain, LEGEND_SIZE).width)); + const width = Math.max(layout.width, legendWidth) + 2 * MARGIN; + const left = MARGIN + (width - 2 * MARGIN - layout.width) / 2; + const legendTop = MARGIN + layout.height + (layout.height > 0 ? GAP : 0); + const height = legendTop + LEGEND_LINE * legend.length + MARGIN; + + const lines = [ + ``, + `codegraph vocabulary — ${escape(label)}`, + ``, + ``, + ``, + ...layout.words.map( + (word) => + `${escape(word.text)}`, + ), + ``, + ``, + ...legend.map( + (line, i) => `${line.markup}`, + ), + ``, + ``, + ]; + return `${lines.join("\n")}\n`; +} + +/** 1/100 px, and never `-0`. */ +function num(value: number): string { + const rounded = Math.round(value * 100) / 100; + return String(rounded === 0 ? 0 : rounded); +} + +function escape(text: string): string { + return text.replace(/&/gu, "&").replace(//gu, ">").replace(/"/gu, """); +} + +function plural(count: number, noun: string): string { + return `${count} ${noun}${count === 1 ? "" : "s"}`; +} diff --git a/packages/vocabulary/test/cloud.test.ts b/packages/vocabulary/test/cloud.test.ts new file mode 100644 index 0000000..c1c80a0 --- /dev/null +++ b/packages/vocabulary/test/cloud.test.ts @@ -0,0 +1,183 @@ +import fc from "fast-check"; +import { describe, expect, it } from "vitest"; +import { + CLOUD_DEFAULTS, + layoutCloud, + monospaceMeasure, + type CloudLayout, + type CloudWord, + type Measure, + type PlacedWord, +} from "../src/cloud.js"; +import { TERM_CLASSES } from "../src/classify.js"; + +/** + * The cloud's geometry (PLAN.md §20.2): pure, no randomness. Terms by weight, + * sized on the square root of the weight, each placed at the first point of a + * spiral from the centre where its box overlaps nothing placed before it. + */ + +const word = (text: string, weight: number, cls: CloudWord["class"] = "domain"): CloudWord => ({ text, weight, class: cls }); + +/** A word's box with its padding: what the layout keeps apart. */ +function box(placed: PlacedWord, padding: number): { x0: number; y0: number; x1: number; y1: number } { + return { + x0: placed.x - placed.width / 2 - padding, + y0: placed.y - placed.height / 2 - padding, + x1: placed.x + placed.width / 2 + padding, + y1: placed.y + placed.height / 2 + padding, + }; +} + +// Touching is not overlapping; the epsilon absorbs the shift to the origin. +const EPSILON = 1e-9; + +function overlaps(a: PlacedWord, b: PlacedWord, padding: number): boolean { + const p = box(a, padding); + const q = box(b, padding); + return p.x0 < q.x1 - EPSILON && q.x0 < p.x1 - EPSILON && p.y0 < q.y1 - EPSILON && q.y0 < p.y1 - EPSILON; +} + +describe("the monospace measurer", () => { + it("advances 0.6 em per code point on a 1.2 em line", () => { + expect(monospaceMeasure("order", 10)).toEqual({ width: 30, height: 12 }); + }); + + // A combining mark takes no cell; a CJK ideograph or a Hangul syllable takes + // two, as a terminal draws it: the box covers what the font draws. + it("counts a mark as nothing and a wide character as two", () => { + expect(monospaceMeasure("été", 10).width).toBe(18); + expect(monospaceMeasure("注文", 10).width).toBe(24); + expect(monospaceMeasure("주문", 10).width).toBe(24); + expect(monospaceMeasure("𠀀", 10).width).toBe(12); + }); +}); + +describe("layoutCloud", () => { + it("lays out nothing as an empty canvas", () => { + expect(layoutCloud([])).toMatchObject({ words: [], width: 0, height: 0 }); + }); + + it("puts one word at the maximum size in the middle of a canvas that just holds it", () => { + const { words, width, height } = layoutCloud([word("order", 7)]); + const { maxSize, padding } = CLOUD_DEFAULTS; + const { width: w, height: h } = monospaceMeasure("order", maxSize); + expect(words).toEqual([{ text: "order", weight: 7, class: "domain", size: maxSize, x: w / 2 + padding, y: h / 2 + padding, width: w, height: h }]); + expect([width, height]).toEqual([w + 2 * padding, h + 2 * padding]); + }); + + // Area reads as magnitude, so the size follows the square root of the weight: + // 100, 25 and 1 are √ 10, 5 and 1, four ninths of the way from 10 px to 20 px. + it("sizes on the square root of the weight, between the minimum and the maximum", () => { + const { words } = layoutCloud([word("a", 100), word("b", 25), word("c", 1)], { minSize: 10, maxSize: 20 }); + expect(words.map((placed) => placed.size)).toEqual([20, 14, 10]); + }); + + it("gives every word the maximum size when the weights are equal", () => { + const { words } = layoutCloud([word("a", 3), word("b", 3)], { minSize: 10, maxSize: 20 }); + expect(words.map((placed) => placed.size)).toEqual([20, 20]); + }); + + it("places by weight descending, ties by text then class", () => { + const { words } = layoutCloud([word("zeta", 1), word("beta", 5), word("alpha", 5, "technical"), word("alpha", 5, "domain")]); + expect(words.map((placed) => `${placed.text}/${placed.class}`)).toEqual(["alpha/domain", "alpha/technical", "beta/domain", "zeta/domain"]); + }); + + it("measures with the measurer it is handed", () => { + const measure: Measure = (text, size) => ({ width: text.length * size, height: size }); + const { words } = layoutCloud([word("order", 4), word("line", 1)], { measure, minSize: 10, maxSize: 20 }); + expect(words.map((placed) => [placed.text, placed.width, placed.height])).toEqual([ + ["order", 100, 20], + ["line", 40, 10], + ]); + }); + + it("refuses a weight or a measure it could not place", () => { + expect(() => layoutCloud([word("order", -1)])).toThrow(/weight/u); + expect(() => layoutCloud([word("order", Number.NaN)])).toThrow(/weight/u); + const infinite: Measure = () => ({ width: Number.POSITIVE_INFINITY, height: 10 }); + expect(() => layoutCloud([word("order", 1)], { measure: infinite })).toThrow(/measure/u); + const negative: Measure = () => ({ width: -1, height: 10 }); + expect(() => layoutCloud([word("order", 1)], { measure: negative })).toThrow(/measure/u); + }); + + it("records the options it laid out with", () => { + expect(layoutCloud([word("order", 1)], { maxSize: 30 }).options).toEqual({ ...CLOUD_DEFAULTS, maxSize: 30 }); + }); +}); + +describe("properties", () => { + const text = fc.stringMatching(/^[a-zé注𠀀]{1,10}$/u); + const cloudWord = fc.record({ text, weight: fc.integer({ min: 0, max: 5000 }), class: fc.constantFrom(...TERM_CLASSES) }); + // One entry per (text, class), as a scope's rows are. + const words = fc.uniqueArray(cloudWord, { maxLength: 40, selector: (w) => `${w.text}/${w.class}` }); + const options = fc.record({ + minSize: fc.integer({ min: 4, max: 20 }), + span: fc.integer({ min: 0, max: 60 }), + padding: fc.integer({ min: 0, max: 4 }), + aspect: fc.double({ min: 0.5, max: 3, noNaN: true }), + }); + const cloud = (ws: readonly CloudWord[], o: { minSize: number; span: number; padding: number; aspect: number }): CloudLayout => + layoutCloud(ws, { minSize: o.minSize, maxSize: o.minSize + o.span, padding: o.padding, aspect: o.aspect }); + + it("never lets two boxes overlap", () => { + fc.assert( + fc.property(words, options, (ws, o) => { + const placed = cloud(ws, o).words; + for (let i = 0; i < placed.length; i++) { + for (let j = i + 1; j < placed.length; j++) { + expect(overlaps(placed[i]!, placed[j]!, o.padding), `${placed[i]?.text} × ${placed[j]?.text}`).toBe(false); + } + } + }), + { numRuns: 60 }, + ); + }); + + it("keeps every box, padding included, inside the canvas it reports", () => { + fc.assert( + fc.property(words, options, (ws, o) => { + const layout = cloud(ws, o); + for (const placed of layout.words) { + const b = box(placed, o.padding); + expect(b.x0).toBeGreaterThanOrEqual(-EPSILON); + expect(b.y0).toBeGreaterThanOrEqual(-EPSILON); + expect(b.x1).toBeLessThanOrEqual(layout.width + EPSILON); + expect(b.y1).toBeLessThanOrEqual(layout.height + EPSILON); + } + }), + { numRuns: 60 }, + ); + }); + + it("lays out the same words the same way, whatever order they arrive in", () => { + fc.assert( + fc.property( + words.chain((ws) => fc.tuple(fc.constant(ws), fc.shuffledSubarray(ws, { minLength: ws.length }))), + options, + ([ws, shuffled], o) => { + expect(cloud(shuffled, o)).toEqual(cloud(ws, o)); + }, + ), + { numRuns: 60 }, + ); + }); + + it("places by weight, and a heavier word is never the smaller", () => { + fc.assert( + fc.property(words, options, (ws, o) => { + const placed = cloud(ws, o).words; + expect(placed.map((w) => w.text).sort()).toEqual(ws.map((w) => w.text).sort()); + for (let i = 1; i < placed.length; i++) { + expect(placed[i]!.weight).toBeLessThanOrEqual(placed[i - 1]!.weight); + expect(placed[i]!.size).toBeLessThanOrEqual(placed[i - 1]!.size); + } + for (const w of placed) { + expect(w.size).toBeGreaterThanOrEqual(o.minSize); + expect(w.size).toBeLessThanOrEqual(o.minSize + o.span); + } + }), + { numRuns: 60 }, + ); + }); +}); diff --git a/packages/vocabulary/test/fixtures.test.ts b/packages/vocabulary/test/fixtures.test.ts index 110419d..c1a793e 100644 --- a/packages/vocabulary/test/fixtures.test.ts +++ b/packages/vocabulary/test/fixtures.test.ts @@ -5,6 +5,7 @@ import { buildGraph, loadModels, type CodeGraph } from "@codegraph/analyzer"; import { describe, expect, it } from "vitest"; import { buildVocabulary } from "../src/aggregate.js"; import { vocabularyToJsonString } from "../src/artifact.js"; +import { vocabularyCloudSvg } from "../src/svg.js"; import { expectClosed } from "./closure.js"; /** @@ -45,6 +46,21 @@ describe.each(FIXTURES)("fixtures/%s", (fixture) => { }); }); +// The corpus cloud of one fixture, byte for byte (PLAN.md §20.2): the layout, +// the sizes, the classes and the legend in one reviewable file. +describe("fixtures/java/expected/vocabulary.svg", () => { + const artifact = buildVocabulary(graphOf("java")); + const corpus = artifact.scopes[0]; + const svg = corpus === undefined ? "" : vocabularyCloudSvg(artifact, corpus); + const snapshot = path("java", "vocabulary.svg"); + if (process.env["CODEGRAPH_UPDATE_SNAPSHOTS"] !== undefined) writeFileSync(snapshot, svg); + + it("reproduces the corpus cloud exactly", () => { + expect(existsSync(snapshot), `${snapshot} is committed`).toBe(true); + expect(svg).toBe(readFileSync(snapshot, "utf8")); + }); +}); + // The words each fixture's README counts by hand, name by name. A snapshot // can be rewritten in one command; these cannot, so a change to the split, the // qualifiers or the weights has to be argued against a count a person made. diff --git a/packages/vocabulary/test/svg.test.ts b/packages/vocabulary/test/svg.test.ts new file mode 100644 index 0000000..3f31811 --- /dev/null +++ b/packages/vocabulary/test/svg.test.ts @@ -0,0 +1,134 @@ +import { describe, expect, it } from "vitest"; +import type { VocabularyArtifact, VocabularyScope, VocabularyTerm } from "../src/artifact.js"; +import { layoutCloud } from "../src/cloud.js"; +import { vocabularyCloudSvg } from "../src/svg.js"; +import { DEFAULT_WEIGHTS } from "../src/weight.js"; + +/** + * The cloud as an SVG (PLAN.md §20.2): size by weight, colour by class and + * nothing else, a legend that says what was counted, deterministic bytes. + */ + +const TERMS: readonly VocabularyTerm[] = [ + { text: "get", class: "technical", modules: 2, types: 3 }, + { text: "line", class: "domain", modules: 1, types: 2 }, + { text: "new", class: "keyword", modules: 1, types: 1 }, + { text: "order", class: "domain", modules: 2, types: 4 }, + { text: "x", class: "short", modules: 1, types: 1 }, + { text: "x<&\">", class: "domain", modules: 1, types: 1 }, +]; + +const ROLES = { module: 0, type: 0, operation: 0, attribute: 0, parameter: 0, local: 0, other: 0 }; + +function scope(rows: VocabularyScope["terms"], extra: Partial = {}): VocabularyScope { + return { + level: "corpus", + name: "shop", + entities: 9, + roles: ROLES, + skipped: { unnamed: 0, conventional: 0, numeric: 0 }, + distinct: rows.length, + distinctDomain: rows.filter(([term]) => TERMS[term]?.class === "domain").length, + weight: 0, + occurrences: 0, + terms: rows, + ...extra, + }; +} + +const CORPUS = scope([ + [3, 40, 12, 12], + [0, 21, 9, 9], + [1, 16, 3, 3], + [2, 3, 1, 1], + [4, 1, 1, 1], +]); + +const ARTIFACT: VocabularyArtifact = { + kind: "codegraph.vocabulary/1", + generatedBy: "@codegraph/vocabulary", + view: { name: "all", filters: [] }, + corpus: { name: "shop", roots: ["shop/src"] }, + langs: ["java"], + weights: DEFAULT_WEIGHTS, + classes: ["domain", "technical", "keyword", "short"], + technical: { words: 138, sha256: "4f34e69f52f5aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" }, + terms: TERMS, + scopes: [CORPUS], + diagnostics: { unplaced: [], withoutLexicon: [] }, +}; + +/** Each drawn word, in document order: its class attribute, its size, its text. */ +function drawn(svg: string): readonly (readonly [cls: string, size: string, text: string])[] { + return [...svg.matchAll(/([^<]*)<\/text>/gu)].map( + (match) => [match[1] ?? "", match[2] ?? "", match[3] ?? ""] as const, + ); +} + +describe("vocabularyCloudSvg", () => { + it("draws the scope's domain words by weight, each at the size the layout gave it", () => { + const svg = vocabularyCloudSvg(ARTIFACT, CORPUS); + const layout = layoutCloud([ + { text: "order", weight: 40, class: "domain" }, + { text: "line", weight: 16, class: "domain" }, + ]); + expect(drawn(svg)).toEqual(layout.words.map((word) => ["domain", String(word.size), word.text])); + }); + + // Meaning controls appearance: the class is the only thing a word's colour + // says, and it says it through one class attribute — no inline fill. + it("colours by class and nothing else", () => { + const svg = vocabularyCloudSvg(ARTIFACT, CORPUS, { termClass: "all" }); + expect(drawn(svg).map(([cls, , text]) => `${text}:${cls}`)).toEqual([ + "order:domain", + "get:technical", + "line:domain", + "new:keyword", + "x:short", + ]); + expect(svg).not.toMatch(/]* (fill|style)=/u); + for (const cls of ["domain", "technical", "keyword", "short"]) expect(svg).toMatch(new RegExp(`\\.${cls}[,{]`, "u")); + }); + + it("names the scope, the class filter, the counts, the weights, the view and the technical list", () => { + const svg = vocabularyCloudSvg(ARTIFACT, CORPUS, { top: 1 }); + expect(svg).toContain(">corpus shop · domain: 1 of 2 terms by weight, size on √weight"); + expect(svg).toContain(">weights module 8 · type 8 · operation 3 · attribute 4 · parameter 1 · local 1 · other 1 · qualifier 1"); + expect(svg).toContain(">view all · technical: 138 shared words (sha256 4f34e69f52f5) + each language's own"); + expect(svg).toContain("codegraph vocabulary — corpus shop"); + }); + + it("keys the colours when it draws more than one class", () => { + const key = /]*>domain<\/tspan> · technical<\/tspan> · keyword<\/tspan> · short<\/tspan><\/text>/u; + expect(vocabularyCloudSvg(ARTIFACT, CORPUS, { termClass: "all" })).toMatch(key); + expect(vocabularyCloudSvg(ARTIFACT, CORPUS)).not.toContain(" { + const module = scope([[3, 8, 1, 1]], { level: "module", id: "java:com.shop", name: "com.shop", kind: "package", parent: 0 }); + expect(vocabularyCloudSvg(ARTIFACT, module)).toContain(">module com.shop · domain: 1 term by weight, size on √weight"); + const operation = { ...module, level: "operation" as const, id: "java:com.shop/Order#total()", name: "total", kind: "method", parent: 0 }; + expect(vocabularyCloudSvg(ARTIFACT, operation)).toContain("codegraph vocabulary — operation total"); + }); + + it("escapes what it writes", () => { + const odd = scope([[5, 2, 1, 1]], { name: "a&b" }); + const svg = vocabularyCloudSvg(ARTIFACT, odd); + expect(drawn(svg).map(([, , text]) => text)).toEqual(["x<&">"]); + expect(svg).toContain("codegraph vocabulary — corpus a&b"); + }); + + it("says so when the scope speaks no word of the class", () => { + const svg = vocabularyCloudSvg(ARTIFACT, scope([[0, 21, 9, 9]])); + expect(drawn(svg)).toEqual([]); + expect(svg).toContain(">corpus shop · domain: no term"); + }); + + it("is the same bytes on every run, and a well-formed document", () => { + const svg = vocabularyCloudSvg(ARTIFACT, CORPUS, { termClass: "all" }); + expect(vocabularyCloudSvg(ARTIFACT, CORPUS, { termClass: "all" })).toBe(svg); + expect(svg).toMatch(/^\n")).toBe(true); + expect(svg).not.toMatch(/NaN|undefined|-0[^.\d]/u); + }); +}); diff --git a/website/content/docs/reference/cli/vocabulary.md b/website/content/docs/reference/cli/vocabulary.md index 4aa7b5b..3feed86 100644 --- a/website/content/docs/reference/cli/vocabulary.md +++ b/website/content/docs/reference/cli/vocabulary.md @@ -12,8 +12,8 @@ Every `TNamed` entity's name is split into words (`OrderLineRepository` → `ord ``` codegraph vocabulary [model.jsonl...] [--level ] [--scope ID] [--top N] [--class ] [--weights KEY=N,…] - [--technical FILE] [--drift FILE] [--out FILE] [--json] - [--internal-only] [--declared-only] [--no-cache] + [--technical FILE] [--drift FILE] [--out FILE] [--svg FILE] + [--json] [--internal-only] [--declared-only] [--no-cache] ``` ## Arguments @@ -34,6 +34,7 @@ codegraph vocabulary [model.jsonl...] [--level ] [--scope ID | `--technical FILE` | Replace the shared technical words with this file's, one word per line; each language's own still apply. | — | | `--drift FILE` | Compare with a glossary, one business term per line: the terms no scope of `--level` says (exit 3), and the domain words no term contains. | — | | `--out FILE` | Also write the [`vocabulary.json`](/docs/reference/artifacts/vocabulary-json/) artifact to this file. | — | +| `--svg FILE` | Also draw one scope's cloud to this SVG file: the corpus, or the `--scope` one; `--class` and `--top` pick its words. | — | | `--json` | Print the same information as a machine-readable JSON object on stdout. | — | | `--internal-only` | Drop stub (external) entities and every edge touching one. | — | | `--declared-only` | Keep only `declared` facts; drop derived and dynamic-candidate edges. | — | @@ -48,11 +49,15 @@ A type's words are those of everything that folds into it, a module's likewise, The glossary is a plain text file, one business term per line; blank lines and `#` comments are skipped. Each term is split by the same universal rules, so `Order Line` meets `order` and `line`. A term is *said* by a scope when every one of its words is, whatever class the code gives the word, so `--level module` asks for all of them in one module. Nothing is stemmed: `lines` is not `line`. The report lists the terms no scope says with their missing words, the terms it says and where, and the `domain` words the code says that no term contains. +## Cloud + +`--svg FILE` draws the cloud of the one scope the report shows, the corpus or the one `--scope` names, from the words `--class` and `--top` pick. A word's size follows the square root of its weight, so its area reads as the weight, and its colour says its class and nothing else: `domain` in ink, `technical` muted, `keyword` and `short` faint, with a dark palette for a dark page. Words are placed heaviest first, each at the first free point of a spiral out from the centre. The legend states the scope, the counts, the weights, the view and the technical list. The font is monospace, so the bytes are the same on every run and any machine. + ## Exit codes `0` ok · `1` internal error (a bug) · `2` usage error · `3` findings. -`3` when the load was not clean, or with `--drift` when a business term is said by no scope of the level. An unknown `--scope` id, an unknown `--weights` key or a `--technical` line that is not one word is a usage error. +`3` when the load was not clean, or with `--drift` when a business term is said by no scope of the level. An unknown `--scope` id, an unknown `--weights` key or a `--technical` line that is not one word is a usage error, and so is `--svg` with `--level module|type` or `--drift`: a cloud is one scope's. ## Example