feat(kg): per-type join policy and derived cross-layer package links by bawoodruff · Pull Request #10 · Cortexa-LLC/mcp

bawoodruff · 2026-08-28T03:04:57Z

Implements #7. Stacked on feat/kg-graph-linking-spec.

What changed

1. The join is keyed on entity type, not just a count. DefaultJoinTypes is package,import. A package name is chosen to be globally unique — that is what a package name is for. A function name is not. --join-types widens or narrows it; --join-types none joins nothing and shows the layers as the disconnected components they are.

2. Real dependency edges. LinkPackages resolves an import named com.depop.auth.client.AuthClient to the package com.depop.auth.client in another layer and adds a DEPENDS_ON edge. Longest prefix wins, three-segment floor, ambiguous targets skipped.

3. Derived edges are visibly derived. GraphEdge.Derived, dashed in mermaid and DOT, counted on its own header line. An inference and a recorded fact must not arrive looking alike.

What it does on the estate

$ kg graph --federated --root package:com.depop.auth.client --depth 1
%% kg graph: 12 node(s), 11 relation(s) of 717633 entities in the project
%% 11 of those are derived cross-layer links, drawn dashed
    subgraph layer0["libraries"]
        n0[["com.depop.auth.client"]]:::kgroot
    end
    subgraph layer1["ads"]
        n1[/"com.depop.auth.client.AccessToken"/]
    end
    ...
    n1 -.->|DEPENDS_ON| n0

Eleven layers depending on the shared auth library — the question --federated was supposed to answer and previously couldn't.

Full run: 2,525 derived links replacing the 67,263 manufactured ones #6 produced. 8.9s, 1.36 GB.

Verification

The spec predicted 845 edges; the implementation produces 2,525. I chased that gap rather than shipping it:

845 = distinct import names that resolve — what my gate script counted.
2,525 = import nodes — the same name is a separate node in each layer that imports it, since imports join only across ≤3 layers.

Confirmed by recomputing the rule independently against the shipped JSON: 2,525 = 2,525, and the name count reproduces 845 exactly. Both numbers were right; the spec quoted the wrong unit, now corrected.

Notes for review

Ambiguity discards the majority: 3,359 imports skipped against 2,525 kept, because the matched package is defined in more than one layer. Skipping is deliberate — guessing would reinvent the coincidence edges this PR removes — but it is the dominant path, so the report prints the count and examples.
Only dotted namespaces resolve. No package entity in the estate has a / in its name, so npm and Go layers get no derived links. That is an indexer gap, recorded in the spec's follow-up; TestLinkPackagesDoesNotResolveSlashNamespaces pins the current behaviour so it surfaces if that changes.
feat(kg): kg graph --federated — one graph across a scope and all its layers #6's tests changed because its fixtures joined type and topic entities, which the new default policy excludes. They now state their policy explicitly, and TestLoadFederatedGraphJoinPolicyByType covers the rule directly.
joinTypePolicy is split out of cobra state so "unset" vs "none" is testable without driving a command — a test that first tried to do it through the flag caught pflag's append-after-first-Set behaviour instead of the logic.

Testing

11 new tests for linking (longest-prefix, segment floor, both ambiguity routes, same-layer, multi-layer imports, determinism over 20 runs, dashed rendering) plus the join-policy and flag tests.
make test green across kglib, kg, markitdown.
Exercised end to end against the real 61-layer estate; every number above is measured.

Implements docs/kg-graph-linking-design.md. The join is now keyed on entity type rather than on a count alone. Names identify packages and imports across repositories because that is what those names are for; they do not identify functions, files or documentation headings. Measured before this change, every one of the 67,263 cross-layer relations in the estate came from the latter kind — Foundation, CodingKeys, print, map, forEach. DefaultJoinTypes is package and import; --join-types widens or narrows it, and an explicitly empty policy joins nothing. Joining says two rows are the same thing, which is still not a dependency. LinkPackages recovers those from what the indexers already record: an import named com.depop.auth.client.AuthClient resolves to the package com.depop.auth.client in another layer, and becomes a DEPENDS_ON edge. Longest prefix wins, with a three-segment floor so com.depop cannot claim every JVM import in the estate, and an ambiguous target is skipped rather than guessed at — on the estate that discards more matches (3,359) than it keeps (2,525). Derived edges carry a flag and render dashed, with their own line in the header. An inference and a recorded fact must not arrive looking alike. On the estate: 2,525 derived links replacing 67,263 manufactured ones, and `--root package:com.depop.auth.client` now draws the eleven layers that depend on the shared auth library.

--join-types none has to mean "join nothing" while an unset flag means "the default policy"; joinTypePolicy keeps those apart, and is separated from cobra state so that distinction is testable without driving a command. The federated report now says which types were eligible to join, how many links were derived, and how many imports were skipped as ambiguous with examples — a graph that quietly dropped the majority of its matches would read as complete.

Records the corrected yield figure too. The gate predicted 845 and the implementation produces 2,525; both are right and the units differ — 845 distinct import names, 2,525 import nodes, since a name appears separately in each layer that imports it. Verified by recomputing the rule independently against the shipped output.

…10) * feat(kg): per-type join policy and derived cross-layer package links Implements docs/kg-graph-linking-design.md. The join is now keyed on entity type rather than on a count alone. Names identify packages and imports across repositories because that is what those names are for; they do not identify functions, files or documentation headings. Measured before this change, every one of the 67,263 cross-layer relations in the estate came from the latter kind — Foundation, CodingKeys, print, map, forEach. DefaultJoinTypes is package and import; --join-types widens or narrows it, and an explicitly empty policy joins nothing. Joining says two rows are the same thing, which is still not a dependency. LinkPackages recovers those from what the indexers already record: an import named com.depop.auth.client.AuthClient resolves to the package com.depop.auth.client in another layer, and becomes a DEPENDS_ON edge. Longest prefix wins, with a three-segment floor so com.depop cannot claim every JVM import in the estate, and an ambiguous target is skipped rather than guessed at — on the estate that discards more matches (3,359) than it keeps (2,525). Derived edges carry a flag and render dashed, with their own line in the header. An inference and a recorded fact must not arrive looking alike. On the estate: 2,525 derived links replacing 67,263 manufactured ones, and `--root package:com.depop.auth.client` now draws the eleven layers that depend on the shared auth library. * feat(kg): --join-types and --no-derived --join-types none has to mean "join nothing" while an unset flag means "the default policy"; joinTypePolicy keeps those apart, and is separated from cobra state so that distinction is testable without driving a command. The federated report now says which types were eligible to join, how many links were derived, and how many imports were skipped as ambiguous with examples — a graph that quietly dropped the majority of its matches would read as complete. * docs(kg): document the join policy and derived links Records the corrected yield figure too. The gate predicted 845 and the implementation produces 2,525; both are right and the units differ — 845 distinct import names, 2,525 import nodes, since a name appears separately in each layer that imports it. Verified by recomputing the rule independently against the shipped output.

…oins Rebasing onto #5 brought this test alongside #10's per-type join policy, which restricts joining to package and import — "this type's names are local to a repository; two matches are a coincidence, not an identity". The fixture used `type`, which no longer joins, so it reported 3 config nodes instead of 2: nothing fused, but only because nothing joined at all. The test would have kept passing while exercising none of the path it was written for. Switch the fixture to two packages named "config" in one layer plus a third in another — internal/api/config alongside internal/worker/config is an ordinary Go layout, so this is a more representative case than the original, not just a working one. Add a premise guard: assert JoinTypes actually covers package before drawing any conclusion from the count. Without it, a future narrowing of the policy silently turns this back into a test that passes for the wrong reason, which is exactly what the rebase just did. Both directions mutation-checked: dropping the same-layer guard fails with "config nodes = 1" (the fusion this test exists to catch), and narrowing DefaultJoinTypes to import alone fails the premise guard rather than passing vacuously. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…10) * feat(kg): per-type join policy and derived cross-layer package links Implements docs/kg-graph-linking-design.md. The join is now keyed on entity type rather than on a count alone. Names identify packages and imports across repositories because that is what those names are for; they do not identify functions, files or documentation headings. Measured before this change, every one of the 67,263 cross-layer relations in the estate came from the latter kind — Foundation, CodingKeys, print, map, forEach. DefaultJoinTypes is package and import; --join-types widens or narrows it, and an explicitly empty policy joins nothing. Joining says two rows are the same thing, which is still not a dependency. LinkPackages recovers those from what the indexers already record: an import named com.depop.auth.client.AuthClient resolves to the package com.depop.auth.client in another layer, and becomes a DEPENDS_ON edge. Longest prefix wins, with a three-segment floor so com.depop cannot claim every JVM import in the estate, and an ambiguous target is skipped rather than guessed at — on the estate that discards more matches (3,359) than it keeps (2,525). Derived edges carry a flag and render dashed, with their own line in the header. An inference and a recorded fact must not arrive looking alike. On the estate: 2,525 derived links replacing 67,263 manufactured ones, and `--root package:com.depop.auth.client` now draws the eleven layers that depend on the shared auth library. * feat(kg): --join-types and --no-derived --join-types none has to mean "join nothing" while an unset flag means "the default policy"; joinTypePolicy keeps those apart, and is separated from cobra state so that distinction is testable without driving a command. The federated report now says which types were eligible to join, how many links were derived, and how many imports were skipped as ambiguous with examples — a graph that quietly dropped the majority of its matches would read as complete. * docs(kg): document the join policy and derived links Records the corrected yield figure too. The gate predicted 845 and the implementation produces 2,525; both are right and the units differ — 845 distinct import names, 2,525 import nodes, since a name appears separately in each layer that imports it. Verified by recomputing the rule independently against the shipped output.

…oins Rebasing onto #5 brought this test alongside #10's per-type join policy, which restricts joining to package and import — "this type's names are local to a repository; two matches are a coincidence, not an identity". The fixture used `type`, which no longer joins, so it reported 3 config nodes instead of 2: nothing fused, but only because nothing joined at all. The test would have kept passing while exercising none of the path it was written for. Switch the fixture to two packages named "config" in one layer plus a third in another — internal/api/config alongside internal/worker/config is an ordinary Go layout, so this is a more representative case than the original, not just a working one. Add a premise guard: assert JoinTypes actually covers package before drawing any conclusion from the count. Without it, a future narrowing of the policy silently turns this back into a test that passes for the wrong reason, which is exactly what the rebase just did. Both directions mutation-checked: dropping the same-layer guard fails with "config nodes = 1" (the fusion this test exists to catch), and narrowing DefaultJoinTypes to import alone fails the premise guard rather than passing vacuously. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* docs(kg): design proposal for cross-layer entity linking Measuring the federated graph PR #6 produces says the join is wrong. Of the 67,263 relations that cross a layer boundary in the real 61-layer estate, all 67,263 exist only because of the (name, type) join — no indexer writes a relation across databases, so the union is genuinely disconnected and every bridge is manufactured. The identifiers manufacturing them are Foundation, CodingKeys, print, map, forEach: Swift and JS boilerplate that evades the --join-max-layers guard by appearing in two or three layers rather than sixty. The proposal is a per-type join policy (names identify packages, not functions) plus real derived DEPENDS_ON edges from import→package resolution, with a measurement gate before implementation because it is not yet known whether that resolution finds anything. Rollup is deliberately deferred to a follow-up: aggregating over manufactured edges would only make them look authoritative. * docs(kg): measurement gate results — prefix matching, not exact The gate the proposal put before its own implementation, run against the 61-layer estate. It changed the design rather than confirming it. Exact package-name matching, which the spec proposed for v1, fails: 71 cross-layer hits, and they are api/auth/client/clients — the boilerplate problem one level up. Longest-dotted-prefix matching with a three-segment minimum finds 845 unambiguous cross-layer dependencies, and they are real: services importing com.depop.auth.client resolving to the package in libraries. Layer pairs come out as a plausible dependency map. Slash-separated ecosystems yield exactly zero, and the reason is upstream: not one of the 5,138 package entity names contains a "/", so npm and Go imports have nothing to resolve against. Recorded as an indexer follow-up rather than worked around here. Expected yield: ~845 derived edges replacing 67,263 manufactured ones. * feat(kg): per-type join policy and derived cross-layer package links (#10) * feat(kg): per-type join policy and derived cross-layer package links Implements docs/kg-graph-linking-design.md. The join is now keyed on entity type rather than on a count alone. Names identify packages and imports across repositories because that is what those names are for; they do not identify functions, files or documentation headings. Measured before this change, every one of the 67,263 cross-layer relations in the estate came from the latter kind — Foundation, CodingKeys, print, map, forEach. DefaultJoinTypes is package and import; --join-types widens or narrows it, and an explicitly empty policy joins nothing. Joining says two rows are the same thing, which is still not a dependency. LinkPackages recovers those from what the indexers already record: an import named com.depop.auth.client.AuthClient resolves to the package com.depop.auth.client in another layer, and becomes a DEPENDS_ON edge. Longest prefix wins, with a three-segment floor so com.depop cannot claim every JVM import in the estate, and an ambiguous target is skipped rather than guessed at — on the estate that discards more matches (3,359) than it keeps (2,525). Derived edges carry a flag and render dashed, with their own line in the header. An inference and a recorded fact must not arrive looking alike. On the estate: 2,525 derived links replacing 67,263 manufactured ones, and `--root package:com.depop.auth.client` now draws the eleven layers that depend on the shared auth library. * feat(kg): --join-types and --no-derived --join-types none has to mean "join nothing" while an unset flag means "the default policy"; joinTypePolicy keeps those apart, and is separated from cobra state so that distinction is testable without driving a command. The federated report now says which types were eligible to join, how many links were derived, and how many imports were skipped as ambiguous with examples — a graph that quietly dropped the majority of its matches would read as complete. * docs(kg): document the join policy and derived links Records the corrected yield figure too. The gate predicted 845 and the implementation produces 2,525; both are right and the units differ — 845 distinct import names, 2,525 import nodes, since a name appears separately in each layer that imports it. Verified by recomputing the rule independently against the shipped output. * test(kg): exercise the same-layer join guard with a type that still joins Rebasing onto #5 brought this test alongside #10's per-type join policy, which restricts joining to package and import — "this type's names are local to a repository; two matches are a coincidence, not an identity". The fixture used `type`, which no longer joins, so it reported 3 config nodes instead of 2: nothing fused, but only because nothing joined at all. The test would have kept passing while exercising none of the path it was written for. Switch the fixture to two packages named "config" in one layer plus a third in another — internal/api/config alongside internal/worker/config is an ordinary Go layout, so this is a more representative case than the original, not just a working one. Add a premise guard: assert JoinTypes actually covers package before drawing any conclusion from the count. Without it, a future narrowing of the policy silently turns this back into a test that passes for the wrong reason, which is exactly what the rebase just did. Both directions mutation-checked: dropping the same-layer guard fails with "config nodes = 1" (the fusion this test exists to catch), and narrowing DefaultJoinTypes to import alone fails the premise guard rather than passing vacuously. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(pr7): resolve an import that equals a package name; normalise JoinTypes Two of the three findings from #7's first review. The Critical is not addressed here — see below. resolvePackage tested only PROPER prefixes: the loop started at len(parts)-1, so it never compared the whole import name against a package name. The documented rule is "the longest package name that is a dotted prefix of it", and a string is a prefix of itself. That gap has a specific, ordinary victim. Java wildcard imports arrive as exactly the package name — extractImportPath keeps only the scoped_identifier and drops the trailing asterisk, so `import com.depop.auth.client.*;` becomes "com.depop.auth.client". Every one of those silently failed to resolve. Kotlin escaped it by accident: its extractor keeps the ".*" verbatim, leaving a longer string that the proper-prefix loop does reach. Reproduced before fixing (Derived = 0, want 1) and mutation-checked: restoring len(parts)-1 fails the new test with that same figure. Also normalised report.JoinTypes to lower case. Matching was already case-insensitive via lowerSet, but the raw input was stored, so `--join-types Package,IMPORT` printed back "Package, IMPORT". Not fixed: the Critical, that minPackageSegments = 3 filters out every package entity this repo's indexer can produce. Verified — EntityTypePackage is written in exactly one place, the Go package_clause handler, and a Go package name is a bare identifier with no dots. No Kotlin/Java/Scala package declaration is indexed at all. So LinkPackages returns Derived: 0 against anything this indexer produced, and the design doc's claim that "the indexers mint package entities only for dotted namespaces" is backwards for this codebase. That one needs a decision rather than a patch: either land JVM package indexing, or correct the doc and re-state where the estate's 2,525 edges actually came from. Both change what this PR claims to be, so they are the author's call. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(pr7): prove cross-layer linking on indexed source; correct the design doc The reviewer's Critical was that this feature could not derive anything from real data: minPackageSegments filtered out every package entity the indexer produced, because Go's bare identifier was the only source and never has a dot. That was accurate. JVM package indexing has since landed separately, so the dependency now exists and the claim can be demonstrated rather than argued. Add an end-to-end test that indexes actual Java source into two scope databases — a library layer declaring com.depop.auth.client, a consumer layer importing from it — federates them, and asserts a derived cross-layer edge. Nothing is hand-built: the graph is what `kg index` produces. Mutation-checked against the pre-indexing world: removing package_declaration from the Java config fails this test with "Derived = 0", which is exactly the symptom the review described. That makes the test a regression guard on the dependency, not just on this code. Correct the design doc's Follow-up, which stated the inverse of reality — that the indexers mint package entities "only for dotted namespaces", when Go's undotted name was the only kind minted. Replaced with a table of what each language actually yields and why Go cannot link, and the remaining npm/go.mod gap restated on that footing. Flag the acceptance figures rather than delete them. The 2,525 edges cannot have come from `kg index` as it shipped when they were recorded, for the reason above, so they are marked as predating JVM package indexing and needing a re-run against a freshly indexed estate. Re-deriving them needs that estate, so it is not something this change can settle. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

The estate was re-indexed with 3ac2f1a (971 s, 61 scopes, no failures, all 420 observations preserved) and every figure re-derived. Re-indexing lowers the derived-edge count rather than raising it: 2,532 -> 2,240, while same-layer resolutions more than double. Both movements are the rule behaving correctly on better data. com.depop.common was indexed only in libraries, so every clients import of it looked like a cross-repo dependency; now that clients declares it too, the name resolves to two layers and the one-layer rule abstains. Elsewhere longest-prefix now finds a specific local package where it previously settled for a shorter one defined elsewhere. Edges drawn because a repository's own copy of a package had not been indexed were never dependencies, so fewer of them is a more truthful graph — and it sharpens the open question about ambiguity, which grows as coverage improves. Also corrects a miscount of my own: "seventy-five .java files and no Kotlin" came from find -maxdepth 4, which misses trees nested deeper, the Android app among them. The estate holds 30,906 .scala, 7,591 .kt and 1,835 .java files, so 5211bc2 gave package declarations to about nine thousand files. CHANGELOG: --federated was #6 not #5, linking was #10 not #7, and its package-indexing entry repeated the same wrong Scala claim this branch corrects. The kg graph flag table was missing --scope and --personal, both of which its own examples use.

…ion (#16) * docs(kg): re-measure the estate figures, and withdraw a wrong correction The pending-re-measurement warning said the derived-edge figures could not have come from indexed data, because Go's package_clause was supposedly the only source of package entities before 5211bc2 and Go names carry no dots. Two facts say otherwise: package_clause is Scala's tree-sitter node as well as Go's, and indexer_treesitter.go matched it generically at 5211bc2^; and the estate's databases, indexed by v0.1.0-34, hold 4,385 package entities with three or more dotted segments, one of which resolves to .scala files. 5211bc2 added Java and Kotlin, not Scala. So the figures were sound. Re-derived at 3ac2f1a they are 2,532 derived edges (was 2,525), 3,355 ambiguous (was 3,359), 5,046 same-layer (was 5,031). The +7 is the wildcard-import fix: a Java `import com.x.y.*;` reaches the resolver as a string identical to the package name, and the original loop tested only proper prefixes, so those were dropped silently. Also records what the warning's remedy would actually buy. Package entities are minted at index time, so a newer binary over older databases changes nothing; re-indexing is what is required. By file count that is 75 .java files and no Kotlin against 12,537 already-indexed .scala — the unindexed mass is Python, TypeScript and Go, which 5211bc2 did not touch. * docs(kg): re-measure on a re-indexed estate; correct two miscounts The estate was re-indexed with 3ac2f1a (971 s, 61 scopes, no failures, all 420 observations preserved) and every figure re-derived. Re-indexing lowers the derived-edge count rather than raising it: 2,532 -> 2,240, while same-layer resolutions more than double. Both movements are the rule behaving correctly on better data. com.depop.common was indexed only in libraries, so every clients import of it looked like a cross-repo dependency; now that clients declares it too, the name resolves to two layers and the one-layer rule abstains. Elsewhere longest-prefix now finds a specific local package where it previously settled for a shorter one defined elsewhere. Edges drawn because a repository's own copy of a package had not been indexed were never dependencies, so fewer of them is a more truthful graph — and it sharpens the open question about ambiguity, which grows as coverage improves. Also corrects a miscount of my own: "seventy-five .java files and no Kotlin" came from find -maxdepth 4, which misses trees nested deeper, the Android app among them. The estate holds 30,906 .scala, 7,591 .kt and 1,835 .java files, so 5211bc2 gave package declarations to about nine thousand files. CHANGELOG: --federated was #6 not #5, linking was #10 not #7, and its package-indexing entry repeated the same wrong Scala claim this branch corrects. The kg graph flag table was missing --scope and --personal, both of which its own examples use. * docs(kg): bring the open questions in line with the re-measurement Question 3 still quoted the pre-re-index split (3,204 against 845) while the section above it reported 2,240 kept against 3,346 discarded, which read as the document contradicting itself. It also understated the problem. The discarded share grows as indexing coverage improves — a package indexed in more repositories resolves to more layers — so the linking weakens over time rather than strengthening. Records the option the re-measurement suggests: preferring the importing layer's own definition, which would have made com.depop.common same-layer and drawn nothing, instead of discarding it as ambiguous.

bawoodruff added 3 commits August 27, 2026 20:04

bawoodruff merged commit a63a985 into feat/kg-graph-linking-spec Aug 28, 2026

bawoodruff deleted the feat/kg-graph-linking branch August 28, 2026 03:11

bawoodruff mentioned this pull request Aug 30, 2026

docs(kg): re-measure the estate figures, and withdraw a wrong correction #16

Merged

Navigation Menu

feat(kg): per-type join policy and derived cross-layer package links - #10

feat(kg): per-type join policy and derived cross-layer package links#10
bawoodruff merged 3 commits into
feat/kg-graph-linking-specfrom
feat/kg-graph-linking

bawoodruff commented Aug 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

bawoodruff commented Aug 28, 2026

What changed

What it does on the estate

Verification

Notes for review

Testing

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant