Skip to content

fix(refs): attribute/enum/workflow/mapping edges, deduped refs & impact, context related entities - #720

Merged
ako merged 14 commits into
mainfrom
fix/refs-graph-gaps
Sep 27, 2026
Merged

ako merged 14 commits into
mainfrom
fix/refs-graph-gaps

Conversation

@ako

@ako ako commented Sep 26, 2026

Copy link
Copy Markdown
Owner

Symptoms

Found by an agent-orientation audit on Evora Factory Management (MPR v1, Mendix 10.24):

  1. Missing targets/edges. impact DigitalTwin.Machine.NumberOfIncidents said (no impact - element is not referenced) although a change activity sets it and DigitalTwin.Machine_Details displays it. impact DigitalTwin.ENUM_MachineStatus said the same. callers AltairIntegration.WF_ScheduleTechnicianAppointment said (no callers found) although ACT_TechnicianTicket_SumbitTriggerWF calls it. An agent that trusts these answers deletes live elements.
  2. Repeated rows. refs/impact DigitalTwin.Machine listed FactoryManagement.ProductionLine_Reset | retrieve twice.
  3. Impact counted rows. The summary said MICROFLOW: 9 over six microflows, and printed the types in map order, which changed between runs.
  4. context DigitalTwin.Machine showed Related Entities: (none found) for an entity with five associations.

Causes

  1. The refs graph stopped at documents. It had no ATTRIBUTE, ENUMERATION or ENUMERATION_VALUE targets. microflowActionRef had no case for WorkflowCallAction. Nothing emitted page→association or mapping→entity edges. Page and snippet XPath constraints were also stored with an empty TargetEntity (52 of 52 on Evora), because resolveEntityRefFromBSON read EntityRef.QualifiedName, a key that no stored EntityRef has.
  2. / 3. The queries had no DISTINCT. The summary counted rows by ranging over a map.
  3. The related-entities query read refs rows whose source is an entity. An association edge's source is the ASSOCIATION, so the query could only ever find generalizations.

Fix (one commit per concern)

  • fix(catalog)
    • Walks the raw documents of microflows, nanoflows, rules, pages, snippets, workflows and import/export mappings, and matches every string value against the names the model declares.
      • A whole-string match is a structured reference: MemberChange.Attribute, AttributeRef.Attribute, EntityRefStep.Association, EnumerationType.Enumeration, ObjectMappingElement.Entity.
      • Inside expressions it matches association paths and qualified enumeration values.
      • Nothing here is per-type code: the walk reaches text-template parameters, conditional visibility, pluggable-widget attribute properties and mapping elements without a case for each.
    • XPath constraints are resolved against their target entity and its generalizations. This covers bare attribute names, association paths, a predicate after a path, and an enum attribute compared with a literal. resolveEntityRefFromBSON now reads the stored shapes.
    • New edges:
      • entity → enumeration from attribute types (type)
      • WorkflowCallAction → WORKFLOW (call, so callers sees it)
    • New kinds: member, xpath, type, value, mapping.
    • New types: ATTRIBUTE, ENUMERATION, ENUMERATION_VALUE, IMPORT_MAPPING, EXPORT_MAPPING. They are published in the lint-rule vocabulary and the write-lint-rules skill.
    • The member kinds stay out of graphRefKinds, and attribute/value targets are kept off graph_god_nodes' asset side. Attributes are members, not assets, so community and centrality results do not change.
    • CatalogSchemaVersion 15. CatalogTx gains Query, which *sql.Tx already has.
  • fix(refs)
    • select distinct with a total order. The summary counts distinct elements per type, in a fixed order, and the footer gives both numbers.
    • refs/impact on an enumeration include the uses of its values, with a Target column. The LIKE is escaped, because ENUM_ names contain _.
    • The empty answer no longer says "not referenced". For an attribute or an enumeration value it lists what was checked and what is not resolved, and says to run search.
  • fix(context)
    • Related entities come from the associations table (both ends, with direction), plus the generalization and the specializations.
    • The callers lists are distinct.
    • The enumeration context uses the same edge set as impact.
  • test(refs): an MDL repro and the findings.

Not covered (said so in the empty-answer message)

  • An attribute named only through a variable in a free-text expression ($Order/Total, $dataView1/MachineStatus). Resolving it needs the variable's type.
  • An enumeration value used only as a decision branch. EnumerationCase stores the bare value name.
  • XPath inside a page whose data source is not a direct/indirect entity ref. It is still resolved where the path is qualified.

Tests, and proof they detect the bugs

Test Layer Fails before the fix with
TestReferencesReachMembersEnumerationsWorkflowsAndMappings (mdl/catalog) builder: buildXPathExpressions + buildReferences over raw BSON fixtures shaped like Evora's units all 16 expected edges missing
TestScanPaths XPath/expression resolver new function; covers keyword, function-name, %token%, predicate context, literal-in-string controls
TestMicroflowActionRef/WorkflowCallAction unit ok = false
TestResolveEntityRefFromBSON (stored shapes) unit "" for DirectEntityRef / IndirectEntityRef (re-checked by restoring the old function)
TestShowReferencesListsEachEdgeOnce, TestShowImpactCountsElementsNotRows, TestShowImpactIsDeterministic, TestShowImpactOfEnumerationIncludesItsValues, TestShowImpactOfUnreferencedAttributeSaysWhatWasChecked (mdl/executor) executor over a seeded catalog 3 rows / missing summary / run 3 differs / missing value uses / "is not referenced"
TestAssembleEntityContextListsAssociatedEntities, TestAssembleMicroflowContextListsEachCallerOnce, TestAssembleEnumerationContextIncludesValueUses executor (none found) / caller listed 3× / value uses missing (re-checked by restoring the old cmd_context.go)

Controls:

  • An unused attribute in the catalog fixture must stay unreferenced, and a name in Documentation must not produce an edge.
  • mxcli lint on Evora gives identical per-rule counts before and after: 231 issues each.

Evora, before → after

impact DigitalTwin.Machine.NumberOfIncidents

before: (no impact - element is not referenced)
after:  Found 4 affected element(s) (4 reference(s))
        EXPORT_MAPPING DigitalTwin.EMM_DigitalTwinData          member
        IMPORT_MAPPING DigitalTwin.IMM_ProductionLine           member
        MICROFLOW      DigitalTwin.Machine_UpdateIncidentNumber member
        PAGE           DigitalTwin.Machine_Details              member

impact DigitalTwin.ENUM_MachineStatus

before: (no impact - element is not referenced)
after:  ENTITY DigitalTwin.Machine                        type  DigitalTwin.ENUM_MachineStatus
        PAGE   DigitalTwin.Machine_Details                value DigitalTwin.ENUM_MachineStatus.Critical
        PAGE   DigitalTwin.ProductionLine_MachineOverview value DigitalTwin.ENUM_MachineStatus.Critical

callers AltairIntegration.WF_ScheduleTechnicianAppointment

before: (no callers found)
after:  AltairIntegration.ACT_TechnicianTicket_SumbitTriggerWF | 1

impact DigitalTwin.Machine

before: Summary: PAGE: 5 / ASSOCIATION: 5 / MICROFLOW: 9 (map order)   Found 19 affected element(s)
        ProductionLine_Reset | retrieve  listed twice
after:  Summary: ASSOCIATION: 5 / EXPORT_MAPPING: 3 / IMPORT_MAPPING: 3 / MICROFLOW: 6 / PAGE: 4
        Found 21 affected element(s) (24 reference(s))   -- each (source, kind) once; mappings now included

context DigitalTwin.Machine → Related Entities

before: (none found)
after:  - AltairIntegration.MachineProperties (association AltairIntegration.MachineProperties_Machine: AltairIntegration.MachineProperties -> DigitalTwin.Machine)
        - DigitalTwin.Incident (association DigitalTwin.Incident_Machine: DigitalTwin.Incident -> DigitalTwin.Machine)
        - DigitalTwin.MachineImage (association DigitalTwin.Machine_MachineImage: DigitalTwin.Machine -> DigitalTwin.MachineImage)
        - DigitalTwin.ProductionLine (association DigitalTwin.Machine_ProductionLine: DigitalTwin.Machine -> DigitalTwin.ProductionLine)
        - DigitalTwin.TechnicianTicket (association DigitalTwin.TechnicianTicket_Machine: DigitalTwin.TechnicianTicket -> DigitalTwin.Machine)

impact on an attribute with no edge (FactoryManagement.AgentExtension.DisplayName)

before: (no impact - element is not referenced)
after:  (no references found to attribute FactoryManagement.AgentExtension.DisplayName)
        Checked: attribute bindings and member changes in microflows, nanoflows, rules, pages,
        snippets, workflows and import/export mappings, and XPath constraints.
        Not checked: the attribute named through a variable in an expression ($Object/DisplayName).
        Run `search 'DisplayName'` before treating it as unused.

Refs on Evora: 11,024 → 18,263. New: 4,043 member→ATTRIBUTE, 1,480 member→ASSOCIATION, 558 value, 356/306/67 xpath (attribute/association/value), 284 type, 144 mapping, 1 call→WORKFLOW. refresh catalog full went from 23s to 28s.

The repro mdl-examples/bug-tests/refs-graph-members-enums-workflows.mdl was exec'd into a copy of Evora. Its expected outputs are in its header, and all of them reproduce with the old binary and are fixed with the new one.

Validation

  • make build: passes.
  • make lint: passes (Go and TypeScript).
  • make test: 22 tests fail, all in widget/.mpk code this PR does not touch (TestParseMPK*, TestFindMPK*, TestDescribeWidget_*, TestValidateWidgetKind_*, …). They come from the sparse checkout omitting *.mpk blobs (open …/*.mpk: no such file, "0 enumeration properties learned from testdata .mpk files"), in cmd/mxcli, mdl/executor, modelsdk/widgets/mpk and sdk/widgets/mpk. They are unrelated to this change.
  • This is a read-side-only change, so no Studio Pro or mx check validation is needed: the catalog is never written back to the model.

🤖 Generated with Claude Code

ako and others added 8 commits September 26, 2026 19:54
The visitor stores DescribeFragmentFromStmt.ContainerType as "PAGE"/"SNIPPET"
while describeFragmentFrom switched on "page"/"snippet" with no default, so
neither branch ran and every widget was reported missing ("not found in page
M.P" — without even naming the widget). Same casing split as ALTER PAGE (#402)
and ALTER STYLING (#631).

Normalise with strings.ToLower (the convention of the other consumers), make
an unrecognised container type an error instead of an empty widget list, and
name the widget in the not-found message. The new tests parse the statement
and dispatch it through the registry, so they pin the visitor/executor casing
contract that a hand-built lowercase AST could not see.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`search` needs only a full catalog, which indexes string literals but not
MDL source. On a project where `refresh catalog full source` had never run,
the source half of every search came back empty with no word — read by
agents as "no microflow/page mentions this".

search now checks the build mode the catalog records (not the row count, so
a built-but-empty index stays silent) and, below "source", prints a warning
naming `refresh catalog full source` on a new ExecContext.Diagnostics writer
(nil = stderr), keeping --format json stdout pure. The old unconditional
"Tip: refresh catalog source" on stdout is replaced by it. Table format now
delegates to execSearch up front instead of querying twice.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`show structure depth 1` (and its JSON form) filtered the catalog on
MicroflowType = 'microflow' / 'nanoflow', while the catalog builder
stores 'MICROFLOW' / 'NANOFLOW'. SQLite's `=` is case-sensitive, so both
counts were always empty, and the summary omits zero counts, so every
module appeared to have no flows at all.

The builder's values are now exported constants
(catalog.MicroflowTypeMicroflow/Nanoflow/Rule), used by the writer, by
the structure query and by the linter's DocumentNoun switch, so reader
and writer share one spelling.

The test runs the real catalog builder over a MockBackend and reads the
counts back through structureDepth1 / structureDepth1JSON, so it detects
a casing drift between writer and reader; reverting only the reader to
lower case makes it fail with the reported symptom.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…and mapped entities

The refs graph stopped at documents. `impact Module.Entity.Attr` answered
"not referenced" for an attribute a microflow writes and a page displays
(Evora: DigitalTwin.Machine.NumberOfIncidents), an enumeration had no inbound
edge at all, a workflow started only by a microflow had no caller, a page
navigating an association and a mapping mapping an entity were invisible.

- A raw-document walk over microflows, nanoflows, rules, pages, snippets,
  workflows and import/export mappings matches every string value against the
  names the model declares: whole-string matches are structured references
  (MemberChange.Attribute, AttributeRef.Attribute, EntityRefStep.Association,
  EnumerationType.Enumeration, ObjectMappingElement.Entity); inside
  expressions, association paths and qualified enumeration values.
  New kinds: member, type, value, mapping.
- XPath constraints resolve bare attribute names against their target entity
  (and its generalizations), association paths, and enum attributes compared
  to a literal (kind xpath). Page/snippet constraints had no target entity
  because resolveEntityRefFromBSON read a key no stored EntityRef carries.
- Entities -> enumerations from attribute types (kind type).
- WorkflowCallAction -> WORKFLOW (kind call).
- New types ATTRIBUTE, ENUMERATION, ENUMERATION_VALUE, IMPORT_MAPPING,
  EXPORT_MAPPING published in the lint-rule vocabulary; members kept off the
  graph_god_nodes asset side; CatalogSchemaVersion 15.

Not covered: a bare member named through a variable in a free-text
expression ($Order/Total), whose type is not known to the catalog.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…y answer says what was checked

`refs` and `impact` printed a row per edge, so a microflow with two retrieve
activities over an entity appeared twice (Evora: ProductionLine_Reset on
DigitalTwin.Machine); the impact summary counted those rows (MICROFLOW: 9 over
six microflows) and printed the types in map order, different between runs.

- select distinct, with a total order; the summary counts distinct elements
  per type, in type order, and the footer gives both numbers.
- impact/refs on an enumeration include the edges to its values, with a
  Target column naming which value.
- "(no impact - element is not referenced)" is gone. For an attribute or an
  enumeration value the message lists the sites that were checked and the
  ones that are not resolved (a member named through a variable in an
  expression; a decision branch on an enum), and says to run search first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e listed once

`context DigitalTwin.Machine` said "Related Entities: (none found)" for an
entity with five associations. The section read refs rows whose SOURCE is an
entity, but an association edge's source is the ASSOCIATION, so only
generalizations could ever match. It now reads both ends from the
associations table (present in a fast catalog too), plus the generalization
and the specializations.

Also: Direct Callers / Shown By / workflow callers list each source once, and
the enumeration context reads the same edge set as impact (type and value
uses), grouped into entities, flows and pages.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ring it

Studio Pro stores expressions exactly as typed, and a trailing newline left
in the expression editor is common. describe interpolated the stored text
verbatim, so `change $X (Status = Mod.Enum.Val` / `);` put the closing paren
or semicolon on a line of its own (297 such lines in Evora Factory
Management's microflows alone).

One helper, describeExpr (TrimSpace; interior newlines kept), now renders
every stored expression the describer emits: change/create members, set,
change-list values, aggregate/reduce, list operations, call microflow /
nanoflow / java / javascript / external action arguments, show page args,
log node + template params, show message params, REST/web service/DB query
params, while, decision and rule arguments, and page widget Visible/Editable
conditions, action arguments, datasource arguments and client template
parameters. It replaces the five ad-hoc TrimSuffix/TrimRight calls that each
covered one slot. The stored model is untouched.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ako added a commit that referenced this pull request Sep 27, 2026
…efix

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ako
ako merged commit 10ea207 into main Sep 27, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant