Skip to content

fix: describe -> exec keeps what Studio Pro stored (#705) - #729

Merged
ako merged 33 commits into
mainfrom
fix/describe-roundtrip-losses
Sep 27, 2026
Merged

ako merged 33 commits into
mainfrom
fix/describe-roundtrip-losses

Conversation

@ako

@ako ako commented Sep 26, 2026 •

Copy link
Copy Markdown
Owner

Fixes #705. Adds a focused round-trip suite that is a step toward #703 (it does not close it: see "Scope").

What was wrong

Running a Studio Pro document's own describe output unchanged lost stored content. The audit's five items all reproduce on the documents mx create-project produces: the Blank template's FeedbackModule / Administration units are byte-identical to the audited PedApp ones, so the integration base project is the Studio Pro-authored fixture.

#705 item Cause Fix
5. create instead of create or modify (association, Java action, module role) describers printed plain create print create or modify (also describe module's role lines)
3. Java action ExportLevel → Public, return name lost executor hardcoded "Public" and "" carry both on modify. A new action gets Studio Pro's Hidden / ReturnValueName. "Public" is not a member of JavaActionsExportLevel or JavaScriptActionsExportLevel (API | Hidden), so every Java and JavaScript action mxcli created was invalid. The JavaScript path is fixed too.
4. Snippet Type Web → '' snippetToGen wrote header constants, and "" is not a PagesType carry the stored Type/ExportLevel/canvas, mirroring carryStoredPageHeader (#541). A new snippet gets Web.
2. Nanoflow annotation flows 3 → 0; ExportLevel, UseListParameterByReference, ReturnVariableName lost nanoflowToGen never wrote AnnotationFlows (ruleToGen neither) or the header keys write annotation flows (nanoflows and rules). Carry the header keys only when the stored unit has them. Read, print and write returns T as $Var. Carry MarkAsUsed, which was hardcoded to false.
1. Page texts lost four causes, found by measuring per owning widget rather than as a multiset see below

Page texts:

  • The language fallback became a write. When the default language was present but '', describe fell back to another language, and exec wrote that value into the default language (en_US '' + nl_NL 'Knop' → en_US 'Knop'). A present-but-empty value now wins.
  • Textarea placeholder. A textarea's placeholder was never read, printed or built. That accounts for most of the audit's 113 → 102 (nine translations).
  • TextTooLongMessage. It was never written, so the carry had nowhere to put the stored translations.
  • Translation pairing. canon.CarryTranslations now pairs by the named element that owns a text: its $Type, Name, and the path from that element. Two conditions keep this exact: the address must be unique in both documents, and no list index may sit between the element and the text. The old pairings failed here. Positional pairing breaks once a DataGrid2 is rebuilt, and source pairing cannot resolve (en_US, "") or a shared English source.

Found on the way (same path, now fixed)

Fixing the verb made Java action output re-execute, and that surfaced three losses in the .java file that no BSON comparison sees:

  • describe never read a body. Its markers had been lowercased ("// begin user CODE") by the lowercase-keywords sweep (00b80f3). It always printed the placeholder, so executing the output would have replaced the code.
  • Regeneration kept only the user code. Studio Pro's banner says the import list and EXTRA CODE are retained too. XSS_Sanitizer's user code calls sanitize() in its extra code, so a rewrite produced Java that doesn't compile. WriteJavaSourceFile now retains both.
  • The describe placeholder body is no longer written as source.

The test

mdl/executor/studiopro_roundtrip_test.go (-tags integration) runs describe → exec → canonical compare per unit. Each case has six checks: exec, document (GetPut + PutGet), header, texts, annotations, source. The ledger is keyed per check and fails in both directions. Every fix commit strikes its entries off, so each one shows the ledger entry failing before the fix and passing after.

Controls: TestStudioProRoundTripControl edits one caption and asserts that both the canonical comparison and the texts check see it. The source check was run with the Java fix reverted and fails on XSS_Sanitizer.

Final state: the Java action, snippet and module role round-trip exactly. Both nanoflows and both pages pass every #705 slice. Their whole-document verdicts stay ledgered, with reasons, for losses outside #705 that this suite found:

Scope

Validation

  • make build, go test ./..., make lint-go, make check-mdl: all pass
  • go test -tags integration ./mdl/executor/ (whole package, mxbuild 11.13.0): passes
  • Integration suite green on mxbuild 11.13.0
  • mdl-examples/bug-tests/describe-roundtrip-705-carried-properties.mdl on a fresh 11.13.0 blank app: the create half builds with mx check at 0 errors, and re-executing its describes reports "4 documents already in sync".
  • Not yet opened in Studio Pro. Please check that NF_Annotated's note is attached and that Note_Edit's textarea shows its placeholder.

Findings recorded (7), and the rewrite-drops-unauthored-state pattern page gains the three insights that recurred.

🤖 Generated with Claude Code

Also closes #704: the review added the association storage carry (commit 6688f78), with a test.

ako and others added 18 commits September 26, 2026 19:54
The visitor stores DescribeFragmentFromStmt.ContainerType as "PAGE"/"SNIPPET"
while describeFragmentFrom switched on "page"/"snippet" with no default, so
neither branch ran and every widget was reported missing ("not found in page
M.P" — without even naming the widget). Same casing split as ALTER PAGE (#402)
and ALTER STYLING (#631).

Normalise with strings.ToLower (the convention of the other consumers), make
an unrecognised container type an error instead of an empty widget list, and
name the widget in the not-found message. The new tests parse the statement
and dispatch it through the registry, so they pin the visitor/executor casing
contract that a hand-built lowercase AST could not see.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`search` needs only a full catalog, which indexes string literals but not
MDL source. On a project where `refresh catalog full source` had never run,
the source half of every search came back empty with no word — read by
agents as "no microflow/page mentions this".

search now checks the build mode the catalog records (not the row count, so
a built-but-empty index stays silent) and, below "source", prints a warning
naming `refresh catalog full source` on a new ExecContext.Diagnostics writer
(nil = stderr), keeping --format json stdout pure. The old unconditional
"Tip: refresh catalog source" on stdout is replaced by it. Table format now
delegates to execSearch up front instead of querying twice.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`show structure depth 1` (and its JSON form) filtered the catalog on
MicroflowType = 'microflow' / 'nanoflow', while the catalog builder
stores 'MICROFLOW' / 'NANOFLOW'. SQLite's `=` is case-sensitive, so both
counts were always empty, and the summary omits zero counts, so every
module appeared to have no flows at all.

The builder's values are now exported constants
(catalog.MicroflowTypeMicroflow/Nanoflow/Rule), used by the writer, by
the structure query and by the linter's DocumentNoun switch, so reader
and writer share one spelling.

The test runs the real catalog builder over a MockBackend and reads the
counts back through structureDepth1 / structureDepth1JSON, so it detects
a casing drift between writer and reader; reverting only the reader to
lower case makes it fail with the reported symptom.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…and mapped entities

The refs graph stopped at documents. `impact Module.Entity.Attr` answered
"not referenced" for an attribute a microflow writes and a page displays
(Evora: DigitalTwin.Machine.NumberOfIncidents), an enumeration had no inbound
edge at all, a workflow started only by a microflow had no caller, a page
navigating an association and a mapping mapping an entity were invisible.

- A raw-document walk over microflows, nanoflows, rules, pages, snippets,
  workflows and import/export mappings matches every string value against the
  names the model declares: whole-string matches are structured references
  (MemberChange.Attribute, AttributeRef.Attribute, EntityRefStep.Association,
  EnumerationType.Enumeration, ObjectMappingElement.Entity); inside
  expressions, association paths and qualified enumeration values.
  New kinds: member, type, value, mapping.
- XPath constraints resolve bare attribute names against their target entity
  (and its generalizations), association paths, and enum attributes compared
  to a literal (kind xpath). Page/snippet constraints had no target entity
  because resolveEntityRefFromBSON read a key no stored EntityRef carries.
- Entities -> enumerations from attribute types (kind type).
- WorkflowCallAction -> WORKFLOW (kind call).
- New types ATTRIBUTE, ENUMERATION, ENUMERATION_VALUE, IMPORT_MAPPING,
  EXPORT_MAPPING published in the lint-rule vocabulary; members kept off the
  graph_god_nodes asset side; CatalogSchemaVersion 15.

Not covered: a bare member named through a variable in a free-text
expression ($Order/Total), whose type is not known to the catalog.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…y answer says what was checked

`refs` and `impact` printed a row per edge, so a microflow with two retrieve
activities over an entity appeared twice (Evora: ProductionLine_Reset on
DigitalTwin.Machine); the impact summary counted those rows (MICROFLOW: 9 over
six microflows) and printed the types in map order, different between runs.

- select distinct, with a total order; the summary counts distinct elements
  per type, in type order, and the footer gives both numbers.
- impact/refs on an enumeration include the edges to its values, with a
  Target column naming which value.
- "(no impact - element is not referenced)" is gone. For an attribute or an
  enumeration value the message lists the sites that were checked and the
  ones that are not resolved (a member named through a variable in an
  expression; a decision branch on an enum), and says to run search first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e listed once

`context DigitalTwin.Machine` said "Related Entities: (none found)" for an
entity with five associations. The section read refs rows whose SOURCE is an
entity, but an association edge's source is the ASSOCIATION, so only
generalizations could ever match. It now reads both ends from the
associations table (present in a fast catalog too), plus the generalization
and the specializations.

Also: Direct Callers / Shown By / workflow callers list each source once, and
the enumeration context reads the same edge set as impact (type and value
uses), grouped into entities, flows and pages.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ring it

Studio Pro stores expressions exactly as typed, and a trailing newline left
in the expression editor is common. describe interpolated the stored text
verbatim, so `change $X (Status = Mod.Enum.Val` / `);` put the closing paren
or semicolon on a line of its own (297 such lines in Evora Factory
Management's microflows alone).

One helper, describeExpr (TrimSpace; interior newlines kept), now renders
every stored expression the describer emits: change/create members, set,
change-list values, aggregate/reduce, list operations, call microflow /
nanoflow / java / javascript / external action arguments, show page args,
log node + template params, show message params, REST/web service/DB query
params, while, decision and rule arguments, and page widget Visible/Editable
conditions, action arguments, datasource arguments and client template
parameters. It replaces the five ad-hoc TrimSuffix/TrimRight calls that each
covered one slot. The stored model is untouched.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every existing round-trip test starts from MDL, so the document it compares
holds only what MDL can say. The losses live in what Studio Pro stores and MDL
does not mention. This suite describes documents that ship in Mendix's Blank
template (what `mx create-project` produces; byte-identical to the PedApp
units audited for #705), re-executes that output unchanged, and
compares the stored unit canonically.

A whole-document verdict is too coarse to track: a page loses a dozen
unrelated things at once. So each case is held to five checks -- exec,
document, header, texts, annotations -- and the ledger is keyed per check,
failing in both directions. It starts as exactly the #705 losses, plus the
whole-document losses this suite found beyond them (input-widget writer
constants and a DataGrid2 rebuilt from its template on pages; activity sizes
on a nanoflow).

A control edits one caption through ALTER PAGE and asserts both the canonical
comparison and the texts check see it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`--json` output did not parse. refs, callers, callees, impact, structure,
show, search and `-c "select …"` printed "Connected to:", catalog
load/build progress, a header and a "Found N" count ahead of the payload;
an empty answer was a sentence ("(no references found)") instead of [];
`context` ignored --json; `check --format json` wrote its document to
stderr, one per phase; `diff --json` printed a text diff.

The executor writes the payload and its commentary through the same
ctx.Output, so the mendixlabs#904 mechanism (pick the executor's writer at the cmd
layer) cannot separate them for commands whose payload the executor
itself prints. Add ExecContext.progress(): Output in text mode (so
interactive output is unchanged), the diagnostics stream (stderr) when
Output carries JSON. The Diagnostics field is the one #716
introduces, with the same semantics.

- Status/progress in connect, catalog load/build, catalog queries and the
  refs/callers/callees/impact handlers goes through progress().
- Empty results go through writeEmptyResult: [] in JSON mode.
- context --json wraps the markdown in {name, type, depth, context}.
- show catalog status / show widgets gain a JSON path; empty listings of
  data transformers, import/export mappings and navigation follow the
  existing `&& ctx.Format != FormatJSON` idiom.
- search: --json is the canonical spelling; --format json is kept as a
  deprecated alias and now sets the executor format, so it is as clean.
- A cold-cache catalog build under -q moves its per-table lines to stderr
  instead of stdout (search -q --format names).
- check: structured formats emit one document on stdout covering every
  phase; the exit code is unchanged.
- diff / diff-local refuse --json (no JSON output) instead of ignoring it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…r modify

DESCRIBE describes a document that exists, so its output has to re-execute
against the project it came from. These three emitted a plain `create`, which
fails there with "already exists": the round trip died at the first statement
instead of reporting what changed (#705 item 5). Every other
describer already emits `create or modify`; `describe module` prints its roles
in the same shape as `describe module role`, so it changes with it.

With the verb fixed the round-trip suite sees past the refusal: a module role
now round-trips exactly; a Java action shows the ExportLevel and
ActionDefaultReturnName losses (#705 item 3, next); an association shows
#704's Table -> Column flip, and that the domain-model rewrite also
drops empty MemberAccess refs and NoGeneralization flags. Both are ledgered.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…p writing "Public"

Neither property has an MDL spelling -- DESCRIBE prints them as comments at
most -- so CREATE OR MODIFY must carry them. The Java action path hardcoded
ExportLevel "Public" and ActionDefaultReturnName "", so describe -> exec of
FeedbackModule.XSS_Sanitizer turned Hidden into Public and dropped
"ReturnValueName" (#705 item 3).

"Public" is not a member of JavaActionsExportLevel or
JavaScriptActionsExportLevel (both API | Hidden), so every Java and
JavaScript action mxcli CREATED carried an enum value the metamodel does not
declare. The JavaScript path had the same hardcode and is fixed with it. A new
action now gets what Studio Pro writes on all 69 actions in the Blank
template: Hidden and ReturnValueName.

The round-trip suite's Java action case now passes every check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ype "Web" on create

snippetToGen wrote the header as constants, and the Type constant was "",
which is not a member of PagesType (Native | Web). describe -> exec of the
Blank template's FeedbackModule._ReadMe turned Type "Web" into ""
(#705 item 4), and every snippet mxcli created carried the invalid
value.

None of the four header properties has an MDL spelling, so UpdateSnippet now
carries them off the stored unit, mirroring carryStoredPageHeader (#541) --
including its width-agnostic canvas read. Type matters most: it says whether
the snippet is for web or native pages, which a rewrite must not decide. A
new snippet gets what Studio Pro writes on every snippet of that template:
Web, Hidden, 800 x 600.

The round-trip suite's snippet case now passes every check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eference, return variable and mark-as-used

describe -> exec of the Blank template's ACT_Feedback_UploadImage lost its
three annotation connectors and three header properties (#705
item 2):

- nanoflowToGen never wrote ObjectCollection.AnnotationFlows, though the
  reader fills it and the shared flow builder produces it; microflowToGen
  always did. With the connectors gone the notes float free and the next
  DESCRIBE attaches them to nothing. ruleToGen had the same omission and is
  fixed with it.
- ExportLevel and UseListParameterByReference have no MDL spelling and were
  never written. UpdateNanoflow now carries them off the stored unit -- and
  only when the stored unit has them, since the writer has never emitted them
  on a new nanoflow and a key the project's metamodel does not declare makes
  the document unopenable.
- ReturnVariableName is authorable (`returns T as $Var`) but the nanoflow path
  neither read, printed nor wrote it. It now does all three, the way the
  microflow path does, and a statement without `as` carries the stored one.
- MarkAsUsed was hardcoded false on the rebuild, clearing Studio Pro's "Mark
  as used"; it is now read and carried like the microflow's.

The round-trip suite gains SUB_Feedback_GetOrCreate, which stores a non-empty
return variable. Both nanoflows now pass exec, header, texts and annotations;
their whole-document verdicts stay ledgered for losses outside #705 (activity
sizes, auto-captions, regenerated merges).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
describe -> exec of the Blank template's ShareFeedback and Account_Overview
lost translations (#705 item 1). Measuring per owning widget rather
than as a multiset showed four distinct causes:

- DESCRIBE fell back to another language when the default one was present but
  EMPTY, and exec writes what describe prints into the default language: a
  caption stored as en_US "" + nl_NL "Knop" came back as en_US "Knop". A
  present-but-empty value now wins; the fallbacks remain for a text with no
  entry in that language (#702's case).
- A textarea's placeholder was never read, printed or built -- only a
  textbox's was -- so it vanished with its nine translations. That was most
  of the audit's 113 -> 102.
- The TextArea writer never emitted TextTooLongMessage, which Studio Pro
  stores on every textarea; with no text in the rebuild there was nothing for
  the translation carry to fill.
- canon.CarryTranslations paired by whole-document path (lost as soon as a
  DataGrid2 is rebuilt from its template) or by source string, which cannot
  tell apart the many (en_US, "") texts or two texts sharing an English
  source, and cannot look up a rebuilt empty text at all. It now pairs by the
  named element that owns the text -- ($Type, Name, path from it) -- where
  that is exact: unique in both documents, and no list index between element
  and text, since a rebuilt pluggable widget reorders its Properties.

Both pages now keep every translation; their whole-document verdicts stay
ledgered for losses outside #705.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s and extra code a rewrite cannot express

With describe now emitting `create or modify java action`, its output
re-executes -- and that exposed three ways it destroyed the action's code,
none of which any BSON comparison sees:

- DESCRIBE never read a Java body. Its markers had been lowercased
  ("// begin user CODE") by the sweep that made MDL keywords lowercase, so no
  Studio Pro file matched and the placeholder was always printed; executing it
  replaced the code. Markers now match case-insensitively, like the
  JavaScript describer's.
- The regenerated file kept only the user code. Studio Pro's banner states
  the contract -- the import list, user code and extra code are retained --
  and FeedbackModule.XSS_Sanitizer's user code calls sanitize() in its EXTRA
  CODE section, so a rewrite produced Java that does not compile. MDL cannot
  spell extra code, so WriteJavaSourceFile now retains the existing file's
  import list and extra code (javaactions.RetainSections), adding the
  statement's imports and taking its extra code when it has one.
- Where DESCRIBE still cannot read a source (add-on modules ship none), the
  placeholder body it prints is no longer written as the action's source.

The user code is normalised on read (CRLF, surrounding blank lines, common
indentation) so a second DESCRIBE prints the same as the first.

The round-trip suite gains a `source` check comparing the file's retained
sections; with this change reverted it fails on XSS_Sanitizer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Six findings (the describe verb, the Java/JavaScript ExportLevel enum,
snippet header constants, nanoflow and rule annotation flows, element-anchored
translation pairing, the Java source file) plus one on the round-trip suite
itself. The rewrite-drops-unauthored-state pattern page gains the three
insights that recurred: a hardcoded constant is usually wrong on create as
well, a fallback in DESCRIBE is a write, and a describe that cannot
re-execute hides every loss behind it.

The bug-test script covers the create half and ends in the describes whose
output must re-execute as unchanged (4 of 4 on 11.13.0, mx check 0 errors).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ako and others added 11 commits September 27, 2026 06:51
… statement omits it

describe omits the storage clause for table storage, and since this branch
prints create or modify, re-executing its unchanged output flipped a Studio
Pro table association to column storage (#704) - a schema change the plain
create used to refuse. The OR MODIFY arms now keep the stored StorageFormat
unless the statement states one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… category

describe prints a parameter description as a comment the parser drops, so
re-executing create or modify output deleted it (PedApp
FeedbackModule.ValidateEmail). Carried from the stored parameter of the same
name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sliceBetweenFold sliced the source at indices found in strings.ToLower(s),
which is not length-preserving (U+0130, the Kelvin sign), so such a character
above the markers cut the user or extra code a byte off.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… trimming expression whitespace

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	mdl/executor/cmd_search.go
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ako and others added 4 commits September 27, 2026 07:00
…ions, module roles, snippet Type, nanoflow export level); association now getput only (#721 B)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…efix

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ako
ako merged commit 1765eec into main Sep 27, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant