fix: describe -> exec keeps what Studio Pro stored (#705) - #729
Merged
Merged
Conversation
The visitor stores DescribeFragmentFromStmt.ContainerType as "PAGE"/"SNIPPET"
while describeFragmentFrom switched on "page"/"snippet" with no default, so
neither branch ran and every widget was reported missing ("not found in page
M.P" — without even naming the widget). Same casing split as ALTER PAGE (#402)
and ALTER STYLING (#631).
Normalise with strings.ToLower (the convention of the other consumers), make
an unrecognised container type an error instead of an empty widget list, and
name the widget in the not-found message. The new tests parse the statement
and dispatch it through the registry, so they pin the visitor/executor casing
contract that a hand-built lowercase AST could not see.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`search` needs only a full catalog, which indexes string literals but not MDL source. On a project where `refresh catalog full source` had never run, the source half of every search came back empty with no word — read by agents as "no microflow/page mentions this". search now checks the build mode the catalog records (not the row count, so a built-but-empty index stays silent) and, below "source", prints a warning naming `refresh catalog full source` on a new ExecContext.Diagnostics writer (nil = stderr), keeping --format json stdout pure. The old unconditional "Tip: refresh catalog source" on stdout is replaced by it. Table format now delegates to execSearch up front instead of querying twice. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`show structure depth 1` (and its JSON form) filtered the catalog on MicroflowType = 'microflow' / 'nanoflow', while the catalog builder stores 'MICROFLOW' / 'NANOFLOW'. SQLite's `=` is case-sensitive, so both counts were always empty, and the summary omits zero counts, so every module appeared to have no flows at all. The builder's values are now exported constants (catalog.MicroflowTypeMicroflow/Nanoflow/Rule), used by the writer, by the structure query and by the linter's DocumentNoun switch, so reader and writer share one spelling. The test runs the real catalog builder over a MockBackend and reads the counts back through structureDepth1 / structureDepth1JSON, so it detects a casing drift between writer and reader; reverting only the reader to lower case makes it fail with the reported symptom. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…and mapped entities The refs graph stopped at documents. `impact Module.Entity.Attr` answered "not referenced" for an attribute a microflow writes and a page displays (Evora: DigitalTwin.Machine.NumberOfIncidents), an enumeration had no inbound edge at all, a workflow started only by a microflow had no caller, a page navigating an association and a mapping mapping an entity were invisible. - A raw-document walk over microflows, nanoflows, rules, pages, snippets, workflows and import/export mappings matches every string value against the names the model declares: whole-string matches are structured references (MemberChange.Attribute, AttributeRef.Attribute, EntityRefStep.Association, EnumerationType.Enumeration, ObjectMappingElement.Entity); inside expressions, association paths and qualified enumeration values. New kinds: member, type, value, mapping. - XPath constraints resolve bare attribute names against their target entity (and its generalizations), association paths, and enum attributes compared to a literal (kind xpath). Page/snippet constraints had no target entity because resolveEntityRefFromBSON read a key no stored EntityRef carries. - Entities -> enumerations from attribute types (kind type). - WorkflowCallAction -> WORKFLOW (kind call). - New types ATTRIBUTE, ENUMERATION, ENUMERATION_VALUE, IMPORT_MAPPING, EXPORT_MAPPING published in the lint-rule vocabulary; members kept off the graph_god_nodes asset side; CatalogSchemaVersion 15. Not covered: a bare member named through a variable in a free-text expression ($Order/Total), whose type is not known to the catalog. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…y answer says what was checked `refs` and `impact` printed a row per edge, so a microflow with two retrieve activities over an entity appeared twice (Evora: ProductionLine_Reset on DigitalTwin.Machine); the impact summary counted those rows (MICROFLOW: 9 over six microflows) and printed the types in map order, different between runs. - select distinct, with a total order; the summary counts distinct elements per type, in type order, and the footer gives both numbers. - impact/refs on an enumeration include the edges to its values, with a Target column naming which value. - "(no impact - element is not referenced)" is gone. For an attribute or an enumeration value the message lists the sites that were checked and the ones that are not resolved (a member named through a variable in an expression; a decision branch on an enum), and says to run search first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e listed once `context DigitalTwin.Machine` said "Related Entities: (none found)" for an entity with five associations. The section read refs rows whose SOURCE is an entity, but an association edge's source is the ASSOCIATION, so only generalizations could ever match. It now reads both ends from the associations table (present in a fast catalog too), plus the generalization and the specializations. Also: Direct Callers / Shown By / workflow callers list each source once, and the enumeration context reads the same edge set as impact (type and value uses), grouped into entities, flows and pages. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ring it Studio Pro stores expressions exactly as typed, and a trailing newline left in the expression editor is common. describe interpolated the stored text verbatim, so `change $X (Status = Mod.Enum.Val` / `);` put the closing paren or semicolon on a line of its own (297 such lines in Evora Factory Management's microflows alone). One helper, describeExpr (TrimSpace; interior newlines kept), now renders every stored expression the describer emits: change/create members, set, change-list values, aggregate/reduce, list operations, call microflow / nanoflow / java / javascript / external action arguments, show page args, log node + template params, show message params, REST/web service/DB query params, while, decision and rule arguments, and page widget Visible/Editable conditions, action arguments, datasource arguments and client template parameters. It replaces the five ad-hoc TrimSuffix/TrimRight calls that each covered one slot. The stored model is untouched. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every existing round-trip test starts from MDL, so the document it compares holds only what MDL can say. The losses live in what Studio Pro stores and MDL does not mention. This suite describes documents that ship in Mendix's Blank template (what `mx create-project` produces; byte-identical to the PedApp units audited for #705), re-executes that output unchanged, and compares the stored unit canonically. A whole-document verdict is too coarse to track: a page loses a dozen unrelated things at once. So each case is held to five checks -- exec, document, header, texts, annotations -- and the ledger is keyed per check, failing in both directions. It starts as exactly the #705 losses, plus the whole-document losses this suite found beyond them (input-widget writer constants and a DataGrid2 rebuilt from its template on pages; activity sizes on a nanoflow). A control edits one caption through ALTER PAGE and asserts both the canonical comparison and the texts check see it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`--json` output did not parse. refs, callers, callees, impact, structure,
show, search and `-c "select …"` printed "Connected to:", catalog
load/build progress, a header and a "Found N" count ahead of the payload;
an empty answer was a sentence ("(no references found)") instead of [];
`context` ignored --json; `check --format json` wrote its document to
stderr, one per phase; `diff --json` printed a text diff.
The executor writes the payload and its commentary through the same
ctx.Output, so the mendixlabs#904 mechanism (pick the executor's writer at the cmd
layer) cannot separate them for commands whose payload the executor
itself prints. Add ExecContext.progress(): Output in text mode (so
interactive output is unchanged), the diagnostics stream (stderr) when
Output carries JSON. The Diagnostics field is the one #716
introduces, with the same semantics.
- Status/progress in connect, catalog load/build, catalog queries and the
refs/callers/callees/impact handlers goes through progress().
- Empty results go through writeEmptyResult: [] in JSON mode.
- context --json wraps the markdown in {name, type, depth, context}.
- show catalog status / show widgets gain a JSON path; empty listings of
data transformers, import/export mappings and navigation follow the
existing `&& ctx.Format != FormatJSON` idiom.
- search: --json is the canonical spelling; --format json is kept as a
deprecated alias and now sets the executor format, so it is as clean.
- A cold-cache catalog build under -q moves its per-table lines to stderr
instead of stdout (search -q --format names).
- check: structured formats emit one document on stdout covering every
phase; the exit code is unchanged.
- diff / diff-local refuse --json (no JSON output) instead of ignoring it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…r modify DESCRIBE describes a document that exists, so its output has to re-execute against the project it came from. These three emitted a plain `create`, which fails there with "already exists": the round trip died at the first statement instead of reporting what changed (#705 item 5). Every other describer already emits `create or modify`; `describe module` prints its roles in the same shape as `describe module role`, so it changes with it. With the verb fixed the round-trip suite sees past the refusal: a module role now round-trips exactly; a Java action shows the ExportLevel and ActionDefaultReturnName losses (#705 item 3, next); an association shows #704's Table -> Column flip, and that the domain-model rewrite also drops empty MemberAccess refs and NoGeneralization flags. Both are ledgered. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…p writing "Public" Neither property has an MDL spelling -- DESCRIBE prints them as comments at most -- so CREATE OR MODIFY must carry them. The Java action path hardcoded ExportLevel "Public" and ActionDefaultReturnName "", so describe -> exec of FeedbackModule.XSS_Sanitizer turned Hidden into Public and dropped "ReturnValueName" (#705 item 3). "Public" is not a member of JavaActionsExportLevel or JavaScriptActionsExportLevel (both API | Hidden), so every Java and JavaScript action mxcli CREATED carried an enum value the metamodel does not declare. The JavaScript path had the same hardcode and is fixed with it. A new action now gets what Studio Pro writes on all 69 actions in the Blank template: Hidden and ReturnValueName. The round-trip suite's Java action case now passes every check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ype "Web" on create snippetToGen wrote the header as constants, and the Type constant was "", which is not a member of PagesType (Native | Web). describe -> exec of the Blank template's FeedbackModule._ReadMe turned Type "Web" into "" (#705 item 4), and every snippet mxcli created carried the invalid value. None of the four header properties has an MDL spelling, so UpdateSnippet now carries them off the stored unit, mirroring carryStoredPageHeader (#541) -- including its width-agnostic canvas read. Type matters most: it says whether the snippet is for web or native pages, which a rewrite must not decide. A new snippet gets what Studio Pro writes on every snippet of that template: Web, Hidden, 800 x 600. The round-trip suite's snippet case now passes every check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eference, return variable and mark-as-used describe -> exec of the Blank template's ACT_Feedback_UploadImage lost its three annotation connectors and three header properties (#705 item 2): - nanoflowToGen never wrote ObjectCollection.AnnotationFlows, though the reader fills it and the shared flow builder produces it; microflowToGen always did. With the connectors gone the notes float free and the next DESCRIBE attaches them to nothing. ruleToGen had the same omission and is fixed with it. - ExportLevel and UseListParameterByReference have no MDL spelling and were never written. UpdateNanoflow now carries them off the stored unit -- and only when the stored unit has them, since the writer has never emitted them on a new nanoflow and a key the project's metamodel does not declare makes the document unopenable. - ReturnVariableName is authorable (`returns T as $Var`) but the nanoflow path neither read, printed nor wrote it. It now does all three, the way the microflow path does, and a statement without `as` carries the stored one. - MarkAsUsed was hardcoded false on the rebuild, clearing Studio Pro's "Mark as used"; it is now read and carried like the microflow's. The round-trip suite gains SUB_Feedback_GetOrCreate, which stores a non-empty return variable. Both nanoflows now pass exec, header, texts and annotations; their whole-document verdicts stay ledgered for losses outside #705 (activity sizes, auto-captions, regenerated merges). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
describe -> exec of the Blank template's ShareFeedback and Account_Overview lost translations (#705 item 1). Measuring per owning widget rather than as a multiset showed four distinct causes: - DESCRIBE fell back to another language when the default one was present but EMPTY, and exec writes what describe prints into the default language: a caption stored as en_US "" + nl_NL "Knop" came back as en_US "Knop". A present-but-empty value now wins; the fallbacks remain for a text with no entry in that language (#702's case). - A textarea's placeholder was never read, printed or built -- only a textbox's was -- so it vanished with its nine translations. That was most of the audit's 113 -> 102. - The TextArea writer never emitted TextTooLongMessage, which Studio Pro stores on every textarea; with no text in the rebuild there was nothing for the translation carry to fill. - canon.CarryTranslations paired by whole-document path (lost as soon as a DataGrid2 is rebuilt from its template) or by source string, which cannot tell apart the many (en_US, "") texts or two texts sharing an English source, and cannot look up a rebuilt empty text at all. It now pairs by the named element that owns the text -- ($Type, Name, path from it) -- where that is exact: unique in both documents, and no list index between element and text, since a rebuilt pluggable widget reorders its Properties. Both pages now keep every translation; their whole-document verdicts stay ledgered for losses outside #705. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s and extra code a rewrite cannot express
With describe now emitting `create or modify java action`, its output
re-executes -- and that exposed three ways it destroyed the action's code,
none of which any BSON comparison sees:
- DESCRIBE never read a Java body. Its markers had been lowercased
("// begin user CODE") by the sweep that made MDL keywords lowercase, so no
Studio Pro file matched and the placeholder was always printed; executing it
replaced the code. Markers now match case-insensitively, like the
JavaScript describer's.
- The regenerated file kept only the user code. Studio Pro's banner states
the contract -- the import list, user code and extra code are retained --
and FeedbackModule.XSS_Sanitizer's user code calls sanitize() in its EXTRA
CODE section, so a rewrite produced Java that does not compile. MDL cannot
spell extra code, so WriteJavaSourceFile now retains the existing file's
import list and extra code (javaactions.RetainSections), adding the
statement's imports and taking its extra code when it has one.
- Where DESCRIBE still cannot read a source (add-on modules ship none), the
placeholder body it prints is no longer written as the action's source.
The user code is normalised on read (CRLF, surrounding blank lines, common
indentation) so a second DESCRIBE prints the same as the first.
The round-trip suite gains a `source` check comparing the file's retained
sections; with this change reverted it fails on XSS_Sanitizer.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Six findings (the describe verb, the Java/JavaScript ExportLevel enum, snippet header constants, nanoflow and rule annotation flows, element-anchored translation pairing, the Java source file) plus one on the round-trip suite itself. The rewrite-drops-unauthored-state pattern page gains the three insights that recurred: a hardcoded constant is usually wrong on create as well, a fallback in DESCRIBE is a write, and a describe that cannot re-execute hides every loss behind it. The bug-test script covers the create half and ends in the describes whose output must re-execute as unchanged (4 of 4 on 11.13.0, mx check 0 errors). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
11 tasks
… statement omits it describe omits the storage clause for table storage, and since this branch prints create or modify, re-executing its unchanged output flipped a Studio Pro table association to column storage (#704) - a schema change the plain create used to refuse. The OR MODIFY arms now keep the stored StorageFormat unless the statement states one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… category describe prints a parameter description as a comment the parser drops, so re-executing create or modify output deleted it (PedApp FeedbackModule.ValidateEmail). Carried from the stored parameter of the same name. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sliceBetweenFold sliced the source at indices found in strings.ToLower(s), which is not length-preserving (U+0130, the Kelvin sign), so such a character above the markers cut the user or extra code a byte off. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… trimming expression whitespace Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # mdl/executor/cmd_search.go
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ions, module roles, snippet Type, nanoflow export level); association now getput only (#721 B) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…efix Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #705. Adds a focused round-trip suite that is a step toward #703 (it does not close it: see "Scope").
What was wrong
Running a Studio Pro document's own
describeoutput unchanged lost stored content. The audit's five items all reproduce on the documentsmx create-projectproduces: the Blank template's FeedbackModule / Administration units are byte-identical to the audited PedApp ones, so the integration base project is the Studio Pro-authored fixture.createinstead ofcreate or modify(association, Java action, module role)createcreate or modify(alsodescribe module's role lines)"Public"and""Hidden/ReturnValueName."Public"is not a member ofJavaActionsExportLevelorJavaScriptActionsExportLevel(API | Hidden), so every Java and JavaScript action mxcli created was invalid. The JavaScript path is fixed too.Web→''snippetToGenwrote header constants, and""is not aPagesTypecarryStoredPageHeader(#541). A new snippet getsWeb.nanoflowToGennever wroteAnnotationFlows(ruleToGenneither) or the header keysreturns T as $Var. Carry MarkAsUsed, which was hardcoded to false.Page texts:
'', describe fell back to another language, and exec wrote that value into the default language (en_US '' + nl_NL 'Knop'→en_US 'Knop'). A present-but-empty value now wins.canon.CarryTranslationsnow pairs by the named element that owns a text: its$Type,Name, and the path from that element. Two conditions keep this exact: the address must be unique in both documents, and no list index may sit between the element and the text. The old pairings failed here. Positional pairing breaks once a DataGrid2 is rebuilt, and source pairing cannot resolve(en_US, "")or a shared English source.Found on the way (same path, now fixed)
Fixing the verb made Java action output re-execute, and that surfaced three losses in the
.javafile that no BSON comparison sees:"// begin user CODE") by the lowercase-keywords sweep (00b80f3). It always printed the placeholder, so executing the output would have replaced the code.XSS_Sanitizer's user code callssanitize()in its extra code, so a rewrite produced Java that doesn't compile.WriteJavaSourceFilenow retains both.The test
mdl/executor/studiopro_roundtrip_test.go(-tags integration) runs describe → exec → canonical compare per unit. Each case has six checks: exec, document (GetPut + PutGet), header, texts, annotations, source. The ledger is keyed per check and fails in both directions. Every fix commit strikes its entries off, so each one shows the ledger entry failing before the fix and passing after.Controls:
TestStudioProRoundTripControledits one caption and asserts that both the canonical comparison and the texts check see it. Thesourcecheck was run with the Java fix reverted and fails on XSS_Sanitizer.Final state: the Java action, snippet and module role round-trip exactly. Both nanoflows and both pages pass every #705 slice. Their whole-document verdicts stay ledgered, with reasons, for losses outside #705 that this suite found:
Scope
describe snippetdrops a dataview'sDataSource: $Param.Validation
make build,go test ./...,make lint-go,make check-mdl: all passgo test -tags integration ./mdl/executor/(whole package, mxbuild 11.13.0): passesmdl-examples/bug-tests/describe-roundtrip-705-carried-properties.mdlon a fresh 11.13.0 blank app: the create half builds withmx checkat 0 errors, and re-executing its describes reports "4 documents already in sync".Findings recorded (7), and the
rewrite-drops-unauthored-statepattern page gains the three insights that recurred.🤖 Generated with Claude Code
Also closes #704: the review added the association storage carry (commit 6688f78), with a test.