diff --git a/.claude/skills/fix-issue/findings/mdl-executor.jsonl b/.claude/skills/fix-issue/findings/mdl-executor.jsonl index 3d0969c6e4..d080cddc4f 100644 --- a/.claude/skills/fix-issue/findings/mdl-executor.jsonl +++ b/.claude/skills/fix-issue/findings/mdl-executor.jsonl @@ -515,3 +515,23 @@ {"area": "mdl/executor", "date": "2026-09-04", "symptom": "A navigation menu's LOG-OUT item could not be authored and did not survive a round trip. MDL's `menu item` took PAGE or MICROFLOW only, so there was no spelling for it; and ako/TestApp's sign-out menu item read back as a plain `menu item 'Item 5';`, so DESCRIBE -> exec turned a working log-out entry into a dead one \u2014 silently, with `mx check` clean.", "cause": "A menu item's action goes through FOUR places that share no code with the button path: menuActionToGen (menu document, modelsdk), navMenuAction (navigation profile, raw BSON), resolveMenuAction (modelsdk read) and parseNavMenuItem (legacy read). Both writers ended in a NoAction default and both readers left the type name unmapped. Added SIGN_OUT to navMenuItemDef in the grammar (it consumes no qualifiedName, so it is read separately from the PAGE/MICROFLOW switch or an ICON after it is mis-assigned), carried it as ActionType \"SignOutAction\" / NavMenuItemSpec.SignOut, and wired all four. Studio Pro stores the same Forms$SignOutClientAction a button carries: DisabledDuringExecution true, nothing else.", "file": "`mdl/grammar/MDLParser.g4` (navMenuItemDef), `mdl/ast/ast_navigation.go`, `mdl/visitor/visitor_navigation.go`, `mdl/executor/cmd_menus.go` + `cmd_navigation.go` (conversion + printMenuMDL + the show summary), `mdl/types/navigation.go`, `mdl/backend/modelsdk/menu_write.go` + `navigation_write.go` + `navigation_read.go`, `sdk/mpr/parser_misc.go`; example `mdl-examples/bug-tests/captrack-10-sign-out-menu-item.mdl`", "insight": "A round trip closes only if the READER produces the exact string the WRITER consumes \u2014 here both readers had a raw-type-name fallback that looked like it preserved information (ActionType became \"Forms$SignOutClientAction\") while breaking the round trip, because DESCRIBE and the writers key on \"SignOutAction\". A fallback that stores the raw name is not the same as handling the case, and it hides the gap better than a NoAction default would. Also: the same logical action reaches storage through four unrelated switches (two writers x two constructs, two readers), so fixing the button path proved nothing about the menu path \u2014 grep for every switch on the action before calling such a fix complete. Controlled by neutralising both readers and re-reading TestApp: `Item 5 -> sign out` goes back to `Item 5`."} {"area": "mdl/executor", "date": "2026-09-04", "symptom": "`ACTIONBUTTON \u2026 (Action: OPEN_LINK 'https://\u2026')` was written by neither engine: modelsdk refused it, legacy fell through to its quiet default and wrote Forms$NoAction, so the button rendered and did nothing with check, exec and mx check all clean.", "cause": "Same missing-case defect as SIGN_OUT, but with two traps a reference settled and reasoning would not. (1) The STORAGE NAME is Forms$OpenLinkClientAction, while the semantic type is LinkClientAction and the executor stamped `Forms$LinkClientAction` \u2014 a wrong $Type that never reached disk only because nothing could write the action. (2) The address is not a string field but a nested Forms$StaticOrDynamicString. Pinned against 31 Studio Pro link buttons (ako/TestApp, FeedbackModule): exactly five keys, LinkType \"Web\" in all 31, and 6 of 31 DYNAMIC (IsDynamic true + AttributeRef + empty Value). MDL authors the static form only, so DESCRIBE flags a dynamic one instead of printing its address as a literal.", "file": "`mdl/backend/modelsdk/widget_write.go` (clientActionToGen + staticAddressToGen), `sdk/mpr/writer_widgets_action.go`, `mdl/executor/cmd_pages_builder_v3.go` ($Type), `mdl/executor/cmd_pages_describe_output.go`, `cmd/mxcli/syntax/features_page.go`; example `mdl-examples/bug-tests/captrack-10-open-link-action.mdl`", "insight": "gen declares a fourth property on Forms$StaticOrDynamicString \u2014 `Attribute` \u2014 that not one of the 31 stored documents carries. Writing it would be the 'never invent a key' failure: a document mxbuild accepts and Studio Pro cannot open. When gen offers more properties than the references show, the references win. Second lesson, about controls: the SIGN_OUT commit used LinkClientAction as its 'still unimplemented' control, and implementing OPEN_LINK silently invalidated it \u2014 the test then failed for a good reason, but a control naming a specific unimplemented feature has a shelf life. Point it at something structurally unwritable instead (ShowHomePageClientAction: no gen type, no metamodel counterpart, no MDL statement that builds one)."} {"area": "mdl/executor", "date": "2026-09-04", "symptom": "A generated domain model opens in Studio Pro as ONE horizontal line of entities, boxes touching, unreadable at any zoom. Reported on a 40-entity model (ako/CapTrackV2, Mendix 11.13).", "cause": "The default position for a CREATE ENTITY with no `@Position` was `model.Point{X: 100 + len(dm.Entities)*150, Y: 100}` \u2014 same y for every entity ever created, x stepping by 150. 40 entities = a 6,950px row; and 150px is narrower than an entity box, so they also overlapped. Replaced with a wrapping grid in the new `mdl/dmlayout` package, and added `mxcli layout` for a real layered layout off the association graph.", "file": "`mdl/executor/cmd_entities.go` (the default), `mdl/dmlayout/dmlayout.go` (new: GridSlot + Plan), `cmd/mxcli/cmd_layout.go` (new command)", "insight": "The default could not have been much better than a grid, and that is the design point: the first entity of a script is placed before the last one exists, so no create-time rule can see the graph. Layout needs the whole model, so it belongs in a separate pass, not as a side effect of authoring \u2014 and because it necessarily overwrites hand-arranged positions it has to be opt-in with a dry run. Two constraints that are easy to miss: an entity stores only Location and NO Size (Studio Pro derives the box when it draws), so spacing must be estimated from name length and attribute count; and a Mendix position is the box's CENTRE, not its top-left, so placement adds half a box. Determinism is load-bearing rather than cosmetic \u2014 an unsorted walk gives a different diagram every run, which rewrites the unit every time and is exactly the churn ADR-0008 exists to prevent (the test catches it on run 0)."} +{"area": "mdl/executor", "date": "2026-09-04", "symptom": "`exec` fails with `no definition for widget com.mendix.widget.web.fileuploader.FileUploader (run 'mxcli widget init -p app.mpr')` and running that command changes nothing. `check` passes. Also affects Events, Google Tag and Markdown viewer.", "cause": "The project has no widget package for it in widgets/, so there is nothing to extract — and the remedy named in the error (widget init) scans exactly that directory.", "file": "mdl/executor/cmd_pages_builder_v3.go", "insight": "Branch the message on whether the package is installed, using the same FindMPK lookup the template loader makes. The important correction is to the PREMISE: these are not Studio-Pro-bundled widgets mxcli should ship definitions for. Measured — a blank 11.13 project ships 33 widgets and none of these four; installing File Uploader (Marketplace module 235351) takes widgets/ from 33 to 34 and the page then builds with NO widget init, because initPluggableEngine refreshes definitions from installed packages itself. A widget whose package is absent is one Studio Pro cannot use either, so there is nothing to ship. Two dead ends first, both of which looked settled: the .mpk files are NOT in Mendix.Modeler.Core.dll (690 embedded zips, zero widgets.mendix.com hits — the hit was a bare ID string), and a .def.json alone is insufficient because getOrGenerateTemplate derives the template from the .mpk in widgets/, so it only moves the error to 'template not found'. Both were chased before anyone asked whether a Studio Pro user could use the widget at all.", "refs": ["mendixlabs/mxcli#1036"]} +{"area": "mdl/executor", "date": "2026-09-04", "symptom": "A widget is the only MDL extension point with no in-language DESCRIBE: microflows, nanoflows, Java actions and JavaScript actions all have `DESCRIBE Module.Name`, while a widget needs the CLI `mxcli widget describe`.", "cause": "No DESCRIBE WIDGET statement existed, so `mxcli widget init` generated markdown to fill the gap — and that generated documentation could drift from what the parser accepts, which is what mendixlabs/mxcli#1036 reported.", "file": "mdl/executor/widget_describe.go", "insight": "Move the description builder OUT of cmd/ into the executor and have both the statement and the CLI call it, so the two cannot disagree — cmd already imports executor, so the dependency direction was already right. Two details worth copying: DESCRIBE WIDGET must work with NO project (DescribeFragment was the existing precedent for the exemption in execDescribe), because 'what can I write here?' is asked before anything is open; and the refactor needs a byte-for-byte output comparison against the pre-refactor binary, which is fiddlier than it looks — `git stash -- ` silently no-ops on an untracked file, and a fresh worktree cannot build because generated embed dirs (skillpacks) are absent. Move the files aside and `git checkout HEAD -- ` in place instead.", "refs": ["mendixlabs/mxcli#1036"]} +{"area": "mdl/executor", "date": "2026-09-04", "symptom": "The generated widget .md leads with an 'MDL Example' that fails on its own first line — `mismatched input 'tagcontentcontainer' expecting '}'` — because the generator derives keywords from the .mpk while the grammar accepts a hardcoded nine.", "cause": "The example was assembled from the widget definition with no reference to what the parser accepts, so it could promise any syntax the def implied.", "file": "mdl/executor/widget_describe.go", "insight": "An example is only worth emitting if it is PARSE-VERIFIED. Build it, then run it through visitor.Build and refuse to emit it if it fails; choose the head form and each container by the same probe. That makes the example unable to promise syntax that fails, and makes it widen on its own when the grammar gains ground — no second list to keep in sync, which is the whole defect class here. Two traps found while writing it: numbering matters (two child slots both named slot1 PARSE but are invalid on one page — the parser does not check names, and the .md generator had the identical bug), and required properties of type datasource/attribute/action/expression must be omitted-and-named rather than filled with invented values, since a made-up entity name parses fine and then fails at exec.", "refs": ["mendixlabs/mxcli#1036"]} +{"area": "mdl/executor", "date": "2026-09-04", "symptom": "DESCRIBE WIDGET's example asked for eleven bindings on Combo box, across options-source modes that are mutually exclusive — overstating what a reader must supply. Separately, the same example emitted NO properties at all when described without a project.", "cause": "Two independent defects. (1) A widget's properties are 'required' only where the editor shows them, and the required list was used raw. (2) The two description sources spell property types differently — a project .mpk gives 'datasource', the embedded template 'DataSource' — and the example's type switch matched only the lowercase spelling.", "file": "mdl/executor/widget_describe.go", "insight": "Prune required bindings by the visibility rules the description already reports, evaluated against the configuration the example itself describes; types.WidgetVisibilityCondition.Hidden is already exported and MDL-WIDGET10 uses it the same way. Be conservative in the direction of over-listing: an indeterminable condition must NOT prune, and a nested (object-list item) rule must never prune the widget's own property. The ceiling is rule EXTRACTION coverage, not the pruning — combobox reports '16 of 32 editor hide-rules recognized', which is why 11 becomes 6 rather than 2, and why attributeEnumeration survives with zero recognized rules. The casing bug is the more general lesson: the two sources' type vocabularies differ, and a test that only checked 'the example parses' passed vacuously against an example that had been emptied — a control asserting the example still ASKS for a visible binding is what caught it.", "refs": ["mendixlabs/mxcli#1036"]} +{"area": "mdl/executor", "date": "2026-09-04", "symptom": "`mxcli check` PASSES on two widget mistakes that fail at exec: an unknown widget id (`pluggablewidget 'com.acme.NotAWidget' w1`) and a real container keyword on a widget that has no such container (`group` inside HTML Element). Only a typo'd widget KIND is caught, and that is the parser, not the validator.", "cause": "widgetTypeV3 is effectively the widget-kind validator — the grammar's allow-list is what rejects an unknown kind, so the validator never needed an independent notion of one. validateWidgetTreeIn already computes both facts (parentObjectLists[w.Type] for the container, lookupWidgetDef for the kind) but reports neither; the branch routes to validateStaticWidgetUnknownProps, which checks the properties of a presumed static widget instead of questioning the kind.", "file": "mdl/executor/validate_widgets.go", "insight": "Found while settling Open Question 1 of PROPOSAL_def_driven_widget_bodies.md — whether making the widget body def-driven would cost error quality. It would, but the more useful finding is that the hole is ALREADY open for everything that reaches the validator, so closing it is an improvement today and independent of any grammar change. The general lesson: when a grammar's allow-list is doing validation work, removing it needs the semantic check written FIRST, and the cheapest way to discover what the validator really catches is to find an input that already bypasses the parser — here `pluggablewidget ''`, which parses today and lands on exactly the path the generic form would create.", "refs": ["mendixlabs/mxcli#1036"]} +{"area": "mdl/executor", "date": "2026-09-04", "symptom": "Two widget mistakes passed `mxcli check` and failed only at `exec`: an explicit widget id resolving to nothing (`pluggablewidget 'com.acme.NotAWidget' w1`), and a real container keyword on a parent with no such container (`group` inside HTML Element).", "cause": "The GRAMMAR was the widget-kind validator — widgetTypeV3 is an allow-list, so an unknown kind could not parse and the validator never grew an independent notion of one. Neither of these is a keyword the parser checks. isUniversalObjectListKeyword actively SUPPRESSED the second case by treating a container keyword as always-an-item wherever it appeared.", "file": "mdl/executor/validate_widget_kind.go", "insight": "THREE guards make these rules safe rather than a false-positive storm, and each was found by a control or a CI target failing. (1) With NO project the registry holds only the nine embedded widgets, so every real project widget looks unknown — MDL-WIDGET25 must stay silent without -p. Measured the hard way: one example file produced 14 violations, and `make check-mdl` broke seven files, because the corpus is checked WITHOUT a project while I had measured only with one. (2) With a project, LoadWidgetRegistry reads only .mxcli/widgets/*.def.json and does NOT refresh from installed .mpk files, so an id whose package IS installed must be treated as real (same FindMPK lookup slice 1's error message uses). (3) A container is never judged against a parent whose definition could not be resolved. Scope guard (1) to the widget-id branch only — the container rule needs a resolvable PARENT, not a project, and a blanket early return kills it. Two process traps: CHECK THE RULE-ID SPACE first (MDL-WIDGET23 was taken by validate_widget_onclick.go, and its own test caught the collision), and a rule that needs a project CANNOT be demonstrated by a .fail.mdl — make check-mdl runs without one, so the file reports 'negative test unexpectedly passed' and makes a working rule look regressed (the Makefile documents this as #891/#892). Keep the repro a plain .mdl and cover the rule with unit tests. Finally, isUniversalObjectListKeyword was a FOURTH incomplete keyword list (7 against the grammar's 9, missing SCALECOLOR/CUSTOMBUTTON/ALLOWEDFILEFORMAT); the replacement derives the set from the registry.", "refs": ["mendixlabs/mxcli#1036"]} +{"area": "mdl/executor", "date": "2026-09-05", "symptom": "Two runs of the SAME mxcli binary over mdl-examples/ produced different `mxcli check` output on 11 of 515 scripts. Same warnings, different order.", "cause": "validateStaticWidgetUnknownProps (MDL-WIDGET07) and the WIDGET17/WIDGET18 validators append one violation per property while ranging over w.Properties directly; Go randomises map iteration.", "file": "mdl/executor/validate_widgets.go", "insight": "Sort the keys (sortedPropertyKeys) at the three sites that emit PER KEY. Leave the two loops that do a case-insensitive LOOKUP and break on first hit: they are only order-sensitive when a widget carries two keys differing solely in case, and either answer is correct. Found not by a bug report but by needing check output as a MEASUREMENT instrument for a grammar change - the noise floor was larger than the signal. A validator that emits per map key is nondeterministic output, and the PR checklist's 'map iteration is deterministic' item covers exactly this. CONTROL: revert the sort and the 8-property regression test fails on the first comparison.", "issue": "mendixlabs/mxcli#1036"} +{"area": "mdl/executor", "date": "2026-09-05", "symptom": "Once a bare identifier was accepted as a widget type, the typo `htmlelemnt frame (tagName: 'div')` reported '0 errors, 1 warning' - a MDL-WIDGET07 warning about `tagName` - while the correct spelling was completely clean. check exited 0.", "cause": "An unresolved generic widget type falls through to validateStaticWidgetUnknownProps, which validates properties against the builtin vocabulary. It complains about the property because it has already assumed the kind is a built-in.", "file": "mdl/executor/validate_widget_kind.go", "insight": "The AST must record WHICH grammar alternative matched (ast.WidgetV3.TypeIsGeneric), set from the parse tree in the visitor - never inferred by comparing the type text against a list of known widget names, which would reintroduce the list the change exists to remove. A generic type that resolves to nothing is MDL-WIDGET25/26 (kind is wrong), and property validation must be SUPPRESSED for it or the message points at the wrong token. Generalises: when a grammar is loosened, the check the parser used to perform must move to the validator in the same change, or a parse error silently becomes a wrong answer. CONTROL: the correct spelling must stay completely clean under the same conditions.", "issue": "mendixlabs/mxcli#1036"} +{"area": "mdl/executor", "date": "2026-09-05", "symptom": "DESCRIBE PAGE on a page mxcli itself authored emitted only the widget head: `htmlelement frame (tagName: 'div', ...)`. The child slot `tagcontentcontainer body { dynamictext t }` and the object list `attribute a1` were absent, silently, at exit 0 \u2014 so describe -> edit -> exec DELETED the widget's body.", "cause": "The write path is correct (the stored BSON carries tagContentContainer with its DynamicText and attributes with the data). DESCRIBE's generic pluggable branch reconstructs only object lists of the chart shape extractObjectLists was built for, and reconstructs child slots not at all. The gap predates the fix but became reachable the moment slices 2-3 made those containers writable from MDL.", "file": "mdl/executor/cmd_pages_describe_omitted.go", "insight": "Until reconstruction exists, emit the gap as an MDL comment rather than a bare head that reads as complete (unreconstructedContainers). Two traps in doing that. (1) getBsonArrayElements STRIPS the leading typed-array marker, so an empty container is length 0 after stripping and length 1 in the raw BSON \u2014 checking the raw length reports every widget as lossy. (2) Warn on CHILD SLOTS HOLDING WIDGETS only, not object lists: a widget template ships DEFAULT entries in its lists that are structurally identical to a user's, so the first version named `event` on a page that never wrote one. A note that fires on defaults is noise and trains people to ignore the notes that matter. CONTROL both ways: 16 real pages in testdata/expr-checker produce 0 notes, and the authored page produces exactly one naming tagcontentcontainer.", "issue": "mendixlabs/mxcli#1036"} +{"area": "mdl/executor", "date": "2026-09-05", "symptom": "DESCRIBE PAGE kept emitting `pluggablewidget '' name` for every widget after being changed to prefer the MDL name, with no error.", "cause": "LoadWidgetRegistry wants the .mpr PATH; LoadUserDefinitions takes filepath.Dir of it internally. Passing filepath.Dir(ctx.MprPath) looked one level above the project and found no definitions, so every lookup missed and the code fell back exactly as designed.", "file": "mdl/executor/exec_context.go", "insight": "A correct fallback hides a wiring bug perfectly: the output stays valid, so nothing fails and no test goes red. When adding a 'prefer X, else Y' path, assert the X branch is actually reached on a real project, not just that the output parses. Related: the registry is keyed BY MDL NAME, so two definitions claiming one name leave Get and GetByWidgetID disagreeing and All() cannot see the collision at all \u2014 guard by round-tripping the name through the same lookup the builder uses, never by counting definitions.", "issue": "mendixlabs/mxcli#1036"} +{"area": "mdl/executor", "date": "2026-09-05", "symptom": "DESCRIBE PAGE dropped an entire object list from a pluggable widget: HTML Element's `attributes` (holding a real value) and `events` were absent from the output, silently, at exit 0. Chart series described fine, so it looked widget-specific.", "cause": "extractObjectListItem tests a value's fields in order and the ACTION branch continued UNCONDITIONALLY once `value[\"Action\"]` merely existed. A widget value carries every field it could have, and Action is always present as a Forms$NoAction \u2014 so that branch consumed all six sub-properties of an item, the item ended with zero Props, and the caller's `len(item.Props) > 0` filter dropped it, taking the whole list with it.", "file": "mdl/executor/cmd_pages_describe_objectlist.go", "insight": "A branch that consumes on KEY EXISTENCE rather than on EXTRACTION is the bug shape; guard on len(map) > 0 and continue only when something was produced. Two measurement lessons cost more than the fix. (1) My first control used python str.replace with no assertion, matched nothing, and was VACUOUS \u2014 it 'passed' while changing no code, the same trap as a test that only skips. Assert the replacement applied. (2) The second control flipped the three conditions but missed that a `continue` had MOVED inside the Action branch, so it reverted the wrong thing and pointed at DataSource; only `git diff -U0` filtered of comments revealed the moved line. Isolating one branch at a time gave the answer: reverting Action alone takes object lists 2 -> 0. A confident wrong root cause in a comment is worse than none.", "issue": "mendixlabs/mxcli#1036"} +{"area": "mdl/executor", "date": "2026-09-05", "symptom": "A child slot on any pluggable widget other than Gallery was absent from DESCRIBE PAGE, so describe -> exec DELETED the widget's body. Measured: 4 widgets before the round trip, 1 after.", "cause": "DESCRIBE reconstructed a Gallery's `content` and `filtersPlaceholder` by asking for those property keys BY NAME (extractGalleryWidgetsByPropertyKey) and had nothing generic. Invisible until slices 2-3 made such a slot writable from MDL.", "file": "mdl/executor/cmd_pages_describe_childslots.go", "insight": "A child slot is ANY property whose Value holds a Widgets array \u2014 read it off the document instead of looking the key up by name, and a widget nobody has thought about round-trips for free. Skip empty ones: getBsonArrayElements strips the typed-array marker, so empty is length 0 here and length 1 in raw BSON, and emitting them puts a `slot { }` on nearly every widget. NOTE the round trip converges rather than being a fixed point on the first pass: the writer emits an item's properties in a different order than the original document, so describe #1 != describe #2 but describe #2 == describe #3. Describe itself is deterministic (5 identical runs). A fixed-point assertion alone would NOT have caught the original bug \u2014 an empty describe is also a fixed point; assert that the CONTENT survives.", "issue": "mendixlabs/mxcli#1036"} +{"area": "mdl/executor", "date": "2026-09-05", "symptom": "`ValueAttribute: Total` on a PieChart/HeatMap was reported MDL-WIDGET01 \"has no property `ValueAttribute`\", and because exec refuses a script with errors the page could not be written at all. The value itself persisted correctly (DESCRIBE returns `seriesValueAttribute: Total`).", "cause": "allowedWidgetProperties built its set from each PropertyMapping's PropertyKey and Source, never its MdlAliases. The def.json declares mdlAliases:[\"ValueAttribute\"], the BUILDER resolves through it (widget_engine.go), and the knownProperties set in widget_defs.go walks it — the validator was the only one of three readers of the same def.json that did not.", "file": "mdl/executor/validate_widgets.go", "insight": "When a def.json field is consumed in more than one place, the checker is the one that gets forgotten, because a checker that is too strict still 'works' until someone writes the documented syntax. Grep for every reader of a field before adding a fourth. The give-away here: the file's own header claimed `mxcli check`-clean, which was true only because `make check-mdl` runs WITHOUT a project, so no widget definition loads and the whole rule is inert.", "issue": "mendixlabs/mxcli#1036"} +{"area": "mdl/executor", "date": "2026-09-05", "symptom": "MDL-WIDGET10 warned `dynamicDataSource is hidden when its own dataSet is \"static\" — the value will be ignored` on every chart series written the documented way. 11 warnings on 34-chart-widget-examples.mdl, one per series, none of them real.", "cause": "A chart's two datasource sub-properties SHARE the Source name \"DataSource\" (measured on linechart.def.json: staticDataSource and dynamicDataSource both declare it, neither declares an alias). itemValueMap resolved by Source, so the one friendly `DataSource:` marked BOTH explicit. buildObjectListItem routes on dataSet (seriesDataSourceMatchesMode) and writes only one.", "file": "mdl/executor/validate_widget_hidden.go", "insight": "A Source is not a unique key. Where two mappings share one, the checker has to reproduce whatever disambiguates them in the WRITER — here the dataSet mode — or it reports properties the script never wrote. Scope the gate to the case the writer scopes it to (chart series datasources); a general 'skip shared sources' would blind the rule everywhere else. Control that catches over-reach: a non-chart item property sharing a Source must still resolve.", "issue": "mendixlabs/mxcli#1036"} +{"area": "mdl/executor", "date": "2026-09-05", "symptom": "`check -p` on a view entity reported \"attribute 'Units': declared as Integer but OQL expression 'sum(s.Units)' returns Decimal. Fix: change to 'Units: Decimal'\" — and following that hint BREAKS the build.", "cause": "inferAggregateType's SUM branch fell back to Decimal when the argument type could not be resolved. The argument is unresolvable exactly when the source entity is created by the same script, since check skips references to script-created objects — i.e. the common shape for a view entity. inferTypeStatic's own SUM branch already said 'return Unknown, do not guess Decimal'; the project-aware path disagreed with it.", "file": "mdl/executor/oql_type_inference.go", "insight": "Measured on mxbuild 11.6.6, two views over the same sum(s.Units) where Units is Integer: declared Integer = 0 errors, declared Decimal = CE6770. The diagnostic inverted the truth, so it did not merely cry wolf — its Fix: walked a working project into a broken one. When a fallback has to guess, return Unknown; a skipped column is a missed error, a wrong guess is a manufactured one. The control that makes the mxbuild evidence mean anything: a deliberately wrong column type in the same app DOES fail CE6770, so mxbuild was really validating view entities.", "issue": "mendixlabs/mxcli#1036"} +{"area": "mdl/executor", "date": "2026-09-05", "symptom": "Three charts on one page, each with a series the author named `s`, failed check with \"duplicate widget name 's' (used 3 times) — Mendix requires unique widget names per page (CE0495)\". mxbuild 11.6.6 reports 0 errors on the same page.", "cause": "checkDuplicateWidgetNames counted every named node in the tree. An object-list item (a chart `series`, a gallery `customitem`) is a WidgetV3 child in the AST but a WidgetObject in the model, and the model stores no name for it.", "file": "mdl/executor/validate_page_context.go", "insight": "The proof that a name is not stored is free: author `series sRegion` and DESCRIBE it back — it returns `series series1`, because DESCRIBE has to synthesise what the document does not carry. A name the model does not hold cannot be a CE0495 duplicate; the same reasoning already excluded rows and columns in widgetKindsWithoutStoredNames. Read the container keywords from the parent's def.json objectLists rather than listing them — the containers are def-driven, so a keyword table would drift the moment a widget ships a new list. The rule needs the registry threaded into validatePageContextTree, and must fall back to its old behaviour when there is none (check runs with no project in CI).", "issue": "mendixlabs/mxcli#1036"} +{"area": "mdl/executor", "date": "2026-09-05", "symptom": "DESCRIBE WIDGET's generated example named properties mxcli's own validator rejected with MDL-WIDGET01 \"has no property\" — 33 across combobox/gallery/barcodescanner/image, 78 over the whole property surface. Since exec refuses a script with errors, barcodescanner had no legal spelling: name them and exec refuses, omit them and mxbuild reports CE0463.", "cause": "Two readers of one widget. DESCRIBE parses the project's .mpk; the validator reads the WidgetDefinition. They agree for most widgets because the .def.json cache is GENERATED from the .mpk — but nine widgets are hand-crafted in sdk/widgets/definitions/ and deliberately never extracted per-project, so their property list is whatever someone typed (combobox: 73 in the .mpk, 7 mapped + 4 known).", "file": "mdl/executor/widget_known_props_from_mpk.go", "insight": "The fix is KNOWN, not ALLOWED: an unmapped .mpk property becomes MDL-WIDGET06 (\"recognized but not yet persisted; a non-default value will be dropped\"), never silently accepted by a write path that does not exist — trading a false error for a silent drop is the worse bug. Apply it to every definition rather than to the nine: recomputing a generated def reproduces what generation put there, so it is idempotent where redundant, and naming the nine is the same hand-maintained list one layer up. The guard to write is the ROUND TRIP between the two readers (every property DESCRIBE emits is one the validator does not call nonexistent), not \"the example parses\" — it already parsed.", "issue": "mendixlabs/mxcli#1036"} +{"area": "mdl/executor", "date": "2026-09-05", "symptom": "MDL-WIDGET10 warned that a property \"is hidden … the value will be ignored\" on 32 of mxcli's OWN generated widget examples — every warning naming a property the example had just written itself (videoplayer emits heightUnit: 'aspectRatio' then the height that choice hides).", "cause": "Two independent bugs with one shape. (1) hiddenUnder gated only the branch asking for a BINDING and not the scalar branch that writes literals, so half the properties skipped the narrowing. (2) exampleValues resolved values from the .mpk while the validator resolves them from the WidgetDefinition's mapping — a selection property has no defaultValue in the .mpk, so gallery's itemSelection looked indeterminable.", "file": "mdl/executor/widget_describe.go", "insight": "When a generator and a checker implement the same rule, they must read the same FACTS, not just run the same logic — mirror the checker's value resolution exactly, including its implicit fallbacks (a selection with no default is written as None; that is the builder's behaviour, not a guess). Two tests are needed because one is blind: the structural one can only see what the generator itself considers hidden, so the residue from cause 2 is invisible to it — only building the example, parsing it and running the REAL validator catches that. And the control must assert pruning FIRES (29 scalars pruned across 36 widgets): \"emits nothing hidden\" is also satisfied by emitting nothing, and by a hiddenUnder that always returns false.", "issue": "mendixlabs/mxcli#1036"} +{"area": "mdl/executor", "date": "2026-09-05", "symptom": "MDL-WIDGET08 rejected mxcli's own generated example: \"property `dataSet` has invalid value `…` — valid values are static, dynamic\", 11 of 42 widgets. The example block claims 'parses as written' and did parse; it did not CHECK as written.", "cause": "The object-list item line was built as ItemKeys[0] + \": '…'\" — a hardcoded placeholder — instead of deriving a literal the way the widget's own scalars do.", "file": "mdl/executor/widget_describe.go", "insight": "Two sources had to be consulted and the second is the lesson: propsFromMPK carries item sub-properties as Children with their enums, which covers most widgets, but ParseMPKForWidget returns 0 children for a PopupMenu's basicItems while the DEFINITION carries {\"propertyKey\":\"itemType\",\"value\":\"item\",\"enumValues\":[...]}. The definition wins, because it is what MDL-WIDGET08 checks against. Scope the assertion to what the checker rejects — \"no MDL-WIDGET08\", NOT \"no ellipsis anywhere\": a free-text sub-property has no correct value to invent and the validator accepts any string, so a placeholder there is honest output and testing for the character would be testing the wrong thing.", "issue": "mendixlabs/mxcli#1036"} +{"area": "mdl/executor", "date": "2026-09-05", "symptom": "CI-only test failure: 'no widget has an authorable object list with item properties — nothing here would exercise the item literal'. Green locally, red on a fresh checkout.", "cause": "The test built its registry from testdata/expr-checker, whose .mxcli/widgets/*.def.json cache is DERIVED and gitignored. Locally that supplies 33 definitions (charts, DataGrid2, HTML Element — the ones with object lists); in CI only the hand-crafted definitions in sdk/widgets/definitions/ load, and none of those declares an authorable object list with item properties.", "file": "mdl/executor/widget_example_item_literal_test.go", "insight": "Reproduce a CI-only failure by moving the gitignored artifact aside (`mv testdata/expr-checker/.mxcli/widgets /tmp/...`) — instant and exact, no pushing to find out. The deeper rule: a fixture's COMMITTED inputs (widgets/*.mpk) are fair game, its derived caches are not, and the split is invisible until CI. Structure the pair as hermetic-test-carries-the-guarantee + end-to-end-SKIPS-when-it-cannot-run, rather than a single test that Fatals on an environment it does not control. Note the failure was the test's own vacuity control firing correctly — that is the control doing its job, not a false alarm, and the fix is to make the assertion runnable everywhere rather than to weaken the control.", "issue": "mendixlabs/mxcli#1036"} diff --git a/.claude/skills/fix-issue/findings/mdl-grammar.jsonl b/.claude/skills/fix-issue/findings/mdl-grammar.jsonl index e0ed86139d..8bed9f5ccc 100644 --- a/.claude/skills/fix-issue/findings/mdl-grammar.jsonl +++ b/.claude/skills/fix-issue/findings/mdl-grammar.jsonl @@ -51,3 +51,5 @@ {"area": "mdl/grammar", "date": "2026-08-27", "raw": "| After ako/mxcli#260, ten of the 327 demo-app mappings still describe into MDL that does not parse — an export root printed as `. {`, and a custom-handler parameter printed as `Suggestion: (Value)` | Two leftovers of the same families. (1) `group as` covers a nested entity-less node (#262) but a ROOT has no member name, so it had no spelling at all. (2) `customHandlerParamText` rendered the stored value path raw, so an array-of-primitives parameter leaked Mendix's `(Wrapper)`/`(Value)` markers | `mdl/grammar/domains/MDLDomainModel.g4` (`exportMappingRootElement`), `mdl/executor/cmd_export_mappings.go` (the entity-less-root branch, and the association-less handling), `mdl/executor/mapping_customhandler.go` (`customHandlerParamText`, `buildCustomHandler`) | **Printing the member is only half of it**: emitting `Value` while the builder concatenates it back as `…|(Wrapper)|Value` gives a path that resolves to nothing, so resolve the parameter path THROUGH the schema index (`resolvePathKind(..., true)`) the way a member reference is resolved. Two things only a real build caught, both invisible to `mxcli check`: an entity-less root fell through to the VALUE branch and produced a project mxbuild cannot LOAD (*\"Type ExportValueMappingElement does not contain a constructor with a parameter of type ExportMapping\"*), because the builder decides object-vs-value on `def.Entity != \"\"`; and an element with NO association cannot be `Find` — **CE0224 \"No association selected for obtaining objects.\"** — it is `Parameter`, which is what CapitalConnector.EM_AttachedDataRequest stores on both its elements. Took the corpus from 317/327 parsing to **327/327**. Repro `mdl-examples/bug-tests/mapping-260b-last-parse-failures.mdl` |", "refs": ["#262", "ako/mxcli#260"], "ce": ["CE0224"]} {"area": "mdl/grammar", "date": "2026-08-28", "raw": "| A JSON structure's ARRAY ITEM element gets a derived name (`LinesItem`, `JsonObject`) that no MDL can change, and every mapping over the structure carries it — `describe` of a Studio Pro mapping then diffs on `ExposedName`. The obvious workaround, `custom name map ('lines\\|(Object)' as 'OrderLine')`, parses, executes and does **nothing** | An array's item is the anonymous `[…]` entry, so it has no JSON key; `customNameMap` is keyed on JSON keys, so the item was unreachable by construction. And an entry matching no key was applied to nothing and reported nothing, so the failed workaround was indistinguishable from success | grammar `mdl/grammar/domains/MDLDomainModel.g4` (`customNameMapping` gains `ITEM OF`), `mdl/ast/ast_jsonstructure.go` (`CustomItemNameMap`), `mdl/visitor/visitor_jsonstructure.go`, `mdl/types/json_utils.go` (`snippetBuilder.itemName`, `SnippetKeys`), `mdl/executor/cmd_jsonstructures.go` (`collectCustomItemNames`, DESCRIBE), `mdl/executor/validate_json_structure_names.go` (`MDL-JSON01`/`MDL-JSON02`) | **Do NOT infer Studio Pro's generation rule from stored documents — they are hand-edited.** Measured across 621 array elements in nine apps: 61% are the generator's `JsonObject`/`Wrapper`[+counter], 35% are a word someone chose, 4% are `Item`. The fingerprint that settles it is `JSON_AutoConfigResponse`, whose eight arrays read *in document order* `Scope, Wrapper_2..Wrapper_6, Claim, CodeChallengeMethods` — a counter with **gaps where a human renamed**; and `JSON_SensorData`'s `Array → SensorData`, a name with no relation to the array's. A version story fitted to a 9-structure sample (\"10.24 singularises, 11.4 does not\") evaporated at corpus scale. So the fix is **expressiveness, not default-matching**: no default can match a corpus that is a third hand-written, and changing the default would rewrite every stored structure's item names plus every mapping bound to one (ExposedName is a resolution key, #882). Design notes: `item of 'key' as 'Name'` rather than folding it into the existing entry, so naming an item does not require restating the array's name and adding one is a one-line diff; the same clause names a primitive array's **Wrapper**, because that wrapper IS the item; `item of 'Root'` for a root array, which has no key. DESCRIBE needs its own collector — an item's path segment is the marker `(Object)`/`(Wrapper)`, so the existing one skips it, and without it a named item was written on CREATE and silently renamed back by describe → exec. ako/mxcli#272 |", "refs": ["#882", "ako/mxcli#272"]} {"area": "mdl/grammar", "date": "2026-08-31", "symptom": "`DESCRIBE MICROFLOW` emits `reduce($list, expr)` (or `all(...)` / `any(...)`) and mxcli's own checker then rejects its own output: \"set 'X' calls 'reduce()', which is not a Mendix expression function [MDL044]\". Note the word **set** — the parser did not reject the call, it read the line as a Change Variable whose value happened to be a function call, and MDL044 was right about the rest", "cause": "DESCRIBE rendered an aggregate as `strings.ToLower(storedEnumValue)`, assuming every value of Mendix's `AggregateFunction` was also an MDL keyword. Mendix has eight, the grammar had five. Underneath sat a quieter defect: Mendix stores a Reduce's seed and result type in `ReduceInitialValueExpression` / `ReduceReturnDataType` and the semantic model had no field for either, so a grammar-only fix would have round-tripped the syntax while deleting the fold", "file": "`mdl/grammar/MDLLexer.g4` (REDUCE/ANY/INITIAL + the `keyword` rule so they stay usable as identifiers), `mdl/grammar/domains/MDLMicroflow.g4` (`listAggregateOperation` + `reduceFoldOptions`), `mdl/ast/ast_microflow.go`, `mdl/visitor/visitor_microflow_actions.go`, `mdl/executor/cmd_microflows_builder_actions.go`, `mdl/executor/cmd_microflows_format_action.go` (`mdlAggregateKeyword`), plus all four read/write paths: `sdk/mpr/parser_microflow.go`, `sdk/mpr/writer_microflow_actions.go`, `mdl/backend/modelsdk/microflow_read_actions.go`, `mdl/backend/modelsdk/microflow_write.go`", "insight": "**A renderer that stringifies an enum outgrows its grammar silently** — the sibling `formatListOperation` switches on concrete types and cannot, which is the shape to prefer. The guard is a describe→parse loop over `microflows.AllAggregateFunctions` (`TestDescribedAggregateParsesBack`), so a ninth Mendix function fails a test rather than a user's script. **Get a reference document before believing the vendor docs**: Mendix's reference guide says a return type is \"not applicable\" to All/Any, but Studio Pro writes `ReduceReturnDataType` as Boolean on both, and `Attribute` as `\"\"` when unused — all three activities now re-serialize byte-identically to Studio Pro's. `mx check` is no help here (0 errors before and after); the controls are the origin/main parse (`reduce`/`all`/`any` → Change Variable, with `sum` → aggregate as the positive control) and reverting the write path (`TestReduceFoldReachesStorage` then reports the two keys missing). #1004", "refs": ["#1004"], "rules": ["MDL044"]} +{"area": "mdl/grammar", "date": "2026-09-04", "symptom": "`container c ()` / `dynamictext t ()` / `pluggablewidget 'id' pw ()` are parse errors, reported at the `)` as though the widget were wrong, while bare `container c` and `container c (x: 'y')` both parse.", "cause": "widgetPropertiesV3 was `LPAREN widgetPropertyV3 (COMMA widgetPropertyV3)* RPAREN` — at least one property required.", "file": "mdl/grammar/domains/MDLPage.g4", "insight": "An empty property list is what an LLM writes for a widget that needs no properties, and the error points at the paren rather than the cause. One-character fix (wrap the list in `( … )?`). Found while measuring something else — the first run of a keyword-parse survey used `()` throughout and mis-scored every keyword as rejected, including ones that worked. If a whole measurement comes back uniformly negative, suspect the harness before the subject.", "refs": ["mendixlabs/mxcli#1036"]} +{"area": "mdl/grammar", "date": "2026-09-05", "symptom": "After adding a generic (IDENTIFIER | keyword) alternative to widgetTypeV3, `slot body` parsed as a widget of type `slot` and `placeholder Main { ... }` as a widget named Main. Both still parsed, `mxcli check` still exited 0, and a diff of check output across all 515 mdl-examples scripts showed ZERO difference.", "cause": "pageBodyV3 listed widgetV3 FIRST, before useFragmentRef / placeholderBlockV3 / slotMarkerV3. SLOT, PLACEHOLDER and USE are all inside the `keyword` rule (655 tokens), so the generic widget alternative matched them before the specific alternative could.", "file": "mdl/grammar/domains/MDLPage.g4", "insight": "Put the specific alternatives BEFORE widgetV3 in pageBodyV3, the same ordering fix widgetV3 already applies internally for `template for`. The transferable lesson is about the MEASUREMENT, not the grammar: a corpus diff of `mxcli check` output compares DIAGNOSTICS, so it is blind to a construct that parses into the wrong AST shape. 515 scripts said nothing; two visitor unit tests caught it immediately. When a grammar change could reinterpret existing syntax rather than reject it, assert on the AST, not on the diagnostics. CONTROL: reorder pageBodyV3 back and TestSpecificPageBodyFormsWinOverTheGenericWidget / TestSlotMarkerWinsOverTheGenericWidget fail.", "issue": "mendixlabs/mxcli#1036"} diff --git a/.claude/skills/fix-issue/findings/mdl-other.jsonl b/.claude/skills/fix-issue/findings/mdl-other.jsonl index afdf7116b0..ad9d243e5f 100644 --- a/.claude/skills/fix-issue/findings/mdl-other.jsonl +++ b/.claude/skills/fix-issue/findings/mdl-other.jsonl @@ -52,3 +52,4 @@ {"area": "mdl/translations", "date": "2026-08-30", "raw": "| `create or modify translations in for ` reports success (\"Set 212 nl_NL translation(s) across 20 document(s)\") and the app's pages switch language while the **menu does not** | `mdl/translations/outofscope.go` (new), `mdl/executor/cmd_translations.go`, `cmd/mxcli/syntax/features_misc.go` | The **navigation is a project-level document**, not a module one, so `in ` never reaches it. Measured on the reporting project: 151 strings scoped against 546 unscoped, and re-running the same file unscoped landed 65 more across 22 further documents. Nothing warned — the document count was the only tell, and only if you knew what number to expect. A scoped run now names **the file's own entries** it did not reach (`translations.OutOfScope`), not \"the project has other strings\", which is true of every scoped run and would warn forever — the per-module workflow is exactly what the scoping exists to support. Second, load-bearing half: those entries were previously swept into the **drift** warning, whose premise (\"no text has this as its source\") is *false* about them — they matched, out of scope. They are subtracted from it, so \"the text may have been deleted\" is only said where it is true. Controls: an unscoped run of the same file reports nothing new and lands the strings; a key matching nothing anywhere is still reported as drift. Reported as ledger #137 |", "refs": ["#137"]} {"area": "mdl/catalog", "date": "2026-09-03", "symptom": "`SHOW LANGUAGES` omits a language the project really has (ar_DZ absent from a list of 8 where the project has 9), and `search ''` returns \"No matches found\" for a string `DESCRIBE TRANSLATIONS` lists. Nothing errors and the catalog builds clean.", "cause": "CATALOG.strings was filled by hand-written per-type extractors reaching five sites (page title, enum caption, three microflow message templates), so a text anywhere else — every widget caption, tooltip, validation message, client template — was never indexed.", "file": "mdl/catalog/builder_strings.go", "insight": "A language present only on an unindexed site is INVISIBLE, not undercounted, so it vanishes from SHOW LANGUAGES entirely and from lint rule QUAL005, which discovers its language set from the same table. The fix is not a sixth case — that is how five was ever the number. Index from the type-agnostic walk DESCRIBE TRANSLATIONS already uses (translations.SitesInUnit over ListRawUnitsByType(\"\")), leaving only non-Texts$Text strings in the typed path (URLs, log nodes, REST paths, documentation, and Microflows$StringTemplate, which holds a plain Text and cannot carry a translation). Derive ObjectType from the unit $Type mechanically rather than via a table. Measured before: 69 of 3265 texts, 8 of 9 languages, 66 en_US of 1045. After: 1496 rows, 9 languages, counts identical to an independent BSON walk. Atlas design templates are ~70% of the corpus and are indexed rather than excluded, because CREATE TRANSLATIONS writes them and a SHOW LANGUAGES that excluded them would reopen the same split. CONTROL: stub the walk and the run reports `strings: 3` with SHOW LANGUAGES reporting nothing at all.", "refs": ["#250"]} {"area": "mdl/linter", "date": "2026-09-03", "symptom": "Lint rule QUAL005 reports no missing translation for an enumeration where only one value is translated (11 real gaps unreported), and likewise for a page's sibling action buttons.", "cause": "The rule grouped by (QualifiedName, StringContext) while ElementId sat unused in the strings table, so every sibling element of one type collapsed into one group and a single translated value made the set look complete.", "file": "mdl/linter/rules/missing_translations.go", "insight": "Add ElementId to the SELECT, the ORDER BY and the elementKey struct. No test caught it because the harness synthesized ElementId from QualifiedName+StringContext, giving every sibling the same value and reproducing the defect inside the fixture — a fixture that encodes the bug cannot detect it. CONTROL: with every sibling translated the run must stay at 0 violations, or the new violation is an artifact of splitting the group rather than the missing translation.", "refs": ["#250"]} +{"area": "mdl/catalog", "date": "2026-09-05", "symptom": "A widget inside a DataGrid2 column, gallery item or chart series was absent from CATALOG.WIDGETS, so a page holding 19 chart sparklines in datagrid columns never appeared under \"which pages use VegaChart?\" while the grid around them did.", "cause": "extractWidgetsRecursive walked a pluggable widget's Object.Properties[].Value.Widgets (a child slot) but not Value.Objects[] (an object list), whose items are themselves property bags holding widgets.", "file": "mdl/catalog/builder_pages.go", "insight": "Wider than the widget edge: CATALOG.REFS is a projection of this table, so an entity or microflow used ONLY inside a column template reported zero references — anything using reference counts to decide 'unused, safe to delete' would delete a document in active use (#940's failure mode, fixed for List View templates and left open for object lists). Recurse (an item is a property bag) rather than special-casing a depth. TWO measurement traps: the unfiltered ref count GREW when the widget edge landed, and growth looks like progress — only a query filtered to the widget actually asked about shows the gap; and .mxcli/catalog.db must be DELETED between runs, since a stale cache made the fixed binary look unfixed and produced a wrong conclusion before the rebuild was forced.", "issue": "mendixlabs/mxcli#1036"} diff --git a/.claude/skills/mendix/create-page/reference/widgets.md b/.claude/skills/mendix/create-page/reference/widgets.md index 33f48bd736..4a462da8c3 100644 --- a/.claude/skills/mendix/create-page/reference/widgets.md +++ b/.claude/skills/mendix/create-page/reference/widgets.md @@ -865,9 +865,9 @@ create page Sales.Dashboard (Title: 'Revenue', Layout: Atlas_Core.Atlas_Default) series sRevenue ( dataSet: static, DataSource: database from Sales.ByRegion, -- or: staticDataSource: database from Sales.ByRegion - StaticXAttribute: Region, - StaticYAttribute: Total, - StaticName: 'Revenue' + staticXAttribute: Region, + staticYAttribute: Total, + staticName: 'Revenue' ) } } diff --git a/.claude/skills/mendix/custom-widgets/SKILL.md b/.claude/skills/mendix/custom-widgets/SKILL.md index 0c4bc76f4d..9b319361ad 100644 --- a/.claude/skills/mendix/custom-widgets/SKILL.md +++ b/.claude/skills/mendix/custom-widgets/SKILL.md @@ -1,10 +1,65 @@ --- name: custom-widgets -description: "MDL syntax for pluggable widgets in CREATE PAGE / ALTER PAGE — GALLERY, COMBOBOX, DataGrid2 and third-party widgets: datasource and column forms, child slots (TEMPLATE/FILTER), adding a widget via .def.json, and the engine internals. Use when placing a pluggable widget on a page, or when `mxcli widget describe` output needs interpreting. For the widgets THIS project actually has, read the generated `widgets` skill." +description: "MDL syntax for pluggable widgets in CREATE PAGE / ALTER PAGE — any installed widget is named by its own name (`htmlelement frame (…) { … }`), with object lists and child slots read from its definition. Covers GALLERY, COMBOBOX, DataGrid2, charts and third-party widgets: datasource and column forms, child slots (TEMPLATE/FILTER), the `pluggablewidget ''` fallback, and adding a widget via .def.json. Use when placing a pluggable widget on a page, or when `mxcli widget describe` output needs interpreting. For the widgets THIS project has, read the generated `widgets` skill." --- # Custom & Pluggable Widgets in MDL +## Any installed widget is named by its own name + +If a widget is installed in `widgets/`, MDL names it directly — no keyword list, +no widget id: + +```sql +htmlelement frame (tagName: 'div', tagContentMode: 'container') { + attribute a1 (attributeName: 'data-testid', attributeValueType: 'expression') + tagcontentcontainer body { + dynamictext caption (Content: 'Inside the element') + } +} +``` + +Three things there are read from the widget's definition, not from anything +hardcoded: the **keyword** (`htmlelement`, the last segment of the widget id), +the **properties** (the widget's own spelling — `tagName`, not `TagName`), and +the **body containers** — `attribute` is an object list (one entry per +repetition), `tagcontentcontainer` a child slot (holds widgets). + +**Ask the widget rather than guessing.** `describe widget ` lists every +property with its type, default and enumeration members; every body container +and whether MDL can express it; and a complete example that parses AND checks as +written: + +```bash +mxcli widget describe htmlelement -p app.mpr +``` + +Do this first when placing an unfamiliar widget. It is faster than reading this +file and it cannot go stale, because it reads the `.mpk` the project actually +has. + +### The id form is the fallback + +```sql +pluggablewidget 'com.mendix.widget.web.htmlelement.HTMLElement' frame (tagName: 'div') +``` + +Use it only when two installed packages ship the same MDL name, or when you have +the id and not the name. Everything below that still shows the id form works +unchanged — the short form is simply the better default. + +### When the name is not found + +A name resolving to no installed definition is an **error** (MDL-WIDGET25, with +near-miss suggestions), and a container the parent does not declare is +MDL-WIDGET26. Both need `-p`: without a project, mxcli knows only its embedded +widgets, so it stays quiet rather than reporting every real widget as unknown. +If a widget you have installed is not found, extract its definition: + +```bash +mxcli widget init -p app.mpr +``` + ## Built-in Pluggable Widgets ### GALLERY @@ -61,9 +116,14 @@ combobox cmbCustomer ( ## Charts (Mendix Charts.mpk) -Charts are pluggable widgets authored by their **package id**. Install `Charts.mpk` -into the project's `widgets/` folder first (any Charts-based app has it); `exec` -auto-generates the `.def.json`. +Charts are pluggable widgets. Install `Charts.mpk` into the project's `widgets/` +folder first (any Charts-based app has it); `exec` auto-generates the +`.def.json`. + +Each is authorable by its **own name** — `barchart`, `linechart`, `piechart`, +`heatmap` — and the examples below use the package id form, which also still +works. The id column is kept because it is what `describe widget` prints and +what identifies the widget unambiguously. **Chart type → widget id → data container:** @@ -79,12 +139,12 @@ auto-generates the `.def.json`. ``` pluggablewidget 'com.mendix.widget.web.barchart.BarChart' chart1 { series s1 ( - DataSet: 'static', + dataSet: 'static', DataSource: database from MyModule.SalesByRegion, -- an OQL VIEW (aggregated) - StaticXAttribute: Region, -- resolves against the series' own datasource - StaticYAttribute: Total, - StaticName: 'Revenue', - Interpolation: 'linear' -- line/area only: linear | smooth + staticXAttribute: Region, -- resolves against the series' own datasource + staticYAttribute: Total, + staticName: 'Revenue', + interpolation: 'linear' -- line/area only: linear | spline ) } ``` @@ -101,15 +161,15 @@ from`, so a microflow-backed series described back as a missing entity.) pluggablewidget 'com.mendix.widget.web.piechart.PieChart' pie1 ( DataSource: database from MyModule.SalesByRegion, ValueAttribute: Total, - SeriesName: 'Sales by Region' -- REQUIRED (CE4899 without it) + seriesName: 'Sales by Region' -- REQUIRED (CE4899 without it) ) pluggablewidget 'com.mendix.widget.web.heatmap.HeatMap' heat1 ( DataSource: database from MyModule.SalesByRegion, ValueAttribute: Total -- REQUIRED (CE0642 without it) ) { - scalecolor scLow (ValuePercentage: 0, ColorValue: '#f7fbff') - scalecolor scHigh (ValuePercentage: 100, ColorValue: '#08306b') + scalecolor scLow (valuePercentage: 0, colorValue: '#f7fbff') + scalecolor scHigh (valuePercentage: 100, colorValue: '#08306b') } ``` diff --git a/CHANGELOG.md b/CHANGELOG.md index 0e7c97413d..8444149325 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,32 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ## [Unreleased] +- **`mxcli check` now catches two widget mistakes that only `exec` caught** — an explicit widget id that resolves to nothing (**MDL-WIDGET25**) and a real container keyword used on a parent that declares no such container (**MDL-WIDGET26**, e.g. `group` inside HTML Element). Both parsed cleanly and reported success at check time, then failed the build. + + The cause is that the **grammar was the widget-kind validator**: `widgetTypeV3` is an allow-list, so an unknown kind could not parse and the validator never needed an independent notion of one — and neither of these mistakes is a keyword the parser checks. `isUniversalObjectListKeyword` actively suppressed the second, treating a container keyword as always-an-item wherever it appeared. + + Both rules stay silent when they cannot be sure, and each guard is load-bearing. **MDL-WIDGET25 needs a project**: with none, the registry holds only mxcli's nine embedded widgets and every real project widget would be called unknown. With one, an id whose `.mpk` **is** installed is real and merely unextracted — the registry `check` uses reads `.mxcli/widgets/` and does not refresh from installed packages. **MDL-WIDGET26 needs a resolvable parent**, since a parent mxcli cannot see declares no containers as far as it knows. Measured: zero false positives across the whole `mdl-examples/doctype-tests/` corpus with a project, and none without. + +- **`DESCRIBE WIDGET` — a widget's definition, in-language** — `describe widget combobox;` or `describe widget 'com.mendix.widget.web.htmlelement.HTMLElement';` reports each property's key, type, caption, category, required flag, default and enumeration values, plus the dynamic rules the widget's editor uses to *hide* properties under some configurations (the ones that cause CE0463 when written into the pruned half). + + A widget was the only MDL extension point without one: a microflow, nanoflow, Java action and JavaScript action all describe in-language against the live project. That gap is *why* `mxcli widget init` generates markdown documentation at all — and why that documentation could drift from what the parser accepts, as reported in mendixlabs/mxcli#1036. The statement and `mxcli widget describe` are now the same function, so they cannot disagree. + + It also reports the widget's **body containers** — child slots and object lists, with each object list's item properties — and marks which are **authorable from MDL today**, since most are not (30 of 46 across a stock project). That answer is derived by parsing a probe, never from a list: the defect behind #1036 was two keyword lists with nothing comparing them, and a third list here would repeat it one layer up. It also means the marks correct themselves when the grammar catches up. + + It emits an **MDL example that parses as written**. The head form (`gallery widget1` vs `pluggablewidget '' widget1`) and every container in it are chosen by probing the real parser, and whatever the grammar cannot yet express is left out *and named* — including required properties that need a real entity or microflow from your project. This is the half of the generated `.md` that was wrong: its example failed on its own first line. Because both halves are derived rather than written down, the example widens on its own as the grammar gains ground. + + Bindings the example cannot fill in — datasource, attribute, action, expression, selection — are **named rather than invented**, and narrowed by the widget's own visibility rules: a property is required only where the editor shows it, so Combo box's eleven drop to six under its default configuration (and would drop further; `16 of 32 editor hide-rules recognized` is the current ceiling). Pruning is conservative by design — an indeterminable condition never prunes, and a rule about an object-list item never prunes the widget's own property. + + It works with **no project open**, answering from mxcli's embedded set — "what can I write here?" is asked before anything is open. With a project the answer is better: the installed `.mpk` is version-accurate and is the only place a Marketplace widget appears. + +- **A widget with no definition no longer names a remedy that cannot work** — `exec` failed with `no definition for widget … (run 'mxcli widget init -p app.mpr')`, and running that command changed nothing, because `widget init` scans `widgets/` and the package was not there. Reported as the postscript to mendixlabs/mxcli#1036, where it cost a debugging session. The message now branches on whether the package is actually installed, using the same `FindMPK` lookup the template loader makes before giving up, and otherwise says to install the widget. + + Measured while fixing it: a widget whose package is absent is one **Studio Pro cannot use either**, so mxcli ships no definitions for these. A blank Mendix 11.13 project carries 33 widgets and none of File Uploader, Events, Google Tag or Markdown viewer; installing File Uploader takes `widgets/` from 33 to 34, and a page using it then builds with no `widget init` at all — `initPluggableEngine` refreshes definitions from installed packages on its own. + +- **`()` is accepted on every widget** — `container c ()`, `dynamictext t ()` and `pluggablewidget 'id' pw ()` were parse errors reported at the closing paren, as though the widget were wrong, while bare `container c` and `container c (x: 'y')` both parsed. `widgetPropertiesV3` required at least one property; an empty list is now allowed, removing an arbitrary difference between two spellings of the same thing. + +- **Generated widget docs emit child slots with their required name** — `mxcli widget init` wrote `tagcontentcontainer { … }`, which even a working slot rejects, so the three child slots that *did* parse were documented in a form that could not. Names are emitted and numbered (`slot1`, `slot2`), since two identically named widgets on one page would collide. + - **A failed build now says which test caused it** (ako/mxcli-sudoku FINDINGS #46 follow-up) — an `@expect` that is syntactically valid but only rejected by MxBuild took down an entire `mxcli test --local` run: no test results at all, valid tests in the same file never executed, and the cause arrived as ~200 lines of mxbuild JSON with the real error among dozens of unrelated Atlas warnings. `BuildResult` parsed only the status and message and left the rest of the response unread, though mxbuild returns every problem with a severity, an error code and a location. Measured on 11.13, a failing build returns **18 problems of which one is the error**, so printing the body meant 11,580 bytes in which nothing marked the line that mattered. Filtering to errors renders it as `[CE0117] Error(s) in expression. — at MxTest / Microflow 'Test_test_3' / Decision '$result = 3'`. diff --git a/cmd/mxcli/cmd_widget_describe.go b/cmd/mxcli/cmd_widget_describe.go index 76912fc597..dcd9a13923 100644 --- a/cmd/mxcli/cmd_widget_describe.go +++ b/cmd/mxcli/cmd_widget_describe.go @@ -4,15 +4,9 @@ package main import ( "encoding/json" - "fmt" - "path/filepath" - "sort" "strings" "github.com/mendixlabs/mxcli/mdl/executor" - "github.com/mendixlabs/mxcli/mdl/types" - mwidgets "github.com/mendixlabs/mxcli/modelsdk/widgets" - mmpk "github.com/mendixlabs/mxcli/modelsdk/widgets/mpk" "github.com/spf13/cobra" ) @@ -24,6 +18,14 @@ properties (key, type, caption, category, required, default, enum options) and t dynamic property rules (which properties the widget's editor hides under which configuration) lifted from the widget package's editorConfig. +It also reports the widget's BODY CONTAINERS — its object lists (a repeated entry, +e.g. a chart series) and child slots (a block of widgets, e.g. a gallery template) +— with whether MDL can express each one, and ends with a complete MDL example +that parses and checks as written. Values in the example are real enumeration +members; bindings it cannot fill (a datasource, an attribute, an action) are named +under "omitted" rather than invented, because a generic example cannot know a name +from your project. + The widget can be named by its MDL keyword (e.g. COMBOBOX, DATAGRID2) or its full widget id (e.g. com.mendix.widget.web.combobox.Combobox). @@ -45,93 +47,18 @@ func init() { } // describedProperty is one property of a widget's discovered format. -type describedProperty struct { - Key string `json:"key"` - Type string `json:"type"` - Caption string `json:"caption,omitempty"` - Category string `json:"category,omitempty"` - Required bool `json:"required"` - Default string `json:"default,omitempty"` - System bool `json:"system,omitempty"` - Enum []string `json:"enum,omitempty"` - Children []describedProperty `json:"children,omitempty"` -} - -// describedRule is one dynamic (visibility) rule of a widget's discovered format. -type describedRule struct { - Property string `json:"property"` - HiddenWhen string `json:"hiddenWhen"` -} - -// widgetDescription is the full inspection result (also the JSON shape). -type widgetDescription struct { - WidgetID string `json:"widgetId"` - MDLName string `json:"mdlName,omitempty"` - Name string `json:"name,omitempty"` - Version string `json:"version,omitempty"` - Source string `json:"source"` // "project .mpk" | "embedded template" - Kind string `json:"kind,omitempty"` - Properties []describedProperty `json:"properties"` - Rules []describedRule `json:"dynamicRules"` - RuleCoverage string `json:"ruleCoverage,omitempty"` -} - +// The description itself is built by executor.DescribeWidget, which the MDL +// statement `DESCRIBE WIDGET x` also calls. One code path on purpose: a widget +// was the only MDL extension point with no in-language DESCRIBE, and the +// generated documentation that filled the gap could drift from what the parser +// accepts (mendixlabs/mxcli#1036). Two implementations would reopen that. func runWidgetDescribe(cmd *cobra.Command, args []string) error { - arg := args[0] projectPath, _ := cmd.Flags().GetString("project") format, _ := cmd.Flags().GetString("format") - registry, err := executor.NewWidgetRegistry() + desc, err := executor.DescribeWidget(args[0], projectPath) if err != nil { - return fmt.Errorf("failed to create widget registry: %w", err) - } - if projectPath != "" { - _ = registry.LoadUserDefinitions(projectPath) - } - - // Resolve the target widget id + optional built-in definition. - widgetID, def := resolveWidgetTarget(registry, arg) - if widgetID == "" { - return widgetNotFoundError(registry, arg) - } - - desc := widgetDescription{WidgetID: widgetID} - if def != nil { - desc.MDLName = def.MDLName - desc.Kind = def.WidgetKind - } - if desc.Kind == "" { - desc.Kind = "pluggable" - } - - // Properties + version: prefer the project's installed .mpk (version-accurate, - // includes marketplace widgets); else fall back to mxcli's embedded template. - if projectPath != "" { - if dir := projectDirOf(projectPath); dir != "" { - if mpkPath, ferr := mmpk.FindMPK(dir, widgetID); ferr == nil && mpkPath != "" { - if wd, perr := mmpk.ParseMPKForWidget(mpkPath, widgetID); perr == nil && wd != nil { - desc.Name = wd.Name - desc.Version = wd.Version - desc.Source = "project .mpk" - desc.Properties = propsFromMPK(wd) - desc.Rules, desc.RuleCoverage = rulesFromProject(mpkPath, widgetID) - } - } - } - } - if desc.Source == "" { - // Embedded template fallback. - tmpl, terr := mwidgets.GetTemplate(widgetID) - if terr != nil || tmpl == nil { - return fmt.Errorf("no installed .mpk and no embedded template for %q — try -p to inspect a project widget", arg) - } - desc.Name = tmpl.Name - desc.Version = tmpl.Version - desc.Source = "embedded template" - desc.Properties = propsFromTemplate(tmpl.Type) - if def != nil { - desc.Rules = rulesFromDef(def.PropertyVisibility) - } + return err } if strings.EqualFold(format, "json") { @@ -139,277 +66,6 @@ func runWidgetDescribe(cmd *cobra.Command, args []string) error { enc.SetIndent("", " ") return enc.Encode(desc) } - printWidgetDescription(cmd, desc) + executor.PrintWidgetDescription(cmd.OutOrStdout(), *desc) return nil } - -// resolveWidgetTarget maps a CLI argument (MDL keyword or widget id) to a widget id -// and, when known, the built-in WidgetDefinition. A dotted argument is treated as a -// widget id directly. -func resolveWidgetTarget(registry *executor.WidgetRegistry, arg string) (string, *executor.WidgetDefinition) { - if strings.Contains(arg, ".") { - if def, ok := registry.GetByWidgetID(arg); ok { - return arg, def - } - return arg, nil // unknown to the registry, but a valid id to look up in the project - } - upper := strings.ToUpper(arg) - if def, ok := registry.Get(upper); ok { - return def.WidgetID, def - } - // Well-known widgets that are special-cased in the executor (no .def.json in the - // registry) but that users still name by keyword. - if id, ok := builtinWidgetAliases[upper]; ok { - def, _ := registry.GetByWidgetID(id) - return id, def - } - return "", nil -} - -// builtinWidgetAliases maps MDL keywords for executor-special-cased widgets (which -// have no .def.json registry entry) to their widget ids, so `widget describe` can -// resolve them by the same friendly names users write in MDL. -var builtinWidgetAliases = map[string]string{ - "DATAGRID": "com.mendix.widget.web.datagrid.Datagrid", - "DATAGRID2": "com.mendix.widget.web.datagrid.Datagrid", -} - -// widgetNotFoundError builds a helpful error listing the known MDL names. -func widgetNotFoundError(registry *executor.WidgetRegistry, arg string) error { - var names []string - for _, d := range registry.All() { - if d.MDLName != "" { - names = append(names, d.MDLName) - } - } - for alias := range builtinWidgetAliases { - names = append(names, strings.ToLower(alias)) - } - sort.Strings(names) - return fmt.Errorf("unknown widget %q — use an MDL keyword (%s) or a full widget id (com.mendix.widget…). Run `mxcli widget list` to see all", - arg, strings.Join(names, ", ")) -} - -// projectDirOf returns the directory containing widgets/ for a project path -// (accepts either the .mpr file or its directory). -func projectDirOf(projectPath string) string { - if strings.EqualFold(filepath.Ext(projectPath), ".mpr") { - return filepath.Dir(projectPath) - } - return projectPath -} - -// propsFromMPK builds described properties from a parsed .mpk definition, in the -// widget's declared order (regular + system interleaved). -func propsFromMPK(wd *mmpk.WidgetDefinition) []describedProperty { - order := wd.AllTopLevel - if len(order) == 0 { - order = wd.Properties - } - out := make([]describedProperty, 0, len(order)) - for _, p := range order { - out = append(out, describedPropFromMPK(p)) - } - return out -} - -func describedPropFromMPK(p mmpk.PropertyDef) describedProperty { - dp := describedProperty{ - Key: p.Key, - Type: p.Type, - Caption: p.Caption, - Category: p.Category, - Required: p.Required, - Default: p.DefaultValue, - System: p.IsSystem, - } - if dp.System && dp.Type == "" { - dp.Type = "system" - } - for _, ev := range p.EnumValues { - dp.Enum = append(dp.Enum, ev.Key) - } - for _, c := range p.Children { - dp.Children = append(dp.Children, describedPropFromMPK(c)) - } - return dp -} - -// propsFromTemplate walks an embedded template's Type map (ObjectType.PropertyTypes) -// to build described properties. Used when no project .mpk is available. -func propsFromTemplate(typ map[string]any) []describedProperty { - objType, _ := typ["ObjectType"].(map[string]any) - pts, _ := objType["PropertyTypes"].([]any) - var out []describedProperty - for _, pt := range pts { - m, ok := pt.(map[string]any) - if !ok { - continue // leading array marker - } - out = append(out, describedPropFromTemplate(m)) - } - return out -} - -func describedPropFromTemplate(m map[string]any) describedProperty { - dp := describedProperty{ - Key: asString(m["PropertyKey"]), - Caption: asString(m["Caption"]), - Category: asString(m["Category"]), - } - vt, _ := m["ValueType"].(map[string]any) - if vt != nil { - dp.Type = asString(vt["Type"]) - dp.Default = asString(vt["DefaultValue"]) - if r, ok := vt["Required"].(bool); ok { - dp.Required = r - } - if evs, ok := vt["EnumerationValues"].([]any); ok { - for _, ev := range evs { - if em, ok := ev.(map[string]any); ok { - if k := asString(em["_Key"]); k != "" { - dp.Enum = append(dp.Enum, k) - } - } - } - } - if nested, ok := vt["ObjectType"].(map[string]any); ok { - if npts, ok := nested["PropertyTypes"].([]any); ok { - for _, npt := range npts { - if nm, ok := npt.(map[string]any); ok { - dp.Children = append(dp.Children, describedPropFromTemplate(nm)) - } - } - } - } - } - dp.System = isSystemPropKey(dp.Key) - return dp -} - -func isSystemPropKey(key string) bool { - switch key { - case "Label", "Visibility", "Editability", "Name", "TabIndex": - return true - } - return false -} - -// rulesFromProject extracts dynamic rules from the project's installed .mpk editor -// config, returning the rules and a coverage note (recognized / total hide-calls). -func rulesFromProject(mpkPath, widgetID string) ([]describedRule, string) { - rules, recognized, total := executor.ExtractWidgetVisibilityStats(mpkPath, widgetID) - coverage := "" - if total > 0 { - coverage = fmt.Sprintf("%d of %d editor hide-rules recognized", recognized, total) - } - return rulesToDescribed(rules), coverage -} - -func rulesFromDef(rules []types.WidgetVisibilityRule) []describedRule { - return rulesToDescribed(rules) -} - -func rulesToDescribed(rules []types.WidgetVisibilityRule) []describedRule { - out := make([]describedRule, 0, len(rules)) - for _, r := range rules { - if r.HiddenWhen == nil { - continue - } - out = append(out, describedRule{Property: r.PropertyKey, HiddenWhen: conditionText(r.HiddenWhen)}) - } - sort.Slice(out, func(i, j int) bool { return out[i].Property < out[j].Property }) - return out -} - -// conditionText renders a visibility condition as readable English. -func conditionText(c *types.WidgetVisibilityCondition) string { - switch c.Operator { - case "eq": - return fmt.Sprintf("%s = %q", c.PropertyKey, c.Value) - case "ne": - return fmt.Sprintf("%s ≠ %q", c.PropertyKey, c.Value) - case "truthy": - return fmt.Sprintf("%s is set", c.PropertyKey) - case "falsy": - return fmt.Sprintf("%s is not set", c.PropertyKey) - default: - return fmt.Sprintf("%s %s %q", c.PropertyKey, c.Operator, c.Value) - } -} - -func asString(v any) string { - s, _ := v.(string) - return s -} - -func printWidgetDescription(cmd *cobra.Command, d widgetDescription) { - out := cmd.OutOrStdout() - title := d.Name - if title == "" { - title = d.WidgetID - } - fmt.Fprintf(out, "Widget: %s", title) - if d.MDLName != "" { - fmt.Fprintf(out, " (%s)", d.MDLName) - } - fmt.Fprintln(out) - fmt.Fprintf(out, " ID: %s\n", d.WidgetID) - if d.Version != "" { - fmt.Fprintf(out, " Version: %s\n", d.Version) - } - fmt.Fprintf(out, " Kind: %s\n", d.Kind) - fmt.Fprintf(out, " Source: %s\n", d.Source) - - fmt.Fprintf(out, "\nProperties (%d):\n", countProps(d.Properties)) - printProps(out, d.Properties, 0) - - fmt.Fprintf(out, "\nDynamic property rules (%d):\n", len(d.Rules)) - if len(d.Rules) == 0 { - fmt.Fprintln(out, " (none discovered)") - } - for _, r := range d.Rules { - fmt.Fprintf(out, " %-40s hidden when %s\n", r.Property, r.HiddenWhen) - } - if d.RuleCoverage != "" { - fmt.Fprintf(out, " — %s\n", d.RuleCoverage) - } -} - -func countProps(props []describedProperty) int { - n := 0 - for _, p := range props { - n++ - n += countProps(p.Children) - } - return n -} - -func printProps(out interface{ Write([]byte) (int, error) }, props []describedProperty, depth int) { - indent := strings.Repeat(" ", depth+1) - for _, p := range props { - req := "" - if p.Required { - req = " required" - } - sys := "" - if p.System { - sys = " [system]" - } - line := fmt.Sprintf("%s%-34s %-13s", indent, p.Key, p.Type) - extra := strings.TrimRight(req+sys, " ") - if p.Default != "" { - extra = strings.TrimSpace(extra + " default=" + p.Default) - } - if len(p.Enum) > 0 { - extra = strings.TrimSpace(extra + " {" + strings.Join(p.Enum, "|") + "}") - } - if p.Category != "" { - extra = strings.TrimSpace(extra + " (" + p.Category + ")") - } - fmt.Fprintf(out, "%s %s\n", strings.TrimRight(line, " "), extra) - if len(p.Children) > 0 { - printProps(out, p.Children, depth+1) - } - } -} diff --git a/cmd/mxcli/cmd_widget_describe_test.go b/cmd/mxcli/cmd_widget_describe_test.go index c7b3cfa173..7d3c1fff70 100644 --- a/cmd/mxcli/cmd_widget_describe_test.go +++ b/cmd/mxcli/cmd_widget_describe_test.go @@ -8,7 +8,6 @@ import ( "testing" "github.com/mendixlabs/mxcli/mdl/executor" - "github.com/mendixlabs/mxcli/mdl/types" ) // TestWidgetDescribe_EmbeddedCombobox runs `widget describe COMBOBOX --format json` @@ -28,7 +27,7 @@ func TestWidgetDescribe_EmbeddedCombobox(t *testing.T) { if err := runWidgetDescribe(cmd, []string{"COMBOBOX"}); err != nil { t.Fatalf("describe COMBOBOX json: %v", err) } - var d widgetDescription + var d executor.WidgetDescription if err := json.Unmarshal([]byte(out.String()), &d); err != nil { t.Fatalf("unmarshal json: %v\n%s", err, out.String()) } @@ -59,35 +58,3 @@ func TestWidgetDescribe_EmbeddedCombobox(t *testing.T) { } // TestWidgetDescribe_UnknownWidget reports a helpful error. -func TestWidgetDescribe_UnknownWidget(t *testing.T) { - reg, err := executor.NewWidgetRegistry() - if err != nil { - t.Fatalf("registry: %v", err) - } - id, _ := resolveWidgetTarget(reg, "NOPE") - if id != "" { - t.Errorf("resolveWidgetTarget(NOPE) = %q, want empty", id) - } - // DATAGRID2 resolves via the builtin alias even without a .def.json entry. - if id, _ := resolveWidgetTarget(reg, "datagrid2"); id != "com.mendix.widget.web.datagrid.Datagrid" { - t.Errorf("resolveWidgetTarget(datagrid2) = %q", id) - } -} - -// TestConditionText renders the four operators as readable English. -func TestConditionText(t *testing.T) { - cases := []struct { - op, val, want string - }{ - {"eq", "None", `itemSelection = "None"`}, - {"ne", "Multi", `itemSelection ≠ "Multi"`}, - {"truthy", "", "itemSelection is set"}, - {"falsy", "", "itemSelection is not set"}, - } - for _, c := range cases { - got := conditionText(&types.WidgetVisibilityCondition{PropertyKey: "itemSelection", Operator: c.op, Value: c.val}) - if got != c.want { - t.Errorf("op %s: got %q, want %q", c.op, got, c.want) - } - } -} diff --git a/cmd/mxcli/syntax/features_page.go b/cmd/mxcli/syntax/features_page.go index 9522aea0c3..7ab1fdbebd 100644 --- a/cmd/mxcli/syntax/features_page.go +++ b/cmd/mxcli/syntax/features_page.go @@ -29,6 +29,83 @@ func init() { SeeAlso: []string{"page", "page.widgets", "page.datasource"}, }) + Register(SyntaxFeature{ + Path: "page.widget-describe", + Summary: "DESCRIBE WIDGET — a widget's properties, enum values and editor rules", + Keywords: []string{ + "describe widget", "widget properties", "widget definition", "what properties", + "enum values", "widget rules", "hidden properties", "pluggable widget properties", + }, + Syntax: `DESCRIBE WIDGET ; +DESCRIBE WIDGET '';`, + Example: `DESCRIBE WIDGET combobox; +DESCRIBE WIDGET 'com.mendix.widget.web.htmlelement.HTMLElement'; + +-- Names the widget by its MDL keyword or its full widget id. +-- +-- Works with NO project open, answering from mxcli's embedded set. With a +-- project the answer is better: the installed .mpk is version-accurate and is +-- the only place a Marketplace widget appears. +-- +-- Reports each property's key, type, caption, category, whether it is required, +-- its default and its enumeration values, plus the dynamic rules the widget's +-- editor uses to HIDE properties under some configurations — the ones that +-- cause CE0463 if written into the pruned half. +-- +-- Also emits an MDL example that PARSES AS WRITTEN: the head form and every +-- container in it are chosen by probing the real parser, and anything the +-- grammar cannot yet express is left out and named. So the example widens on +-- its own as MDL gains ground, and cannot promise syntax that fails. +-- +-- Same output as ` + "`mxcli widget describe`" + `, because it is the same code. + +-- The other direction — which pages already use it — is a reference query, +-- and needs ` + "`refresh catalog full`" + `: +SHOW REFERENCES TO combobox; +SHOW IMPACT OF htmlelement; + +-- Name it as you write it in a page body; the casing does not matter. A +-- built-in Mendix widget (textbox, dynamictext) has no definition and so no +-- reference edge — use SHOW WIDGETS for those.`, + SeeAlso: []string{"page.widgets", "page.create"}, + }) + + Register(SyntaxFeature{ + Path: "page.widget-any", + Summary: "Any widget with a definition, written by its own MDL name", + Keywords: []string{ + "htmlelement", "html element", "fileuploader", "file uploader", "markdown", + "custom widget syntax", "marketplace widget", "pluggable widget name", + "widget not recognized", "mismatched input", "def-driven", "mdl name", + "object list", "child slot", "widget container", "attributes list", + }, + Syntax: ` [( Prop: Value, ... )] [{ }] + [( Prop: Value, ... )] [{ ... }]`, + Example: `-- Any widget with a definition is written by its own MDL name. There is no +-- list of blessed keywords: if ` + "`describe widget `" + ` knows it, you can write it. +CREATE PAGE Sales.Detail (Title: 'Detail', Layout: Atlas_Core.Atlas_Default) { + htmlelement frame (tagName: 'div') { + -- object lists and child slots the widget's own definition declares + attribute a1 (attributeName: 'title', attributeValueType: 'expression') + tagcontentcontainer body { + dynamictext t (Content: 'hello') + } + } + fileuploader up () +} + +-- The names come from the widget itself, so ask it rather than guessing: +-- mxcli widget describe htmlelement -p app.mpr +-- which lists every property, every container, and an example that parses. +-- +-- A name that resolves to no definition is MDL-WIDGET25 (widget) or +-- MDL-WIDGET26 (container), each naming the near misses — but BOTH need a +-- project, because the set of valid widget names IS the project's installed +-- packages. With no -p, ` + "`mxcli check`" + ` cannot tell a typo from a widget it +-- has simply never seen, and says nothing rather than guessing.`, + SeeAlso: []string{"page.widgets", "page.widget-describe", "page.create"}, + }) + Register(SyntaxFeature{ Path: "page.widgets", Summary: "Widget types: containers, data widgets, inputs, actions, display", diff --git a/cmd/mxcli/syntax/widget_keywords_drift_test.go b/cmd/mxcli/syntax/widget_keywords_drift_test.go index 1e3808d9de..622b902ed6 100644 --- a/cmd/mxcli/syntax/widget_keywords_drift_test.go +++ b/cmd/mxcli/syntax/widget_keywords_drift_test.go @@ -60,7 +60,20 @@ func widgetTypeAlternatives(t *testing.T) []string { line = line[:i] } for _, tok := range strings.Split(line, "|") { - if tok = strings.TrimSpace(tok); regexp.MustCompile(`^[A-Z][A-Z0-9]*$`).MatchString(tok) { + tok = strings.TrimSpace(tok) + // IDENTIFIER is the generic alternative added by slice 2 of + // PROPOSAL_def_driven_widget_bodies.md, not a keyword — it is how a + // widget is named by its own MDL name. It lexes as a token class, so + // the all-caps shape below matches it; there is no keyword + // "identifier" to document. Its lower-case sibling `keyword` (slice + // 3) is already skipped by that shape. + // + // Covered instead by TestDefDrivenWidgetNameIsDocumented below, which + // is the assertion that actually applies to it. + if tok == "IDENTIFIER" { + continue + } + if regexp.MustCompile(`^[A-Z][A-Z0-9]*$`).MatchString(tok) { out = append(out, strings.ToLower(tok)) } } @@ -164,3 +177,45 @@ func TestWidgetKeywordGuardCanFail(t *testing.T) { "whole guard is vacuous") } } + +// The enumerated keywords are no longer the whole widget vocabulary: since +// slices 2-3 any widget with a definition can be named by its MDL name, and any +// container a definition declares can be written in a body. That is a bigger +// capability than the list above, and the same reasoning applies to it — a +// capability nobody can find is one people build around. +// +// This asserts both halves are still present in the grammar (so their removal +// is noticed) and that a page.* topic tells the reader about them. +func TestDefDrivenWidgetNameIsDocumented(t *testing.T) { + path := filepath.Join("..", "..", "..", "mdl", "grammar", "domains", "MDLPage.g4") + b, err := os.ReadFile(path) + if err != nil { + t.Fatalf("read %s: %v", path, err) + } + rule := regexp.MustCompile(`(?s)\nwidgetTypeV3\s*\n\s*:(.*?)\n\s*;`).FindStringSubmatch(string(b)) + if rule == nil { + t.Fatal("widgetTypeV3 rule not found") + } + body := rule[1] + for _, alt := range []string{"IDENTIFIER", "keyword"} { + if !regexp.MustCompile(`\|\s*` + alt + `\s*\n`).MatchString(body + "\n") { + t.Errorf("widgetTypeV3 no longer offers the generic `%s` alternative — "+ + "if that was deliberate, 30 of 46 documented widget containers stop parsing again "+ + "(mendixlabs/mxcli#1036); update this guard rather than deleting it", alt) + } + } + + corpus := pageSyntaxCorpus() + if corpus == "" { + t.Fatal("no page.* topics registered — the guard would pass vacuously") + } + // The reader has to be able to learn that a widget can be named by its own + // name. Any of these phrasings satisfies that. + for _, want := range []string{"mdl name", "def-driven", "any widget with a definition"} { + if strings.Contains(corpus, want) { + return + } + } + t.Error("no page.* syntax topic explains that a widget can be written by its own MDL name " + + "(e.g. `htmlelement frame (...)`) — add it to cmd/mxcli/syntax/features_page.go") +} diff --git a/docs-site/src/appendixes/error-messages.md b/docs-site/src/appendixes/error-messages.md index 6c844a7ed8..3aa32df789 100644 --- a/docs-site/src/appendixes/error-messages.md +++ b/docs-site/src/appendixes/error-messages.md @@ -78,7 +78,7 @@ page MyModule.OrderList: widget `cb1` (combobox) has no property **Cause:** The property key written on a pluggable widget is not declared in the widget's `.def.json` (the extracted schema from its `.mpk`). Usually a typo; sometimes a property that exists in a different widget but not this one. **Solution:** -1. Compare the key against the widget's known properties — `mxcli describe widget ` lists them. +1. Compare the key against the widget's known properties — `mxcli widget describe ` lists them (or `describe widget ;` in MDL). 2. Use the suggested replacement if one is offered (Levenshtein-nearest match). 3. If the property genuinely doesn't exist on this widget version, check that `.mxcli/widgets/` has the latest schema: `mxcli refresh catalog -p app.mpr` re-extracts any `.mpk` whose mtime changed. 4. If the property was just added by a `.mpk` upgrade, make sure `mxcli init` or `widget init` was run after the upgrade. diff --git a/docs-site/src/internals/catalog-schema.md b/docs-site/src/internals/catalog-schema.md index 01e3177e26..30fa0eed95 100644 --- a/docs-site/src/internals/catalog-schema.md +++ b/docs-site/src/internals/catalog-schema.md @@ -156,19 +156,68 @@ CREATE TABLE WIDGETS ( ### REFS +The reference graph: one row per edge. Populated by `refresh catalog full`. + ```sql CREATE TABLE REFS ( - SourceName TEXT, -- Referencing document - SourceKind TEXT, -- "Microflow", "Page", etc. - TargetName TEXT, -- Referenced element - TargetKind TEXT, -- "Entity", "Microflow", etc. - RefKind TEXT -- "Call", "DataSource", "Association", etc. + Id INTEGER PRIMARY KEY AUTOINCREMENT, + SourceType TEXT NOT NULL, -- "MICROFLOW", "PAGE", "ENTITY", ... + SourceId TEXT NOT NULL, -- element $ID, or '' where the builder has no id + SourceName TEXT NOT NULL, -- referencing document, module-qualified + TargetType TEXT NOT NULL, -- "ENTITY", "MICROFLOW", "WIDGET", ... + TargetId TEXT, -- element $ID, or the widget ID for a WIDGET target + TargetName TEXT NOT NULL, -- referenced element + RefKind TEXT NOT NULL, -- see the vocabulary below + ModuleName TEXT, + ProjectId TEXT, + SnapshotId TEXT ); -CREATE INDEX idx_refs_source ON REFS(SourceName); -CREATE INDEX idx_refs_target ON REFS(TargetName); +CREATE INDEX idx_refs_source ON refs(SourceType, SourceName); +CREATE INDEX idx_refs_target ON refs(TargetType, TargetName); +CREATE INDEX idx_refs_kind ON refs(RefKind); ``` +`RefKind` values are lower-case, and the current vocabulary is whatever +`CATALOG.GRAPH_REFKIND_DISTRIBUTION` reports for your project — query that +rather than trusting a list here: + +| RefKind | Edge | +|---------|------| +| `call` | flow calls a microflow / nanoflow / rule / Java action / REST operation | +| `create` / `change` / `delete` / `retrieve` | flow acts on an entity object | +| `return` | flow returns an entity type | +| `parameter` | page or flow parameter entity type | +| `generalize` | entity extends entity | +| `associate` | association targets entity | +| `layout` | page uses a layout | +| `datasource` | page or widget reads an entity | +| `action` | widget calls a microflow / nanoflow | +| `show_page` | flow or widget action opens a page | +| `home_page` / `login_page` / `menu_item` | navigation profile references a page | +| `calculate` | calculated attribute uses a microflow | +| `schedule` | scheduled event runs a microflow | +| `validate` | attribute validation rule uses a regular expression | +| `widget` | page or snippet uses a pluggable / custom widget | + +#### WIDGET targets + +A `widget` edge is the odd one out and is worth knowing about before you join +against it: + +- `TargetName` is the widget's **MDL name** (`COMBOBOX`), not its dotted widget + ID. The ID is in `TargetId`. A dotted target would be mis-read as a module by + `GRAPH_MODULE_COUPLING` and friends, which take everything before the first + dot as the module name. +- It is therefore the only `TargetName` that is not module-qualified — a widget + definition belongs to no Mendix module. `GRAPH_GOD_NODES` excludes `WIDGET` + targets from its asset list for that reason, while still counting a page's + out-degree towards the widgets it uses. +- Only widgets with a definition get an edge. A built-in Mendix widget + (`Forms$DynamicText`) has none, so it produces no row; use `CATALOG.WIDGETS` + for those. +- One edge per page x widget, not per widget instance. + ### PERMISSIONS ```sql @@ -234,6 +283,16 @@ WHERE AttributeCount > 20 ORDER BY AttributeCount DESC; SELECT SourceName, RefKind FROM CATALOG.REFS WHERE TargetName = 'Sales.Customer'; +-- Which pages use a given pluggable widget? +SELECT SourceType, SourceName FROM CATALOG.REFS +WHERE RefKind = 'widget' AND TargetName = 'COMBOBOX'; + +-- Which installed widget packages does nothing use? +-- (MDL's SELECT has no NOT EXISTS / NOT IN — use an anti-join.) +SELECT d.MdlName, d.WidgetId FROM CATALOG.WIDGET_DEFINITIONS d +LEFT JOIN CATALOG.REFS r ON r.TargetId = d.WidgetId AND r.RefKind = 'widget' +WHERE r.Id IS NULL; + -- Full-text search SELECT name, kind, snippet(STRINGS, 2, '', '', '...', 20) FROM CATALOG.STRINGS WHERE strings MATCH 'validation error'; diff --git a/docs-site/src/language/widget-types.md b/docs-site/src/language/widget-types.md index 72ed451bf5..ed29b93255 100644 --- a/docs-site/src/language/widget-types.md +++ b/docs-site/src/language/widget-types.md @@ -1,11 +1,13 @@ # Widget Types -MDL supports a comprehensive set of widget types for building Mendix pages. Each widget is declared with a type keyword, a unique name, properties in parentheses, and optional child widgets in braces. +A widget is declared with a type keyword, a unique name, properties in parentheses, and optional child widgets in braces. ```sql WIDGET_TYPE widgetName (Property: value, ...) [{ children }] ``` +**The list below is not the boundary.** The built-in Mendix widgets are documented here because they have fixed, hand-written property sets. Every **pluggable or custom widget** installed in the project is also written by its own name, with a body derived from the widget's definition — see [Any installed widget](#any-installed-widget) below. If a widget is in `widgets/`, MDL can name it. + ## Widget Categories | Category | Widgets | @@ -445,6 +447,74 @@ NAVIGATIONLIST navMain { } ``` +## Any installed widget + +Everything above is a **built-in** widget: its keyword and properties are fixed +by Mendix and by mxcli. A **pluggable or custom widget** — DataGrid 2, Combo box, +Gallery, HTML Element, the charts, anything from the Marketplace, anything your +team built — is named the same way, by its own MDL name: + +```sql +htmlelement frame (tagName: 'div', tagContentMode: 'container') { + attribute a1 (attributeName: 'data-testid', attributeValueType: 'expression') + tagcontentcontainer body { + dynamictext caption (Content: 'Inside the element') + } +} +``` + +Three things there come from the widget's own definition rather than from any +list in mxcli: + +- **The keyword** `htmlelement` — the widget's MDL name, which is the last + segment of its widget id. +- **The properties** `tagName`, `tagContentMode` — written with the widget's own + spelling, exactly as `DESCRIBE WIDGET` reports them. +- **The body containers** `attribute` (an object list, one entry per repetition) + and `tagcontentcontainer` (a child slot, holding widgets). + +### Finding the names + +Ask the widget: + +```sql +DESCRIBE WIDGET htmlelement; +``` + +It lists every property with its type, default and enumeration members, every +body container and whether MDL can express it, and a complete example that +parses and checks as written. See +[DESCRIBE WIDGET](../reference/query/describe-widget.md). + +### The explicit form + +A widget can also be named by its full id. This is the fallback, not the norm — +use it when two installed packages ship the same MDL name, or when you have the +id in hand and not the name: + +```sql +pluggablewidget 'com.mendix.widget.web.htmlelement.HTMLElement' frame ( + tagName: 'div' +) +``` + +`DESCRIBE PAGE` emits the short form wherever it round-trips, and falls back to +the id form when the name would be ambiguous. + +### When a widget is not found + +A name that resolves to no installed definition is an error, not a silently +accepted widget — MDL-WIDGET25, with the nearest known names suggested. A +container keyword the parent widget does not declare is MDL-WIDGET26. Both need +a project open (`-p`), since without one mxcli knows only its embedded widgets. + +If a widget you have installed is not found, its definition has not been +extracted yet: + +```bash +mxcli widget init -p app.mpr # extract definitions for every widget in widgets/ +``` + ## Common Widget Properties These properties are shared across many widget types: @@ -464,3 +534,5 @@ These properties are shared across many widget types: - [Page Structure](./page-structure.md) -- layout selection and data sources - [Data Binding](./data-binding.md) -- connecting widgets to attributes - [ALTER PAGE](./alter-page.md) -- modifying widgets in existing pages +- [DESCRIBE WIDGET](../reference/query/describe-widget.md) -- inspect any installed widget's properties and body containers +- [Pluggable Widgets Across Versions](../guides/pluggable-widgets.md) -- how mxcli keeps widget definitions version-correct diff --git a/docs-site/src/reference/query/README.md b/docs-site/src/reference/query/README.md index 331dc23679..57815af5e9 100644 --- a/docs-site/src/reference/query/README.md +++ b/docs-site/src/reference/query/README.md @@ -27,6 +27,7 @@ Statements for browsing and inspecting project elements. Query statements are re | [DESCRIBE ENUMERATION](describe-enumeration.md) | Show enumeration values and documentation | | [DESCRIBE MICROFLOW](describe-microflow.md) | Show complete MDL source for a microflow or nanoflow | | [DESCRIBE PAGE](describe-page.md) | Show complete MDL source for a page or snippet | +| [DESCRIBE WIDGET](describe-widget.md) | Show a widget's properties, body containers and a working MDL example | ## Search diff --git a/docs-site/src/reference/query/describe-widget.md b/docs-site/src/reference/query/describe-widget.md new file mode 100644 index 0000000000..969d7be7c4 --- /dev/null +++ b/docs-site/src/reference/query/describe-widget.md @@ -0,0 +1,113 @@ +# DESCRIBE WIDGET + +## Synopsis + + DESCRIBE WIDGET + + DESCRIBE WIDGET '' + +## Description + +Shows the format mxcli has discovered for a pluggable or custom widget: its +properties (key, type, caption, category, required, default, enumeration +members), the **body containers** it accepts, the editor rules that hide a +property under some configurations, and a complete MDL example. + +A widget was the only MDL extension point without a `DESCRIBE`. That is why +`mxcli widget init` writes markdown documentation at all — and why the two could +drift. `DESCRIBE WIDGET` and `mxcli widget describe` are the same function, so +they cannot disagree. + +Unlike the other `DESCRIBE` statements, this one **works with no project open**: +"what can I write here?" is a question asked before anything is open. With `-p`, +the properties and rules come from the widget package actually installed in the +project (`widgets/*.mpk`) — version-accurate, and the only place a Marketplace +widget appears at all. Without it, they come from mxcli's embedded template. + +## Parameters + +*keyword* +: The widget's MDL name, as written in a page body — `combobox`, `htmlelement`, + `datagrid`. Case-insensitive. + +*widget id* +: The full widget id as a quoted string — + `'com.mendix.widget.web.htmlelement.HTMLElement'`. Use this form for a widget + whose MDL name is ambiguous, or when you have the id in hand from a `.mpk`. + +## Examples + +```sql +DESCRIBE WIDGET htmlelement; +``` + +Example output, abbreviated: + +``` +Widget: HTML Element (htmlelement) + ID: com.mendix.widget.web.htmlelement.HTMLElement + Version: 1.2.2 + Kind: pluggable + Source: project .mpk + +Properties (23): + tagName enumeration required default=div {div|span|p|ul|…} + tagContentMode enumeration required default=container {container|innerHTML} + attributes object (General::HTML attributes) + attributeName string required + attributeValueType enumeration required default=expression {expression|template} + +Body containers (4): + attribute object list -> attributes authorable + items: attributeName, attributeValueType, … + event object list -> events authorable + items: eventName, eventAction, … + tagcontentcontainer child slot -> tagContentContainer authorable + tagcontentrepeatcontainer child slot -> tagContentRepeatContainer authorable + +MDL example (parses as written): + htmlelement widget1 ( + tagName: 'div', + tagUseRepeat: false, + tagContentMode: 'container' + ) { + attribute item1 (attributeValueType: 'expression') -- one entry of `attributes` + event item2 (eventName: 'onClick') -- one entry of `events` + tagcontentcontainer slot3 { + -- widgets for `tagContentContainer` + } + } +``` + +By widget id: + +```sql +DESCRIBE WIDGET 'com.mendix.widget.web.htmlelement.HTMLElement'; +``` + +## Notes + +**Body containers report whether MDL can express them.** `authorable` is derived +by parsing a probe against the live grammar, never read from a list — so the mark +is correct by construction rather than by maintenance. A container reported as +not authorable is one to set in Studio Pro. + +**The MDL example parses and checks as written.** The whole block is parsed +before it is emitted, and its property values are real members rather than +placeholders. Bindings it cannot fill — a datasource, an attribute, an action — +are **named rather than invented**, under `-- omitted:`, because a generic +example cannot know a name from your project. + +**The example is narrowed by the widget's own editor rules.** A property the +widget hides under the configuration the example picked is left out, so what you +see is what that configuration actually supports. The footer reports how many of +the widget's hide-rules were recognised; an unrecognised rule never prunes. + +**`LIST WIDGETS` does not exist**, deliberately. `SHOW WIDGETS` already means +widget *instances placed on pages*, and the definitions are +`SELECT * FROM CATALOG.WIDGET_DEFINITIONS`. + +## See Also + +[SHOW WIDGETS](show-widgets.md), [DESCRIBE PAGE](describe-page.md), +[Pluggable Widgets Across Versions](../../guides/pluggable-widgets.md) diff --git a/docs/01-project/MDL_QUICK_REFERENCE.md b/docs/01-project/MDL_QUICK_REFERENCE.md index 9ffb5dff36..16d5843dd9 100644 --- a/docs/01-project/MDL_QUICK_REFERENCE.md +++ b/docs/01-project/MDL_QUICK_REFERENCE.md @@ -1238,6 +1238,7 @@ MDL uses explicit property declarations for pages: | Pop-up dimensions | `PopupWidth: n, PopupHeight: n, PopupResizable: bool` | `(Layout: Atlas_Core.PopupLayout, PopupWidth: 800, PopupHeight: 480, PopupResizable: true)` — case-sensitive; default 600×600 | | Page CSS class / style | `Class: 'css-class', Style: 'css: rule'` | `(Title: 'Home', Class: 'container-fluid bg-light', Style: 'min-height: 100vh')` — the page's Appearance | | Page variables | `variables: { $name: type = 'expr' }` | `variables: { $show: boolean = 'true' }` | +| Inspect a widget | `describe widget ;` | `describe widget combobox;` — properties, enum values, defaults and the editor rules that HIDE properties under some configurations. Works with no project open; with one, reads the installed `.mpk` (version-accurate, and the only place a Marketplace widget appears). Same output as `mxcli widget describe` | | Widget name | Required after type | `textbox txtName (...)` | | Attribute binding | `attribute: AttrName` | `textbox txt (label: 'Name', attribute: Name)` | | Variable binding | `datasource: $Var` | `dataview dv (datasource: $Product) { ... }` | @@ -1617,6 +1618,15 @@ Cross-reference commands require `refresh catalog full` to populate reference da `show callers` covers invocation only. A document that merely *uses a type* — an entity as a page datasource, a microflow parameter, an entity's generalization — is not a caller of it; `show references to` lists those. +A **pluggable or custom widget** is a reference target too, so "which pages use this widget?" is one query — the same question about a Java action always was: + +```mdl +show references to combobox; -- pages and snippets that place a Combo box +show impact of htmlelement; -- the same, grouped by document type +``` + +Name the widget the way you write it in a page body. The target is stored as the widget's MDL name and matched case-insensitively when the exact spelling finds nothing, so `combobox`, `ComboBox` and `COMBOBOX` all resolve; the resolved spelling is printed. A built-in Mendix widget (`textbox`, `dynamictext`) has no definition and therefore no edge — use `show widgets` for those. + ## Connection & Session | Statement | Syntax | Notes | diff --git a/docs/11-proposals/PROPOSAL_def_driven_widget_bodies.md b/docs/11-proposals/PROPOSAL_def_driven_widget_bodies.md new file mode 100644 index 0000000000..02fa4d79ea --- /dev/null +++ b/docs/11-proposals/PROPOSAL_def_driven_widget_bodies.md @@ -0,0 +1,604 @@ +--- +title: Widgets as first-class MDL, not a second dialect +status: draft +date: 2026-09-04 +related: + - PROPOSAL_mcp_pluggable_widget_authoring.md + - PROPOSAL_multi_version_pluggable_widgets.md + - PROPOSAL_widget_property_visibility.md +--- + +# Proposal: Widgets as first-class MDL, not a second dialect + +**Status:** Draft +**Date:** 2026-09-04 + +Using a widget should feel like calling a Java action. It does not. A widget is +named differently, described differently, and is invisible to the reference +graph — and where every other extension point resolves against the project, a +widget resolves against a hardcoded list in the grammar. This proposal closes +the four gaps, in order of how much each one costs. + +## Problem Statement + +### What a user hit + +[mendixlabs/mxcli#1036](https://github.com/mendixlabs/mxcli/issues/1036). A team +needed a sandboxed iframe: the HTML Element widget's `attributes` object list +would carry `sandbox` and `srcdoc`. MDL could not express it, so they fell back +to `tagContentHTML`, **which executes same-origin — precisely the risk the +sandbox exists to prevent.** + +The generated documentation is what cost them the time. `widget init` writes +`.claude/skills/widgets/htmlelement.md`, whose lead example is: + +```sql +PLUGGABLEWIDGET 'com.mendix.widget.web.htmlelement.HTMLElement' widget1 { + tagcontentcontainer { ... } + attribute item1 -- one entry of `attributes` + event item1 -- one entry of `events` +} +``` + +Fed back into `mxcli check` verbatim, it fails on the first line of its own body: + +``` +line 4:4 mismatched input 'tagcontentcontainer' expecting '}' +``` + +The file is written *for LLM agents to follow* and carries the widget's real +property table, so it reads as authoritative. Multiple sessions concluded the +feature existed. + +### The real shape of the problem + +The reported bug is one symptom of a widget being a second dialect inside MDL. +Every other extension point — a submicroflow, a Java action, a JavaScript +action — is referenced by qualified name, described in-language, and findable in +the reference graph: + +```mdl +$R = CALL MICROFLOW Module.Name (Param = value) +$R = CALL NANOFLOW Module.Name (Param = value) +$R = CALL JAVA ACTION Module.Name (Param = value) +$R = CALL JAVASCRIPT ACTION Module.Name (Param = value) +``` + +A widget matches none of that: + +| | reference | `DESCRIBE` | "who uses it?" | +|---|---|---|---| +| microflow / nanoflow | `Module.Name` | MDL statement | `call` edge | +| java action | `Module.Name` | MDL statement, round-trips | `call` edge | +| javascript action | `Module.Name` | MDL statement | `call` edge | +| **widget** | keyword *if blessed*, else a string ID | **CLI command only** | **no edge at all** | + +Four gaps follow, and each is measured below. + +### Gap 1 — Two reference forms, gated on a hardcoded list + +``` +htmlelement.def.json: mdlName: HTMLELEMENT widgetId: com.mendix.widget.web.htmlelement.HTMLElement + +htmlelement h (tagName: 'div') -- REJECTED +combobox c (Attribute: Name) -- PARSES +``` + +A widget gets a keyword only if it appears in the grammar's `widgetTypeV3` list. +Everything else must be written by string ID. + +**The resolution machinery already exists and is already wired.** +`cmd_pages_builder_v3.go:425` tries the MDL name *first*: + +```go +// Try by MDL name first +if def, ok := pb.widgetRegistry.Get(strings.ToUpper(w.Type)); ok { + return pb.buildPluggable(def, w) +} +``` + +Every `.def.json` carries an `mdlName`; `WidgetRegistry.Get(mdlName)` exists. +The only thing missing is a parser that will produce `w.Type == "htmlelement"`. +Today `MDLName` is read **only to build error messages**. + +### Gap 2 — Object lists and child slots, the same defect one level down + +Inside a widget body, containers are gated on nine hardcoded keywords +(`MDLPage.g4:402–410`), while the doc generator derives one for **every** object +list and child slot mechanically (`deriveObjectListKeyword` singularises any +property key; child slots use `strings.ToUpper(child.Key)`). Two lists, nothing +comparing them. + +Measured against the fixture project (33 widget defs, 46 documented constructs): + +| | parses | rejected | +|---|---|---| +| Object lists | 13/16 | `attribute`, `event`, `attr` | +| Child slots (named, under `pluggablewidget`) | 3/30 | 27 | +| **Total** | **16/46** | **30 (65%)** | + +20 of 33 widgets document at least one keyword that cannot parse. **Control** — +identical widget, identical body shape, only list membership differs: + +```mdl +group g1 (headerText: 'x') -- PARSES +attribute a1 (attributeName: 'title') -- REJECTED +``` + +### Gap 3 — No `DESCRIBE WIDGET`, which is why the generated doc exists at all + +`DESCRIBE JAVA ACTION FeedbackModule.ValidateEmail` emits re-executable MDL. +There is no MDL equivalent for a widget — only `mxcli widget describe`, a CLI +command. + +This reframes the reported bug. Actions need no generated documentation because +`DESCRIBE` answers in-language, against the live project. **The widget `.md` is a +workaround for a missing statement**, and the reason it could drift is that +nothing else could answer the question. Fixing the generator alone treats the +symptom. + +### Gap 4 — Widgets are absent from the reference graph + +`CATALOG.REFS` carries 15 edge kinds and none is widget use: + +``` +action associate call change create datasource delete generalize +home_page layout menu_item parameter retrieve return show_page +``` + +So `show references to ` returns +`MICROFLOW FeedbackModule.VAL_Feedback | call`, and the same question about a +widget returns nothing. Impact analysis for a widget upgrade is unanswerable. +This is the same class as the scheduled-event gap recorded in CLAUDE.md, where a +microflow run only by a scheduled event read as dead until a `schedule` edge was +added. + +### Why honest documentation is not sufficient + +Option 2 of the issue (emit only what the grammar accepts) converts a silent +failure into a documented dead end. Worth doing, and it is slice 1 below — but +it would not have unblocked the reporter. There is no other route: `ALTER PAGE` +rejects the same construct, verified with a working control. + +```mdl +alter page M.Sandbox { + insert into frame { dynamictext t1 (Content: 'hello') } -- PARSES (control) +}; +alter page M.Sandbox { + insert into frame { attribute a1 (attributeName: 'sandbox') } -- REJECTED +}; +``` + +They would have had accurate documentation of a capability gap, and shipped the +same-origin fallback anyway. + +## BSON Structure + +**No new BSON and no new write path.** This is what makes the proposal small, +and it should be re-verified before code is written, because everything rests on +it. + +`PluggableWidgetEngine.applyObjectLists` (`widget_engine.go:1068`) is already +fully def-driven: + +```go +byContainer[strings.ToUpper(lists[i].MDLContainer)] = &lists[i] +... +mapping, ok := byContainer[strings.ToUpper(child.Type)] +``` + +It matches the AST child's `Type` **string** against whatever the def declares, +and knows nothing about the nine keywords. The visitor sets that string from the +token's literal text (`visitor_page_v3.go:554`): + +```go +widget.Type = strings.ToLower(typeCtx.GetText()) +``` + +So the whole pipeline below the parser is text-driven and generic. The keyword +lists exist **only so ANTLR has a token to match** — an artefact of the parser +generator leaking out as a capability boundary. + +`PLUGGABLEWIDGET` and `CUSTOMWIDGET` are additionally **already the same thing**: +both take the `buildPluggable` branch (`cmd_pages_builder_v3.go:430`) and both +store `CustomWidgets$CustomWidget`. + +## Proposed MDL Syntax + +### A widget is named like everything else + +```mdl +create page Sales.Frame ( Title: 'Frame', Layout: Atlas_Core.Atlas_Default ) +{ + htmlelement frame ( tagName: 'div' ) { + attribute sandboxAttr ( + attributeName: 'sandbox', + attributeValueType: 'template', + attributeValueTemplate: 'allow-scripts' + ) + event onClickEvent ( eventName: 'onClick' ) + tagcontentcontainer content { + dynamictext note ( Content: 'Sandboxed' ) + } + } +} +``` + +Nothing here is new syntax — it is the shape `combobox`, `datagrid` and `group` +already use, applied to every widget instead of a blessed subset. `DESCRIBE PAGE` +emits this form, replacing today's `pluggablewidget '' frame ( … )`. + +### Describing a widget is a statement + +```mdl +describe widget htmlelement; -- property table, enums, containers +list widgets; -- every widget with a definition +show references to widget htmlelement; -- which pages use it +``` + +`DESCRIBE WIDGET` is the statement that retires the drift risk: once the answer +is available in-language and against the live project, the generated `.md` stops +being the only source and can be regenerated from — or replaced by — it. + +### The ID form remains, as an escape hatch + +```mdl +widget 'com.acme.widget.Unlisted' w1 ( someProp: 'x' ) +``` + +For a widget whose definition is not loaded, or to be explicit. After slice 2 +this is rarely written and never emitted by `DESCRIBE`. + +### Design notes against `design-mdl-syntax.md` + +- **Reuse existing keywords first.** This proposal goes further: it stops + *adding* them. Today every new widget with an object list needs a new reserved + word — an unbounded cost paid in name collisions (#619's quoting escape hatch + exists for exactly this). +- **One way to do each thing.** Two spellings collapse to one: `combobox c (…)` + and `pluggablewidget 'com.mendix.widget.web.combobox.Combobox' c (…)` are the + same widget today, and `CUSTOMWIDGET` is a third spelling of the same + behaviour. +- **No implicit context**: a container name resolves against the parent widget's + own definition, which is as explicit as a qualified name. +- **One example is enough for an LLM** — and the example the generator already + emits becomes the correct one. + +### The one required doc change + +The generated example omits the **name**, which even a working slot requires: + +``` +tagcontentcontainer { ... } -- as generated today, rejected +tagcontentcontainer content { } -- correct +``` + +So the three child slots that *could* parse today are documented in a form that +cannot either. + +## Implementation Plan + +Six slices. Each ships alone; 1 is independent of the rest. + +### Slice 1 — Stop the bleeding + +Correct whether or not anything else lands. **Items 2–4 are implemented.** +Item 1 was dropped: the premise behind it turned out to be false (below). + +1. ~~**Ship the four missing built-in widgets.**~~ **Dropped — the premise was + wrong.** The draft assumed `events`, `fileuploader`, `googletag` and + `markdown` are bundled with Studio Pro and therefore unreachable to mxcli. + Measured instead: + + | | | + |---|---| + | A blank Mendix 11.13 project (`mx create-project`) | 33 widgets, **none of the four** | + | Installing File Uploader (Marketplace module 235351) | `widgets/` goes 33 → 34, `.mpk` present | + | `exec` of a page using it, straight after | **builds** — no `widget init` needed | + + So a widget whose package is absent is one **Studio Pro cannot use either**; + it is not a gap mxcli can paper over, and there is nothing to ship. The + moment the widget is usable at all, the `.mpk` is in the project and mxcli + picks it up on its own, because `initPluggableEngine` refreshes definitions + from installed packages before reading them. + + Two wrong turns are worth recording, since both looked settled at the time. + Their `.mpk`s are **not** inside `Mendix.Modeler.Core.dll` — that came from a + bare ID-string match, and the assembly has 690 embedded zips with *zero* + `widgets.mendix.com` hits. And embedding a `.def.json` would not have worked + anyway: `getOrGenerateTemplate` (`modelsdk/widgets/loader.go:215`) derives the + template from the `.mpk` **in `widgets/`**, so a def alone only moves the + error to `template not found: fileuploader` — verified before the premise + itself was checked, which is the lesson. The remaining item below is the + whole fix. + +2. **Fix that error message.** It currently says + `(run 'mxcli widget init -p app.mpr')` — a remedy that **provably cannot + work**, since `widget init` scans `widgets/` and these are not there. +3. **Generated examples include the name** on child slots. +4. **`()` is accepted.** `widgetPropertiesV3` requires at least one property, so + `container c ()`, `text t ()` and `pluggablewidget … pw ()` are all parse + errors while bare `pw` and `pw (x: 'y')` are fine. One-line grammar fix, found + in the same investigation. + +### Slice 2 — The widget keyword is def-driven — **implemented** + +5. **Done.** `widgetTypeV3` gains a generic `IDENTIFIER` alternative, ordered + last, so `htmlelement frame ( … )` works for every widget with a definition. +6. `DESCRIBE PAGE` emits the keyword form — **not done here**, see below. + +Not "grammar and visitor only", which was the estimate. The load-bearing half +was the validator, exactly as Open Question 1 warned: + +- The visitor records **which alternative matched** (`ast.WidgetV3.TypeIsGeneric`), + because `Type` alone cannot tell a typo from a built-in — both are a lowercase + string. It is taken from the parse tree, never by comparing the text against a + list of known names, which would reintroduce the list this proposal removes. +- **MDL-WIDGET25 grew a second branch** for a generic type that resolves to no + definition. Without it, measured: `htmlelemnt frame (tagName: 'div')` gave + *"0 errors, 1 warning"* — a warning about `tagName` — while the correct + spelling was completely clean. A typo had traded a parse error for a wrong + answer, which is worse than what it replaced. +- **MDL-WIDGET07 is suppressed** for a generic type that did not resolve. + Reporting its properties on top of the kind error points at the wrong token. + +Item 6 is left for its own change: `DESCRIBE PAGE` currently emits the +`pluggablewidget ''` form, which still round-trips, so this is a readability +improvement rather than a capability and does not belong in the same commit. + +### Slice 0 — The validator knows what a widget is (blocks slices 2–3) + +Report at `check` time what only `exec` catches today: an unknown widget kind or +id, and a container the parent's definition does not declare — each naming the +near misses. The detection already exists in `validateWidgetTreeIn`; only the +reporting is missing. Worth shipping on its own, and the thing that makes +slices 2–3 safe (Open Question 1). + +### Slice 3 — The widget body is def-driven — **implemented** + +7. **Grammar — simpler than proposed.** No parallel `pluggableBodyV3` / + `genericContainerV3` rule was needed: adding `keyword` beside `IDENTIFIER` in + the same last-ordered `widgetTypeV3` alternative covers containers, because a + container and a widget occupy the same position in a body. + + `keyword` is load-bearing, as the draft said: `attribute` lexes as the + `ATTRIBUTE` token and never as `IDENTIFIER`, so slice 2 alone cannot reach the + case that motivated the issue. + + **One ordering fix was required, and it is the finding of this slice.** + `pageBodyV3` listed `widgetV3` *first*. `SLOT`, `PLACEHOLDER` and `USE` are + all inside `keyword` (655 tokens), so the generic alternative swallowed them: + `slot body` became a widget of type `slot`, and `placeholder Main { … }` a + widget named `Main`. The specific alternatives now precede `widgetV3`. + +8. **Validator — done.** A container the parent does not declare is reported + against the parent, naming what it *does* declare: + + ``` + `attribut` is not a container of `htmlelement` — it declares: attribute, + event, tagcontentcontainer, tagcontentrepeatcontainer + ``` + + Inside a resolvable parent this beats "not a widget in this project", so the + generic branch routes to MDL-WIDGET26 when a parent definition is available + and to MDL-WIDGET25 when it is not. + +9. ~~**Delete the nine** from `widgetTypeV3`.~~ **Deferred, deliberately.** They + are no longer a capability boundary — anything else parses too — so what + remains is redundancy, not drift risk: a new container keyword needs no + maintenance because the generic path already accepts it. Removing them would + flip nine keywords to `TypeIsGeneric`, routing them through the generic + branch, where a container inside an *unresolvable* parent would newly report + MDL-WIDGET25. That is a false-positive class in exchange for tidiness. + +**Result.** The construct from the issue parses: + +```mdl +htmlelement frame (tagName: 'div') { + attribute a1 (attributeName: 'data-role', attributeValueType: 'expression') + tagcontentcontainer body { dynamictext t (Content: 'hi') } +} +``` + +Measured across the fixture's definitions: **50 of 50 containers authorable**, +from 16 of 46. With a project, properties are validated against the real +definition — an invented `attributeValueType: 'static'` is MDL-WIDGET08, *valid +values are expression, template*. + +### Slice 4 — `DESCRIBE WIDGET` / `LIST WIDGETS` + +10. An MDL statement returning what `mxcli widget describe` returns, from the + live project. This is what makes the generated `.md` optional rather than + load-bearing, and it is the slice that actually retires #1036's failure mode. + `LIST WIDGETS` follows the repo convention (`list`, not `show`, for new + commands); the existing `SHOW WIDGETS` is unrelated — it lists widget + *instances on pages*, not definitions. + +### Slice 5 — A `widget` edge in `CATALOG.REFS` — **implemented** + +11. ~~Emit one edge per widget instance on a page~~ — **one edge per page (or + snippet) x widget definition.** `show references to combobox` and + `show impact of htmlelement` work. Catalog work, not grammar; independent + of slices 2–4. + + Three details differ from the sketch above, each settled by measurement + rather than by the wording: + + - **Per container, not per instance.** DISTINCT collapses a page's seven + comboboxes into one edge, matching the four sibling widget projections in + `buildReferences`. Per-instance rows say nothing `SHOW REFERENCES` or + `SHOW IMPACT` can use, and `CATALOG.WIDGETS` already holds the instances. + + - **`TargetName` is the MDL name (`COMBOBOX`), not the widget ID.** The + dotted ID poisons the module-derived graph views, which take everything + before the first dot as the module: measured on `testdata/expr-checker`, + it invented a module `com` carrying 14 edges from three real modules, and + listed `com.mendix.widget.web.image.Image` in `graph_god_nodes` with + `ModuleName` `com`. The ID lives in `TargetId`. A widget is consequently + the **first non-dotted target** in the table (0 of 248 before), so + `graph_god_nodes` now excludes `WIDGET` from its asset side. + + - **Only widgets that resolve to a definition.** A built-in stores its BSON + `$Type` in the same column and has none, so it gets no edge — an edge + should point at something describable. + + The syntax in the original sketch (`show references to widget htmlelement`) + was not needed: `show references to ` already parses both a bare word + and a dotted ID, so no grammar changed. It did need one fix on the executor + side — the stored name is SHOUTED while MDL keywords are written in lower + case, so the natural spelling answered "(no references found)", a wrong + answer rather than a missing one. + + Not done, and deliberately: a widget definition is **not** added to + `objects`, so an unused `.mpk` does not appear in `GRAPH_DEAD_ASSETS`. The + anti-join that answers it is documented in `catalog-schema.md` instead. + +### Slice 6 — `PLUGGABLEWIDGET` → `WIDGET` (and collapse `CUSTOMWIDGET`) + +12. Deliberately last: after slice 2 the ID form is rarely written and never + emitted, so this is a readability change on an escape hatch rather than a + headline. Its real value is collapsing `CUSTOMWIDGET` — which already takes + the identical code path and writes the identical BSON — into one keyword, + removing a genuine "two ways to do one thing". + + Old spellings stay accepted (never emitted), so 164 occurrences across 42 + example, skill and doc files keep working and can be migrated at leisure. + + **The cost to weigh**: `WIDGET` already appears in + `ALTER/DESCRIBE STYLING ON PAGE … WIDGET name`, where it means *the widget + named X* rather than *a widget of type X*. The positions are disjoint so it + parses, and both readings are still "widget" — the same way `PAGE` is both + declared and referenced — but it is the argument against, and it should be + made explicitly rather than discovered. + +### Files to modify/create + +| File | Change | +|------|--------| +| `modelsdk/widgets/definitions/{events,fileuploader,googletag,markdown}.def.json` | **new** — the four missing built-ins (slice 1) | +| `mdl/executor/widget_engine.go` | correct the `no definition for widget` remedy (slice 1) | +| `mdl/executor/widget_defs.go` | emit the name in generated examples (slice 1) | +| `mdl/grammar/domains/MDLPage.g4` | `()` accepted (1); identifier widget type (2); `pluggableBodyV3` + `genericContainerV3`, remove the nine (3) | +| `mdl/visitor/visitor_page_v3.go` | set `Type` from an identifier widget type and a generic container | +| `mdl/executor/validate_widgets.go` | resolve containers against the def; suggest near-misses | +| `mdl/executor/cmd_pages_describe_output.go` | emit the keyword form (slice 2) | +| `mdl/grammar/domains/MDLCatalog.g4`, `mdl/ast/`, `mdl/executor/cmd_widgets.go` | `DESCRIBE WIDGET` / `LIST WIDGETS` (slice 4) | +| `mdl/catalog/builder_references.go` | `widget` edge (slice 5) | +| `mdl-examples/doctype-tests/`, `mdl-examples/bug-tests/1036-*.mdl` | the reporter's four cases; HTML Element attributes + events + a child slot | +| `docs/01-project/MDL_QUICK_REFERENCE.md`, `cmd/mxcli/syntax/features_*.go` | the widget vocabulary is per-project, not fixed | + +## Version Compatibility + +None. MDL-side only: no new BSON, no Mendix API, no feature-registry entry, no +`checkFeature()` gate. A widget's object lists come from its own `.mpk`, so +version differences are already carried by the def +(`PROPOSAL_multi_version_pluggable_widgets.md`). + +## Test Plan + +- **The guard that makes this unreintroducible.** Derive every widget keyword, + object-list keyword and child-slot keyword from every `.def.json` in the + fixture, generate a minimal page per keyword, and assert it parses. This test's + absence *is* the bug: two lists and nothing comparing them. Present numbers — + 16 of 46 containers, and 1 of 33 widget keywords — become all of them. +- **The reporter's four cases**, verbatim, in `mdl-examples/bug-tests/`. +- **Round-trip**: `create` → `describe` → `exec` for a widget with both an object + list and a child slot, asserting the description re-parses **and** that + `DESCRIBE` now emits the keyword form. +- **`mx check` at 0 errors** on a project carrying an HTML Element with + `attributes`, plus a Studio Pro open. The BSON path is unchanged, but the claim + that it is unchanged deserves one measurement. +- **Error-quality tests** (the regression risk): an unknown container names the + valid ones for *that* widget; an unknown widget keyword names near-misses; a + typo'd built-in widget still fails informatively. +- **Controls, per CLAUDE.md.** Reverting the validator must make the typo case + report the raw parse error; reverting the grammar must reproduce + `mismatched input 'attribute' expecting '}'`. A test that only passes against + fixed code has not been shown to detect anything. + +## Open Questions + +1. ~~**Does error quality actually survive?**~~ **Settled: no, not as things + stand — and the fix is a prerequisite slice, not a risk to accept.** + + The question assumed the validator would need to match what the parser + catches. Measured, the validator catches **less than assumed**, and the gap + is already open for every case that reaches it today: + + | written | today's verdict | + |---|---| + | `contaner c1 (…)` — typo'd widget kind | **parse error** (the parser is the allow-list) | + | `pluggablewidget 'com.acme.NotAWidget' w1` — unknown widget | `check` **passes**; fails at `exec` | + | `group g1 (…)` inside HTML Element — real keyword, wrong widget | `check` **passes**; fails at `exec` | + + So **`widgetTypeV3` is currently the widget-kind validator.** The validator + has no independent notion of "is this a real widget kind" and does not need + one, because nothing else can parse. Slices 2–3 remove that enforcement, so + as written they would move *every* container mistake into the hole the last + two rows already occupy: `check` green, failure at `exec`. + + **But the detection is already computed.** `validateWidgetTreeIn` holds the + parent's declared object lists and looks the child up in them + (`mapping := parentObjectLists[strings.ToUpper(w.Type)]`), and + `lookupWidgetDef` says whether the type is a known widget. Nothing *reports* + when both miss — the branch routes to `validateStaticWidgetUnknownProps`, + which checks properties of a presumed static widget rather than questioning + the kind. + + That reframes the work. Closing the hole is an improvement **today**, + independent of any grammar change: it makes `check` catch two mistakes that + currently reach `exec`. And once `check` reports them, this question is + answered affirmatively **by construction**, because the semantic error exists + before the parse error is given up. + + **Resolution: a new Slice 0 — "the validator knows what a widget is" — lands + before slices 2–3, and they are blocked on it rather than on a decision.** + It must report, with near-miss suggestions: an unknown widget kind or id, and + a container the parent's definition does not declare. Its own control is that + a *correct* widget and a *correct* container stay silent. + +2. ~~**How far does the ambiguity reach?**~~ **Settled by measurement.** Built + the grammar and diffed `mxcli check` output across all 515 scripts in + `mdl-examples/`, at each step: + + | | verdict changes | message changes | + |---|---|---| + | `IDENTIFIER` (slice 2) | 0 of 515 | 0 | + | `+ keyword` (slice 3, 655 tokens) | 0 of 515 | 0 | + + Ordering the alternative last contains it, and ALL(\*) resolves the rest. + + **But the diff was not sufficient, and that is the more useful finding.** It + compares DIAGNOSTICS, and the real damage was to the AST: `slot body` and + `placeholder Main { … }` were silently reparsed as widgets, still exiting 0. + Two visitor unit tests caught what 515 scripts could not. A corpus diff of + `check` output cannot see a construct that parses into the wrong shape. + + Running it also required fixing the tool: three validators emitted one + violation per property while ranging over a map, so two runs of the *same* + binary disagreed on 11 of 515 scripts — a noise floor larger than the signal. +3. **Should child slots stay named?** Consistency says yes and existing documents + require it. But a slot is a fixed property, not a repeating item, so its name + is never referenced — `tagcontentcontainer { }` reads better and is what the + generator emits today. Allowing both would be two spellings. + Recommend: keep the name, fix the doc. +4. **Does slice 4 make the generated `.md` redundant?** It should, for an agent + that can run `mxcli`. It does not for one reading a repo cold, which is the + case `widget init` was built for. Likely answer: keep generating, but from the + same code path `DESCRIBE WIDGET` uses, so they cannot disagree. +5. ~~**What is a widget edge's source granularity?**~~ **Settled by slice 5: per + (container, widget definition).** Per instance says nothing `SHOW REFERENCES` + or `SHOW IMPACT` can use — both list sources, and `CATALOG.WIDGETS` already + holds the instances — so the extra rows buy nothing at any project size. It + also matches the four sibling widget projections in `buildReferences`, which + have collapsed with DISTINCT all along. + + The question that turned out to matter was not granularity but **what to put + in `TargetName`**, which the draft did not ask. The dotted widget ID poisons + every module-derived graph view; the MDL name does not. See slice 5. +6. ~~**The built-in census.**~~ **Moot — the premise it rested on was wrong.** + It asked whether the four widget IDs scraped out of `Mendix.Modeler.Core.dll` + were a complete list of Studio Pro's bundled widgets. Slice 1 established + they are not bundled at all: a blank 11.13 project ships 33 widgets, none of + them, and a widget whose `.mpk` is absent is one Studio Pro cannot use + either. There is no list to complete. diff --git a/docs/11-proposals/README.md b/docs/11-proposals/README.md index 5ad06d2367..ce98eec667 100644 --- a/docs/11-proposals/README.md +++ b/docs/11-proposals/README.md @@ -135,6 +135,7 @@ for display in this README): | [Replace Generated Playwright Tests with playwright-cli](proposal-playwright-cli.md) | Draft | The current approach (documented in proposal-playwright-testing.md) has Claude Code generate TypeScript test files (.spec.ts), then run them | | [Self-Describing Syntax Feature Registry](syntax-feature-registry.md) | Draft | Branch: research/recursive-help-discovery | | [Structured description of irreducible microflow graphs](PROPOSAL_structured_microflow_description.md) | Draft | DESCRIBE MICROFLOW renders a microflow's control flow as nested if/then/else. | +| [Widgets as first-class MDL, not a second dialect](PROPOSAL_def_driven_widget_bodies.md) | Draft | A widget is named, described and tracked differently from every other MDL extension point, and resolves against hardcoded grammar lists rather than the project. | | [Translations — preserve, describe, author, and auto-translate](PROPOSAL_translations.md) | Partial | A Mendix app ships its user-visible strings in every language it supports. | | [Version-Aware Agent Support](PROPOSAL_version_aware_agent_support.md) | Draft | Three use cases require mxcli to be version-aware at the MDL level: | | [warm dev loop — Docker-free run and iPad split-screen preview](PROPOSAL_mxcli_dev_warm_loop.md) | Draft | Relates to: PROPOSAL_check_mxbuild_gap_heuristics.md (the static-check gate that | diff --git a/mdl-examples/bug-tests/1036-def-driven-widget-body.mdl b/mdl-examples/bug-tests/1036-def-driven-widget-body.mdl new file mode 100644 index 0000000000..111383ed4f --- /dev/null +++ b/mdl-examples/bug-tests/1036-def-driven-widget-body.mdl @@ -0,0 +1,80 @@ +-- mendixlabs/mxcli#1036 — slices 2 and 3 of PROPOSAL_def_driven_widget_bodies.md +-- +-- `mxcli widget init` generates widget documentation for agents to follow. Fed +-- back into `mxcli check` verbatim, the generated htmlelement.md failed on the +-- first line of its own example: +-- +-- line 4:4 mismatched input 'tagcontentcontainer' expecting '}' +-- +-- The doc generator derived a keyword for every object list and child slot in a +-- widget definition; the grammar accepted nine hardcoded ones. Two lists, and +-- nothing comparing them. Measured on the fixture's definitions: 16 of 46 +-- documented constructs parsed. It is 50 of 50 now. +-- +-- The cost was not the wasted sessions. Unable to express the `attributes` list +-- below, the reporter's team fell back to tagContentHTML, which executes +-- same-origin — the exact risk the sandbox they wanted was for. +-- +-- Nothing here is a new keyword. Every name in this file comes from a widget's +-- own definition, which is why `mxcli widget describe htmlelement` lists them. + +create module DefDriven; + +-- A widget named by its own MDL name. `htmlelement` was never in the grammar's +-- widget-type list, though the builder has always resolved it: the page builder +-- tries widgetRegistry.Get(ToUpper(type)) FIRST. Only ANTLR needed a token. +create page DefDriven.HtmlElementBody ( + Title: 'HTML Element with attributes', + Layout: Atlas_Core.Atlas_Default +) { + htmlelement frame (tagName: 'div') { + + -- Object list. `attribute` lexes as the ATTRIBUTE keyword token and never + -- as IDENTIFIER, so accepting a bare identifier as a widget type is NOT + -- enough to reach it — this is the case slice 3 exists for. + attribute a1 ( + attributeName: 'data-role', + attributeValueType: 'expression' + ) + attribute a2 ( + attributeName: 'aria-label', + attributeValueType: 'expression' + ) + + -- A second object list on the same widget. + event e1 ( + eventName: 'onClick' + ) + + -- Child slot: a fixed property holding widgets, not a repeating item. + tagcontentcontainer body { + dynamictext greeting (Content: 'Hello from inside an HTML Element') + } + } +} + +-- A pluggable widget with no properties at all. `()` on a def-driven widget +-- parses for the same reason it does on a built-in (slice 1). +create page DefDriven.BareWidget ( + Title: 'Bare', + Layout: Atlas_Core.Atlas_Default +) { + htmlelement plain () +} + +-- Containers on widgets that already had keywords keep working unchanged — +-- the enumerated alternatives are still tried first, so nothing about an +-- ordinary widget body moved. +create page DefDriven.StillWorks ( + Title: 'Unchanged', + Layout: Atlas_Core.Atlas_Default +) { + accordion acc () { + group g1 (headerText: 'Section one') { + dynamictext t1 (Content: 'inside a group') + } + } + container plain (Class: 'row') { + dynamictext t2 (Content: 'inside a container') + } +} diff --git a/mdl-examples/bug-tests/1036-slice0-validator-knows-widgets.mdl b/mdl-examples/bug-tests/1036-slice0-validator-knows-widgets.mdl new file mode 100644 index 0000000000..3b1d9eff8e --- /dev/null +++ b/mdl-examples/bug-tests/1036-slice0-validator-knows-widgets.mdl @@ -0,0 +1,54 @@ +-- Slice 0 of PROPOSAL_def_driven_widget_bodies.md: the validator knows what a +-- widget is. Both statements below used to pass `mxcli check` and fail only at +-- `exec`. +-- +-- Until now the GRAMMAR was the widget-kind validator: `widgetTypeV3` is an +-- allow-list, so an unknown kind could not parse and the validator never needed +-- an independent notion of one. Two mistakes are not keywords, so neither was +-- caught: +-- +-- MDL-WIDGET25 an explicit widget id that resolves to nothing +-- MDL-WIDGET26 a real container keyword on a parent that has no such container +-- +-- Reporting them is worth doing on its own. It is also what makes slices 2-3 +-- safe: those give up the parser's enforcement, so the semantic check has to +-- exist first (Open Question 1). +-- +-- Both rules are deliberately silent when they cannot be sure: +-- * an id whose .mpk IS installed is real, just not extracted yet — the +-- registry used by `check` reads .mxcli/widgets/ and does NOT refresh from +-- installed packages, so without this guard every widget in a project that +-- never ran `widget init` would be called unknown +-- * a container is never judged against a parent whose definition could not +-- be resolved, for the same reason +-- +-- NOT a .fail.mdl, deliberately. `make check-mdl` runs `check` WITHOUT a +-- project, and both rules need one: +-- +-- MDL-WIDGET25 cannot tell an unknown widget from one installed in a project +-- it cannot see, so with no -p it stays silent +-- MDL-WIDGET26 needs the PARENT's definition resolved, and without a project +-- the registry holds only the nine embedded widgets +-- +-- So this file passes check with no project and reports both rules with one. +-- That is the case the Makefile warns about (#891, #892): naming it .fail.mdl +-- would report "negative test unexpectedly passed" and make a working rule look +-- regressed. The rules are covered by unit tests in validate_widget_kind_test.go +-- instead; this file is the human-readable repro. +-- +-- Run it against a project to see them: +-- mxcli check mdl-examples/bug-tests/1036-slice0-validator-knows-widgets.mdl -p app.mpr + +create page Bug1036.SliceZero ( + title: 'Slice 0', + layout: Atlas_Core.Atlas_Default, + folder: 'Bug1036' +) { + -- MDL-WIDGET25: no definition, and no package in widgets/ either. + pluggablewidget 'com.acme.widget.NotAWidget' w1 (someProp: 'x') + + -- MDL-WIDGET26: `group` is Accordion's container, not HTML Element's. + pluggablewidget 'com.mendix.widget.web.htmlelement.HTMLElement' h (tagName: 'div') { + group g1 (headerText: 'HTML Element has no groups') + } +} diff --git a/mdl-examples/bug-tests/1036-widget-discovery-honesty.mdl b/mdl-examples/bug-tests/1036-widget-discovery-honesty.mdl new file mode 100644 index 0000000000..a2b2d2813b --- /dev/null +++ b/mdl-examples/bug-tests/1036-widget-discovery-honesty.mdl @@ -0,0 +1,40 @@ +-- Slice 1 of PROPOSAL_def_driven_widget_bodies.md — the parts of +-- mendixlabs/mxcli#1036 that need no design work. +-- +-- Three independent defects, all found while reproducing the report: +-- +-- 1. `()` was a parse error on EVERY widget kind. `container c` parsed and +-- `container c (x: 'y')` parsed, but `container c ()` — what an LLM writes +-- for a widget that needs no properties — failed at the `)` with an error +-- that read as though the widget itself were wrong. +-- +-- 2. The `no definition for widget` error always said "run 'mxcli widget init +-- -p app.mpr'". For a widget Studio Pro BUNDLES rather than installs (File +-- Uploader, Events, Google Tag, Markdown viewer) that is worse than +-- unhelpful: widget init scans widgets/, the .mpk is not there, and +-- re-running it can never help. It is now branched on whether the package is +-- actually installed — the same question FindMPK answers for the template +-- loader. +-- +-- 3. `mxcli widget init` generated child-slot examples with NO NAME +-- (`tagcontentcontainer { … }`), which even a working slot rejects — so the +-- three slots that DID parse were documented in a form that could not. +-- Names are now emitted and numbered, since two `slot1`s on one page would +-- collide. +-- +-- Not in this file: the capability gap itself (30 of 46 documented constructs +-- do not parse). That is slices 2-3 of the proposal. + +create page Bug1036.EmptyProps ( + title: 'Empty property lists', + layout: Atlas_Core.Atlas_Default, + folder: 'Bug1036' +) { + -- All three spellings are now accepted, and mean the same thing. + container outer () { + container inner { + dynamictext note ( Content: 'both spellings parse' ) + } + dynamictext empty () + } +} diff --git a/mdl-examples/doctype-tests/34-chart-widget-examples.mdl b/mdl-examples/doctype-tests/34-chart-widget-examples.mdl index 7a722d9974..ca630b2691 100644 --- a/mdl-examples/doctype-tests/34-chart-widget-examples.mdl +++ b/mdl-examples/doctype-tests/34-chart-widget-examples.mdl @@ -180,7 +180,7 @@ create page ChartExamples.P_Line ( StaticXAttribute: Period, StaticYAttribute: Total, StaticName: 'Revenue', - Interpolation: 'smooth' + Interpolation: 'spline' ) } } diff --git a/mdl-examples/doctype-tests/44-describe-widget-examples.mdl b/mdl-examples/doctype-tests/44-describe-widget-examples.mdl new file mode 100644 index 0000000000..9e97276c1a --- /dev/null +++ b/mdl-examples/doctype-tests/44-describe-widget-examples.mdl @@ -0,0 +1,43 @@ +-- DESCRIBE WIDGET — a widget definition, in-language. +-- +-- Slice 4 of PROPOSAL_def_driven_widget_bodies.md. A widget was the only MDL +-- extension point with no DESCRIBE: a microflow, nanoflow, Java action and +-- JavaScript action all describe in-language, against the live project. A +-- widget did not, which is WHY `mxcli widget init` generates markdown +-- documentation at all — and why that documentation could drift from what the +-- parser accepts (mendixlabs/mxcli#1036). +-- +-- The statement and `mxcli widget describe` are the same function +-- (executor.DescribeWidget), so they cannot disagree. + +-- By MDL keyword. +describe widget combobox; + +-- By full widget id — what a widget package, a page's BSON and the generated +-- docs all carry, and the only name a widget without a keyword has. +describe widget 'com.mendix.widget.web.htmlelement.HTMLElement'; + +-- Unlike every other DESCRIBE there is no qualified name: a widget definition +-- is not a document in the model. It comes from a package in the project, or +-- from mxcli's embedded set — which is also why this works with NO project +-- open, the state an agent is in when it asks "what can I write here?". +-- +-- The report covers each property's key, type, caption, category, whether it is +-- required, its default and its enumeration values, plus the dynamic rules the +-- widget's editor uses to hide properties under some configurations. Those +-- rules matter: writing into a hidden property is the CE0463 that +-- mdl-examples/bug-tests/956-fileuploader-six-action-slots.mdl documents. + +-- The report ends with an MDL example that PARSES AS WRITTEN. Both the head +-- form and the containers in it are chosen by probing the real parser, so: +-- +-- gallery -> `gallery widget1 (…) { filter slot1 {…} template slot2 {…} }` +-- its `emptyplaceholder` slot is omitted and named, because +-- the grammar has no keyword for it yet +-- htmlelement -> the `pluggablewidget '' …` form, because `htmlelement` +-- is not yet a widget keyword, and all four of its containers +-- are omitted and named +-- +-- This is the half of the generated .md that was wrong — its example failed on +-- its own first line. Deriving it means it cannot promise syntax that fails, +-- and that it widens on its own once slices 2-3 land. diff --git a/mdl/ast/ast_page_v3.go b/mdl/ast/ast_page_v3.go index 5ee5ff6129..923329d5ce 100644 --- a/mdl/ast/ast_page_v3.go +++ b/mdl/ast/ast_page_v3.go @@ -107,6 +107,23 @@ type WidgetV3 struct { // no name, and carries nothing but this entity and its widgets. Empty for // every other widget, including a Gallery's named `template ` slot. Specialization string + + // TypeIsGeneric records that Type came from the grammar's generic + // IDENTIFIER alternative rather than one of the enumerated widget-type + // tokens (slice 2 of PROPOSAL_def_driven_widget_bodies.md). + // + // The distinction is invisible in Type — both arrive as a lowercase string — + // but it is what tells a typo from a built-in. `htmlelemnt` can ONLY be a + // misspelt widget definition, because a real built-in has its own token; a + // generic type that resolves to no definition is therefore MDL-WIDGET25 + // rather than a static widget to be validated on the builtin property + // vocabulary. Without it the typo passes `check` with a warning about the + // wrong thing. + // + // Set by the visitor from the parse tree, never inferred from a list of + // known widget names — inferring it would reintroduce the list this + // proposal exists to remove. + TypeIsGeneric bool } // DataSourceV3 represents a V3 datasource expression. diff --git a/mdl/ast/ast_query.go b/mdl/ast/ast_query.go index 82cd695249..ee115cf89c 100644 --- a/mdl/ast/ast_query.go +++ b/mdl/ast/ast_query.go @@ -348,8 +348,11 @@ const ( DescribeMenu // DESCRIBE MENU Module.Name (standalone Menus$MenuDocument) DescribeQueue // DESCRIBE QUEUE Module.Name DescribeScheduledEvent // DESCRIBE SCHEDULED EVENT Module.Name - DescribeRegularExpression // DESCRIBE REGULAR EXPRESSION Module.Name - DescribeAuto // DESCRIBE Module.Name — type auto-detected at execution time + // DescribeWidget is not a document — it is a widget DEFINITION, named by + // MDL keyword or widget id. Name carries whichever was written. + DescribeWidget // DESCRIBE WIDGET combobox | DESCRIBE WIDGET 'com.mendix.widget.web.combobox.Combobox' + DescribeRegularExpression // DESCRIBE REGULAR EXPRESSION Module.Name + DescribeAuto // DESCRIBE Module.Name — type auto-detected at execution time ) // String returns the human-readable name of the describe object type. diff --git a/mdl/catalog/builder_pages.go b/mdl/catalog/builder_pages.go index 555feb57ac..ef9ed48daa 100644 --- a/mdl/catalog/builder_pages.go +++ b/mdl/catalog/builder_pages.go @@ -554,21 +554,9 @@ func extractWidgetsRecursive(w map[string]any) []rawWidgetInfo { } } - // Handle CustomWidget nested widgets in properties + // Handle CustomWidget nested widgets in properties — both kinds of container. if obj, ok := w["Object"].(map[string]any); ok { - props := getBsonArrayElements(obj["Properties"]) - for _, prop := range props { - if propMap, ok := prop.(map[string]any); ok { - if value, ok := propMap["Value"].(map[string]any); ok { - propWidgets := getBsonArrayElements(value["Widgets"]) - for _, pw := range propWidgets { - if pwMap, ok := pw.(map[string]any); ok { - result = append(result, extractWidgetsRecursive(pwMap)...) - } - } - } - } - } + result = append(result, widgetsInPropertyBag(obj)...) } // Handle NavigationList items @@ -587,6 +575,51 @@ func extractWidgetsRecursive(w map[string]any) []rawWidgetInfo { return result } +// widgetsInPropertyBag walks a pluggable widget's stored property bag and +// indexes every widget inside it, through BOTH kinds of container: +// +// Value.Widgets a child slot — a Gallery's content, an HTML Element's body +// Value.Objects an object list — a DataGrid2 column, a chart series +// +// Only the first was walked, so anything placed in a column, a gallery item or +// a series was invisible to the catalog. Measured on a real project: a page +// holding 19 chart sparklines inside datagrid columns did not appear under +// "which pages use VegaChart?", while the grid around them did. +// +// The consequence is wider than the widget edge, because CATALOG.REFS is a +// projection of this table: an entity or microflow used ONLY inside a column +// template reported zero references, so anything using reference counts to +// decide "unused, safe to delete" would delete a document in active use. That is +// issue #940's failure mode — fixed for List View templates, left open here. +// +// An object-list item is itself a property bag, so the walk recurses: a column +// holding a nested widget that has its own object list is covered without a +// second case. +func widgetsInPropertyBag(bag map[string]any) []rawWidgetInfo { + var result []rawWidgetInfo + for _, prop := range getBsonArrayElements(bag["Properties"]) { + propMap, ok := prop.(map[string]any) + if !ok { + continue + } + value, ok := propMap["Value"].(map[string]any) + if !ok { + continue + } + for _, pw := range getBsonArrayElements(value["Widgets"]) { + if pwMap, ok := pw.(map[string]any); ok { + result = append(result, extractWidgetsRecursive(pwMap)...) + } + } + for _, obj := range getBsonArrayElements(value["Objects"]) { + if objMap, ok := obj.(map[string]any); ok { + result = append(result, widgetsInPropertyBag(objMap)...) + } + } + } + return result +} + // extractSnippetWidgets extracts all widgets from raw snippet BSON data. func extractSnippetWidgets(rawData map[string]any) []rawWidgetInfo { // Handle both snippet formats: diff --git a/mdl/catalog/builder_pages_objectlist_test.go b/mdl/catalog/builder_pages_objectlist_test.go new file mode 100644 index 0000000000..9709365a5e --- /dev/null +++ b/mdl/catalog/builder_pages_objectlist_test.go @@ -0,0 +1,144 @@ +// SPDX-License-Identifier: Apache-2.0 + +package catalog + +import "testing" + +// gridWithWidgetInAColumn is the BSON shape of a pluggable widget whose +// object-list ITEM holds widgets: a DataGrid2 column with custom content, a +// gallery item, a chart series. The item lives in `Value.Objects`, and each +// object carries its own `Properties[].Value.Widgets`. +func gridWithWidgetInAColumn() map[string]any { + return map[string]any{ + "$ID": "grid-1", + "Name": "grid1", + "$Type": "CustomWidgets$CustomWidget", + "Type": map[string]any{"WidgetId": "com.mendix.widget.web.datagrid.Datagrid"}, + "Object": map[string]any{ + "Properties": []any{ + int32(3), + map[string]any{ + "TypePointer": "t-columns", + "Value": map[string]any{ + "Objects": []any{ + int32(3), + map[string]any{ + "Properties": []any{ + int32(3), + map[string]any{ + "TypePointer": "t-content", + "Value": map[string]any{ + "Widgets": []any{ + int32(3), + map[string]any{ + "$ID": "pb-1", + "Name": "pb1", + "$Type": "CustomWidgets$CustomWidget", + "Type": map[string]any{ + "WidgetId": "com.mendix.widget.custom.progressbar.ProgressBar", + }, + }, + }, + }, + }, + }, + }, + }, + }, + }, + }, + }, + } +} + +// A widget inside an object-list item must be indexed. +// +// The catalog walked a pluggable widget's `Object.Properties[].Value.Widgets` +// (a child slot) but not `Value.Objects[]` (an object list), so anything placed +// in a DataGrid2 column, a gallery item or a chart series was invisible. +// +// Measured on a real project by an external test: a page holding 19 chart +// sparklines inside datagrid columns did not appear under "which pages use +// VegaChart?", while the grid around them did. Reproduced here on the fixture — +// a progressbar in a `column c2 { … }` gave one row (the grid), ground truth two. +// +// It matters beyond the widget edge: CATALOG.REFS is a projection of this table, +// so an entity or microflow used ONLY inside a column template reported zero +// references, and anything using reference counts to decide "unused, safe to +// delete" would delete a document in active use. That is issue #940's failure +// mode, which was fixed for List View templates and left open here. +func TestExtractWidgetsRecursive_ObjectListItemWidgets(t *testing.T) { + got := extractWidgetsRecursive(gridWithWidgetInAColumn()) + + var sawGrid, sawNested bool + for _, w := range got { + switch w.WidgetType { + case "com.mendix.widget.web.datagrid.Datagrid": + sawGrid = true + case "com.mendix.widget.custom.progressbar.ProgressBar": + sawNested = true + } + } + if !sawGrid { + t.Error("the grid itself was not indexed") + } + if !sawNested { + t.Errorf("the widget inside the column was not indexed; got %d widgets: %+v", + len(got), got) + } +} + +// The control: a child slot (Value.Widgets, no Objects) must keep working. A fix +// that swapped one traversal for the other would pass the test above and lose +// every Gallery content widget. +func TestExtractWidgetsRecursive_ChildSlotStillWalked(t *testing.T) { + w := map[string]any{ + "$ID": "g-1", "Name": "gal1", "$Type": "CustomWidgets$CustomWidget", + "Type": map[string]any{"WidgetId": "com.mendix.widget.web.gallery.Gallery"}, + "Object": map[string]any{ + "Properties": []any{ + int32(3), + map[string]any{ + "TypePointer": "t-content", + "Value": map[string]any{ + "Widgets": []any{ + int32(3), + map[string]any{"$ID": "t-1", "Name": "txt1", "$Type": "Forms$DynamicText"}, + }, + }, + }, + }, + }, + } + var sawText bool + for _, got := range extractWidgetsRecursive(w) { + if got.WidgetType == "Forms$DynamicText" { + sawText = true + } + } + if !sawText { + t.Error("a widget in a child slot stopped being indexed") + } +} + +// The second control: an object list with no widgets in it must not invent rows, +// and must not panic on the marker-only array. getBsonArrayElements strips the +// leading typed-array marker, so an empty list is length 0 here and length 1 raw. +func TestExtractWidgetsRecursive_EmptyObjectList(t *testing.T) { + w := map[string]any{ + "$ID": "g-1", "Name": "c1", "$Type": "CustomWidgets$CustomWidget", + "Type": map[string]any{"WidgetId": "com.acme.Thing"}, + "Object": map[string]any{ + "Properties": []any{ + int32(3), + map[string]any{ + "TypePointer": "t", + "Value": map[string]any{"Objects": []any{int32(3)}}, + }, + }, + }, + } + if got := extractWidgetsRecursive(w); len(got) != 1 { + t.Errorf("got %d widgets, want 1 (the widget itself): %+v", len(got), got) + } +} diff --git a/mdl/catalog/builder_references.go b/mdl/catalog/builder_references.go index d516e88c20..2e32c485aa 100644 --- a/mdl/catalog/builder_references.go +++ b/mdl/catalog/builder_references.go @@ -34,6 +34,7 @@ const ( RefKindReturn = "return" // Microflow/nanoflow returns an entity type RefKindSchedule = "schedule" // Scheduled event runs a microflow RefKindValidate = "validate" // Attribute validation rule uses a regular expression + RefKindWidget = "widget" // Page/snippet uses a pluggable or custom widget ) // collectActionActivities returns all ActionActivity objects from an ObjectCollection, @@ -426,6 +427,12 @@ func (b *Builder) buildReferences() error { } } } + + // Page/snippet -> widget definition. buildWidgetDefinitions runs before + // this pass, so the join target is populated. + if n, werr := insertWidgetRefs(b.tx, projectID, snapshotID); werr == nil { + refCount += n + } } // Extract navigation references (home pages, menu items, login pages) diff --git a/mdl/catalog/builder_widget_refs.go b/mdl/catalog/builder_widget_refs.go new file mode 100644 index 0000000000..8ef654ee24 --- /dev/null +++ b/mdl/catalog/builder_widget_refs.go @@ -0,0 +1,77 @@ +// SPDX-License-Identifier: Apache-2.0 + +package catalog + +// insertWidgetRefs emits the `widget` edge: one row per (page or snippet) x +// widget definition actually used on it. +// +// Slice 5 of PROPOSAL_def_driven_widget_bodies.md. A widget was the one MDL +// extension point with no edge in CATALOG.REFS, so "which pages use this +// widget?" was unanswerable while the same question about a Java action was one +// query. It is also the question an upgrade asks: a .mpk shipped in widgets/ +// that no page uses is dead weight, and one used on forty pages is not +// something to swap lightly. +// +// Both halves are already in the catalog, so this is a projection and costs no +// extra parse: widgets_data.WidgetType carries the widget ID for a pluggable or +// custom widget (buildPages resolves Type.WidgetId out of the +// CustomWidgets$CustomWidget wrapper), and widget_definitions_data is keyed by +// that same ID. +// +// # Only widgets that resolve to a definition +// +// The join is the filter. A built-in Mendix widget stores its BSON $Type in the +// same column (Forms$DynamicText, Forms$ActionButton, ...) and has no +// definition, so it gets no edge — deliberately. An edge is a pointer to +// something describable, and `Forms$TextBox` is a language primitive, not a +// document: emitting one would put a target in the graph that nothing can +// resolve. "Which pages have a text box?" is already answerable directly from +// CATALOG.WIDGETS. +// +// # TargetName is the MDL name, not the widget ID +// +// Measured on testdata/expr-checker (15 widget edges either way), with the +// widget ID as TargetName: +// +// graph_module_coupling gains a module "com" with 14 edges, from three +// different source modules +// graph_god_nodes reports com.mendix.widget.web.image.Image with +// ModuleName "com" +// +// Those views derive a module by taking everything before the FIRST dot, which +// is sound for a qualified name and nonsense for a dotted widget ID. Using the +// MDL name (IMAGE, COMBOBOX) leaves graph_module_coupling identical to the +// baseline, because its existing instr(TargetName, '.') > 0 guard skips a +// non-dotted target for free. It is also the spelling a user has in hand: +// `show references to combobox` is what you type after `describe widget +// combobox`, and it is the keyword the page body uses. +// +// The widget ID is not lost — it goes in TargetId, which is what that column is +// for. Two packages shipping the same MDL name would share a TargetName and be +// told apart by TargetId; SHOW REFERENCES would list both, which is a better +// failure than being unable to name the widget at all. +// +// # One edge per page, not per instance +// +// DISTINCT collapses the seven comboboxes on a page into one edge, matching the +// four sibling projections in buildReferences. Per-instance rows would say +// nothing SHOW REFERENCES or SHOW IMPACT could use, and CATALOG.WIDGETS already +// holds the instances. +func insertWidgetRefs(tx CatalogTx, projectID, snapshotID string) (int, error) { + res, err := tx.Exec( + `INSERT INTO refs (SourceType, SourceId, SourceName, TargetType, TargetId, TargetName, RefKind, ModuleName, ProjectId, SnapshotId) + SELECT DISTINCT w.ContainerType, '', w.ContainerQualifiedName, + 'WIDGET', d.WidgetId, d.MdlName, ?, w.ModuleName, ?, ? + FROM widgets_data w + JOIN widget_definitions_data d ON d.WidgetId = w.WidgetType + WHERE w.ContainerQualifiedName != '' AND d.MdlName != ''`, + RefKindWidget, projectID, snapshotID) + if err != nil { + return 0, err + } + n, err := res.RowsAffected() + if err != nil { + return 0, err + } + return int(n), nil +} diff --git a/mdl/catalog/builder_widget_refs_test.go b/mdl/catalog/builder_widget_refs_test.go new file mode 100644 index 0000000000..5f8b996d2b --- /dev/null +++ b/mdl/catalog/builder_widget_refs_test.go @@ -0,0 +1,252 @@ +// SPDX-License-Identifier: Apache-2.0 + +package catalog + +import ( + "testing" +) + +// seedWidgetRefFixture builds a small project in an in-memory catalog: +// +// Sales.OrderList 3 comboboxes, 1 datagrid, 2 dynamic texts (built-in) +// Sales.OrderForm 1 combobox +// Sales.AddressSnip 1 combobox (a SNIPPET, not a page) +// +// The three comboboxes on one page are what proves DISTINCT; the dynamic texts +// are what proves a built-in gets no edge. +func seedWidgetRefFixture(t *testing.T) *Catalog { + t.Helper() + cat, err := New() + if err != nil { + t.Fatalf("New: %v", err) + } + t.Cleanup(func() { cat.Close() }) + db := cat.CatalogDB() + + defs := []struct{ id, mdl string }{ + {"com.mendix.widget.web.combobox.Combobox", "COMBOBOX"}, + {"com.mendix.widget.web.datagrid.Datagrid", "DATAGRID"}, + {"com.acme.widget.Unused.Unused", "UNUSED"}, + } + for _, d := range defs { + if _, err := db.Exec( + `INSERT INTO widget_definitions_data (WidgetId, MdlName, WidgetKind, ProjectId, SnapshotId) + VALUES (?, ?, 'pluggable', 'p', 's')`, d.id, d.mdl); err != nil { + t.Fatalf("seed definition %s: %v", d.id, err) + } + } + + widgets := []struct{ id, wtype, container, ctype string }{ + {"w1", "com.mendix.widget.web.combobox.Combobox", "Sales.OrderList", "PAGE"}, + {"w2", "com.mendix.widget.web.combobox.Combobox", "Sales.OrderList", "PAGE"}, + {"w3", "com.mendix.widget.web.combobox.Combobox", "Sales.OrderList", "PAGE"}, + {"w4", "com.mendix.widget.web.datagrid.Datagrid", "Sales.OrderList", "PAGE"}, + {"w5", "Forms$DynamicText", "Sales.OrderList", "PAGE"}, + {"w6", "Forms$DynamicText", "Sales.OrderList", "PAGE"}, + {"w7", "com.mendix.widget.web.combobox.Combobox", "Sales.OrderForm", "PAGE"}, + {"w8", "com.mendix.widget.web.combobox.Combobox", "Sales.AddressSnip", "SNIPPET"}, + } + for _, w := range widgets { + if _, err := db.Exec( + `INSERT INTO widgets_data (Id, Name, WidgetType, ContainerQualifiedName, ContainerType, ModuleName, ProjectId, SnapshotId) + VALUES (?, ?, ?, ?, ?, 'Sales', 'p', 's')`, + w.id, w.id, w.wtype, w.container, w.ctype); err != nil { + t.Fatalf("seed widget %s: %v", w.id, err) + } + } + return cat +} + +func runInsertWidgetRefs(t *testing.T, cat *Catalog) int { + t.Helper() + tx, err := cat.CatalogDB().Begin() + if err != nil { + t.Fatalf("Begin: %v", err) + } + n, err := insertWidgetRefs(tx, "p", "s") + if err != nil { + tx.Rollback() + t.Fatalf("insertWidgetRefs: %v", err) + } + if err := tx.Commit(); err != nil { + t.Fatalf("Commit: %v", err) + } + return n +} + +// The edge itself: one row per container x widget definition, named by the MDL +// name, carrying the widget ID, and covering snippets as well as pages. +func TestInsertWidgetRefs_EmitsOneEdgePerContainer(t *testing.T) { + cat := seedWidgetRefFixture(t) + if n := runInsertWidgetRefs(t, cat); n != 4 { + t.Fatalf("inserted %d edges, want 4: OrderList x COMBOBOX, OrderList x DATAGRID, OrderForm x COMBOBOX, AddressSnip x COMBOBOX", n) + } +} + +func TestInsertWidgetRefs_Rows(t *testing.T) { + cat := seedWidgetRefFixture(t) + runInsertWidgetRefs(t, cat) + + rows, err := cat.CatalogDB().Query( + `SELECT SourceType, SourceName, TargetType, TargetName, TargetId, RefKind + FROM refs ORDER BY SourceName, TargetName`) + if err != nil { + t.Fatalf("query refs: %v", err) + } + defer rows.Close() + + type row struct{ srcType, src, tgtType, tgt, tgtID, kind string } + var got []row + for rows.Next() { + var r row + if err := rows.Scan(&r.srcType, &r.src, &r.tgtType, &r.tgt, &r.tgtID, &r.kind); err != nil { + t.Fatalf("scan: %v", err) + } + got = append(got, r) + } + + want := []row{ + {"SNIPPET", "Sales.AddressSnip", "WIDGET", "COMBOBOX", "com.mendix.widget.web.combobox.Combobox", RefKindWidget}, + {"PAGE", "Sales.OrderForm", "WIDGET", "COMBOBOX", "com.mendix.widget.web.combobox.Combobox", RefKindWidget}, + {"PAGE", "Sales.OrderList", "WIDGET", "COMBOBOX", "com.mendix.widget.web.combobox.Combobox", RefKindWidget}, + {"PAGE", "Sales.OrderList", "WIDGET", "DATAGRID", "com.mendix.widget.web.datagrid.Datagrid", RefKindWidget}, + } + if len(got) != len(want) { + t.Fatalf("got %d rows, want %d:\n got: %v\nwant: %v", len(got), len(want), got, want) + } + for i := range want { + if got[i] != want[i] { + t.Errorf("row %d:\n got %+v\nwant %+v", i, got[i], want[i]) + } + } +} + +// Three comboboxes on one page are one edge, not three. Without DISTINCT this +// test sees 6 rows. +func TestInsertWidgetRefs_CollapsesInstances(t *testing.T) { + cat := seedWidgetRefFixture(t) + runInsertWidgetRefs(t, cat) + + var n int + if err := cat.CatalogDB().QueryRow( + `SELECT COUNT(*) FROM refs WHERE SourceName = 'Sales.OrderList' AND TargetName = 'COMBOBOX'`, + ).Scan(&n); err != nil { + t.Fatalf("count: %v", err) + } + if n != 1 { + t.Errorf("Sales.OrderList -> COMBOBOX = %d edges, want 1 — the page has three combobox instances", n) + } +} + +// A built-in widget stores its BSON $Type in the same column and has no +// definition. It must not produce an edge to a target nothing can resolve. The +// control is in the same fixture: the pluggable widgets on that same page DO +// get edges, so "no built-in edge" cannot pass by emitting nothing at all. +func TestInsertWidgetRefs_SkipsBuiltins(t *testing.T) { + cat := seedWidgetRefFixture(t) + runInsertWidgetRefs(t, cat) + + var builtin, pluggable int + if err := cat.CatalogDB().QueryRow( + `SELECT COUNT(*) FROM refs WHERE TargetName LIKE 'Forms$%' OR TargetName = 'DynamicText'`, + ).Scan(&builtin); err != nil { + t.Fatalf("count builtin: %v", err) + } + if builtin != 0 { + t.Errorf("built-in widgets produced %d edges, want 0", builtin) + } + if err := cat.CatalogDB().QueryRow( + `SELECT COUNT(*) FROM refs WHERE SourceName = 'Sales.OrderList'`, + ).Scan(&pluggable); err != nil { + t.Fatalf("count pluggable: %v", err) + } + if pluggable != 2 { + t.Errorf("control: Sales.OrderList has %d edges, want 2 — if this is 0 the test above proves nothing", pluggable) + } +} + +// An installed .mpk no page uses gets no edge, which is what makes "unused +// widget package" answerable. +func TestInsertWidgetRefs_UnusedDefinitionHasNoEdge(t *testing.T) { + cat := seedWidgetRefFixture(t) + runInsertWidgetRefs(t, cat) + + var n int + if err := cat.CatalogDB().QueryRow( + `SELECT COUNT(*) FROM refs WHERE TargetName = 'UNUSED'`).Scan(&n); err != nil { + t.Fatalf("count: %v", err) + } + if n != 0 { + t.Errorf("UNUSED has %d inbound edges, want 0", n) + } +} + +// Why TargetName is the MDL name and not the widget ID. +// +// graph_module_coupling and graph_module_cohesion derive a module by taking +// everything before the FIRST dot. That is sound for a qualified name and +// nonsense for a dotted widget ID: measured on testdata/expr-checker, using the +// widget ID invented a module called "com" carrying 14 edges from three real +// modules. A non-dotted MDL name is skipped by those views' own +// instr(TargetName, '.') > 0 guard. +// +// The control is the second half: with the widget ID written into the same +// fixture, the fake module DOES appear — so this asserts a property of the +// choice, not of the fixture. +func TestInsertWidgetRefs_MdlNameKeepsModuleViewsClean(t *testing.T) { + cat := seedWidgetRefFixture(t) + runInsertWidgetRefs(t, cat) + + countCoupling := func(target string) int { + var n int + if err := cat.CatalogDB().QueryRow( + `SELECT COUNT(*) FROM graph_module_coupling WHERE TargetModule = ?`, target, + ).Scan(&n); err != nil { + t.Fatalf("query graph_module_coupling: %v", err) + } + return n + } + + if n := countCoupling("com"); n != 0 { + t.Errorf("graph_module_coupling has %d rows for a module 'com', want 0", n) + } + + // Control: the widget ID as TargetName does invent that module. + if _, err := cat.CatalogDB().Exec( + `INSERT INTO refs (SourceType, SourceId, SourceName, TargetType, TargetId, TargetName, RefKind, ModuleName, ProjectId, SnapshotId) + VALUES ('PAGE', '', 'Sales.OrderList', 'WIDGET', '', 'com.mendix.widget.web.combobox.Combobox', ?, 'Sales', 'p', 's')`, + RefKindWidget); err != nil { + t.Fatalf("seed control row: %v", err) + } + if n := countCoupling("com"); n == 0 { + t.Error("control: a dotted widget ID as TargetName should invent a module 'com' — if it does not, this test cannot detect the problem it exists for") + } +} + +// A widget definition belongs to no Mendix module, so it must not be listed as +// an asset in graph_god_nodes, where every other row is a module-qualified +// document. The page's out-degree still counts it. +func TestWidgetRefsStayOffTheGodNodeAssetList(t *testing.T) { + cat := seedWidgetRefFixture(t) + runInsertWidgetRefs(t, cat) + + var asAsset int + if err := cat.CatalogDB().QueryRow( + `SELECT COUNT(*) FROM graph_god_nodes WHERE Asset IN ('COMBOBOX', 'DATAGRID')`, + ).Scan(&asAsset); err != nil { + t.Fatalf("query graph_god_nodes: %v", err) + } + if asAsset != 0 { + t.Errorf("graph_god_nodes lists %d widget definitions as assets, want 0", asAsset) + } + + var outDeg int + if err := cat.CatalogDB().QueryRow( + `SELECT OutDegree FROM graph_god_nodes WHERE Asset = 'Sales.OrderList'`, + ).Scan(&outDeg); err != nil { + t.Fatalf("query OutDegree: %v", err) + } + if outDeg != 2 { + t.Errorf("Sales.OrderList OutDegree = %d, want 2 — the page's dependency on the widgets it uses is real and must survive", outDeg) + } +} diff --git a/mdl/catalog/tables.go b/mdl/catalog/tables.go index d4ef34b350..63adc5250c 100644 --- a/mdl/catalog/tables.go +++ b/mdl/catalog/tables.go @@ -7,6 +7,12 @@ package catalog // // History: // +// 11 — the `widget` edge in refs (page/snippet -> widget definition) and the +// graph_god_nodes change that keeps widget targets off the asset side. +// Both need the bump for the same reason: refs are only written by +// REFRESH CATALOG FULL and a view is CREATE VIEW IF NOT EXISTS, so a +// cached catalog would answer `show references to combobox` with +// "(no references found)" — a wrong answer, not a missing one. // 10 — the three lookups expression type checking needs and the catalog could // not answer: attributes_data.EnumerationQualifiedName (DataType says only // "Enumeration", losing which one), enumeration_values_data (the table @@ -23,7 +29,7 @@ package catalog // SnapshotSource / SourceId / SourceBranch / SourceRevision columns // from every row (issue #576). // 1 — initial flat schema with denormalized snapshot columns on every row. -const CatalogSchemaVersion = "10" +const CatalogSchemaVersion = "11" // MetaSchemaVersion is the catalog_meta key that records the schema version // the cache was built against. @@ -1297,8 +1303,14 @@ func (c *Catalog) createTables() error { // graph_god_nodes — degree centrality (most depended-upon / highest fan-out). `CREATE VIEW IF NOT EXISTS graph_god_nodes AS WITH deg AS ( + -- WIDGET targets are excluded from the ASSET side: a widget + -- definition belongs to no Mendix module, and every other row here + -- is a module-qualified document, so it would list as an asset whose + -- ModuleName is its own name (the ELSE d.Asset fallback below) and + -- whose ObjectType is NULL. A page's OUT-degree still counts the + -- widgets it uses, which is a real dependency. SELECT TargetName AS Asset, COUNT(*) AS InDeg, 0 AS OutDeg - FROM refs WHERE TargetName != '' GROUP BY TargetName + FROM refs WHERE TargetName != '' AND TargetType != 'WIDGET' GROUP BY TargetName UNION ALL SELECT SourceName AS Asset, 0 AS InDeg, COUNT(*) AS OutDeg FROM refs WHERE SourceName != '' GROUP BY SourceName diff --git a/mdl/executor/cmd_pages_builder_missing_widget_test.go b/mdl/executor/cmd_pages_builder_missing_widget_test.go new file mode 100644 index 0000000000..bbb2b0d21b --- /dev/null +++ b/mdl/executor/cmd_pages_builder_missing_widget_test.go @@ -0,0 +1,55 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "os" + "path/filepath" + "strings" + "testing" +) + +// A widget whose package is NOT in widgets/ must not be told to run +// `widget init`. That command scans widgets/, so for a widget Studio Pro +// bundles rather than installs it can never help — which is what cost the +// reporter of mendixlabs/mxcli#1036 a debugging session. +func TestMissingWidgetMessage_UninstalledWidgetDoesNotRecommendWidgetInit(t *testing.T) { + dir := t.TempDir() + if err := os.MkdirAll(filepath.Join(dir, "widgets"), 0o755); err != nil { + t.Fatal(err) + } + got := missingWidgetMessage(dir, "com.mendix.widget.web.fileuploader.FileUploader") + + if strings.Contains(got, "run 'mxcli widget init") { + t.Errorf("recommends a command that cannot work:\n%s", got) + } + if !strings.Contains(got, "cannot help") { + t.Errorf("does not say why widget init is not the remedy:\n%s", got) + } + if !strings.Contains(got, "com.mendix.widget.web.fileuploader.FileUploader") { + t.Errorf("does not name the widget:\n%s", got) + } +} + +// The control: when the package IS installed, `widget init` is exactly the +// right remedy and must still be named. Without this, the test above passes +// against a build that simply deleted the recommendation. +func TestMissingWidgetMessage_InstalledWidgetStillRecommendsWidgetInit(t *testing.T) { + // FindMPK PARSES each .mpk rather than matching on its name, so this needs + // a real package — the fixture project ships Accordion among 33 others. + // (An empty file named after the widget is silently skipped, which is what + // the first version of this control got wrong.) + dir := filepath.Join("..", "..", "testdata", "expr-checker") + if _, err := os.Stat(filepath.Join(dir, "widgets")); err != nil { + t.Skipf("fixture widgets/ not available: %v", err) + } + + got := missingWidgetMessage(dir, "com.mendix.widget.web.accordion.Accordion") + + if !strings.Contains(got, "mxcli widget init") { + t.Errorf("installed package should still point at widget init:\n%s", got) + } + if !strings.Contains(got, "not extracted yet") { + t.Errorf("does not say the package is present but unextracted:\n%s", got) + } +} diff --git a/mdl/executor/cmd_pages_builder_v3.go b/mdl/executor/cmd_pages_builder_v3.go index d448c0587c..a892b87857 100644 --- a/mdl/executor/cmd_pages_builder_v3.go +++ b/mdl/executor/cmd_pages_builder_v3.go @@ -5,6 +5,7 @@ package executor import ( "fmt" "log" + "path/filepath" "regexp" "strings" @@ -16,6 +17,7 @@ import ( "github.com/mendixlabs/mxcli/sdk/domainmodel" "github.com/mendixlabs/mxcli/sdk/microflows" "github.com/mendixlabs/mxcli/sdk/pages" + "github.com/mendixlabs/mxcli/sdk/widgets/mpk" ) // ============================================================================ @@ -432,7 +434,7 @@ func (pb *pageBuilder) buildWidgetV3(w *ast.WidgetV3) (pages.Widget, error) { if def, ok := pb.widgetRegistry.GetByWidgetID(widgetType); ok { return pb.buildPluggable(def, w) } - return nil, mdlerrors.NewNotFoundMsg("widget", widgetType, "no definition for widget "+widgetType+" (run 'mxcli widget init -p app.mpr')") + return nil, mdlerrors.NewNotFoundMsg("widget", widgetType, pb.missingWidgetMessage(widgetType)) } } } @@ -2544,3 +2546,49 @@ func (pb *pageBuilder) buildMenuBarV3(w *ast.WidgetV3) (pages.Widget, error) { NavigationProfile: w.GetStringProp("Profile"), }, nil } + +// missingWidgetMessage explains why a widget has no definition, and — the part +// that matters — names a remedy that can actually work. +// +// The old message always said "run 'mxcli widget init -p app.mpr'". When the +// widget's package is not in the project at all — File Uploader, Events, Google +// Tag and Markdown viewer are in no blank project, measured on 11.13 — that is +// worse than unhelpful: `widget init` scans `widgets/`, the .mpk is not there, +// and re-running it can never help. Reported as the postscript to +// mendixlabs/mxcli#1036, where it cost the reporter a debugging session. +// +// The remedy is to install the widget, which is also the only way to use it in +// Studio Pro. Measured: a blank 11.13 project ships 33 widgets and none of those +// four; installing File Uploader takes widgets/ from 33 to 34, and mxcli then +// builds the page with no further action, because initPluggableEngine refreshes +// definitions from installed packages on its own. mxcli therefore ships no +// definitions for them — there is nothing to ship that the project does not +// already carry once the widget is usable at all. +// +// The distinguishing question is exactly the one FindMPK answers, and it is the +// same lookup the template loader makes before giving up. +func (pb *pageBuilder) missingWidgetMessage(widgetID string) string { + projectDir := "" + if pb.backend != nil { + projectDir = filepath.Dir(pb.backend.Path()) + } + return missingWidgetMessage(projectDir, widgetID) +} + +// missingWidgetMessage is the pure form, so the branch can be tested without +// standing up a whole backend. +func missingWidgetMessage(projectDir, widgetID string) string { + if projectDir != "" { + if found, err := mpk.FindMPK(projectDir, widgetID); err == nil && found != "" { + // The package is installed; the definition just has not been + // extracted from it yet. This is the case `widget init` exists for. + return "no definition for widget " + widgetID + + " — its package is installed but not extracted yet (run 'mxcli widget init -p app.mpr')" + } + } + return "no definition for widget " + widgetID + + " — the project has no widget package for it in widgets/." + + " 'mxcli widget init' cannot help: it scans widgets/, and the package is not there." + + " Install the widget or its module from the Marketplace; that puts the .mpk in widgets/," + + " after which mxcli picks it up automatically." +} diff --git a/mdl/executor/cmd_pages_describe.go b/mdl/executor/cmd_pages_describe.go index 21e00be477..3a7a7d0439 100644 --- a/mdl/executor/cmd_pages_describe.go +++ b/mdl/executor/cmd_pages_describe.go @@ -670,6 +670,16 @@ type rawWidget struct { // Object-list child blocks (for generic PLUGGABLEWIDGET output): chart series, // lines, scale colors, etc. Reconstructed from the widget's WidgetObject lists. ObjectLists []rawObjectList + + // ChildSlots are the widget's reconstructed child slots — fixed properties + // holding widgets, as opposed to ObjectLists' repeated items. + ChildSlots []rawChildSlot + + // OmittedContainers names container-shaped properties present in the stored + // document that DESCRIBE could not reproduce. Emitted as a comment so a + // describe -> exec round trip cannot silently delete a widget's body. + // See unreconstructedContainers. + OmittedContainers []string // Data container context: entity qualified name provided by this container EntityContext string // Full widget ID (e.g. "com.mendix.widget.custom.switch.Switch") diff --git a/mdl/executor/cmd_pages_describe_childslots.go b/mdl/executor/cmd_pages_describe_childslots.go new file mode 100644 index 0000000000..9ae003189c --- /dev/null +++ b/mdl/executor/cmd_pages_describe_childslots.go @@ -0,0 +1,119 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "fmt" + "sort" + "strings" +) + +// rawChildSlot is one child-slot property of a pluggable widget: a fixed +// property that holds widgets, as opposed to an object list that holds repeated +// items. +type rawChildSlot struct { + // Keyword is the MDL container keyword, derived from the property key the + // same way an object list's is (tagContentContainer -> tagcontentcontainer). + Keyword string + // PropertyKey is the stored key, kept so the emitted MDL can be matched back + // to the document when debugging. + PropertyKey string + Widgets []rawWidget +} + +// extractChildSlots reconstructs every child slot of a pluggable widget. +// +// # Why +// +// DESCRIBE reconstructed a Gallery's `content` and `filtersPlaceholder` by +// asking for those property keys BY NAME (extractGalleryWidgetsByPropertyKey), +// and nothing at all for any other widget. So a child slot on an arbitrary +// pluggable widget was dropped from the describe output — silently, at exit 0. +// +// That was invisible while slices 2-3 were unwritten, because MDL could not +// express such a slot in the first place. Once it could, describe -> edit -> +// exec started DELETING a widget's body: measured on a page mxcli authored +// itself, `tagcontentcontainer body { dynamictext t }` was stored correctly and +// came back as a bare head. +// +// This generalises the Gallery reader in the way the rest of this work +// generalises everything else: a child slot is any property whose Value holds a +// `Widgets` array, read off the document rather than looked up in a table of +// known widgets. A widget nobody has thought about round-trips for free. +// +// # Empty slots are skipped +// +// getBsonArrayElements strips the leading typed-array marker, so an EMPTY slot +// is length 0 here and length 1 in the raw BSON. Emitting empty slots would put +// a `slot { }` block on nearly every pluggable widget — correct, and unreadable. +func extractChildSlots(ctx *ExecContext, w map[string]any, entityContext string) []rawChildSlot { + obj, ok := w["Object"].(map[string]any) + if !ok { + return nil + } + keyMap := buildPropertyTypeKeyMap(w, true) + if len(keyMap) == 0 { + return nil + } + + var out []rawChildSlot + for _, prop := range getBsonArrayElements(obj["Properties"]) { + propMap, ok := prop.(map[string]any) + if !ok { + continue + } + value, ok := propMap["Value"].(map[string]any) + if !ok { + continue + } + widgetsArr := getBsonArrayElements(value["Widgets"]) + if len(widgetsArr) == 0 { + continue + } + key := keyMap[extractBinaryID(propMap["TypePointer"])] + if key == "" { + continue + } + + var widgets []rawWidget + for _, wgt := range widgetsArr { + wgtMap, ok := wgt.(map[string]any) + if !ok { + continue + } + widgets = append(widgets, parseRawWidget(ctx, wgtMap, entityContext)...) + } + if len(widgets) == 0 { + continue + } + + kw := strings.ToLower(deriveObjectListKeyword(key)) + if kw == "" { + kw = strings.ToLower(key) + } + out = append(out, rawChildSlot{Keyword: kw, PropertyKey: key, Widgets: widgets}) + } + + // Stable order: the BSON property order is not guaranteed to be meaningful, + // and an unstable describe makes diffs unusable (the same reasoning as the + // MDL-WIDGET07 ordering fix). + sort.Slice(out, func(i, j int) bool { return out[i].Keyword < out[j].Keyword }) + return out +} + +// outputChildSlots emits each reconstructed slot as an MDL container block. +// +// The slot NAME is synthesised. A child slot is a fixed property, not a named +// element — the document stores no name for it, and MDL requires one, so +// DESCRIBE WIDGET's usage example generates `slot1`, `slot2` for the same +// reason. Names are derived from the keyword so a re-describe is stable rather +// than renumbering on every run. +func outputChildSlots(ctx *ExecContext, slots []rawChildSlot, prefix string, indent int) { + for _, s := range slots { + fmt.Fprintf(ctx.Output, "%s%s %s {\n", prefix, s.Keyword, s.Keyword+"1") + for _, child := range s.Widgets { + outputWidgetMDLV3(ctx, child, indent+1) + } + fmt.Fprintf(ctx.Output, "%s}\n", prefix) + } +} diff --git a/mdl/executor/cmd_pages_describe_containers_test.go b/mdl/executor/cmd_pages_describe_containers_test.go new file mode 100644 index 0000000000..1cdcd5f884 --- /dev/null +++ b/mdl/executor/cmd_pages_describe_containers_test.go @@ -0,0 +1,147 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "testing" +) + +// fullWidgetValue builds a widget-value sub-document the way the encoder +// actually writes one: EVERY field present, almost all of them empty. +// +// That shape is the whole point. extractObjectListItem tests the fields in +// order and used to `continue` as soon as a KEY EXISTED, so the first branch +// (DataSource) consumed every property and the ones below it never ran. The +// item came back with no props and the caller dropped it, taking the entire +// object list out of the DESCRIBE output. +func fullWidgetValue(typePointer string, set map[string]any) map[string]any { + v := map[string]any{ + "$ID": "v-" + typePointer, + "$Type": "CustomWidgets$WidgetValue", + "Action": map[string]any{"$Type": "Forms$NoAction"}, + "AttributeRef": map[string]any{}, + "DataSource": map[string]any{}, + "EntityRef": map[string]any{}, + "Expression": "", + "PrimitiveValue": "", + "TextTemplate": map[string]any{}, + "TranslatableValue": map[string]any{}, + "Widgets": []any{int32(3)}, + "Objects": []any{int32(3)}, + } + for k, val := range set { + v[k] = val + } + return map[string]any{"TypePointer": typePointer, "Value": v} +} + +// An item whose sub-properties carry real values must produce them, even though +// every other field of each value is present-but-empty. +func TestExtractObjectListItem_EmptyFieldsDoNotSwallowTheProperty(t *testing.T) { + nested := map[string]string{ + "p1": "attributeName", + "p2": "attributeValueType", + } + item := map[string]any{ + "Properties": []any{ + int32(3), + fullWidgetValue("p1", map[string]any{"PrimitiveValue": "data-x"}), + fullWidgetValue("p2", map[string]any{"PrimitiveValue": "expression"}), + }, + } + + got := extractObjectListItem(&ExecContext{}, item, nested) + if len(got.Props) != 2 { + t.Fatalf("got %d props, want 2 — an empty DataSource/Action/AttributeRef must not "+ + "consume the property before PrimitiveValue is reached:\n%+v", len(got.Props), got.Props) + } + byKey := map[string]string{} + for _, p := range got.Props { + byKey[p.Key] = p.Value + } + if byKey["attributeName"] != "data-x" { + t.Errorf("AttributeName = %q, want %q (props: %+v)", byKey["attributeName"], "data-x", got.Props) + } + if byKey["attributeValueType"] != "expression" { + t.Errorf("AttributeValueType = %q, want %q", byKey["attributeValueType"], "expression") + } +} + +// The control for the branch ordering: a property that genuinely IS a +// datasource must still be taken by the datasource branch and must NOT fall +// through to PrimitiveValue. Without this, "stop consuming on empty" could be +// implemented by removing the branches altogether and the test above would +// still pass. +func TestExtractObjectListItem_RealDataSourceStillWins(t *testing.T) { + nested := map[string]string{"p1": "staticDataSource"} + item := map[string]any{ + "Properties": []any{ + int32(3), + fullWidgetValue("p1", map[string]any{ + "PrimitiveValue": "SHOULD NOT BE READ", + "DataSource": map[string]any{ + "$Type": "Forms$ListenTargetSource", + "Widget": "someWidget", + }, + }), + }, + } + + got := extractObjectListItem(&ExecContext{}, item, nested) + for _, p := range got.Props { + if p.Value == "SHOULD NOT BE READ" { + t.Errorf("a real datasource property fell through to PrimitiveValue: %+v", got.Props) + } + } +} + +// A child slot is any property whose Value holds widgets — read off the +// document rather than looked up by name, so a widget nobody has thought about +// round-trips too. DESCRIBE previously reconstructed only a Gallery's `content` +// and `filtersPlaceholder`, by asking for those keys BY NAME. +func TestExtractChildSlots(t *testing.T) { + w := map[string]any{ + "Type": map[string]any{ + "ObjectType": map[string]any{ + "PropertyTypes": []any{ + int32(3), + map[string]any{"$ID": "t1", "PropertyKey": "tagContentContainer"}, + map[string]any{"$ID": "t2", "PropertyKey": "emptySlot"}, + }, + }, + }, + "Object": map[string]any{ + "Properties": []any{ + int32(3), + map[string]any{ + "TypePointer": "t1", + "Value": map[string]any{"Widgets": []any{int32(3), + map[string]any{"$Type": "Forms$DynamicText", "Name": "t"}}}, + }, + // Empty: the marker only. Emitting these would put a `slot { }` + // block on nearly every pluggable widget. + map[string]any{ + "TypePointer": "t2", + "Value": map[string]any{"Widgets": []any{int32(3)}}, + }, + }, + }, + } + + got := extractChildSlots(&ExecContext{}, w, "") + if len(got) != 1 { + t.Fatalf("got %d slots, want 1 (the populated one only): %+v", len(got), got) + } + if got[0].Keyword != "tagcontentcontainer" { + t.Errorf("Keyword = %q, want %q", got[0].Keyword, "tagcontentcontainer") + } + if len(got[0].Widgets) == 0 { + t.Error("slot reconstructed with no widgets — the recursion into parseRawWidget did not run") + } +} + +func TestExtractChildSlots_NoContainers(t *testing.T) { + if got := extractChildSlots(&ExecContext{}, map[string]any{}, ""); len(got) != 0 { + t.Errorf("got %+v, want none", got) + } +} diff --git a/mdl/executor/cmd_pages_describe_objectlist.go b/mdl/executor/cmd_pages_describe_objectlist.go index 0eea7dc9f6..66940caced 100644 --- a/mdl/executor/cmd_pages_describe_objectlist.go +++ b/mdl/executor/cmd_pages_describe_objectlist.go @@ -149,10 +149,34 @@ func extractObjectListItem(ctx *ExecContext, itemObj map[string]any, nestedMap m } // Per-item datasource (e.g. chart series `staticDataSource`). - if ds, ok := value["DataSource"].(map[string]any); ok && ds != nil { + // These branches must consume the property only when they actually + // EXTRACTED something. A widget value carries every field it could + // possibly have — Action, AttributeRef, DataSource, Expression, + // TextTemplate, PrimitiveValue — most of them empty, so a branch that + // `continue`s merely because its key EXISTS swallows the property and + // the branches below it never run. + // + // The measured culprit is the ACTION branch below: `value["Action"]` is + // present on every sub-property as a Forms$NoAction, and it continued + // unconditionally. On an HTML Element authored by mxcli, that consumed + // all six sub-properties of an `attribute` item; the item ended with + // zero Props, the caller's `len(item.Props) > 0` filter dropped it, and + // the whole `attributes` list vanished from DESCRIBE — while the list + // itself resolved perfectly (probe: list="attributes" objects=1 + // nested=6). Isolated by reverting that one branch: object lists go + // 2 -> 0. + // + // DataSource and AttributeRef are the same latent shape and are guarded + // the same way. Neither is load-bearing for the measured case. + if ds, ok := value["DataSource"].(map[string]any); ok && len(ds) > 0 { if rds := parseCustomWidgetDataSource(ctx, ds); rds != nil && rds.Reference != "" { item.DataSource = rds } + // Consume it either way. A datasource that is PRESENT but could not + // be rendered must not fall through to the scalar branches below — + // they would describe it as its PrimitiveValue, which is a wrong + // answer rather than a missing one. `len(ds) > 0` is the whole + // change: an EMPTY map means the field is simply unset. continue } // Child widgets (an Accordion group's `content` slot). A Widgets-typed @@ -173,17 +197,20 @@ func extractObjectListItem(ctx *ExecContext, itemObj map[string]any, nestedMap m // which is how MDL addresses an item action slot — there is no alias // (#956). A NoAction is the unset default and is skipped, so an // untouched item describes exactly as it did before. - if action, ok := value["Action"].(map[string]any); ok && action != nil { + if action, ok := value["Action"].(map[string]any); ok && len(action) > 0 { if t := extractString(action["$Type"]); t != "Forms$NoAction" && t != "Pages$NoAction" { if mdl := renderClientActionMDL(ctx, action); mdl != "" { item.Props = append(item.Props, rawExplicitProp{ Key: objectListMDLKey(key), Value: mdl, IsRef: true}) } + // A real action, rendered or not, is not a scalar. + continue } - continue + // A NoAction is the unset default: fall through, since the property + // may carry its value in one of the fields below. } // Attribute binding (staticXAttribute, staticYAttribute, …). - if attrRef, ok := value["AttributeRef"].(map[string]any); ok && attrRef != nil { + if attrRef, ok := value["AttributeRef"].(map[string]any); ok && len(attrRef) > 0 { if a := extractString(attrRef["Attribute"]); a != "" { item.Props = append(item.Props, rawExplicitProp{Key: objectListMDLKey(key), Value: shortAttributeName(a), IsRef: true}) } @@ -223,13 +250,25 @@ func extractObjectListItem(ctx *ExecContext, itemObj map[string]any, nestedMap m return item } -// objectListMDLKey maps a widget schema sub-property key to the MDL property name -// the DESCRIBE output uses. MDL property names are case-insensitive, so the -// canonical PascalCase form (first letter upper) round-trips to the same schema -// key on re-exec (staticXAttribute→StaticXAttribute, staticName→StaticName). +// objectListMDLKey is the MDL property name DESCRIBE emits for a widget schema +// sub-property: the schema key, verbatim. +// +// It used to upper-case the first letter. That round-tripped — MDL property +// names are case-insensitive, so `StaticName` and `staticName` both resolve — +// but it made DESCRIBE PAGE the only surface using that spelling: +// +// mxcli widget describe htmlelement attributeName (from the .mpk) +// what you write in a page attributeName +// DESCRIBE PAGE, before AttributeName +// +// Three surfaces, two spellings, for no benefit. It was invisible while only +// chart series reached this code; slice 3 put it on every widget with an object +// list, which is what made it worth fixing. +// +// Emitting the key verbatim also keeps a real distinction visible that +// PascalCase erased: `DataSource:` stays capitalised because it is MDL's own +// keyword, not a widget schema key, so the two kinds of name no longer look +// alike. func objectListMDLKey(schemaKey string) string { - if schemaKey == "" { - return schemaKey - } - return strings.ToUpper(schemaKey[:1]) + schemaKey[1:] + return schemaKey } diff --git a/mdl/executor/cmd_pages_describe_objectlist_test.go b/mdl/executor/cmd_pages_describe_objectlist_test.go index 53768116a5..0a37b04cdb 100644 --- a/mdl/executor/cmd_pages_describe_objectlist_test.go +++ b/mdl/executor/cmd_pages_describe_objectlist_test.go @@ -4,13 +4,20 @@ package executor import "testing" +// The key is emitted VERBATIM. It used to be PascalCased, which round-tripped +// (MDL property names are case-insensitive) but made DESCRIBE PAGE the only +// surface spelling it that way — `describe widget` documents `staticName` and +// that is what a person writes. func TestObjectListMDLKey(t *testing.T) { cases := map[string]string{ - "staticXAttribute": "StaticXAttribute", - "staticName": "StaticName", - "dataSet": "DataSet", - "interpolation": "Interpolation", + "staticXAttribute": "staticXAttribute", + "staticName": "staticName", + "dataSet": "dataSet", + "interpolation": "interpolation", "": "", + // Already-capitalised keys are untouched too: verbatim means verbatim, + // not lower-cased. A widget is free to name a property `Foo`. + "Foo": "Foo", } for in, want := range cases { if got := objectListMDLKey(in); got != want { @@ -67,13 +74,13 @@ func TestExtractObjectListItem_ChartSeries(t *testing.T) { isRef bool }{p.Value, p.IsRef} } - if p, ok := got["DataSet"]; !ok || p.val != "static" || p.isRef { + if p, ok := got["dataSet"]; !ok || p.val != "static" || p.isRef { t.Errorf("DataSet prop = %+v, want {static,false}", p) } - if p, ok := got["StaticXAttribute"]; !ok || p.val != "Region" || !p.isRef { + if p, ok := got["staticXAttribute"]; !ok || p.val != "Region" || !p.isRef { t.Errorf("StaticXAttribute prop = %+v, want {Region,true}", p) } - if p, ok := got["StaticName"]; !ok || p.val != "Revenue" || p.isRef { + if p, ok := got["staticName"]; !ok || p.val != "Revenue" || p.isRef { t.Errorf("StaticName prop = %+v, want {Revenue,false}", p) } // The datasource sub-property must NOT also appear as a scalar prop. diff --git a/mdl/executor/cmd_pages_describe_omitted.go b/mdl/executor/cmd_pages_describe_omitted.go new file mode 100644 index 0000000000..ca4451eaf0 --- /dev/null +++ b/mdl/executor/cmd_pages_describe_omitted.go @@ -0,0 +1,135 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "fmt" + "io" + "sort" + "strings" +) + +// unreconstructedContainers names the container-shaped properties a pluggable +// widget carries in its stored BSON that DESCRIBE did not reproduce. +// +// # Why this exists +// +// Slices 2-3 (PROPOSAL_def_driven_widget_bodies.md) made a widget's object +// lists and child slots WRITABLE from MDL. DESCRIBE PAGE cannot yet read them +// back for an arbitrary pluggable widget: extractObjectLists reconstructs the +// chart-shaped lists it was built for and returns nothing for HTML Element's +// `attributes`, and nothing reconstructs a child slot at all. +// +// Measured on a page mxcli itself authored: +// +// written htmlelement frame { attribute a1 (…); tagcontentcontainer body { … } } +// stored BSON carries `attributes` with data-x AND `tagContentContainer` +// with its DynamicText — the write path is correct +// described htmlelement frame (tagName: 'div', …) <- body gone, silently +// +// So describe -> edit -> exec deleted a widget's body and said nothing. That is +// the #965 failure class (an annotation emptying the loop body it sits in), and +// it became reachable the moment the construct could be written. +// +// Reconstructing them is a separate piece of work. Until then the honest +// behaviour is the one slice 4's usage example already follows: say what was +// left out rather than pretend the output is complete. A visible gap is a +// nuisance; a silent one is data loss. +// +// # Detection is from the document, not from a list +// +// A container is a property whose Value holds an `Objects` array (object list) +// or a `Widgets` array (child slot). That is read off the stored BSON, so it +// covers widgets nobody has thought about — the same reason the rest of this +// work reads definitions rather than maintaining keyword tables. +// +// Anything already reconstructed is excluded, so a chart's series list — which +// DESCRIBE does emit — produces no note. +func unreconstructedContainers(w map[string]any, reconstructed []rawObjectList, slots []rawChildSlot) []string { + obj, ok := w["Object"].(map[string]any) + if !ok { + return nil + } + keyMap := buildPropertyTypeKeyMap(w, true) + if len(keyMap) == 0 { + return nil + } + + done := make(map[string]bool, len(reconstructed)+len(slots)) + for _, ol := range reconstructed { + if ol.Keyword != "" { + done[ol.Keyword] = true + } + } + // A slot DESCRIBE now reproduces is not a loss, so it must not be named as + // one — otherwise the note would fire on exactly the case that was fixed. + for _, cs := range slots { + if cs.Keyword != "" { + done[cs.Keyword] = true + } + } + + seen := make(map[string]bool) + var out []string + for _, prop := range getBsonArrayElements(obj["Properties"]) { + propMap, ok := prop.(map[string]any) + if !ok { + continue + } + value, ok := propMap["Value"].(map[string]any) + if !ok { + continue + } + // CHILD SLOTS ONLY, and only when they hold widgets. + // + // An object list is deliberately excluded even though it is equally + // unreconstructed, because it cannot be reported without crying wolf: a + // widget template ships DEFAULT entries in its lists, and they are + // structurally identical to a user's. Measured on the probe page — + // which wrote one `attribute` and no `event` at all — the stored + // document carries one object in each, so warning on object lists named + // `event` too. A note that fires on defaults is noise, and noise trains + // people to ignore the notes that matter. + // + // A WIDGET inside a slot has no such ambiguity: a template never puts + // one there, so its presence means someone did. That is the case where + // a describe -> exec round trip destroys real work, and it is the case + // worth interrupting for. + // + // (getBsonArrayElements strips the leading typed-array marker, so a + // length of 0 here really is empty — the raw BSON array is length 1.) + if len(getBsonArrayElements(value["Widgets"])) == 0 { + continue + } + key := keyMap[extractBinaryID(propMap["TypePointer"])] + if key == "" { + continue + } + // Report the MDL keyword the author would write, not the raw property + // key, so the note names something they can act on. + kw := strings.ToLower(deriveObjectListKeyword(key)) + if kw == "" { + kw = strings.ToLower(key) + } + if done[kw] || seen[kw] { + continue + } + seen[kw] = true + out = append(out, kw) + } + sort.Strings(out) + return out +} + +// writeOmittedContainerNote emits the gap as an MDL comment, so the output +// still parses and re-executes — it simply does not carry the body, and says +// so where the body would have been. +func writeOmittedContainerNote(out io.Writer, prefix string, omitted []string) { + if len(omitted) == 0 { + return + } + fmt.Fprintf(out, "%s-- NOT SHOWN: %s — this widget stores content DESCRIBE cannot yet\n", + prefix, strings.Join(omitted, ", ")) + fmt.Fprintf(out, "%s-- reproduce. Re-running this script would DROP it. Inspect the widget with\n", prefix) + fmt.Fprintf(out, "%s-- `mxcli widget describe ` and re-add the block by hand.\n", prefix) +} diff --git a/mdl/executor/cmd_pages_describe_output.go b/mdl/executor/cmd_pages_describe_output.go index 64d1d55d9c..613bd8f48a 100644 --- a/mdl/executor/cmd_pages_describe_output.go +++ b/mdl/executor/cmd_pages_describe_output.go @@ -661,7 +661,9 @@ func outputWidgetMDLV3(ctx *ExecContext, w rawWidget, indent int) { w.OnChange != "" || len(w.NamedActions) > 0) && w.WidgetID != "" { // Generic pluggable widget with explicit properties, object-list child // blocks (chart series/lines/scaleColors), and/or an onClick action. - header := fmt.Sprintf("pluggablewidget '%s' %s", w.WidgetID, mdlIdent(w.Name)) + // The widget's own MDL name where that round-trips, else the + // explicit id form. See pluggableWidgetHeader. + header := pluggableWidgetHeader(ctx.GetWidgetRegistry(), w.WidgetID, w.Name) props := []string{} if w.Caption != "" { props = append(props, fmt.Sprintf("Label: %s", mdlQuote(w.Caption))) @@ -689,8 +691,14 @@ func outputWidgetMDLV3(ctx *ExecContext, w rawWidget, indent int) { } props = appendNamedActionProps(props, w) props = appendAppearanceProps(props, w) - if len(w.ObjectLists) == 0 { + if len(w.ObjectLists) == 0 && len(w.ChildSlots) == 0 && len(w.OmittedContainers) == 0 { formatWidgetProps(ctx.Output, prefix, header, props, "\n") + } else if len(w.ObjectLists) == 0 { + // Child slots and/or a gap to name, but no object lists. + formatWidgetProps(ctx.Output, prefix, header, props, " {\n") + outputChildSlots(ctx, w.ChildSlots, prefix+" ", indent+1) + writeOmittedContainerNote(ctx.Output, prefix+" ", w.OmittedContainers) + fmt.Fprintf(ctx.Output, "%s}\n", prefix) } else { // Emit the widget with a body holding its object-list items. formatWidgetProps(ctx.Output, prefix, header, props, " {\n") @@ -725,6 +733,11 @@ func outputWidgetMDLV3(ctx *ExecContext, w rawWidget, indent int) { formatWidgetProps(ctx.Output, childPrefix, itemHeader, itemProps, "\n") } } + // A widget can carry both kinds of container — HTML Element has + // `attributes`/`events` AND `tagContentContainer` — so the slots + // belong in this branch too, not only the one above. + outputChildSlots(ctx, w.ChildSlots, childPrefix, indent+1) + writeOmittedContainerNote(ctx.Output, childPrefix, w.OmittedContainers) fmt.Fprintf(ctx.Output, "%s}\n", prefix) } } else { diff --git a/mdl/executor/cmd_pages_describe_parse.go b/mdl/executor/cmd_pages_describe_parse.go index 77e439ae4e..96a4911730 100644 --- a/mdl/executor/cmd_pages_describe_parse.go +++ b/mdl/executor/cmd_pages_describe_parse.go @@ -455,6 +455,8 @@ func parseRawWidget(ctx *ExecContext, w map[string]any, parentEntityContext ...s if !isKnownCustomWidgetType(widget.RenderMode) { widget.ExplicitProperties = extractExplicitProperties(ctx, w) widget.ObjectLists = extractObjectLists(ctx, w) + widget.ChildSlots = extractChildSlots(ctx, w, widget.EntityContext) + widget.OmittedContainers = unreconstructedContainers(w, widget.ObjectLists, widget.ChildSlots) // onClick action (ledger #67 — reported on CustomChart): read the client // action back with full parameter mappings so a describe round-trip // re-emits it (the finding's original widget goes through this path). diff --git a/mdl/executor/cmd_search.go b/mdl/executor/cmd_search.go index dec48316fa..de3e283dda 100644 --- a/mdl/executor/cmd_search.go +++ b/mdl/executor/cmd_search.go @@ -198,8 +198,14 @@ func execShowReferences(ctx *ExecContext, s *ast.ShowStmt) error { return err } - targetName := s.Name.String() - fmt.Fprintf(ctx.Output, "\nReferences to %s\n", targetName) + typed := s.Name.String() + fmt.Fprintf(ctx.Output, "\nReferences to %s\n", typed) + + // A widget's TargetName is stored SHOUTED (COMBOBOX) while MDL keywords are + // written in lower case, so an exact-only match answers the natural spelling + // with "(no references found)" — wrong, not missing. See resolveReferenceTarget. + targetName, loose := resolveReferenceTarget(ctx, typed) + reportResolvedTarget(ctx, typed, targetName, loose) // Find all references to this target query := ` @@ -209,7 +215,7 @@ func execShowReferences(ctx *ExecContext, s *ast.ShowStmt) error { ORDER by RefKind, SourceType, SourceName ` - result, err := ctx.Catalog.Query(strings.Replace(query, "?", "'"+targetName+"'", 1)) + result, err := ctx.Catalog.Query(strings.Replace(query, "?", "'"+escapeSQLString(targetName)+"'", 1)) if err != nil { return mdlerrors.NewBackend("query references", err) } @@ -236,8 +242,11 @@ func execShowImpact(ctx *ExecContext, s *ast.ShowStmt) error { return err } - targetName := s.Name.String() - fmt.Fprintf(ctx.Output, "\nImpact analysis for %s\n", targetName) + typed := s.Name.String() + fmt.Fprintf(ctx.Output, "\nImpact analysis for %s\n", typed) + + targetName, loose := resolveReferenceTarget(ctx, typed) + reportResolvedTarget(ctx, typed, targetName, loose) // Find all direct references to this target directQuery := ` @@ -247,7 +256,7 @@ func execShowImpact(ctx *ExecContext, s *ast.ShowStmt) error { ORDER by SourceType, SourceName ` - result, err := ctx.Catalog.Query(strings.Replace(directQuery, "?", "'"+targetName+"'", 1)) + result, err := ctx.Catalog.Query(strings.Replace(directQuery, "?", "'"+escapeSQLString(targetName)+"'", 1)) if err != nil { return mdlerrors.NewBackend("query impact", err) } diff --git a/mdl/executor/describe_widget_header.go b/mdl/executor/describe_widget_header.go new file mode 100644 index 0000000000..adcfae4779 --- /dev/null +++ b/mdl/executor/describe_widget_header.go @@ -0,0 +1,70 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "fmt" + "strings" +) + +// pluggableWidgetHeader returns the MDL header DESCRIBE PAGE should emit for a +// pluggable widget: its own MDL name where that round-trips, and the explicit +// `pluggablewidget ''` form otherwise. +// +// Item 6 of slice 2 in PROPOSAL_def_driven_widget_bodies.md. Until slices 2-3 +// the keyword form did not parse for most widgets, so DESCRIBE had no choice. +// Now that it does, emitting +// +// htmlelement frame (tagName: 'div') +// +// instead of +// +// pluggablewidget 'com.mendix.widget.web.htmlelement.HTMLElement' frame (…) +// +// makes describe → edit → exec produce the form a person would have written, +// which is the point of DESCRIBE being re-executable at all. +// +// # It falls back rather than guessing +// +// The bar is not "shorter" but "rebuilds the SAME widget". Two cases fail that +// and take the id form: +// +// 1. **The id resolves to no definition.** Without a project the registry +// holds only the embedded widgets, so most real widgets are unknown here — +// and an MDL name mxcli invented would not resolve on the way back in. +// 2. **Two definitions share an MDL name.** Then the name is ambiguous: the +// builder resolves it by `registry.Get(ToUpper(name))`, which can only +// return one of them, so emitting the name would silently retarget the +// widget. The id is unambiguous by construction. +// +// Case 2 is not hypothetical in principle — an MDL name is the last segment of +// a widget id, and two vendors can ship `…​.Slider`. It costs one map to rule +// out, and the alternative failure is a describe that rewrites a page onto a +// different widget. +func pluggableWidgetHeader(registry *WidgetRegistry, widgetID, name string) string { + idForm := fmt.Sprintf("pluggablewidget '%s' %s", widgetID, mdlIdent(name)) + if registry == nil || widgetID == "" { + return idForm + } + def, ok := registry.GetByWidgetID(widgetID) + if !ok || def == nil || def.MDLName == "" { + return idForm + } + // Does emitting this name rebuild the SAME widget? Ask, rather than infer. + // + // The builder resolves a bare name with registry.Get(ToUpper(name)), and the + // registry is keyed BY MDL NAME — so when two definitions claim one name the + // map keeps only the last, while GetByWidgetID keeps both. Get and + // GetByWidgetID then disagree, and emitting the name would rebuild the page + // onto the other widget. An MDL name is the last segment of a widget id, so + // two vendors shipping `….Slider` is not hypothetical. + // + // Counting definitions cannot see this (All() iterates the by-name map, so + // the loser is already gone). Round-tripping the name through the same + // lookup the builder uses can. + back, ok := registry.Get(def.MDLName) + if !ok || back == nil || back.WidgetID != widgetID { + return idForm + } + return fmt.Sprintf("%s %s", strings.ToLower(def.MDLName), mdlIdent(name)) +} diff --git a/mdl/executor/describe_widget_header_test.go b/mdl/executor/describe_widget_header_test.go new file mode 100644 index 0000000000..c71832af46 --- /dev/null +++ b/mdl/executor/describe_widget_header_test.go @@ -0,0 +1,157 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" +) + +// Item 6 of slice 2: DESCRIBE emits the widget's own MDL name now that the +// keyword form parses for every widget with a definition. +func TestPluggableWidgetHeader_UsesTheMDLName(t *testing.T) { + registry := LoadWidgetRegistry("") + if registry == nil { + t.Fatal("no registry — the embedded definitions must load, or this test proves nothing") + } + + def, ok := registry.Get("COMBOBOX") + if !ok || def == nil || def.WidgetID == "" { + t.Fatal("COMBOBOX is not an embedded definition; pick another widget for this test") + } + + got := pluggableWidgetHeader(registry, def.WidgetID, "cmb1") + want := "combobox cmb1" + if got != want { + t.Errorf("header = %q, want %q", got, want) + } +} + +// The fallbacks. Each must produce the id form, which always round-trips. +func TestPluggableWidgetHeader_FallsBackToTheIDForm(t *testing.T) { + registry := LoadWidgetRegistry("") + if registry == nil { + t.Fatal("no registry") + } + + cases := []struct { + name string + registry *WidgetRegistry + widgetID string + why string + }{ + {"unknown id", registry, "com.acme.NotInstalled.Thing", + "an id with no definition has no MDL name to emit; inventing one would not resolve on the way back in"}, + {"no registry", nil, "com.mendix.widget.web.combobox.Combobox", + "without definitions there is nothing to resolve against"}, + {"empty id", registry, "", + "nothing to look up"}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + got := pluggableWidgetHeader(tc.registry, tc.widgetID, "w1") + if !strings.HasPrefix(got, "pluggablewidget '") { + t.Errorf("header = %q, want the pluggablewidget id form — %s", got, tc.why) + } + }) + } +} + +// The ambiguity guard. Two definitions sharing an MDL name make the name +// unusable: the builder resolves it with registry.Get, which can return only +// one of them, so emitting the name could silently retarget the widget onto a +// different one. An MDL name is the last segment of a widget id, so two vendors +// shipping `….Slider` is not hypothetical. +func TestPluggableWidgetHeader_AmbiguousNameFallsBack(t *testing.T) { + registry := LoadWidgetRegistry("") + if registry == nil { + t.Fatal("no registry") + } + def, ok := registry.Get("COMBOBOX") + if !ok { + t.Fatal("COMBOBOX missing") + } + + // Control first: unique today, so the name IS emitted. + if got := pluggableWidgetHeader(registry, def.WidgetID, "w1"); !strings.HasPrefix(got, "combobox ") { + t.Fatalf("control: header = %q, want the name form before a collision is introduced", got) + } + + // Introduce a second definition claiming the same MDL name. This is what a + // colliding install looks like in the registry: byWidgetID keeps both, + // byMDLName keeps only the last, so the two lookups disagree. + other := &WidgetDefinition{WidgetID: "com.acme.other.Combobox", MDLName: "combobox"} + registry.byWidgetID[other.WidgetID] = other + registry.byMDLName["COMBOBOX"] = other + if got := pluggableWidgetHeader(registry, def.WidgetID, "w1"); !strings.HasPrefix(got, "pluggablewidget '") { + t.Errorf("header = %q, want the id form once two definitions claim the MDL name — "+ + "emitting the ambiguous name could rebuild the page onto the other widget", got) + } +} + +// unreconstructedContainers turns silent data loss into a visible gap. +// +// DESCRIBE cannot yet read a child slot back for an arbitrary pluggable widget, +// so describe -> exec deleted a widget's body and said nothing. Measured on a +// page mxcli authored itself: the stored BSON carried `tagContentContainer` +// with a DynamicText, and the describe output was a bare head. +func TestUnreconstructedContainers(t *testing.T) { + // A minimal pluggable widget document: two properties, one child slot with + // a widget in it and one empty. Arrays carry the leading typed-array marker, + // which is why an EMPTY container is length 1 and not length 0 — getting + // that wrong reports every widget as lossy. + widget := map[string]any{ + "Type": map[string]any{ + "PropertyTypes": []any{ + int32(3), + map[string]any{"$ID": "t1", "PropertyKey": "tagContentContainer"}, + map[string]any{"$ID": "t2", "PropertyKey": "emptySlot"}, + }, + }, + "Object": map[string]any{ + "Properties": []any{ + int32(3), + map[string]any{ + "TypePointer": "t1", + "Value": map[string]any{ + "Widgets": []any{int32(3), map[string]any{"$Type": "Forms$DynamicText"}}, + }, + }, + map[string]any{ + "TypePointer": "t2", + "Value": map[string]any{"Widgets": []any{int32(3)}}, + }, + }, + }, + } + + got := unreconstructedContainers(widget, nil, nil) + + // The populated slot must be reported. + var sawPopulated, sawEmpty bool + for _, g := range got { + if strings.Contains(g, "tagcontent") { + sawPopulated = true + } + if strings.Contains(g, "empty") { + sawEmpty = true + } + } + if !sawPopulated { + t.Errorf("a child slot holding a widget was not reported; got %v — "+ + "this is the case where describe -> exec destroys real work", got) + } + // The control: an EMPTY slot must not be reported, or the note fires on + // every widget and stops being read. + if sawEmpty { + t.Errorf("an empty child slot was reported as lost; got %v — an array of length 1 "+ + "is the typed-array marker alone, i.e. no content", got) + } +} + +// Nothing to report on a document with no containers at all. +func TestUnreconstructedContainers_Empty(t *testing.T) { + if got := unreconstructedContainers(map[string]any{}, nil, nil); len(got) != 0 { + t.Errorf("got %v, want none", got) + } +} diff --git a/mdl/executor/exec_context.go b/mdl/executor/exec_context.go index dda34b5bf1..b62b36906e 100644 --- a/mdl/executor/exec_context.go +++ b/mdl/executor/exec_context.go @@ -62,6 +62,12 @@ type ExecContext struct { // ThemeRegistry holds cached theme design property definitions (lazy init). ThemeRegistry *ThemeRegistry + // widgetRegistry caches the widget definitions for this session. Loaded + // once via GetWidgetRegistry, because DESCRIBE PAGE consults it per widget + // and LoadWidgetRegistry reads .def.json files off disk. + widgetRegistry *WidgetRegistry + widgetRegistryLoaded bool + // Settings holds session-scoped key-value settings (SET command). Settings map[string]any @@ -285,3 +291,28 @@ func (ctx *ExecContext) ensureSqlMgr() *sqllib.Manager { } return ctx.SqlMgr } + +// GetWidgetRegistry returns the session's widget registry, loading it on first +// use. It is cached because DESCRIBE PAGE asks per widget and the load reads +// every .def.json in the project — file I/O in a per-widget path is exactly +// what the review checklist warns against. +// +// A nil result is normal and means "no definitions available": with no project +// only the embedded widgets exist, and callers must degrade rather than treat +// it as an error. +func (ctx *ExecContext) GetWidgetRegistry() *WidgetRegistry { + if ctx == nil { + return nil + } + if ctx.widgetRegistryLoaded { + return ctx.widgetRegistry + } + ctx.widgetRegistryLoaded = true + // LoadWidgetRegistry wants the .mpr PATH, not its directory — + // LoadUserDefinitions takes filepath.Dir of it internally. Passing the + // directory looks one level too high and silently finds no definitions, + // which shows up as DESCRIBE falling back to the widget-id form for every + // widget rather than as an error. + ctx.widgetRegistry = LoadWidgetRegistry(ctx.MprPath) + return ctx.widgetRegistry +} diff --git a/mdl/executor/executor_query.go b/mdl/executor/executor_query.go index e8cbfc72ae..672285bc1d 100644 --- a/mdl/executor/executor_query.go +++ b/mdl/executor/executor_query.go @@ -169,7 +169,13 @@ func execShow(ctx *ExecContext, s *ast.ShowStmt) error { } func execDescribe(ctx *ExecContext, s *ast.DescribeStmt) error { - if !ctx.Connected() && s.ObjectType != ast.DescribeFragment { + // DESCRIBE WIDGET joins DESCRIBE FRAGMENT in not needing a project: a widget + // definition is not a document in the model, and mxcli's embedded set can + // answer for a built-in widget with nothing open. With a project the answer + // is better — the installed .mpk is version-accurate and covers Marketplace + // widgets — but requiring one would make the statement useless for exactly + // the "what can I write here?" question it exists to answer. + if !ctx.Connected() && s.ObjectType != ast.DescribeFragment && s.ObjectType != ast.DescribeWidget { return mdlerrors.NewNotConnected() } @@ -243,6 +249,8 @@ func execDescribe(ctx *ExecContext, s *ast.DescribeStmt) error { return describeExternalEntity(ctx, s.Name) case ast.DescribeNavigation: return describeNavigation(ctx, s.Name) + case ast.DescribeWidget: + return describeWidgetStmt(ctx, s.Name.Name) case ast.DescribeWorkflow: return describeWorkflow(ctx, s.Name) case ast.DescribeBusinessEventService: @@ -340,6 +348,8 @@ func describeObjectTypeLabel(t ast.DescribeObjectType) string { return "externalentity" case ast.DescribeNavigation: return "navigation" + case ast.DescribeWidget: + return "widget" case ast.DescribeWorkflow: return "workflow" case ast.DescribeBusinessEventService: diff --git a/mdl/executor/oql_sum_unknown_test.go b/mdl/executor/oql_sum_unknown_test.go new file mode 100644 index 0000000000..a8cdad767e --- /dev/null +++ b/mdl/executor/oql_sum_unknown_test.go @@ -0,0 +1,93 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "errors" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/backend/mock" + "github.com/mendixlabs/mxcli/sdk/domainmodel" +) + +// unresolvableCtx is the real shape of the bug: a project the checker can read, +// in which the view's SOURCE entity does not exist — because the script creates +// it in the same run, and `check --references` skips script-created objects. +func unresolvableCtx() *ExecContext { + return &ExecContext{Backend: &mock.MockBackend{ + ListDomainModelsFunc: func() ([]*domainmodel.DomainModel, error) { + return nil, errors.New("no domain models") + }, + }} +} + +// SUM over an argument whose type could not be resolved must be Unknown, not +// Decimal — the same rule inferTypeStatic already states in its own comment and +// the project-aware path contradicted. +// +// # The failure this caused +// +// A view entity whose source entity is created BY THE SAME SCRIPT cannot have +// its attribute types resolved (check skips references to script-created +// objects), so `sum(s.Units)` resolved to Unknown and fell through to Decimal. +// mxcli then reported +// +// attribute 'Units': declared as Integer but OQL expression 'sum(s.Units)' +// returns Decimal. Fix: change to 'Units: Decimal' +// +// Measured against mxbuild 11.6.6, that hint INVERTS the truth. Two views over +// the same `sum(s.Units)` where Units is an Integer attribute: +// +// declared Integer -> 0 errors (what mxcli flagged) +// declared Decimal -> CE6770 (what mxcli told the user to write) +// +// The control that the check is not simply inert: a view declaring a String +// column over `sum(s.Amount)` DOES fail CE6770 on the same mxbuild, so mxbuild +// really does validate these and the 0-errors result above means something. +// +// So the diagnostic did not merely cry wolf, it walked a working project into a +// broken one. An unresolvable type has to stay unresolved. +func TestInferAggregateType_SumOfUnknownIsUnknown(t *testing.T) { + ctx := unresolvableCtx() + aliasMap := map[string]string{"s": "ChartExamples.Sales"} + + got := inferAggregateType(ctx, "sum(s.Units)", &OQLColumnInfo{}, aliasMap) + if got.Kind != ast.TypeUnknown { + t.Errorf("sum() over an unresolvable argument inferred %s, want Unknown — "+ + "guessing Decimal makes the checker demand a declaration mxbuild rejects (CE6770)", + formatDataTypeForError(got)) + } +} + +// The control for the fix: SUM must still PROPAGATE a type it can resolve, or +// "return Unknown" degenerates into "never check sum() at all" and the rule +// stops detecting the real CE6770 it was written for. +func TestInferAggregateType_SumPropagatesAKnownType(t *testing.T) { + ctx := unresolvableCtx() + // A literal resolves without a project, so this exercises the propagation + // branch rather than the entity lookup. + if got := inferAggregateType(ctx, "sum(1.5)", &OQLColumnInfo{}, nil); got.Kind != ast.TypeDecimal { + t.Errorf("sum(1.5) inferred %s, want Decimal — a resolvable argument type must "+ + "still propagate", formatDataTypeForError(got)) + } + if got := inferAggregateType(ctx, "sum(2)", &OQLColumnInfo{}, nil); got.Kind != ast.TypeInteger { + t.Errorf("sum(2) inferred %s, want Integer", formatDataTypeForError(got)) + } +} + +// The second control: the neighbouring aggregates keep their own rules. COUNT is +// Integer whatever its argument, AVG is Decimal whatever its argument — a fix +// that made every aggregate Unknown would pass the first test and silently turn +// the whole rule off. +func TestInferAggregateType_NeighbouringAggregatesUnchanged(t *testing.T) { + ctx := unresolvableCtx() + aliasMap := map[string]string{"s": "ChartExamples.Sales"} + + if got := inferAggregateType(ctx, "count(s.ID)", &OQLColumnInfo{}, aliasMap); got.Kind != ast.TypeInteger { + t.Errorf("count() inferred %s, want Integer", formatDataTypeForError(got)) + } + if got := inferAggregateType(ctx, "avg(s.Units)", &OQLColumnInfo{}, aliasMap); got.Kind != ast.TypeDecimal { + t.Errorf("avg() inferred %s, want Decimal", formatDataTypeForError(got)) + } +} diff --git a/mdl/executor/oql_type_inference.go b/mdl/executor/oql_type_inference.go index e13758b8c0..b90e57bc7c 100644 --- a/mdl/executor/oql_type_inference.go +++ b/mdl/executor/oql_type_inference.go @@ -694,18 +694,32 @@ func inferAggregateType(ctx *ExecContext, expr string, col *OQLColumnInfo, alias return ast.DataType{Kind: ast.TypeInteger} } - // SUM(expression) → preserves input type (Integer→Integer, else Decimal) + // SUM(expression) → preserves the input type, and stays UNKNOWN when that + // type could not be resolved. + // + // Falling back to Decimal looks harmless and is not: the argument is + // unresolvable exactly when the source entity is created by the same script + // (check skips references to script-created objects), which is the common + // shape for a view entity. Measured against mxbuild 11.6.6 on `sum(s.Units)` + // where Units is an Integer attribute: + // + // declared Integer -> 0 errors <- what mxcli flagged + // declared Decimal -> CE6770 <- what mxcli's hint told the user to write + // + // So the guess inverted the truth and the "Fix:" would break a working + // project. inferTypeStatic's SUM branch already says this in its own comment; + // this path disagreed with it. if strings.HasPrefix(upperExpr, "SUM(") { col.IsAggregate = true col.AggregateFunc = "sum" innerArg := extractFunctionArg(expr) if innerArg != "" { innerType := inferTypeFromExpression(ctx, innerArg, &OQLColumnInfo{}, aliasMap) - if innerType.Kind == ast.TypeInteger || innerType.Kind == ast.TypeLong { + if innerType.Kind != ast.TypeUnknown { return innerType } } - return ast.DataType{Kind: ast.TypeDecimal} + return ast.DataType{Kind: ast.TypeUnknown} } // AVG(expression) → always Decimal diff --git a/mdl/executor/reference_target.go b/mdl/executor/reference_target.go new file mode 100644 index 0000000000..a458c3f8f0 --- /dev/null +++ b/mdl/executor/reference_target.go @@ -0,0 +1,68 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "fmt" + "strings" +) + +// resolveReferenceTarget returns the spelling of `name` that CATALOG.REFS +// actually stores, and whether it differs from what the user typed. +// +// SHOW REFERENCES TO and SHOW IMPACT OF match TargetName exactly. Every target +// used to be a module-qualified name, which a user copies verbatim from +// `show entities`, so exact matching was right and nothing needed this. +// +// The `widget` edge (slice 5 of PROPOSAL_def_driven_widget_bodies.md) breaks +// that assumption: its TargetName is the widget's MDL name, which is stored +// SHOUTED (COMBOBOX) because that is how widget_definitions_data holds it, +// while MDL keywords are case-insensitive and every example writes them in +// lower case. So the natural +// +// show references to combobox +// +// found nothing — and reported "(no references found)", which is a WRONG +// answer rather than a missing one. That is the failure mode worth spending a +// lookup to avoid: a user cannot tell it from a widget genuinely being unused. +// +// The fallback is deliberately second, never first. An exact match is returned +// untouched, so no existing answer can change; only a query that would have +// returned nothing gets a second chance. Callers report the resolved spelling +// so the user can see which name was actually matched. +func resolveReferenceTarget(ctx *ExecContext, name string) (resolved string, matchedLoosely bool) { + if ctx == nil || ctx.Catalog == nil || name == "" { + return name, false + } + + exact, err := ctx.Catalog.Query(fmt.Sprintf( + `SELECT 1 FROM refs WHERE TargetName = '%s' LIMIT 1`, escapeSQLString(name))) + if err == nil && exact.Count > 0 { + return name, false + } + + // Nothing under that spelling. Try case-insensitively, and only accept the + // answer when it is unambiguous — two targets differing only in case are a + // question this cannot answer for the user, so leave the exact (empty) + // result rather than guessing at one of them. + loose, err := ctx.Catalog.Query(fmt.Sprintf( + `SELECT DISTINCT TargetName FROM refs WHERE lower(TargetName) = lower('%s')`, + escapeSQLString(name))) + if err != nil || loose.Count != 1 || len(loose.Rows) != 1 || len(loose.Rows[0]) == 0 { + return name, false + } + match, ok := loose.Rows[0][0].(string) + if !ok || match == "" || match == name { + return name, false + } + return match, true +} + +// reportResolvedTarget tells the user which stored spelling was matched, when +// it is not the one they typed. Silent on an exact match. +func reportResolvedTarget(ctx *ExecContext, typed, resolved string, matchedLoosely bool) { + if !matchedLoosely || ctx == nil || ctx.Output == nil { + return + } + fmt.Fprintf(ctx.Output, "(matched %s)\n", strings.TrimSpace(resolved)) +} diff --git a/mdl/executor/reference_target_test.go b/mdl/executor/reference_target_test.go new file mode 100644 index 0000000000..657e48ede9 --- /dev/null +++ b/mdl/executor/reference_target_test.go @@ -0,0 +1,116 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "bytes" + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/catalog" +) + +// seedRefTargets builds an in-memory catalog holding one widget edge (stored +// SHOUTED, as widget_definitions_data holds MDL names) and one ordinary +// module-qualified target, so the exact path and the fallback are exercised +// against the same catalog. +func seedRefTargets(t *testing.T, targets ...string) *ExecContext { + t.Helper() + cat, err := catalog.New() + if err != nil { + t.Fatalf("catalog.New: %v", err) + } + t.Cleanup(func() { cat.Close() }) + for _, tgt := range targets { + if _, err := cat.CatalogDB().Exec( + `INSERT INTO refs (SourceType, SourceId, SourceName, TargetType, TargetId, TargetName, RefKind, ProjectId, SnapshotId) + VALUES ('PAGE', '', 'Sales.OrderList', 'WIDGET', '', ?, 'widget', 'p', 's')`, tgt); err != nil { + t.Fatalf("seed %q: %v", tgt, err) + } + } + return &ExecContext{Catalog: cat, Output: &bytes.Buffer{}} +} + +func TestResolveReferenceTarget(t *testing.T) { + ctx := seedRefTargets(t, "COMBOBOX", "Sales.Order") + + cases := []struct { + name string + typed string + want string + wantLoose bool + reasonWhen string + }{ + {"exact widget name", "COMBOBOX", "COMBOBOX", false, + "an exact match must be returned untouched, so no existing answer changes"}, + {"lower-case widget name", "combobox", "COMBOBOX", true, + "the spelling every MDL example uses must find the stored SHOUTED name"}, + {"mixed-case widget name", "ComboBox", "COMBOBOX", true, + "MDL keywords are case-insensitive, so any casing must resolve"}, + {"exact qualified name", "Sales.Order", "Sales.Order", false, + "an ordinary target keeps exact-match behaviour"}, + {"wrong-case qualified name", "sales.order", "Sales.Order", true, + "the fallback is not widget-specific; it can only turn an empty answer into a right one"}, + {"genuine typo", "Sales.Ordr", "Sales.Ordr", false, + "a name that matches nothing in any casing must not be rewritten to something else"}, + {"unknown widget", "gallery", "gallery", false, + "a widget with no edges must stay unresolved rather than borrow another's name"}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + got, loose := resolveReferenceTarget(ctx, tc.typed) + if got != tc.want || loose != tc.wantLoose { + t.Errorf("resolveReferenceTarget(%q) = (%q, %v), want (%q, %v) — %s", + tc.typed, got, loose, tc.want, tc.wantLoose, tc.reasonWhen) + } + }) + } +} + +// Two targets differing only in case are a question this cannot answer, so it +// must decline rather than pick one. Without the Count != 1 guard it would +// silently return whichever the database ordered first. +func TestResolveReferenceTarget_AmbiguousCaseDeclines(t *testing.T) { + ctx := seedRefTargets(t, "Sales.Order", "sales.ORDER") + + got, loose := resolveReferenceTarget(ctx, "SALES.order") + if loose || got != "SALES.order" { + t.Errorf("resolveReferenceTarget = (%q, %v), want (%q, false) — two case-variant targets must not be guessed between", + got, loose, "SALES.order") + } + + // Control: with only one variant present, the same input DOES resolve — so + // the decline above is the ambiguity guard, not a broken lookup. + single := seedRefTargets(t, "Sales.Order") + if got, loose := resolveReferenceTarget(single, "SALES.order"); !loose || got != "Sales.Order" { + t.Errorf("control: resolveReferenceTarget = (%q, %v), want (\"Sales.Order\", true)", got, loose) + } +} + +// The resolved spelling is reported, because a user who typed `combobox` and +// got results under `COMBOBOX` should be able to see which name matched. +func TestReportResolvedTarget(t *testing.T) { + var buf bytes.Buffer + ctx := &ExecContext{Output: &buf} + + reportResolvedTarget(ctx, "COMBOBOX", "COMBOBOX", false) + if buf.Len() != 0 { + t.Errorf("exact match printed %q, want nothing", buf.String()) + } + + reportResolvedTarget(ctx, "combobox", "COMBOBOX", true) + if got := buf.String(); !strings.Contains(got, "COMBOBOX") { + t.Errorf("loose match printed %q, want it to name COMBOBOX", got) + } +} + +// A nil catalog must not panic — `show references` reaches here only after +// ensureCatalog, but the helper is small enough to be called elsewhere. +func TestResolveReferenceTarget_NoCatalog(t *testing.T) { + if got, loose := resolveReferenceTarget(&ExecContext{}, "combobox"); got != "combobox" || loose { + t.Errorf("no catalog = (%q, %v), want (\"combobox\", false)", got, loose) + } + if got, loose := resolveReferenceTarget(nil, "combobox"); got != "combobox" || loose { + t.Errorf("nil ctx = (%q, %v), want (\"combobox\", false)", got, loose) + } +} diff --git a/mdl/executor/validate.go b/mdl/executor/validate.go index 85e03184df..37091aff45 100644 --- a/mdl/executor/validate.go +++ b/mdl/executor/validate.go @@ -549,7 +549,7 @@ func validateWithContext(ctx *ExecContext, stmt ast.Statement, sc *scriptContext s.Name.String(), strings.Join(refErrors, "\n - ")) } // Validate page context tree (parameter/selection/attribute bindings) - if ctxErrors := validatePageContextTree(s.Parameters, s.Widgets); len(ctxErrors) > 0 { + if ctxErrors := validatePageContextTree(ctx, s.Parameters, s.Widgets); len(ctxErrors) > 0 { return mdlerrors.NewValidationf("page '%s' has context errors:\n - %s", s.Name.String(), strings.Join(ctxErrors, "\n - ")) } @@ -576,7 +576,7 @@ func validateWithContext(ctx *ExecContext, stmt ast.Statement, sc *scriptContext s.Name.String(), strings.Join(argErrors, "\n - ")) } // Validate snippet context tree (parameter/selection/attribute bindings) - if ctxErrors := validatePageContextTree(s.Parameters, s.Widgets); len(ctxErrors) > 0 { + if ctxErrors := validatePageContextTree(ctx, s.Parameters, s.Widgets); len(ctxErrors) > 0 { return mdlerrors.NewValidationf("snippet '%s' has context errors:\n - %s", s.Name.String(), strings.Join(ctxErrors, "\n - ")) } diff --git a/mdl/executor/validate_dup_widget_test.go b/mdl/executor/validate_dup_widget_test.go index 1682a55164..164117aed6 100644 --- a/mdl/executor/validate_dup_widget_test.go +++ b/mdl/executor/validate_dup_widget_test.go @@ -17,7 +17,7 @@ func TestCheckDuplicateWidgetNames_Unit(t *testing.T) { {Type: "listview", Name: "ruTop"}, }}, } - errs := checkDuplicateWidgetNames(widgets) + errs := checkDuplicateWidgetNames(widgets, nil) if len(errs) != 1 || !strings.Contains(errs[0], "ruTop") { t.Fatalf("expected one duplicate error for ruTop, got %v", errs) } @@ -40,7 +40,7 @@ func TestCheckDuplicateWidgetNames_Parsed(t *testing.T) { if !ok { t.Fatalf("statement 0 = %T, want *ast.CreatePageStmtV3", prog.Statements[0]) } - dup := checkDuplicateWidgetNames(pg.Widgets) + dup := checkDuplicateWidgetNames(pg.Widgets, nil) if len(dup) != 1 || !strings.Contains(dup[0], "ruTop") { t.Fatalf("expected duplicate ruTop error from parsed page, got %v (widget names may not be populated for containers)", dup) } diff --git a/mdl/executor/validate_page_context.go b/mdl/executor/validate_page_context.go index 34a9010cff..46ec9969cb 100644 --- a/mdl/executor/validate_page_context.go +++ b/mdl/executor/validate_page_context.go @@ -16,7 +16,7 @@ import ( // // This runs at check time (no MPR needed) and catches issues that would otherwise // only surface as CE errors in Studio Pro. -func validatePageContextTree(params []ast.PageParameter, widgets []*ast.WidgetV3) []string { +func validatePageContextTree(ctx *ExecContext, params []ast.PageParameter, widgets []*ast.WidgetV3) []string { // Build param name set paramNames := make(map[string]bool, len(params)) for _, p := range params { @@ -29,11 +29,21 @@ func validatePageContextTree(params []ast.PageParameter, widgets []*ast.WidgetV3 // Walk the widget tree with context tracking var errors []string - errors = append(errors, checkDuplicateWidgetNames(widgets)...) + errors = append(errors, checkDuplicateWidgetNames(widgets, pageContextWidgetRegistry(ctx))...) walkWidgetsWithContext(widgets, paramNames, widgetNames, false, &errors) return errors } +// pageContextWidgetRegistry returns the widget registry for the run, or nil. +// The duplicate-name rule needs it to tell an object-list item from a widget; +// every other check here is registry-free, and a nil ctx (unit tests) is fine. +func pageContextWidgetRegistry(ctx *ExecContext) *WidgetRegistry { + if ctx == nil { + return nil + } + return ctx.GetWidgetRegistry() +} + // widgetKindsWithoutStoredNames are the widget kinds Mendix stores with no Name // at all, so MDL's name for one is mxcli's own — derived at describe time to give // ALTER PAGE something to address. @@ -51,29 +61,70 @@ var widgetKindsWithoutStoredNames = map[string]bool{ "column": true, } +// objectListContainerKinds returns the lowercased MDL container keywords the +// widget's definition declares as object lists (a BarChart's `series`, an +// Accordion's `group`). Children with one of those types are ITEMS, not widgets. +// +// Returns nil for anything that does not resolve — a built-in widget, an unknown +// name, or no registry at all — so the caller's behaviour is unchanged there. +func objectListContainerKinds(registry *WidgetRegistry, w *ast.WidgetV3) map[string]bool { + if registry == nil || w == nil { + return nil + } + def := lookupWidgetDef(w, registry) + if def == nil { + return nil + } + out := make(map[string]bool, len(def.ObjectLists)) + for _, ol := range def.ObjectLists { + if ol.MDLContainer != "" { + out[strings.ToLower(ol.MDLContainer)] = true + } + } + // WidgetMode carries no ObjectLists — object lists are declared once on the + // definition — so there is nothing mode-scoped to add here. + return out +} + // checkDuplicateWidgetNames flags any widget name that appears more than once on a // page. Mendix requires widget names to be unique per page and rejects duplicates // with CE0495 "Duplicate name" — which mxcli check otherwise passed (FINDINGS #15). // Each duplicate name is reported once, in first-seen order. // // Widget kinds Mendix stores without a name are skipped: see -// widgetKindsWithoutStoredNames. -func checkDuplicateWidgetNames(widgets []*ast.WidgetV3) []string { +// widgetKindsWithoutStoredNames. So are OBJECT-LIST ITEMS — a chart `series`, a +// gallery `customitem` — for the same reason and with the same evidence: a page +// authored with `series sRegion (…)` comes back from DESCRIBE as +// `series series1 (…)`, because the stored WidgetObject carries no name and +// DESCRIBE has to synthesise one. Measured on mxbuild 11.6.6, three charts on +// one page each holding a `series s` is 0 errors; mxcli called it CE0495 and, +// since a reference error fails the run, refused to execute the script. +// +// registry may be nil (check runs with no project in CI). Then no parent +// resolves, itemKinds is empty everywhere, and the rule behaves as it did +// before — a false positive is preferable to a silent one, and the enumerated +// widget types are unaffected either way. +func checkDuplicateWidgetNames(widgets []*ast.WidgetV3, registry *WidgetRegistry) []string { counts := make(map[string]int) var order []string - var walk func(ws []*ast.WidgetV3) - walk = func(ws []*ast.WidgetV3) { + // itemKinds are the container keywords the ENCLOSING widget declares as + // object lists. Read from that widget's definition rather than from a list + // of keywords: the containers are def-driven, so a table here would be the + // second list this proposal exists to remove. + var walk func(ws []*ast.WidgetV3, itemKinds map[string]bool) + walk = func(ws []*ast.WidgetV3, itemKinds map[string]bool) { for _, w := range ws { - if w.Name != "" && !widgetKindsWithoutStoredNames[strings.ToLower(w.Type)] { + kind := strings.ToLower(w.Type) + if w.Name != "" && !widgetKindsWithoutStoredNames[kind] && !itemKinds[kind] { if counts[w.Name] == 0 { order = append(order, w.Name) } counts[w.Name]++ } - walk(w.Children) + walk(w.Children, objectListContainerKinds(registry, w)) } } - walk(widgets) + walk(widgets, nil) var errors []string for _, name := range order { diff --git a/mdl/executor/validate_page_context_objectlist_test.go b/mdl/executor/validate_page_context_objectlist_test.go new file mode 100644 index 0000000000..bae7d2688e --- /dev/null +++ b/mdl/executor/validate_page_context_objectlist_test.go @@ -0,0 +1,117 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" +) + +// chartRegistry is a registry holding one definition that declares a `series` +// object list — the shape a real chart's .def.json has. Built in memory because +// NO embedded widget definition declares an object list (measured: 0 of them), +// so a test using LoadWidgetRegistry("") would skip everywhere, and the .def.json +// cache a project builds from its .mpk files is gitignored, so a test reading one +// would skip in CI. +func chartRegistry() *WidgetRegistry { + def := &WidgetDefinition{ + WidgetID: "com.mendix.widget.web.barchart.BarChart", + MDLName: "barchart", + ObjectLists: []ObjectListMapping{ + {PropertyKey: "series", MDLContainer: "SERIES"}, + }, + } + return &WidgetRegistry{ + byMDLName: map[string]*WidgetDefinition{"BARCHART": def}, + byWidgetID: map[string]*WidgetDefinition{def.WidgetID: def}, + } +} + +// dashboardWithThreeSeries mirrors the shape 34-chart-widget-examples.mdl uses: +// three separate charts on one page, each with a series the author called `s`. +func dashboardWithThreeSeries() []*ast.WidgetV3 { + chart := func(name, container, itemName string) *ast.WidgetV3 { + return &ast.WidgetV3{ + Type: "pluggablewidget", Name: name, + Properties: map[string]any{"WidgetType": "com.mendix.widget.web.barchart.BarChart"}, + Children: []*ast.WidgetV3{{Type: container, Name: itemName}}, + } + } + return []*ast.WidgetV3{ + chart("dashBar", "series", "s"), + chart("dashColumn", "series", "s"), + chart("dashArea", "series", "s"), + } +} + +// An object-list ITEM's name is mxcli's own, not the model's — the same reason +// widgetKindsWithoutStoredNames already excludes rows and columns. +// +// Proof the model does not hold it: a page authored with `series sRegion (…)` +// comes back from DESCRIBE as `series series1 (…)`. DESCRIBE synthesises the +// name because the stored WidgetObject has none, so no two of them can collide +// under CE0495. +// +// Proof mxbuild agrees: the page above, exec'd into a real 11.6.6 project and +// run through `mx check`, reports 0 errors. The control that the check is not +// inert on that project: a view entity with a deliberately wrong column type in +// the same app fails CE6770, so mxbuild was really validating. +// +// mxcli reported `duplicate widget name 's' (used 3 times)` and, because a +// reference error fails the run, refused to execute the script at all. +func TestCheckDuplicateWidgetNames_ObjectListItemsAreNotWidgets(t *testing.T) { + got := checkDuplicateWidgetNames(dashboardWithThreeSeries(), chartRegistry()) + for _, e := range got { + if strings.Contains(e, "'s'") { + t.Errorf("object-list item names counted as widget names: %q", e) + } + } +} + +// The control: real duplicate WIDGET names must still be reported. A fix that +// stopped descending into a pluggable widget's children — or that skipped every +// child of one — would pass the test above and turn CE0495 detection off for +// everything inside a chart or a gallery. +func TestCheckDuplicateWidgetNames_RealDuplicatesStillReported(t *testing.T) { + widgets := []*ast.WidgetV3{ + {Type: "dynamictext", Name: "dup"}, + {Type: "container", Name: "c", Children: []*ast.WidgetV3{ + {Type: "dynamictext", Name: "dup"}, + }}, + } + got := checkDuplicateWidgetNames(widgets, chartRegistry()) + if len(got) != 1 || !strings.Contains(got[0], "'dup'") { + t.Errorf("a genuine duplicate widget name was not reported: %v", got) + } +} + +// The second control: a child of a pluggable widget that is NOT one of its +// object-list containers is a real widget in a child slot, and two of those +// sharing a name IS CE0495. +func TestCheckDuplicateWidgetNames_ChildSlotWidgetsStillCount(t *testing.T) { + widgets := []*ast.WidgetV3{ + {Type: "pluggablewidget", Name: "w1", + Properties: map[string]any{"WidgetType": "com.mendix.widget.web.barchart.BarChart"}, + Children: []*ast.WidgetV3{{Type: "dynamictext", Name: "dup"}}}, + {Type: "dynamictext", Name: "dup"}, + } + got := checkDuplicateWidgetNames(widgets, chartRegistry()) + if len(got) != 1 || !strings.Contains(got[0], "'dup'") { + t.Errorf("a duplicate inside a pluggable widget's child slot was not reported: %v", got) + } +} + +// The third control: without a registry nothing can be resolved, and the rule +// must fall back to its previous behaviour rather than silently accepting +// everything. `check` runs with no project in CI, so this is the common path. +func TestCheckDuplicateWidgetNames_NoRegistryStillReports(t *testing.T) { + widgets := []*ast.WidgetV3{ + {Type: "dynamictext", Name: "dup"}, + {Type: "dynamictext", Name: "dup"}, + } + if got := checkDuplicateWidgetNames(widgets, nil); len(got) != 1 { + t.Errorf("with no registry, a plain duplicate must still be reported: %v", got) + } +} diff --git a/mdl/executor/validate_page_context_test.go b/mdl/executor/validate_page_context_test.go index 5dfee86b5b..230662ad54 100644 --- a/mdl/executor/validate_page_context_test.go +++ b/mdl/executor/validate_page_context_test.go @@ -25,7 +25,7 @@ func TestValidatePageContextTree_ParameterDSValid(t *testing.T) { }, } - errors := validatePageContextTree(params, widgets) + errors := validatePageContextTree(nil, params, widgets) if len(errors) > 0 { t.Errorf("Expected no errors, got: %v", errors) } @@ -44,7 +44,7 @@ func TestValidatePageContextTree_ParameterDSInvalid(t *testing.T) { }, } - errors := validatePageContextTree(params, widgets) + errors := validatePageContextTree(nil, params, widgets) if len(errors) != 1 { t.Fatalf("Expected 1 error, got %d: %v", len(errors), errors) } @@ -72,7 +72,7 @@ func TestValidatePageContextTree_SelectionDSValid(t *testing.T) { }, } - errors := validatePageContextTree(nil, widgets) + errors := validatePageContextTree(nil, nil, widgets) if len(errors) > 0 { t.Errorf("Expected no errors, got: %v", errors) } @@ -88,7 +88,7 @@ func TestValidatePageContextTree_SelectionDSInvalid(t *testing.T) { }, } - errors := validatePageContextTree(nil, widgets) + errors := validatePageContextTree(nil, nil, widgets) if len(errors) != 1 { t.Fatalf("Expected 1 error, got %d: %v", len(errors), errors) } @@ -102,7 +102,7 @@ func TestValidatePageContextTree_AttributeWithoutContext(t *testing.T) { {Type: "textbox", Name: "txtName", Properties: map[string]any{"Attribute": "Name"}}, } - errors := validatePageContextTree(nil, widgets) + errors := validatePageContextTree(nil, nil, widgets) if len(errors) != 1 { t.Fatalf("Expected 1 error, got %d: %v", len(errors), errors) } @@ -125,14 +125,14 @@ func TestValidatePageContextTree_AttributeInsideDataView(t *testing.T) { }, } - errors := validatePageContextTree(nil, widgets) + errors := validatePageContextTree(nil, nil, widgets) if len(errors) > 0 { t.Errorf("Expected no errors, got: %v", errors) } } func TestValidatePageContextTree_NoErrors(t *testing.T) { - errors := validatePageContextTree(nil, nil) + errors := validatePageContextTree(nil, nil, nil) if len(errors) > 0 { t.Errorf("Expected no errors for nil widgets, got: %v", errors) } diff --git a/mdl/executor/validate_page_unnamed_widgets_test.go b/mdl/executor/validate_page_unnamed_widgets_test.go index 4d7dc743c8..77c3f49bbf 100644 --- a/mdl/executor/validate_page_unnamed_widgets_test.go +++ b/mdl/executor/validate_page_unnamed_widgets_test.go @@ -57,7 +57,7 @@ func TestCheckDuplicateWidgetNames_IgnoresUnnamedWidgetKinds(t *testing.T) { ), } - if errs := checkDuplicateWidgetNames(page); len(errs) != 0 { + if errs := checkDuplicateWidgetNames(page, nil); len(errs) != 0 { t.Errorf("a widget kind whose name is not stored cannot be a CE0495 duplicate; got:\n %s", strings.Join(errs, "\n ")) } @@ -77,7 +77,7 @@ func TestCheckDuplicateWidgetNames_StillCatchesRealDuplicates(t *testing.T) { ), } - errs := checkDuplicateWidgetNames(page) + errs := checkDuplicateWidgetNames(page, nil) if len(errs) != 1 { t.Fatalf("got %d errors, want 1:\n %s", len(errs), strings.Join(errs, "\n ")) } @@ -95,7 +95,7 @@ func TestCheckDuplicateWidgetNames_NamedWidgetCollidingWithADerivedName(t *testi namedWidget("container", "row1"), } - errs := checkDuplicateWidgetNames(page) + errs := checkDuplicateWidgetNames(page, nil) if len(errs) != 1 || !strings.Contains(errs[0], "row1") { t.Fatalf("two containers named row1 are a real duplicate; got %d errors:\n %s", len(errs), strings.Join(errs, "\n ")) diff --git a/mdl/executor/validate_widget_aliases_test.go b/mdl/executor/validate_widget_aliases_test.go new file mode 100644 index 0000000000..2ae48f412e --- /dev/null +++ b/mdl/executor/validate_widget_aliases_test.go @@ -0,0 +1,111 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" +) + +// A property mapping's MdlAliases are the names a person is meant to WRITE. +// They must be accepted by the property validator, or the validator rejects the +// only spelling the documentation offers. +// +// # The defect +// +// A PieChart binds its data at the widget level, so its def.json carries +// +// {"propertyKey": "seriesValueAttribute", "source": "Attribute", +// "operation": "attribute", "mdlAliases": ["ValueAttribute"]} +// +// and the builder resolves `ValueAttribute:` through that alias — measured: the +// stored page comes back from DESCRIBE as `seriesValueAttribute: Total`, so the +// value persists. allowedWidgetProperties built its set from PropertyKey and +// Source only, so `ValueAttribute` was unknown and MDL-WIDGET01 fired. Since +// exec refuses to run a script with errors, the false positive did not merely +// warn — it blocked the page from being written at all. +// +// This is the "two lists, nothing comparing them" class again, and the sibling +// list in widget_defs.go (`mapped`, for knownProperties) already walks +// MdlAliases — so the two disagreed about the same def.json. +func TestAllowedWidgetProperties_IncludesMdlAliases(t *testing.T) { + def := &WidgetDefinition{ + WidgetID: "com.acme.Test", + MDLName: "test", + PropertyMappings: []PropertyMapping{ + {PropertyKey: "seriesValueAttribute", Source: "Attribute", + Operation: "attribute", MdlAliases: []string{"ValueAttribute"}}, + }, + } + + allowed, keys := allowedWidgetProperties(def) + + if !allowed["valueattribute"] { + t.Errorf("the alias `ValueAttribute` is not an allowed property; allowed keys: %v", keys) + } + // The control: the schema key must STILL be allowed. A fix that swapped one + // name for the other would pass the assertion above and break every script + // written against the schema key — including DESCRIBE's own output, which + // emits `seriesValueAttribute`. + if !allowed["seriesvalueattribute"] { + t.Errorf("the schema key `seriesValueAttribute` stopped being allowed; allowed keys: %v", keys) + } + // The suggestion list is what "did you mean" reads, so a typo'd alias should + // point back at the alias, not only at the schema key. + var sawAlias bool + for _, k := range keys { + if k == "ValueAttribute" { + sawAlias = true + } + } + if !sawAlias { + t.Errorf("`ValueAttribute` missing from the suggestion list %v — a typo would be told to "+ + "use the internal name instead of the documented one", keys) + } +} + +// Mode-scoped mappings carry aliases too, and go through the same helper. +// +// The alias here is deliberately NOT a case variant of the property key. The +// PieChart's real pair is `seriesName` / `SeriesName`, which collapses to one +// entry once lowercased — so a test using it passes against the broken code and +// proves nothing. +func TestAllowedWidgetProperties_IncludesMdlAliasesInModes(t *testing.T) { + def := &WidgetDefinition{ + WidgetID: "com.acme.Test", + MDLName: "test", + Modes: []WidgetMode{{ + PropertyMappings: []PropertyMapping{ + {PropertyKey: "seriesSortAttribute", Source: "Attribute", + Operation: "attribute", MdlAliases: []string{"SortAttribute"}}, + }, + }}, + } + allowed, keys := allowedWidgetProperties(def) + if !allowed["sortattribute"] { + t.Errorf("mode-scoped alias `SortAttribute` not allowed; keys: %v", keys) + } +} + +// The end-to-end shape of the bug, against the real embedded/installed +// definitions rather than a hand-built one: a PieChart written the documented +// way must not produce MDL-WIDGET01. +// +// Skips when the PieChart definition is not available (it ships in Charts.mpk, +// not in the embedded set), so this is a bonus assertion — the two above are the +// ones that always run. +func TestPieChartValueAttributeIsAccepted(t *testing.T) { + registry := LoadWidgetRegistry("") + if registry == nil { + t.Skip("no registry") + } + def, ok := registry.GetByWidgetID("com.mendix.widget.web.piechart.PieChart") + if !ok || def == nil { + t.Skip("PieChart definition not available without a project") + } + allowed, keys := allowedWidgetProperties(def) + if !allowed["valueattribute"] { + t.Errorf("PieChart rejects `ValueAttribute:`, the name propertyAliases registers "+ + "and the builder resolves; allowed: %s", strings.Join(keys, ", ")) + } +} diff --git a/mdl/executor/validate_widget_hidden.go b/mdl/executor/validate_widget_hidden.go index fc40e14a98..5d86fbf328 100644 --- a/mdl/executor/validate_widget_hidden.go +++ b/mdl/executor/validate_widget_hidden.go @@ -158,6 +158,20 @@ func registryProjectPath(registry *WidgetRegistry) string { return registry.projectPath } +// sharedSourceOffMode reports whether an item property's shared Source lookup +// must be skipped because the item's dataSet mode selects its sibling. +// +// Scoped to a chart series' datasource pair, which is the one place two item +// properties claim the same Source. Anything wider would blind the +// hidden-property rule on every other object list, so the gate mirrors the +// builder's own condition rather than generalising it. +func sharedSourceOffMode(mapping *ObjectListMapping, m ItemPropertyMapping, dataSetMode string) bool { + if m.Operation != "datasource" || !isChartSeriesContainer(mapping.MDLContainer) { + return false + } + return !seriesDataSourceMatchesMode(m.PropertyKey, dataSetMode) +} + // itemValueMap resolves an object-list item's sub-property values (keyed by // lowercased schema key) and reports which the MDL set explicitly. The item form // of widgetValueMap. @@ -165,10 +179,24 @@ func itemValueMap(item *ast.WidgetV3, mapping *ObjectListMapping) (values map[st values = map[string]string{} explicit = map[string]bool{} + // A chart series' two datasource sub-properties SHARE the Source name + // "DataSource" (measured on linechart.def.json: staticDataSource and + // dynamicDataSource both declare it, neither declares an alias), so the + // mode selects which one the friendly `DataSource:` lands in. The builder + // routes on it — buildObjectListItem consults "DataSource" only for the + // property seriesDataSourceMatchesMode picks — so the checker has to as + // well, or it reports a property the script never wrote. + itemDataSetMode := "static" + if v, ok := lookupProperty(item.Properties, "dataSet"); ok { + if s := stringifyAny(v); s != "" { + itemDataSetMode = s + } + } + for _, m := range mapping.ItemProperties { key := strings.ToLower(m.PropertyKey) val, set := "", false - if m.Source != "" { + if m.Source != "" && !sharedSourceOffMode(mapping, m, itemDataSetMode) { if v, ok := lookupWidgetProp(item, m.Source); ok { val, set = v, true } diff --git a/mdl/executor/validate_widget_hidden_dataset_test.go b/mdl/executor/validate_widget_hidden_dataset_test.go new file mode 100644 index 0000000000..9ebfebfdcc --- /dev/null +++ b/mdl/executor/validate_widget_hidden_dataset_test.go @@ -0,0 +1,114 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" +) + +// chartSeriesItem builds a chart `series`/`line` item written the documented way: +// one friendly `DataSource:` plus the dataSet mode that selects which schema +// property it lands in. +func chartSeriesItem(dataSet string) *ast.WidgetV3 { + return &ast.WidgetV3{ + Name: "s1", + Properties: map[string]any{ + "DataSet": dataSet, + "DataSource": &ast.DataSourceV3{Type: "database", Reference: "Mod.View"}, + }, + } +} + +// chartSeriesMapping mirrors the shape a chart's def.json actually has: BOTH +// datasource sub-properties declare Source "DataSource" and neither declares an +// alias, so a lookup by Source alone cannot tell them apart. +func chartSeriesMapping() *ObjectListMapping { + return &ObjectListMapping{ + MDLContainer: "SERIES", + PropertyKey: "series", + ItemProperties: []ItemPropertyMapping{ + {PropertyKey: "dataSet", Operation: "primitive", Value: "static"}, + {PropertyKey: "staticDataSource", Source: "DataSource", Operation: "datasource"}, + {PropertyKey: "dynamicDataSource", Source: "DataSource", Operation: "datasource"}, + }, + } +} + +// The friendly `DataSource:` on a chart series is routed BY dataSet mode when the +// item is built: buildObjectListItem looks the alias up only for the property +// seriesDataSourceMatchesMode selects, so `dataSet: 'static'` writes +// staticDataSource and leaves dynamicDataSource unset. +// +// itemValueMap resolved it by Source instead, which both properties share, so it +// reported dynamicDataSource as explicitly set — and MDL-WIDGET10 then warned +// that a value the script never wrote "will be ignored". Measured: 11 such +// warnings on 34-chart-widget-examples.mdl, one per series in the file, on the +// only syntax the examples and skills document. +func TestItemValueMap_FriendlyDataSourceIsModeRouted(t *testing.T) { + _, explicit := itemValueMap(chartSeriesItem("static"), chartSeriesMapping()) + + if !explicit["staticdatasource"] { + t.Error("staticDataSource not explicit under dataSet 'static' — the mode-matching " + + "property is the one the builder writes, so the checker must see it set") + } + if explicit["dynamicdatasource"] { + t.Error("dynamicDataSource reported as explicitly set under dataSet 'static'; " + + "buildObjectListItem never writes it, so MDL-WIDGET10 warns about a value " + + "that does not exist") + } +} + +// The control: the routing must follow the mode rather than always preferring +// `static`. A fix that hardcoded "ignore dynamic*" would pass the test above and +// break every dynamic series. +func TestItemValueMap_FriendlyDataSourceFollowsDynamicMode(t *testing.T) { + _, explicit := itemValueMap(chartSeriesItem("dynamic"), chartSeriesMapping()) + + if !explicit["dynamicdatasource"] { + t.Error("dynamicDataSource not explicit under dataSet 'dynamic' — this is the " + + "property the builder writes in that mode") + } + if explicit["staticdatasource"] { + t.Error("staticDataSource reported as explicitly set under dataSet 'dynamic'") + } +} + +// The second control: the gate is scoped to chart series. A non-chart object list +// whose sub-properties share a Source must keep resolving by Source, or the +// hidden-property rule goes blind on every other widget. +func TestItemValueMap_NonChartContainerStillResolvesBySource(t *testing.T) { + mapping := &ObjectListMapping{ + MDLContainer: "COLUMN", + PropertyKey: "columns", + ItemProperties: []ItemPropertyMapping{ + {PropertyKey: "attribute", Source: "Attribute", Operation: "attribute"}, + }, + } + item := &ast.WidgetV3{Name: "c1", Properties: map[string]any{"Attribute": "Name"}} + + _, explicit := itemValueMap(item, mapping) + if !explicit["attribute"] { + t.Error("a non-chart item property stopped resolving through its Source") + } +} + +// The third control: an item that names a datasource by its SCHEMA key keeps +// working regardless of mode. Someone writing `dynamicDataSource:` explicitly +// means it, and the builder honours it (the PropertyKey lookup runs before the +// mode-aware fallback), so the checker must see it set. +func TestItemValueMap_ExplicitSchemaKeyIgnoresMode(t *testing.T) { + item := &ast.WidgetV3{ + Name: "s1", + Properties: map[string]any{ + "DataSet": "static", + "dynamicDataSource": &ast.DataSourceV3{Type: "database", Reference: "Mod.View"}, + }, + } + _, explicit := itemValueMap(item, chartSeriesMapping()) + if !explicit["dynamicdatasource"] { + t.Error("an explicitly named dynamicDataSource was dropped by the mode gate — " + + "the gate applies to the shared Source lookup, not to the schema key") + } +} diff --git a/mdl/executor/validate_widget_kind.go b/mdl/executor/validate_widget_kind.go new file mode 100644 index 0000000000..1949167472 --- /dev/null +++ b/mdl/executor/validate_widget_kind.go @@ -0,0 +1,282 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "fmt" + "path/filepath" + "sort" + "strings" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/linter" + "github.com/mendixlabs/mxcli/sdk/widgets/mpk" +) + +// Slice 0 of PROPOSAL_def_driven_widget_bodies.md: teach the validator what a +// widget is. +// +// Until now the GRAMMAR was the widget-kind validator — `widgetTypeV3` is an +// allow-list, so an unknown kind could not parse and the validator never needed +// an independent notion of one. Two mistakes already slip past that, because +// neither is a keyword the parser checks, and both reached `exec` with `check` +// reporting success: +// +// pluggablewidget 'com.acme.NotAWidget' w1 -- the id is a string literal +// group g1 (…) inside HTML Element -- a real keyword, wrong parent +// +// Reporting them is worth doing on its own. It is also what makes the +// def-driven body (slices 2-3) safe: those give up the parser's enforcement, so +// the semantic check has to exist first. + +// validateWidgetKind reports a widget whose kind mxcli cannot resolve, and a +// container keyword the parent's definition does not declare. +func validateWidgetKind(w *ast.WidgetV3, registry *WidgetRegistry, parentDef *WidgetDefinition, + parentObjectLists map[string]*ObjectListMapping, locationPrefix string) []linter.Violation { + if w == nil || registry == nil { + return nil + } + + // An explicit widget id that resolves to nothing. Only reachable through the + // `pluggablewidget ''` / `customwidget ''` forms, where the id is a + // string literal the parser cannot check. + if id, ok := w.Properties["WidgetType"].(string); ok && id != "" { + // With no project there is nothing to be unknown RELATIVE TO: the + // registry holds only the embedded widgets, so every real project + // widget would be reported. `mxcli check` with no -p is the common + // case — it is how the example corpus is checked in CI — and without + // this one example file alone produced 14 violations. + // + // Scoped to this branch on purpose: the container rule below needs a + // resolvable PARENT, not a project, and works from a definition alone. + if registryProjectPath(registry) == "" { + return nil + } + if _, known := registry.GetByWidgetID(id); !known && !packageInstalledFor(registry, id) { + return []linter.Violation{{ + RuleID: "MDL-WIDGET25", + Severity: linter.SeverityError, + Message: fmt.Sprintf("%s: widget `%s` has no definition for %q%s", + locationPrefix, w.Name, id, nearestWidgetIDs(registry, id)), + Suggestion: "install the widget from the Marketplace so its .mpk lands in widgets/, or check the id for a typo", + }} + } + return nil + } + + // A generic widget type — one the grammar accepted as a bare IDENTIFIER + // rather than as an enumerated widget token (slice 2). It can only be a + // widget definition's MDL name, so failing to resolve one is a mistake, + // not a built-in mxcli happens not to know. + // + // Without this the typo `htmlelemnt` reaches validateStaticWidgetUnknownProps + // and is reported as an unrecognized PROPERTY (MDL-WIDGET07, a warning), so + // `check` exits 0 having complained about the wrong thing. Measured before + // this branch existed: `htmlelemnt frame (tagName: 'div')` gave + // "0 errors, 1 warning" about `tagName`, while the correct spelling was + // completely clean. + // + // Same project requirement as the id branch below, and for the same reason: + // with no project the registry holds only the embedded widgets, so every + // real one would be reported. + if w.TypeIsGeneric && registryProjectPath(registry) != "" { + if _, known := registry.Get(strings.ToUpper(w.Type)); known { + return nil + } + // A container the parent declares is not a widget and never resolves in + // the registry — that is what makes it a container. Accept it here and + // let the object-list engine validate its properties. + if parentDef != nil && + (parentObjectLists[strings.ToUpper(w.Type)] != nil || parentDeclaresSlot(parentDef, w.Type)) { + return nil + } + // Inside a resolvable parent, "not a container of " beats "not a + // widget": it names what the parent DOES declare, which is the answer + // the author needs. `attribut` inside an htmlelement is a misspelt + // container, not a missing widget package. + if parentDef != nil { + return []linter.Violation{{ + RuleID: "MDL-WIDGET26", + Severity: linter.SeverityError, + Message: fmt.Sprintf("%s: `%s` is not a container of %s%s", + locationPrefix, strings.ToLower(w.Type), parentLabel(parentDef), declaredContainers(parentDef)), + Suggestion: "use one of the parent widget's own containers, or move this out of the widget's body", + }} + } + return []linter.Violation{{ + RuleID: "MDL-WIDGET25", + Severity: linter.SeverityError, + Message: fmt.Sprintf("%s: `%s` is not a widget in this project%s", + locationPrefix, strings.ToLower(w.Type), nearestWidgetNames(registry, w.Type)), + Suggestion: "run `mxcli widget init` if the package was just installed, or `describe widget ` to see what is available", + }} + } + + // A container keyword used where the parent does not declare it. These + // keywords mean nothing on their own — `group` is not a widget — so one + // outside a parent that declares it can only be a mistake. + if !isObjectListContainerKeyword(w.Type, registry) { + return nil + } + // Never judge a container against a parent that cannot be resolved. In a + // project that has not run `widget init` the registry knows only the + // embedded widgets, so every real parent looks container-less and every + // container would be reported. Silence is the honest answer there. + if parentDef == nil { + return nil + } + if parentObjectLists[strings.ToUpper(w.Type)] != nil || parentDeclaresSlot(parentDef, w.Type) { + return nil + } + return []linter.Violation{{ + RuleID: "MDL-WIDGET26", + Severity: linter.SeverityError, + Message: fmt.Sprintf("%s: `%s` is not a container of %s%s", + locationPrefix, strings.ToLower(w.Type), parentLabel(parentDef), declaredContainers(parentDef)), + Suggestion: "use one of the parent widget's own containers, or move this out of the widget's body", + }} +} + +// isObjectListContainerKeyword reports whether a keyword is an object-list +// container rather than a widget in its own right. +// +// Derived from the registry — the union of every known definition's container +// keywords — rather than restated. A hardcoded list here was already wrong: +// isUniversalObjectListKeyword names seven where the grammar has nine, missing +// SCALECOLOR, CUSTOMBUTTON and ALLOWEDFILEFORMAT, which is the same list-drift +// this proposal exists to remove. +func isObjectListContainerKeyword(widgetType string, registry *WidgetRegistry) bool { + if widgetType == "" || registry == nil { + return false + } + up := strings.ToUpper(widgetType) + // A keyword that names a widget is a widget, whatever else it may be. + if _, isWidget := registry.Get(up); isWidget { + return false + } + for _, def := range registry.All() { + for _, ol := range def.ObjectLists { + if strings.EqualFold(ol.MDLContainer, widgetType) { + return true + } + } + } + return isUniversalObjectListKeyword(widgetType) +} + +func parentLabel(parentDef *WidgetDefinition) string { + if parentDef == nil { + return "this widget" + } + if parentDef.MDLName != "" { + return "`" + strings.ToLower(parentDef.MDLName) + "`" + } + return "`" + parentDef.WidgetID + "`" +} + +// declaredContainers names what the parent DOES declare, so the reader is not +// left to guess. A parent with none says so, which is the more useful answer. +func declaredContainers(parentDef *WidgetDefinition) string { + if parentDef == nil { + return "" + } + var names []string + for _, ol := range parentDef.ObjectLists { + names = append(names, strings.ToLower(ol.MDLContainer)) + } + for _, cs := range parentDef.ChildSlots { + names = append(names, strings.ToLower(cs.MDLContainer)) + } + if len(names) == 0 { + return ", which declares no containers" + } + sort.Strings(names) + return " — it declares: " + strings.Join(names, ", ") +} + +// nearestWidgetIDs suggests known ids sharing the unknown one's last segment, +// which is where a typo or a wrong vendor prefix usually shows. +func nearestWidgetIDs(registry *WidgetRegistry, id string) string { + last := id + if i := strings.LastIndex(id, "."); i >= 0 { + last = id[i+1:] + } + var hits []string + for _, def := range registry.All() { + if strings.EqualFold(def.WidgetID, id) { + continue + } + if strings.Contains(strings.ToLower(def.WidgetID), strings.ToLower(last)) { + hits = append(hits, def.WidgetID) + } + } + if len(hits) == 0 { + return "" + } + sort.Strings(hits) + if len(hits) > 3 { + hits = hits[:3] + } + return " — did you mean " + strings.Join(hits, ", ") + "?" +} + +// packageInstalledFor reports whether the project has a widget package for this +// id, even though no definition has been extracted from it yet. +// +// Load-bearing against a false-positive storm. LoadWidgetRegistry reads only +// `.mxcli/widgets/*.def.json`; unlike the page builder's registry it does NOT +// refresh those from installed .mpk files. So in a project that has never run +// `widget init`, the validator knows the nine embedded widgets and nothing else +// — and calling every real project widget "unknown" would be worse than the +// silence this rule replaces. +// +// Asking whether the package is installed is the same question slice 1's error +// message asks, and it separates "mxcli has not looked at this widget yet" from +// "this widget does not exist". +func packageInstalledFor(registry *WidgetRegistry, widgetID string) bool { + dir := registryProjectPath(registry) + if dir == "" { + return false + } + found, err := mpk.FindMPK(filepath.Dir(dir), widgetID) + return err == nil && found != "" +} + +// parentDeclaresSlot reports whether the parent declares a CHILD SLOT by this +// name. Object lists alone are not the whole vocabulary of a widget body, and +// treating a declared slot as undeclared would report correct MDL. +func parentDeclaresSlot(parentDef *WidgetDefinition, keyword string) bool { + if parentDef == nil { + return false + } + for _, cs := range parentDef.ChildSlots { + if strings.EqualFold(cs.MDLContainer, keyword) { + return true + } + } + return false +} + +// nearestWidgetNames suggests known MDL names close to an unresolved one, +// using the same edit-distance helper MDL-WIDGET07 uses for property keys. +// +// A hand-rolled prefix heuristic was tried first and is not good enough: the +// commonest real mistake is a dropped letter in the MIDDLE (`htmlelemnt`), +// which shares no useful prefix with `htmlelement` past the typo, so it +// suggested nothing at all on the very case that motivated the rule. +func nearestWidgetNames(registry *WidgetRegistry, name string) string { + if registry == nil || name == "" { + return "" + } + var candidates []string + for _, def := range registry.All() { + if def.MDLName != "" { + candidates = append(candidates, strings.ToLower(def.MDLName)) + } + } + sort.Strings(candidates) + if best := nearestKey(name, candidates); best != "" { + return " — did you mean `" + best + "`?" + } + return "" +} diff --git a/mdl/executor/validate_widget_kind_test.go b/mdl/executor/validate_widget_kind_test.go new file mode 100644 index 0000000000..b8a227df2b --- /dev/null +++ b/mdl/executor/validate_widget_kind_test.go @@ -0,0 +1,182 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" +) + +// widgetKindViolations runs the widget-tree validator over one page body. +func widgetKindViolations(t *testing.T, projectPath string, widgets []*ast.WidgetV3) []string { + t.Helper() + registry := LoadWidgetRegistry(projectPath) + if registry == nil { + t.Fatal("no widget registry") + } + var out []string + for _, v := range validateWidgetTree(widgets, registry, "page X") { + out = append(out, v.RuleID+": "+v.Message) + } + return out +} + +func pluggable(id, name string, children ...*ast.WidgetV3) *ast.WidgetV3 { + return &ast.WidgetV3{ + Type: "pluggablewidget", + Name: name, + Properties: map[string]any{"WidgetType": id}, + Children: children, + } +} + +// An unknown widget id reaches `exec` and fails there, while `check` passes. +// The parser cannot catch it — the id is a string literal — so the validator is +// the only thing that could, and today it says nothing. +func TestValidateWidgetKind_UnknownWidgetIDIsReported(t *testing.T) { + // Needs a project: with none, the rule cannot tell an unknown widget from + // one installed in a project it cannot see — see the no-project control. + got := widgetKindViolations(t, "../../testdata/expr-checker/minimal.mpr", []*ast.WidgetV3{ + pluggable("com.acme.widget.NotAWidget", "w1"), + }) + if !containsRule(got, "MDL-WIDGET25") { + t.Fatalf("unknown widget id not reported; got %v", got) + } + if !containsText(got, "com.acme.widget.NotAWidget") { + t.Errorf("the message does not name the widget: %v", got) + } +} + +// The control for the above: a widget mxcli knows must stay silent, or the rule +// is simply "always complain". +func TestValidateWidgetKind_KnownWidgetIDIsSilent(t *testing.T) { + got := widgetKindViolations(t, "", []*ast.WidgetV3{ + pluggable("com.mendix.widget.web.combobox.Combobox", "w1"), + }) + if containsRule(got, "MDL-WIDGET25") { + t.Errorf("a known widget was reported as unknown: %v", got) + } +} + +// A container keyword the parent does not declare — `group` on a widget whose +// definition has no such object list. It parses (GROUP is in the grammar) and +// the validator used to SKIP it, because isUniversalObjectListKeyword treats +// the keyword as always-an-item wherever it appears. +// +// Tested against a synthetic definition rather than through the tree walk: no +// EMBEDDED widget declares an object list, and the fixture project has no +// extracted defs, so neither route can express the control below. +func TestValidateWidgetKind_ContainerNotDeclaredByTheParentIsReported(t *testing.T) { + registry := LoadWidgetRegistry("") + parent := &WidgetDefinition{ + MDLName: "PARENTWIDGET", + ObjectLists: []ObjectListMapping{ + {MDLContainer: "SERIES", PropertyKey: "series"}, + }, + } + got := validateWidgetKind(&ast.WidgetV3{Type: "group", Name: "g1"}, + registry, parent, objectListMappingSet(parent), "page X") + + if len(got) == 0 || got[0].RuleID != "MDL-WIDGET26" { + t.Fatalf("an undeclared container was not reported; got %v", got) + } + if !strings.Contains(got[0].Message, "group") { + t.Errorf("the message does not name the container: %s", got[0].Message) + } + // It must say what the parent DOES declare, or the reader is left guessing. + if !strings.Contains(got[0].Message, "series") { + t.Errorf("the message does not name the parent's real containers: %s", got[0].Message) + } +} + +// The control: the same keyword on a parent that DOES declare it stays silent. +// Without this, the rule above passes against a build that rejects everything. +func TestValidateWidgetKind_ContainerDeclaredByTheParentIsSilent(t *testing.T) { + registry := LoadWidgetRegistry("") + parent := &WidgetDefinition{ + MDLName: "PARENTWIDGET", + ObjectLists: []ObjectListMapping{ + {MDLContainer: "GROUP", PropertyKey: "groups"}, + }, + } + if got := validateWidgetKind(&ast.WidgetV3{Type: "group", Name: "g1"}, + registry, parent, objectListMappingSet(parent), "page X"); len(got) != 0 { + t.Errorf("the parent declares `group`, so it must not be reported: %v", got) + } +} + +// A declared CHILD SLOT is equally legitimate. Object lists are not the whole +// vocabulary of a widget body. +func TestValidateWidgetKind_DeclaredChildSlotIsSilent(t *testing.T) { + registry := LoadWidgetRegistry("") + parent := &WidgetDefinition{ + MDLName: "PARENTWIDGET", + ChildSlots: []ChildSlotMapping{{MDLContainer: "GROUP", PropertyKey: "groups"}}, + } + if got := validateWidgetKind(&ast.WidgetV3{Type: "group", Name: "g1"}, + registry, parent, nil, "page X"); len(got) != 0 { + t.Errorf("a declared child slot must not be reported: %v", got) + } +} + +// An unresolvable parent must silence the rule. In a project that never ran +// `widget init` the registry knows only the embedded widgets, so every real +// parent looks container-less — reporting there would bury correct MDL. +func TestValidateWidgetKind_UnresolvableParentSilencesTheContainerRule(t *testing.T) { + registry := LoadWidgetRegistry("") + if got := validateWidgetKind(&ast.WidgetV3{Type: "group", Name: "g1"}, + registry, nil, nil, "page X"); len(got) != 0 { + t.Errorf("with no resolvable parent the rule must stay silent: %v", got) + } +} + +// An ordinary widget nested inside a pluggable widget's slot is legitimate and +// must not be mistaken for an undeclared container. +func TestValidateWidgetKind_OrdinaryNestedWidgetIsSilent(t *testing.T) { + const fixture = "../../testdata/expr-checker/minimal.mpr" + got := widgetKindViolations(t, fixture, []*ast.WidgetV3{ + pluggable("com.mendix.widget.web.htmlelement.HTMLElement", "h", + &ast.WidgetV3{Type: "container", Name: "c1"}), + }) + if containsRule(got, "MDL-WIDGET26") { + t.Errorf("a plain container widget was reported as an undeclared container: %v", got) + } +} + +func containsRule(msgs []string, rule string) bool { + for _, m := range msgs { + if strings.Contains(m, rule) { + return true + } + } + return false +} + +func containsText(msgs []string, text string) bool { + for _, m := range msgs { + if strings.Contains(m, text) { + return true + } + } + return false +} + +// With NO project the registry holds only the embedded widgets, so every real +// project widget would look unknown. `mxcli check` without -p is the common +// case — it is how the example corpus is checked in CI — and an earlier version +// of this rule produced 14 violations in a single example file. +// +// The guard that was there (is the .mpk installed?) could not help: with no +// project there is no widgets/ to look in. This is the control that measurement +// with -p could not provide. +func TestValidateWidgetKind_NoProjectMeansNoUnknownWidgetClaims(t *testing.T) { + got := widgetKindViolations(t, "", []*ast.WidgetV3{ + pluggable("com.mendix.widget.web.htmlelement.HTMLElement", "h"), + pluggable("com.acme.widget.NotAWidget", "w1"), + }) + if containsRule(got, "MDL-WIDGET25") { + t.Errorf("claimed a widget is unknown with no project to judge against: %v", got) + } +} diff --git a/mdl/executor/validate_widgets.go b/mdl/executor/validate_widgets.go index 7ee78a8ae0..954d7f76cb 100644 --- a/mdl/executor/validate_widgets.go +++ b/mdl/executor/validate_widgets.go @@ -62,6 +62,11 @@ func LoadWidgetRegistry(projectPath string) *WidgetRegistry { if projectPath != "" { _ = registry.LoadUserDefinitions(projectPath) registry.projectPath = projectPath + // The validator and DESCRIBE WIDGET must agree about which properties a + // widget has; they read different sources, so the definition is topped up + // from the same .mpk DESCRIBE parses. See + // widget_known_props_from_mpk.go for why this is not a list of nine. + enrichKnownPropertiesFromMPK(registry, projectPath) } return registry } @@ -116,6 +121,9 @@ func validateWidgetTreeIn(widgets []*ast.WidgetV3, registry *WidgetRegistry, loc } mapping := parentObjectLists[strings.ToUpper(w.Type)] isObjectListItem := mapping != nil || isUniversalObjectListKeyword(w.Type) + // Slice 0: is this a widget at all, and does the parent declare this + // container? Both were previously left to `exec`. + out = append(out, validateWidgetKind(w, registry, lookupWidgetDef(parent, registry), parentObjectLists, locationPrefix)...) out = append(out, validatePluggableWidgetProperties(w, registry, locationPrefix)...) // #928: contentparams with no `{N}` placeholder to consume them. if lookupWidgetDef(w, registry) != nil { @@ -135,7 +143,14 @@ func validateWidgetTreeIn(widgets []*ast.WidgetV3, registry *WidgetRegistry, loc // widgets get the stricter def.json check (MDL-WIDGET01) above, and // object-list items are validated by the object-list engine. def := lookupWidgetDef(w, registry) - if def == nil && !isObjectListItem { + // A generic widget type that resolved to nothing is already reported as + // MDL-WIDGET25 (the kind is wrong). Validating its properties on top of + // that says the kind is fine and the property is not, which points at + // the wrong token — measured on `htmlelemnt frame (tagName: 'div')`, + // which drew a `tagName` warning beside the real error. A built-in + // (TypeIsGeneric false) keeps the check, since its properties are the + // only thing that can be wrong about it. + if def == nil && !isObjectListItem && !w.TypeIsGeneric { out = append(out, validateStaticWidgetUnknownProps(w, locationPrefix)...) // #928: `editable:` on a widget Mendix gives no editability — same // "silently dropped on write" family, but the flat property @@ -642,7 +657,7 @@ func isKnownStaticWidgetProp(key string) bool { // separately (MDL-WIDGET01) and must not reach here. func validateStaticWidgetUnknownProps(w *ast.WidgetV3, locationPrefix string) []linter.Violation { var out []linter.Violation - for key := range w.Properties { + for _, key := range sortedPropertyKeys(w) { if isKnownStaticWidgetProp(key) { continue } @@ -695,7 +710,7 @@ func validateDynamicTextFormatting(w *ast.WidgetV3, locationPrefix string) []lin // (1) Format keys placed at the widget level are silently dropped on write — // formatting is per-parameter. Flag them with the correct location. if strings.EqualFold(w.Type, "dynamictext") { - for key := range w.Properties { + for _, key := range sortedPropertyKeys(w) { if paramFormatKeys[strings.ToLower(key)] { out = append(out, linter.Violation{ RuleID: "MDL-WIDGET18", @@ -1044,7 +1059,7 @@ func validatePluggableWidgetProperties(w *ast.WidgetV3, registry *WidgetRegistry knownUnmapped := knownUnmappedProperties(def, allowed) var out []linter.Violation - for key := range w.Properties { + for _, key := range sortedPropertyKeys(w) { // Builtin property names (Label, Class, Visible, DataSource, …) are // MDL-recognized keywords that the widget engine routes via a // dedicated path rather than via propertyMappings. Accept them @@ -1153,6 +1168,16 @@ func addMappingNames(add func(string), m PropertyMapping) { add(m.PropertyKey) } add(m.Source) + // The aliases are the names people are TOLD to write, so they have to be + // accepted here — the builder already resolves them (widget_engine.go), and + // the knownProperties set in widget_defs.go already walks them. Leaving them + // out made the validator the odd one out of three readers of the same + // def.json: `ValueAttribute: Total` on a PieChart persisted correctly and + // was still reported as MDL-WIDGET01 "has no property", which — because exec + // refuses a script with errors — blocked the page from being written at all. + for _, a := range m.MdlAliases { + add(a) + } } // readsFixedASTSlot reports whether an operation's value is resolved from a @@ -1525,3 +1550,33 @@ func mappingOperationFor(def *WidgetDefinition, propertyKey string) string { } return "" } + +// sortedPropertyKeys returns a widget's property keys in a stable order. +// +// A validator that appends one violation per property key was iterating the map +// directly, so `mxcli check` printed the same warnings in a different order from +// one run to the next. Measured before the fix: two runs of the same binary over +// mdl-examples/ disagreed on 11 of 515 scripts. +// +// Nothing was wrong with the diagnostics — but "the output is stable" is what +// makes a before/after diff of `check` usable as a measurement, and it was not. +// This surfaced while diffing check output across the corpus to size the +// grammar change for slices 2-3: the noise floor of the tool was larger than +// the signal being looked for. +// +// The three call sites are the ones that emit PER KEY (MDL-WIDGET07, WIDGET17, +// WIDGET18). Two other loops over w.Properties do a case-insensitive LOOKUP and +// break on the first hit; those are left alone, since they are only +// order-sensitive when a widget carries two keys differing solely in case, and +// picking either is equally correct. +func sortedPropertyKeys(w *ast.WidgetV3) []string { + if w == nil { + return nil + } + out := make([]string, 0, len(w.Properties)) + for k := range w.Properties { + out = append(out, k) + } + sort.Strings(out) + return out +} diff --git a/mdl/executor/validate_widgets_order_test.go b/mdl/executor/validate_widgets_order_test.go new file mode 100644 index 0000000000..117161d134 --- /dev/null +++ b/mdl/executor/validate_widgets_order_test.go @@ -0,0 +1,103 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "sort" + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/linter" +) + +// widgetWithUnknownProps builds a static widget carrying several properties no +// builder consumes, so every one produces an MDL-WIDGET07 warning. Eight keys +// make an accidental pass vanishingly unlikely: with unsorted map iteration the +// chance of Go handing back the same order twice is 1/8!. +func widgetWithUnknownProps() *ast.WidgetV3 { + return &ast.WidgetV3{ + Type: "container", + Name: "c1", + Properties: map[string]any{ + "zeta": "1", "alpha": "2", "mike": "3", "delta": "4", + "omega": "5", "bravo": "6", "kilo": "7", "sierra": "8", + }, + } +} + +func messagesOf(vs []linter.Violation) []string { + out := make([]string, 0, len(vs)) + for _, v := range vs { + out = append(out, v.Message) + } + return out +} + +// `mxcli check` printed the same warnings in a different order from one run to +// the next, because the validator iterated w.Properties directly. Nothing was +// wrong with the diagnostics, but it made a before/after diff of check output +// useless as a measurement — two runs of the same binary over mdl-examples/ +// disagreed on 11 of 515 scripts, which was more noise than the grammar change +// being measured produced signal. +func TestStaticWidgetUnknownProps_OrderIsStable(t *testing.T) { + first := messagesOf(validateStaticWidgetUnknownProps(widgetWithUnknownProps(), "page M.P")) + if len(first) != 8 { + t.Fatalf("got %d violations, want 8 — the fixture must produce one per unknown key, or this test proves nothing", len(first)) + } + + for i := 0; i < 50; i++ { + got := messagesOf(validateStaticWidgetUnknownProps(widgetWithUnknownProps(), "page M.P")) + if len(got) != len(first) { + t.Fatalf("run %d produced %d violations, want %d", i, len(got), len(first)) + } + for j := range first { + if got[j] != first[j] { + t.Fatalf("run %d differs at position %d:\n got %q\nwant %q\nwidget property warnings must not depend on map iteration order", + i, j, got[j], first[j]) + } + } + } +} + +// Stable is necessary but not sufficient — it must also be a stable order a +// reader can predict. Sorted by property key is the one the fix chose. +func TestStaticWidgetUnknownProps_OrderIsSorted(t *testing.T) { + w := widgetWithUnknownProps() + got := messagesOf(validateStaticWidgetUnknownProps(w, "page M.P")) + + keys := make([]string, 0, len(w.Properties)) + for k := range w.Properties { + keys = append(keys, k) + } + sort.Strings(keys) + + if len(got) != len(keys) { + t.Fatalf("got %d violations for %d properties", len(got), len(keys)) + } + for i, k := range keys { + if !strings.Contains(got[i], "`"+k+"`") { + t.Errorf("violation %d = %q, want it to be about property %q — warnings should follow sorted key order", i, got[i], k) + } + } +} + +// sortedPropertyKeys is the shared helper; a nil widget must not panic, since +// the validators are called on trees built from partial parses. +func TestSortedPropertyKeys(t *testing.T) { + if got := sortedPropertyKeys(nil); got != nil { + t.Errorf("sortedPropertyKeys(nil) = %v, want nil", got) + } + if got := sortedPropertyKeys(&ast.WidgetV3{}); len(got) != 0 { + t.Errorf("sortedPropertyKeys(no properties) = %v, want empty", got) + } + w := &ast.WidgetV3{Properties: map[string]any{"b": 1, "a": 2, "c": 3}} + got := sortedPropertyKeys(w) + want := []string{"a", "b", "c"} + for i := range want { + if got[i] != want[i] { + t.Errorf("sortedPropertyKeys = %v, want %v", got, want) + break + } + } +} diff --git a/mdl/executor/widget_defs.go b/mdl/executor/widget_defs.go index 7f838d14be..7882892b36 100644 --- a/mdl/executor/widget_defs.go +++ b/mdl/executor/widget_defs.go @@ -796,12 +796,17 @@ func widgetDocMarkdown(mpkDef *mpk.WidgetDefinition, def *WidgetDefinition, mdlN buf.WriteString(fmt.Sprintf("%s '%s' widget1", prefix, mpkDef.ID)) if def != nil && (len(def.ChildSlots) > 0 || len(def.ObjectLists) > 0) { buf.WriteString(" {\n") - for _, slot := range def.ChildSlots { - buf.WriteString(fmt.Sprintf(" %s {\n -- widgets for `%s`\n }\n", strings.ToLower(slot.MDLContainer), slot.PropertyKey)) + // Names are required — `controlbar cb1 { … }`, never `controlbar { … }` + // — and must be unique within the page, so they are numbered. Emitting + // the nameless form made even the slots that DO parse unusable as + // written (mendixlabs/mxcli#1036). + for i, slot := range def.ChildSlots { + buf.WriteString(fmt.Sprintf(" %s slot%d {\n -- widgets for `%s`\n }\n", + strings.ToLower(slot.MDLContainer), i+1, slot.PropertyKey)) } - for _, ol := range def.ObjectLists { - itemKw := strings.ToLower(ol.MDLContainer) - buf.WriteString(fmt.Sprintf(" %s item1 -- one entry of `%s`\n", itemKw, ol.PropertyKey)) + for i, ol := range def.ObjectLists { + buf.WriteString(fmt.Sprintf(" %s item%d -- one entry of `%s`\n", + strings.ToLower(ol.MDLContainer), i+1, ol.PropertyKey)) } buf.WriteString("}\n") } else { diff --git a/mdl/executor/widget_describe.go b/mdl/executor/widget_describe.go new file mode 100644 index 0000000000..a61b14b839 --- /dev/null +++ b/mdl/executor/widget_describe.go @@ -0,0 +1,834 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "fmt" + "io" + "path/filepath" + "sort" + "strings" + + mdlerrors "github.com/mendixlabs/mxcli/mdl/errors" + "github.com/mendixlabs/mxcli/mdl/types" + "github.com/mendixlabs/mxcli/mdl/visitor" + mwidgets "github.com/mendixlabs/mxcli/modelsdk/widgets" + mmpk "github.com/mendixlabs/mxcli/modelsdk/widgets/mpk" +) + +// DescribeWidget assembles everything mxcli knows about one widget: its +// properties (key, type, caption, category, required, default, enum options) +// and the dynamic rules its editor uses to hide properties under some +// configurations. +// +// It exists in the executor rather than in cmd/ so that the MDL statement +// `DESCRIBE WIDGET x` and the CLI `mxcli widget describe x` are the same code. +// That is the point of the statement: a widget was the one MDL extension point +// with no in-language DESCRIBE, which is why `mxcli widget init` had to generate +// documentation at all — and why that documentation could drift from what the +// parser accepts (mendixlabs/mxcli#1036). +// +// arg is an MDL keyword (COMBOBOX), a widget id +// (com.mendix.widget.web.combobox.Combobox), or one of a few built-in aliases. +// projectPath may be empty, in which case only mxcli's embedded knowledge is +// available and the answer is correspondingly thinner. +func DescribeWidget(arg, projectPath string) (*WidgetDescription, error) { + registry, err := NewWidgetRegistry() + if err != nil { + return nil, mdlerrors.NewBackend("widget registry init", err) + } + if projectPath != "" { + _ = registry.LoadUserDefinitions(projectPath) + } + + widgetID, def := resolveWidgetTarget(registry, arg) + if widgetID == "" { + return nil, widgetNotFoundError(registry, arg) + } + + desc := WidgetDescription{WidgetID: widgetID} + if def != nil { + desc.MDLName = def.MDLName + desc.Kind = def.WidgetKind + } + if desc.Kind == "" { + desc.Kind = "pluggable" + } + + // Properties + version: prefer the project's installed .mpk (version-accurate, + // and the only place a Marketplace widget appears); else mxcli's embedded + // template. + if projectPath != "" { + if dir := projectDirOf(projectPath); dir != "" { + if mpkPath, ferr := mmpk.FindMPK(dir, widgetID); ferr == nil && mpkPath != "" { + if wd, perr := mmpk.ParseMPKForWidget(mpkPath, widgetID); perr == nil && wd != nil { + desc.Name = wd.Name + desc.Version = wd.Version + desc.Source = "project .mpk" + desc.Properties = propsFromMPK(wd) + desc.Rules, desc.RuleCoverage = rulesFromProject(mpkPath, widgetID) + } + } + } + } + desc.Containers = describeContainers(def) + + if desc.Source == "" { + tmpl, terr := mwidgets.GetTemplate(widgetID) + if terr != nil || tmpl == nil { + return nil, mdlerrors.NewNotFoundMsg("widget", arg, + "no installed .mpk and no embedded template for "+arg+ + " — open the project with -p to inspect a widget it has installed") + } + desc.Name = tmpl.Name + desc.Version = tmpl.Version + desc.Source = "embedded template" + desc.Properties = propsFromTemplate(tmpl.Type) + if def != nil { + desc.Rules = rulesFromDef(def.PropertyVisibility) + } + } + desc.defDefaults = definitionDefaults(def) + desc.Example, desc.OmittedFromExample = buildUsageExample(desc) + return &desc, nil +} + +type DescribedProperty struct { + Key string `json:"key"` + Type string `json:"type"` + Caption string `json:"caption,omitempty"` + Category string `json:"category,omitempty"` + Required bool `json:"required"` + Default string `json:"default,omitempty"` + System bool `json:"system,omitempty"` + Enum []string `json:"enum,omitempty"` + Children []DescribedProperty `json:"children,omitempty"` +} + +// DescribedRule is one dynamic (visibility) rule of a widget's discovered format. +type DescribedRule struct { + Property string `json:"property"` + HiddenWhen string `json:"hiddenWhen"` + // Cond is the same condition in machine form. Kept alongside the English + // so the usage example can EVALUATE it: a widget's required properties are + // required only where visible, and Combo box lists eleven bindings of which + // its mutually exclusive options-source modes leave about two. + Cond *types.WidgetVisibilityCondition `json:"-"` + // Nested marks a rule about an object-list ITEM's property rather than the + // widget's own. Those are evaluated against the item, never the widget. + Nested bool `json:"-"` +} + +// WidgetDescription is the full inspection result (also the JSON shape). +type WidgetDescription struct { + WidgetID string `json:"widgetId"` + MDLName string `json:"mdlName,omitempty"` + Name string `json:"name,omitempty"` + Version string `json:"version,omitempty"` + Source string `json:"source"` // "project .mpk" | "embedded template" + Kind string `json:"kind,omitempty"` + Properties []DescribedProperty `json:"properties"` + Rules []DescribedRule `json:"dynamicRules"` + RuleCoverage string `json:"ruleCoverage,omitempty"` + // Containers a widget's body can hold: child slots (a curly-brace block of + // widgets) and object lists (repeating entries). Reported with whether MDL + // can currently express each one, which is NOT a given — see + // DescribedContainer.Authorable. + Containers []DescribedContainer `json:"containers,omitempty"` + // Example is re-executable MDL placing this widget, and OmittedFromExample + // says what it leaves out. Both are derived by parsing, so the example is + // guaranteed to parse and widens on its own as the grammar does. + Example string `json:"example,omitempty"` + OmittedFromExample []string `json:"omittedFromExample,omitempty"` + + // defDefaults is the value mxcli's own WidgetDefinition gives each property + // when a script does not set one — the mapping's `default`, else a primitive + // mapping's `value`. Unexported, so it never reaches the JSON output; it + // exists only so the example's hide-rule narrowing resolves a property to + // the SAME value MDL-WIDGET10 will. + // + // The .mpk alone is not enough: a selection property declares no + // defaultValue there, so gallery's `itemSelection` looked indeterminable + // and the example emitted the `keepSelection` that "Single" hides. + defDefaults map[string]string +} + +// DescribedContainer is one child slot or object list of a widget. +type DescribedContainer struct { + Keyword string `json:"keyword"` + PropertyKey string `json:"propertyKey"` + Kind string `json:"kind"` // "child slot" | "object list" + ItemKeys []string `json:"itemKeys,omitempty"` + // Authorable reports whether ` name (…)` actually parses inside a + // widget body today. It is derived by parsing a probe, never from a list: + // the bug this description exists to help with (mendixlabs/mxcli#1036) was + // two lists of keywords with nothing comparing them, and a third list here + // would be the same mistake one layer up. + Authorable bool `json:"authorable"` + + // items carries each sub-property's writable value as the WIDGET DEFINITION + // records it — the mapping's `default`/`value` and its enumValues. Unexported, + // so the JSON shape is unchanged. + // + // The .mpk is not always enough: ParseMPKForWidget returns 0 children for a + // PopupMenu's `basicItems`, while the definition carries + // {"propertyKey":"itemType","value":"item","enumValues":["item","divider"]}. + // MDL-WIDGET08 reads the definition, so the example has to as well or the two + // disagree about the same sub-property. + items []DescribedProperty +} + +func resolveWidgetTarget(registry *WidgetRegistry, arg string) (string, *WidgetDefinition) { + if strings.Contains(arg, ".") { + if def, ok := registry.GetByWidgetID(arg); ok { + return arg, def + } + return arg, nil // unknown to the registry, but a valid id to look up in the project + } + upper := strings.ToUpper(arg) + if def, ok := registry.Get(upper); ok { + return def.WidgetID, def + } + // Well-known widgets that are special-cased in the executor (no .def.json in the + // registry) but that users still name by keyword. + if id, ok := builtinWidgetAliases[upper]; ok { + def, _ := registry.GetByWidgetID(id) + return id, def + } + return "", nil +} + +// builtinWidgetAliases maps MDL keywords for executor-special-cased widgets (which +// have no .def.json registry entry) to their widget ids, so `widget describe` can +// resolve them by the same friendly names users write in MDL. +var builtinWidgetAliases = map[string]string{ + "DATAGRID": "com.mendix.widget.web.datagrid.Datagrid", + "DATAGRID2": "com.mendix.widget.web.datagrid.Datagrid", +} + +// widgetNotFoundError builds a helpful error listing the known MDL names. +func widgetNotFoundError(registry *WidgetRegistry, arg string) error { + var names []string + for _, d := range registry.All() { + if d.MDLName != "" { + names = append(names, d.MDLName) + } + } + for alias := range builtinWidgetAliases { + names = append(names, strings.ToLower(alias)) + } + sort.Strings(names) + return fmt.Errorf("unknown widget %q — use an MDL keyword (%s) or a full widget id (com.mendix.widget…). Run `mxcli widget list` to see all", + arg, strings.Join(names, ", ")) +} + +// projectDirOf returns the directory containing widgets/ for a project path +// (accepts either the .mpr file or its directory). +func projectDirOf(projectPath string) string { + if strings.EqualFold(filepath.Ext(projectPath), ".mpr") { + return filepath.Dir(projectPath) + } + return projectPath +} + +// propsFromMPK builds described properties from a parsed .mpk definition, in the +// widget's declared order (regular + system interleaved). +func propsFromMPK(wd *mmpk.WidgetDefinition) []DescribedProperty { + order := wd.AllTopLevel + if len(order) == 0 { + order = wd.Properties + } + out := make([]DescribedProperty, 0, len(order)) + for _, p := range order { + out = append(out, describedPropFromMPK(p)) + } + return out +} + +func describedPropFromMPK(p mmpk.PropertyDef) DescribedProperty { + dp := DescribedProperty{ + Key: p.Key, + Type: p.Type, + Caption: p.Caption, + Category: p.Category, + Required: p.Required, + Default: p.DefaultValue, + System: p.IsSystem, + } + if dp.System && dp.Type == "" { + dp.Type = "system" + } + for _, ev := range p.EnumValues { + dp.Enum = append(dp.Enum, ev.Key) + } + for _, c := range p.Children { + dp.Children = append(dp.Children, describedPropFromMPK(c)) + } + return dp +} + +// propsFromTemplate walks an embedded template's Type map (ObjectType.PropertyTypes) +// to build described properties. Used when no project .mpk is available. +func propsFromTemplate(typ map[string]any) []DescribedProperty { + objType, _ := typ["ObjectType"].(map[string]any) + pts, _ := objType["PropertyTypes"].([]any) + var out []DescribedProperty + for _, pt := range pts { + m, ok := pt.(map[string]any) + if !ok { + continue // leading array marker + } + out = append(out, describedPropFromTemplate(m)) + } + return out +} + +func describedPropFromTemplate(m map[string]any) DescribedProperty { + dp := DescribedProperty{ + Key: asString(m["PropertyKey"]), + Caption: asString(m["Caption"]), + Category: asString(m["Category"]), + } + vt, _ := m["ValueType"].(map[string]any) + if vt != nil { + dp.Type = asString(vt["Type"]) + dp.Default = asString(vt["DefaultValue"]) + if r, ok := vt["Required"].(bool); ok { + dp.Required = r + } + if evs, ok := vt["EnumerationValues"].([]any); ok { + for _, ev := range evs { + if em, ok := ev.(map[string]any); ok { + if k := asString(em["_Key"]); k != "" { + dp.Enum = append(dp.Enum, k) + } + } + } + } + if nested, ok := vt["ObjectType"].(map[string]any); ok { + if npts, ok := nested["PropertyTypes"].([]any); ok { + for _, npt := range npts { + if nm, ok := npt.(map[string]any); ok { + dp.Children = append(dp.Children, describedPropFromTemplate(nm)) + } + } + } + } + } + dp.System = isSystemPropKey(dp.Key) + return dp +} + +func isSystemPropKey(key string) bool { + switch key { + case "Label", "Visibility", "Editability", "Name", "TabIndex": + return true + } + return false +} + +// rulesFromProject extracts dynamic rules from the project's installed .mpk editor +// config, returning the rules and a coverage note (recognized / total hide-calls). +func rulesFromProject(mpkPath, widgetID string) ([]DescribedRule, string) { + rules, recognized, total := ExtractWidgetVisibilityStats(mpkPath, widgetID) + coverage := "" + if total > 0 { + coverage = fmt.Sprintf("%d of %d editor hide-rules recognized", recognized, total) + } + return rulesToDescribed(rules), coverage +} + +func rulesFromDef(rules []types.WidgetVisibilityRule) []DescribedRule { + return rulesToDescribed(rules) +} + +func rulesToDescribed(rules []types.WidgetVisibilityRule) []DescribedRule { + out := make([]DescribedRule, 0, len(rules)) + for _, r := range rules { + if r.HiddenWhen == nil { + continue + } + out = append(out, DescribedRule{ + Property: r.PropertyKey, + HiddenWhen: conditionText(r.HiddenWhen), + Cond: r.HiddenWhen, + Nested: r.Nested(), + }) + } + sort.Slice(out, func(i, j int) bool { return out[i].Property < out[j].Property }) + return out +} + +// conditionText renders a visibility condition as readable English. +func conditionText(c *types.WidgetVisibilityCondition) string { + switch c.Operator { + case "eq": + return fmt.Sprintf("%s = %q", c.PropertyKey, c.Value) + case "ne": + return fmt.Sprintf("%s ≠ %q", c.PropertyKey, c.Value) + case "truthy": + return fmt.Sprintf("%s is set", c.PropertyKey) + case "falsy": + return fmt.Sprintf("%s is not set", c.PropertyKey) + default: + return fmt.Sprintf("%s %s %q", c.PropertyKey, c.Operator, c.Value) + } +} + +func PrintWidgetDescription(out io.Writer, d WidgetDescription) { + title := d.Name + if title == "" { + title = d.WidgetID + } + fmt.Fprintf(out, "Widget: %s", title) + if d.MDLName != "" { + fmt.Fprintf(out, " (%s)", d.MDLName) + } + fmt.Fprintln(out) + fmt.Fprintf(out, " ID: %s\n", d.WidgetID) + if d.Version != "" { + fmt.Fprintf(out, " Version: %s\n", d.Version) + } + fmt.Fprintf(out, " Kind: %s\n", d.Kind) + fmt.Fprintf(out, " Source: %s\n", d.Source) + + fmt.Fprintf(out, "\nProperties (%d):\n", countProps(d.Properties)) + printProps(out, d.Properties, 0) + + fmt.Fprintf(out, "\nDynamic property rules (%d):\n", len(d.Rules)) + if len(d.Rules) == 0 { + fmt.Fprintln(out, " (none discovered)") + } + for _, r := range d.Rules { + fmt.Fprintf(out, " %-40s hidden when %s\n", r.Property, r.HiddenWhen) + } + if d.RuleCoverage != "" { + fmt.Fprintf(out, " — %s\n", d.RuleCoverage) + } + + if d.Example != "" { + fmt.Fprintf(out, "\nMDL example (parses as written):\n") + for _, line := range strings.Split(d.Example, "\n") { + fmt.Fprintf(out, " %s\n", line) + } + if len(d.OmittedFromExample) > 0 { + fmt.Fprintf(out, " -- omitted: %s\n", strings.Join(d.OmittedFromExample, "; ")) + } + } + + if len(d.Containers) > 0 { + fmt.Fprintf(out, "\nBody containers (%d):\n", len(d.Containers)) + for _, c := range d.Containers { + mark := " authorable" + if !c.Authorable { + mark = " NOT authorable from MDL yet" + } + fmt.Fprintf(out, " %-34s %-12s -> %s%s\n", c.Keyword, c.Kind, c.PropertyKey, mark) + if len(c.ItemKeys) > 0 { + fmt.Fprintf(out, " %-34s items: %s\n", "", strings.Join(c.ItemKeys, ", ")) + } + } + } +} + +func countProps(props []DescribedProperty) int { + n := 0 + for _, p := range props { + n++ + n += countProps(p.Children) + } + return n +} + +func printProps(out interface{ Write([]byte) (int, error) }, props []DescribedProperty, depth int) { + indent := strings.Repeat(" ", depth+1) + for _, p := range props { + req := "" + if p.Required { + req = " required" + } + sys := "" + if p.System { + sys = " [system]" + } + line := fmt.Sprintf("%s%-34s %-13s", indent, p.Key, p.Type) + extra := strings.TrimRight(req+sys, " ") + if p.Default != "" { + extra = strings.TrimSpace(extra + " default=" + p.Default) + } + if len(p.Enum) > 0 { + extra = strings.TrimSpace(extra + " {" + strings.Join(p.Enum, "|") + "}") + } + if p.Category != "" { + extra = strings.TrimSpace(extra + " (" + p.Category + ")") + } + fmt.Fprintf(out, "%s %s\n", strings.TrimRight(line, " "), extra) + if len(p.Children) > 0 { + printProps(out, p.Children, depth+1) + } + } +} + +// describeWidgetStmt is the DESCRIBE WIDGET handler. It prints exactly what +// `mxcli widget describe` prints, because it is the same function — see +// DescribeWidget for why that matters. +// +// The widget is named by MDL keyword or widget id; unlike every other DESCRIBE +// there is no qualified name, because a widget definition is not a document in +// the model. It comes from a package in the project (or from mxcli's embedded +// set), which is also why this reads no backend and works with no project open. +func describeWidgetStmt(ctx *ExecContext, name string) error { + if name == "" { + return mdlerrors.NewValidation("DESCRIBE WIDGET needs a widget: an MDL keyword (combobox) or a widget id ('com.mendix.widget.web.combobox.Combobox')") + } + projectPath := "" + if ctx != nil && ctx.Backend != nil { + projectPath = ctx.Backend.Path() + } + desc, err := DescribeWidget(name, projectPath) + if err != nil { + return err + } + PrintWidgetDescription(ctx.Output, *desc) + return nil +} + +// describeContainers lists a widget's child slots and object lists, each marked +// with whether MDL can currently express it. +func describeContainers(def *WidgetDefinition) []DescribedContainer { + if def == nil { + return nil + } + var out []DescribedContainer + for _, cs := range def.ChildSlots { + kw := strings.ToLower(cs.MDLContainer) + out = append(out, DescribedContainer{ + Keyword: kw, PropertyKey: cs.PropertyKey, Kind: "child slot", + Authorable: containerKeywordParses(kw, true), + }) + } + for _, ol := range def.ObjectLists { + kw := strings.ToLower(ol.MDLContainer) + c := DescribedContainer{ + Keyword: kw, PropertyKey: ol.PropertyKey, Kind: "object list", + Authorable: containerKeywordParses(kw, false), + } + for _, ip := range ol.ItemProperties { + c.ItemKeys = append(c.ItemKeys, ip.PropertyKey) + def := ip.Default + if def == "" && ip.Operation == "primitive" { + def = ip.Value + } + c.items = append(c.items, DescribedProperty{ + Key: ip.PropertyKey, Type: ip.Operation, Default: def, Enum: ip.EnumValues, + }) + } + out = append(out, c) + } + sort.Slice(out, func(i, j int) bool { return out[i].Keyword < out[j].Keyword }) + return out +} + +// containerKeywordParses answers "can I write this inside a widget body?" by +// parsing a minimal page and checking for errors — deriving the answer from the +// grammar itself rather than restating it. +func containerKeywordParses(keyword string, slot bool) bool { + if keyword == "" { + return false + } + body := keyword + " probe1 (x: 'y')" + if slot { + body = keyword + " probe1 { dynamictext t (Content: 'x') }" + } + src := "create page Probe.P (Title: 'P', Layout: Atlas_Core.Atlas_Default) {\n" + + " pluggablewidget 'probe.Widget' pw {\n " + body + "\n }\n}\n" + _, errs := visitor.Build(src) + return len(errs) == 0 +} + +// buildUsageExample renders MDL that places this widget, and returns it with a +// list of what it left out. +// +// Two rules make it worth printing at all, both learned from the generated .md +// this replaces (mendixlabs/mxcli#1036): +// +// 1. It emits only what PARSES. The head form, and every container, is chosen +// by probing the real parser — so the example corrects itself as the grammar +// gains ground, and cannot drift the way a hand-written template did. +// 2. It says what it omitted and why. The .md's example silently included +// containers that could not be written, which is what made it misleading +// rather than merely incomplete. +// +// The result is verified by parsing it before returning; if it somehow does not +// parse, the caller is told rather than handed a broken snippet. +func buildUsageExample(d WidgetDescription) (example string, omitted []string) { + name := "widget1" + + // Head: the widget's own keyword when the grammar takes it, else the + // explicit-id form. Probed, never assumed. + head := "pluggablewidget '" + d.WidgetID + "' " + name + if kw := strings.ToLower(d.MDLName); kw != "" && widgetKeywordParses(kw) { + head = kw + " " + name + } + + // Scalars: only those whose value can be written as a literal. A datasource, + // attribute, action or expression needs a real name from the project, and + // inventing one would produce an example that parses but cannot run. + var props []string + var needBinding []string + for _, p := range d.Properties { + if p.System { + continue + } + // The two sources spell property types differently — a project .mpk + // gives "datasource", the embedded template "DataSource" — so this + // folds case. Matching only one spelling silently emptied the example + // for every widget described without a project. + // hiddenUnder applies to BOTH branches. It used to gate only the + // binding branch, so the example emitted literals its own configuration + // hides — `heightUnit: 'aspectRatio'` followed by the `height` that + // choice hides — and mxcli's own MDL-WIDGET10 then warned about 32 of + // them. The generator and the checker implement the same editorConfig + // rules; disagreeing is worse than either alone, because the example is + // what a reader copies. + if hiddenUnder(d, p.Key) { + continue + } + switch strings.ToLower(p.Type) { + case "boolean", "integer", "enumeration", "string", "texttemplate": + if p.Required { + props = append(props, " "+p.Key+": "+exampleLiteral(p)) + } + case "attribute", "datasource", "action", "expression", "selection": + if p.Required { + needBinding = append(needBinding, p.Key+" ("+strings.ToLower(p.Type)+")") + } + } + } + + // Names are numbered across the whole body: two widgets sharing a name on + // one page is invalid, and the parser does not catch it — the same defect + // the generated .md had. + var body []string + n := 0 + for _, c := range d.Containers { + if !c.Authorable { + omitted = append(omitted, c.Keyword) + continue + } + n++ + if c.Kind == "child slot" { + body = append(body, fmt.Sprintf(" %s slot%d {\n -- widgets for `%s`\n }", c.Keyword, n, c.PropertyKey)) + continue + } + item := fmt.Sprintf(" %s item%d", c.Keyword, n) + if k, lit := itemExampleLiteral(d, c); k != "" { + item += " (" + k + ": " + lit + ")" + } + body = append(body, item+" -- one entry of `"+c.PropertyKey+"`") + } + + var sb strings.Builder + sb.WriteString(head) + if len(props) > 0 { + sb.WriteString(" (\n" + strings.Join(props, ",\n") + "\n)") + } + if len(body) > 0 { + sb.WriteString(" {\n" + strings.Join(body, "\n") + "\n}") + } + out := sb.String() + + if !pageBodyParses(out) { + return "", append(omitted, "(example could not be generated for this widget)") + } + for _, n := range needBinding { + omitted = append(omitted, n+" — needs a name from your project") + } + return out, omitted +} + +// itemExampleLiteral picks the sub-property to show on an object-list item, and +// a value for it that the validator will accept. +// +// It used to take ItemKeys[0] and write `'…'`. For an ENUMERATION sub-property +// that is simply a wrong value, and mxcli's own MDL-WIDGET08 said so — "property +// `dataSet` has invalid value `…` — valid values are static, dynamic" — on 11 of +// the fixture's examples. The block claims to parse as written, and it did; it +// just did not CHECK as written, which is the more useful promise. +// +// The values were already in hand: propsFromMPK carries an object-list +// property's sub-properties as Children, with their enums and defaults, so the +// same exampleLiteral used for the widget's own scalars applies here. +// +// Preference order is deliberate: a sub-property with a derivable literal (a +// default, or an enumeration's first member) beats one without, because for a +// free-text sub-property there is no correct value to invent and a placeholder +// is the honest output — and the validator accepts any string, so it costs +// nothing. +func itemExampleLiteral(d WidgetDescription, c DescribedContainer) (key, literal string) { + if len(c.ItemKeys) == 0 { + return "", "" + } + children := map[string]DescribedProperty{} + for _, p := range d.Properties { + if !strings.EqualFold(p.Key, c.PropertyKey) { + continue + } + for _, ch := range p.Children { + children[strings.ToLower(ch.Key)] = ch + } + } + // The definition WINS, for the same reason it does in exampleValues: it is + // what MDL-WIDGET08 checks the value against. + for _, it := range c.items { + if it.Default == "" && len(it.Enum) == 0 { + continue + } + children[strings.ToLower(it.Key)] = it + } + for _, k := range c.ItemKeys { + ch, ok := children[strings.ToLower(k)] + if !ok { + continue + } + if ch.Default == "" && len(ch.Enum) == 0 { + continue // nothing to derive; keep looking for one that has something + } + return k, exampleLiteral(ch) + } + // No sub-property offers a value. Show the first key with a placeholder — + // it is a free-text slot, which the validator accepts. + return c.ItemKeys[0], "'…'" +} + +// exampleLiteral picks a writable value for a scalar property: its default when +// it has one, else the first enumeration value, else a placeholder. +func exampleLiteral(p DescribedProperty) string { + switch strings.ToLower(p.Type) { + case "boolean": + if p.Default != "" { + return p.Default + } + return "false" + case "integer": + if p.Default != "" { + return p.Default + } + return "0" + } + if p.Default != "" { + return "'" + p.Default + "'" + } + if len(p.Enum) > 0 { + return "'" + p.Enum[0] + "'" + } + return "'…'" +} + +// widgetKeywordParses reports whether ` name (…)` is accepted as a +// widget in a page body — the head-form half of the same probe the containers use. +func widgetKeywordParses(keyword string) bool { + return pageBodyParses(keyword + " probe1 (someProp: 'x')") +} + +// pageBodyParses puts a fragment in a minimal page and reports whether it parses. +func pageBodyParses(body string) bool { + src := "create page Probe.P (Title: 'P', Layout: Atlas_Core.Atlas_Default) {\n" + body + "\n}\n" + _, errs := visitor.Build(src) + return len(errs) == 0 +} + +// exampleValues is the configuration the example describes: each scalar +// property's default, which is also what the example writes for the required +// ones. Visibility rules are evaluated against this. +func exampleValues(d WidgetDescription) map[string]string { + values := map[string]string{} + var walk func(props []DescribedProperty) + walk = func(props []DescribedProperty) { + for _, p := range props { + if p.Default != "" { + values[p.Key] = p.Default + } else if len(p.Enum) > 0 { + // An enumeration with no declared default takes its first value, + // which is what Mendix shows in the editor. + values[p.Key] = p.Enum[0] + } + walk(p.Children) + } + } + walk(d.Properties) + // The definition's default WINS over the package's. It is what mxcli writes + // when a script is silent, and therefore what the validator concludes the + // property holds — which is the whole point of resolving it here. + for k, v := range d.defDefaults { + if v != "" { + values[k] = v + } + } + return values +} + +// definitionDefaults collects the value each mapping falls back to, mirroring +// widgetValueMap's own order: an explicit `default`, else a primitive mapping's +// `value` (the widget XML's defaultValue, which is where the generator and the +// checker have to agree). +func definitionDefaults(def *WidgetDefinition) map[string]string { + if def == nil { + return nil + } + out := map[string]string{} + collect := func(mappings []PropertyMapping) { + for _, m := range mappings { + if m.PropertyKey == "" { + continue + } + switch { + case m.Default != "": + out[m.PropertyKey] = m.Default + case m.Operation == "primitive" && m.Value != "": + out[m.PropertyKey] = m.Value + case m.Operation == "selection": + // An omitted `Selection:` is WRITTEN as None — the builder's own + // behaviour, not a guess — and a selection property declares no + // defaultValue in the .mpk, which is why the generator saw + // DataGrid2's `itemSelection` as indeterminable and emitted the + // `itemSelectionMethod` that "None" hides. Same reasoning, and + // same three branches, as widgetValueMap. + out[m.PropertyKey] = "None" + } + if m.Operation == "selection" { + out[m.PropertyKey] = canonicalSelection(out[m.PropertyKey]) + } + } + } + collect(def.PropertyMappings) + for _, mode := range def.Modes { + collect(mode.PropertyMappings) + } + return out +} + +// hiddenUnder reports whether a property is hidden in the configuration the +// example describes, so its binding need not be asked for. +// +// Conservative in the direction of asking too much rather than too little: a +// rule whose condition property has no determinable value does NOT prune, and +// nested rules (about an object-list item) never apply to the widget itself. +// Over-listing a binding costs the reader a moment; hiding one they actually +// need would send them to a build error, which is the failure this whole area +// keeps producing. +func hiddenUnder(d WidgetDescription, propertyKey string) bool { + values := exampleValues(d) + for _, r := range d.Rules { + if r.Nested || r.Cond == nil || !strings.EqualFold(r.Property, propertyKey) { + continue + } + if _, known := values[r.Cond.PropertyKey]; !known { + continue // indeterminable — do not guess, keep asking for it + } + if r.Cond.Hidden(values) { + return true + } + } + return false +} diff --git a/mdl/executor/widget_describe_moved_test.go b/mdl/executor/widget_describe_moved_test.go new file mode 100644 index 0000000000..43691354e3 --- /dev/null +++ b/mdl/executor/widget_describe_moved_test.go @@ -0,0 +1,42 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "testing" + + "github.com/mendixlabs/mxcli/mdl/types" +) + +func TestWidgetDescribe_UnknownWidget(t *testing.T) { + reg, err := NewWidgetRegistry() + if err != nil { + t.Fatalf("registry: %v", err) + } + id, _ := resolveWidgetTarget(reg, "NOPE") + if id != "" { + t.Errorf("resolveWidgetTarget(NOPE) = %q, want empty", id) + } + // DATAGRID2 resolves via the builtin alias even without a .def.json entry. + if id, _ := resolveWidgetTarget(reg, "datagrid2"); id != "com.mendix.widget.web.datagrid.Datagrid" { + t.Errorf("resolveWidgetTarget(datagrid2) = %q", id) + } +} + +// TestConditionText renders the four operators as readable English. +func TestConditionText(t *testing.T) { + cases := []struct { + op, val, want string + }{ + {"eq", "None", `itemSelection = "None"`}, + {"ne", "Multi", `itemSelection ≠ "Multi"`}, + {"truthy", "", "itemSelection is set"}, + {"falsy", "", "itemSelection is not set"}, + } + for _, c := range cases { + got := conditionText(&types.WidgetVisibilityCondition{PropertyKey: "itemSelection", Operator: c.op, Value: c.val}) + if got != c.want { + t.Errorf("op %s: got %q, want %q", c.op, got, c.want) + } + } +} diff --git a/mdl/executor/widget_describe_test.go b/mdl/executor/widget_describe_test.go new file mode 100644 index 0000000000..6395ba9a94 --- /dev/null +++ b/mdl/executor/widget_describe_test.go @@ -0,0 +1,372 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/types" +) + +// A widget was the only MDL extension point with no in-language DESCRIBE, which +// is why `widget init` had to generate documentation — and why that +// documentation could drift from what the parser accepts. The statement is only +// worth having if it answers without a project, which is the state an agent is +// in when it asks "what can I write here?". +func TestDescribeWidget_AnswersFromEmbeddedKnowledgeWithNoProject(t *testing.T) { + desc, err := DescribeWidget("combobox", "") + if err != nil { + t.Fatalf("DescribeWidget with no project: %v", err) + } + if desc.WidgetID != "com.mendix.widget.web.combobox.Combobox" { + t.Errorf("WidgetID = %q", desc.WidgetID) + } + if desc.Source != "embedded template" { + t.Errorf("Source = %q, want the embedded fallback", desc.Source) + } + if len(desc.Properties) == 0 { + t.Error("no properties — an empty description answers nothing") + } +} + +// The full widget id must work as well as the MDL keyword: it is what a widget +// package, a page's BSON and the generated docs all carry, and for a widget with +// no keyword it is the only name there is. +func TestDescribeWidget_AcceptsTheWidgetIdAsWellAsTheKeyword(t *testing.T) { + byKeyword, err := DescribeWidget("combobox", "") + if err != nil { + t.Fatal(err) + } + byID, err := DescribeWidget("com.mendix.widget.web.combobox.Combobox", "") + if err != nil { + t.Fatal(err) + } + if byKeyword.WidgetID != byID.WidgetID { + t.Errorf("keyword gave %q, id gave %q", byKeyword.WidgetID, byID.WidgetID) + } + if len(byKeyword.Properties) != len(byID.Properties) { + t.Errorf("property counts differ: %d vs %d", len(byKeyword.Properties), len(byID.Properties)) + } +} + +// An unknown widget must say so rather than returning an empty description that +// reads as "this widget has no properties". +func TestDescribeWidget_UnknownWidgetIsAnError(t *testing.T) { + if _, err := DescribeWidget("notawidget", ""); err == nil { + t.Fatal("want an error for an unknown widget, got none") + } +} + +// The project's installed .mpk is preferred over the embedded template, because +// it is version-accurate and is the only place a Marketplace widget appears. +// This is the control for the no-project test above: without it, "embedded +// template" there is equally consistent with the .mpk path never running. +func TestDescribeWidget_PrefersTheProjectPackageOverTheEmbeddedTemplate(t *testing.T) { + desc, err := DescribeWidget("combobox", "../../testdata/expr-checker/minimal.mpr") + if err != nil { + t.Skipf("fixture unavailable: %v", err) + } + if desc.Source != "project .mpk" { + t.Errorf("Source = %q, want the project package to win", desc.Source) + } +} + +// The rendered form is what a reader actually sees, and it is shared with +// `mxcli widget describe` — so a change that broke it would break both. +func TestPrintWidgetDescription_RendersTheHeaderAndProperties(t *testing.T) { + desc, err := DescribeWidget("combobox", "") + if err != nil { + t.Fatal(err) + } + var sb strings.Builder + PrintWidgetDescription(&sb, *desc) + out := sb.String() + for _, want := range []string{"Widget:", "ID:", "Kind:", "Properties ("} { + if !strings.Contains(out, want) { + t.Errorf("rendered output missing %q:\n%s", want, out) + } + } +} + +// The containers are the half of a widget's shape that DESCRIBE WIDGET was +// missing relative to the generated .md — and the half whose MDL syntax is +// currently wrong for most widgets, so reporting them without saying which are +// reachable would repeat the .md's mistake. +// +// Gallery is used because it is an EMBEDDED definition carrying containers on +// both sides of the answer, so this needs no project and cannot skip. An +// earlier version pointed at the fixture project and skipped every run, since +// the fixture has no extracted defs — a test that only ever skips proves +// nothing (see #808 in fix-issue.md). +func TestDescribeWidget_ReportsContainersAndWhetherTheyAreAuthorable(t *testing.T) { + desc, err := DescribeWidget("gallery", "") + if err != nil { + t.Fatalf("DescribeWidget(gallery): %v", err) + } + byKeyword := map[string]DescribedContainer{} + for _, c := range desc.Containers { + byKeyword[c.Keyword] = c + } + if len(byKeyword) == 0 { + t.Fatal("no containers reported for a widget that has three") + } + + // Both of these used to sit on opposite sides of the answer: `template` was + // in the grammar's container vocabulary and `emptyplaceholder` was not. + // Slices 2-3 removed that boundary, so both are authorable now — measured + // across the fixture's definitions, 50 of 50 containers are. + // + // The assertion is kept pointing at the SAME two containers deliberately. + // It is the regression test for the capability: if `emptyplaceholder` ever + // reports unauthorable again, the def-driven body has been lost. + for _, kw := range []string{"emptyplaceholder", "template"} { + c, ok := byKeyword[kw] + if !ok { + t.Errorf("%s missing; got %v", kw, desc.Containers) + continue + } + if !c.Authorable { + t.Errorf("%s reported unauthorable — since slices 2-3 every container a definition "+ + "declares can be written (mendixlabs/mxcli#1036)", kw) + } + } +} + +// Authorability is derived by parsing, never from a list. A list here would be +// the same defect the whole proposal is about, one layer up — so an invented +// keyword must come back false through the same path a real one comes back true. +func TestContainerKeywordParses_DerivesTheAnswerRatherThanListingIt(t *testing.T) { + if !containerKeywordParses("group", false) { + t.Error("group should parse as an object-list container") + } + // An invented keyword now PARSES — that is what slices 2-3 did, and it is + // why the wrong-name check moved to the validator (MDL-WIDGET25/26), which + // can consult the parent's definition where the parser cannot. + if !containerKeywordParses("definitelynotakeyword", false) { + t.Error("since slice 3 any name parses in a container position; the check that it is a " + + "REAL container belongs to MDL-WIDGET26, not to the parser") + } + // The probe must still be a real probe. A body that is malformed for a + // reason unrelated to the keyword has to come back false, or "authorable" + // would be a constant dressed up as a derivation — the exact defect this + // test exists to prevent, one layer up. + if containerKeywordParses("group (", false) { + t.Error("a malformed probe reported authorable — the parse probe is not actually running") + } + if containerKeywordParses("", false) { + t.Error("an empty keyword must not report as authorable") + } +} + +// The example is the half of the generated .md that was WRONG — its version +// promised syntax that failed on its own first line. This one is built from +// probes, so the guarantee is testable: feed it back and it parses. +func TestDescribeWidget_ExampleParsesAsWritten(t *testing.T) { + for _, w := range []string{"gallery", "combobox", "image"} { + desc, err := DescribeWidget(w, "") + if err != nil { + t.Fatalf("DescribeWidget(%s): %v", w, err) + } + if desc.Example == "" { + t.Errorf("%s: no example emitted", w) + continue + } + if !pageBodyParses(desc.Example) { + t.Errorf("%s: emitted example does not parse:\n%s", w, desc.Example) + } + } +} + +// A container MDL cannot express must be left out AND named. Silently including +// it is what made the .md misleading rather than merely incomplete; silently +// dropping it would be almost as bad, since the reader would never learn the +// widget has it. +func TestDescribeWidget_ExampleOmitsWhatItCannotFillAndSaysSo(t *testing.T) { + desc, err := DescribeWidget("gallery", "") + if err != nil { + t.Fatal(err) + } + + // Since slices 2-3 no container is omitted for being unwritable — all three + // of Gallery's appear. This half is the capability regression test: it fails + // if the def-driven body is lost. + for _, kw := range []string{"emptyplaceholder", "filter", "template"} { + if !strings.Contains(desc.Example, kw) { + t.Errorf("container %q missing from the example — every container a definition declares "+ + "should now be writable:\n%s", kw, desc.Example) + } + } + + // The omission machinery still has a job: a BINDING cannot be invented, + // because it needs a name from the reader's own project. Those must be left + // out and named, which is what stopped the generated .md from promising + // syntax that failed. + if len(desc.OmittedFromExample) == 0 { + t.Fatal("nothing reported as omitted — Gallery's datasource cannot be filled in, " + + "so the example must say so rather than invent one") + } + var named bool + for _, o := range desc.OmittedFromExample { + if strings.Contains(strings.ToLower(o), "datasource") { + named = true + } + } + if !named { + t.Errorf("the unfillable datasource is not named among the omissions; got %v", desc.OmittedFromExample) + } + if strings.Contains(desc.Example, "DataSource:") { + t.Errorf("the example invented a datasource instead of omitting it:\n%s", desc.Example) + } +} + +// Two widgets sharing a name on one page is invalid, and the parser does not +// catch it — so the example has to number them itself. The .md generator had +// this same defect. +func TestDescribeWidget_ExampleNamesAreUnique(t *testing.T) { + desc, err := DescribeWidget("gallery", "") + if err != nil { + t.Fatal(err) + } + seen := map[string]bool{} + for _, line := range strings.Split(desc.Example, "\n") { + fields := strings.Fields(strings.TrimSpace(line)) + if len(fields) < 2 { + continue + } + name := fields[1] + if !strings.HasPrefix(name, "slot") && !strings.HasPrefix(name, "item") { + continue + } + if seen[name] { + t.Errorf("duplicate widget name %q in example:\n%s", name, desc.Example) + } + seen[name] = true + } + if len(seen) < 2 { + t.Skip("widget has fewer than two named containers; nothing to collide") + } +} + +// The head form is probed too: a widget whose keyword the grammar accepts uses +// it, and one whose keyword it does not falls back to the explicit-id form. +// Both halves in one test, so neither can pass vacuously. +func TestDescribeWidget_ExampleHeadFormFollowsTheGrammar(t *testing.T) { + authorable, err := DescribeWidget("gallery", "") + if err != nil { + t.Fatal(err) + } + if !strings.HasPrefix(authorable.Example, "gallery widget1") { + t.Errorf("gallery's keyword parses, so the example should use it:\n%s", authorable.Example) + } + + notYet, err := DescribeWidget("image", "") + if err != nil { + t.Fatal(err) + } + if strings.HasPrefix(notYet.Example, "pluggablewidget") == strings.HasPrefix(notYet.Example, "image widget1") { + t.Fatalf("indeterminate head form:\n%s", notYet.Example) + } +} + +// A widget's properties are "required" only where the editor shows them. Combo +// box declares eleven bindings across mutually exclusive options-source modes, +// so listing them all overstates what a reader must supply — the same class of +// misinformation as the generated .md, quieter: not syntax that fails, but work +// that is not needed. +// +// Needs a project: the rules come from the .mpk's editorConfig, so the embedded +// template carries none. Asserted below rather than left to be discovered. +func TestDescribeWidget_BindingsHiddenUnderTheExampleAreNotAskedFor(t *testing.T) { + const fixture = "../../testdata/expr-checker/minimal.mpr" + desc, err := DescribeWidget("combobox", fixture) + if err != nil { + t.Fatalf("DescribeWidget(combobox, fixture): %v", err) + } + if len(desc.Rules) == 0 { + t.Fatal("no visibility rules from the project .mpk — nothing could be pruned, so this would pass vacuously") + } + omitted := strings.Join(desc.OmittedFromExample, "; ") + + // attributeBoolean is hidden when optionsSourceType ≠ "boolean", and the + // default is "association" — so the rule fires and it must not be asked for. + if strings.Contains(omitted, "attributeBoolean") { + t.Errorf("attributeBoolean is hidden under the example's configuration but was still asked for:\n%s", omitted) + } +} + +// The control. Pruning must not simply drop every binding: a widget whose +// datasource nothing hides still has to ask for it, or the example would look +// complete while being unusable. +func TestDescribeWidget_VisibleBindingsAreStillAskedFor(t *testing.T) { + for _, w := range []string{"gallery", "datagrid"} { + desc, err := DescribeWidget(w, "") + if err != nil { + t.Fatalf("%s: %v", w, err) + } + if !strings.Contains(strings.Join(desc.OmittedFromExample, "; "), "datasource") { + t.Errorf("%s: its datasource is not hidden by any rule, so it must still be asked for; got %v", + w, desc.OmittedFromExample) + } + } +} + +// Conservatism, stated as a test: a rule whose condition property has no +// determinable value must NOT prune. Over-listing costs the reader a moment; +// hiding a binding they actually need sends them to a build error, which is the +// failure this area keeps producing. +func TestHiddenUnder_DoesNotPruneOnAnIndeterminableCondition(t *testing.T) { + d := WidgetDescription{ + Properties: []DescribedProperty{ + {Key: "someBinding", Type: "datasource", Required: true}, + // `mode` has no default and no enum, so its value is unknowable. + {Key: "mode", Type: "enumeration"}, + }, + Rules: []DescribedRule{{ + Property: "someBinding", + Cond: &types.WidgetVisibilityCondition{PropertyKey: "mode", Operator: "ne", Value: "x"}, + }}, + } + if hiddenUnder(d, "someBinding") { + t.Error("pruned on a condition whose value cannot be determined") + } + + // The control: give `mode` a default the condition matches, and it prunes. + d.Properties[1].Default = "y" // "y" != "x", so `ne` fires + if !hiddenUnder(d, "someBinding") { + t.Error("did not prune when the condition is determinable and fires") + } +} + +// A rule about an object-list ITEM's property must never prune the WIDGET's +// binding of the same name — they are different properties on different objects. +func TestHiddenUnder_IgnoresNestedItemRules(t *testing.T) { + d := WidgetDescription{ + Properties: []DescribedProperty{ + {Key: "caption", Type: "attribute", Required: true}, + {Key: "mode", Type: "enumeration", Default: "y"}, + }, + Rules: []DescribedRule{{ + Property: "caption", + Nested: true, + Cond: &types.WidgetVisibilityCondition{PropertyKey: "mode", Operator: "ne", Value: "x"}, + }}, + } + if hiddenUnder(d, "caption") { + t.Error("a nested item rule pruned the widget's own binding") + } +} + +// The limitation behind the test above, stated so it is not rediscovered: with +// no project there is no .mpk, so no editorConfig, so no rules — and nothing to +// prune with. The description is still useful, it just cannot narrow the +// bindings. +func TestDescribeWidget_NoProjectMeansNoVisibilityRules(t *testing.T) { + desc, err := DescribeWidget("combobox", "") + if err != nil { + t.Fatal(err) + } + if len(desc.Rules) != 0 { + t.Errorf("embedded combobox unexpectedly carries %d rules; the pruning test's project requirement may be stale", len(desc.Rules)) + } +} diff --git a/mdl/executor/widget_describe_validator_agreement_test.go b/mdl/executor/widget_describe_validator_agreement_test.go new file mode 100644 index 0000000000..e06de80582 --- /dev/null +++ b/mdl/executor/widget_describe_validator_agreement_test.go @@ -0,0 +1,140 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "os" + "path/filepath" + "sort" + "strings" + "testing" +) + +// fixtureProject is a real project with widgets/*.mpk committed. The .def.json +// CACHE under .mxcli/ is gitignored, so a test must not depend on it — which is +// the point here: the nine hand-crafted widgets never get a .def.json anyway, +// and they are exactly the ones this test is about. +func fixtureProject(t *testing.T) string { + t.Helper() + p, err := filepath.Abs(filepath.Join("..", "..", "testdata", "expr-checker", "minimal.mpr")) + if err != nil { + t.Skipf("cannot resolve fixture: %v", err) + } + if _, err := os.Stat(p); err != nil { + t.Skipf("fixture project not present: %v", err) + } + return p +} + +// DESCRIBE WIDGET and the property validator must agree about which properties a +// widget has. They read different sources — DESCRIBE reads the project's +// installed .mpk, the validator reads the WidgetDefinition — and they disagreed. +// +// # The defect +// +// Nine widgets (combobox, gallery, image, barcodescanner, the four data-grid +// filters, dropdownsort) have hand-crafted definitions in sdk/widgets/definitions/ +// and are deliberately never extracted per-project, so no .def.json is ever +// generated for them. Those hand-written definitions cover a fraction of the +// widget: +// +// combobox 73 properties in the .mpk, 7 mapped + 4 known +// gallery 44 12 +// image 37 14 +// barcodescanner 12 1 +// +// So `DESCRIBE WIDGET` emitted an example it labels "parses as written" — and it +// does parse — naming 33 properties that mxcli's OWN validator then rejected with +// MDL-WIDGET01. Since exec refuses before writing, barcodescanner could not be +// placed from MDL at all: include the five properties and exec refuses, omit them +// and mxbuild reports CE0463 "the definition of this widget has changed". +// Combobox was the sharpest case — its describe output marks `source` REQUIRED +// and its validator said the widget has no such property. +// +// Reported by an external test project against 41c55d09 + this PR, retested at +// bca5466e, and reproduced here at 64055caa: combobox 17, gallery 7, +// barcodescanner 5, image 4. +// +// # Why this is the right assertion +// +// Not "the example parses" — it already did. The generator and the validator are +// two readers of one widget, so the invariant is that a property one of them +// emits is not one the other calls nonexistent. +func TestDescribeWidgetPropertiesAreAcceptedByTheValidator(t *testing.T) { + project := fixtureProject(t) + registry := LoadWidgetRegistry(project) + if registry == nil { + t.Fatal("no registry") + } + + // The nine hand-crafted widgets are the population this is about; assert the + // fixture actually has some of them installed, or the test proves nothing. + targets := []string{"combobox", "gallery", "image", "barcodescanner"} + var checked int + var bad []string + + for _, name := range targets { + desc, err := DescribeWidget(name, project) + if err != nil { + continue + } + if desc.Source != "project .mpk" { + // Falling back to the embedded template means the .mpk is absent, and + // then both sides read the same thin data and cannot disagree. + continue + } + def, ok := registry.Get(name) + if !ok || def == nil { + continue + } + checked++ + + allowed, _ := allowedWidgetProperties(def) + known := knownUnmappedProperties(def, allowed) + for _, p := range desc.Properties { + if p.Key == "" || isSystemPropKey(p.Key) { + continue + } + k := strings.ToLower(p.Key) + if allowed[k] || known[k] { + continue + } + bad = append(bad, name+"."+p.Key) + } + } + + if checked == 0 { + t.Skip("none of the hand-crafted widgets are installed in the fixture with a .mpk") + } + if len(bad) > 0 { + sort.Strings(bad) + t.Errorf("%d properties DESCRIBE WIDGET emits are rejected by the validator "+ + "(MDL-WIDGET01 \"has no property\"); the two read different sources and must not "+ + "disagree:\n %s", len(bad), strings.Join(bad, "\n ")) + } +} + +// The control: enrichment must not make the validator accept ANYTHING. A property +// no widget declares is still an error, or MDL-WIDGET01 stops detecting typos — +// which is the rule's whole job. +func TestValidatorStillRejectsAPropertyTheMPKDoesNotDeclare(t *testing.T) { + project := fixtureProject(t) + registry := LoadWidgetRegistry(project) + if registry == nil { + t.Fatal("no registry") + } + def, ok := registry.Get("combobox") + if !ok || def == nil { + t.Skip("combobox not in registry") + } + allowed, _ := allowedWidgetProperties(def) + known := knownUnmappedProperties(def, allowed) + + for _, bogus := range []string{"notarealproperty", "sourceX", "optionsSourceTypo"} { + k := strings.ToLower(bogus) + if allowed[k] || known[k] { + t.Errorf("%q was accepted; enrichment must add the widget's REAL properties, "+ + "not open the gate", bogus) + } + } +} diff --git a/mdl/executor/widget_example_hidden_props_test.go b/mdl/executor/widget_example_hidden_props_test.go new file mode 100644 index 0000000000..2504c2ef3e --- /dev/null +++ b/mdl/executor/widget_example_hidden_props_test.go @@ -0,0 +1,181 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "regexp" + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/visitor" +) + +var examplePropLine = regexp.MustCompile(`(?m)^\s{2}([A-Za-z][A-Za-z0-9_]*)\s*:`) + +// exampleScalarKeys returns the property keys the generated example writes in the +// widget's own head — the ` key: value,` lines, not container item properties. +func exampleScalarKeys(example string) []string { + head := example + if i := strings.Index(head, ") {"); i >= 0 { + head = head[:i] + } + var out []string + for _, m := range examplePropLine.FindAllStringSubmatch(head, -1) { + out = append(out, m[1]) + } + return out +} + +// The generator narrows its example by the widget's editorConfig hide-rules, and +// the validator implements the same rules as MDL-WIDGET10. They disagreed: 32 +// warnings over the fixture's widgets, every one of them naming a property the +// example ITSELF had just emitted. +// +// videoplayer example: heightUnit: 'aspectRatio', … height: 500 +// validator: property `height` is hidden when `heightUnit` is +// "aspectRatio" — the value will be ignored +// +// The generator chose `heightUnit: 'aspectRatio'` and then wrote the `height` +// its own rule hides. Cause: hiddenUnder was consulted only on the branch that +// asks for a BINDING (attribute/datasource/action/expression/selection) and not +// on the scalar branch that emits literals — so the narrowing existed and half +// the properties skipped it. +// +// Reported by an external test project (14 of 14 warnings were self-inflicted +// there); reproduced here at 668ad9ae over all 42 definitions. +func TestUsageExampleDoesNotEmitItsOwnHiddenProperties(t *testing.T) { + project := fixtureProject(t) + registry := LoadWidgetRegistry(project) + if registry == nil { + t.Fatal("no registry") + } + + var offenders []string + var checked int + for _, def := range registry.All() { + if def == nil || def.MDLName == "" { + continue + } + desc, err := DescribeWidget(def.MDLName, project) + if err != nil || desc == nil || desc.Example == "" { + continue + } + checked++ + for _, key := range exampleScalarKeys(desc.Example) { + if hiddenUnder(*desc, key) { + offenders = append(offenders, def.MDLName+"."+key) + } + } + } + if checked == 0 { + t.Skip("no widgets described") + } + if len(offenders) > 0 { + t.Errorf("%d properties are emitted by the example and hidden by that same "+ + "example's configuration — the generator and MDL-WIDGET10 must not disagree:\n %s", + len(offenders), strings.Join(offenders, "\n ")) + } +} + +// The control. "Emit nothing hidden" is trivially satisfied by emitting nothing, +// and it is also satisfied if hiddenUnder always returns false — in which case +// the test above proves nothing at all. Assert that pruning REALLY fires: some +// widget must have a scalar property that hiddenUnder reports hidden under the +// example's own configuration, i.e. something was actually removed. +func TestUsageExamplePruningActuallyFires(t *testing.T) { + project := fixtureProject(t) + registry := LoadWidgetRegistry(project) + if registry == nil { + t.Fatal("no registry") + } + + var pruned int + var nonEmpty int + for _, def := range registry.All() { + if def == nil || def.MDLName == "" { + continue + } + desc, err := DescribeWidget(def.MDLName, project) + if err != nil || desc == nil || desc.Example == "" { + continue + } + if len(exampleScalarKeys(desc.Example)) > 0 { + nonEmpty++ + } + for _, p := range desc.Properties { + if p.System || !p.Required { + continue + } + switch strings.ToLower(p.Type) { + case "boolean", "integer", "enumeration", "string", "texttemplate": + if hiddenUnder(*desc, p.Key) { + pruned++ + } + } + } + } + if nonEmpty == 0 { + t.Fatal("no example emits any property — the first test would pass vacuously") + } + if pruned == 0 { + t.Fatal("hiddenUnder never fires on a required scalar, so the assertion in " + + "TestUsageExampleDoesNotEmitItsOwnHiddenProperties is vacuous") + } + t.Logf("%d required scalars pruned across %d widgets with a non-empty example", pruned, nonEmpty) +} + +// The end-to-end form of the same invariant, and the one that matches how the +// disagreement was reported: build each widget's example, parse it into a real +// page statement, and run the property validator over it. Zero MDL-WIDGET10. +// +// This is stronger than the structural test above, which can only see what the +// generator itself considers hidden. The residue it catches is the OTHER +// direction of the same split: the validator resolves a property's value from +// the widget DEFINITION's mapping defaults (gallery's `itemSelection` defaults +// to "Single", so `keepSelection` is hidden), while exampleValues read only the +// .mpk, where a selection property carries no defaultValue — so the generator +// called it indeterminable and emitted the property the validator then warned +// about. Two value sources for one question. +func TestGeneratedExamplesProduceNoHiddenPropertyWarnings(t *testing.T) { + project := fixtureProject(t) + registry := LoadWidgetRegistry(project) + if registry == nil { + t.Fatal("no registry") + } + + var offenders []string + var validated int + for _, def := range registry.All() { + if def == nil || def.MDLName == "" { + continue + } + desc, err := DescribeWidget(def.MDLName, project) + if err != nil || desc == nil || desc.Example == "" { + continue + } + src := "create page Probe.P_" + def.MDLName + + " (Title: 'P', Layout: Atlas_Core.Atlas_Default) {\n" + desc.Example + "\n}\n" + prog, errs := visitor.Build(src) + if len(errs) > 0 || prog == nil { + continue // the example-parses guarantee is a different test's business + } + for _, stmt := range prog.Statements { + for _, v := range ValidateWidgetPropertiesForStatement(stmt, registry) { + if v.RuleID == "MDL-WIDGET10" { + offenders = append(offenders, def.MDLName+": "+v.Message) + } + } + } + validated++ + } + + if validated == 0 { + t.Skip("no example validated") + } + if len(offenders) > 0 { + t.Errorf("%d hidden-property warnings on mxcli's OWN generated examples "+ + "(%d widgets validated) — the generator and MDL-WIDGET10 read the same "+ + "editorConfig rules and must reach the same answer:\n %s", + len(offenders), validated, strings.Join(offenders, "\n ")) + } +} diff --git a/mdl/executor/widget_example_item_literal_test.go b/mdl/executor/widget_example_item_literal_test.go new file mode 100644 index 0000000000..ab8ee89728 --- /dev/null +++ b/mdl/executor/widget_example_item_literal_test.go @@ -0,0 +1,189 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/visitor" +) + +// The example writes one sub-property on each object-list item to show the +// shape. It wrote `'…'` — a literal ellipsis — and mxcli's own MDL-WIDGET08 +// then rejected it: +// +// widget `item2` (series) property `dataSet` has invalid value `…` +// — valid values are static, dynamic +// +// 11 across the fixture's 42 widgets. `'…'` reads as a placeholder to a person +// and is simply a wrong value to the checker, so the example was not runnable +// as printed even though it parsed — and "parses as written" is exactly what +// the block claims. +// +// The values are already in hand: propsFromMPK carries an object-list +// property's item properties as Children, with their enums and defaults, so +// exampleLiteral can pick a real one from the same data. +// +// The assertion is "no MDL-WIDGET08", NOT "no ellipsis anywhere". A free-text +// sub-property has no correct value to invent, and the validator accepts any +// string, so a placeholder there is the honest output — narrowing the rule to +// what the checker actually rejects keeps the test about the defect rather than +// about a character. +func TestGeneratedExamplesUseRealItemValues(t *testing.T) { + project := fixtureProject(t) + registry := LoadWidgetRegistry(project) + if registry == nil { + t.Fatal("no registry") + } + + var offenders []string + var validated, withItems int + for _, def := range registry.All() { + if def == nil || def.MDLName == "" { + continue + } + desc, err := DescribeWidget(def.MDLName, project) + if err != nil || desc == nil || desc.Example == "" { + continue + } + src := "create page Probe.P_" + def.MDLName + + " (Title: 'P', Layout: Atlas_Core.Atlas_Default) {\n" + desc.Example + "\n}\n" + prog, errs := visitor.Build(src) + if len(errs) > 0 || prog == nil { + continue + } + for _, stmt := range prog.Statements { + for _, v := range ValidateWidgetPropertiesForStatement(stmt, registry) { + if v.RuleID == "MDL-WIDGET08" { + offenders = append(offenders, def.MDLName+": "+v.Message) + } + } + } + validated++ + for _, c := range desc.Containers { + if c.Kind == "object list" && c.Authorable && len(c.ItemKeys) > 0 { + withItems++ + } + } + } + + // Without this the assertion is satisfied by a build where no example has an + // object list at all — so when that is the case, this test proves nothing and + // says so rather than passing quietly. + // + // It SKIPS rather than fails, because the empty case is legitimate and is + // exactly what CI sees: .mxcli/widgets/*.def.json is derived and gitignored, + // so a fresh checkout has only the hand-crafted definitions in + // sdk/widgets/definitions/, none of which declares an authorable object list + // with item properties. The guarantee comes from + // TestItemExampleLiteral_* below, which is hermetic and runs everywhere; + // this test is the end-to-end confirmation where the environment can give it. + if withItems == 0 { + t.Skip("no authorable object list with item properties in this environment " + + "(the .def.json cache is gitignored, so CI has only the hand-crafted " + + "definitions) — see TestItemExampleLiteral_* for the hermetic assertion") + } + if len(offenders) > 0 { + t.Errorf("%d generated examples carry a placeholder value the validator rejects "+ + "(%d validated, %d authorable object lists):\n %s", + len(offenders), validated, withItems, strings.Join(offenders, "\n ")) + } +} + +func firstLineContaining(s, needle string) string { + for _, line := range strings.Split(s, "\n") { + if strings.Contains(line, needle) { + return strings.TrimSpace(line) + } + } + return "" +} + +// The hermetic half: itemExampleLiteral's contract, asserted on a synthetic +// description so it runs in CI as well as locally. +// +// The end-to-end test above cannot carry this on its own — it needs a widget +// definition with an authorable object list, and CI has none, because the +// .def.json cache those come from is derived and gitignored. A test that only +// runs on a developer's machine is not a guard. +func TestItemExampleLiteral_PrefersASubPropertyWithADerivableValue(t *testing.T) { + desc := WidgetDescription{ + Properties: []DescribedProperty{{ + Key: "series", Type: "object", + Children: []DescribedProperty{ + {Key: "staticName", Type: "texttemplate"}, + {Key: "dataSet", Type: "enumeration", Enum: []string{"static", "dynamic"}}, + }, + }}, + } + c := DescribedContainer{ + Keyword: "series", PropertyKey: "series", Kind: "object list", + ItemKeys: []string{"staticName", "dataSet"}, + } + + key, lit := itemExampleLiteral(desc, c) + if key != "dataSet" { + t.Errorf("key = %q, want dataSet — a sub-property with a derivable value must be "+ + "preferred over a free-text one, or the example shows `'…'` where a real "+ + "member was available", key) + } + if lit != "'static'" { + t.Errorf("literal = %q, want 'static' (the enumeration's first member); `'…'` is "+ + "what MDL-WIDGET08 rejects", lit) + } +} + +// The definition wins over the .mpk, because the definition is what +// MDL-WIDGET08 checks the value against. Measured cause: ParseMPKForWidget +// returns 0 children for a PopupMenu's `basicItems`, while its definition +// carries itemType with enumValues [item, divider]. +func TestItemExampleLiteral_DefinitionBeatsThePackage(t *testing.T) { + desc := WidgetDescription{ + Properties: []DescribedProperty{{ + Key: "basicItems", Type: "object", + // The package knows the key but nothing about its values. + Children: []DescribedProperty{{Key: "itemType", Type: "enumeration"}}, + }}, + } + c := DescribedContainer{ + Keyword: "item", PropertyKey: "basicItems", Kind: "object list", + ItemKeys: []string{"itemType"}, + items: []DescribedProperty{ + {Key: "itemType", Type: "primitive", Default: "item", Enum: []string{"item", "divider"}}, + }, + } + + key, lit := itemExampleLiteral(desc, c) + if key != "itemType" || lit != "'item'" { + t.Errorf("got (%q, %q), want (itemType, 'item') — the definition carries the value "+ + "the package omits, and it is the source the validator reads", key, lit) + } +} + +// A container whose sub-properties are ALL free text keeps the placeholder. +// There is no correct value to invent, the validator accepts any string, and +// inventing one would be worse than admitting the gap. +func TestItemExampleLiteral_FreeTextKeepsThePlaceholder(t *testing.T) { + desc := WidgetDescription{ + Properties: []DescribedProperty{{ + Key: "attributes", Type: "object", + Children: []DescribedProperty{{Key: "attributeName", Type: "string"}}, + }}, + } + c := DescribedContainer{ + Keyword: "attribute", PropertyKey: "attributes", Kind: "object list", + ItemKeys: []string{"attributeName"}, + } + key, lit := itemExampleLiteral(desc, c) + if key != "attributeName" || lit != "'…'" { + t.Errorf("got (%q, %q), want (attributeName, '…')", key, lit) + } +} + +// No sub-properties at all: nothing to write, and no panic on the empty slice. +func TestItemExampleLiteral_NoItemKeys(t *testing.T) { + if key, lit := itemExampleLiteral(WidgetDescription{}, DescribedContainer{}); key != "" || lit != "" { + t.Errorf("got (%q, %q), want empty", key, lit) + } +} diff --git a/mdl/executor/widget_item_action_slot_test.go b/mdl/executor/widget_item_action_slot_test.go index 2e15ca4a9e..96078d932d 100644 --- a/mdl/executor/widget_item_action_slot_test.go +++ b/mdl/executor/widget_item_action_slot_test.go @@ -191,7 +191,7 @@ func TestExtractObjectListItem_ReadsAnAction(t *testing.T) { var got string for _, p := range item.Props { - if p.Key == "StaticOnClickAction" { + if p.Key == "staticOnClickAction" { if !p.IsRef { t.Error("the action was quoted — `staticOnClickAction: 'microflow …'` does not parse back") } diff --git a/mdl/executor/widget_item_template_params_test.go b/mdl/executor/widget_item_template_params_test.go index b6c257565c..af3fad1319 100644 --- a/mdl/executor/widget_item_template_params_test.go +++ b/mdl/executor/widget_item_template_params_test.go @@ -42,12 +42,12 @@ func TestExtractObjectListItem_EmitsTextTemplateParameters(t *testing.T) { var text, params string for _, p := range item.Props { switch p.Key { - case "ButtonCaption": + case "buttonCaption": text = p.Value - case "ButtonCaptionParams": + case "buttonCaptionParams": params = p.Value if !p.IsRef { - t.Error("the parameter list was quoted — `ButtonCaptionParams: '[...]'` does not parse") + t.Error("the parameter list was quoted — `buttonCaptionParams: '[...]'` does not parse") } } } @@ -78,7 +78,7 @@ func TestBuildObjectListItem_ReadsParamsCompanionCaseInsensitively(t *testing.T) }, } child := &ast.WidgetV3{Name: "b1", Properties: map[string]any{ - "ButtonCaption": "Hello {1}", + "buttonCaption": "Hello {1}", "ButtonCaptionParams": []ast.ParamAssignmentV3{{Index: 1, Value: "'abc'"}}, }} diff --git a/mdl/executor/widget_known_props_from_mpk.go b/mdl/executor/widget_known_props_from_mpk.go new file mode 100644 index 0000000000..2f0e38cfe7 --- /dev/null +++ b/mdl/executor/widget_known_props_from_mpk.go @@ -0,0 +1,123 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "path/filepath" + "sort" + "strings" + + "github.com/mendixlabs/mxcli/modelsdk/widgets/mpk" +) + +// enrichKnownPropertiesFromMPK fills each definition's KnownProperties from the +// widget's INSTALLED package, so the property validator recognises every +// property the widget actually declares. +// +// # Why +// +// mxcli reads a widget from two places. DESCRIBE WIDGET parses the project's +// .mpk (version-accurate, and the only place a Marketplace widget appears). The +// property validator reads the WidgetDefinition. For most widgets those agree, +// because the .def.json cache is GENERATED from the .mpk and its +// knownProperties already carry everything unmapped. +// +// Nine widgets are the exception. COMBOBOX, GALLERY, IMAGE, BARCODESCANNER, the +// four data-grid filters and DROPDOWNSORT have hand-crafted definitions in +// sdk/widgets/definitions/ and are deliberately never extracted per-project, so +// their .def.json never exists and their hand-written property list is whatever +// someone typed. Measured against the packages in testdata/expr-checker: +// +// combobox 73 properties in the .mpk, 7 mapped + 4 known +// gallery 44 12 +// image 37 14 +// barcodescanner 12 1 +// +// So DESCRIBE emitted an example it labels "parses as written" — and it does +// parse — naming properties the validator then rejected as nonexistent. Because +// exec refuses a script with errors, BARCODESCANNER could not be placed from MDL +// in any legal form: name the five properties and exec refuses, omit them and +// mxbuild reports CE0463 "the definition of this widget has changed". +// +// # Known, not allowed +// +// A .mpk property with no mapping is added to KnownProperties, NOT to the +// allowed set. That is the honest distinction the validator already draws: +// knownUnmappedProperties turns it into MDL-WIDGET06 — "recognized but not yet +// persisted by mxcli; a non-default value will be dropped" — rather than +// silently accepting a value nothing writes. Promoting them to "allowed" would +// trade a false error for a silent drop. +// +// MDL-WIDGET01 keeps its job: a name no package declares is still an error, so +// typos are still caught. +// +// # Applied to every definition, not to a list of nine +// +// Recomputing KnownProperties for a generated definition produces what +// generation already put there, so the enrichment is idempotent where it is +// redundant. Naming the nine would be the same hand-maintained-list defect one +// layer up — the defect this whole line of work exists to remove. +func enrichKnownPropertiesFromMPK(r *WidgetRegistry, projectPath string) { + if r == nil || projectPath == "" { + return + } + byID := mpkPropertiesByWidgetID(filepath.Dir(projectPath)) + if len(byID) == 0 { + return + } + for _, def := range r.byWidgetID { + props, ok := byID[def.WidgetID] + if !ok { + continue + } + allowed, _ := allowedWidgetProperties(def) + seen := make(map[string]bool, len(def.KnownProperties)) + for _, k := range def.KnownProperties { + seen[strings.ToLower(k)] = true + } + var added []string + for _, p := range props { + if p.Key == "" || p.IsSystem { + continue + } + l := strings.ToLower(p.Key) + if allowed[l] || seen[l] { + continue + } + seen[l] = true + added = append(added, p.Key) + } + if len(added) == 0 { + continue + } + sort.Strings(added) + def.KnownProperties = append(def.KnownProperties, added...) + } +} + +// mpkPropertiesByWidgetID parses every package in the project's widgets/ folder +// once and indexes the properties by widget id. +// +// ParseAll rather than ParseMPK: a bundled package (Charts.mpk) carries many +// widgets and ParseMPK returns only the first, which is the #679 bug — here it +// would silently leave every chart but one unenriched. +func mpkPropertiesByWidgetID(projectDir string) map[string][]mpk.PropertyDef { + matches, err := filepath.Glob(filepath.Join(projectDir, "widgets", "*.mpk")) + if err != nil || len(matches) == 0 { + return nil + } + out := make(map[string][]mpk.PropertyDef, len(matches)) + for _, path := range matches { + defs, err := mpk.ParseAll(path) + if err != nil { + continue // a package we cannot read enriches nothing; it is not an error + } + for _, d := range defs { + if d == nil || d.ID == "" { + continue + } + out[d.ID] = d.Properties + } + } + return out +} diff --git a/mdl/grammar/domains/MDLCatalog.g4 b/mdl/grammar/domains/MDLCatalog.g4 index 5eb25b5225..02b1b5b3df 100644 --- a/mdl/grammar/domains/MDLCatalog.g4 +++ b/mdl/grammar/domains/MDLCatalog.g4 @@ -172,6 +172,8 @@ describeStatement | DESCRIBE ODATA SERVICE qualifiedName // DESCRIBE ODATA SERVICE Module.ServiceName | DESCRIBE EXTERNAL ENTITY qualifiedName // DESCRIBE EXTERNAL ENTITY Module.EntityName | DESCRIBE NAVIGATION (qualifiedName | IDENTIFIER)? // DESCRIBE NAVIGATION [profile] + | DESCRIBE WIDGET identifierOrKeyword // DESCRIBE WIDGET combobox | DESCRIBE WIDGET 'com.mendix…' + | DESCRIBE WIDGET STRING_LITERAL // …by full widget id, which contains dots | DESCRIBE STYLING ON (PAGE | SNIPPET) qualifiedName (WIDGET IDENTIFIER)? // DESCRIBE STYLING ON PAGE Module.Page [WIDGET name] | DESCRIBE CATALOG DOT (catalogTableName) // DESCRIBE CATALOG.ENTITIES | DESCRIBE BUSINESS EVENT SERVICE qualifiedName // DESCRIBE BUSINESS EVENT SERVICE Module.Name diff --git a/mdl/grammar/domains/MDLPage.g4 b/mdl/grammar/domains/MDLPage.g4 index 23d75f7e9b..c003801d1c 100644 --- a/mdl/grammar/domains/MDLPage.g4 +++ b/mdl/grammar/domains/MDLPage.g4 @@ -249,7 +249,18 @@ snippetHeaderPropertyV3 // `placeholder { … }` block binds its widgets to that named layout // placeholder (issue #532 — pages over a layout with >1 placeholder). pageBodyV3 - : (widgetV3 | useFragmentRef | useBuildingBlockRef | placeholderBlockV3 | slotMarkerV3)* + // ORDER IS LOAD-BEARING since slices 2-3. widgetV3's last alternative is a + // generic (IDENTIFIER | keyword) widget type, and SLOT, PLACEHOLDER and USE + // are all in `keyword` — so with widgetV3 first, `slot content` parsed as a + // widget of type `slot` named `content`, and `placeholder Main { … }` as a + // widget named Main. Both still PARSED and still exited 0, which is why a + // diff of `mxcli check` output across all 515 example scripts did not show + // it; the damage is to the AST, not to the diagnostics. Two visitor unit + // tests caught it. + // + // The specific alternatives therefore go first, the same ordering fix + // widgetV3 already applies internally for `template for`. + : (useFragmentRef | useBuildingBlockRef | placeholderBlockV3 | slotMarkerV3 | widgetV3)* ; // SLOT [name] — a content placeholder inside a `define fragment` body. When the @@ -412,11 +423,38 @@ widgetTypeV3 // the dojo-based native Forms$DataGrid even on Mendix 11+; useful for // migrated projects that still have native datagrids on the page. | LEGACYDATAGRID + // Any widget with a definition, named by its MDL name — `htmlelement frame + // (...)`, `fileuploader up (...)`. Slice 2 of + // PROPOSAL_def_driven_widget_bodies.md (mendixlabs/mxcli#1036). + // + // The list above was never a capability boundary: cmd_pages_builder_v3.go's + // default branch already resolves widgetRegistry.Get(ToUpper(w.Type)) FIRST, + // and every .def.json declares an mdlName. Only ANTLR needed a token, so a + // widget mxcli could build was one MDL could not spell. + // + // ORDERED LAST so every enumerated type keeps winning its own alternative, + // and the widget's NAME is still a direct IDENTIFIER child of widgetV3 — + // this one is nested inside widgetTypeV3, so wCtx.IDENTIFIER() is unaffected. + // + // An unknown name is no longer a parse error; it is MDL-WIDGET25, which + // slice 0 added for exactly this reason. + | IDENTIFIER + // Slice 3: the same, for a container whose name lexes as a KEYWORD token. + // This is not defensive — it is the case that motivated the whole issue. + // `attribute` lexes as ATTRIBUTE and never as IDENTIFIER, so the + // alternative above cannot match `attribute a1 (...)`, which is the HTML + // Element object list the reporter could not write. + | keyword ; // V3 Widget properties: (Prop: Value, Prop: Value) +// The list may be EMPTY. `container c ()` is what an LLM writes when a widget +// needs no properties, and rejecting it gave a parse error at the `)` that read +// as though the widget itself were wrong. Bare `container c` already parsed, so +// this only removes an arbitrary difference between two spellings of the same +// thing (mendixlabs/mxcli#1036). widgetPropertiesV3 - : LPAREN widgetPropertyV3 (COMMA widgetPropertyV3)* RPAREN + : LPAREN (widgetPropertyV3 (COMMA widgetPropertyV3)*)? RPAREN ; widgetPropertyV3 diff --git a/mdl/visitor/visitor_page_generic_widget_test.go b/mdl/visitor/visitor_page_generic_widget_test.go new file mode 100644 index 0000000000..e5019566c3 --- /dev/null +++ b/mdl/visitor/visitor_page_generic_widget_test.go @@ -0,0 +1,136 @@ +// SPDX-License-Identifier: Apache-2.0 + +package visitor + +import ( + "fmt" + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" +) + +func buildOnePage(t *testing.T, src string) *ast.CreatePageStmtV3 { + t.Helper() + prog, errs := Build(src) + if len(errs) > 0 { + t.Fatalf("parse errors: %v\nsource:\n%s", errs, src) + } + if len(prog.Statements) != 1 { + t.Fatalf("got %d statements, want 1", len(prog.Statements)) + } + pg, ok := prog.Statements[0].(*ast.CreatePageStmtV3) + if !ok { + t.Fatalf("got %T, want *ast.CreatePageStmtV3", prog.Statements[0]) + } + return pg +} + +// Slice 2: a widget named by its own MDL name reaches the AST as that type, and +// is marked generic so the validator knows it must resolve to a definition. +func TestGenericWidgetTypeReachesTheAST(t *testing.T) { + pg := buildOnePage(t, `create page P.Q (Title: 'x', Layout: A.B) { + htmlelement frame (tagName: 'div') +}`) + if len(pg.Widgets) != 1 { + t.Fatalf("got %d widgets, want 1", len(pg.Widgets)) + } + w := pg.Widgets[0] + if w.Type != "htmlelement" { + t.Errorf("Type = %q, want %q", w.Type, "htmlelement") + } + if w.Name != "frame" { + t.Errorf("Name = %q, want \"frame\" — the name must still be the widget's own IDENTIFIER, "+ + "not swallowed by the generic type alternative", w.Name) + } + if !w.TypeIsGeneric { + t.Error("TypeIsGeneric = false; a name that is not an enumerated widget token must be " + + "marked generic, or MDL-WIDGET25 cannot tell a typo from a built-in") + } +} + +// The control: an enumerated widget keyword must NOT be marked generic, or every +// built-in would be required to resolve to a widget definition. +func TestEnumeratedWidgetTypeIsNotGeneric(t *testing.T) { + pg := buildOnePage(t, `create page P.Q (Title: 'x', Layout: A.B) { + container c1 (Class: 'x') +}`) + w := pg.Widgets[0] + if w.Type != "container" { + t.Fatalf("Type = %q, want container", w.Type) + } + if w.TypeIsGeneric { + t.Error("TypeIsGeneric = true for `container`, an enumerated widget token — " + + "the flag must come from which grammar alternative matched, not from a name lookup") + } +} + +// Slice 3: a container whose keyword lexes as a KEYWORD token, not IDENTIFIER. +// This is the construct from mendixlabs/mxcli#1036 — `attribute` inside an HTML +// Element — which failed with "mismatched input" before. +func TestKeywordContainerParsesInsideAWidgetBody(t *testing.T) { + pg := buildOnePage(t, `create page P.Q (Title: 'x', Layout: A.B) { + htmlelement frame (tagName: 'div') { + attribute a1 (attributeName: 'title') + event e1 (eventName: 'onClick') + } +}`) + w := pg.Widgets[0] + if len(w.Children) != 2 { + t.Fatalf("got %d children, want 2 (attribute, event)", len(w.Children)) + } + for i, want := range []string{"attribute", "event"} { + if w.Children[i].Type != want { + t.Errorf("child %d Type = %q, want %q", i, w.Children[i].Type, want) + } + if !w.Children[i].TypeIsGeneric { + t.Errorf("child %d (%s) not marked generic", i, want) + } + } +} + +// pageBodyV3's alternative ORDER is load-bearing since slices 2-3, and this is +// the trap that a diff of `mxcli check` output cannot detect. +// +// SLOT, PLACEHOLDER and USE are all inside the `keyword` rule, so widgetV3's +// generic alternative can match them. With widgetV3 ordered first, `slot body` +// parsed as a widget of type `slot` named `body`, and `placeholder Main { … }` +// as a widget named Main. Both still parsed and `check` still exited 0 — the +// damage is to the AST, not to the diagnostics, which is why 515 example +// scripts showed zero difference while two visitor tests failed. +// +// Anyone reordering pageBodyV3 for tidiness reintroduces it silently. +func TestSpecificPageBodyFormsWinOverTheGenericWidget(t *testing.T) { + pg := buildOnePage(t, `create page P.Q (Title: 'x', Layout: A.B) { + placeholder Main { + dynamictext t (Content: 'hi') + } +}`) + if len(pg.Placeholders) != 1 { + t.Fatalf("placeholder blocks = %d, want 1 — `placeholder` was swallowed by the generic "+ + "widget alternative; the specific alternatives must precede widgetV3 in pageBodyV3", + len(pg.Placeholders)) + } + if len(pg.Widgets) != 0 { + t.Errorf("bare widgets = %d, want 0 — the placeholder block was parsed as a widget", len(pg.Widgets)) + } +} + +func TestSlotMarkerWinsOverTheGenericWidget(t *testing.T) { + prog, errs := Build(`DEFINE FRAGMENT Card AS { + CONTAINER wrap (Class: 'c') { + SLOT content + } + };`) + if len(errs) > 0 { + t.Fatalf("parse errors: %v", errs) + } + if len(prog.Statements) != 1 { + t.Fatalf("got %d statements, want 1", len(prog.Statements)) + } + // `slot` is in the `keyword` rule, so the generic widget alternative can + // match it. The fragment must still see a slot, not a widget named `body`. + if got := fmt.Sprintf("%#v", prog.Statements[0]); strings.Contains(strings.ToLower(got), `type:"slot"`) { + t.Errorf("`slot body` became a widget of type slot — slotMarkerV3 must precede widgetV3 in pageBodyV3:\n%s", got) + } +} diff --git a/mdl/visitor/visitor_page_v3.go b/mdl/visitor/visitor_page_v3.go index 4713b5a2d6..099dfef20c 100644 --- a/mdl/visitor/visitor_page_v3.go +++ b/mdl/visitor/visitor_page_v3.go @@ -552,6 +552,18 @@ func buildWidgetV3(ctx parser.IWidgetV3Context, b *Builder) *ast.WidgetV3 { widget.Properties["WidgetType"] = unquoteString(wCtx.STRING_LITERAL().GetText()) } else if typeCtx := wCtx.WidgetTypeV3(); typeCtx != nil { widget.Type = strings.ToLower(typeCtx.GetText()) + // Which alternative matched, taken from the parse tree rather than by + // comparing the text against a list of known widget names. A generic + // type must resolve to a widget definition; an enumerated one is a + // built-in. See ast.WidgetV3.TypeIsGeneric. + // Both generic alternatives count. IDENTIFIER covers `htmlelement` + // (slice 2); Keyword covers a container whose name lexes as a keyword + // token, such as `attribute` (slice 3) — the case that motivated the + // issue. An enumerated widget type is a direct token alternative of + // widgetTypeV3 and matches neither accessor. + if typeCtx.IDENTIFIER() != nil || typeCtx.Keyword() != nil { + widget.TypeIsGeneric = true + } } // Get required identifier. The name may be quoted (QUOTED_IDENTIFIER) when it diff --git a/mdl/visitor/visitor_query.go b/mdl/visitor/visitor_query.go index 88e9b6a460..8d042a31de 100644 --- a/mdl/visitor/visitor_query.go +++ b/mdl/visitor/visitor_query.go @@ -1011,6 +1011,24 @@ func (b *Builder) ExitDescribeStatement(ctx *parser.DescribeStatementContext) { return } + // DESCRIBE WIDGET — a widget DEFINITION, not a + // document, so there is no qualified name. The STYLING branch above + // returns before this, but WIDGET also appears there (DESCRIBE STYLING … + // WIDGET name), so the guard stays explicit rather than resting on order. + if ctx.WIDGET() != nil && ctx.STYLING() == nil { + name := "" + if lit := ctx.STRING_LITERAL(); lit != nil { + name = unquoteString(lit.GetText()) + } else if id := ctx.IdentifierOrKeyword(0); id != nil { + name = id.GetText() + } + b.statements = append(b.statements, &ast.DescribeStmt{ + ObjectType: ast.DescribeWidget, + Name: ast.QualifiedName{Name: name}, + }) + return + } + // Handle DESCRIBE NAVIGATION [profile] if ctx.NAVIGATION() != nil { stmt := &ast.DescribeStmt{ObjectType: ast.DescribeNavigation} diff --git a/testdata/expr-checker/.gitignore b/testdata/expr-checker/.gitignore index 9ea33a1009..f2018de286 100644 --- a/testdata/expr-checker/.gitignore +++ b/testdata/expr-checker/.gitignore @@ -23,6 +23,9 @@ /mprcontents/mprjournal* .claude/settings.local.json +# Derived from widgets/*.mpk and rewritten by `refresh catalog full` +# (executor.RegenerateWidgetDocs), not authored here. +.claude/skills/widgets/ mxcli.exe mxcli .mxcli