diff --git a/.claude/skills/fix-issue/findings/mdl-backend.jsonl b/.claude/skills/fix-issue/findings/mdl-backend.jsonl index 6abb5c2cc8..0678e54085 100644 --- a/.claude/skills/fix-issue/findings/mdl-backend.jsonl +++ b/.claude/skills/fix-issue/findings/mdl-backend.jsonl @@ -136,3 +136,4 @@ {"area": "mdl/backend", "date": "2026-09-26", "symptom": "`ALTER PAGE … SET ImageUrl = '…' ON img1` (pluggable Image) prints \"Altered page\" and changes nothing: DESCRIBE still shows the old URL. Same silent success for SET on any pluggable widget property of kind Expression, Image, Icon, Action, DataSource, Attribute or Widgets", "cause": "setPluggableWidgetPropertyMut (mdl/backend/pagemutator/mutator.go) wrote every value to Value.PrimitiveValue. imageUrl is a TextTemplate; readers take Value.TextTemplate. The DataGrid 2 column setter had the identical defect fixed on 2026-08-18 (columnValueField) — the widget-level setter sitting next to it was never given the same schema dispatch.", "file": "mdl/backend/pagemutator/mutator.go (setPluggableWidgetPropertyMut, buildPropKindMap), test pluggable_property_kind_test.go, example mdl-examples/bug-tests/1201-alter-page-set-pluggable-texttemplate.mdl", "insight": "When a fix lands on one setter for 'write the field the schema declares', grep the sibling setters in the same file for the same always-PrimitiveValue write — a WidgetValue carries every variant field, so the wrong write never errors and every signal stays green. The live control is cheap without mxbuild: copy testdata/expr-checker, create an image with URL A, ALTER to B, describe (pre-fix binary shows A). A null TextTemplate means the slot is hidden (#574); ALTER does not re-run visibility, so refuse rather than build an envelope there. Not fixed here: a pluggable boolean SET stores \"yes\"/\"no\" — check what CREATE stores before changing it.", "refs": ["mendixlabs/mxcli#1201", "mendixlabs/mxcli#1069", "mendixlabs/mxcli#750", "mendixlabs/mxcli#574"], "rules": []} {"area": "mdl/backend", "date": "2026-09-26", "symptom": "describe → exec of a snippet turns its Type 'Web' into '' (and would reset a non-default ExportLevel or canvas size)", "cause": "snippetToGen wrote Type, ExportLevel, CanvasWidth and CanvasHeight as constants on every write; the Type constant was \"\"", "file": "mdl/backend/modelsdk/snippet_write.go", "insight": "The snippet twin of #541 (carryStoredPageHeader). \"\" is not a member of PagesType (Native | Web), so every snippet mxcli created was also invalid. Carry the stored header on UpdateSnippet (width-agnostic canvas read — Studio Pro stores int64); a new snippet gets Web/Hidden/800×600, measured on all four Blank-template snippets", "refs": ["ako/mxcli#705", "#541"]} {"area": "mdl/backend", "date": "2026-09-26", "symptom": "describe → exec of a nanoflow detaches every annotation from its activity (AnnotationFlows 3 → 0, so the next describe prints the notes as free-floating) and deletes ExportLevel, UseListParameterByReference and ReturnVariableName; MarkAsUsed is cleared. Rules drop their annotation flows too", "cause": "nanoflowToGen and ruleToGen never wrote ObjectCollection.AnnotationFlows (microflowToGen did), and nanoflowToGen never set the three header keys; nanoflowFromGen did not read ReturnVariableName or MarkAsUsed and the builder hardcoded MarkAsUsed false", "file": "mdl/backend/modelsdk/nanoflow_write.go, rule_write.go, microflow.go; mdl/executor/cmd_microflows_build.go", "insight": "Three flow flavours share the object/flow converters but each has its own top-level writer, and each dropped a different subset — grep every *ToGen for the same list loop, not just the reported one. For keys with no MDL spelling and unknown version floors (Nanoflow.ExportLevel, UseListParameterByReference), carry ONLY the keys the stored unit has: nothing was ever written on create, Studio Pro fills an absent key, and an undeclared one makes the unit unopenable", "refs": ["ako/mxcli#705"]} +{"area": "mdl/backend", "date": "2026-09-27", "symptom": "Running unchanged `describe user role` output (create or modify user role) rewrites Security$ProjectSecurity: /UserRoles[0] marker 2 -> 3; same for DemoUsers on a demo-user rewrite", "cause": "The encoder's default PartList marker is 3; Studio Pro writes 2 for ProjectSecurity.UserRoles and .DemoUsers (PedApp, TestApp, expr-checker agree). An untouched list passes through raw, so it only shows once an element in it is rewritten", "file": "mdl/backend/modelsdk/security_write.go", "insight": "A marker bug hides until the first rewrite of an element in that list; census the stored marker per (owner $Type, key) across Studio Pro fixtures and register it with RegisterPropertyListMarker", "refs": ["ako/mxcli#731"]} diff --git a/.claude/skills/fix-issue/findings/mdl-executor.jsonl b/.claude/skills/fix-issue/findings/mdl-executor.jsonl index 92a6984382..bdafc54863 100644 --- a/.claude/skills/fix-issue/findings/mdl-executor.jsonl +++ b/.claude/skills/fix-issue/findings/mdl-executor.jsonl @@ -728,3 +728,8 @@ {"area": "mdl/executor", "date": "2026-09-26", "symptom": "`describe fragment from page M.P widget w` (and `from snippet`) fails for every container and every widget — including ones `describe page` prints — with `not found in page M.P`, a message that does not even name the widget", "cause": "Visitor stores DescribeFragmentFromStmt.ContainerType as \"PAGE\"/\"SNIPPET\"; describeFragmentFrom switched on \"page\"/\"snippet\" with no default, so neither branch ran, the widget list stayed empty, and the fall-through reported the widget missing. Third instance of this split: ALTER PAGE (#402), DESCRIBE/ALTER STYLING (#631)", "file": "`mdl/executor/cmd_fragments.go` (`describeFragmentFrom`)", "insight": "The mismatch hid behind a plausible error because a switch on the discriminator had no default: an unmatched container type looked like an empty container, and an empty container looks like a missing widget. Normalise with strings.ToLower where the discriminator is consumed (the house convention — cmd_styling, cmd_alter_page, validate_alter_* all do) AND make the default an error, so the next casing drift fails loudly instead of reporting the wrong thing. The existing mock tests hand-built the AST and so agreed with the handler; only a test that goes visitor.Build → NewRegistry().Dispatch pins the contract between the two layers (cmd_fragments_from_test.go). Verified on Evora: Administration.Account_Edit/textBox6 and AgentCommons.Snippet_Agent_Details/dataView7 now describe", "refs": ["#402", "#631"]} {"area": "mdl/executor", "date": "2026-09-26", "symptom": "describe output that does not re-parse or loses data (ako/mxcli#707): an entity string default or validation message containing ' was emitted unescaped; so were module-role descriptions, published OData/REST Path/Version/Namespace/Summary/Folder, and REST client BaseUrl/Path/header values; an agent `mcp service` block with a Description lacked the comma after `Enabled`; workflow decision / parallel split captions came back only as `-- caption` comments (replay reset them to 'Decision' / 'Parallel split'); `describe demo user` emitted `password '***'`, which replay stored as the password; `describe settings` printed `DatabasePassword = ''`.", "cause": "Hand-rolled `'%s'` emit sites that put the quotes and the escaping in different places (the #1006 source-scan guard covered only cmd_workflows.go); a block emitter with no separator logic, unlike its sibling; captions treated as commentary although the grammar has `comment '…'` for both activities; secrets printed as data, with a placeholder the writer took literally.", "file": "mdl/executor/cmd_entities_describe.go, cmd_security.go, cmd_security_write.go, cmd_odata.go, cmd_published_rest.go, cmd_rest_clients.go, cmd_agenteditor_agents.go, cmd_workflows.go, cmd_settings.go", "fix": "Every emit site uses mdlQuoted; TestDescribers_HaveNoHandRolledStringLiterals now scans all seven describer files. MCP block writes the comma like the tool block. workflowCaptionClauses emits `comment '…'` for a non-default caption and computes the name clause against the caption the writer will store. DatabasePassword is omitted with a comment (create or modify is a patch, so replay keeps it). Demo users are described as `create or modify … password '***'`, and the executor treats '***' as 'keep the stored password', refusing it for a user that does not exist.", "insight": "Assert round trips by reparsing describe output with the real visitor and comparing the AST value to the stored one, not by substring. For secrets the right placeholder is one the WRITER understands: omission works where the create is a patch (configuration); where the grammar requires the value (demo user) give the placeholder a meaning (keep stored) and refuse it where that meaning is empty, so a replay can neither leak nor silently set a credential.", "test": "mdl/executor/issue707_describe_roundtrip_test.go"} {"area": "mdl/executor", "date": "2026-09-26", "symptom": "describe microflow \u2026 with handles (ako/mxcli#713) printed no handle for an activity inside an `on error { \u2026 }` block, and the alter-target resolver counted such activities after the whole main flow: on SUB_Feedback_PostToAppInsights `return * @1` picked `return $Response`, although describe prints the handler's `return empty` first. Also `$Response` did not address a REST call whose output is on its result handling (and cast, create list, web service, workflow, XML/JSON, database-query outputs).", "cause": "Error-handler bodies are rendered by collectErrorHandlerStatements, a second describer that returned bare strings and never wrote the source map, so those nodes had no line to rank or print a handle at; unranked candidates were appended last. The output-variable switch was copied from actionOutputVariableName, which had drifted from the formatter.", "file": "mdl/executor/cmd_microflows_show_helpers.go; mdl/backend/mfmutator/target.go", "fix": "collectErrorHandlerStatementSpans reports each handler-body object's statement span; emitActivityStatement and emitCommentedErrorHandler record them in the source map (additive entries in ELK sourceMap too). mfmutator.OutputVariable reads the variable where the formatter does, for every action it prints as `$X = \u2026`. A comment rendering (`-- Unsupported \u2026`) is no statement, so it never becomes a handle.", "insight": "Any node the describer prints through a side path must enter the source map, or everything keyed on print order (ordinals, handles, ELK highlighting) silently disagrees with the text. Check ranking against a Studio Pro flow that has a handler body, not just VAL_Feedback.", "test": "TestDescribeWithHandles_ErrorHandlerBody, TestMicroflowTargets_PedAppEveryFlowRanksAndResolves, TestOutputVariable_EveryActionDescribePrintsAnAssignmentFor, TestCandidate_CommentRenderingIsNoStatement"} +{"area": "mdl/executor", "date": "2026-09-27", "symptom": "Once describe emitted create or modify javascript action, re-running it wrote a .js file holding the describe placeholder body, dropped parameter descriptions, and rewrote Studio Pro doc comments (CRLF, blank lines) \u2014 getput and putget failures on 45 PedApp actions", "cause": "The JS write path lacked the Java twin's carries: placeholder guard (javaSourceOmittedBody) and stored-parameter Description/Category; and carriedDocumentation overwrote stored prose with its doc-comment normal form", "file": "mdl/executor/cmd_javascript_actions_write.go, documentation_carry.go", "insight": "Turning a plain create into create or modify is what makes describe output execute at all \u2014 expect the next layer of losses to surface immediately; diff the twin (Java vs JavaScript) write paths for every carry one has and the other lacks", "refs": ["ako/mxcli#731"]} +{"area": "mdl/executor", "date": "2026-09-27", "symptom": "Running unchanged `describe javascript action` output (now `create or modify`) on a project with the .js sources regenerated the file: the import list (`import { Big } from \"big.js\"`) and the EXTRA CODE helpers (deg2rad) were dropped and the signature re-spelled, so NanoflowCommons.GetStraightLineDistance's user code called functions that no longer existed", "cause": "execCreateJavaScriptAction always called WriteJavaScriptSourceFile, and GenerateJavaScriptSource has no slot for the imports or EXTRA CODE (the Java twin retains both, #705). A plain `create` refused on an existing action, which is what had hidden it", "file": "mdl/executor/cmd_javascript_actions_write.go (jsSourceUnchanged)", "insight": "The PedApp round trip could not see this: PedApp ships no javascriptsource, so describe printed the placeholder body and the file path was never exercised. Snapshot the source tree as well as mprcontents when round-tripping a doctype that owns files. Leaving the file alone when the statement restates the stored user code and parameter names fixes describe->exec; an edited body still regenerates without the retained sections", "refs": ["ako/mxcli#731"], "test": "TestJavaScriptActionRewrite_UnchangedCodeKeepsSourceFile"} +{"area": "mdl/executor", "date": "2026-09-27", "symptom": "describe microflow \u2026 with handles printed `-- handle: commit $Order on error {` for an activity with a custom error handler (TestApp Services.SaveOrder). The handle could not be used: an `alter microflow` target ends at the `{` that opens a fragment, so `insert after commit $Order on error { \u2026 }` parsed the handler brace as the fragment.", "cause": "printedStatement ended an action's statement at a line ending in `;` or `{` and kept the `{`, which belongs to the error-handler block describe opens, not to the statement.", "file": "mdl/executor/cmd_microflows_handles.go", "fix": "printedStatement strips the trailing `{` of an error-handler block opener, so the handle is `commit $Order on error`, which parses as a target and still matches the activity.", "insight": "A handle is only useful if it can be written back as a target in the grammar that consumes it; test a printed handle by parsing it, not only by resolving it.", "test": "TestPrintedStatement_ErrorHandlerBlockOpenerIsNotPartOfTheStatement"} +{"area": "mdl/executor", "date": "2026-09-27", "symptom": "alter microflow (ako/mxcli#736) reported \"Altered microflow\" for statements mx check then rejected: two fragments in one statement declaring the same variable gave CE0111 Duplicate variable name; a fragment reading $X in the same statement as `drop $X` (either order) gave CE0109 Undefined variable. A loop fragment was refused as reading its own iterator, and drop/replace of a Studio Pro loop with more than one body activity was refused by the dangling-reference guard.", "cause": "The scope checks compared each operation with the flow as stored, never with what earlier operations of the same statement had declared, read or removed. The iterator of a fragment's loop is on its LoopSource, not an action output. Loop body flows are stored in the unit's Flows list, not in the loop, so removing the loop left them pointing at removed objects.", "file": "mdl/executor/cmd_alter_flow.go; mdl/backend/mfmutator/splice.go", "fix": "alterFlowContext tracks declaredByOps / readByOps / removedByOps across the statement's operations and checks each later operation against them; checkFragmentScope counts a fragment loop's iterator as its own; Drop and Replace remove graph.bodyFlows of a loop with it.", "insight": "A per-operation check against the stored document is only sound for a one-operation statement; every multi-operation test needs a case where operation 2 depends on operation 1. mx check on a copied TestApp is the cheap oracle: the CE numbers appear the moment a hygiene hole is hit.", "test": "TestAlterMicroflow_PedApp_ScopeSpansTheStatement; TestAlterMicroflow_PedApp_LoopFragmentDeclaresItsIterator; TestSplice_DropOrReplaceALoopTakesItsBodyFlows"} +{"area": "mdl/executor", "date": "2026-09-27", "symptom": "describe of a published REST service with no resources printed `create or modify published rest service M.S ( \u2026 )` followed by a bare `;`, which does not parse (`mismatched input ';' expecting '{'`); found while making describe end every statement with `;` (ako/mxcli#744).", "cause": "The resource block `{ \u2026 }` is mandatory in the grammar (publishedRestService: \u2026 RPAREN LBRACE publishedRestResource* RBRACE), but the describer only printed it when the service had resources and otherwise closed the statement with `;` alone.", "file": "mdl/executor/cmd_published_rest.go", "fix": "Always print the block: `{\\n};` when there are no resources.", "insight": "A describe test that asserts substrings never parses the output; feeding every Describe*_Mock output through the parser (assertTerminated, #744) found this at once. The empty-collection branch of a describer is the one no fixture exercises.", "test": "TestDescribePublishedRestService_Mock (assertTerminated)"} diff --git a/.claude/skills/mendix/cheatsheet-errors/SKILL.md b/.claude/skills/mendix/cheatsheet-errors/SKILL.md index 7a97e60e26..8565ba1801 100644 --- a/.claude/skills/mendix/cheatsheet-errors/SKILL.md +++ b/.claude/skills/mendix/cheatsheet-errors/SKILL.md @@ -51,7 +51,7 @@ declare $Products list of Module.Product = empty; ```mdl -- object: a microflow parameter, a retrieve, a create, or a loop iterator -retrieve $Product from Module.Product where Code = $Code limit 1; -- retrieve +retrieve $Product from Module.Product where Code = $Code first; -- retrieve $Product = create Module.Product (Name = $Name); -- create -- or: create microflow M.Save ($Product: Module.Product) ... / loop $Product in $Products ... diff --git a/.claude/skills/mendix/cheatsheet-variables/SKILL.md b/.claude/skills/mendix/cheatsheet-variables/SKILL.md index d3558a53bb..083ea3e47f 100644 --- a/.claude/skills/mendix/cheatsheet-variables/SKILL.md +++ b/.claude/skills/mendix/cheatsheet-variables/SKILL.md @@ -23,7 +23,7 @@ Quick reference for variable declarations in MDL microflows. ## Key Rules 1. **Primitives** (String/Integer/Long/Decimal/Boolean/DateTime/Enumeration): Use `declare $var type = value;` — these are the *only* types `declare` (a Create Variable activity) accepts. -2. **Objects (entities)**: Never `declare` an object. Mendix forbids the Create Variable activity from holding an object (CE0053/CE0038, plus CE7247 on a later `set`; flagged as **MDL043** by `mxcli check`) — bare *or* initialized. Get the object from a microflow parameter, a `retrieve … limit 1`, `$var = create Module.Entity(...)`, or a loop iterator. There is no "empty object variable" and no aliasing activity — reuse the variable you already have. +2. **Objects (entities)**: Never `declare` an object. Mendix forbids the Create Variable activity from holding an object (CE0053/CE0038, plus CE7247 on a later `set`; flagged as **MDL043** by `mxcli check`) — bare *or* initialized. Get the object from a microflow parameter, a `retrieve … first`, `$var = create Module.Entity(...)`, or a loop iterator. There is no "empty object variable" and no aliasing activity — reuse the variable you already have. 3. **Lists**: Never `declare` a list — same Create Variable restriction (CE0053/CE0038, flagged as **MDL040**). Get the list from a microflow parameter, a `retrieve`, or `$var = create list of Module.Entity;` 4. **SET requires DECLARE**: Always declare primitive variables before using SET. 5. **Parameters are pre-declared**: Microflow parameters don't need DECLARE (and a parameter *may* be an object/list type — that restriction is only on `declare`). @@ -40,7 +40,7 @@ declare $Product as Module.Product; -- the AS keyword is also a parse er -- CORRECT: get the object from a source that produces one $Product = create Module.Product (Name = $n); -- create -retrieve $Product from Module.Product where Code = $c limit 1; -- retrieve +retrieve $Product from Module.Product where Code = $c first; -- retrieve -- or accept it as a parameter, or use a loop iterator: loop $Product in $Products ... ``` @@ -93,7 +93,7 @@ begin declare $Result boolean = true; -- Local primitive: must declare -- Need a local object? Don't declare it — create/retrieve it: - $Order = create Module.Order (Reference = $Input); -- or retrieve … limit 1 + $Order = create Module.Order (Reference = $Input); -- or retrieve … first return $Result; end; diff --git a/.claude/skills/mendix/check-syntax/SKILL.md b/.claude/skills/mendix/check-syntax/SKILL.md index 973912c2e6..dbed9603e8 100644 --- a/.claude/skills/mendix/check-syntax/SKILL.md +++ b/.claude/skills/mendix/check-syntax/SKILL.md @@ -239,7 +239,7 @@ Before writing any MDL, verify these requirements: **Supported in Microflows:** - `declare $Var type = value;` (primitives only: String/Integer/Long/Decimal/Boolean/DateTime/Enumeration) -- `$entity = create Module.Entity (...);` / `retrieve $entity from ... limit 1;` (objects — **never** `declare` an object; that fails CE0053/CE0038 and is flagged MDL043) +- `$entity = create Module.Entity (...);` / `retrieve $entity from ... first;` (objects — **never** `declare` an object; that fails CE0053/CE0038 and is flagged MDL043) - `$list = create list of Module.Entity;` (lists — **never** `declare` a list; that fails CE0053/CE0038 and is flagged MDL040) - `set $Var = expression;` - `$Var = create Module.Entity (attr = value);` @@ -260,7 +260,7 @@ Before writing any MDL, verify these requirements: **Now Supported (previously not):** - `rollback $entity [refresh];` - Reverts uncommitted changes -- `retrieve ... limit n` - Returns single entity when `limit 1` +- `retrieve ... first` - Returns a single entity; `limit n [offset n]` returns a list (a bare `limit 1` is the object only without the `mdl 1;` header, and warns MDL-V1-LIMIT1) - `boolean` without `default` - Auto-defaults to `false` - `buttonstyle: warning` and `buttonstyle: info` - Now parse correctly - Keywords as attribute names - `caption`, `label`, `title`, `text`, `content`, `format`, `range`, `source`, `check`, etc. all work unquoted diff --git a/.claude/skills/mendix/manage-security/SKILL.md b/.claude/skills/mendix/manage-security/SKILL.md index a1b69e207e..9a839c4b5f 100644 --- a/.claude/skills/mendix/manage-security/SKILL.md +++ b/.claude/skills/mendix/manage-security/SKILL.md @@ -178,8 +178,9 @@ create module role MyModule.Viewer description 'Read-only access'; -- whole security script stays re-runnable rather than needing a run-once file. create or modify module role MyModule.ApiUser description 'API consumer'; --- Remove a module role +-- Remove a module role (`if exists` makes it a no-op when the role is gone) drop module role MyModule.Viewer; +drop module role if exists MyModule.Legacy; ``` ### Microflow Access diff --git a/.claude/skills/mendix/test-microflows/SKILL.md b/.claude/skills/mendix/test-microflows/SKILL.md index 7157a2f37a..dd02d84055 100644 --- a/.claude/skills/mendix/test-microflows/SKILL.md +++ b/.claude/skills/mendix/test-microflows/SKILL.md @@ -405,9 +405,10 @@ with nothing but "the project cannot be deployed". An `@expect` or `@verify` tha cannot be evaluated is reported here too, as `MDL-TEST01`. One rule to know about, because its symptom is confusing and its shape is common -in tests: `retrieve $x … limit 1` binds a **single object**, not a one-element -list, so `head($x)` is `CE0097` at build time and `MDL-RETRIEVE01` at check time. -Drop the `limit` to get a list, or use the variable as the object it is. +in tests: `retrieve $x … first` — and `retrieve $x … limit 1` in a script without +the `mdl 1;` header — binds a **single object**, not a one-element list, so +`head($x)` is `CE0097` at build time and `MDL-RETRIEVE01` at check time. Drop the +range to get a list, or use the variable as the object it is. --- diff --git a/.claude/skills/mendix/write-microflows/SKILL.md b/.claude/skills/mendix/write-microflows/SKILL.md index 1bb0fe8878..ac77a04dd0 100644 --- a/.claude/skills/mendix/write-microflows/SKILL.md +++ b/.claude/skills/mendix/write-microflows/SKILL.md @@ -257,7 +257,7 @@ declare $status Enumeration(Module.OrderStatus) = Module.OrderStatus.Open; > not you give it an initializer. `mxcli check` now flags it as **MDL043**. There > is **no** "empty object variable" activity. Get objects from one of these: > - a microflow **parameter**: `create microflow M.Save ($Product: Test.Product) ...` -> - a **retrieve**: `retrieve $Product from Test.Product where Code = $c limit 1;` +> - a **retrieve**: `retrieve $Product from Test.Product where Code = $c first;` > - a **create object**: `$Product = create Test.Product (Name = $n);` > - a **loop iterator**: `loop $Product in $Products ...` @@ -283,6 +283,14 @@ declare $status Enumeration(Module.OrderStatus) = Module.OrderStatus.Open; > The `calendar*Between` functions (`calendarMonthsBetween`, `calendarYearsBetween`) > return whole units (Integer) and are fine to assign directly. +### Changing a variable: always `set` + +`set $Counter = $Counter + 1;` changes a variable. `$x = …` without `set` is an +activity that creates `$x` — required under `mdl 1;` (`MDL-V1-SET` otherwise). +List operations and aggregates are one statement per activity and never nest: +`$Open = filter $Orders by Status = M.Status.Open;` then `$N = count $Open;` — +see [`reference/data-operations.md`](reference/data-operations.md#one-statement-per-activity). + ### ❌ INCORRECT Syntax ```mdl @@ -579,7 +587,7 @@ Before executing a microflow script, verify: ```mdl declare $primitive type = value; -- Primitives (String/Integer/Decimal/Boolean/DateTime) declare $status Enumeration(Module.Enum) = …; -- Enumerations are primitives too --- Objects: never declare. Use a parameter, retrieve (limit 1), `$obj = create Module.Entity(...)`, or a loop iterator. +-- Objects: never declare. Use a parameter, retrieve (… first), `$obj = create Module.Entity(...)`, or a loop iterator. -- Lists: never declare. Use a parameter, retrieve, or `$list = create list of Module.Entity;` ``` @@ -676,8 +684,8 @@ close page on error { return; }; of them; `continue` is fine on `declare`, `set`, `retrieve`, `delete` and `call microflow`. Measured on 11.14.0 — note that create-*variable* and change-*variable* accept `continue` while change-*object* does not. -- **The list-operation and aggregate forms of `set`** (`$x = head($l)`, - `$n = count($l)`) have no error handling in Mendix at all — **MDL077**. +- **List operations and aggregates** (`$x = head $l;`, `$n = count $l;`) + have no error handling in Mendix at all — **MDL077**. **In a nanoflow, almost none of them take a clause at all.** `change`, `log`, `show page`, `close page`, `show message` and `validation feedback` are CE6035 diff --git a/.claude/skills/mendix/write-microflows/reference/data-operations.md b/.claude/skills/mendix/write-microflows/reference/data-operations.md index 6ed4590d2f..ea22536277 100644 --- a/.claude/skills/mendix/write-microflows/reference/data-operations.md +++ b/.claude/skills/mendix/write-microflows/reference/data-operations.md @@ -117,18 +117,65 @@ add head($SourceItems) to $Items; Use expression-valued `add` only when the expression returns an object compatible with the target list element type. +### One statement per activity + +Every list operation and aggregate is **one Studio Pro activity**, and it is +written as one statement that mirrors it: the keyword is the operation's name +in the activity's dialog, and the inputs are the ones the dialog asks for. The +operand is always a **variable**, because the dialog selects a variable — so +one activity cannot be nested inside another, just as it cannot be drawn. + +| Studio Pro activity: operation | MDL statement | +|---|---| +| List operation: Filter | `$Open = filter $Orders by Status = Shop.Status.Open;` | +| List operation: Filter by expression | `$Big = filter $Orders where $currentObject/Total > 1000;` | +| List operation: Find | `$Order = find $Orders by Number = $Number;` | +| List operation: Find by expression | `$Late = find $Orders where $currentObject/DueDate < [%CurrentDateTime%];` | +| List operation: Sort | `$Sorted = sort $Orders by OrderDate desc, Number asc;` | +| List operation: Head / Tail | `$First = head $Orders;` / `$Rest = tail $Orders;` | +| List operation: Range | `$Page = range $Orders offset 20 limit 10;` | +| List operation: Union / Intersect / Subtract | `$All = union $A with $B;` / `$Both = intersect $A with $B;` / `$Left = subtract $B from $A;` | +| List operation: Contains / Equals | `$Has = contains $Order in $Orders;` / `$Same = equals $A and $B;` | +| Aggregate list: Count | `$N = count $Orders;` | +| Aggregate list: Sum / Average / Minimum / Maximum | `$Total = sum $Orders by Amount;` or `$Total = sum $Orders of $currentObject/Amount * 1.21;` | +| Aggregate list: All / Any | `$AllPaid = all $Orders where $currentObject/Paid;` | +| Aggregate list: Reduce | `$Csv = reduce $Orders from '' as String using $currentResult + $currentObject/Name;` | + +`by` picks a member (the dialog's attribute or association selector) and `where` +/ `of` take an expression — the "… by expression" variants. `subtract $B from $A` +is A minus B. + +Two statements, never one nested call: + +```mdl +$Approved = filter $Orders where $currentObject/Status = Shop.Status.Approved; +$Count = count $Approved; +``` + +The older **call forms** (`$x = filter($L, …)`, `$n = count($L)`, `sum($L.Attr)`) +still parse and build the same activity, but they are deprecated +(`MDL-DEPR003`, `MDL-DEPR004`) — do not write them. Under the `mdl 1;` header: + +- `set` is **mandatory** to change a variable: `set $Total = $Total + 1;`. A + statement without `set` is only ever an activity (`MDL-V1-SET` under mdl 0). +- `$x = find(…)` and `$x = contains(…)` are refused: the call is also Mendix's + **string** function. `set $Pos = find($Text, 'a');` is the string function; + `$Match = find $L where …;` is the list operation (`MDL-V1-LIST`). +- a nested call, or a list operation after `set`, is refused (`MDL-V1-LIST`). + Without the header a nested operand is refused at check time as `MDL-LISTOP02`. + ### `range` — paging a list -`range` takes the **offset first, then the amount**, and Mendix requires at -least one of them: +`range` takes an `offset` and a `limit` (Studio Pro's *Offset* and *Amount*), and +Mendix requires at least one of them: ```mdl -$Page = range($Sorted, $Offset, $PageSize); -- skip $Offset, take $PageSize -$First = range($Sorted, 0, 10); -- first 10 -$Rest = range($Sorted, $Offset); -- skip $Offset, take the rest +$Page = range $Sorted offset $Offset limit $PageSize; -- skip $Offset, take $PageSize +$First = range $Sorted limit 10; -- first 10 +$Rest = range $Sorted offset $Offset; -- skip $Offset, take the rest ``` -`range($List)` with no bound builds nothing useful and fails with **CE6520** +`range $List;` with no bound builds nothing useful and fails with **CE6520** ("Amount and offset are not specified. Either amount or offset or both must be specified."); `mxcli check` refuses it first as **MDL068**. To use the whole list, drop the activity and use the list variable directly. @@ -142,89 +189,95 @@ retrieve $Page from Sales.Order where [Status = 'Open'] sort by OrderDate desc limit $PageSize offset $Offset; ``` -Note the clause order there is `limit` then `offset` — the reverse of `range`'s -argument order, because each mirrors the Mendix editor it comes from. - -### `filter` / `find` test one item at a time — `$currentObject` +### `filter` / `find` — `by` a member, or `where` an expression over `$currentObject` -A FILTER/FIND predicate is an expression Mendix evaluates once per item, with -the item bound to **`$currentObject`**. That is the only iterator name there is: +`by Member = value` is Studio Pro's *Filter* / *Find*: an attribute or +association of the list's entity and the value it must have. Anything else is +the *by expression* operation, written after `where`, which Mendix evaluates +once per item with the item bound to **`$currentObject`** — the only iterator +name there is: ```mdl -$Pending = filter($Orders, $currentObject/Status = 'Pending'); -$Large = filter($Orders, $currentObject/Amount > 1000); -$Match = find($Orders, $currentObject/OrderNumber = $Wanted); +$Pending = filter $Orders by Status = Shop.Status.Pending; +$Large = filter $Orders where $currentObject/Amount > 1000; +$Match = find $Orders by OrderNumber = $Wanted; ``` -A **bare attribute name** means the same thing — mxcli resolves it against the -list's entity and writes `$currentObject/Attr`: +A **bare attribute name** after `where` means the same thing — mxcli resolves it +against the list's entity and writes `$currentObject/Attr`: ```mdl -$Open = filter($Orders, Status != 'Closed'); -- stored as $currentObject/Status +$Open = filter $Orders where Status != 'Closed'; -- stored as $currentObject/Status ``` -Two things are refused rather than passed through to the build: +`by` accepts only `Member = value`; `filter $L by Amount > 3` is refused with a +pointer to `where`. Two things are refused after `where` rather than passed +through to the build: - a bare name that is **not** a member of the list's entity (this used to reach mxbuild as `CE0117 "Error(s) in expression."`); -- any **other** iterator name — `filter($L, $item/Amount > 0)` is `MDL-LISTOP01`, - pre-empting `CE0109 "Undefined variable 'item'"`. +- any **other** iterator name — `filter $L where $item/Amount > 0` is + `MDL-LISTOP01`, pre-empting `CE0109 "Undefined variable 'item'"`. `$item` is still fine when it is genuinely in scope, which is how the O(N) lookup -idiom is written: inside `loop $item in $L`, `find($Others, Key = $item/Key)` +idiom is written: inside `loop $item in $L`, `find $Others by Key = $item/Key` navigates the **loop's** variable on the right-hand side. -**`sort` is not an expression.** It takes attribute names directly, so a bare -attribute is the only spelling — `sort($Orders, CreateDate desc)`. Writing -`$currentObject/` there is wrong. - +**`sort` is not an expression.** It takes attribute names directly — +`sort $Orders by CreateDate desc` — and any word works as a name there, so an +attribute called `Count` or `Date` needs no quotes. Writing `$currentObject/` +there is wrong. -### `contains` is overloaded — string vs list +### `contains` — string function vs list operation -`contains(a, b)` is both a **string** function (`contains(haystack, needle)` → substring test) and a **list** operation (`contains(list, object)` → membership test). mxcli picks the right serialization automatically: +The list operation is `contains $Object in $List`, and it creates its own +Boolean output variable, so do not declare it first. The **string** function +`contains(haystack, needle)` is an expression, assigned with `set` to a variable +declared first: ```mdl +-- LIST contains — the statement creates $Found +$Found = contains $Item in $Items; + -- STRING contains — assign to a PRE-DECLARED Boolean (a Change Variable action) declare $HasAt Boolean = false; set $HasAt = contains($Email, '@'); - --- LIST contains — do NOT pre-declare the output (the list op creates it) -set $Found = contains($Items, $Item); ``` -The distinction: a **literal or computed** second argument is always the string function. When both arguments are plain variables, the input variable's declared type decides — a **String** input becomes the string function (Change Variable, so declare the Boolean first), anything else stays a list operation (which creates its own output variable, so leave it undeclared). Getting the declare wrong is what triggers `CE0111 "Duplicate variable name"`. +Under `mdl 1;` that is the whole rule. Without the header, the call form +`$Found = contains($Items, $Item)` is still read as the list operation when both +arguments are plain variables and the first is not a declared String — the +guess that `mdl 1` removes. Getting the declare wrong is what triggers `CE0111 +"Duplicate variable name"`. ### Aggregates — all eight, including `reduce`, `all` and `any` An Aggregate list activity folds a list into one value. `count` takes only the -list; the rest take either an **attribute** or an **expression** over -`$currentObject`. +list; `sum`, `average`, `minimum` and `maximum` aggregate an attribute (`by`) or +an expression over `$currentObject` (`of`) — the dialog's *Aggregate with*. ```mdl -$Count = count($Orders); -$Total = sum($Orders.Amount); -- attribute form -$Total = sum($Orders, $currentObject/Amount * 1.21); -- expression form -$Avg = average($Orders.Amount); -$Min = minimum($Orders.Amount); -$Max = maximum($Orders.Amount); +$Count = count $Orders; +$Total = sum $Orders by Amount; -- attribute +$Total = sum $Orders of $currentObject/Amount * 1.21; -- expression +$Avg = average $Orders by Amount; +$Min = minimum $Orders by Amount; +$Max = maximum $Orders by Amount; -- Boolean predicates over every item. No seed, always Boolean. -$AllPaid = all($Orders, $currentObject/Paid); -$AnyLate = any($Orders, $currentObject/DueDate < [%CurrentDateTime%]); +$AllPaid = all $Orders where $currentObject/Paid; +$AnyLate = any $Orders where $currentObject/DueDate < [%CurrentDateTime%]; -- REDUCE folds with a running total. $currentResult is the accumulator. -$Discounted = reduce( - $Orders, - $currentResult + $currentObject/Amount * 0.9, - initial: 0, - returns: Decimal -); +$Discounted = reduce $Orders from 0 as Decimal + using $currentResult + $currentObject/Amount * 0.9; ``` -**`reduce` needs `initial:` and `returns:` and neither can be inferred.** Mendix -stores both beside the expression, and the fold is meaningless without a seed -and a result type — so MDL makes them mandatory rather than guessing. `all` and -`any` take neither: they never accumulate, and always fold to Boolean. +**`reduce` needs the initial value (`from`) and the return type (`as`), and +neither can be inferred.** Mendix stores both beside the expression, and the +fold is meaningless without a seed and a result type — so MDL makes them +mandatory rather than guessing. `all` and `any` take neither: they never +accumulate, and always fold to Boolean. Do not reach for `reduce` where `sum` will do. It exists for folds Mendix has no dedicated function for — running a string together, or carrying a value forward @@ -254,9 +307,10 @@ retrieve $Product from Test.Product **Important**: - Use `from Module.Entity` (fully qualified) -- RETRIEVE with `limit 1` returns a **single entity** -- RETRIEVE without `limit 1` returns a **list** (`list of Module.Entity`) -- Use `limit 1` when you expect exactly one result (e.g., lookup by unique key) +- RETRIEVE with `first` returns a **single entity** +- RETRIEVE without a range, or with `limit n [offset n]`, returns a **list** (`list of Module.Entity`) +- Use `first` when you expect exactly one result (e.g., lookup by unique key). `limit 1` + is a list of one under `mdl 1;` and the object without the header (warning `MDL-V1-LIMIT1`) **Sorting and paging** — use `sort by`, **not** `order by`: diff --git a/.claude/skills/mendix/write-microflows/reference/pitfalls.md b/.claude/skills/mendix/write-microflows/reference/pitfalls.md index be632b05ed..01e0dd4abb 100644 --- a/.claude/skills/mendix/write-microflows/reference/pitfalls.md +++ b/.claude/skills/mendix/write-microflows/reference/pitfalls.md @@ -22,7 +22,7 @@ declare $Product as Test.Product; -- AS keyword also not supported create microflow Test.Save ($Product: Test.Product) returns boolean as $ok ... -- from a retrieve (single object) -retrieve $Product from Test.Product where Code = $Code limit 1; +retrieve $Product from Test.Product where Code = $Code first; -- from a create object $Product = create Test.Product (Name = $Name); @@ -361,17 +361,21 @@ rollback $Order refresh; **Use Case**: Revert uncommitted changes to an object. Useful when validation fails and you want to restore the object to its database state. -### RETRIEVE with LIMIT (Supported!) +### RETRIEVE: `first` for an object, `limit` for a list ```mdl --- CORRECT: LIMIT is supported -retrieve $Product from Module.Product where IsActive = true limit 1; +-- FIRST binds a single object (Mendix's "First object" range) +retrieve $Product from Module.Product where IsActive = true first; --- LIMIT 1 returns a single entity (not a list) --- Without LIMIT, returns a list +-- Without a range, or with LIMIT/OFFSET, it binds a list retrieve $ProductList from Module.Product where IsActive = true; +retrieve $Page from Module.Product sort by Name asc limit 20 offset 40; ``` +`limit 1` without `offset` changes meaning with the language version: a list of +one under `mdl 1;`, but in a script without the header the object, with warning +`MDL-V1-LIMIT1`. Write `first` whenever you mean the object. + ### WHILE Loop ```mdl diff --git a/CHANGELOG.md b/CHANGELOG.md index 6c7f46a673..a3a71284cb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Changed +- **`retrieve … first` binds one object; under `mdl 1;` `retrieve … limit 1` is a list of one** (ako/mxcli#734, ADR-0011) — `first` is Mendix's "First object" range in every language version, the same split as `import from mapping … first | limit n`. Under the `mdl 1;` preview header a bare `limit 1` is a Custom range, a list of one. Without the header it keeps its old meaning, the object, and `check` / `exec` warn **MDL-V1-LIMIT1**. `describe` prints the object range as `first`, so its output means the same whatever header runs it; measured on the Studio Pro-authored PedApp fixture (describe → exec keeps both ranges as stored) and with `mx check` 11.13 on a copy of a Studio Pro project (a loop over an `mdl 1` `limit 1` builds at 0 errors; the headerless script is CE0100). **MDL-RETRIEVE01** now keys on the object range the writer stores and names CE0100 for a loop. `first` on an association retrieve is refused: Mendix gives that source no range. **Migrating a script:** nothing to do — without the header every script behaves as before; replace an object-meaning `limit 1` with `first` to silence the warning and to keep the meaning under `mdl 1;`. + - **`alter page`, `alter snippet` and `alter layout` are the first document types on the generic ALTER** (ako/mxcli#712, ADR-0012) — one grammar, `alter <type> Module.Name { set (Key: value) on <target>; insert before|after|into <target> { … } replace <target> with { … } drop <target>; }`, whose target is resolved by the document type (a new `backend.AlterTargetResolver`, implemented by the page mutator on the modelsdk and `--mcp` backends). Properties are written as in `create`: `set (Caption: 'Save') on btnSave`, `set (Title: 'Edit')` for the page itself. A target is a widget name, `grid.Column` or `layoutContainer.top`; a quoted-caption target or `@n` is refused on a page, whose elements have names — by `check` too, as **MDL-ALTER01**. **Migrating a script:** nothing breaks — `set Key = value`, `set Key: value` without parentheses and `drop widget a` still run and build the identical change, and `check` / `exec` warn with **MDL-DEPR101**, **MDL-DEPR102** and **MDL-DEPR103** naming the rewrite. - **`DynamicClasses` and a column's `DynamicCellClass` are written as Mendix expressions** (mendixlabs/mxcli#750) — the expression is written as-is, so the doubled-quote spelling is gone: `dynamicclasses: if $currentObject/Featured then 'is-featured' else ''`, and `dynamicclasses: 'is-featured'` is the string — the class `is-featured`. The same rule as the OData client's credentials. `create page`, `alter page … set` and `describe` all use it, and a describe → exec round trip stores identical values (measured on a Mendix 11.14.0 project). **Migrating a script:** the old spelling, the expression's text in quotes (`'if … then ''a'' else '''''`), still parses but would now store that text as a class name, so `check` and `exec` refuse it as **MDL-WIDGET33** and give the unquoted expression. An expression in any other widget property is an error rather than an empty value; a pluggable property whose schema kind is Expression (a column's `Visible`, for one) keeps the quoted form until a following change. diff --git a/CLAUDE.md b/CLAUDE.md index 7384391049..6afee35f56 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -363,7 +363,7 @@ These rules apply whenever generating microflow or nanoflow MDL. Violations are 2. **NEVER use nested LOOPs for list matching.** Loop over the primary list and use `$match = FIND($TargetList, key = $item/key)` for an O(N) in-memory lookup. A plain `retrieve … where` **cannot** filter a list variable (only a database/association source), so `retrieve $match from $TargetList where …` is a parse error — use `FIND`/`FILTER`. Nested loops are O(N^2). The `$item` there is the enclosing loop's iterator and stays valid — MDL-LISTOP01 flags a predicate variable that is *not in scope*, not the name. Inside the predicate itself, the item under test is `$currentObject` (a bare attribute name resolves to it). 3. **NEVER nest one list operation inside another.** Each of HEAD/TAIL/FIND/FILTER/SORT/UNION/INTERSECT/SUBTRACT/RANGE and the aggregates is a separate **activity**, and an activity stores its list as a **variable reference** — there is no slot for a nested computation. `$n = COUNT(FILTER($reqs, …))` parses, and used to drop the inner call entirely and write an activity with an empty list: `check` clean, `exec` reporting "Created microflow", then CE0012 / CE0096 at build time — and `sort(filter(…), Attr)` made mxbuild abort outright, because the sort attribute resolves against the now-absent list's entity. One statement each: `$approved = FILTER($reqs, …); $n = COUNT($approved);`. `mxcli check` now refuses the nested form as MDL-LISTOP02 (mendixlabs/mxcli#1101). 4. **Use append logic when merging**, not overwrite: `$Existing/Field + '\n' + $New/Field` inside an `if $New/Field != empty` guard. -5. **`retrieve … limit 1` binds a single OBJECT, not a one-element list** — it is Mendix's "First object" range, so `head()`, `count()` or a `loop` over that variable is **CE0097**. Drop the `limit` for a list; `limit 1 offset n` and every other `limit` ARE lists. Note the same word means the opposite on `import from mapping`, where `first` binds the object and `limit 1` a one-element list. `describe` re-emits `limit 1` either way, so the source of an object retrieve and a list retrieve are identical text and only MDL-RETRIEVE01 distinguishes them before a build (mendixlabs/mxcli#1103). +5. **Write `retrieve … first` for a single OBJECT** — Mendix's "First object" range; `head()`/`count()` over it is **CE0097** and a `loop` **CE0100** (MDL-RETRIEVE01 at check time). `limit n [offset n]` is a list. **`limit 1` without `offset` changes meaning with the language version** (ako/mxcli#734): a list of one under `mdl 1;`, the object in a headerless script, where it warns MDL-V1-LIMIT1. `describe` prints `first` for the object range, matching `import from mapping … first | limit n`. 6. **Read `.claude/skills/patterns-data-processing.md`** for delta merge, batch processing, and list operation patterns. **Always validate before presenting to user:** diff --git a/cmd/mxcli/cmd_fmt.go b/cmd/mxcli/cmd_fmt.go index 928c364a10..1bf949c83e 100644 --- a/cmd/mxcli/cmd_fmt.go +++ b/cmd/mxcli/cmd_fmt.go @@ -6,10 +6,13 @@ import ( "fmt" "io" "os" + "sort" "strings" + "github.com/mendixlabs/mxcli/mdl/deprecation" "github.com/mendixlabs/mxcli/mdl/formatter" "github.com/mendixlabs/mxcli/mdl/langver" + "github.com/mendixlabs/mxcli/mdl/upgrade" "github.com/mendixlabs/mxcli/mdl/visitor" "github.com/spf13/cobra" ) @@ -35,10 +38,31 @@ Examples: # Format from stdin (pipe) mxcli describe microflow Mod.MF | mxcli fmt mxcli describe microflow Mod.MF | mxcli fmt - + +Upgrading (--upgrade): + Rewrites every deprecated spelling registered in the deprecation registry to + its canonical form (create or replace -> create or modify, show -> list, ...) + and changes nothing else: comments, layout and keyword case are kept, and the + heuristic formatting above is not applied. A deprecated use that has no + mechanical rewrite is reported on stderr and left in place. + + --header also adds the language header (mdl 1;), after rewriting every + construct whose meaning the header would change. A construct without such a + rewrite blocks the header, and fmt fails rather than change the script's + meaning. While mdl 1 is a preview the header is added only when asked. + + # Upgrade in place + mxcli fmt --upgrade -w script.mdl + mxcli fmt --upgrade --header -w script.mdl `, Args: cobra.MaximumNArgs(1), RunE: func(cmd *cobra.Command, args []string) error { writeInPlace, _ := cmd.Flags().GetBool("write") + doUpgrade, _ := cmd.Flags().GetBool("upgrade") + addHeader, _ := cmd.Flags().GetBool("header") + if cmd.Flags().Changed("header") && !doUpgrade { + return fmt.Errorf("--header needs --upgrade") + } // Determine source: stdin when no arg or "-" is passed. fromStdin := len(args) == 0 || args[0] == "-" @@ -84,13 +108,34 @@ Examples: return fmt.Errorf("no valid MDL statements found in %s", label) } - formatted := formatter.Format(string(data)) + var formatted string + if doUpgrade { + opts := upgrade.DefaultOptions() + if cmd.Flags().Changed("header") { + opts.AddHeader = addHeader + } + res, err := upgrade.Upgrade(string(data), opts) + if err != nil { + return fmt.Errorf("%s: %w", label, err) + } + reportUpgrade(os.Stderr, label, res) + formatted = res.Source + } else { + formatted = formatter.Format(string(data)) + } if writeInPlace { + if doUpgrade && formatted == string(data) { + return nil // nothing to upgrade: leave the file untouched + } if err := os.WriteFile(filePath, []byte(formatted), 0644); err != nil { return fmt.Errorf("failed to write file: %w", err) } - fmt.Fprintf(os.Stderr, "Formatted %s\n", filePath) + verb := "Formatted" + if doUpgrade { + verb = "Upgraded" + } + fmt.Fprintf(os.Stderr, "%s %s\n", verb, filePath) } else { fmt.Print(formatted) } @@ -101,6 +146,39 @@ Examples: func init() { fmtCmd.Flags().BoolP("write", "w", false, "Write result to source file instead of stdout") + fmtCmd.Flags().Bool("upgrade", false, "Rewrite deprecated spellings to their canonical form, changing nothing else") + fmtCmd.Flags().Bool("header", false, "With --upgrade: add the mdl 1 language header (opt-in while mdl 1 is a preview)") +} + +// reportUpgrade prints what an upgrade did, and what it left, on w. +func reportUpgrade(w io.Writer, label string, res upgrade.Result) { + codes := func(m map[string]int) string { + var parts []string + for c, n := range m { + parts = append(parts, fmt.Sprintf("%s x%d", c, n)) + } + sort.Strings(parts) + return strings.Join(parts, ", ") + } + if len(res.Rewritten) > 0 { + fmt.Fprintf(w, "%s: rewrote %s\n", label, codes(res.Rewritten)) + } + if len(res.GatedRewritten) > 0 { + fmt.Fprintf(w, "%s: rewrote for the header %s\n", label, codes(res.GatedRewritten)) + } + if res.HeaderAdded { + fmt.Fprintf(w, "%s: added the language header\n", label) + } + for _, d := range res.Unrewritten { + msg := d.Code + if e, ok := deprecation.Lookup(d.Code); ok { + msg = fmt.Sprintf("%s (%s -> %s)", d.Code, e.Old, e.Canonical) + } + if d.NoFix != "" { + msg += ": " + d.NoFix + } + fmt.Fprintf(w, "%s:%d:%d: not upgraded, no mechanical rewrite: %s\n", label, d.Line, d.Column+1, msg) + } } // hasSubstantiveContent reports whether s contains at least one non-blank, diff --git a/cmd/mxcli/cmd_fmt_upgrade_test.go b/cmd/mxcli/cmd_fmt_upgrade_test.go new file mode 100644 index 0000000000..4ea239e944 --- /dev/null +++ b/cmd/mxcli/cmd_fmt_upgrade_test.go @@ -0,0 +1,90 @@ +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "bytes" + "context" + "os" + "path/filepath" + "strings" + "testing" +) + +func runFmt(t *testing.T, args ...string) (string, error) { + t.Helper() + // cobra keeps flag values between runs in one process. + for _, f := range []string{"write", "upgrade", "header"} { + _ = fmtCmd.Flags().Set(f, "false") + fmtCmd.Flags().Lookup(f).Changed = false + } + var out bytes.Buffer + rootCmd.SetOut(&out) + rootCmd.SetErr(&out) + rootCmd.SetArgs(append([]string{"fmt"}, args...)) + err := rootCmd.ExecuteContext(context.Background()) + return out.String(), err +} + +func TestFmtUpgrade_WritesOnlyTheRewrites(t *testing.T) { + path := filepath.Join(t.TempDir(), "s.mdl") + src := "-- keep me\nCREATE OR REPLACE entity M.User (Name: String);\n show entities;\n" + if err := os.WriteFile(path, []byte(src), 0o644); err != nil { + t.Fatal(err) + } + if _, err := runFmt(t, "--upgrade", "-w", path); err != nil { + t.Fatal(err) + } + got, _ := os.ReadFile(path) + // Only the deprecated keywords change: no heuristic upper-casing (which + // would turn M.User into M.USER), no re-indentation, and no header while + // mdl 1 is a preview. + want := "-- keep me\nCREATE OR MODIFY entity M.User (Name: String);\n list entities;\n" + if string(got) != want { + t.Fatalf("got:\n%s\nwant:\n%s", got, want) + } + + if _, err := runFmt(t, "--upgrade", "--header", "-w", path); err != nil { + t.Fatal(err) + } + got, _ = os.ReadFile(path) + if !strings.HasPrefix(string(got), "mdl 1;\n-- keep me\n") { + t.Fatalf("--header did not add the header:\n%s", got) + } + + // Idempotent: a second run leaves the file as it is. + before := string(got) + if _, err := runFmt(t, "--upgrade", "--header", "-w", path); err != nil { + t.Fatal(err) + } + if got, _ = os.ReadFile(path); string(got) != before { + t.Fatalf("second fmt --upgrade changed the file:\n%s", got) + } +} + +func TestFmtUpgrade_HeaderNeedsUpgrade(t *testing.T) { + path := filepath.Join(t.TempDir(), "s.mdl") + if err := os.WriteFile(path, []byte("show entities;\n"), 0o644); err != nil { + t.Fatal(err) + } + if _, err := runFmt(t, "--header", path); err == nil || !strings.Contains(err.Error(), "--upgrade") { + t.Fatalf("--header without --upgrade: %v", err) + } +} + +// A construct with no mechanical rewrite is reported with its reason, the +// header is refused, and the file is left as it was. +func TestFmtUpgrade_ReportsWhatItCannotRewrite(t *testing.T) { + path := filepath.Join(t.TempDir(), "s.mdl") + src := "create microflow M.F ($L: List of M.E) begin\n $n = count(filter($L, Name = 'x'));\nend\n/\n" + if err := os.WriteFile(path, []byte(src), 0o644); err != nil { + t.Fatal(err) + } + _, err := runFmt(t, "--upgrade", "--header", "-w", path) + if err == nil || !strings.Contains(err.Error(), "MDL-V1-LIST") || !strings.Contains(err.Error(), "nested call") { + t.Fatalf("want the nested list operation reported with its reason, got %v", err) + } + if got, _ := os.ReadFile(path); string(got) != src { + t.Fatalf("the file was changed although the header was refused:\n%s", got) + } +} diff --git a/cmd/mxcli/lsp_completions_gen.go b/cmd/mxcli/lsp_completions_gen.go index 739fd48d89..19568e7d73 100644 --- a/cmd/mxcli/lsp_completions_gen.go +++ b/cmd/mxcli/lsp_completions_gen.go @@ -166,6 +166,7 @@ var mdlGeneratedKeywords = []protocol.CompletionItem{ {Label: "REDUCE", Kind: protocol.CompletionItemKindKeyword, Detail: "Microflow keyword"}, {Label: "ANY", Kind: protocol.CompletionItemKindKeyword, Detail: "Microflow keyword"}, {Label: "INITIAL", Kind: protocol.CompletionItemKindKeyword, Detail: "Microflow keyword"}, + {Label: "USING", Kind: protocol.CompletionItemKindKeyword, Detail: "Microflow keyword"}, {Label: "LIST", Kind: protocol.CompletionItemKindKeyword, Detail: "Microflow keyword"}, {Label: "REMOVE", Kind: protocol.CompletionItemKindKeyword, Detail: "Microflow keyword"}, {Label: "EQUALS", Kind: protocol.CompletionItemKindKeyword, Detail: "Microflow keyword"}, diff --git a/cmd/mxcli/syntax/features_microflow.go b/cmd/mxcli/syntax/features_microflow.go index bc7dc3a74a..74d1d3af0e 100644 --- a/cmd/mxcli/syntax/features_microflow.go +++ b/cmd/mxcli/syntax/features_microflow.go @@ -84,23 +84,24 @@ func init() { }, // Retrieve-by-association was missing here, so it read as unsupported // even though it works and the write-microflows skill documents it. - Syntax: "-- From the database\nRETRIEVE $Var FROM Module.Entity\n [WHERE condition]\n [SORT BY attr ASC|DESC]\n [LIMIT n] [OFFSET n];\n\n-- Sort over an association: one `/` per hop, the last segment is the attribute\nRETRIEVE $Var FROM Module.Entity\n SORT BY Module.Assoc/Module.Other.Attr ASC;\n\n-- Over an association, from an object you already have\nRETRIEVE $Var FROM $Object/Module.Association;", - Example: "-- LIMIT 1 binds a single OBJECT (Mendix's \"First object\" range), not a\n" + + Syntax: "-- From the database\nRETRIEVE $Var FROM Module.Entity\n [WHERE condition]\n [SORT BY attr ASC|DESC]\n [FIRST | [LIMIT n] [OFFSET n]];\n\n-- Sort over an association: one `/` per hop, the last segment is the attribute\nRETRIEVE $Var FROM Module.Entity\n SORT BY Module.Assoc/Module.Other.Attr ASC;\n\n-- Over an association, from an object you already have\nRETRIEVE $Var FROM $Object/Module.Association;", + Example: "-- FIRST binds a single OBJECT (Mendix's \"First object\" range), not a\n" + "-- one-element list — hence the singular variable name here.\n" + - "RETRIEVE $Customer FROM MyModule.Customer\n WHERE Code = $CustomerCode\n LIMIT 1;\n\n" + - "-- Any other LIMIT is a bounded range, which is a list.\n" + + "RETRIEVE $Customer FROM MyModule.Customer\n WHERE Code = $CustomerCode\n FIRST;\n\n" + + "-- LIMIT/OFFSET is a bounded range, which is a list.\n" + "RETRIEVE $Orders FROM MyModule.Order\n WHERE Status = 'Pending'\n SORT BY CreateDate DESC\n LIMIT 10 OFFSET 0;\n\n" + "-- Follow an association rather than querying the database\nRETRIEVE $Orders FROM $Customer/MyModule.Order_Customer;\nRETRIEVE $Customer FROM $Order/MyModule.Order_Customer;\n\n" + "-- Sort on an attribute of an associated entity. Name the association\n" + "-- when two of them reach the same entity.\n" + "RETRIEVE $Orders FROM MyModule.Order\n SORT BY MyModule.Order_BillTo/MyModule.Address.City ASC;\n\n" + "-- Notes:\n" + - "-- * LIMIT 1 with no OFFSET is the one form that binds an object. HEAD(),\n" + - "-- COUNT() or a LOOP over it is CE0097 at build time; mxcli reports it as\n" + - "-- MDL-RETRIEVE01 at check time.\n" + - "-- * LIMIT 1 OFFSET n is a bounded range, so that one IS a list.\n" + - "-- * `import from mapping … limit 1` means the opposite — a one-element\n" + - "-- list — and `… first` is its object form.\n" + + "-- * FIRST is the one form that binds an object, in every language version.\n" + + "-- HEAD() or COUNT() over it is CE0097 at build time and a LOOP CE0100;\n" + + "-- mxcli reports both as MDL-RETRIEVE01 at check time.\n" + + "-- * LIMIT 1 without OFFSET depends on the language version: under `mdl 1;`\n" + + "-- it is a list of one, as in `import from mapping … limit 1`. Without the\n" + + "-- header it keeps its old meaning, the object, and warns MDL-V1-LIMIT1 —\n" + + "-- write FIRST for the object. LIMIT 1 OFFSET n is always a list.\n" + "-- * SORT BY may navigate associations. Name the hop when more than one\n" + "-- reaches the same entity — mxcli infers a single hop, but it cannot\n" + "-- tell Order_ShipTo from Order_BillTo, and the wrong one builds\n" + @@ -176,8 +177,8 @@ func init() { "-- FEEDBACK -> MDL076. A custom handler IS accepted on all of them, and\n" + "-- CONTINUE is fine on DECLARE, SET, RETRIEVE, DELETE and CALL MICROFLOW.\n" + "--\n" + - "-- The list-operation and aggregate forms of SET ($x = head($l),\n" + - "-- $n = count($l)) have no error handling in Mendix at all -> MDL077.\n" + + "-- List operations and aggregates ($x = head $l, $n = count $l) have\n" + + "-- no error handling in Mendix at all -> MDL077.\n" + "--\n" + "-- IN A NANOFLOW only DECLARE and SET take a clause at all. CHANGE, LOG,\n" + "-- SHOW PAGE, CLOSE PAGE, SHOW MESSAGE and VALIDATION FEEDBACK are CE6035\n" + @@ -383,9 +384,67 @@ func init() { // A CE number in the cached `syntax --json` index leads an agent // debugging the build error back to the right topic (issue #1002). "$currentObject", "predicate", "CE0117", "CE0109", "MDL-LISTOP01", + "contains", "equals", "by", "where", "MDL-DEPR003", "MDL-DEPR004", "MDL-V1-LIST", }, - Syntax: "$List = CREATE LIST OF Module.Entity;\nADD $Item TO $List;\nREMOVE $Item FROM $List;\n$Result = HEAD($List);\n$Result = TAIL($List);\n$Result = FIND($List, predicate);\n$Result = FILTER($List, predicate);\n$Result = SORT($List, attr ASC);\n$Result = UNION($L1, $L2);\n$Result = INTERSECT($L1, $L2);\n$Result = SUBTRACT($L1, $L2);\n$Result = RANGE($List, offset, amount);\n$Result = RANGE($List, offset);\n\n-- Aggregates. Mendix has eight; each takes an attribute or an expression\n-- over $currentObject.\n$Count = COUNT($List);\n$Sum = SUM($List.Attr);\n$Sum = SUM($List, expression);\n$Avg = AVERAGE($List.Attr);\n$Min = MINIMUM($List.Attr);\n$Max = MAXIMUM($List.Attr);\n$AllMatch = ALL($List, boolean-expression);\n$AnyMatch = ANY($List, boolean-expression);\n\n-- REDUCE folds the list into one value. $currentResult is the running\n-- total; both INITIAL and RETURNS are required and cannot be inferred.\n$Folded = REDUCE($List, expression, initial: value, returns: Type);\n\n-- RANGE takes OFFSET first, then AMOUNT, and needs at least ONE of them:\n-- RANGE($L, $Offset, $Amount) page: skip $Offset, take $Amount\n-- RANGE($L, 0, $Amount) first $Amount\n-- RANGE($L, $Offset) skip $Offset, take the rest\n-- RANGE($L) with no bound is CE6520 at build time (mxcli check: MDL068).\n\nA FIND/FILTER predicate is evaluated once per item, and Mendix binds the\nitem to $currentObject -- the same variable the aggregate expressions above\nuse, and the only iterator name there is:\n\n FILTER($Orders, $currentObject/Amount > 0)\n\nA bare attribute name means the same thing; mxcli resolves it against the\nlist's entity and writes $currentObject/Attr. A name that is not a member\nof that entity is refused, and naming any other variable is MDL-LISTOP01.\n\nSORT is not an expression -- it takes attribute names directly, so a bare\nattribute is the only spelling there.\n\nEvery list operation above is a separate ACTIVITY, and an activity stores\nits list as a VARIABLE. They do not nest: COUNT(FILTER($L, ...)) is not a\nshorter spelling of two statements, it is a list argument Mendix cannot\nstore. mxcli refuses it as MDL-LISTOP02; give the inner operation its own\nstatement and pass the variable:\n\n $Approved = FILTER($Orders, $currentObject/Status = 'Approved');\n $Count = COUNT($Approved);", - Example: "$AllOrders = CREATE LIST OF MyModule.Order;\nADD $NewOrder TO $AllOrders;\n$First = HEAD($AllOrders);\n\n-- The item under test is $currentObject\n$Pending = FILTER($AllOrders, $currentObject/Status = 'Pending');\n$Large = FILTER($AllOrders, $currentObject/Amount > 1000);\n\n-- A bare attribute name is resolved against the list's entity\n$Open = FILTER($AllOrders, Status != 'Closed');\n\n-- SORT takes attribute names, not an expression\n$Sorted = SORT($Pending, CreateDate DESC);\n$Page = RANGE($Sorted, $Offset, $PageSize);\n$Total = SUM($AllOrders.Amount);\n$AllPaid = ALL($AllOrders, $currentObject/Paid);\n$AnyLate = ANY($AllOrders, $currentObject/DueDate < [%CurrentDateTime%]);\n\n-- List operations do not nest -- one statement each (MDL-LISTOP02)\n-- WRONG: $Count = COUNT(FILTER($AllOrders, $currentObject/Paid));\n$Paid = FILTER($AllOrders, $currentObject/Paid);\n$Count = COUNT($Paid);\n$Discounted = REDUCE(\n $AllOrders,\n $currentResult + $currentObject/Amount * 0.9,\n initial: 0,\n returns: Decimal\n);", + Syntax: "$List = CREATE LIST OF Module.Entity;\nADD $Item TO $List;\nREMOVE $Item FROM $List;\n\n" + + "-- One statement per Studio Pro activity. The keyword is the operation's\n" + + "-- name and the operand is always a variable, as in the activity's dialog.\n" + + "-- List operation:\n" + + "$Result = HEAD $List;\n$Result = TAIL $List;\n" + + "$Result = FIND $List BY Member = value; -- Find (attribute or association)\n" + + "$Result = FIND $List WHERE expression; -- Find by expression\n" + + "$Result = FILTER $List BY Member = value; -- Filter\n" + + "$Result = FILTER $List WHERE expression; -- Filter by expression\n" + + "$Result = SORT $List BY Attr DESC, Attr2 ASC;\n" + + "$Result = UNION $L1 WITH $L2;\n$Result = INTERSECT $L1 WITH $L2;\n" + + "$Result = SUBTRACT $L2 FROM $L1; -- $L1 minus $L2\n" + + "$Bool = CONTAINS $Object IN $List;\n$Bool = EQUALS $L1 AND $L2;\n" + + "$Result = RANGE $List OFFSET offset LIMIT amount;\n\n" + + "-- Aggregate list: BY an attribute, or OF an expression over $currentObject.\n" + + "$Count = COUNT $List;\n$Sum = SUM $List BY Attr;\n$Sum = SUM $List OF expression;\n" + + "$Avg = AVERAGE $List BY Attr;\n$Min = MINIMUM $List BY Attr;\n$Max = MAXIMUM $List BY Attr;\n" + + "$AllMatch = ALL $List WHERE boolean-expression;\n$AnyMatch = ANY $List WHERE boolean-expression;\n\n" + + "-- REDUCE folds the list into one value. $currentResult is the running\n" + + "-- total; the initial value and the type are required and cannot be inferred.\n" + + "$Folded = REDUCE $List FROM initial AS Type USING expression;\n\n" + + "-- RANGE needs at least ONE bound:\n" + + "-- RANGE $L OFFSET $Offset LIMIT $Amount page: skip $Offset, take $Amount\n" + + "-- RANGE $L LIMIT $Amount first $Amount\n" + + "-- RANGE $L OFFSET $Offset skip $Offset, take the rest\n" + + "-- RANGE with no bound is CE6520 at build time (mxcli check: MDL068).\n\n" + + "BY names a member and the value it must have; anything else goes after\n" + + "WHERE, which is evaluated once per item with the item bound to\n" + + "$currentObject -- the same variable the aggregate expressions use:\n\n" + + " FILTER $Orders WHERE $currentObject/Amount > 0\n\n" + + "A bare attribute name after WHERE means the same thing; mxcli resolves it\n" + + "against the list's entity and writes $currentObject/Attr. Naming any\n" + + "other variable there is MDL-LISTOP01.\n\n" + + "Because the operand is a variable, activities cannot nest. Give each\n" + + "its own statement:\n\n" + + " $Approved = FILTER $Orders WHERE $currentObject/Status = 'Approved';\n" + + " $Count = COUNT $Approved;\n\n" + + "The call forms ($x = FILTER($L, ...), $n = COUNT($L)) are deprecated\n" + + "aliases (MDL-DEPR003, MDL-DEPR004). $x = FIND(...) and CONTAINS(...) are\n" + + "also Mendix's string functions: under `mdl 1;` the call form is refused\n" + + "and `SET $x = find($Text, 'a')` is always the string function\n" + + "(MDL-V1-LIST). Without the header a nested call is refused as\n" + + "MDL-LISTOP02.", + Example: "$AllOrders = CREATE LIST OF MyModule.Order;\nADD $NewOrder TO $AllOrders;\n$First = HEAD $AllOrders;\n\n" + + "-- Filter by member, and by expression over $currentObject\n" + + "$Pending = FILTER $AllOrders BY Status = MyModule.OrderStatus.Pending;\n" + + "$Large = FILTER $AllOrders WHERE $currentObject/Amount > 1000;\n" + + "$Match = FIND $AllOrders BY OrderNumber = $Number;\n\n" + + "-- SORT takes attribute names, not an expression\n" + + "$Sorted = SORT $Pending BY CreateDate DESC;\n" + + "$Page = RANGE $Sorted OFFSET $Offset LIMIT $PageSize;\n" + + "$Total = SUM $AllOrders BY Amount;\n" + + "$AllPaid = ALL $AllOrders WHERE $currentObject/Paid;\n" + + "$AnyLate = ANY $AllOrders WHERE $currentObject/DueDate < [%CurrentDateTime%];\n\n" + + "-- One statement per activity\n" + + "$Paid = FILTER $AllOrders WHERE $currentObject/Paid;\n" + + "$Count = COUNT $Paid;\n" + + "$Discounted = REDUCE $AllOrders FROM 0 AS Decimal\n" + + " USING $currentResult + $currentObject/Amount * 0.9;", SeeAlso: []string{"microflow.retrieve"}, }) @@ -400,6 +459,41 @@ func init() { Example: "LOG INFO NODE 'OrderService' 'Order created successfully';\nLOG WARNING 'Customer not found';\nLOG ERROR 'Failed to process {1}' WITH (\n {1} = $OrderNumber\n);", }) + Register(SyntaxFeature{ + Path: "microflow.alter", + Summary: "Patch a stored microflow or nanoflow: insert, replace or drop activities in place", + Keywords: []string{ + "alter microflow", "alter nanoflow", "insert after", "insert before", + "replace", "drop activity", "patch microflow", "splice", "handle", + }, + Syntax: "ALTER MICROFLOW|NANOFLOW Module.Name {\n" + + " INSERT AFTER|BEFORE <target> { <statements> }\n" + + " REPLACE <target> WITH { <statements> }\n" + + " DROP <target>;\n" + + "};\n\n" + + "-- <target> addresses one activity by content, as `describe microflow ... with handles` prints it:\n" + + "-- $Var the activity that outputs $Var\n" + + "-- 'Caption' a decision or an activity with a custom caption\n" + + "-- <statement> a statement pattern; * matches any run of tokens\n" + + "-- followed by @n when it matches more than one. Targets are resolved against the\n" + + "-- stored flow before any operation runs; an ambiguous or unknown target is an error.\n" + + "-- Only the new activities, the rewired flows and the objects moved to make room change;\n" + + "-- every other element keeps its $ID, position and curve.\n" + + "-- Refused: insert after a decision, insert before an activity several flows enter,\n" + + "-- drop/replace of a decision or of an activity with an error handler, anything inside\n" + + "-- a loop body, a fragment that returns, and a fragment variable that clashes with one\n" + + "-- the flow has or reads one not declared on the path. Over --mcp only insert is supported.", + Example: "alter microflow FeedbackModule.VAL_Feedback {\n" + + " insert after $IsValidEmail { log info node 'Feedback' 'Email checked'; }\n" + + " replace set $ValidFeedback = false @3 with {\n" + + " set $ValidFeedback = false;\n" + + " log warning node 'Feedback' 'Email rejected';\n" + + " }\n" + + " drop log debug node 'Feedback' *;\n" + + "};", + SeeAlso: []string{"microflow"}, + }) + Register(SyntaxFeature{ Path: "microflow.show-page", Summary: "Open and close pages from microflows", diff --git a/cmd/mxcli/syntax/features_misc.go b/cmd/mxcli/syntax/features_misc.go index 3bb1e41eb6..2334c55fa4 100644 --- a/cmd/mxcli/syntax/features_misc.go +++ b/cmd/mxcli/syntax/features_misc.go @@ -12,7 +12,7 @@ func init() { Summary: "mdl <n>; — the MDL language version a script is written in", Keywords: []string{ "mdl 1", "mdl 0", "language version", "header", "edition", - "preview", "beta", "version-gated", "meaning", + "preview", "beta", "version-gated", "meaning", "fmt --upgrade", "upgrade", }, Syntax: "mdl <n>;\n\n" + "-- Optional, and only as the FIRST statement of a script. It declares the\n" + @@ -25,9 +25,28 @@ func init() { "-- describe and fmt do not emit it.\n" + "-- mdl 2; refused: this mxcli does not know that version.\n" + "--\n" + + "-- Under mdl 1 parsing is strict (each is a warning without the header):\n" + + "-- every statement ends with ';' and '/' is not a terminator; '' is the\n" + + "-- only string escape, so a backslash is an ordinary character; and an\n" + + "-- unknown or mis-shaped property key in a REST, business event or agent\n" + + "-- property list is an error. A trailing comma is allowed in every\n" + + "-- bracketed list, with or without the header.\n" + + "--\n" + "-- A script's meaning never depends on which mxcli release runs it: a\n" + "-- change of meaning applies only under the version that introduces it.\n" + - "-- The header is independent of the Mendix version the project targets.", + "-- The header is independent of the Mendix version the project targets.\n" + + "--\n" + + "-- `mxcli fmt --upgrade --header` adds it, after rewriting every construct\n" + + "-- whose meaning it would change: it adds each missing `;`, deletes `/`\n" + + "-- lines, writes backslash escapes as the characters they stood for,\n" + + "-- turns `retrieve … limit 1` into `first`, adds `set` to reassignments,\n" + + "-- and writes list operations one statement per activity. It refuses, and\n" + + "-- says why, when a construct has no rewrite: an unknown or mis-shaped\n" + + "-- property (MDL-V1-PROP/PROPVALUE), `create or replace view entity`\n" + + "-- (MDL-V1-REPLACE01), a nested list operation, find/contains on a variable\n" + + "-- whose type the script does not state, and an escaped line break inside\n" + + "-- an expression.\n" + + "-- `mxcli fmt --upgrade` alone rewrites deprecated spellings (MDL-DEPRnnn).", Example: "mdl 1;\n\ncreate persistent entity MyModule.Customer (\n Name: String(200)\n);", SeeAlso: []string{"create-modifiers"}, }) diff --git a/docs-site/src/appendixes/quick-reference.md b/docs-site/src/appendixes/quick-reference.md index 564e37586a..5747f21e2c 100644 --- a/docs-site/src/appendixes/quick-reference.md +++ b/docs-site/src/appendixes/quick-reference.md @@ -180,7 +180,7 @@ AUTHENTICATION Basic, Session | `TRY ... CATCH ... END TRY` | `ON ERROR { ... }` blocks | Use error handlers on specific activities | **Notes:** -- `RETRIEVE ... LIMIT n` IS supported. `LIMIT 1` returns a single entity, otherwise returns a list. +- `RETRIEVE ... FIRST` returns a single entity; `RETRIEVE ... LIMIT n [OFFSET n]` returns a list. A bare `LIMIT 1` is a list of one under `mdl 1;`, and without the header the single entity, with warning `MDL-V1-LIMIT1`. - `ROLLBACK $Entity [REFRESH];` IS supported. Rolls back uncommitted changes to an object. ## Project Organization diff --git a/docs-site/src/language/activity-types.md b/docs-site/src/language/activity-types.md index f8ec49841c..9ea237b1d7 100644 --- a/docs-site/src/language/activity-types.md +++ b/docs-site/src/language/activity-types.md @@ -78,7 +78,7 @@ Retrieves objects from the database using an optional XPath-style WHERE clause: -- Retrieve a single object RETRIEVE $Customer FROM Sales.Customer WHERE Email = $InputEmail - LIMIT 1; + FIRST; -- Retrieve a list of objects RETRIEVE $ActiveOrders FROM Sales.Order @@ -94,7 +94,7 @@ RETRIEVE $RecentOrders FROM Sales.Order LIMIT 50; ``` -When `LIMIT 1` is specified, the result is a single entity object. Otherwise, the result is a list. +With `FIRST` the result is a single entity object (Mendix's "First object" range). Otherwise the result is a list, including with `LIMIT`/`OFFSET`. A bare `LIMIT 1` depends on the language version: a list of one under `mdl 1;`, and without the header the object, with warning `MDL-V1-LIMIT1`. `describe` prints the object range as `FIRST`. ### RETRIEVE by Association diff --git a/docs-site/src/language/basics.md b/docs-site/src/language/basics.md index cd09b83366..b33d43e2ea 100644 --- a/docs-site/src/language/basics.md +++ b/docs-site/src/language/basics.md @@ -18,22 +18,23 @@ INDEX (Name); ## Statement Termination -Statements are terminated with a semicolon (`;`) or a forward slash (`/`) on its own line (Oracle-style, useful for multi-line statements): +Every statement ends with a semicolon (`;`): ```sql --- Semicolon terminator CREATE MODULE OrderManagement; --- Forward-slash terminator (useful for long statements) CREATE PERSISTENT ENTITY Sales.Order ( OrderId: AutoNumber NOT NULL UNIQUE, - OrderDate: DateTime NOT NULL + OrderDate: DateTime NOT NULL, ) INDEX (OrderDate DESC); -/ ``` -Simple commands such as `HELP`, `EXIT`, `STATUS`, `SHOW`, and `DESCRIBE` do not require a terminator. +Under `mdl 1;` a missing `;` is an error, and so is the Oracle SQL*Plus-style `/` on its own line. A script without the header still accepts both, and `check` warns `MDL-V1-SEMI` / `MDL-V1-SLASH` for each one. At the REPL, where there is no header, a single command such as `SHOW ENTITIES` still needs no terminator. + +## Trailing Commas + +A trailing comma is allowed in every bracketed list — attributes, enumeration values, parameters, property lists, `{ … }` blocks — under every language version. `()` is the only way to write an empty list: `(,)` and `(a,,)` are errors. ## Language Version Header @@ -52,6 +53,39 @@ create persistent entity Sales.Customer ( - The header must be the **first** statement. A version this mxcli does not know is refused. - It is independent of the Mendix version your project targets. +What `mdl 1` makes strict (each is a warning without the header, with the code shown): + +| Under `mdl 1;` | Without the header | +|---|---| +| A statement without `;` is an error. | Accepted; `MDL-V1-SEMI`. | +| A `/` terminator line is an error. | Accepted; `MDL-V1-SLASH`. | +| `''` is the only string escape; a backslash is an ordinary character, so `'C:\temp'` is that path. | `\n`, `\r`, `\t`, `\\` and `\'` are escapes; `MDL-V1-ESCAPE` for each literal whose value would change. | +| In a REST client, published REST service, business event service, model, knowledge base, consumed MCP service or agent, an unknown property key is an error that names the key it most likely meant, and so is a value its key does not take (`Response: json from $X`). | The property is ignored, or read by its shape as before; `MDL-V1-PROP` / `MDL-V1-PROPVALUE`. | + +### Upgrading a script: `mxcli fmt --upgrade` + +`mxcli fmt --upgrade` rewrites every deprecated spelling (the `MDL-DEPRnnn` warnings) to its canonical form — `create or replace` becomes `create or modify`, `show entities` becomes `list entities` — and changes nothing else: comments, layout and keyword case are kept. A deprecated use with no mechanical rewrite is reported and left in place. + +```bash +mxcli fmt --upgrade script.mdl # print the upgraded script +mxcli fmt --upgrade -w script.mdl # upgrade in place +mxcli fmt --upgrade --header -w script.mdl # also add `mdl 1;` +``` + +`--header` adds `mdl 1;` after rewriting every construct whose meaning the header would change, so the script keeps doing what it did: + +| Code | Rewrite | +|---|---| +| `MDL-V1-SEMI` | adds the missing `;` | +| `MDL-V1-SLASH` | deletes the `/` line | +| `MDL-V1-ESCAPE` | writes the string's value with `''` as the only escape (`'it\'s\t'` becomes `'it''s` + a tab + `'`) | +| `MDL-V1-LIMIT1` | `retrieve … limit 1` (one object) becomes `retrieve … first` | +| `MDL-V1-SET` | `$x = …` becomes `set $x = …` | +| `MDL-V1-LIST`, `MDL-DEPR003`, `MDL-DEPR004` | a list operation or aggregate call becomes its statement form (`$x = filter($L, …)` → `$x = filter $L where …`); `find`/`contains` on a declared String keeps the call and gains `set` | +| `MDL-V1-REPLACE02` | `create or replace user role` / `demo user` becomes a plain `create` | + +A construct with no mechanical rewrite is reported with the reason, and `fmt` refuses to add the header rather than change the script's meaning: an unknown or mis-shaped property (`MDL-V1-PROP`, `MDL-V1-PROPVALUE`), `create or replace view entity` (`MDL-V1-REPLACE01`), a nested list operation such as `count(filter(…))`, `find`/`contains` on a variable whose type the script does not state, and an escaped line break (`\n`) inside an expression. While `mdl 1` is a preview, the header is added only when asked. Running `fmt --upgrade` on its own output changes nothing. + The design is in [ADR-0011](https://github.com/mendixlabs/mxcli/blob/main/docs/13-decisions/0011-mdl-language-versioning.md); `mxcli syntax language-header` has the details. ## Case Insensitivity diff --git a/docs-site/src/language/lexical-structure.md b/docs-site/src/language/lexical-structure.md index 6b9e346040..0034bb0c79 100644 --- a/docs-site/src/language/lexical-structure.md +++ b/docs-site/src/language/lexical-structure.md @@ -58,7 +58,9 @@ String literals use single quotes: 'it''s here' -- doubled single quote to escape ``` -The only escape sequence is `''` (two single quotes) to represent a literal single quote. Backslash escaping is **not** supported. +The only escape sequence is `''` (two single quotes) to represent a literal single quote, as in a Mendix expression. Under `mdl 1;` a backslash is an ordinary character: `'C:\temp'` is that path. + +A script without the header (`mdl 0`) still reads a backslash before `n`, `r`, `t`, `\` or `'` as an escape — `'C:\temp'` holds a tab — and `check` warns `MDL-V1-ESCAPE` for each literal whose value changes under `mdl 1`. Write the backslash-free form (`'it''s'`) to mean the same under both. ### Numeric Literals diff --git a/docs-site/src/reference/microflow/create-microflow.md b/docs-site/src/reference/microflow/create-microflow.md index 041689bda3..24d4e4be43 100644 --- a/docs-site/src/reference/microflow/create-microflow.md +++ b/docs-site/src/reference/microflow/create-microflow.md @@ -65,13 +65,13 @@ Every one of these modifiers is optional, and leaving one out always means Mendi ```sql -- Database retrieve with optional XPath constraint -RETRIEVE $Var FROM Module.Entity [ WHERE condition ] [ LIMIT n ]; +RETRIEVE $Var FROM Module.Entity [ WHERE condition ] [ FIRST | [ LIMIT n ] [ OFFSET n ] ]; -- Retrieve by association RETRIEVE $List FROM $Parent/Module.AssocName; ``` -`RETRIEVE ... LIMIT 1` returns a single entity. Without `LIMIT` or with `LIMIT` greater than 1, it returns a list. Retrieve by association traverses an association from a known object. +`RETRIEVE ... FIRST` returns a single entity. Without a range, or with `LIMIT`/`OFFSET`, it returns a list. A bare `LIMIT 1` is a list of one under `mdl 1;`; without the header it keeps its old meaning, a single entity, and warns `MDL-V1-LIMIT1`. Retrieve by association traverses an association from a known object. **Calls** diff --git a/docs/01-project/MDL_QUICK_REFERENCE.md b/docs/01-project/MDL_QUICK_REFERENCE.md index 1abf197309..fdf93b2abd 100644 --- a/docs/01-project/MDL_QUICK_REFERENCE.md +++ b/docs/01-project/MDL_QUICK_REFERENCE.md @@ -483,6 +483,8 @@ rather than updating the first. | Describe microflow | `describe microflow Module.Name;` | Full MDL with activities | | Describe microflow (normalized) | `describe microflow Module.Name normalized;` | Folds crossed branches into one condition instead of flattening them. Opt-in: the output re-executes to an equivalent graph with fewer nodes and a different layout | | Describe microflow (with handles) | `describe microflow Module.Name with handles;` | Prints `-- handle: <target>` above each activity: its content address for `alter microflow` — output `$Var`, `'Caption'`, or a statement pattern with `*` wildcards (anchored at both ends), plus `@n` when several match. Comments only; cannot be combined with `normalized` | +| Insert into a stored microflow | `alter microflow Module.Name { insert after <target> { <statements> } };` | Also `insert before`, and `alter nanoflow`. A graph splice into the stored flow, not a rebuild: only the new activities, the two rewired flows and the objects moved to make room change; every other element keeps its `$ID`, position and curve. `<target>` is a handle from `describe … with handles`, resolved before any operation runs. Refused: after a decision, before an activity several flows enter, inside a loop body, a fragment that returns, a variable the flow already has or one not declared on the path | +| Replace or drop an activity | `alter microflow Module.Name { replace <target> with { <statements> } drop <target>; };` | Flows into the activity are re-pointed at the replacement (or at its successor, for `drop`). Refused for a decision, an end event, an activity with an error handler, and an activity whose output variable is still read. Over `--mcp` only `insert` is supported | | Describe nanoflow | `describe nanoflow Module.Name;` | Full MDL with activities | | Rename microflow | `rename microflow Module.Old to New;` | Updates all references | | Rename nanoflow | `rename nanoflow Module.Old to New;` | Updates all references | @@ -524,7 +526,7 @@ it is for pages. | Commit | `commit $entity [without events] [refresh];` | **Omitted = with events**, matching Studio Pro's default. `without events` is the deviation and the only form that changes the stored value; `with events` still parses and means the default | | Delete | `delete $entity [refresh];` | | | Rollback | `rollback $entity [refresh];` | Reverts uncommitted changes | -| Retrieve (DB) | `retrieve $Var from Module.Entity [where condition] [sort by Attr asc\|desc, ...] [limit n [offset n]];` | Database XPath retrieve. `limit 1` with no `offset` binds a single **object**, not a one-element list (MDL-RETRIEVE01) | +| Retrieve (DB) | `retrieve $Var from Module.Entity [where condition] [sort by Attr asc\|desc, ...] [first \| [limit n] [offset n]];` | Database XPath retrieve. `first` binds a single **object** (Mendix's "First object" range); `limit`/`offset` bind a list. `limit 1` without `offset` is a list of one under `mdl 1;`, and without the header keeps its old meaning, the object, with warning MDL-V1-LIMIT1 | | Retrieve (DB), sorted | `sort by Attr asc` / `sort by Module.Other.Attr asc` / `sort by Module.Assoc/Module.Other.Attr asc` | A bare name is qualified with the entity **declaring** it, which may be an ancestor. A sort may also navigate associations — one `/` per hop, the last segment is the attribute — and mxcli stores the hops as the `EntityRef` Mendix needs; without them the build is **CE7247**. **Name the hop when more than one association reaches the same entity**: a bare `Module.Other.Attr` is resolved by inference, which walks the generalization chain across modules (`Administration.Account` reaches `System.Language.Code` through `System.User_Language`) but cannot tell `Order_ShipTo` from `Order_BillTo` — measured, a sort on the billing address round-tripped into one on the shipping address at 0 errors both sides (mendixlabs/mxcli#1152). The same spelling works in a page datasource's `sort by` | | Retrieve (Assoc) | `retrieve $list from $Parent/Module.AssocName;` | Retrieve by association | | Add to list | `add expression to $list;` | Also accepts existing `add $item to $list;` form | @@ -617,12 +619,13 @@ and `mxbuild` were all clean. Only the running app showed it. | `TRY ... CATCH ... end TRY` | `on error { ... }` blocks | Use error handlers on specific activities | **Notes:** -- `retrieve ... limit n` IS supported. **`limit 1` with no `offset` binds a single OBJECT**, not a - one-element list: it is Mendix's "First object" range. Every other `limit` (including - `limit 1 offset n`) is a bounded range, which is a list. Using a `limit 1` variable as a list — - `head()`, `count()`, a `loop` — is **CE0097** at build time and **MDL-RETRIEVE01** at check time. - Note this is the opposite of the import-mapping clause above, where `first` binds an object and - `limit 1` a one-element list. +- `retrieve ... first` binds a single OBJECT: Mendix's "First object" range. `retrieve ... limit n + [offset n]` is a bounded range, which is a list — the same split as the import-mapping clause above. + **`limit 1` without `offset` depends on the language version** (ako/mxcli#734): under `mdl 1;` + it is a list of one; without the header it keeps its old meaning, the object, and warns + `MDL-V1-LIMIT1`. Write `first` for an object. Using an object as a list — `head()`, `count()` + (**CE0097**) or a `loop` (**CE0100**) — is **MDL-RETRIEVE01** at check time. `describe` prints + the object range as `first`. - `rollback $entity [refresh];` IS supported. Rolls back uncommitted changes to an object. ## Project Organization diff --git a/docs/05-mdl-specification/01-language-reference.md b/docs/05-mdl-specification/01-language-reference.md index fc788f49a3..246db16717 100644 --- a/docs/05-mdl-specification/01-language-reference.md +++ b/docs/05-mdl-specification/01-language-reference.md @@ -988,9 +988,12 @@ Creates a new module role within a module. **Syntax:** ```sql -create module role <module>.<role> [description '<text>'] +create [or modify] module role <module>.<role> [description '<text>'] ``` +`or modify` updates the description of a role that already exists instead of +failing. + **Example:** ```sql create module role Shop.Admin description 'Full administrative access'; @@ -1003,9 +1006,11 @@ Removes a module role. **Syntax:** ```sql -drop module role <module>.<role> +drop module role [if exists] <module>.<role> ``` +`if exists` makes dropping a role that is not there a no-op. + ### GRANT EXECUTE ON MICROFLOW Grants execute access on a microflow to one or more module roles. diff --git a/mdl-examples/bug-tests/1103-retrieve-limit-one-is-an-object.fail.mdl b/mdl-examples/bug-tests/1103-retrieve-limit-one-is-an-object.fail.mdl index 467ffe1c16..e0484e5059 100644 --- a/mdl-examples/bug-tests/1103-retrieve-limit-one-is-an-object.fail.mdl +++ b/mdl-examples/bug-tests/1103-retrieve-limit-one-is-an-object.fail.mdl @@ -1,11 +1,11 @@ -- mendixlabs/mxcli#1103 — `retrieve … limit 1` binds a single OBJECT. -- --- The executor maps `limit 1` with no offset to Mendix's "First object" range, --- so $Requests is an object and head() cannot take it. Nothing said so before --- the build: `mxcli check --references` passed and `describe` re-emits --- `limit 1`, so an object retrieve and a list retrieve are identical text. --- mxbuild rejected it as CE0097 at the far end of a build — and inside a --- .test.mdl file, not even that: the injected test simply failed to build. +-- This script has no `mdl 1;` header, so `limit 1` with no offset keeps its +-- alpha meaning, Mendix's "First object" range (and warns MDL-V1-LIMIT1, +-- ako/mxcli#734): $Requests is an object and head() cannot take it. mxbuild +-- rejects that as CE0097 — and inside a .test.mdl file, not even that: the +-- injected test simply failed to build. `describe` now prints the object +-- range as `first`; under `mdl 1;` `limit 1` is a list of one. -- -- MDL-RETRIEVE01 reports it at check time. This script must FAIL check. create module Probe; diff --git a/mdl/ast/ast.go b/mdl/ast/ast.go index 223a8ef2d1..6def502fef 100644 --- a/mdl/ast/ast.go +++ b/mdl/ast/ast.go @@ -87,6 +87,11 @@ type DeprecatedSpelling struct { // Subject says what the spelling was used on, in MDL's own words ("entity", // "microflow", …); empty when there is nothing more specific to say. Subject string + // Fix is the structural rewrite of this use to the canonical form, for an + // entry whose rewrite is not a keyword swap (deprecation.Rewrite.Structural). + // Nil when this use has none; NoFix then says why. + Fix *Fix + NoFix string } // LanguageNote is one construct whose meaning depends on the language version, @@ -95,6 +100,28 @@ type LanguageNote struct { Line int // 1-based source line of the construct Code string // the langver.Change's rule ID Message string + // Fix rewrites the construct so that under the new version it still means + // what it means here — what `fmt --upgrade --header` applies before adding + // the header. Nil when there is no mechanical rewrite; NoFix then says why. + // A Fix with no edits is a construct already spelled so that it keeps its + // meaning under the new version. + Fix *Fix + NoFix string +} + +// Fix is a mechanical source rewrite: edits in the coordinates the parser read +// the script in, computed from the parse tree by the visitor that recorded the +// construct. Edits may not overlap. +type Fix struct { + Edits []TextEdit +} + +// TextEdit replaces the runes [Start, Stop) of the script with Text. Offsets +// count runes (code points) from the start of the script, as ANTLR's character +// stream does; Start == Stop is an insertion. +type TextEdit struct { + Start, Stop int + Text string } // DocumentAnnotation is one annotation written before a CREATE statement. diff --git a/mdl/ast/ast_alter_flow.go b/mdl/ast/ast_alter_flow.go new file mode 100644 index 0000000000..b6534dbd44 --- /dev/null +++ b/mdl/ast/ast_alter_flow.go @@ -0,0 +1,54 @@ +// SPDX-License-Identifier: Apache-2.0 + +package ast + +// ============================================================================ +// ALTER MICROFLOW / ALTER NANOFLOW — a graph splice into the stored flow +// ============================================================================ + +// AlterFlowStmt represents: +// +// alter microflow|nanoflow Module.Name { +// insert after|before <target> { <statements> } +// replace <target> with { <statements> } +// drop <target>; +// } +// +// (ADR-0012 decision 3). Targets are content addresses, resolved against the +// flow as stored before any operation applies. +type AlterFlowStmt struct { + Nanoflow bool + Name QualifiedName + Operations []*AlterFlowOperation +} + +func (s *AlterFlowStmt) isStatement() {} + +// Kind is "microflow" or "nanoflow". +func (s *AlterFlowStmt) Kind() string { + if s.Nanoflow { + return "nanoflow" + } + return "microflow" +} + +// AlterFlowOpKind names an operation of an AlterFlowStmt. +type AlterFlowOpKind string + +const ( + AlterFlowInsertAfter AlterFlowOpKind = "insert after" + AlterFlowInsertBefore AlterFlowOpKind = "insert before" + AlterFlowReplace AlterFlowOpKind = "replace" + AlterFlowDrop AlterFlowOpKind = "drop" +) + +// AlterFlowOperation is one operation of an AlterFlowStmt. +type AlterFlowOperation struct { + Op AlterFlowOpKind + // Target is the content address as written (`$IsValidEmail`, + // `'Email is Valid?'`, `log * node 'Debug' *`, with an optional `@n`). + // mfmutator.ParseTarget reads it. + Target string + // Body is the fragment, for insert and replace. + Body []MicroflowStatement +} diff --git a/mdl/ast/ast_microflow.go b/mdl/ast/ast_microflow.go index 277f7c5f8d..0f1207460a 100644 --- a/mdl/ast/ast_microflow.go +++ b/mdl/ast/ast_microflow.go @@ -551,16 +551,21 @@ type RollbackStmt struct { func (s *RollbackStmt) isMicroflowStatement() {} -// RetrieveStmt represents: RETRIEVE $Var FROM Entity [WHERE condition] [SORT BY ...] [LIMIT n] [OFFSET n] [ON ERROR ...] +// RetrieveStmt represents: RETRIEVE $Var FROM Entity [WHERE condition] [SORT BY ...] [FIRST | [LIMIT n] [OFFSET n]] [ON ERROR ...] // or: RETRIEVE $Var FROM $Parent/Module.Association (association retrieve) type RetrieveStmt struct { - Variable string // Output variable - Source QualifiedName // Entity (database) or Association (association retrieve) - StartVariable string // Non-empty for association retrieve: the starting variable name - Where Expression // Optional WHERE condition - SortColumns []SortColumnDef // Optional SORT BY columns - Limit string // Optional LIMIT expression (empty = no limit) - Offset string // Optional OFFSET expression (empty = no offset) + Variable string // Output variable + Source QualifiedName // Entity (database) or Association (association retrieve) + StartVariable string // Non-empty for association retrieve: the starting variable name + Where Expression // Optional WHERE condition + SortColumns []SortColumnDef // Optional SORT BY columns + // First is Mendix's "First object" range: the output is ONE object, not a + // list. It is set by `first`, and by `limit 1` without offset in an mdl 0 + // script (the alpha meaning, ako/mxcli#734); Limit and Offset are then empty. + // Everything downstream reads this, never the limit text. + First bool + Limit string // Optional LIMIT expression of the Custom range (empty = no limit) + Offset string // Optional OFFSET expression of the Custom range (empty = no offset) ErrorHandling *ErrorHandlingClause // Optional ON ERROR clause Annotations *ActivityAnnotations // Optional @position, @caption, @color, @annotation } @@ -851,21 +856,30 @@ type SortSpec struct { Ascending bool // True for ASC, false for DESC } -// ListOperationStmt represents list operations like HEAD, TAIL, FIND, etc. -// $Var = HEAD($List) -// $Var = FIND($List, condition) -// $Var = SORT($List, attr ASC) -// $Var = UNION($List1, $List2) +// ListOperationStmt is one Studio Pro List operation activity: +// $Var = head $List +// $Var = find $List by Member = value | where condition +// $Var = sort $List by attr asc +// $Var = union $List1 with $List2 +// The call forms ($Var = HEAD($List), …) build the same node. type ListOperationStmt struct { - OutputVariable string // Output variable name - Operation ListOperationType // Operation type - InputVariable string // Input list variable (first operand) - SecondVariable string // Second operand for UNION, INTERSECT, SUBTRACT, CONTAINS, EQUALS - Condition Expression // Condition for FIND/FILTER - SortSpecs []SortSpec // Sort specifications for SORT - OffsetExpr Expression // Offset expression for RANGE - LimitExpr Expression // Limit expression for RANGE - Annotations *ActivityAnnotations // Optional @position, @caption, @color, @annotation + OutputVariable string // Output variable name + Operation ListOperationType // Operation type + InputVariable string // Input list variable (first operand) + SecondVariable string // Second operand for UNION, INTERSECT, SUBTRACT, CONTAINS, EQUALS + Condition Expression // Condition for FIND/FILTER + // ByExpression is set for FIND/FILTER whose condition is an expression over + // $currentObject (Studio Pro's "Find by expression" / "Filter by + // expression"). When false, a condition of the shape `Member = value` (see + // IsMemberEquality) is the "by member" operation, and any other condition + // falls back to the expression operation, which is what the function form + // has always done. The statement form sets it from the linking word: `where` + // sets it, `by` does not. + ByExpression bool + SortSpecs []SortSpec // Sort specifications for SORT + OffsetExpr Expression // Offset expression for RANGE + LimitExpr Expression // Limit expression for RANGE + Annotations *ActivityAnnotations // Optional @position, @caption, @color, @annotation // ErrorHandling is recorded only so the clause can be REFUSED. Mendix's // ListOperationsAction has no ErrorHandlingType, so an ON ERROR here has // nowhere to go; parsing it and reporting it beats dropping it silently. @@ -877,6 +891,24 @@ type ListOperationStmt struct { func (s *ListOperationStmt) isMicroflowStatement() {} +// IsMemberEquality reports whether a FIND/FILTER condition names a member of +// the list's entity and a value, `Member = value` — the shape Studio Pro's Find +// and Filter operations take (an attribute or association, and the value it must +// have). It is the single test both the visitor (which linking word a function +// form is a respelling of) and the flow builder (which operation to write) use, +// so the two cannot disagree. +func IsMemberEquality(cond Expression) bool { + binary, ok := cond.(*BinaryExpr) + if !ok || binary.Operator != "=" { + return false + } + switch binary.Left.(type) { + case *IdentifierExpr, *QualifiedNameExpr: + return true + } + return false +} + // UnresolvedOperand is a list operand that the visitor could not reduce to a // variable name. // @@ -932,10 +964,11 @@ func (t AggregateListOperationType) String() string { } } -// AggregateListStmt represents aggregate operations: COUNT, SUM, AVERAGE, etc. -// $Count = COUNT($List) -// $Sum = SUM($List/Attr) -// $Sum = SUM($List, $currentObject/Price * 2) // expression form +// AggregateListStmt is one Studio Pro Aggregate list activity: +// $Count = count $List +// $Sum = sum $List by Attr +// $Sum = sum $List of $currentObject/Price * 2 // expression form +// The call forms ($Count = COUNT($List), …) build the same node. type AggregateListStmt struct { OutputVariable string // Output variable name Operation AggregateListOperationType // Operation type diff --git a/mdl/ast/ast_security.go b/mdl/ast/ast_security.go index 3aa17531ae..4f152b76af 100644 --- a/mdl/ast/ast_security.go +++ b/mdl/ast/ast_security.go @@ -18,9 +18,11 @@ type CreateModuleRoleStmt struct { func (s *CreateModuleRoleStmt) isStatement() {} -// DropModuleRoleStmt represents: DROP MODULE ROLE Module.RoleName +// DropModuleRoleStmt represents: DROP MODULE ROLE [IF EXISTS] Module.RoleName type DropModuleRoleStmt struct { - Name QualifiedName + // IfExists downgrades "not found" to a no-op; see DropUserRoleStmt. + IfExists bool + Name QualifiedName } func (s *DropModuleRoleStmt) isStatement() {} diff --git a/mdl/backend/backend.go b/mdl/backend/backend.go index a57e80e3a1..f9a0cd760d 100644 --- a/mdl/backend/backend.go +++ b/mdl/backend/backend.go @@ -37,5 +37,6 @@ type FullBackend interface { AgentEditorBackend PageMutationBackend WorkflowMutationBackend + MicroflowMutationBackend WidgetBuilderBackend } diff --git a/mdl/backend/mcp/microflow_mutator.go b/mdl/backend/mcp/microflow_mutator.go new file mode 100644 index 0000000000..4bbe61ad18 --- /dev/null +++ b/mdl/backend/mcp/microflow_mutator.go @@ -0,0 +1,499 @@ +// SPDX-License-Identifier: Apache-2.0 + +package mcp + +import ( + "encoding/json" + "fmt" + "strconv" + "strings" + + "go.mongodb.org/mongo-driver/bson" + "go.mongodb.org/mongo-driver/bson/primitive" + + "github.com/mendixlabs/mxcli/mdl/backend" + "github.com/mendixlabs/mxcli/mdl/backend/mfmutator" + "github.com/mendixlabs/mxcli/mdl/types" + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/sdk/microflows" +) + +// ALTER MICROFLOW / NANOFLOW over MCP (plan item 4.2f). +// +// The splice itself is the shared engine (mfmutator), run on the stored form of +// the flow as the local .mpr holds it — the same decisions, placement and +// refusals as on the modelsdk backend. What differs is the write: Studio Pro +// takes no unit bytes, so Save diffs the spliced document against the stored +// one and sends the difference as ped_update_document path operations on the +// live document: positions set, new objects and flows added, rewired flow ends +// set, removed elements removed. PED addresses list entries by index, and those +// indexes are the stored order — so before anything is sent, the live document +// is read back and compared with the local one, and a mismatch (the .mpr is +// behind Studio Pro) refuses the write rather than patching the wrong element. + +// OpenMicroflowForMutation opens a microflow or nanoflow for splicing. +func (b *Backend) OpenMicroflowForMutation(unitID model.ID) (backend.MicroflowMutator, error) { + raw, err := b.reader.GetRawUnitBytes(unitID) + if err != nil { + return nil, fmt.Errorf("alter over MCP reads the stored flow from the local project: %w", err) + } + var d bson.D + if err := bson.Unmarshal(raw, &d); err != nil { + return nil, err + } + docType, _ := docValue(d, "$Type").(string) + name, _ := docValue(d, "Name").(string) + qn, err := b.flowQualifiedName(unitID, docType, name) + if err != nil { + return nil, err + } + deps := &mcpFlowDeps{b: b, docType: docType, qn: qn, stored: d, objects: map[string]microflows.MicroflowObject{}, + flows: map[string]*microflows.SequenceFlow{}, annotations: map[string]*microflows.AnnotationFlow{}} + var work bson.D + if err := bson.Unmarshal(raw, &work); err != nil { + return nil, err + } + m, err := mfmutator.New(work, unitID, deps) + if err != nil { + return nil, err + } + return &mcpFlowMutator{Mutator: m, qn: qn}, nil +} + +// mcpFlowMutator is the shared splice, limited to what PED applies safely: +// inserts. A PED update that fails is rolled back EXCEPT for its removals, +// which persist (PED's own warning, and measured: a failed update left the +// removed activity gone together with the flows attached to it). A drop or a +// replace needs a removal, so over MCP they are refused rather than risked on +// the live model; run them against the .mpr instead. +type mcpFlowMutator struct { + *mfmutator.Mutator + qn string +} + +func (m *mcpFlowMutator) Replace(model.ID, *backend.MicroflowFragment) error { + return fmt.Errorf("replace is not supported by the MCP backend yet: Studio Pro does not roll back a removal when an update fails; run without --mcp to alter %s in the .mpr", m.qn) +} + +func (m *mcpFlowMutator) Drop(model.ID) error { + return fmt.Errorf("drop is not supported by the MCP backend yet: Studio Pro does not roll back a removal when an update fails; run without --mcp to alter %s in the .mpr", m.qn) +} + +func (b *Backend) flowQualifiedName(id model.ID, docType, name string) (string, error) { + var container model.ID + switch docType { + case microflowDocType: + mf, err := b.GetMicroflow(id) + if err != nil { + return "", err + } + container = mf.ContainerID + case "Microflows$Nanoflow": + nfs, err := b.reader.ListNanoflows() + if err != nil { + return "", err + } + for _, nf := range nfs { + if nf.ID == id { + container = nf.ContainerID + } + } + default: + return "", fmt.Errorf("unit %s is a %s, not a microflow or nanoflow", id, docType) + } + mod, err := b.moduleNameForContainer(container) + if err != nil { + return "", err + } + return mod + "." + name, nil +} + +// mcpFlowDeps is mfmutator.Deps for the MCP backend. Serialize produces only +// what the splice reads (identity, type, geometry, pointers) and keeps the +// domain object, which is what PED is sent; SaveUnit turns the spliced +// document into PED operations. +type mcpFlowDeps struct { + b *Backend + docType string + qn string + stored bson.D + + objects map[string]microflows.MicroflowObject + flows map[string]*microflows.SequenceFlow + annotations map[string]*microflows.AnnotationFlow +} + +func binaryOf(id model.ID) primitive.Binary { + return primitive.Binary{Subtype: 0, Data: types.UUIDToBlob(string(id))} +} + +func (d *mcpFlowDeps) SerializeObject(obj microflows.MicroflowObject) (bson.D, error) { + d.objects[string(obj.GetID())] = obj + p := obj.GetPosition() + w, h := 0, 0 + if s, ok := obj.(interface{ GetSize() model.Size }); ok { + w, h = s.GetSize().Width, s.GetSize().Height + } + return bson.D{ + {Key: "$ID", Value: binaryOf(obj.GetID())}, + {Key: "$Type", Value: "mcp-new-object"}, + {Key: "RelativeMiddlePoint", Value: fmt.Sprintf("%d;%d", p.X, p.Y)}, + {Key: "Size", Value: fmt.Sprintf("%d;%d", w, h)}, + }, nil +} + +func (d *mcpFlowDeps) SerializeSequenceFlow(f *microflows.SequenceFlow) (bson.D, error) { + d.flows[string(f.ID)] = f + return bson.D{ + {Key: "$ID", Value: binaryOf(f.ID)}, + {Key: "$Type", Value: "Microflows$SequenceFlow"}, + {Key: "DestinationConnectionIndex", Value: int32(f.DestinationConnectionIndex)}, + {Key: "DestinationPointer", Value: binaryOf(f.DestinationID)}, + {Key: "IsErrorHandler", Value: f.IsErrorHandler}, + {Key: "Line", Value: bson.D{ + {Key: "$Type", Value: "Microflows$BezierCurve"}, + {Key: "DestinationControlVector", Value: f.DestinationControlVector}, + {Key: "OriginControlVector", Value: f.OriginControlVector}, + }}, + {Key: "OriginConnectionIndex", Value: int32(f.OriginConnectionIndex)}, + {Key: "OriginPointer", Value: binaryOf(f.OriginID)}, + }, nil +} + +func (d *mcpFlowDeps) SerializeAnnotationFlow(f *microflows.AnnotationFlow) (bson.D, error) { + d.annotations[string(f.ID)] = f + return bson.D{ + {Key: "$ID", Value: binaryOf(f.ID)}, + {Key: "$Type", Value: "Microflows$AnnotationFlow"}, + {Key: "DestinationPointer", Value: binaryOf(f.DestinationID)}, + {Key: "OriginPointer", Value: binaryOf(f.OriginID)}, + }, nil +} + +// SaveUnit sends the difference between the stored and the spliced document. +func (d *mcpFlowDeps) SaveUnit(_ string, contents []byte) error { + var spliced bson.D + if err := bson.Unmarshal(contents, &spliced); err != nil { + return err + } + ops, err := d.b.flowPatchOps(d, spliced) + if err != nil { + return err + } + if len(ops) == 0 { + return nil + } + if err := d.b.checkLiveFlowMatches(d.docType, d.qn, d.stored); err != nil { + return err + } + // Only inserts reach here (mcpFlowMutator refuses the rest), so there is + // nothing to remove; a removal would mean the splice did something this + // path was not built for, and PED would not roll it back on a failure. + for _, op := range ops { + if op.Operation.Type == "remove" { + return fmt.Errorf("%s: refusing to send a removal over MCP", d.qn) + } + } + // One update: PED applies it all or, on a failure, nothing. + if err := d.b.pedUpdateDoc(d.docType, d.qn, ops...); err != nil { + return err + } + return d.b.pedCheckDocument(d.docType, d.qn) +} + +// flowList returns a unit's top-level objects or flows, in stored order, with +// their $IDs. +func flowList(d bson.D, objects bool) (ids []string, docs []bson.D) { + list := docValue(d, "Flows") + if objects { + oc, _ := docValue(d, "ObjectCollection").(bson.D) + list = docValue(oc, "Objects") + } + a, _ := list.(bson.A) + for i, el := range a { + if i == 0 { + if _, marker := el.(int32); marker { + continue + } + } + e, ok := el.(bson.D) + if !ok { + continue + } + b, _ := docValue(e, "$ID").(primitive.Binary) + ids = append(ids, types.BlobToUUID(b.Data)) + docs = append(docs, e) + } + return ids, docs +} + +func docValue(d bson.D, key string) any { + for _, e := range d { + if e.Key == key { + return e.Value + } + } + return nil +} + +func pointerID(d bson.D, key string) string { + b, _ := docValue(d, key).(primitive.Binary) + return types.BlobToUUID(b.Data) +} + +func storedPointToPED(s string) map[string]int { + x, y, _ := strings.Cut(s, ";") + px, _ := strconv.Atoi(x) + py, _ := strconv.Atoi(y) + return map[string]int{"x": px, "y": py} +} + +func pedVector(s string) map[string]int { + p := storedPointToPED(s) + return map[string]int{"width": p["x"], "height": p["y"]} +} + +var pedSides = []string{"Top", "Right", "Bottom", "Left"} + +// pedQualifiedFlowType maps the storage $Type of a flow object to the +// qualified name PED reports for it, where the two differ (CLAUDE.md, "BSON +// Storage Names vs Qualified Names"). +var pedQualifiedFlowType = map[string]string{ + "Microflows$MicroflowParameter": "Microflows$MicroflowParameterObject", +} + +// flowPatchOps derives the PED operations that turn the stored document into +// the spliced one. Order matters, because PED addresses entries by index: +// everything that uses a stored index (positions, rewired flows) and every +// append goes first, while indexes are still the stored ones; removals go +// last, highest index first. +func (b *Backend) flowPatchOps(d *mcpFlowDeps, spliced bson.D) ([]pedOpEntry, error) { + var ops []pedOpEntry + set := func(path string, v any) { + ops = append(ops, pedOpEntry{Path: path, Operation: pedOperation{Type: "set", Value: v}}) + } + oldObjIDs, oldObjs := flowList(d.stored, true) + newObjIDs, newObjs := flowList(spliced, true) + oldFlowIDs, oldFlows := flowList(d.stored, false) + newFlowIDs, newFlows := flowList(spliced, false) + + path := map[string]string{} + oldObjIndex := map[string]int{} + for i, id := range oldObjIDs { + oldObjIndex[id] = i + path[id] = fmt.Sprintf("/objectCollection/objects/%d", i) + } + survivingObj := map[string]bool{} + var added []string + for i, id := range newObjIDs { + if idx, ok := oldObjIndex[id]; ok { + survivingObj[id] = true + before, _ := docValue(oldObjs[idx], "RelativeMiddlePoint").(string) + after, _ := docValue(newObjs[i], "RelativeMiddlePoint").(string) + if before != after { + set(path[id]+"/relativeMiddlePoint", storedPointToPED(after)) + } + continue + } + added = append(added, id) + } + for j, id := range added { + path[id] = fmt.Sprintf("/objectCollection/objects/%d", len(oldObjIDs)+j) + } + for _, id := range added { + obj, ok := d.objects[id] + if !ok { + return nil, fmt.Errorf("internal: spliced object %s was not serialized", id) + } + idPath := map[model.ID]string{} + m, err := b.mapObjectTree(obj, path[id], idPath) + if err != nil { + return nil, err + } + var sets []pedOpEntry + skeletonObject(m, path[id], func(p string, v any) { + sets = append(sets, pedOpEntry{Path: p, Operation: pedOperation{Type: "set", Value: v}}) + }) + ops = append(ops, pedOpEntry{Path: "/objectCollection/objects", Operation: pedOperation{Type: "add", Value: m}}) + ops = append(ops, sets...) + // The skeleton constructor ignores a position; set it on the stored element. + p := obj.GetPosition() + set(path[id]+"/relativeMiddlePoint", map[string]int{"x": p.X, "y": p.Y}) + for nested, np := range idPath { + path[string(nested)] = np + } + } + ref := func(id string) (string, error) { + p, ok := path[id] + if !ok { + return "", fmt.Errorf("a flow points at %s, which is not a top-level object PED can address", id) + } + return "$id(" + p + ")", nil + } + + oldFlowIndex := map[string]int{} + for i, id := range oldFlowIDs { + oldFlowIndex[id] = i + } + survivingFlow := map[string]bool{} + var addedFlows []string + for i, id := range newFlowIDs { + idx, ok := oldFlowIndex[id] + if !ok { + addedFlows = append(addedFlows, id) + continue + } + survivingFlow[id] = true + fp := fmt.Sprintf("/flows/%d", idx) + o, n := oldFlows[idx], newFlows[i] + for _, end := range []string{"Origin", "Destination"} { + key := strings.ToLower(end) + if pointerID(o, end+"Pointer") != pointerID(n, end+"Pointer") { + r, err := ref(pointerID(n, end+"Pointer")) + if err != nil { + return nil, err + } + set(fp+"/"+key, r) + } + if fmt.Sprint(docValue(o, end+"ConnectionIndex")) != fmt.Sprint(docValue(n, end+"ConnectionIndex")) { + set(fp+"/"+key+"ConnectionIndex", docValue(n, end+"ConnectionIndex")) + } + ol, _ := docValue(o, "Line").(bson.D) + nl, _ := docValue(n, "Line").(bson.D) + if ov, nv := fmt.Sprint(docValue(ol, end+"ControlVector")), fmt.Sprint(docValue(nl, end+"ControlVector")); ov != nv && nv != "" { + set(fp+"/line/"+key+"ControlVector", pedVector(nv)) + } + } + } + for j, id := range addedFlows { + fp := fmt.Sprintf("/flows/%d", len(oldFlowIDs)+j) + if af, ok := d.annotations[id]; ok { + o, err := ref(string(af.OriginID)) + if err != nil { + return nil, err + } + dst, err := ref(string(af.DestinationID)) + if err != nil { + return nil, err + } + ops = append(ops, pedOpEntry{Path: "/flows", Operation: pedOperation{Type: "add", Value: map[string]any{ + "$Type": "Microflows$AnnotationFlow", "originId": o, "destinationId": dst}}}) + continue + } + f, ok := d.flows[id] + if !ok { + return nil, fmt.Errorf("internal: spliced flow %s was not serialized", id) + } + o, err := ref(string(f.OriginID)) + if err != nil { + return nil, err + } + dst, err := ref(string(f.DestinationID)) + if err != nil { + return nil, err + } + v := map[string]any{"$Type": "Microflows$SequenceFlow", "originId": o, "destinationId": dst} + if f.OriginConnectionIndex >= 0 && f.OriginConnectionIndex < 4 { + v["originConnectionSide"] = pedSides[f.OriginConnectionIndex] + } + if f.DestinationConnectionIndex >= 0 && f.DestinationConnectionIndex < 4 { + v["destinationConnectionSide"] = pedSides[f.DestinationConnectionIndex] + } + cv, err := mapCaseValue(f.CaseValue) + if err != nil { + return nil, err + } + if cv != nil { + v["caseValue"] = cv + } + ops = append(ops, pedOpEntry{Path: "/flows", Operation: pedOperation{Type: "add", Value: v}}) + if f.IsErrorHandler { + set(fp+"/isErrorHandler", true) + } + if f.OriginControlVector != "" { + set(fp+"/line/originControlVector", pedVector(f.OriginControlVector)) + } + if f.DestinationControlVector != "" { + set(fp+"/line/destinationControlVector", pedVector(f.DestinationControlVector)) + } + } + + for i := len(oldFlowIDs) - 1; i >= 0; i-- { + if !survivingFlow[oldFlowIDs[i]] { + idx := i + ops = append(ops, pedOpEntry{Path: "/flows", Operation: pedOperation{Type: "remove", Index: &idx}}) + } + } + for i := len(oldObjIDs) - 1; i >= 0; i-- { + if !survivingObj[oldObjIDs[i]] { + idx := i + ops = append(ops, pedOpEntry{Path: "/objectCollection/objects", Operation: pedOperation{Type: "remove", Index: &idx}}) + } + } + return ops, nil +} + +// checkLiveFlowMatches compares the live document's objects and flows with the +// stored ones the splice ran on: the same count, and each object the same type +// at the same position. PED addresses them by index, so a live document that +// has moved on since the .mpr was saved would have the patch applied to other +// elements than the ones the splice chose. +func (b *Backend) checkLiveFlowMatches(docType, qn string, stored bson.D) error { + res, err := b.client.CallTool("ped_read_document", map[string]any{ + "documentType": docType, + "documentName": qn, + "paths": []string{"/objectCollection/objects", "/flows"}, + }) + if err != nil { + return err + } + if res.IsError { + return fmt.Errorf("ped_read_document %s: %s", qn, pedStripReminder(res.Text)) + } + var body struct { + Results []struct { + Path string `json:"path"` + Result json.RawMessage `json:"result"` + } `json:"results"` + } + if err := json.Unmarshal([]byte(pedStripReminder(res.Text)), &body); err != nil { + return fmt.Errorf("read %s from Studio Pro: %w", qn, err) + } + stale := func(what string) error { + return fmt.Errorf("%s in Studio Pro no longer matches the local project (%s); save in Studio Pro so the .mpr is current, then retry", qn, what) + } + _, objs := flowList(stored, true) + _, flows := flowList(stored, false) + for _, r := range body.Results { + var live []map[string]any + if err := json.Unmarshal(r.Result, &live); err != nil { + return fmt.Errorf("read %s %s from Studio Pro: %w", qn, r.Path, err) + } + switch r.Path { + case "/flows": + if len(live) != len(flows) { + return stale(fmt.Sprintf("%d flows live, %d stored", len(live), len(flows))) + } + case "/objectCollection/objects": + if len(live) != len(objs) { + return stale(fmt.Sprintf("%d objects live, %d stored", len(live), len(objs))) + } + for i, o := range objs { + typ, _ := docValue(o, "$Type").(string) + if q, ok := pedQualifiedFlowType[typ]; ok { + typ = q + } + if live[i]["$Type"] != typ { + return stale(fmt.Sprintf("object %d is a %v live, a %s stored", i, live[i]["$Type"], typ)) + } + rp, _ := docValue(o, "RelativeMiddlePoint").(string) + want := storedPointToPED(rp) + pt, _ := live[i]["relativeMiddlePoint"].(map[string]any) + if fmt.Sprint(pt["x"]) != strconv.Itoa(want["x"]) || fmt.Sprint(pt["y"]) != strconv.Itoa(want["y"]) { + return stale(fmt.Sprintf("object %d is at %v live, at %s stored", i, pt, rp)) + } + } + } + } + return nil +} diff --git a/mdl/backend/mcp/microflow_mutator_test.go b/mdl/backend/mcp/microflow_mutator_test.go new file mode 100644 index 0000000000..7fad8b9900 --- /dev/null +++ b/mdl/backend/mcp/microflow_mutator_test.go @@ -0,0 +1,207 @@ +// SPDX-License-Identifier: Apache-2.0 + +package mcp + +import ( + "encoding/json" + "fmt" + "strings" + "testing" + + "go.mongodb.org/mongo-driver/bson" + "go.mongodb.org/mongo-driver/bson/primitive" + + "github.com/mendixlabs/mxcli/mdl/backend" + "github.com/mendixlabs/mxcli/mdl/backend/mfmutator" + "github.com/mendixlabs/mxcli/mdl/types" + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/sdk/microflows" +) + +func mfID(name string) primitive.Binary { + b := make([]byte, 16) + copy(b, name) + return primitive.Binary{Data: b} +} + +func mfUUID(name string) string { return types.BlobToUUID(mfID(name).Data) } + +func mfObj(name, typ string, x int) bson.D { + return bson.D{ + {Key: "$ID", Value: mfID(name)}, + {Key: "$Type", Value: typ}, + {Key: "RelativeMiddlePoint", Value: fmt.Sprintf("%d;200", x)}, + {Key: "Size", Value: "120;60"}, + } +} + +func mfFlow(name, from, to string) bson.D { + return bson.D{ + {Key: "$ID", Value: mfID(name)}, + {Key: "$Type", Value: "Microflows$SequenceFlow"}, + {Key: "DestinationConnectionIndex", Value: int32(3)}, + {Key: "DestinationPointer", Value: mfID(to)}, + {Key: "IsErrorHandler", Value: false}, + {Key: "Line", Value: bson.D{{Key: "$Type", Value: "Microflows$BezierCurve"}, + {Key: "DestinationControlVector", Value: "-30;0"}, {Key: "OriginControlVector", Value: "30;0"}}}, + {Key: "OriginConnectionIndex", Value: int32(1)}, + {Key: "OriginPointer", Value: mfID(from)}, + } +} + +// storedReduce is the shape of TestApp's Microflows.MicroflowReduce, the flow +// the live check ran on: a start, three activities 170 apart, an end. +func storedReduce() bson.D { + return bson.D{ + {Key: "$ID", Value: mfID("unit")}, + {Key: "$Type", Value: "Microflows$Microflow"}, + {Key: "Flows", Value: bson.A{int32(3), mfFlow("f0", "start", "a"), mfFlow("f1", "a", "b"), mfFlow("f2", "b", "end")}}, + {Key: "Name", Value: "Reduce"}, + {Key: "ObjectCollection", Value: bson.D{ + {Key: "$ID", Value: mfID("oc")}, + {Key: "$Type", Value: "Microflows$MicroflowObjectCollection"}, + {Key: "Objects", Value: bson.A{int32(3), + mfObj("start", "Microflows$StartEvent", -98), + mfObj("end", "Microflows$EndEvent", 564), + mfObj("a", "Microflows$ActionActivity", 214), + mfObj("b", "Microflows$ActionActivity", 404), + }}, + }}, + } +} + +func openFlowMutator(t *testing.T, b *Backend) (*mcpFlowMutator, *mcpFlowDeps) { + t.Helper() + stored := storedReduce() + raw, _ := bson.Marshal(stored) + var storedD, work bson.D + _ = bson.Unmarshal(raw, &storedD) + _ = bson.Unmarshal(raw, &work) + deps := &mcpFlowDeps{b: b, docType: microflowDocType, qn: "M.Reduce", stored: storedD, + objects: map[string]microflows.MicroflowObject{}, flows: map[string]*microflows.SequenceFlow{}, + annotations: map[string]*microflows.AnnotationFlow{}} + m, err := mfmutator.New(work, "unit", deps) + if err != nil { + t.Fatal(err) + } + return &mcpFlowMutator{Mutator: m, qn: "M.Reduce"}, deps +} + +func logFragment() *backend.MicroflowFragment { + id := model.ID(types.GenerateID()) + act := &microflows.ActionActivity{ + BaseActivity: microflows.BaseActivity{BaseMicroflowObject: microflows.BaseMicroflowObject{ + BaseElement: model.BaseElement{ID: id}, + Position: model.Point{X: 360, Y: 200}, + Size: model.Size{Width: 120, Height: 60}, + }}, + Action: &microflows.LogMessageAction{LogLevel: "Info", LogNodeName: "'Reduce'", + MessageTemplate: &model.Text{Translations: map[string]string{"en_US": "reduced"}}}, + } + return &backend.MicroflowFragment{Objects: []microflows.MicroflowObject{act}, Entry: id, Exit: id} +} + +// The splice becomes PED path operations addressed by the stored indexes: +// the moved end, the new activity (added bare, then its action and position +// set, as PED's skeleton constructor requires), the rewired flow's end, and +// the new flow — and no removal. +func TestFlowPatchOps_InsertAfter(t *testing.T) { + var sent []any + f := newFakePED(t, func(name string, args map[string]any) (string, bool) { + switch name { + case "ped_read_document": + return `{"results":[{"path":"/objectCollection/objects","result":[` + + `{"$Type":"Microflows$StartEvent","relativeMiddlePoint":{"x":-98,"y":200}},` + + `{"$Type":"Microflows$EndEvent","relativeMiddlePoint":{"x":564,"y":200}},` + + `{"$Type":"Microflows$ActionActivity","relativeMiddlePoint":{"x":214,"y":200}},` + + `{"$Type":"Microflows$ActionActivity","relativeMiddlePoint":{"x":404,"y":200}}]},` + + `{"path":"/flows","result":[{},{},{}]}]}`, false + case "ped_update_document": + sent = append(sent, args["operations"]) + return "SUCCESS: All operations have been performed successfully.", false + case "ped_check_errors": + return "No errors found.", false + } + return "SUCCESS", false + }) + b := &Backend{client: f.connectClient(t)} + m, _ := openFlowMutator(t, b) + if err := m.InsertAfter(model.ID(mfUUID("a")), logFragment()); err != nil { + t.Fatal(err) + } + if err := m.Save(); err != nil { + t.Fatalf("save: %v", err) + } + if len(sent) != 1 { + t.Fatalf("want one ped_update_document, got %d", len(sent)) + } + js, _ := json.Marshal(sent[0]) + got := string(js) + var ops []pedOpEntry + if err := json.Unmarshal(js, &ops); err != nil { + t.Fatal(err) + } + has := func(path, typ, value string) bool { + for _, op := range ops { + v, _ := json.Marshal(op.Operation.Value) + if op.Path == path && op.Operation.Type == typ && strings.Contains(string(v), value) { + return true + } + } + return false + } + for _, w := range []struct{ path, typ, value string }{ + {"/objectCollection/objects/3/relativeMiddlePoint", "set", `"x":534`}, // b moved along + {"/objectCollection/objects", "add", `"Microflows$ActionActivity"`}, + {"/objectCollection/objects/4/action", "set", `"Microflows$LogMessageAction"`}, + {"/flows/1/destination", "set", `$id(/objectCollection/objects/4)`}, + {"/flows", "add", `"destinationId":"$id(/objectCollection/objects/3)"`}, + {"/flows", "add", `"originId":"$id(/objectCollection/objects/4)"`}, + } { + if !has(w.path, w.typ, w.value) { + t.Errorf("no %s %s with %s in:\n%s", w.typ, w.path, w.value, got) + } + } + if strings.Contains(got, `"remove"`) { + t.Errorf("an insert sent a removal:\n%s", got) + } +} + +// PED addresses by index, so a live document that differs from the stored one +// must refuse the write before anything is sent. +func TestFlowPatchOps_StaleLiveDocumentIsRefused(t *testing.T) { + updates := 0 + f := newFakePED(t, func(name string, _ map[string]any) (string, bool) { + switch name { + case "ped_read_document": + return `{"results":[{"path":"/objectCollection/objects","result":[{},{},{},{},{}]},{"path":"/flows","result":[{},{},{}]}]}`, false + case "ped_update_document": + updates++ + } + return "SUCCESS", false + }) + b := &Backend{client: f.connectClient(t)} + m, _ := openFlowMutator(t, b) + if err := m.InsertAfter(model.ID(mfUUID("a")), logFragment()); err != nil { + t.Fatal(err) + } + err := m.Save() + if err == nil || !strings.Contains(err.Error(), "no longer matches the local project") { + t.Fatalf("want the stale-document refusal, got %v", err) + } + if updates != 0 { + t.Error("an update was sent to a document that did not match") + } +} + +// Drop and replace need a removal, which PED does not roll back when an +// update fails; over MCP they are refused. +func TestMCPFlowMutator_RefusesRemovals(t *testing.T) { + m, _ := openFlowMutator(t, &Backend{}) + if err := m.Drop(model.ID(mfUUID("a"))); err == nil || !strings.Contains(err.Error(), "not supported by the MCP backend") { + t.Errorf("drop: %v", err) + } + if err := m.Replace(model.ID(mfUUID("a")), logFragment()); err == nil || !strings.Contains(err.Error(), "not supported by the MCP backend") { + t.Errorf("replace: %v", err) + } +} diff --git a/mdl/backend/mcp/unsupported_gen.go b/mdl/backend/mcp/unsupported_gen.go index a1dc96a9c1..abae5e3a3a 100644 --- a/mdl/backend/mcp/unsupported_gen.go +++ b/mdl/backend/mcp/unsupported_gen.go @@ -992,6 +992,11 @@ func (unsupportedBackend) MoveViewEntitySourceDocument(_ string, _ model.ID, _ s return } +func (unsupportedBackend) OpenMicroflowForMutation(_ model.ID) (r0 backend.MicroflowMutator, err1 error) { + err1 = errUnsupported("OpenMicroflowForMutation") + return +} + func (unsupportedBackend) OpenPageForMutation(_ model.ID) (r0 backend.PageMutator, err1 error) { err1 = errUnsupported("OpenPageForMutation") return diff --git a/mdl/backend/mfmutator/splice.go b/mdl/backend/mfmutator/splice.go new file mode 100644 index 0000000000..011cf3e1e7 --- /dev/null +++ b/mdl/backend/mfmutator/splice.go @@ -0,0 +1,962 @@ +// SPDX-License-Identifier: Apache-2.0 + +package mfmutator + +// # The graph splice (plan item 4.2b) +// +// An `alter microflow` operation edits the STORED document, never a rebuild of +// it. The rebuild (`UpdateMicroflow`, then re-pairing IDs by type and position) +// is what deleted merges, reset curves and moved $IDs onto other nodes on a +// Studio Pro-authored flow (ADR-0012, Context). So the splice works on the raw +// BSON tree of the unit as read from storage: +// +// - the fragment's objects are the only elements it builds, and they are +// appended to the collection that holds the target; +// - the sequence flows around the target are rewired by changing their +// pointers, connection sides and the control vector at the rewired end — +// every other property of a rewired flow (its $ID, case value, error-handler +// flag, the curve at its other end) is kept; +// - nodes are moved only to make room, and only by a translation; +// - nothing else is touched, and no $ID is ever rewritten. A removed element +// leaves no reference behind: Save refuses a document in which any binary +// still names an element that was removed (CLAUDE.md rule 1), and one in +// which two elements share an $ID. +// +// What the splice cannot do safely it refuses, with the reason: an insert +// after a decision (which branch?), before an activity several flows enter +// (which path?), a drop or replace of an activity with an error handler (the +// handler would be orphaned), and — for now — anything inside a loop body, +// whose coordinates are relative to the loop box. + +import ( + "bytes" + "fmt" + "strconv" + "strings" + + "go.mongodb.org/mongo-driver/bson" + "go.mongodb.org/mongo-driver/bson/primitive" + + "github.com/mendixlabs/mxcli/mdl/backend" + "github.com/mendixlabs/mxcli/mdl/types" + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/sdk/microflows" +) + +// Deps supplies the engine-specific steps: turning a new object or flow into +// its stored BSON form, and writing the unit back. Everything between — which +// flows to rewire, where to place, what to remove — is decided here, once, for +// every storage engine. +type Deps interface { + SerializeObject(obj microflows.MicroflowObject) (bson.D, error) + SerializeSequenceFlow(f *microflows.SequenceFlow) (bson.D, error) + SerializeAnnotationFlow(f *microflows.AnnotationFlow) (bson.D, error) + // SaveUnit writes the patched unit. The implementation must go through the + // storage engine's reconciling write (canon.Reconcile), like every write. + SaveUnit(unitID string, contents []byte) error +} + +// Mutator splices fragments into one microflow or nanoflow unit. +type Mutator struct { + deps Deps + unitID model.ID + doc bson.D + // removed holds the $IDs of every element an operation took out, so Save + // can prove that nothing still points at one of them. + removed map[string]bool +} + +var _ backend.MicroflowMutator = (*Mutator)(nil) + +// New returns a Mutator over a decoded Microflows$Microflow or +// Microflows$Nanoflow unit. +func New(doc bson.D, unitID model.ID, deps Deps) (*Mutator, error) { + switch t := dString(doc, "$Type"); t { + case "Microflows$Microflow", "Microflows$Nanoflow": + default: + return nil, fmt.Errorf("unit %s is a %s, not a microflow or nanoflow", unitID, t) + } + if dDoc(doc, "ObjectCollection") == nil { + return nil, fmt.Errorf("unit %s has no object collection", unitID) + } + return &Mutator{deps: deps, unitID: unitID, doc: doc, removed: map[string]bool{}}, nil +} + +// Bytes returns the patched unit, after the integrity checks Save applies. +func (m *Mutator) Bytes() ([]byte, error) { + if err := m.checkIntegrity(); err != nil { + return nil, err + } + return bson.Marshal(m.doc) +} + +// Save writes the patched unit through Deps. +func (m *Mutator) Save() error { + out, err := m.Bytes() + if err != nil { + return err + } + return m.deps.SaveUnit(string(m.unitID), out) +} + +// --------------------------------------------------------------------------- +// The graph view +// --------------------------------------------------------------------------- + +// node is one object of the flow as stored. +type node struct { + id string // model ID form (types.BlobToUUID of the $ID) + typ string + doc bson.D + pos point + size point + // loop is the $ID of the loop whose body holds the node, "" at top level. + loop string +} + +// flowRef is one entry of the unit's Flows list. +type flowRef struct { + id string + typ string + doc bson.D + origin string + dest string + isErr bool +} + +func (f flowRef) isSequence() bool { return f.typ == "Microflows$SequenceFlow" } + +type graph struct { + nodes map[string]*node + // order is the nodes in storage order, so a shift visits them the same + // way every run. + order []*node + flows []flowRef +} + +func (m *Mutator) graph() *graph { + g := &graph{nodes: map[string]*node{}} + var walk func(oc bson.D, loop string) + walk = func(oc bson.D, loop string) { + for _, el := range arrayElements(dGet(oc, "Objects")) { + d, ok := el.(bson.D) + if !ok { + continue + } + n := &node{ + id: binaryID(dGet(d, "$ID")), + typ: dString(d, "$Type"), + doc: d, + pos: parsePoint(dString(d, "RelativeMiddlePoint")), + size: parsePoint(dString(d, "Size")), + loop: loop, + } + if n.id != "" { + g.nodes[n.id] = n + g.order = append(g.order, n) + } + if n.typ == "Microflows$LoopedActivity" { + if inner := dDoc(d, "ObjectCollection"); inner != nil { + walk(inner, n.id) + } + } + } + } + walk(dDoc(m.doc, "ObjectCollection"), "") + for _, el := range arrayElements(dGet(m.doc, "Flows")) { + d, ok := el.(bson.D) + if !ok { + continue + } + isErr, _ := dGet(d, "IsErrorHandler").(bool) + g.flows = append(g.flows, flowRef{ + id: binaryID(dGet(d, "$ID")), + typ: dString(d, "$Type"), + doc: d, + origin: binaryID(dGet(d, "OriginPointer")), + dest: binaryID(dGet(d, "DestinationPointer")), + isErr: isErr, + }) + } + return g +} + +// outgoing returns the sequence flows leaving id, split into normal and +// error-handler flows. +func (g *graph) outgoing(id string) (normal, errs []flowRef) { + for _, f := range g.flows { + if f.isSequence() && f.origin == id { + if f.isErr { + errs = append(errs, f) + } else { + normal = append(normal, f) + } + } + } + return normal, errs +} + +// incoming returns the sequence flows entering id. +func (g *graph) incoming(id string) []flowRef { + var out []flowRef + for _, f := range g.flows { + if f.isSequence() && f.dest == id { + out = append(out, f) + } + } + return out +} + +// annotationFlows returns the annotation flows attached to id. +func (g *graph) annotationFlows(id string) []flowRef { + var out []flowRef + for _, f := range g.flows { + if !f.isSequence() && (f.origin == id || f.dest == id) { + out = append(out, f) + } + } + return out +} + +func (g *graph) node(id model.ID) (*node, error) { + n, ok := g.nodes[string(id)] + if !ok { + return nil, fmt.Errorf("activity %s is not in the stored flow (was it dropped by an earlier operation?)", id) + } + if n.loop != "" { + return nil, fmt.Errorf("%s is inside a loop body; alter does not splice inside a loop yet — "+ + "address the loop itself, or rewrite the loop with create or modify", describeNode(n)) + } + return n, nil +} + +// --------------------------------------------------------------------------- +// Operations +// --------------------------------------------------------------------------- + +// InsertAfter splices frag onto the flow leaving target. +func (m *Mutator) InsertAfter(target model.ID, frag *backend.MicroflowFragment) error { + g := m.graph() + x, err := g.node(target) + if err != nil { + return err + } + normal, _ := g.outgoing(x.id) + switch { + case len(normal) == 0: + return fmt.Errorf("nothing follows %s: it ends the flow; insert before it instead", describeNode(x)) + case len(normal) > 1: + return fmt.Errorf("%s has %d outgoing flows (it is a decision): insert after it would not say which branch; "+ + "insert before the first activity of the branch instead", describeNode(x), len(normal)) + } + return m.spliceOnFlow(g, normal[0], frag) +} + +// InsertBefore splices frag onto the flow entering target. +func (m *Mutator) InsertBefore(target model.ID, frag *backend.MicroflowFragment) error { + g := m.graph() + y, err := g.node(target) + if err != nil { + return err + } + in := g.incoming(y.id) + switch { + case len(in) == 0: + return fmt.Errorf("no flow enters %s; there is no path to insert on", describeNode(y)) + case len(in) > 1: + return fmt.Errorf("%d flows enter %s: insert before it would not say on which path; "+ + "insert after one of its predecessors instead", len(in), describeNode(y)) + } + return m.spliceOnFlow(g, in[0], frag) +} + +// spliceOnFlow puts frag on flow f (origin X, destination Y): f keeps its +// origin end and now enters the fragment's entry; a new flow leaves the +// fragment's exit and enters Y where f used to. +func (m *Mutator) spliceOnFlow(g *graph, f flowRef, frag *backend.MicroflowFragment) error { + x, y := g.nodes[f.origin], g.nodes[f.dest] + if x == nil || y == nil { + return fmt.Errorf("flow %s points at an object that is not in the flow", f.id) + } + if x.loop != "" || y.loop != "" { + return fmt.Errorf("the flow runs inside a loop body; alter does not splice inside a loop yet") + } + fb, err := fragmentGeometry(frag) + if err != nil { + return err + } + ax, s, err := flowAxis(x, y) + if err != nil { + return err + } + + // Make room: the fragment plus a gap on either side has to fit between + // X's and Y's facing edges; what is missing is added by moving everything + // past the midpoint of that gap further along the axis. + a := x.pos.get(ax) + s*x.size.get(ax)/2 + b := y.pos.get(ax) - s*y.size.get(ax)/2 + need := fb.length(ax) + 2*minGap + if deficit := need - s*(b-a); deficit > 0 { + m.shift(g, ax, s, float64(a+b)/2, s*deficit, "") + b += s * deficit + } + + // Place the fragment centred in the gap, its entry level with the flow. + cross := ax.other() + mid := point{} + mid = mid.set(ax, (a+b)/2) + mid = mid.set(cross, (x.pos.get(cross)+y.pos.get(cross))/2) + fb.placeCentred(ax, mid) + if err := fb.checkRoom(g, ""); err != nil { + return err + } + + entrySide, exitSide := side(ax, -s), side(ax, s) + destIdx, destVec := flowEnd(f.doc, "Destination") + if err := m.addFragment(frag); err != nil { + return err + } + // f now enters the fragment. + setPointer(f.doc, "DestinationPointer", frag.Entry) + setInt(f.doc, "DestinationConnectionIndex", entrySide) + setVector(f.doc, "DestinationControlVector", sideVector(entrySide)) + // And a new flow carries on from the fragment to Y. + return m.addFlow(&microflows.SequenceFlow{ + BaseElement: model.BaseElement{ID: model.ID(types.GenerateID())}, + OriginID: frag.Exit, + DestinationID: model.ID(y.id), + OriginConnectionIndex: exitSide, + DestinationConnectionIndex: destIdx, + OriginControlVector: sideVector(exitSide), + DestinationControlVector: destVec, + }) +} + +// Replace puts frag where target is: every flow that entered target enters +// the fragment, the flow that left it leaves the fragment's exit, and +// annotations attached to target are attached to the fragment's entry. +func (m *Mutator) Replace(target model.ID, frag *backend.MicroflowFragment) error { + g := m.graph() + x, err := g.node(target) + if err != nil { + return err + } + out, err := removable(g, x, "replace") + if err != nil { + return err + } + y := g.nodes[out.dest] + if y == nil { + return fmt.Errorf("flow %s points at an object that is not in the flow", out.id) + } + fb, err := fragmentGeometry(frag) + if err != nil { + return err + } + ax, s, err := flowAxis(x, y) + if err != nil { + return err + } + // The fragment starts where target started; if it is longer, everything + // past target moves along by the difference. + near := x.pos.get(ax) - s*x.size.get(ax)/2 + if extra := fb.length(ax) - x.size.get(ax); extra > 0 { + m.shift(g, ax, s, float64(x.pos.get(ax)), s*extra, x.id) + } + centre := point{} + centre = centre.set(ax, near+s*fb.length(ax)/2) + centre = centre.set(ax.other(), x.pos.get(ax.other())) + fb.placeCentred(ax, centre) + if err := fb.checkRoom(g, x.id); err != nil { + return err + } + + if err := m.addFragment(frag); err != nil { + return err + } + for _, in := range g.incoming(x.id) { + setPointer(in.doc, "DestinationPointer", frag.Entry) + } + setPointer(out.doc, "OriginPointer", frag.Exit) + for _, af := range g.annotationFlows(x.id) { + key := "DestinationPointer" + if af.origin == x.id { + key = "OriginPointer" + } + setPointer(af.doc, key, frag.Entry) + } + m.removeFlows(g.bodyFlows(x.id)) + return m.removeObject(x) +} + +// Drop removes target and joins the flows that entered it to the object it +// led to. Annotation lines attached to it go with it; the annotations stay. +func (m *Mutator) Drop(target model.ID) error { + g := m.graph() + x, err := g.node(target) + if err != nil { + return err + } + out, err := removable(g, x, "drop") + if err != nil { + return err + } + destIdx, destVec := flowEnd(out.doc, "Destination") + destPtr := dGet(out.doc, "DestinationPointer") + for _, in := range g.incoming(x.id) { + if in.origin == out.dest { + return fmt.Errorf("dropping %s would leave a flow from an object to itself", describeNode(x)) + } + } + for _, in := range g.incoming(x.id) { + setBinary(in.doc, "DestinationPointer", destPtr) + setInt(in.doc, "DestinationConnectionIndex", destIdx) + setVector(in.doc, "DestinationControlVector", destVec) + } + drop := g.bodyFlows(x.id) + drop[out.id] = true + for _, af := range g.annotationFlows(x.id) { + drop[af.id] = true + } + m.removeFlows(drop) + return m.removeObject(x) +} + +// bodyFlows returns the flows that run inside loop's body, at any depth. They +// are stored in the unit's Flows list, not in the loop, so taking the loop out +// has to take them too; left behind they would point at removed objects. +func (g *graph) bodyFlows(loop string) map[string]bool { + inside := func(id string) bool { + for n := g.nodes[id]; n != nil && n.loop != ""; n = g.nodes[n.loop] { + if n.loop == loop { + return true + } + } + return false + } + out := map[string]bool{} + for _, f := range g.flows { + if inside(f.origin) || inside(f.dest) { + out[f.id] = true + } + } + return out +} + +// removable checks that x can be taken out of the flow and returns the one +// flow that leaves it. +func removable(g *graph, x *node, verb string) (flowRef, error) { + switch x.typ { + case "Microflows$ActionActivity", "Microflows$LoopedActivity": + default: + return flowRef{}, fmt.Errorf("cannot %s %s: only an activity or a loop can be, because a decision or an end event "+ + "changes the shape of the flow", verb, describeNode(x)) + } + normal, errs := g.outgoing(x.id) + if len(errs) > 0 { + return flowRef{}, fmt.Errorf("cannot %s %s: it has an error handler, which would be left with no activity; "+ + "rewrite the flow with create or modify instead", verb, describeNode(x)) + } + if len(normal) != 1 { + return flowRef{}, fmt.Errorf("cannot %s %s: it has %d outgoing flows, not one", verb, describeNode(x), len(normal)) + } + return normal[0], nil +} + +// --------------------------------------------------------------------------- +// Tree edits +// --------------------------------------------------------------------------- + +// addFragment serializes the fragment and appends it: its objects to the top +// level collection (appended, so no stored object changes list position), its +// flows to the Flows list. +func (m *Mutator) addFragment(frag *backend.MicroflowFragment) error { + oc := dDoc(m.doc, "ObjectCollection") + objs := arrayElements(dGet(oc, "Objects")) + for _, obj := range frag.Objects { + d, err := m.deps.SerializeObject(obj) + if err != nil { + return fmt.Errorf("serialize %T: %w", obj, err) + } + if d == nil { + return fmt.Errorf("cannot write a %T into a flow", obj) + } + objs = append(objs, d) + } + if !setArray(oc, "Objects", objs) { + return fmt.Errorf("the object collection has no Objects list") + } + for _, f := range frag.Flows { + if err := m.addFlow(f); err != nil { + return err + } + } + for _, af := range frag.AnnotationFlows { + d, err := m.deps.SerializeAnnotationFlow(af) + if err != nil { + return fmt.Errorf("serialize annotation flow: %w", err) + } + if err := m.appendFlowDoc(d); err != nil { + return err + } + } + return nil +} + +func (m *Mutator) addFlow(f *microflows.SequenceFlow) error { + d, err := m.deps.SerializeSequenceFlow(f) + if err != nil { + return fmt.Errorf("serialize sequence flow: %w", err) + } + return m.appendFlowDoc(d) +} + +func (m *Mutator) appendFlowDoc(d bson.D) error { + if d == nil { + return fmt.Errorf("a flow serialized to nothing") + } + flows := arrayElements(dGet(m.doc, "Flows")) + flows = append(flows, d) + if !setArray(m.doc, "Flows", flows) { + return fmt.Errorf("the unit has no Flows list") + } + return nil +} + +func (m *Mutator) removeFlows(ids map[string]bool) { + var keep []any + for _, el := range arrayElements(dGet(m.doc, "Flows")) { + if d, ok := el.(bson.D); ok && ids[binaryID(dGet(d, "$ID"))] { + m.markRemoved(d) + continue + } + keep = append(keep, el) + } + setArray(m.doc, "Flows", keep) +} + +// removeObject takes x out of the top-level collection. +func (m *Mutator) removeObject(x *node) error { + oc := dDoc(m.doc, "ObjectCollection") + var keep []any + found := false + for _, el := range arrayElements(dGet(oc, "Objects")) { + if d, ok := el.(bson.D); ok && binaryID(dGet(d, "$ID")) == x.id { + m.markRemoved(d) + found = true + continue + } + keep = append(keep, el) + } + if !found { + return fmt.Errorf("%s is not in the top-level collection", describeNode(x)) + } + setArray(oc, "Objects", keep) + return nil +} + +// markRemoved records the $ID of d and of every element nested in it. +func (m *Mutator) markRemoved(v any) { + switch t := v.(type) { + case bson.D: + for _, e := range t { + if e.Key == "$ID" { + if b, ok := e.Value.(primitive.Binary); ok { + m.removed[string(b.Data)] = true + } + continue + } + m.markRemoved(e.Value) + } + case bson.A: + for _, el := range t { + m.markRemoved(el) + } + } +} + +// shift moves every top-level node past the cut (along axis ax, in the +// direction of s) by delta. The sweep keeps everything on either side of the +// cut exactly as it was relative to its neighbours, so only the flows that +// cross the cut get longer; a flow's control vectors are relative to its ends, +// so no curve changes. skip is a node that is about to be removed. +func (m *Mutator) shift(g *graph, ax axis, s int, cut float64, delta int, skip string) { + for _, n := range g.order { + if n.loop != "" || n.id == skip { + continue + } + if float64(s)*(float64(n.pos.get(ax))-cut) <= 0 { + continue + } + n.pos = n.pos.set(ax, n.pos.get(ax)+delta) + dSet(n.doc, "RelativeMiddlePoint", n.pos.String()) + } +} + +// checkIntegrity is the rule-1 guard: no binary anywhere in the unit may +// still name an element that was removed, and no two elements may share an +// $ID. Either would make the document unopenable, and neither is caught by +// anything cheaper than Studio Pro. +func (m *Mutator) checkIntegrity() error { + seen := map[string]bool{} + var dup, dangling string + var walk func(v any, key string) + walk = func(v any, key string) { + switch t := v.(type) { + case bson.D: + for _, e := range t { + walk(e.Value, e.Key) + } + case bson.A: + for _, el := range t { + walk(el, key) + } + case primitive.Binary: + if len(t.Data) != 16 { + return + } + k := string(t.Data) + if key == "$ID" { + if seen[k] && dup == "" { + dup = types.BlobToUUID(t.Data) + } + seen[k] = true + return + } + if m.removed[k] && dangling == "" { + dangling = key + " -> " + types.BlobToUUID(t.Data) + } + } + } + walk(m.doc, "") + if dup != "" { + return fmt.Errorf("refusing to write: two elements would share $ID %s", dup) + } + if dangling != "" { + return fmt.Errorf("refusing to write: a reference still points at a removed element (%s)", dangling) + } + return nil +} + +// --------------------------------------------------------------------------- +// Geometry +// --------------------------------------------------------------------------- + +// minGap is the free space kept on each side of an inserted fragment, the +// edge-to-edge distance mxcli's own layout leaves between activities. +const minGap = 40 + +type axis int + +const ( + axisX axis = iota + axisY +) + +func (a axis) other() axis { return 1 - a } + +type point struct{ X, Y int } + +func (p point) get(a axis) int { + if a == axisX { + return p.X + } + return p.Y +} + +func (p point) set(a axis, v int) point { + if a == axisX { + p.X = v + } else { + p.Y = v + } + return p +} + +func (p point) String() string { return strconv.Itoa(p.X) + ";" + strconv.Itoa(p.Y) } + +func parsePoint(s string) point { + x, y, ok := strings.Cut(s, ";") + if !ok { + return point{} + } + px, _ := strconv.Atoi(strings.TrimSpace(x)) + py, _ := strconv.Atoi(strings.TrimSpace(y)) + return point{px, py} +} + +// flowAxis is the direction a flow from x to y runs: the axis along which the +// centres are further apart, and the sign along it. +func flowAxis(x, y *node) (axis, int, error) { + dx, dy := y.pos.X-x.pos.X, y.pos.Y-x.pos.Y + switch { + case dx == 0 && dy == 0: + return 0, 0, fmt.Errorf("%s and %s are drawn on the same spot; cannot tell which way the flow runs", describeNode(x), describeNode(y)) + case abs(dx) >= abs(dy): + return axisX, sign(dx), nil + default: + return axisY, sign(dy), nil + } +} + +// Connection indexes, as Mendix numbers the sides of a box. +const ( + sideTop = 0 + sideRight = 1 + sideBottom = 2 + sideLeft = 3 +) + +// side is the side of a box that faces direction s along ax. +func side(ax axis, s int) int { + switch { + case ax == axisX && s > 0: + return sideRight + case ax == axisX: + return sideLeft + case s > 0: + return sideBottom + default: + return sideTop + } +} + +// sideVector is the control vector Studio Pro draws a straight flow end with: +// perpendicular to the side, pointing out of the box. +func sideVector(sd int) string { + switch sd { + case sideTop: + return "0;-15" + case sideRight: + return "15;0" + case sideBottom: + return "0;15" + default: + return "-15;0" + } +} + +// fragmentBox is the fragment's layout as the builder produced it, to be +// translated into place. +type fragmentBox struct { + frag *backend.MicroflowFragment + min, max point +} + +func fragmentGeometry(frag *backend.MicroflowFragment) (*fragmentBox, error) { + if frag == nil || len(frag.Objects) == 0 || frag.Entry == "" || frag.Exit == "" { + return nil, fmt.Errorf("the fragment is empty") + } + fb := &fragmentBox{frag: frag} + first := true + for _, obj := range frag.Objects { + p, sz := obj.GetPosition(), objectSize(obj) + lo := point{p.X - sz.X/2, p.Y - sz.Y/2} + hi := point{p.X + sz.X/2, p.Y + sz.Y/2} + if first { + fb.min, fb.max, first = lo, hi, false + continue + } + fb.min = point{min(fb.min.X, lo.X), min(fb.min.Y, lo.Y)} + fb.max = point{max(fb.max.X, hi.X), max(fb.max.Y, hi.Y)} + } + return fb, nil +} + +func (fb *fragmentBox) length(ax axis) int { return fb.max.get(ax) - fb.min.get(ax) } + +// placeCentred translates the fragment so that its box is centred on c along +// ax, and its entry object sits on c across it. +func (fb *fragmentBox) placeCentred(ax axis, c point) { + var entry point + for _, obj := range fb.frag.Objects { + if obj.GetID() == fb.frag.Entry { + entry = point{obj.GetPosition().X, obj.GetPosition().Y} + } + } + d := point{} + d = d.set(ax, c.get(ax)-(fb.min.get(ax)+fb.max.get(ax))/2) + d = d.set(ax.other(), c.get(ax.other())-entry.get(ax.other())) + for _, obj := range fb.frag.Objects { + p := obj.GetPosition() + obj.SetPosition(model.Point{X: p.X + d.X, Y: p.Y + d.Y}) + } + fb.min = point{fb.min.X + d.X, fb.min.Y + d.Y} + fb.max = point{fb.max.X + d.X, fb.max.Y + d.Y} +} + +// checkRoom refuses a placement that would draw the fragment over an object +// that stays where it is. The shift clears the space past the cut; an object +// that already sat inside the gap (a branch drawn below the main line, say) +// is not moved, and drawing over it would hide it in Studio Pro. skip is the +// object a replace removes. +func (fb *fragmentBox) checkRoom(g *graph, skip string) error { + for _, n := range g.order { + if n.loop != "" || n.id == skip { + continue + } + lo := point{n.pos.X - n.size.X/2, n.pos.Y - n.size.Y/2} + hi := point{n.pos.X + n.size.X/2, n.pos.Y + n.size.Y/2} + if lo.X < fb.max.X && fb.min.X < hi.X && lo.Y < fb.max.Y && fb.min.Y < hi.Y { + return fmt.Errorf("there is no free room for the fragment: at (%d, %d)-(%d, %d) it would be drawn over %s; "+ + "move that object aside in Studio Pro first", fb.min.X, fb.min.Y, fb.max.X, fb.max.Y, describeNode(n)) + } + } + return nil +} + +func objectSize(obj microflows.MicroflowObject) point { + if s, ok := obj.(interface{ GetSize() model.Size }); ok { + sz := s.GetSize() + return point{sz.Width, sz.Height} + } + return point{} +} + +// flowEnd reads a flow's connection index and control vector at one end +// ("Origin" or "Destination"). +func flowEnd(d bson.D, end string) (int, string) { + idx := 0 + switch v := dGet(d, end+"ConnectionIndex").(type) { + case int32: + idx = int(v) + case int64: + idx = int(v) + case int: + idx = v + } + vec := "" + if line := dDoc(d, "Line"); line != nil { + vec = dString(line, end+"ControlVector") + } + return idx, vec +} + +func abs(v int) int { + if v < 0 { + return -v + } + return v +} + +func sign(v int) int { + if v < 0 { + return -1 + } + return 1 +} + +func describeNode(n *node) string { + name := strings.TrimPrefix(n.typ, "Microflows$") + if c := dString(n.doc, "Caption"); c != "" && name != "ActionActivity" { + name += " '" + c + "'" + } + return fmt.Sprintf("the %s at (%d, %d)", name, n.pos.X, n.pos.Y) +} + +// --------------------------------------------------------------------------- +// bson.D helpers. The unit is decoded with bson v1 into bson.D, whose nested +// documents are bson.D and whose arrays are bson.A with Mendix's leading +// int32 list marker. +// --------------------------------------------------------------------------- + +func dGet(d bson.D, key string) any { + for _, e := range d { + if e.Key == key { + return e.Value + } + } + return nil +} + +func dDoc(d bson.D, key string) bson.D { + v, _ := dGet(d, key).(bson.D) + return v +} + +func dString(d bson.D, key string) string { + v, _ := dGet(d, key).(string) + return v +} + +func dSet(d bson.D, key string, v any) bool { + for i := range d { + if d[i].Key == key { + d[i].Value = v + return true + } + } + return false +} + +// arrayElements returns the elements of a Mendix list, without its marker. +func arrayElements(v any) []any { + a, ok := v.(bson.A) + if !ok || len(a) == 0 { + return nil + } + if _, isMarker := a[0].(int32); isMarker { + return append([]any(nil), a[1:]...) + } + return append([]any(nil), a...) +} + +// setArray replaces a list's elements, keeping its stored marker. +func setArray(d bson.D, key string, elements []any) bool { + a, ok := dGet(d, key).(bson.A) + if !ok { + return false + } + out := bson.A{} + if len(a) > 0 { + if marker, isMarker := a[0].(int32); isMarker { + out = append(out, marker) + } + } + return dSet(d, key, append(out, elements...)) +} + +func binaryID(v any) string { + b, ok := v.(primitive.Binary) + if !ok || len(b.Data) != 16 { + return "" + } + return types.BlobToUUID(b.Data) +} + +// setPointer points key at the element id, keeping the binary subtype the +// stored pointer uses. +func setPointer(d bson.D, key string, id model.ID) { + sub := byte(0) + if b, ok := dGet(d, key).(primitive.Binary); ok { + sub = b.Subtype + } + dSet(d, key, primitive.Binary{Subtype: sub, Data: types.UUIDToBlob(string(id))}) +} + +func setBinary(d bson.D, key string, v any) { + if b, ok := v.(primitive.Binary); ok { + dSet(d, key, primitive.Binary{Subtype: b.Subtype, Data: bytes.Clone(b.Data)}) + } +} + +// setInt writes an integer property with the width it is stored in. +func setInt(d bson.D, key string, v int) { + switch dGet(d, key).(type) { + case int64: + dSet(d, key, int64(v)) + default: + dSet(d, key, int32(v)) + } +} + +// setVector sets a control vector on the flow's line. A flow without a +// Bezier line (a pre-10 document) has nothing to set. +func setVector(d bson.D, key, v string) { + if v == "" { + return + } + if line := dDoc(d, "Line"); line != nil { + dSet(line, key, v) + } +} diff --git a/mdl/backend/mfmutator/splice_test.go b/mdl/backend/mfmutator/splice_test.go new file mode 100644 index 0000000000..a665bff523 --- /dev/null +++ b/mdl/backend/mfmutator/splice_test.go @@ -0,0 +1,404 @@ +// SPDX-License-Identifier: Apache-2.0 + +package mfmutator + +import ( + "bytes" + "fmt" + "strings" + "testing" + + "go.mongodb.org/mongo-driver/bson" + "go.mongodb.org/mongo-driver/bson/primitive" + + "github.com/mendixlabs/mxcli/mdl/backend" + "github.com/mendixlabs/mxcli/mdl/types" + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/sdk/microflows" +) + +// These tests run the splice on hand-built documents, for the shapes the +// Studio Pro fixture does not have (loops, error handlers, vertical flows). +// The acceptance test on a Studio Pro-drawn flow is in the executor package +// (cmd_alter_flow_pedapp_test.go). + +// uid returns a deterministic element id for a short name. +func uid(name string) string { + b := make([]byte, 16) + copy(b, name) + return types.BlobToUUID(b) +} + +func bin(name string) primitive.Binary { + return primitive.Binary{Subtype: 0, Data: types.UUIDToBlob(uid(name))} +} + +func obj(name, typ string, x, y int) bson.D { + w, h := 120, 60 + if typ != "Microflows$ActionActivity" && typ != "Microflows$LoopedActivity" { + w, h = 20, 20 + } + return bson.D{ + {Key: "$ID", Value: bin(name)}, + {Key: "$Type", Value: typ}, + {Key: "RelativeMiddlePoint", Value: fmt.Sprintf("%d;%d", x, y)}, + {Key: "Size", Value: fmt.Sprintf("%d;%d", w, h)}, + } +} + +func flow(name, from, to string, fromSide, toSide int32, isErr bool) bson.D { + return bson.D{ + {Key: "$ID", Value: bin(name)}, + {Key: "$Type", Value: "Microflows$SequenceFlow"}, + {Key: "CaseValues", Value: bson.A{int32(2), bson.D{{Key: "$ID", Value: bin(name + "c")}, {Key: "$Type", Value: "Microflows$NoCase"}}}}, + {Key: "DestinationConnectionIndex", Value: toSide}, + {Key: "DestinationPointer", Value: bin(to)}, + {Key: "IsErrorHandler", Value: isErr}, + {Key: "Line", Value: bson.D{ + {Key: "$ID", Value: bin(name + "l")}, + {Key: "$Type", Value: "Microflows$BezierCurve"}, + {Key: "DestinationControlVector", Value: "-30;0"}, + {Key: "OriginControlVector", Value: "30;0"}, + }}, + {Key: "OriginConnectionIndex", Value: fromSide}, + {Key: "OriginPointer", Value: bin(from)}, + } +} + +func unit(objects []bson.D, flows []bson.D) bson.D { + objs := bson.A{int32(3)} + for _, o := range objects { + objs = append(objs, o) + } + fl := bson.A{int32(3)} + for _, f := range flows { + fl = append(fl, f) + } + return bson.D{ + {Key: "$ID", Value: bin("unit")}, + {Key: "$Type", Value: "Microflows$Microflow"}, + {Key: "Flows", Value: fl}, + {Key: "ObjectCollection", Value: bson.D{ + {Key: "$ID", Value: bin("oc")}, + {Key: "$Type", Value: "Microflows$MicroflowObjectCollection"}, + {Key: "Objects", Value: objs}, + }}, + } +} + +// fakeDeps serializes the way the codec does for the properties the splice +// reads, and records what was saved. +type fakeDeps struct{ saved []byte } + +func (d *fakeDeps) SerializeObject(o microflows.MicroflowObject) (bson.D, error) { + typ := "Microflows$ActionActivity" + if _, ok := o.(*microflows.ExclusiveMerge); ok { + typ = "Microflows$ExclusiveMerge" + } + p := o.GetPosition() + sz := objectSize(o) + return bson.D{ + {Key: "$ID", Value: primitive.Binary{Data: types.UUIDToBlob(string(o.GetID()))}}, + {Key: "$Type", Value: typ}, + {Key: "RelativeMiddlePoint", Value: fmt.Sprintf("%d;%d", p.X, p.Y)}, + {Key: "Size", Value: fmt.Sprintf("%d;%d", sz.X, sz.Y)}, + }, nil +} + +func (d *fakeDeps) SerializeSequenceFlow(f *microflows.SequenceFlow) (bson.D, error) { + return bson.D{ + {Key: "$ID", Value: primitive.Binary{Data: types.UUIDToBlob(string(f.ID))}}, + {Key: "$Type", Value: "Microflows$SequenceFlow"}, + {Key: "DestinationConnectionIndex", Value: int32(f.DestinationConnectionIndex)}, + {Key: "DestinationPointer", Value: primitive.Binary{Data: types.UUIDToBlob(string(f.DestinationID))}}, + {Key: "IsErrorHandler", Value: f.IsErrorHandler}, + {Key: "Line", Value: bson.D{ + {Key: "DestinationControlVector", Value: f.DestinationControlVector}, + {Key: "OriginControlVector", Value: f.OriginControlVector}, + }}, + {Key: "OriginConnectionIndex", Value: int32(f.OriginConnectionIndex)}, + {Key: "OriginPointer", Value: primitive.Binary{Data: types.UUIDToBlob(string(f.OriginID))}}, + }, nil +} + +func (d *fakeDeps) SerializeAnnotationFlow(f *microflows.AnnotationFlow) (bson.D, error) { + return bson.D{ + {Key: "$ID", Value: primitive.Binary{Data: types.UUIDToBlob(string(f.ID))}}, + {Key: "$Type", Value: "Microflows$AnnotationFlow"}, + {Key: "DestinationPointer", Value: primitive.Binary{Data: types.UUIDToBlob(string(f.DestinationID))}}, + {Key: "OriginPointer", Value: primitive.Binary{Data: types.UUIDToBlob(string(f.OriginID))}}, + }, nil +} + +func (d *fakeDeps) SaveUnit(_ string, contents []byte) error { + d.saved = contents + return nil +} + +func newMutator(t *testing.T, doc bson.D) (*Mutator, *fakeDeps) { + t.Helper() + raw, err := bson.Marshal(doc) + if err != nil { + t.Fatal(err) + } + var d bson.D + if err := bson.Unmarshal(raw, &d); err != nil { + t.Fatal(err) + } + deps := &fakeDeps{} + m, err := New(d, "unit", deps) + if err != nil { + t.Fatal(err) + } + return m, deps +} + +// oneActivity is a single-activity fragment in builder coordinates. +func oneActivity() *backend.MicroflowFragment { + id := model.ID(types.GenerateID()) + act := &microflows.ActionActivity{BaseActivity: microflows.BaseActivity{BaseMicroflowObject: microflows.BaseMicroflowObject{ + BaseElement: model.BaseElement{ID: id}, + Position: model.Point{X: 360, Y: 200}, + Size: model.Size{Width: 120, Height: 60}, + }}} + return &backend.MicroflowFragment{Objects: []microflows.MicroflowObject{act}, Entry: id, Exit: id} +} + +func line() []bson.D { + return []bson.D{ + obj("start", "Microflows$StartEvent", 100, 200), + obj("a", "Microflows$ActionActivity", 250, 200), + obj("b", "Microflows$ActionActivity", 420, 200), + obj("end", "Microflows$EndEvent", 600, 200), + } +} + +func lineFlows() []bson.D { + return []bson.D{ + flow("f1", "start", "a", 1, 3, false), + flow("f2", "a", "b", 1, 3, false), + flow("f3", "b", "end", 1, 3, false), + } +} + +// The control every splice test leans on: decoding a unit and encoding it +// again with nothing spliced gives back the stored bytes exactly. +func TestSplice_UntouchedUnitRoundTripsExactly(t *testing.T) { + raw, _ := bson.Marshal(unit(line(), lineFlows())) + m, _ := newMutator(t, unit(line(), lineFlows())) + out, err := m.Bytes() + if err != nil { + t.Fatal(err) + } + if !bytes.Equal(out, raw) { + t.Error("an unmodified unit did not round-trip byte for byte") + } +} + +func TestSplice_InsertAfterKeepsTheErrorHandlerFlow(t *testing.T) { + objs := append(line(), obj("handler", "Microflows$ActionActivity", 250, 350)) + flows := append(lineFlows(), flow("err", "a", "handler", 2, 0, true)) + m, _ := newMutator(t, unit(objs, flows)) + errBefore, _ := bson.Marshal(flow("err", "a", "handler", 2, 0, true)) + + frag := oneActivity() + if err := m.InsertAfter(model.ID(uid("a")), frag); err != nil { + t.Fatal(err) + } + g := m.graph() + for _, f := range g.flows { + switch f.id { + case uid("err"): + got, _ := bson.Marshal(f.doc) + if !bytes.Equal(got, errBefore) { + t.Error("the error-handler flow changed") + } + case uid("f2"): + if f.dest != string(frag.Entry) { + t.Error("the normal flow out of a was not rewired to the fragment") + } + } + } +} + +func TestSplice_VerticalFlowUsesTopAndBottom(t *testing.T) { + objs := []bson.D{ + obj("a", "Microflows$ActionActivity", 200, 100), + obj("b", "Microflows$ActionActivity", 200, 200), + obj("below", "Microflows$ActionActivity", 200, 300), + obj("beside", "Microflows$ActionActivity", 500, 100), + } + flows := []bson.D{flow("f", "a", "b", 2, 0, false), flow("g", "b", "below", 2, 0, false)} + m, _ := newMutator(t, unit(objs, flows)) + frag := oneActivity() + if err := m.InsertAfter(model.ID(uid("a")), frag); err != nil { + t.Fatal(err) + } + g := m.graph() + if p := g.nodes[uid("b")].pos; p.X != 200 || p.Y <= 200 { + t.Errorf("b should have moved down, is at %v", p) + } + if p := g.nodes[uid("beside")].pos; p.X != 500 || p.Y != 100 { + t.Errorf("an object above the cut moved to %v", p) + } + for _, f := range g.flows { + if f.id == uid("f") { + if idx, _ := flowEnd(f.doc, "Destination"); idx != sideTop { + t.Errorf("the rewired flow enters the fragment on side %d, want top", idx) + } + } + if f.origin == string(frag.Exit) { + if idx, _ := flowEnd(f.doc, "Origin"); idx != sideBottom { + t.Errorf("the new flow leaves the fragment on side %d, want bottom", idx) + } + } + } +} + +func TestSplice_Refusals(t *testing.T) { + loop := obj("loop", "Microflows$LoopedActivity", 420, 200) + loop = append(loop, bson.E{Key: "ObjectCollection", Value: bson.D{ + {Key: "$ID", Value: bin("loopoc")}, + {Key: "$Type", Value: "Microflows$MicroflowObjectCollection"}, + {Key: "Objects", Value: bson.A{int32(3), obj("inner", "Microflows$ActionActivity", 100, 60), obj("inner2", "Microflows$ActionActivity", 260, 60)}}, + }}) + withLoop := []bson.D{obj("start", "Microflows$StartEvent", 100, 200), obj("a", "Microflows$ActionActivity", 250, 200), loop} + loopFlows := []bson.D{flow("f1", "start", "a", 1, 3, false), flow("f2", "a", "loop", 1, 3, false), flow("fi", "inner", "inner2", 1, 3, false)} + + twoIn := append(line(), obj("c", "Microflows$ActionActivity", 250, 350)) + twoInFlows := append(lineFlows(), flow("f4", "c", "b", 0, 2, false)) + + withHandler := append(line(), obj("handler", "Microflows$ActionActivity", 250, 350)) + handlerFlows := append(lineFlows(), flow("err", "a", "handler", 2, 0, true)) + + cases := []struct { + name string + objs []bson.D + flows []bson.D + op func(m *Mutator) error + want string + }{ + {"insert inside a loop", withLoop, loopFlows, + func(m *Mutator) error { return m.InsertAfter(model.ID(uid("inner")), oneActivity()) }, "inside a loop"}, + {"drop inside a loop", withLoop, loopFlows, + func(m *Mutator) error { return m.Drop(model.ID(uid("inner"))) }, "inside a loop"}, + {"insert before a join", twoIn, twoInFlows, + func(m *Mutator) error { return m.InsertBefore(model.ID(uid("b")), oneActivity()) }, "2 flows enter"}, + {"insert after the end", line(), lineFlows(), + func(m *Mutator) error { return m.InsertAfter(model.ID(uid("end")), oneActivity()) }, "ends the flow"}, + {"drop an activity with an error handler", withHandler, handlerFlows, + func(m *Mutator) error { return m.Drop(model.ID(uid("a"))) }, "error handler"}, + {"replace an activity with an error handler", withHandler, handlerFlows, + func(m *Mutator) error { return m.Replace(model.ID(uid("a")), oneActivity()) }, "error handler"}, + {"drop the end event", line(), lineFlows(), + func(m *Mutator) error { return m.Drop(model.ID(uid("end"))) }, "cannot drop"}, + {"an empty fragment", line(), lineFlows(), + func(m *Mutator) error { return m.InsertAfter(model.ID(uid("a")), &backend.MicroflowFragment{}) }, "empty"}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + m, _ := newMutator(t, unit(tc.objs, tc.flows)) + before, _ := bson.Marshal(m.doc) + err := tc.op(m) + if err == nil || !strings.Contains(err.Error(), tc.want) { + t.Fatalf("want an error containing %q, got %v", tc.want, err) + } + after, _ := bson.Marshal(m.doc) + if !bytes.Equal(before, after) { + t.Error("a refused operation changed the document") + } + }) + } +} + +// CLAUDE.md rule 1: an element is never removed while anything still points +// at it. A pointer the splice does not know about (here, a made-up property +// on another object) is found by value, and the write is refused. +func TestSplice_DropRefusesADanglingReference(t *testing.T) { + objs := line() + objs[3] = append(objs[3], bson.E{Key: "SomePointer", Value: bin("a")}) + m, deps := newMutator(t, unit(objs, lineFlows())) + if err := m.Drop(model.ID(uid("a"))); err != nil { + t.Fatalf("drop: %v", err) + } + err := m.Save() + if err == nil || !strings.Contains(err.Error(), "still points at a removed element") { + t.Fatalf("want the dangling-reference refusal, got %v", err) + } + if deps.saved != nil { + t.Error("a unit with a dangling reference was saved") + } +} + +// Control for the test above: the same drop without the stray pointer saves. +func TestSplice_DropJoinsTheFlows(t *testing.T) { + m, deps := newMutator(t, unit(line(), lineFlows())) + if err := m.Drop(model.ID(uid("a"))); err != nil { + t.Fatalf("drop: %v", err) + } + if err := m.Save(); err != nil { + t.Fatalf("save: %v", err) + } + if bytes.Contains(deps.saved, types.UUIDToBlob(uid("a"))) { + t.Error("the dropped activity's $ID is still in the unit") + } + g := m.graph() + for _, f := range g.flows { + if f.id == uid("f1") && f.dest != uid("b") { + t.Errorf("the flow into the dropped activity now enters %s, want b", f.dest) + } + if f.id == uid("f2") { + t.Error("the flow out of the dropped activity is still there") + } + } +} + +// A loop's body flows are stored in the unit's Flows list, not in the loop. +// Dropping or replacing the loop takes them with it; left behind, they point at +// the removed body objects and Save refuses the unit (a Studio Pro-authored +// loop with two body activities, ACT_ConflictedWorkflowHelper_ApplyJumpTo in +// TestApp, hit exactly that). +func TestSplice_DropOrReplaceALoopTakesItsBodyFlows(t *testing.T) { + build := func() ([]bson.D, []bson.D) { + loop := obj("loop", "Microflows$LoopedActivity", 420, 200) + loop = append(loop, bson.E{Key: "ObjectCollection", Value: bson.D{ + {Key: "$ID", Value: bin("loopoc")}, + {Key: "$Type", Value: "Microflows$MicroflowObjectCollection"}, + {Key: "Objects", Value: bson.A{int32(3), obj("inner", "Microflows$ActionActivity", 100, 60), obj("inner2", "Microflows$ActionActivity", 260, 60)}}, + }}) + objs := []bson.D{ + obj("start", "Microflows$StartEvent", 100, 200), + obj("a", "Microflows$ActionActivity", 250, 200), + loop, + obj("end", "Microflows$EndEvent", 700, 200), + } + flows := []bson.D{ + flow("f1", "start", "a", 1, 3, false), + flow("f2", "a", "loop", 1, 3, false), + flow("fi", "inner", "inner2", 1, 3, false), + flow("f3", "loop", "end", 1, 3, false), + } + return objs, flows + } + ops := map[string]func(m *Mutator) error{ + "drop": func(m *Mutator) error { return m.Drop(model.ID(uid("loop"))) }, + "replace": func(m *Mutator) error { return m.Replace(model.ID(uid("loop")), oneActivity()) }, + } + for name, op := range ops { + t.Run(name, func(t *testing.T) { + objs, flows := build() + m, deps := newMutator(t, unit(objs, flows)) + if err := op(m); err != nil { + t.Fatalf("%s: %v", name, err) + } + if err := m.Save(); err != nil { + t.Fatalf("save: %v", err) + } + for _, gone := range []string{"loop", "inner", "inner2", "fi"} { + if bytes.Contains(deps.saved, types.UUIDToBlob(uid(gone))) { + t.Errorf("%s is still in the unit", gone) + } + } + }) + } +} diff --git a/mdl/backend/microflow_mutation.go b/mdl/backend/microflow_mutation.go new file mode 100644 index 0000000000..2737d66c93 --- /dev/null +++ b/mdl/backend/microflow_mutation.go @@ -0,0 +1,49 @@ +// SPDX-License-Identifier: Apache-2.0 + +package backend + +import ( + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/sdk/microflows" +) + +// MicroflowFragment is what an `alter microflow` insert or replace splices in: +// the objects and flows a fragment of MDL builds to, written exactly as +// `create microflow` writes the same statements, and the two objects the +// surrounding flow connects to. Entry is where the flow into the fragment +// ends; Exit is where the flow out of it starts. For a one-activity fragment +// they are the same object. +// +// Positions are the builder's own; the mutator translates the fragment into +// place (plan item 4.2c) before writing it. +type MicroflowFragment struct { + Objects []microflows.MicroflowObject + Flows []*microflows.SequenceFlow + AnnotationFlows []*microflows.AnnotationFlow + Entry, Exit model.ID +} + +// MicroflowMutator splices into one stored microflow or nanoflow (ADR-0012 +// decision 3). Targets are activity IDs, already resolved from their content +// address by mfmutator.Resolve. Every operation edits the stored document in +// place: untouched elements stay byte-identical, and nothing is rebuilt. +// Call Save to persist. +type MicroflowMutator interface { + // InsertAfter splices frag onto the one flow that leaves target. + InsertAfter(target model.ID, frag *MicroflowFragment) error + // InsertBefore splices frag onto the one flow that enters target. + InsertBefore(target model.ID, frag *MicroflowFragment) error + // Replace puts frag in target's place and removes target. + Replace(target model.ID, frag *MicroflowFragment) error + // Drop removes target and joins its incoming flows to its successor. + Drop(target model.ID) error + // Save writes the patched unit. + Save() error +} + +// MicroflowMutationBackend opens a microflow or nanoflow for splicing. +type MicroflowMutationBackend interface { + // OpenMicroflowForMutation loads a Microflows$Microflow or + // Microflows$Nanoflow unit and returns a mutator over its stored form. + OpenMicroflowForMutation(unitID model.ID) (MicroflowMutator, error) +} diff --git a/mdl/backend/mock/backend.go b/mdl/backend/mock/backend.go index b5fa2477a4..cef40cf4a4 100644 --- a/mdl/backend/mock/backend.go +++ b/mdl/backend/mock/backend.go @@ -341,6 +341,9 @@ type MockBackend struct { // WorkflowMutationBackend OpenWorkflowForMutationFunc func(unitID model.ID) (backend.WorkflowMutator, error) + // MicroflowMutationBackend + OpenMicroflowForMutationFunc func(unitID model.ID) (backend.MicroflowMutator, error) + // WidgetSerializationBackend // WidgetBuilderBackend diff --git a/mdl/backend/mock/mock_mutation.go b/mdl/backend/mock/mock_mutation.go index f9d9f53c61..a0ee642bca 100644 --- a/mdl/backend/mock/mock_mutation.go +++ b/mdl/backend/mock/mock_mutation.go @@ -32,6 +32,17 @@ func (m *MockBackend) OpenWorkflowForMutation(unitID model.ID) (backend.Workflow return nil, fmt.Errorf("MockBackend.OpenWorkflowForMutation not configured") } +// --------------------------------------------------------------------------- +// MicroflowMutationBackend +// --------------------------------------------------------------------------- + +func (m *MockBackend) OpenMicroflowForMutation(unitID model.ID) (backend.MicroflowMutator, error) { + if m.OpenMicroflowForMutationFunc != nil { + return m.OpenMicroflowForMutationFunc(unitID) + } + return nil, fmt.Errorf("MockBackend.OpenMicroflowForMutation not configured") +} + // --------------------------------------------------------------------------- // WidgetSerializationBackend // --------------------------------------------------------------------------- diff --git a/mdl/backend/modelsdk/microflow_mutator_write.go b/mdl/backend/modelsdk/microflow_mutator_write.go new file mode 100644 index 0000000000..21f8a28104 --- /dev/null +++ b/mdl/backend/modelsdk/microflow_mutator_write.go @@ -0,0 +1,91 @@ +// SPDX-License-Identifier: Apache-2.0 + +package modelsdkbackend + +import ( + "fmt" + + "go.mongodb.org/mongo-driver/bson" + + "github.com/mendixlabs/mxcli/mdl/backend" + "github.com/mendixlabs/mxcli/mdl/backend/mfmutator" + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/modelsdk/codec" + "github.com/mendixlabs/mxcli/modelsdk/element" + genMf "github.com/mendixlabs/mxcli/modelsdk/gen/microflows" + "github.com/mendixlabs/mxcli/sdk/microflows" +) + +// OpenMicroflowForMutation loads a microflow or nanoflow unit and returns the +// shared graph splice (mfmutator) over its stored bytes. New objects and flows +// are encoded with the same converters `create microflow` uses, so a fragment +// is written exactly as the same statements would be in a create; the patched +// unit is written through the reconciling writer as a patch. +func (b *Backend) OpenMicroflowForMutation(unitID model.ID) (backend.MicroflowMutator, error) { + if b.writer == nil { + return nil, fmt.Errorf("OpenMicroflowForMutation: not connected for writing") + } + raw, err := b.reader.GetRawUnitBytes(string(unitID)) + if err != nil { + return nil, fmt.Errorf("OpenMicroflowForMutation: load unit: %w", err) + } + var d bson.D + if err := bson.Unmarshal(raw, &d); err != nil { + return nil, fmt.Errorf("OpenMicroflowForMutation: unmarshal: %w", err) + } + return mfmutator.New(d, unitID, codecMicroflowDeps{b: b}) +} + +// codecMicroflowDeps implements mfmutator.Deps for the modelsdk (codec) backend. +type codecMicroflowDeps struct{ b *Backend } + +var _ mfmutator.Deps = codecMicroflowDeps{} + +func (d codecMicroflowDeps) SerializeObject(obj microflows.MicroflowObject) (bson.D, error) { + el := microflowObjectToGen(obj) + if el == nil { + return nil, nil + } + assignFlowObjectIDs(el) + return encodeFlowElementToD(el) +} + +func (d codecMicroflowDeps) SerializeSequenceFlow(f *microflows.SequenceFlow) (bson.D, error) { + el := sequenceFlowToGen(f, d.b.majorVersion()) + assignID(el) + if sf, ok := el.(*genMf.SequenceFlow); ok { + for _, cv := range sf.CaseValuesItems() { + assignID(cv) + } + assignID(sf.Line()) + } + return encodeFlowElementToD(el) +} + +func (d codecMicroflowDeps) SerializeAnnotationFlow(f *microflows.AnnotationFlow) (bson.D, error) { + el := annotationFlowToGen(f, d.b.majorVersion()) + assignID(el) + if af, ok := el.(*genMf.AnnotationFlow); ok { + assignID(af.Line()) + } + return encodeFlowElementToD(el) +} + +// SaveUnit writes the spliced unit as a patch: the reconciling writer elides +// an unchanged unit and guards storage GUIDs as for every write, but does not +// re-pair element $IDs the splice deliberately kept (canon.ContentsOwnElementIDs). +func (d codecMicroflowDeps) SaveUnit(unitID string, contents []byte) error { + return d.b.writer.UpdateRawUnitPatch(unitID, contents) +} + +func encodeFlowElementToD(el element.Element) (bson.D, error) { + raw, err := (&codec.Encoder{}).Encode(el) + if err != nil { + return nil, err + } + var out bson.D + if err := bson.Unmarshal(raw, &out); err != nil { + return nil, err + } + return out, nil +} diff --git a/mdl/backend/modelsdk/security_markers_test.go b/mdl/backend/modelsdk/security_markers_test.go new file mode 100644 index 0000000000..7bba02aa10 --- /dev/null +++ b/mdl/backend/modelsdk/security_markers_test.go @@ -0,0 +1,67 @@ +// SPDX-License-Identifier: Apache-2.0 + +package modelsdkbackend + +import ( + "testing" + + "go.mongodb.org/mongo-driver/bson" +) + +// Studio Pro writes Security$ProjectSecurity's UserRoles and DemoUsers lists +// with typed-array marker 2 (measured: PedApp 11.13.0, TestApp 11.14.0 and the +// expr-checker fixture all agree). The encoder's default is 3, so rewriting the +// document for `create or modify user role` with an unchanged definition, which +// is what running `describe user role` output does, changed the marker and +// wrote the unit (#731). The control re-reads the untouched document. +func TestProjectSecurity_UserRoleAndDemoUserListMarkers(t *testing.T) { + proj := copyFixture(t) + b := New() + if err := b.Connect(proj); err != nil { + t.Fatalf("connect: %v", err) + } + t.Cleanup(func() { _ = b.Disconnect() }) + + ps, err := b.GetProjectSecurity() + if err != nil { + t.Fatalf("GetProjectSecurity: %v", err) + } + if len(ps.UserRoles) == 0 { + t.Fatal("fixture has no user roles") + } + markers := func() (int32, int32) { + raw, err := b.GetRawUnitBytes(ps.ID) + if err != nil { + t.Fatalf("GetRawUnitBytes: %v", err) + } + var d bson.D + if err := bson.Unmarshal(raw, &d); err != nil { + t.Fatalf("unmarshal: %v", err) + } + return markerOf(t, d, "UserRoles"), markerOf(t, d, "DemoUsers") + } + + // Control: the Studio Pro-authored document carries marker 2 for both. + if u, d := markers(); u != 2 || d != 2 { + t.Fatalf("fixture markers UserRoles=%d DemoUsers=%d; want 2 and 2 — the control does not hold", u, d) + } + + ur := ps.UserRoles[0] + if err := b.AddUserRole(ps.ID, ur.Name, append([]string{}, ur.ModuleRoles...), !ur.ManageAllRoles); err != nil { + t.Fatalf("AddUserRole: %v", err) + } + if u, d := markers(); u != 2 || d != 2 { + t.Errorf("after rewriting user role %s: markers UserRoles=%d DemoUsers=%d; want 2 and 2", ur.Name, u, d) + } + + if len(ps.DemoUsers) == 0 { + t.Fatal("fixture has no demo users") + } + du := ps.DemoUsers[0] + if err := b.AddDemoUser(ps.ID, du.UserName, du.Password+"x", du.Entity, append([]string{}, du.UserRoles...)); err != nil { + t.Fatalf("AddDemoUser: %v", err) + } + if u, d := markers(); u != 2 || d != 2 { + t.Errorf("after rewriting demo user %s: markers UserRoles=%d DemoUsers=%d; want 2 and 2", du.UserName, u, d) + } +} diff --git a/mdl/backend/modelsdk/security_write.go b/mdl/backend/modelsdk/security_write.go index 522520d260..102cee7c6a 100644 --- a/mdl/backend/modelsdk/security_write.go +++ b/mdl/backend/modelsdk/security_write.go @@ -12,6 +12,17 @@ import ( genSec "github.com/mendixlabs/mxcli/modelsdk/gen/security" ) +// Studio Pro writes Security$ProjectSecurity's UserRoles and DemoUsers lists +// with typed-array marker 2, not the encoder's default 3 (measured on PedApp +// 11.13.0, TestApp 11.14.0 and the expr-checker fixture: every document +// agrees). Without these, rewriting a user role or demo user with an unchanged +// definition changed the marker, so running `describe user role` output wrote +// the unit (#731). +func init() { + codec.RegisterPropertyListMarker("Security$ProjectSecurity", "UserRoles", 2) + codec.RegisterPropertyListMarker("Security$ProjectSecurity", "DemoUsers", 2) +} + // loadModuleSecurityGen decodes a Security$ModuleSecurity unit by ID. func (b *Backend) loadModuleSecurityGen(unitID model.ID) (*genSec.ModuleSecurity, error) { raw, err := b.reader.GetRawUnitBytes(string(unitID)) diff --git a/mdl/backend/modelsdk/unimplemented_gen.go b/mdl/backend/modelsdk/unimplemented_gen.go index 232eed4a25..b7b3f4f78a 100644 --- a/mdl/backend/modelsdk/unimplemented_gen.go +++ b/mdl/backend/modelsdk/unimplemented_gen.go @@ -891,6 +891,11 @@ func (unimplemented) MoveViewEntitySourceDocument(_ string, _ model.ID, _ string return errUnimplemented("MoveViewEntitySourceDocument") } +func (unimplemented) OpenMicroflowForMutation(_ model.ID) (backend.MicroflowMutator, error) { + var r0 backend.MicroflowMutator + return r0, errUnimplemented("OpenMicroflowForMutation") +} + func (unimplemented) OpenPageForMutation(_ model.ID) (backend.PageMutator, error) { var r0 backend.PageMutator return r0, errUnimplemented("OpenPageForMutation") @@ -1118,6 +1123,10 @@ func (unimplemented) UpdateJsonStructure(_ *types.JsonStructure) error { return errUnimplemented("UpdateJsonStructure") } +func (unimplemented) UpdateLayout(_ *pages.Layout) error { + return errUnimplemented("UpdateLayout") +} + func (unimplemented) UpdateMenuDocument(_ *types.MenuDocument) error { return errUnimplemented("UpdateMenuDocument") } @@ -1224,3 +1233,8 @@ func (unimplemented) WriteJavaScriptSourceFile(_ string, _ string, _ string, _ [ func (unimplemented) WriteJavaSourceFile(_ string, _ string, _ string, _ []*types.JavaActionParameter, _ types.CodeActionReturnType, _ []string, _ string) error { return errUnimplemented("WriteJavaSourceFile") } + +func (unimplemented) WriteViewEntitySourceDocument(_ model.ID, _ string, _ string, _ string, _ string) (model.ID, error) { + var r0 model.ID + return r0, errUnimplemented("WriteViewEntitySourceDocument") +} diff --git a/mdl/deprecation/deprecation.go b/mdl/deprecation/deprecation.go index 50d5c47eb8..853c6a9fcf 100644 --- a/mdl/deprecation/deprecation.go +++ b/mdl/deprecation/deprecation.go @@ -62,20 +62,39 @@ type Entry struct { CanonicalExample string } -// Rewrite replaces one keyword token of the deprecated form with another. It is -// the only shape the seeded entries need; a richer one is added with the first -// entry that needs it. +// Rewrite is the mechanical rewrite from the deprecated form to the canonical +// one. It is either a keyword swap (Token and Replacement) or, where the two +// forms differ in shape rather than in one word, a structural rewrite that +// Structural names. A structural rewrite is implemented against the parse tree, +// never as text substitution, and its correctness rests on the same test as a +// swap: Example and CanonicalExample must build the same statements. type Rewrite struct { // Token is the keyword to replace, lower-case. Token string // Replacement is the keyword written in its place, lower-case. Replacement string + // Structural describes a rewrite that is not a keyword swap, e.g. + // "call form to statement form". Empty for a keyword swap. + Structural string } +// IsZero reports whether r is empty: the entry has no mechanical rewrite, and +// `fmt --upgrade` reports its uses instead of guessing at one. A structural +// rewrite is computed per use by the visitor that records it (ast.Fix), and a +// use it cannot rewrite is reported the same way. +func (r Rewrite) IsZero() bool { return r.Token == "" && r.Replacement == "" && r.Structural == "" } + // Codes of the registered entries, for the visitor to record. const ( CreateOrReplace = "MDL-DEPR001" Show = "MDL-DEPR002" + // ListOperationFunctionForm is `$x = head($L)` and the other list + // operations written as calls; find and contains are excluded, because the + // call form clashes with the string functions (see mdl/visitor, MDL-V1-LIST). + ListOperationFunctionForm = "MDL-DEPR003" + // AggregateFunctionForm is `$n = count($L)` and the other aggregates + // written as calls. + AggregateFunctionForm = "MDL-DEPR004" ) // entries is the registry. Append only: a code is never reused or renumbered, @@ -87,10 +106,10 @@ var entries = []Entry{ Canonical: "create or modify …", Rewrite: Rewrite{Token: "replace", Replacement: "modify"}, RemovedIn: 2, - Note: "Not reported where `or replace` means something else today: " + - "`create or replace view entity` (drops and recreates), " + - "`create or replace translations` (replaces the whole set), and " + - "`create or replace user role` / `demo user` (the `replace` is ignored).", + Note: "Not reported for `create or replace translations`, which replaces the " + + "whole set. Under mdl 0 (no header) it is not reported for a view entity " + + "(drops and recreates) or a user role / demo user (a plain create) either: " + + "those warn MDL-V1-REPLACE01/02 instead, and are aliases from `mdl 1;` on.", Example: "create or replace enumeration M.Color (Red 'Red');", CanonicalExample: "create or modify enumeration M.Color (Red 'Red');", }, @@ -108,6 +127,31 @@ var entries = []Entry{ Example: "show entities in M;", CanonicalExample: "list entities in M;", }, + { + Code: ListOperationFunctionForm, + Old: "$x = <operation>($List, …)", + Canonical: "$x = <operation> $List …", + Rewrite: Rewrite{Structural: "call form to statement form: head/tail $L; filter/find $L by Member = v " + + "(when the condition has that shape) or where <expr>; sort $L by …; union/intersect $A with $B; " + + "subtract($A, $B) -> subtract $B from $A; equals $A and $B; range($L, o, n) -> range $L offset o limit n"}, + RemovedIn: 2, + Note: "A list operation is one Studio Pro activity whose operand is a variable, so the statement form " + + "cannot nest. find(…) and contains(…) are not reported here: the call form is also the string " + + "function, so they are version-gated instead (MDL-V1-LIST).", + Example: "create microflow M.F ($L: List of M.E) begin $H = head($L); end;", + CanonicalExample: "create microflow M.F ($L: List of M.E) begin $H = head $L; end;", + }, + { + Code: AggregateFunctionForm, + Old: "$n = <function>($List, …)", + Canonical: "$n = <function> $List …", + Rewrite: Rewrite{Structural: "call form to statement form: count $L; sum|average|minimum|maximum " + + "$L by Attr (for $L.Attr) or of <expr>; all|any $L where <expr>; reduce $L from <initial> as <type> using <expr>"}, + RemovedIn: 2, + Note: "An aggregate is one Studio Pro Aggregate list activity whose operand is a variable.", + Example: "create microflow M.F ($L: List of M.E) begin $N = count($L); end;", + CanonicalExample: "create microflow M.F ($L: List of M.E) begin $N = count $L; end;", + }, } // All returns every registered entry, in code order. diff --git a/mdl/deprecation/deprecation_test.go b/mdl/deprecation/deprecation_test.go index 8084f5c439..de608703f2 100644 --- a/mdl/deprecation/deprecation_test.go +++ b/mdl/deprecation/deprecation_test.go @@ -27,8 +27,11 @@ func TestRegistryEntriesAreWellFormed(t *testing.T) { if got, ok := Lookup(e.Code); !ok || got.Code != e.Code { t.Errorf("Lookup(%q) = %v, %v", e.Code, got.Code, ok) } - if e.Old == "" || e.Canonical == "" || e.Rewrite.Token == "" || e.Rewrite.Replacement == "" || - e.Example == "" || e.CanonicalExample == "" { + swap := e.Rewrite.Token != "" && e.Rewrite.Replacement != "" + if swap == (e.Rewrite.Structural != "") { + t.Errorf("%s: a rewrite is either a keyword swap or structural, exactly one: %+v", e.Code, e.Rewrite) + } + if e.Old == "" || e.Canonical == "" || e.Example == "" || e.CanonicalExample == "" { t.Errorf("%s is incomplete: %+v", e.Code, e) } // ADR-0011: an alias warns under the version that deprecates it (1) diff --git a/mdl/executor/cmd_agenteditor_agents.go b/mdl/executor/cmd_agenteditor_agents.go index 008c51e0a5..c2766868c4 100644 --- a/mdl/executor/cmd_agenteditor_agents.go +++ b/mdl/executor/cmd_agenteditor_agents.go @@ -85,7 +85,7 @@ func describeAgentEditorAgent(ctx *ExecContext, name ast.QualifiedName) error { fmt.Fprintf(ctx.Output, "/**\n * %s\n */\n", a.Documentation) } - fmt.Fprintf(ctx.Output, "create agent %s%s (\n", qualifiedName, describeFolderClause(ctx, a.ContainerID)) + fmt.Fprintf(ctx.Output, "create or modify agent %s%s (\n", qualifiedName, describeFolderClause(ctx, a.ContainerID)) // Build property lines. User-set properties are emitted in a stable // order; empty values are omitted. @@ -163,7 +163,6 @@ func describeAgentEditorAgent(ctx *ExecContext, name ast.QualifiedName) error { } else { fmt.Fprintln(ctx.Output, ");") } - fmt.Fprintln(ctx.Output, "/") return nil } diff --git a/mdl/executor/cmd_agenteditor_kbs.go b/mdl/executor/cmd_agenteditor_kbs.go index f2c9bbf2f8..26eeacd8f8 100644 --- a/mdl/executor/cmd_agenteditor_kbs.go +++ b/mdl/executor/cmd_agenteditor_kbs.go @@ -83,7 +83,7 @@ func describeAgentEditorKnowledgeBase(ctx *ExecContext, name ast.QualifiedName) fmt.Fprintf(ctx.Output, "/**\n * %s\n */\n", k.Documentation) } - fmt.Fprintf(ctx.Output, "create knowledge base %s%s (\n", qualifiedName, describeFolderClause(ctx, k.ContainerID)) + fmt.Fprintf(ctx.Output, "create or modify knowledge base %s%s (\n", qualifiedName, describeFolderClause(ctx, k.ContainerID)) var lines []string if k.Provider != "" { @@ -120,7 +120,6 @@ func describeAgentEditorKnowledgeBase(ctx *ExecContext, name ast.QualifiedName) } fmt.Fprintln(ctx.Output, ");") - fmt.Fprintln(ctx.Output, "/") return nil } diff --git a/mdl/executor/cmd_agenteditor_mcpservices.go b/mdl/executor/cmd_agenteditor_mcpservices.go index eb9b159c13..3a8d3ad0c7 100644 --- a/mdl/executor/cmd_agenteditor_mcpservices.go +++ b/mdl/executor/cmd_agenteditor_mcpservices.go @@ -79,7 +79,7 @@ func describeAgentEditorConsumedMCPService(ctx *ExecContext, name ast.QualifiedN fmt.Fprintf(ctx.Output, "/**\n * %s\n */\n", c.Documentation) } - fmt.Fprintf(ctx.Output, "create consumed mcp service %s%s (\n", qualifiedName, describeFolderClause(ctx, c.ContainerID)) + fmt.Fprintf(ctx.Output, "create or modify consumed mcp service %s%s (\n", qualifiedName, describeFolderClause(ctx, c.ContainerID)) var lines []string if c.ProtocolVersion != "" { @@ -104,7 +104,6 @@ func describeAgentEditorConsumedMCPService(ctx *ExecContext, name ast.QualifiedN } fmt.Fprintln(ctx.Output, ");") - fmt.Fprintln(ctx.Output, "/") return nil } diff --git a/mdl/executor/cmd_agenteditor_mock_test.go b/mdl/executor/cmd_agenteditor_mock_test.go index 2ac1af1372..0a273b8fe2 100644 --- a/mdl/executor/cmd_agenteditor_mock_test.go +++ b/mdl/executor/cmd_agenteditor_mock_test.go @@ -344,7 +344,8 @@ func TestDescribeAgentEditorModel_Mock(t *testing.T) { assertNoError(t, describeAgentEditorModel(ctx, ast.QualifiedName{Module: "M", Name: "GPT4"})) out := buf.String() - assertContainsStr(t, out, "create model") + assertContainsStr(t, out, "create or modify model") + assertTerminated(t, out) // #744 assertContainsStr(t, out, "Provider") assertContainsStr(t, out, "Key") } @@ -405,7 +406,8 @@ func TestDescribeAgentEditorAgent_Mock(t *testing.T) { assertNoError(t, describeAgentEditorAgent(ctx, ast.QualifiedName{Module: "M", Name: "MyAgent"})) out := buf.String() - assertContainsStr(t, out, "create agent") + assertContainsStr(t, out, "create or modify agent") + assertTerminated(t, out) // #744 assertContainsStr(t, out, "UsageType") assertContainsStr(t, out, "Model") } @@ -465,7 +467,8 @@ func TestDescribeAgentEditorKnowledgeBase_Mock(t *testing.T) { assertNoError(t, describeAgentEditorKnowledgeBase(ctx, ast.QualifiedName{Module: "M", Name: "MyKB"})) out := buf.String() - assertContainsStr(t, out, "create knowledge base") + assertContainsStr(t, out, "create or modify knowledge base") + assertTerminated(t, out) // #744 assertContainsStr(t, out, "Provider") } @@ -524,7 +527,8 @@ func TestDescribeAgentEditorConsumedMCPService_Mock(t *testing.T) { assertNoError(t, describeAgentEditorConsumedMCPService(ctx, ast.QualifiedName{Module: "M", Name: "MySvc"})) out := buf.String() - assertContainsStr(t, out, "create consumed mcp service") + assertContainsStr(t, out, "create or modify consumed mcp service") + assertTerminated(t, out) // #744 // ProtocolVersion with a date value must be quoted so the output is re-parseable (issue #435) assertContainsStr(t, out, "ProtocolVersion: '2025-03-26'") diff --git a/mdl/executor/cmd_agenteditor_models.go b/mdl/executor/cmd_agenteditor_models.go index 743937a0d3..071bdaece7 100644 --- a/mdl/executor/cmd_agenteditor_models.go +++ b/mdl/executor/cmd_agenteditor_models.go @@ -87,7 +87,7 @@ func describeAgentEditorModel(ctx *ExecContext, name ast.QualifiedName) error { fmt.Fprintf(ctx.Output, "/**\n * %s\n */\n", m.Documentation) } - fmt.Fprintf(ctx.Output, "create model %s%s (\n", qualifiedName, describeFolderClause(ctx, m.ContainerID)) + fmt.Fprintf(ctx.Output, "create or modify model %s%s (\n", qualifiedName, describeFolderClause(ctx, m.ContainerID)) // Emit properties in stable order. User-set properties (Provider, Key) // come first; Portal-populated metadata comes last and only if non-empty. @@ -127,7 +127,6 @@ func describeAgentEditorModel(ctx *ExecContext, name ast.QualifiedName) error { } fmt.Fprintln(ctx.Output, ");") - fmt.Fprintln(ctx.Output, "/") return nil } diff --git a/mdl/executor/cmd_alter_flow.go b/mdl/executor/cmd_alter_flow.go new file mode 100644 index 0000000000..d8aac43098 --- /dev/null +++ b/mdl/executor/cmd_alter_flow.go @@ -0,0 +1,504 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "fmt" + "regexp" + "sort" + "strings" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/backend" + "github.com/mendixlabs/mxcli/mdl/backend/mfmutator" + mdlerrors "github.com/mendixlabs/mxcli/mdl/errors" + "github.com/mendixlabs/mxcli/mdl/types" + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/sdk/microflows" +) + +// execAlterFlow handles `alter microflow|nanoflow Module.Name { … }`: a patch +// of the stored flow (ADR-0012 decision 3), never a rebuild. +// +// Every target is resolved against the flow AS STORED, before any operation +// runs, so an address means what `describe … with handles` showed: an +// ambiguity or a miss refuses the whole statement before anything changes, and +// a later operation cannot address what an earlier one inserted. +func execAlterFlow(ctx *ExecContext, s *ast.AlterFlowStmt) error { + if !ctx.Connected() { + return mdlerrors.NewNotConnected() + } + if !ctx.ConnectedForWrite() { + return mdlerrors.NewNotConnectedWrite() + } + a, err := loadAlterFlow(ctx, s) + if err != nil { + return err + } + + targets := make([]mfmutator.Candidate, len(s.Operations)) + for i, op := range s.Operations { + c, err := mfmutator.ResolveText(a.cands, op.Target) + if err != nil { + return mdlerrors.NewValidation(fmt.Sprintf("alter %s %s: %s %s: %v", s.Kind(), s.Name, op.Op, op.Target, err)) + } + targets[i] = c + } + + mut, err := ctx.Backend.OpenMicroflowForMutation(a.mf.ID) + if err != nil { + return mdlerrors.NewBackend("open "+s.Kind()+" for alter", err) + } + for i, op := range s.Operations { + target := targets[i] + fail := func(err error) error { + return mdlerrors.NewValidation(fmt.Sprintf("alter %s %s: %s %s: %v", s.Kind(), s.Name, op.Op, op.Target, err)) + } + if op.Op == ast.AlterFlowDrop { + if err := a.checkOutputUnused(target, nil); err != nil { + return fail(err) + } + if err := mut.Drop(target.ID); err != nil { + return fail(err) + } + a.noteRemoved(target, nil) + continue + } + frag, err := a.buildFragment(ctx, op.Body) + if err != nil { + return fail(err) + } + if err := a.checkFragmentScope(ctx, op, target, frag); err != nil { + return fail(err) + } + switch op.Op { + case ast.AlterFlowInsertAfter: + err = mut.InsertAfter(target.ID, frag) + case ast.AlterFlowInsertBefore: + err = mut.InsertBefore(target.ID, frag) + case ast.AlterFlowReplace: + if err = a.checkOutputUnused(target, frag); err == nil { + err = mut.Replace(target.ID, frag) + } + default: + err = fmt.Errorf("unknown operation") + } + if err != nil { + return fail(err) + } + if op.Op == ast.AlterFlowReplace { + a.noteRemoved(target, frag) + } + a.noteFragment(ctx, frag) + } + if err := mut.Save(); err != nil { + return mdlerrors.NewBackend("save altered "+s.Kind(), err) + } + fmt.Fprintf(ctx.Output, "Altered %s %s\n", s.Kind(), s.Name) + return nil +} + +// alterFlowContext is what the operations of one statement share: the stored +// flow (a nanoflow wrapped as a microflow, the way describe renders one), its +// addressable activities, and the name maps rendering needs. +type alterFlowContext struct { + stmt *ast.AlterFlowStmt + mf *microflows.Microflow + cands []mfmutator.Candidate + entityNames map[model.ID]string + microflowNames map[model.ID]string + + // What the statement's earlier operations did to the variables, so a later + // one is checked against the flow as it will be written, not as stored: + // the variables their fragments declare, the ones their fragments read + // (with who reads them), and the stored outputs they took away. + declaredByOps map[string]bool + readByOps map[string][]string + removedByOps map[string]bool +} + +// noteRemoved records that target's output is gone, unless the fragment that +// replaces it declares it again. +func (a *alterFlowContext) noteRemoved(target mfmutator.Candidate, replacement *backend.MicroflowFragment) { + v := target.OutputVariable + if v == "" || fragmentDeclares(replacement, v) { + return + } + a.removedByOps[v] = true +} + +// noteFragment records what an inserted or replacing fragment declares and +// reads. +func (a *alterFlowContext) noteFragment(ctx *ExecContext, frag *backend.MicroflowFragment) { + for _, obj := range frag.Objects { + if act, ok := obj.(*microflows.ActionActivity); ok { + if v := mfmutator.OutputVariable(act.Action); v != "" { + a.declaredByOps[v] = true + } + } + text := formatActivity(ctx, obj, a.entityNames, a.microflowNames) + for _, m := range variableRef.FindAllStringSubmatch(text, -1) { + a.readByOps[m[1]] = append(a.readByOps[m[1]], text) + } + } +} + +func fragmentDeclares(frag *backend.MicroflowFragment, v string) bool { + if frag == nil { + return false + } + for _, obj := range frag.Objects { + if act, ok := obj.(*microflows.ActionActivity); ok && mfmutator.OutputVariable(act.Action) == v { + return true + } + } + return false +} + +func loadAlterFlow(ctx *ExecContext, s *ast.AlterFlowStmt) (*alterFlowContext, error) { + h, err := getHierarchy(ctx) + if err != nil { + return nil, mdlerrors.NewBackend("build hierarchy", err) + } + a := &alterFlowContext{stmt: s, entityNames: getEntityNames(ctx, h), + declaredByOps: map[string]bool{}, readByOps: map[string][]string{}, removedByOps: map[string]bool{}} + // A copy: nanoflow names are added below, and the cached map is shared. + a.microflowNames = map[model.ID]string{} + for id, n := range getMicroflowNames(ctx, h) { + a.microflowNames[id] = n + } + inModule := func(container model.ID, name string) bool { + return h.GetModuleName(h.FindModuleID(container)) == s.Name.Module && name == s.Name.Name + } + if s.Nanoflow { + nfs, err := ctx.Backend.ListNanoflows() + if err != nil { + return nil, mdlerrors.NewBackend("list nanoflows", err) + } + for _, nf := range nfs { + a.microflowNames[nf.ID] = h.GetQualifiedName(nf.ContainerID, nf.Name) + } + nf, ok := pickLive(nfs, + func(nf *microflows.Nanoflow) bool { return inModule(nf.ContainerID, nf.Name) }, + func(nf *microflows.Nanoflow) bool { return nf.Excluded }) + if !ok { + return nil, mdlerrors.NewNotFound("nanoflow", s.Name.String()) + } + a.mf = &microflows.Microflow{ + BaseElement: nf.BaseElement, + ContainerID: nf.ContainerID, + Name: nf.Name, + Parameters: nf.Parameters, + ReturnType: nf.ReturnType, + ReturnVariableName: nf.ReturnVariableName, + ObjectCollection: nf.ObjectCollection, + } + } else { + mfs, err := ctx.Backend.ListMicroflows() + if err != nil { + return nil, mdlerrors.NewBackend("list microflows", err) + } + mf, ok := pickLive(mfs, + func(mf *microflows.Microflow) bool { return inModule(mf.ContainerID, mf.Name) }, + func(mf *microflows.Microflow) bool { return mf.Excluded }) + if !ok { + return nil, mdlerrors.NewNotFound("microflow", s.Name.String()) + } + a.mf = mf + } + if a.mf.ObjectCollection == nil { + return nil, mdlerrors.NewValidation(fmt.Sprintf("%s %s has no flow to alter", s.Kind(), s.Name)) + } + a.cands, _, _, _ = microflowTargets(ctx, a.mf, a.entityNames, a.microflowNames) + return a, nil +} + +// buildFragment builds a fragment's statements with the builder `create +// microflow` uses, seeded with the variables the stored flow declares, and +// cuts it out of the start and end events the builder wraps it in. +func (a *alterFlowContext) buildFragment(ctx *ExecContext, body []ast.MicroflowStatement) (*backend.MicroflowFragment, error) { + if len(body) == 0 { + return nil, fmt.Errorf("the fragment is empty; use drop to remove an activity") + } + varTypes, declared := a.storedVariables(ctx) + hierarchy, _ := getHierarchy(ctx) + restServices, _ := loadRestServices(ctx) + fb := &flowBuilder{ + textLang: authoringLanguage(ctx), + posX: 200, + posY: 200, + baseY: 200, + spacing: HorizontalSpacing, + varTypes: varTypes, + declaredVars: declared, + measurer: &layoutMeasurer{varTypes: varTypes}, + backend: ctx.Backend, + hierarchy: hierarchy, + restServices: restServices, + isNanoflow: a.stmt.Nanoflow, + } + oc := fb.buildFlowGraph(body, nil) + if errs := fb.GetErrors(); len(errs) > 0 { + return nil, fmt.Errorf("the fragment has errors:\n - %s", strings.Join(errs, "\n - ")) + } + if fb.endsWithReturn { + return nil, fmt.Errorf("the fragment ends the flow with a return, so nothing would lead on to the rest of it; " + + "a return inside an inserted fragment is not supported yet") + } + return cutFragment(oc) +} + +// cutFragment removes the builder's start event and final end event, and says +// where the fragment is entered and left. Several paths reaching the end (an +// if without a merge before it, an error handler that rejoins at the end) are +// joined by a merge, which becomes the exit. +func cutFragment(oc *microflows.MicroflowObjectCollection) (*backend.MicroflowFragment, error) { + var start, end microflows.MicroflowObject + ends := 0 + for _, obj := range oc.Objects { + switch obj.(type) { + case *microflows.StartEvent: + start = obj + case *microflows.EndEvent: + ends++ + end = obj + } + } + if start == nil || end == nil { + return nil, fmt.Errorf("the fragment does not continue: its last statement ends the flow, so nothing would lead on to the rest of it") + } + if ends > 1 { + return nil, fmt.Errorf("the fragment returns; a return inside an inserted fragment is not supported yet") + } + frag := &backend.MicroflowFragment{} + var intoEnd []*microflows.SequenceFlow + for _, f := range oc.Flows { + switch { + case f.OriginID == start.GetID(): + if frag.Entry != "" { + return nil, fmt.Errorf("the fragment starts with more than one flow") + } + frag.Entry = f.DestinationID + case f.DestinationID == end.GetID(): + intoEnd = append(intoEnd, f) + default: + frag.Flows = append(frag.Flows, f) + } + } + for _, obj := range oc.Objects { + if obj != start && obj != end { + frag.Objects = append(frag.Objects, obj) + } + } + frag.AnnotationFlows = oc.AnnotationFlows + switch { + case frag.Entry == "" || frag.Entry == end.GetID() || len(frag.Objects) == 0: + return nil, fmt.Errorf("the fragment builds no activity") + case len(intoEnd) == 0: + return nil, fmt.Errorf("no path through the fragment leads on to the rest of the flow") + case len(intoEnd) == 1: + frag.Exit = intoEnd[0].OriginID + default: + p := end.GetPosition() + merge := &microflows.ExclusiveMerge{BaseMicroflowObject: microflows.BaseMicroflowObject{ + BaseElement: model.BaseElement{ID: model.ID(types.GenerateID())}, + Position: p, + Size: model.Size{Width: MergeSize, Height: MergeSize}, + }} + for _, f := range intoEnd { + f.DestinationID = merge.ID + frag.Flows = append(frag.Flows, f) + } + frag.Objects = append(frag.Objects, merge) + frag.Exit = merge.ID + } + return frag, nil +} + +// storedVariables returns the variables the stored flow declares, in the two +// maps the builder keeps: entity-typed ones with their entity (a change or a +// member access resolves attributes through it), and the rest as declared. +func (a *alterFlowContext) storedVariables(ctx *ExecContext) (varTypes, declared map[string]string) { + varTypes, declared = map[string]string{}, map[string]string{} + add := func(name string, dt microflows.DataType) { + if name == "" { + return + } + t := "Unknown" + if dt != nil { + t = formatMicroflowDataType(ctx, dt, a.entityNames) + } + switch dt.(type) { + case *microflows.ObjectType, *microflows.ListType: + varTypes[name] = t + default: + declared[name] = t + } + } + for _, p := range a.mf.Parameters { + add(p.Name, p.Type) + } + for _, c := range a.cands { + act, ok := c.Object.(*microflows.ActionActivity) + if !ok || c.OutputVariable == "" { + continue + } + switch x := act.Action.(type) { + case *microflows.CreateVariableAction: + add(c.OutputVariable, x.DataType) + case *microflows.CreateObjectAction: + if x.EntityQualifiedName != "" { + varTypes[c.OutputVariable] = x.EntityQualifiedName + } else if n, ok := a.entityNames[x.EntityID]; ok { + varTypes[c.OutputVariable] = n + } else { + declared[c.OutputVariable] = "Object" + } + default: + declared[c.OutputVariable] = "Unknown" + } + } + return varTypes, declared +} + +// systemVariables are in scope everywhere they exist at all; the platform +// reports a misuse (a $latestError outside an error handler) itself. +var systemVariables = map[string]bool{ + "currentUser": true, "currentSession": true, "currentObject": true, "currentDeviceType": true, + "latestError": true, "latestHttpResponse": true, "latestSoapFault": true, +} + +var variableRef = regexp.MustCompile(`\$([A-Za-z_][A-Za-z0-9_]*)`) + +// checkFragmentScope is plan item 4.2d: the fragment is checked in the scope +// of its insertion point. A variable it declares that the flow already has is +// an error (it would shadow or clash with the stored one); a variable it uses +// that is not declared upstream of where it goes, nor by the fragment itself, +// is an error too, since the fragment would read something that does not exist +// yet on that path. +func (a *alterFlowContext) checkFragmentScope(ctx *ExecContext, op *ast.AlterFlowOperation, target mfmutator.Candidate, frag *backend.MicroflowFragment) error { + existing := map[string]bool{} + for _, p := range a.mf.Parameters { + existing[p.Name] = true + } + for _, c := range a.cands { + if c.OutputVariable != "" { + existing[c.OutputVariable] = true + } + } + own := map[string]bool{} + for _, obj := range frag.Objects { + // A loop's iterator exists only inside the loop, which the fragment + // brings along; it reads it there, so it is the fragment's own. + if loop, ok := obj.(*microflows.LoopedActivity); ok { + if src, ok := loop.LoopSource.(*microflows.IterableList); ok && src.VariableName != "" { + own[src.VariableName] = true + } + continue + } + act, ok := obj.(*microflows.ActionActivity) + if !ok { + continue + } + v := mfmutator.OutputVariable(act.Action) + if v == "" { + continue + } + replacingSame := op.Op == ast.AlterFlowReplace && v == target.OutputVariable + if existing[v] && !replacingSame { + return fmt.Errorf("the fragment declares $%s, which the %s already has; choose another name", v, a.stmt.Kind()) + } + if a.declaredByOps[v] { + return fmt.Errorf("the fragment declares $%s, which an earlier operation of this alter already declares; choose another name", v) + } + own[v] = true + } + + inScope := map[string]bool{} + for _, p := range a.mf.Parameters { + inScope[p.Name] = true + } + for id := range a.upstreamOf(target.ID, op.Op == ast.AlterFlowInsertAfter) { + for _, c := range a.cands { + if c.ID == id && c.OutputVariable != "" { + inScope[c.OutputVariable] = true + } + } + } + var missing []string + seen := map[string]bool{} + for _, obj := range frag.Objects { + for _, m := range variableRef.FindAllStringSubmatch(formatActivity(ctx, obj, a.entityNames, a.microflowNames), -1) { + v := m[1] + if seen[v] || systemVariables[v] || own[v] || (inScope[v] && !a.removedByOps[v]) { + continue + } + seen[v] = true + missing = append(missing, "$"+v) + } + } + if len(missing) > 0 { + sort.Strings(missing) + where := "before " + op.Target + if op.Op == ast.AlterFlowInsertAfter { + where = "after " + op.Target + } + return fmt.Errorf("the fragment uses %s, which is not declared on the path %s", strings.Join(missing, ", "), where) + } + return nil +} + +// upstreamOf returns every object from which id can be reached along the +// stored flows — the activities whose outputs exist when the flow gets there. +// id itself is included only when including says so (an insert after it runs +// once it has). +func (a *alterFlowContext) upstreamOf(id model.ID, including bool) map[model.ID]bool { + preds := map[model.ID][]model.ID{} + for _, f := range a.mf.ObjectCollection.Flows { + preds[f.DestinationID] = append(preds[f.DestinationID], f.OriginID) + } + out := map[model.ID]bool{} + queue := append([]model.ID(nil), preds[id]...) + for len(queue) > 0 { + n := queue[0] + queue = queue[1:] + if out[n] { + continue + } + out[n] = true + queue = append(queue, preds[n]...) + } + if including { + out[id] = true + } + return out +} + +// checkOutputUnused refuses to take away an activity whose output variable +// another activity still reads — unless the replacement declares it again. +func (a *alterFlowContext) checkOutputUnused(target mfmutator.Candidate, replacement *backend.MicroflowFragment) error { + v := target.OutputVariable + if v == "" || fragmentDeclares(replacement, v) { + return nil + } + if readers := a.readByOps[v]; len(readers) > 0 { + return fmt.Errorf("$%s is read by what an earlier operation of this alter adds: %s", v, strings.Join(readers, "; ")) + } + ref := regexp.MustCompile(`\$` + regexp.QuoteMeta(v) + `\b`) + var users []string + for _, c := range a.cands { + if c.ID == target.ID { + continue + } + for _, text := range append([]string{c.Statement}, c.Alternates...) { + if ref.MatchString(text) { + users = append(users, c.Statement) + break + } + } + } + if len(users) > 0 { + return fmt.Errorf("$%s is still used by: %s", v, strings.Join(users, "; ")) + } + return nil +} diff --git a/mdl/executor/cmd_alter_flow_pedapp_test.go b/mdl/executor/cmd_alter_flow_pedapp_test.go new file mode 100644 index 0000000000..791677917e --- /dev/null +++ b/mdl/executor/cmd_alter_flow_pedapp_test.go @@ -0,0 +1,645 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "bytes" + "context" + "fmt" + "os" + "path/filepath" + "strconv" + "strings" + "testing" + + "go.mongodb.org/mongo-driver/bson" + "go.mongodb.org/mongo-driver/bson/primitive" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/backend" + modelsdkbackend "github.com/mendixlabs/mxcli/mdl/backend/modelsdk" + "github.com/mendixlabs/mxcli/mdl/types" + "github.com/mendixlabs/mxcli/mdl/visitor" + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/sdk/microflows" +) + +// The acceptance test of plan item 4.2 (ako/mxcli#736) runs on the Studio +// Pro-authored PedApp fixture, because only a flow Studio Pro drew can show +// what a rebuild loses: its merges, its curves, its object order, its $IDs. + +// openPedAppFixture opens a private copy of testdata/pedapp. +func openPedAppFixture(t *testing.T) (*Executor, *bytes.Buffer) { + t.Helper() + src := filepath.Join("..", "..", "testdata", "pedapp") + if _, err := os.Stat(filepath.Join(src, "PedApp.mpr")); err != nil { + t.Skipf("PedApp fixture not found: %v", err) + } + dir := t.TempDir() + if err := copyPedAppFile(filepath.Join(src, "PedApp.mpr"), filepath.Join(dir, "PedApp.mpr")); err != nil { + t.Fatal(err) + } + if err := copyPedAppTree(filepath.Join(src, "mprcontents"), filepath.Join(dir, "mprcontents")); err != nil { + t.Fatal(err) + } + out := &bytes.Buffer{} + exec := New(out) + exec.SetBackendFactory(func() backend.FullBackend { return modelsdkbackend.New() }) + if err := exec.Execute(&ast.ConnectStmt{Path: filepath.Join(dir, "PedApp.mpr")}); err != nil { + t.Fatalf("connect: %v", err) + } + t.Cleanup(func() { _ = exec.Execute(&ast.DisconnectStmt{}) }) + return exec, out +} + +func afRun(t *testing.T, exec *Executor, src string) error { + t.Helper() + prog, errs := visitor.Build(src) + if len(errs) > 0 { + t.Fatalf("parse %q: %v", src, errs[0]) + } + for _, s := range prog.Statements { + if err := exec.Execute(s); err != nil { + return err + } + } + return nil +} + +// valFeedbackUnit returns VAL_Feedback's unit ID and stored bytes. +func valFeedbackUnit(t *testing.T, exec *Executor) (model.ID, []byte) { + t.Helper() + ctx := exec.newExecContext(context.Background()) + h, err := getHierarchy(ctx) + if err != nil { + t.Fatal(err) + } + all, err := ctx.Backend.ListMicroflows() + if err != nil { + t.Fatal(err) + } + for _, m := range all { + if m.Name == "VAL_Feedback" && h.GetModuleName(h.FindModuleID(m.ContainerID)) == "FeedbackModule" { + raw, err := ctx.Backend.GetRawUnitBytes(m.ID) + if err != nil { + t.Fatal(err) + } + return m.ID, append([]byte(nil), raw...) + } + } + t.Fatal("FeedbackModule.VAL_Feedback not found") + return "", nil +} + +// flowView indexes a stored flow's top-level objects and flows by $ID. +type flowView struct { + doc bson.D + objs map[string]bson.D + flows map[string]bson.D + order []string // object ids in storage order +} + +func parseFlowView(t *testing.T, raw []byte) flowView { + t.Helper() + var d bson.D + if err := bson.Unmarshal(raw, &d); err != nil { + t.Fatal(err) + } + v := flowView{doc: d, objs: map[string]bson.D{}, flows: map[string]bson.D{}} + oc, _ := afGet(d, "ObjectCollection").(bson.D) + for _, el := range afList(afGet(oc, "Objects")) { + o := el.(bson.D) + id := afIDOf(afGet(o, "$ID")) + v.objs[id] = o + v.order = append(v.order, id) + } + for _, el := range afList(afGet(d, "Flows")) { + f := el.(bson.D) + v.flows[afIDOf(afGet(f, "$ID"))] = f + } + return v +} + +func afGet(d bson.D, k string) any { + for _, e := range d { + if e.Key == k { + return e.Value + } + } + return nil +} + +func afList(v any) []any { + a, _ := v.(bson.A) + if len(a) > 0 { + if _, ok := a[0].(int32); ok { + return a[1:] + } + } + return a +} + +func afIDOf(v any) string { + b, _ := v.(primitive.Binary) + return types.BlobToUUID(b.Data) +} + +func afMarshal(t *testing.T, v any) []byte { + t.Helper() + b, err := bson.Marshal(v) + if err != nil { + t.Fatal(err) + } + return b +} + +// without returns d minus the named keys, for comparing the rest. +func afWithout(d bson.D, keys ...string) bson.D { + var out bson.D + for _, e := range d { + skip := false + for _, k := range keys { + skip = skip || e.Key == k + } + if !skip { + out = append(out, e) + } + } + return out +} + +// changedKeys lists the keys whose values differ between two elements (one +// level deep, plus the Line's vectors, which is where a flow keeps its curve). +func afChangedKeys(t *testing.T, a, b bson.D) []string { + t.Helper() + var out []string + keys := map[string]bool{} + for _, e := range a { + keys[e.Key] = true + } + for _, e := range b { + keys[e.Key] = true + } + for k := range keys { + av, bv := afGet(a, k), afGet(b, k) + if k == "Line" { + al, _ := av.(bson.D) + bl, _ := bv.(bson.D) + for _, lk := range []string{"OriginControlVector", "DestinationControlVector"} { + if fmt.Sprint(afGet(al, lk)) != fmt.Sprint(afGet(bl, lk)) { + out = append(out, "Line."+lk) + } + } + if !bytes.Equal(afMarshal(t, afWithout(al, "OriginControlVector", "DestinationControlVector")), + afMarshal(t, afWithout(bl, "OriginControlVector", "DestinationControlVector"))) { + out = append(out, "Line") + } + continue + } + if !bytes.Equal(afMarshal(t, bson.D{{Key: "v", Value: av}}), afMarshal(t, bson.D{{Key: "v", Value: bv}})) { + out = append(out, k) + } + } + return afSort(out) +} + +func afSort(s []string) []string { + for i := 1; i < len(s); i++ { + for j := i; j > 0 && s[j] < s[j-1]; j-- { + s[j], s[j-1] = s[j-1], s[j] + } + } + return s +} + +func afPoint(d bson.D) (int, int) { + s, _ := afGet(d, "RelativeMiddlePoint").(string) + x, y, _ := strings.Cut(s, ";") + px, _ := strconv.Atoi(x) + py, _ := strconv.Atoi(y) + return px, py +} + +func afSize(d bson.D) (int, int) { + s, _ := afGet(d, "Size").(string) + x, y, _ := strings.Cut(s, ";") + px, _ := strconv.Atoi(x) + py, _ := strconv.Atoi(y) + return px, py +} + +// objectAt returns the id of the stored top-level object at (x, y). +func (v flowView) objectAt(t *testing.T, x, y int, typ string) string { + t.Helper() + for _, id := range v.order { + o := v.objs[id] + if px, py := afPoint(o); px == x && py == y && afGet(o, "$Type") == typ { + return id + } + } + t.Fatalf("no %s at (%d, %d)", typ, x, y) + return "" +} + +// The acceptance test of plan item 4.2 (ako/mxcli#736): one `log` inserted +// after $IsValidEmail in the Studio Pro-drawn VAL_Feedback. Only the new +// activity, the two flows around it and the positions moved to make room may +// differ; every other element — its $ID, its curve, every merge — must come +// through byte-identical. +func TestAlterMicroflow_PedApp_InsertAfterChangesOnlyTheSplice(t *testing.T) { + exec, _ := openPedAppFixture(t) + _, raw := valFeedbackUnit(t, exec) + before := parseFlowView(t, raw) + javaCall := before.objectAt(t, 980, 200, "Microflows$ActionActivity") + split := before.objectAt(t, 1155, 200, "Microflows$ExclusiveSplit") + + script := "alter microflow FeedbackModule.VAL_Feedback {\n" + + " insert after $IsValidEmail { log info node 'Feedback' 'Email checked'; }\n" + + "};" + if n := strings.Count(script, "\n") + 1; n > 5 { + t.Fatalf("the acceptance script is %d lines; the plan allows 5", n) + } + if err := afRun(t, exec, script); err != nil { + t.Fatalf("alter: %v", err) + } + _, rawAfter := valFeedbackUnit(t, exec) + after := parseFlowView(t, rawAfter) + + // The document around the flow is untouched. + if !bytes.Equal(afMarshal(t, afWithout(before.doc, "ObjectCollection", "Flows")), + afMarshal(t, afWithout(after.doc, "ObjectCollection", "Flows"))) { + t.Error("a property of the microflow document itself changed") + } + ocBefore, _ := afGet(before.doc, "ObjectCollection").(bson.D) + ocAfter, _ := afGet(after.doc, "ObjectCollection").(bson.D) + if !bytes.Equal(afMarshal(t, afWithout(ocBefore, "Objects")), afMarshal(t, afWithout(ocAfter, "Objects"))) { + t.Error("a property of the object collection changed") + } + + // Objects: every stored one survives with its $ID and in its place in the + // list; exactly one is new, a log activity; the rest differ at most in + // position, and only by the shift that made room. + for i, id := range before.order { + if after.order[i] != id { + t.Fatalf("stored object %d moved in the list: %s became %s", i, id, after.order[i]) + } + } + if got := len(after.order) - len(before.order); got != 1 { + t.Fatalf("want exactly one new object, got %d", got) + } + newID := after.order[len(after.order)-1] + newObj := after.objs[newID] + if action, _ := afGet(newObj, "Action").(bson.D); afGet(newObj, "$Type") != "Microflows$ActionActivity" || + afGet(action, "$Type") != "Microflows$LogMessageAction" { + t.Fatalf("the new object is not a log activity: %v", afGet(newObj, "$Type")) + } + shifted := 0 + javaX, _ := afPoint(before.objs[javaCall]) + for _, id := range before.order { + b, a := before.objs[id], after.objs[id] + changed := afChangedKeys(t, b, a) + if len(changed) == 0 { + continue + } + if len(changed) != 1 || changed[0] != "RelativeMiddlePoint" { + t.Errorf("object %s (%v) changed more than its position: %v", id, afGet(b, "$Type"), changed) + continue + } + bx, by := afPoint(b) + ax, ay := afPoint(a) + if ay != by || ax <= bx || bx <= javaX { + t.Errorf("object %s moved from (%d,%d) to (%d,%d): only objects past the insertion point may move, and only along the flow", + id, bx, by, ax, ay) + } + shifted++ + } + if shifted == 0 { + t.Error("nothing was shifted, yet the gap after $IsValidEmail is too narrow for an activity: placement did not run") + } + // The new activity overlaps nothing. + nx, ny := afPoint(newObj) + nw, nh := afSize(newObj) + for _, id := range after.order { + if id == newID { + continue + } + o := after.objs[id] + ox, oy := afPoint(o) + ow, oh := afSize(o) + if abs(nx-ox)*2 < nw+ow && abs(ny-oy)*2 < nh+oh { + t.Errorf("the new activity at (%d,%d) overlaps %v at (%d,%d)", nx, ny, afGet(o, "$Type"), ox, oy) + } + } + + // Flows: every stored flow survives; one is new (log -> split); one is + // rewired (java call -> log), and only at its destination end. + var added []string + for id := range after.flows { + if _, ok := before.flows[id]; !ok { + added = append(added, id) + } + } + if len(added) != 1 { + t.Fatalf("want exactly one new flow, got %d", len(added)) + } + nf := after.flows[added[0]] + if afIDOf(afGet(nf, "OriginPointer")) != newID || afIDOf(afGet(nf, "DestinationPointer")) != split { + t.Error("the new flow does not run from the log to 'Email is Valid?'") + } + rewired := 0 + for id, b := range before.flows { + a, ok := after.flows[id] + if !ok { + t.Errorf("stored flow %s is gone", id) + continue + } + changed := afChangedKeys(t, b, a) + if len(changed) == 0 { + continue + } + rewired++ + if afIDOf(afGet(b, "OriginPointer")) != javaCall || afIDOf(afGet(a, "DestinationPointer")) != newID { + t.Errorf("flow %s changed but is not the flow out of $IsValidEmail: %v", id, changed) + } + for _, k := range changed { + switch k { + case "DestinationPointer", "DestinationConnectionIndex", "Line.DestinationControlVector": + default: + t.Errorf("the rewired flow changed %s; only its destination end may change", k) + } + } + } + if rewired != 1 { + t.Errorf("want exactly one rewired flow, got %d", rewired) + } + // The new flow ends where the rewired one used to. + var oldIn bson.D + for id, b := range before.flows { + if afIDOf(afGet(b, "OriginPointer")) == javaCall { + oldIn = b + _ = id + } + } + if fmt.Sprint(afGet(nf, "DestinationConnectionIndex")) != fmt.Sprint(afGet(oldIn, "DestinationConnectionIndex")) { + t.Error("the new flow does not enter 'Email is Valid?' on the side the old flow did") + } + + // And the result reads back: describe shows the log between the two. + var buf bytes.Buffer + exec.output = &buf + if err := afRun(t, exec, "describe microflow FeedbackModule.VAL_Feedback;"); err != nil { + t.Fatal(err) + } + body := buf.String() + iCall := strings.Index(body, "$IsValidEmail = call java action") + iLog := strings.Index(body, "log info node 'Feedback' 'Email checked';") + iSplit := strings.Index(body, "@caption 'Email is Valid?'") + if iCall < 0 || iLog < iCall || iSplit < iLog { + t.Errorf("describe does not show the log between the call and the decision:\n%s", body) + } +} + +// The control: an alter with no operations reads the unit, patches nothing +// and writes nothing — the stored bytes survive the decode/encode round trip +// exactly, so any difference the test above sees is the splice's. +func TestAlterMicroflow_PedApp_EmptyAlterChangesNothing(t *testing.T) { + exec, _ := openPedAppFixture(t) + _, raw := valFeedbackUnit(t, exec) + if err := afRun(t, exec, "alter microflow FeedbackModule.VAL_Feedback { };"); err != nil { + t.Fatalf("alter: %v", err) + } + _, rawAfter := valFeedbackUnit(t, exec) + if !bytes.Equal(raw, rawAfter) { + t.Error("an empty alter changed the stored unit") + } +} + +// Drop joins the flow into the dropped activity to the one after it, and +// takes away only the activity and the flow that left it. +func TestAlterMicroflow_PedApp_Drop(t *testing.T) { + exec, _ := openPedAppFixture(t) + _, raw := valFeedbackUnit(t, exec) + before := parseFlowView(t, raw) + target := before.objectAt(t, 1305, 460, "Microflows$ActionActivity") + mergeAfter := before.objectAt(t, 1460, 460, "Microflows$ExclusiveMerge") + + if err := afRun(t, exec, "alter microflow FeedbackModule.VAL_Feedback { drop set $ValidFeedback = false @3; };"); err != nil { + t.Fatalf("alter: %v", err) + } + _, rawAfter := valFeedbackUnit(t, exec) + after := parseFlowView(t, rawAfter) + + if _, ok := after.objs[target]; ok { + t.Fatal("the dropped activity is still there") + } + if len(after.objs) != len(before.objs)-1 || len(after.flows) != len(before.flows)-1 { + t.Fatalf("want one object and one flow fewer, got %d->%d objects, %d->%d flows", + len(before.objs), len(after.objs), len(before.flows), len(after.flows)) + } + for id, b := range before.objs { + if id == target { + continue + } + if !bytes.Equal(afMarshal(t, b), afMarshal(t, after.objs[id])) { + t.Errorf("object %s changed", id) + } + } + rewired := 0 + for id, b := range before.flows { + a, ok := after.flows[id] + if !ok { + if afIDOf(afGet(b, "OriginPointer")) != target { + t.Errorf("flow %s is gone but did not leave the dropped activity", id) + } + continue + } + if changed := afChangedKeys(t, b, a); len(changed) > 0 { + rewired++ + if afIDOf(afGet(b, "DestinationPointer")) != target || afIDOf(afGet(a, "DestinationPointer")) != mergeAfter { + t.Errorf("flow %s changed but is not the flow into the dropped activity: %v", id, changed) + } + } + } + if rewired != 1 { + t.Errorf("want one rewired flow, got %d", rewired) + } + if bytes.Contains(rawAfter, uuidBytes(target)) { + t.Error("the unit still contains the dropped activity's $ID") + } +} + +func uuidBytes(id string) []byte { return types.UUIDToBlob(id) } + +// Replace puts a two-activity fragment where one activity was: the flows in +// and out are re-pointed, everything past it moves along to make room. +func TestAlterMicroflow_PedApp_Replace(t *testing.T) { + exec, _ := openPedAppFixture(t) + _, raw := valFeedbackUnit(t, exec) + before := parseFlowView(t, raw) + target := before.objectAt(t, 1305, 460, "Microflows$ActionActivity") + + err := afRun(t, exec, `alter microflow FeedbackModule.VAL_Feedback { + replace set $ValidFeedback = false @3 with { + set $ValidFeedback = false; + log warning node 'Feedback' 'Email rejected'; + } + };`) + if err != nil { + t.Fatalf("alter: %v", err) + } + _, rawAfter := valFeedbackUnit(t, exec) + after := parseFlowView(t, rawAfter) + if _, ok := after.objs[target]; ok { + t.Fatal("the replaced activity is still there") + } + if got := len(after.objs) - len(before.objs); got != 1 { + t.Errorf("want one object more (two in, one out), got %+d", got) + } + if got := len(after.flows) - len(before.flows); got != 1 { + t.Errorf("want one flow more (the fragment's own), got %+d", got) + } + for id := range before.flows { + if _, ok := after.flows[id]; !ok { + t.Errorf("stored flow %s is gone; replace keeps the flows in and out", id) + } + } + if bytes.Contains(rawAfter, uuidBytes(target)) { + t.Error("the unit still contains the replaced activity's $ID") + } + var buf bytes.Buffer + exec.output = &buf + if err := afRun(t, exec, "describe microflow FeedbackModule.VAL_Feedback;"); err != nil { + t.Fatal(err) + } + if !strings.Contains(buf.String(), "log warning node 'Feedback' 'Email rejected';") { + t.Errorf("describe does not show the replacement:\n%s", buf.String()) + } +} + +// What the splice cannot do safely, it refuses — before writing anything. +func TestAlterMicroflow_PedApp_Refusals(t *testing.T) { + cases := []struct{ name, op, want string }{ + {"insert after a decision", `insert after 'Email is Valid?' { log info 'x'; }`, "which branch"}, + {"drop a decision", `drop 'Email is Valid?';`, "cannot drop"}, + {"drop a variable still read", `drop $IsValidEmail;`, "still used"}, + {"drop the end", `drop return $ValidFeedback;`, "cannot drop"}, + {"declare an existing variable", `insert after $IsValidEmail { declare $ValidFeedback Boolean = true; }`, "already has"}, + {"use a variable not yet declared", `insert after $ValidFeedback { log info 'x {1}' with ({1} = toString($IsValidEmail)); }`, "not declared on the path"}, + {"ambiguous target", `drop set $ValidFeedback = false;`, "add an ordinal"}, + {"unknown target", `drop $Nope;`, "no activity matches"}, + {"a fragment that returns", `insert after $IsValidEmail { return false; }`, "ends the flow"}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + exec, _ := openPedAppFixture(t) + _, raw := valFeedbackUnit(t, exec) + err := afRun(t, exec, "alter microflow FeedbackModule.VAL_Feedback { "+tc.op+" };") + if err == nil || !strings.Contains(err.Error(), tc.want) { + t.Fatalf("want an error containing %q, got %v", tc.want, err) + } + if _, rawAfter := valFeedbackUnit(t, exec); !bytes.Equal(raw, rawAfter) { + t.Error("a refused alter changed the stored unit") + } + }) + } +} + +var _ = microflows.ActionActivity{} + +// alter nanoflow goes through the same splice: the unit is a +// Microflows$Nanoflow with the same object collection and flows. +func TestAlterNanoflow_PedApp_InsertBefore(t *testing.T) { + exec, _ := openPedAppFixture(t) + err := afRun(t, exec, `alter nanoflow FeedbackModule.ACT_Feedback_ClearImage { + insert before call javascript action * { declare $Cleared Boolean = true; } + };`) + if err != nil { + t.Fatalf("alter: %v", err) + } + var buf bytes.Buffer + exec.output = &buf + if err := afRun(t, exec, "describe nanoflow FeedbackModule.ACT_Feedback_ClearImage;"); err != nil { + t.Fatal(err) + } + body := buf.String() + iChange := strings.Index(body, "change $Feedback") + iDeclare := strings.Index(body, "declare $Cleared Boolean = true;") + iCall := strings.Index(body, "call javascript action") + if iChange < 0 || iDeclare < iChange || iCall < iDeclare { + t.Errorf("describe does not show the declare between the change and the call:\n%s", body) + } +} + +// The scope check covers the whole statement, not each operation against the +// stored flow alone. Measured with mx check 11.14 on TestApp before the fix: +// two inserts declaring the same variable gave CE0111 "Duplicate variable +// name", and an insert reading a variable a drop in the same statement took +// away gave CE0109 "Undefined variable" — both after "Altered microflow". +func TestAlterMicroflow_PedApp_ScopeSpansTheStatement(t *testing.T) { + t.Run("two fragments declare the same variable", func(t *testing.T) { + exec, _ := openPedAppFixture(t) + _, raw := valFeedbackUnit(t, exec) + err := afRun(t, exec, `alter microflow FeedbackModule.VAL_Feedback { + insert after $IsValidEmail { declare $Dup Boolean = true; } + insert after $ValidFeedback { declare $Dup Boolean = false; } + };`) + if err == nil || !strings.Contains(err.Error(), "$Dup") { + t.Fatalf("want the clash on $Dup refused, got %v", err) + } + if _, after := valFeedbackUnit(t, exec); !bytes.Equal(raw, after) { + t.Error("a refused alter changed the stored unit") + } + }) + + // SUB_Feedback_Sanitize reads its nine $Sanitized* outputs in one change; + // replacing that change first leaves $SanitizedPageName unread, so only a + // fragment of the second statement reads it. + const unread = `alter microflow FeedbackModule.SUB_Feedback_Sanitize { + replace change $Feedback (Subject = $SanitizedSubject, Description = $SanitizedDescription, SubmitterUUID = $SanitizedSubmitterUUID, SubmitterEmail = $SanitizedSubmitterEmail, SubmitterDisplayName = $SanitizedSubmitterDisplayName, ActiveUserRoles = $SanitizedActiveUserRoles, PageName = $SanitizedPageName, Browser = $SanitizedBrowser, EnvironmentURL = $SanitizedEnvironmentURL) with { log info 'sanitized'; } + };` + const use = `insert after $SanitizedSubmitterUUID { log info 'page {1}' with ({1} = $SanitizedPageName); }` + const drop = `drop $SanitizedPageName;` + for name, ops := range map[string]string{ + "insert a reader, then drop the producer": use + "\n" + drop, + "drop the producer, then insert a reader": drop + "\n" + use, + } { + t.Run(name, func(t *testing.T) { + exec, _ := openPedAppFixture(t) + if err := afRun(t, exec, unread); err != nil { + t.Fatalf("setup: %v", err) + } + err := afRun(t, exec, "alter microflow FeedbackModule.SUB_Feedback_Sanitize {\n"+ops+"\n};") + if err == nil || !strings.Contains(err.Error(), "$SanitizedPageName") { + t.Fatalf("want the read of a dropped $SanitizedPageName refused, got %v", err) + } + }) + } + // Control: each operation on its own is accepted. + t.Run("control", func(t *testing.T) { + for _, op := range []string{use, drop} { + exec, _ := openPedAppFixture(t) + if err := afRun(t, exec, unread); err != nil { + t.Fatalf("setup: %v", err) + } + if err := afRun(t, exec, "alter microflow FeedbackModule.SUB_Feedback_Sanitize {\n"+op+"\n};"); err != nil { + t.Errorf("%s: %v", op, err) + } + } + }) +} + +// A loop in the fragment declares its iterator; the scope check must count it +// as the fragment's own, or every loop fragment is refused as reading an +// undeclared variable. +func TestAlterMicroflow_PedApp_LoopFragmentDeclaresItsIterator(t *testing.T) { + exec, _ := openPedAppFixture(t) + err := afRun(t, exec, `alter microflow FeedbackModule.VAL_Feedback { + insert after $IsValidEmail { + $Items = create list of FeedbackModule.Feedback; + loop $Item in $Items begin log info 'item'; end loop; + } + };`) + if err != nil { + t.Fatalf("alter: %v", err) + } +} diff --git a/mdl/executor/cmd_associations.go b/mdl/executor/cmd_associations.go index 34561bc3af..4a245ae8dc 100644 --- a/mdl/executor/cmd_associations.go +++ b/mdl/executor/cmd_associations.go @@ -594,7 +594,6 @@ func describeAssociation(ctx *ExecContext, name ast.QualifiedName) error { fmt.Fprintf(ctx.Output, "create or modify association %s.%s\n", module.Name, assoc.Name) fmt.Fprintf(ctx.Output, "from %s to %s\n", fromEntity, toEntity) formatAssocDetails(assoc.Type, assoc.Owner, assoc.StorageFormat, assoc.ChildDeleteBehavior) - fmt.Fprintln(ctx.Output, "/") return nil } } @@ -612,7 +611,6 @@ func describeAssociation(ctx *ExecContext, name ast.QualifiedName) error { fmt.Fprintf(ctx.Output, "create or modify association %s.%s\n", module.Name, ca.Name) fmt.Fprintf(ctx.Output, "from %s to %s\n", fromEntity, ca.ChildRef) formatAssocDetails(ca.Type, ca.Owner, ca.StorageFormat, ca.ChildDeleteBehavior) - fmt.Fprintln(ctx.Output, "/") return nil } } diff --git a/mdl/executor/cmd_constants.go b/mdl/executor/cmd_constants.go index 06cdc1713e..172da93557 100644 --- a/mdl/executor/cmd_constants.go +++ b/mdl/executor/cmd_constants.go @@ -132,7 +132,6 @@ func outputConstantMDL(ctx *ExecContext, c *model.Constant, moduleName string) e } fmt.Fprintln(ctx.Output, ";") - fmt.Fprintln(ctx.Output, "/") return nil } diff --git a/mdl/executor/cmd_contract.go b/mdl/executor/cmd_contract.go index 40ec17fddd..eb76b49150 100644 --- a/mdl/executor/cmd_contract.go +++ b/mdl/executor/cmd_contract.go @@ -331,7 +331,6 @@ func outputContractEntityMDL(ctx *ExecContext, et *types.EdmEntityType, svcQN st } fmt.Fprintln(ctx.Output, ");") - fmt.Fprintln(ctx.Output, "/") return nil } diff --git a/mdl/executor/cmd_datatransformer.go b/mdl/executor/cmd_datatransformer.go index 913d085727..cd20343f5f 100644 --- a/mdl/executor/cmd_datatransformer.go +++ b/mdl/executor/cmd_datatransformer.go @@ -76,7 +76,7 @@ func describeDataTransformer(ctx *ExecContext, name ast.QualifiedName) error { w := ctx.Output // Emit re-executable MDL - fmt.Fprintf(w, "create data transformer %s.%s%s\n", modName, dt.Name, describeFolderClause(ctx, dt.ContainerID)) + fmt.Fprintf(w, "create or modify data transformer %s.%s%s\n", modName, dt.Name, describeFolderClause(ctx, dt.ContainerID)) // Source — collapse newlines into spaces for single-line string sourceContent := strings.ReplaceAll(dt.SourceJSON, "\n", " ") diff --git a/mdl/executor/cmd_datatransformer_mock_test.go b/mdl/executor/cmd_datatransformer_mock_test.go index 696152b844..55ea1931c5 100644 --- a/mdl/executor/cmd_datatransformer_mock_test.go +++ b/mdl/executor/cmd_datatransformer_mock_test.go @@ -103,7 +103,7 @@ func TestDescribeDataTransformer_Mock(t *testing.T) { assertNoError(t, describeDataTransformer(ctx, ast.QualifiedName{Module: "ETL", Name: "TransformOrders"})) out := buf.String() - assertContainsStr(t, out, "create data transformer") + assertContainsStr(t, out, "create or modify data transformer") } func TestCreateDataTransformer_OrModify_PreservesID(t *testing.T) { diff --git a/mdl/executor/cmd_dbconnection.go b/mdl/executor/cmd_dbconnection.go index 78b67b6e2e..afcef20f00 100644 --- a/mdl/executor/cmd_dbconnection.go +++ b/mdl/executor/cmd_dbconnection.go @@ -237,7 +237,7 @@ func describeDatabaseConnection(ctx *ExecContext, name ast.QualifiedName) error // outputDatabaseConnectionMDL outputs a database connection definition in MDL format. func outputDatabaseConnectionMDL(ctx *ExecContext, conn *model.DatabaseConnection, moduleName string) error { - fmt.Fprintf(ctx.Output, "create database connection %s.%s%s\n", moduleName, conn.Name, describeFolderClause(ctx, conn.ContainerID)) + fmt.Fprintf(ctx.Output, "create or modify database connection %s.%s%s\n", moduleName, conn.Name, describeFolderClause(ctx, conn.ContainerID)) fmt.Fprintf(ctx.Output, "type '%s'\n", conn.DatabaseType) // Connection string @@ -303,7 +303,6 @@ func outputDatabaseConnectionMDL(ctx *ExecContext, conn *model.DatabaseConnectio } fmt.Fprintln(ctx.Output, ";") - fmt.Fprintln(ctx.Output, "/") return nil } diff --git a/mdl/executor/cmd_dbconnection_mock_test.go b/mdl/executor/cmd_dbconnection_mock_test.go index 03b81e6766..ec1314608b 100644 --- a/mdl/executor/cmd_dbconnection_mock_test.go +++ b/mdl/executor/cmd_dbconnection_mock_test.go @@ -102,5 +102,5 @@ func TestDescribeDatabaseConnection_Mock(t *testing.T) { assertNoError(t, describeDatabaseConnection(ctx, ast.QualifiedName{Module: "DataMod", Name: "MyDB"})) out := buf.String() - assertContainsStr(t, out, "create database connection") + assertContainsStr(t, out, "create or modify database connection") } diff --git a/mdl/executor/cmd_diff_local.go b/mdl/executor/cmd_diff_local.go index f60bb8025a..4be788f3e9 100644 --- a/mdl/executor/cmd_diff_local.go +++ b/mdl/executor/cmd_diff_local.go @@ -393,7 +393,6 @@ func entityBsonToMDL(ctx *ExecContext, raw map[string]any, qualifiedName string) } lines = append(lines, ");") - lines = append(lines, "/") return strings.Join(lines, "\n") } @@ -573,7 +572,6 @@ func nanoflowBsonToMDL(_ *ExecContext, raw map[string]any, qualifiedName string) lines = append(lines, "begin") lines = append(lines, " -- (nanoflow body)") lines = append(lines, "end;") - lines = append(lines, "/") return strings.Join(lines, "\n") } @@ -607,7 +605,6 @@ func enumerationBsonToMDL(_ *ExecContext, raw map[string]any, qualifiedName stri } lines = append(lines, ");") - lines = append(lines, "/") return strings.Join(lines, "\n") } @@ -638,7 +635,6 @@ func pageBsonToMDL(_ *ExecContext, raw map[string]any, qualifiedName string) str lines = append(lines, "begin") lines = append(lines, " -- (page widgets)") lines = append(lines, "end;") - lines = append(lines, "/") return strings.Join(lines, "\n") } @@ -657,7 +653,6 @@ func snippetBsonToMDL(_ *ExecContext, raw map[string]any, qualifiedName string) lines = append(lines, "begin") lines = append(lines, " -- (snippet widgets)") lines = append(lines, "end;") - lines = append(lines, "/") return strings.Join(lines, "\n") } @@ -676,7 +671,6 @@ func layoutBsonToMDL(_ *ExecContext, raw map[string]any, qualifiedName string) s lines = append(lines, "begin") lines = append(lines, " -- (layout structure)") lines = append(lines, "end;") - lines = append(lines, "/") return strings.Join(lines, "\n") } @@ -693,7 +687,6 @@ func moduleBsonToMDL(_ *ExecContext, raw map[string]any) string { } lines = append(lines, fmt.Sprintf("create module %s;", name)) - lines = append(lines, "/") return strings.Join(lines, "\n") } diff --git a/mdl/executor/cmd_diff_mdl.go b/mdl/executor/cmd_diff_mdl.go index 175143e988..77b1f2d549 100644 --- a/mdl/executor/cmd_diff_mdl.go +++ b/mdl/executor/cmd_diff_mdl.go @@ -92,7 +92,6 @@ func entityStmtToMDL(ctx *ExecContext, s *ast.CreateEntityStmt) string { } lines = append(lines, ";") - lines = append(lines, "/") return strings.Join(lines, "\n") } @@ -128,7 +127,6 @@ func viewEntityStmtToMDL(ctx *ExecContext, s *ast.CreateViewEntityStmt) string { lines = append(lines, " "+line) } lines = append(lines, ");") - lines = append(lines, "/") return strings.Join(lines, "\n") } @@ -154,7 +152,6 @@ func enumerationStmtToMDL(ctx *ExecContext, s *ast.CreateEnumerationStmt) string } lines = append(lines, ");") - lines = append(lines, "/") return strings.Join(lines, "\n") } @@ -192,7 +189,6 @@ func associationStmtToMDL(ctx *ExecContext, s *ast.CreateAssociationStmt) string deleteBehavior = "DELETE_IF_NO_REFERENCES" } lines = append(lines, fmt.Sprintf("delete_behavior %s;", deleteBehavior)) - lines = append(lines, "/") return strings.Join(lines, "\n") } @@ -307,7 +303,6 @@ func entityToMDL(ctx *ExecContext, moduleName string, entity *domainmodel.Entity } lines = append(lines, ";") - lines = append(lines, "/") return strings.Join(lines, "\n") } @@ -341,7 +336,6 @@ func viewEntityFromProjectToMDL(ctx *ExecContext, moduleName string, entity *dom } } lines = append(lines, ");") - lines = append(lines, "/") return strings.Join(lines, "\n") } @@ -373,7 +367,6 @@ func enumerationToMDL(ctx *ExecContext, moduleName string, enum *model.Enumerati } lines = append(lines, ");") - lines = append(lines, "/") return strings.Join(lines, "\n") } @@ -422,7 +415,6 @@ func associationToMDL(ctx *ExecContext, moduleName string, assoc *domainmodel.As } } lines = append(lines, fmt.Sprintf("delete_behavior %s;", deleteBehavior)) - lines = append(lines, "/") return strings.Join(lines, "\n") } diff --git a/mdl/executor/cmd_drop_module_role_test.go b/mdl/executor/cmd_drop_module_role_test.go new file mode 100644 index 0000000000..eff9a96bb5 --- /dev/null +++ b/mdl/executor/cmd_drop_module_role_test.go @@ -0,0 +1,35 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/backend/mock" + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/sdk/security" +) + +// `drop module role if exists` is a no-op for a role that is not there, the +// same as `drop user role if exists` (#731). The control is the same statement +// without IF EXISTS, which still refuses. +func TestDropModuleRole_IfExistsSkipsMissingRole(t *testing.T) { + mod := mkModule("M") + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListModulesFunc: func() ([]*model.Module, error) { return []*model.Module{mod}, nil }, + GetModuleSecurityFunc: func(model.ID) (*security.ModuleSecurity, error) { + return &security.ModuleSecurity{ModuleRoles: []*security.ModuleRole{{Name: "User"}}}, nil + }, + } + ctx, buf := newMockCtx(t, withBackend(mb), withHierarchy(mkHierarchy(mod))) + name := ast.QualifiedName{Module: "M", Name: "Gone"} + + assertNoError(t, execDropModuleRole(ctx, &ast.DropModuleRoleStmt{Name: name, IfExists: true})) + assertContainsStr(t, buf.String(), "does not exist, skipping") + + err := execDropModuleRole(ctx, &ast.DropModuleRoleStmt{Name: name}) + assertError(t, err) + assertContainsStr(t, err.Error(), "not found") +} diff --git a/mdl/executor/cmd_entities_describe.go b/mdl/executor/cmd_entities_describe.go index 60288a822a..2a0f977881 100644 --- a/mdl/executor/cmd_entities_describe.go +++ b/mdl/executor/cmd_entities_describe.go @@ -422,7 +422,6 @@ func describeEntity(ctx *ExecContext, name ast.QualifiedName) error { // Output access rule GRANT statements outputEntityAccessGrants(ctx, entity, name.Module, name.Name) - fmt.Fprintln(ctx.Output, "/") return nil } } diff --git a/mdl/executor/cmd_enumerations.go b/mdl/executor/cmd_enumerations.go index 5e0aa8ec98..0a803a7f9e 100644 --- a/mdl/executor/cmd_enumerations.go +++ b/mdl/executor/cmd_enumerations.go @@ -454,7 +454,6 @@ func describeEnumeration(ctx *ExecContext, name ast.QualifiedName) error { folderClause = fmt.Sprintf(" FOLDER '%s'", fp) } fmt.Fprintf(ctx.Output, ")%s;\n", folderClause) - fmt.Fprintln(ctx.Output, "/") return nil } } diff --git a/mdl/executor/cmd_imagecollections.go b/mdl/executor/cmd_imagecollections.go index 233b3d7cfb..4473689951 100644 --- a/mdl/executor/cmd_imagecollections.go +++ b/mdl/executor/cmd_imagecollections.go @@ -151,7 +151,6 @@ func describeImageCollection(ctx *ExecContext, name ast.QualifiedName) error { fmt.Fprintf(ctx.Output, " export level '%s'", exportLevel) } fmt.Fprintln(ctx.Output, ";") - fmt.Fprintln(ctx.Output, "/") return nil } @@ -184,7 +183,6 @@ func describeImageCollection(ctx *ExecContext, name ast.QualifiedName) error { } fmt.Fprintln(ctx.Output, ");") - fmt.Fprintln(ctx.Output, "/") return nil } diff --git a/mdl/executor/cmd_imagecollections_mock_test.go b/mdl/executor/cmd_imagecollections_mock_test.go index 116c6df993..8848ae2f8f 100644 --- a/mdl/executor/cmd_imagecollections_mock_test.go +++ b/mdl/executor/cmd_imagecollections_mock_test.go @@ -103,6 +103,7 @@ func TestDescribeImageCollection_Mock(t *testing.T) { assertNoError(t, describeImageCollection(ctx, ast.QualifiedName{Module: "Icons", Name: "AppIcons"})) out := buf.String() + assertTerminated(t, out) // #744 assertContainsStr(t, out, "create or modify image collection") } diff --git a/mdl/executor/cmd_javascript_actions.go b/mdl/executor/cmd_javascript_actions.go index b14eacfb36..c771659cf7 100644 --- a/mdl/executor/cmd_javascript_actions.go +++ b/mdl/executor/cmd_javascript_actions.go @@ -64,6 +64,11 @@ func listJavaScriptActions(ctx *ExecContext, moduleName string) error { return writeResult(ctx, result) } +// jsSourceOmittedBody is the body DESCRIBE prints for an action whose .js source +// it cannot read. exec recognises it and writes no source file (#731), as it +// does for the Java twin, javaSourceOmittedBody. +const jsSourceOmittedBody = "// JavaScript source not available from this project; body omitted by DESCRIBE." + // describeJavaScriptAction handles DESCRIBE JAVASCRIPT ACTION command. func describeJavaScriptAction(ctx *ExecContext, name ast.QualifiedName) error { qualifiedName := name.Module + "." + name.Name @@ -88,7 +93,7 @@ func describeJavaScriptAction(ctx *ExecContext, name ast.QualifiedName) error { } // Type parameters - sb.WriteString("create javascript action ") + sb.WriteString("create or modify javascript action ") sb.WriteString(qualifiedName) if len(jsa.TypeParameters) > 0 { sb.WriteString("<") @@ -176,7 +181,7 @@ func describeJavaScriptAction(ctx *ExecContext, name ast.QualifiedName) error { if userCode != "" { sb.WriteString(userCode) } else { - sb.WriteString("// JavaScript source not available from this project; body omitted by DESCRIBE.") + sb.WriteString(jsSourceOmittedBody) } sb.WriteString("\n$$;") diff --git a/mdl/executor/cmd_javascript_actions_mock_test.go b/mdl/executor/cmd_javascript_actions_mock_test.go index f7d7dc0d10..8b7e101707 100644 --- a/mdl/executor/cmd_javascript_actions_mock_test.go +++ b/mdl/executor/cmd_javascript_actions_mock_test.go @@ -56,7 +56,7 @@ func TestDescribeJavaScriptAction_Mock(t *testing.T) { assertNoError(t, describeJavaScriptAction(ctx, ast.QualifiedName{Module: "WebMod", Name: "ShowAlert"})) out := buf.String() - assertContainsStr(t, out, "create javascript action") + assertContainsStr(t, out, "create or modify javascript action") } func TestDescribeJavaScriptAction_NotFound(t *testing.T) { @@ -170,3 +170,102 @@ func TestCreateJavaScriptAction_DuplicateRejected(t *testing.T) { assertError(t, err) assertContainsStr(t, err.Error(), "already exists") } + +// With describe emitting `create or modify javascript action`, its output now +// re-executes (#731). Where describe could not read the .js source it prints a +// placeholder body; that is not code, and writing it would replace the real +// source (or add a stub file to a project that ships none). The Java twin is +// TestJavaActionBody_PlaceholderIsNotWritten. +func TestJavaScriptActionBody_PlaceholderIsNotWritten(t *testing.T) { + mod := mkModule("WebMod") + h := mkHierarchy(mod) + wrote := false + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListModulesFunc: func() ([]*model.Module, error) { return []*model.Module{mod}, nil }, + ListJavaScriptActionsFunc: func() ([]*types.JavaScriptAction, error) { return nil, nil }, + CreateJavaScriptActionFunc: func(*types.JavaScriptAction) error { return nil }, + WriteJavaScriptSourceFileFunc: func(string, string, string, []*types.JavaActionParameter, types.CodeActionReturnType) error { + wrote = true + return nil + }, + } + ctx, _ := newMockCtx(t, withBackend(mb), withHierarchy(h)) + stmt := func(code string) *ast.CreateJavaScriptActionStmt { + return &ast.CreateJavaScriptActionStmt{ + Name: ast.QualifiedName{Module: "WebMod", Name: "DoThing"}, + ReturnType: ast.DataType{Kind: ast.TypeBoolean}, + JavaScriptCode: code, + } + } + + assertNoError(t, execCreateJavaScriptAction(ctx, stmt("\n"+jsSourceOmittedBody+"\n"))) + if wrote { + t.Error("the DESCRIBE placeholder body was written as the action's JavaScript source") + } + + // Control: an authored body is still written. + assertNoError(t, execCreateJavaScriptAction(ctx, stmt("return Promise.resolve(true);"))) + if !wrote { + t.Error("an authored body was not written — the guard is too broad") + } +} + +// A parameter's Description and Category have no MDL spelling — describe +// prints the description as a `--` comment — so `create or modify` carries them +// from the stored parameter of the same name, as the Java twin does. Dropping +// them made running unchanged describe output rewrite the action (#731). +func TestJavaScriptActionRewrite_CarriesParameterDescription(t *testing.T) { + mod := mkModule("WebMod") + h := mkHierarchy(mod) + stored := &types.JavaScriptAction{ + BaseElement: model.BaseElement{ID: nextID("jsa")}, + ContainerID: mod.ID, + Name: "Wait", + Parameters: []*types.JavaActionParameter{{ + Name: "Delay", + Description: "The number of milliseconds to wait.", + Category: "Timing", + IsRequired: true, + ParameterType: &types.IntegerType{}, + }}, + } + withContainer(h, stored.ContainerID, mod.ID) + var captured *types.JavaScriptAction + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListModulesFunc: func() ([]*model.Module, error) { return []*model.Module{mod}, nil }, + ListJavaScriptActionsFunc: func() ([]*types.JavaScriptAction, error) { return []*types.JavaScriptAction{stored}, nil }, + ReadJavaScriptActionByNameFunc: func(string) (*types.JavaScriptAction, error) { + return stored, nil + }, + UpdateJavaScriptActionFunc: func(jsa *types.JavaScriptAction) error { + captured = jsa + return nil + }, + WriteJavaScriptSourceFileFunc: func(string, string, string, []*types.JavaActionParameter, types.CodeActionReturnType) error { + return nil + }, + } + ctx, _ := newMockCtx(t, withBackend(mb), withHierarchy(h)) + assertNoError(t, execCreateJavaScriptAction(ctx, &ast.CreateJavaScriptActionStmt{ + Name: ast.QualifiedName{Module: "WebMod", Name: "Wait"}, + CreateOrModify: true, + Parameters: []ast.JavaActionParam{ + {Name: "Delay", Type: ast.DataType{Kind: ast.TypeInteger}, IsRequired: true}, + {Name: "Fresh", Type: ast.DataType{Kind: ast.TypeString}}, + }, + ReturnType: ast.DataType{Kind: ast.TypeVoid}, + JavaScriptCode: "\n" + jsSourceOmittedBody + "\n", + })) + if captured == nil || len(captured.Parameters) != 2 { + t.Fatalf("UpdateJavaScriptAction not called with two parameters: %+v", captured) + } + if p := captured.Parameters[0]; p.Description != "The number of milliseconds to wait." || p.Category != "Timing" { + t.Errorf("Delay: description %q, category %q — not carried from the stored parameter", p.Description, p.Category) + } + // Control: a parameter with no stored twin gets none. + if p := captured.Parameters[1]; p.Description != "" || p.Category != "" { + t.Errorf("Fresh: description %q, category %q — a new parameter inherited a description", p.Description, p.Category) + } +} diff --git a/mdl/executor/cmd_javascript_actions_source_keep_test.go b/mdl/executor/cmd_javascript_actions_source_keep_test.go new file mode 100644 index 0000000000..e79d5219d3 --- /dev/null +++ b/mdl/executor/cmd_javascript_actions_source_keep_test.go @@ -0,0 +1,115 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "os" + "path/filepath" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/backend/mock" + "github.com/mendixlabs/mxcli/mdl/types" + "github.com/mendixlabs/mxcli/model" +) + +// A Studio Pro .js source holds three things MDL cannot print: the import +// list, the EXTRA CODE section (describe shows it only as `--` comments) and +// the function signature Studio Pro generated. Regenerating the file from the +// statement drops all three — NanoflowCommons.GetStraightLineDistance lost +// `import { Big }` and the deg2rad helpers its user code calls. +const storedGetDistanceJS = `// This file was generated by Mendix Studio Pro. +import { Big } from "big.js"; + +// BEGIN EXTRA CODE +function deg2rad(deg) { + return deg.times(Math.PI / 180); +} +// END EXTRA CODE + +/** + * @param {Big} latitude + * @returns {Promise.<Big>} + */ +export async function GetDistance(latitude) { + // BEGIN USER CODE + return deg2rad(latitude); + // END USER CODE +} +` + +// Since describe emits `create or modify javascript action` (#731), running its +// output unchanged reaches the rewrite. A statement whose code is the stored +// user code, with the stored parameters, says nothing new about the source, so +// the file is left alone. Changing the code or the parameter list still +// regenerates it (the controls). +func TestJavaScriptActionRewrite_UnchangedCodeKeepsSourceFile(t *testing.T) { + dir := t.TempDir() + mprPath := filepath.Join(dir, "App.mpr") + jsDir := filepath.Join(dir, "javascriptsource", "webmod", "actions") + if err := os.MkdirAll(jsDir, 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(jsDir, "GetDistance.js"), []byte(storedGetDistanceJS), 0o644); err != nil { + t.Fatal(err) + } + + mod := mkModule("WebMod") + h := mkHierarchy(mod) + stored := &types.JavaScriptAction{ + BaseElement: model.BaseElement{ID: nextID("jsa")}, + ContainerID: mod.ID, + Name: "GetDistance", + Parameters: []*types.JavaActionParameter{{ + Name: "latitude", IsRequired: true, ParameterType: &types.DecimalType{}, + }}, + } + withContainer(h, stored.ContainerID, mod.ID) + wrote := 0 + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListModulesFunc: func() ([]*model.Module, error) { return []*model.Module{mod}, nil }, + ListJavaScriptActionsFunc: func() ([]*types.JavaScriptAction, error) { return []*types.JavaScriptAction{stored}, nil }, + ReadJavaScriptActionByNameFunc: func(string) (*types.JavaScriptAction, error) { return stored, nil }, + UpdateJavaScriptActionFunc: func(*types.JavaScriptAction) error { return nil }, + WriteJavaScriptSourceFileFunc: func(string, string, string, []*types.JavaActionParameter, types.CodeActionReturnType) error { + wrote++ + return nil + }, + } + ctx, _ := newMockCtx(t, withBackend(mb), withHierarchy(h), withMprPath(mprPath)) + + // The body exactly as describe prints it. + userCode, _ := readJavaScriptActionSource(mprPath, "WebMod", "GetDistance") + if userCode == "" { + t.Fatal("precondition: the stored user code was not read") + } + stmt := func(code string, params ...string) *ast.CreateJavaScriptActionStmt { + s := &ast.CreateJavaScriptActionStmt{ + Name: ast.QualifiedName{Module: "WebMod", Name: "GetDistance"}, + CreateOrModify: true, + ReturnType: ast.DataType{Kind: ast.TypeDecimal}, + JavaScriptCode: code, + } + for _, p := range params { + s.Parameters = append(s.Parameters, ast.JavaActionParam{Name: p, Type: ast.DataType{Kind: ast.TypeDecimal}, IsRequired: true}) + } + return s + } + + assertNoError(t, execCreateJavaScriptAction(ctx, stmt("\n"+userCode+"\n", "latitude"))) + if wrote != 0 { + t.Fatal("unchanged describe output regenerated the .js source, dropping its imports and EXTRA CODE") + } + + // Control: edited code is written. + assertNoError(t, execCreateJavaScriptAction(ctx, stmt("return latitude;", "latitude"))) + if wrote != 1 { + t.Errorf("edited code was not written (writes: %d)", wrote) + } + // Control: a changed parameter list regenerates the signature. + assertNoError(t, execCreateJavaScriptAction(ctx, stmt(userCode, "latitude", "longitude"))) + if wrote != 2 { + t.Errorf("a changed parameter list was not written (writes: %d)", wrote) + } +} diff --git a/mdl/executor/cmd_javascript_actions_write.go b/mdl/executor/cmd_javascript_actions_write.go index a3ff3102f3..16973201e4 100644 --- a/mdl/executor/cmd_javascript_actions_write.go +++ b/mdl/executor/cmd_javascript_actions_write.go @@ -5,6 +5,7 @@ package executor import ( "fmt" + "strings" "github.com/mendixlabs/mxcli/mdl/ast" mdlerrors "github.com/mendixlabs/mxcli/mdl/errors" @@ -54,6 +55,12 @@ func execCreateJavaScriptAction(ctx *ExecContext, s *ast.CreateJavaScriptActionS var existingActionInfo *types.MicroflowActionInfo var existingJSDoc string haveExistingJS := false + // A parameter's Description and Category have no MDL spelling — DESCRIBE + // prints the description as a comment — so a rewrite carries them from the + // stored parameter of the same name, as the Java twin does (#731). + storedParams := map[string]*types.JavaActionParameter{} + // The stored parameter names in order, for sourceUnchanged below. + var storedParamNames []string // No MDL spelling for either, so a rewrite carries them; the defaults are // Studio Pro's. "Public", which this hardcoded, is not a member of // JavaScriptActionsExportLevel (API | Hidden) — see the Java twin. @@ -79,6 +86,17 @@ func execCreateJavaScriptAction(ctx *ExecContext, s *ast.CreateJavaScriptActionS exportLevel = ex.ExportLevel } defaultReturnName = ex.ActionDefaultReturnName + // The list read may not carry parameter details; the full read does. + full := ex + if r, err := ctx.Backend.ReadJavaScriptActionByName(s.Name.Module + "." + s.Name.Name); err == nil && r != nil { + full = r + } + for _, p := range full.Parameters { + if p != nil { + storedParams[p.Name] = p + storedParamNames = append(storedParamNames, p.Name) + } + } } moduleID := containerID @@ -147,6 +165,10 @@ func execCreateJavaScriptAction(ctx *ExecContext, s *ast.CreateJavaScriptActionS default: p.ParameterType = astDataTypeToJavaActionParamType(param.Type) } + if sp := storedParams[param.Name]; sp != nil { + p.Description = sp.Description + p.Category = sp.Category + } jsa.Parameters = append(jsa.Parameters, p) } @@ -180,8 +202,22 @@ func execCreateJavaScriptAction(ctx *ExecContext, s *ast.CreateJavaScriptActionS } } - if err := ctx.Backend.WriteJavaScriptSourceFile(moduleName, s.Name.Name, s.JavaScriptCode, jsa.Parameters, jsa.ReturnType); err != nil { - return mdlerrors.NewBackend("write javascript source file", err) + // The placeholder DESCRIBE prints when it could not read the source is not + // code: writing it would overwrite the real implementation, or add a stub + // file to a project that ships none. The statement says nothing about the + // code, so the file is left alone (#731; the Java twin is #637). + // + // Likewise when the statement restates the stored user code with the stored + // parameters, as unchanged describe output does: regenerating the file from + // it drops the import list, the EXTRA CODE section and Studio Pro's + // signature, none of which MDL can print, so user code that calls an EXTRA + // CODE helper stops working (#731). + keep := strings.TrimSpace(s.JavaScriptCode) == jsSourceOmittedBody || + (existingID != "" && jsSourceUnchanged(ctx, s, storedParamNames)) + if !keep { + if err := ctx.Backend.WriteJavaScriptSourceFile(moduleName, s.Name.Name, s.JavaScriptCode, jsa.Parameters, jsa.ReturnType); err != nil { + return mdlerrors.NewBackend("write javascript source file", err) + } } ctx.InvalidateCache() @@ -232,3 +268,19 @@ func platformOrDefault(p string) string { } return p } + +// jsSourceUnchanged reports whether s restates the existing action's .js source: +// the same user code (as DESCRIBE reads it) and the same parameter names in +// order, so a regenerated file would differ only in what MDL cannot carry. +func jsSourceUnchanged(ctx *ExecContext, s *ast.CreateJavaScriptActionStmt, storedParamNames []string) bool { + if len(s.Parameters) != len(storedParamNames) { + return false + } + for i, p := range s.Parameters { + if p.Name != storedParamNames[i] { + return false + } + } + stored, _ := readJavaScriptActionSource(ctx.MprPath, s.Name.Module, s.Name.Name) + return stored != "" && strings.TrimSpace(stored) == strings.TrimSpace(s.JavaScriptCode) +} diff --git a/mdl/executor/cmd_microflows_builder_actions.go b/mdl/executor/cmd_microflows_builder_actions.go index ec6316e905..1e3bc4e855 100644 --- a/mdl/executor/cmd_microflows_builder_actions.go +++ b/mdl/executor/cmd_microflows_builder_actions.go @@ -1182,16 +1182,18 @@ func (fb *flowBuilder) addRetrieveAction(s *ast.RetrieveStmt) model.ID { EntityQualifiedName: entityQN, } - // Set range if LIMIT is specified - if s.Limit != "" { - rangeType := microflows.RangeTypeCustom - // LIMIT 1 with no offset uses RangeTypeFirst for single object retrieval - if s.Limit == "1" && s.Offset == "" { - rangeType = microflows.RangeTypeFirst + // The range. The visitor has already resolved what `limit 1` means in + // the script's language version (ako/mxcli#734): First is the object + // range, a limit or offset the Custom range, which is always a list. + if s.First { + dbSource.Range = &microflows.Range{ + BaseElement: model.BaseElement{ID: model.ID(types.GenerateID())}, + RangeType: microflows.RangeTypeFirst, } + } else if s.Limit != "" { dbSource.Range = &microflows.Range{ BaseElement: model.BaseElement{ID: model.ID(types.GenerateID())}, - RangeType: rangeType, + RangeType: microflows.RangeTypeCustom, Limit: s.Limit, Offset: s.Offset, } @@ -1306,14 +1308,12 @@ func (fb *flowBuilder) addRetrieveAction(s *ast.RetrieveStmt) model.ID { source = dbSource - // Register variable type for CHANGE statements - // RETRIEVE with LIMIT 1 returns a single entity, otherwise returns a List + // Register variable type for CHANGE statements: the object range binds + // one object, every other range a list. if fb.varTypes != nil { - if s.Limit == "1" { - // LIMIT 1 returns a single entity + if s.First { fb.varTypes[s.Variable] = entityQN } else { - // No LIMIT or LIMIT > 1 returns a list fb.varTypes[s.Variable] = "List of " + entityQN } } @@ -1803,10 +1803,12 @@ func (fb *flowBuilder) addListOperationAction(s *ast.ListOperationStmt) model.ID } func (fb *flowBuilder) listAttributeOperation(s *ast.ListOperationStmt, filter bool) microflows.ListOperation { - binary, ok := s.Condition.(*ast.BinaryExpr) - if !ok || binary.Operator != "=" { + // `find $L where …` / `filter $L where …` is always the by-expression + // operation, even when the expression happens to read `Member = value`. + if s.ByExpression || !ast.IsMemberEquality(s.Condition) { return nil } + binary := s.Condition.(*ast.BinaryExpr) fieldName, ok := listOperationFieldName(binary.Left) if !ok || fieldName == "" { return nil diff --git a/mdl/executor/cmd_microflows_builder_list_activity_test.go b/mdl/executor/cmd_microflows_builder_list_activity_test.go new file mode 100644 index 0000000000..90faa87665 --- /dev/null +++ b/mdl/executor/cmd_microflows_builder_list_activity_test.go @@ -0,0 +1,54 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/sdk/microflows" +) + +// `filter $L where Status = 'Open'` is Studio Pro's "Filter by expression" +// even though its expression reads `Member = value`; `filter $L by …` with the +// same condition is "Filter" by member (#733). The linking word decides, not +// the shape of the condition. +func TestFilterWhereIsAlwaysByExpression(t *testing.T) { + build := func(byExpression bool) microflows.ListOperation { + fb := predicateBuilder(t) + stmt := &ast.ListOperationStmt{ + Operation: ast.ListOpFilter, + InputVariable: "L", + OutputVariable: "R", + Condition: bare("Status", "=", "'Open'"), + ByExpression: byExpression, + } + oc := fb.buildFlowGraph([]ast.MicroflowStatement{stmt}, + &ast.MicroflowReturnType{Type: ast.DataType{Kind: ast.TypeBoolean}}) + if errs := fb.GetErrors(); len(errs) > 0 { + t.Fatalf("build errors: %v", errs) + } + for _, obj := range oc.Objects { + if act, ok := obj.(*microflows.ActionActivity); ok { + if lo, ok := act.Action.(*microflows.ListOperationAction); ok { + return lo.Operation + } + } + } + t.Fatal("no list operation built") + return nil + } + + op, ok := build(true).(*microflows.FilterOperation) + if !ok { + t.Fatalf("where: built %T, want *microflows.FilterOperation", build(true)) + } + if !strings.Contains(op.Expression, "$currentObject/Status") { + t.Errorf("where: expression %q does not read the member off $currentObject", op.Expression) + } + // Control: the same condition after `by` (or in the call form) is by member. + if got, ok := build(false).(*microflows.FilterByAttributeOperation); !ok || got.Attribute != "Shop.Order.Status" { + t.Errorf("by: built %#v, want Filter by attribute Shop.Order.Status", build(false)) + } +} diff --git a/mdl/executor/cmd_microflows_builder_validate.go b/mdl/executor/cmd_microflows_builder_validate.go index edef1c2b25..3768066005 100644 --- a/mdl/executor/cmd_microflows_builder_validate.go +++ b/mdl/executor/cmd_microflows_builder_validate.go @@ -302,7 +302,7 @@ func (fb *flowBuilder) validateStatement(stmt ast.MicroflowStatement) { if s.StartVariable != "" { // Association retrieve always returns a list fb.varTypes[s.Variable] = "List of " + s.Source.Module + "." + s.Source.Name - } else if s.Limit == "1" { + } else if s.First { fb.varTypes[s.Variable] = s.Source.Module + "." + s.Source.Name } else { fb.varTypes[s.Variable] = "List of " + s.Source.Module + "." + s.Source.Name diff --git a/mdl/executor/cmd_microflows_format_action.go b/mdl/executor/cmd_microflows_format_action.go index e8de99f433..b7600d7e35 100644 --- a/mdl/executor/cmd_microflows_format_action.go +++ b/mdl/executor/cmd_microflows_format_action.go @@ -10,6 +10,7 @@ import ( "sort" "strings" + "github.com/mendixlabs/mxcli/mdl/langver" "github.com/mendixlabs/mxcli/mdl/visitor" "github.com/mendixlabs/mxcli/model" "github.com/mendixlabs/mxcli/sdk/microflows" @@ -441,6 +442,9 @@ func formatAction( attrName = parts[len(parts)-1] } } + if describeLanguage(ctx) >= langver.V1 { + return formatAggregateActivity(ctx, a, fn, attrName, outputVar, entityNames) + } // REDUCE carries the fold Mendix stores beside the expression. Both parts // are required, so they are rendered even when empty rather than dropped — // a reduce that describes without them cannot be executed back (#1004). @@ -543,7 +547,11 @@ func formatAction( if dbSource.Range != nil { switch dbSource.Range.RangeType { case microflows.RangeTypeFirst: - stmt += "\n limit 1" + // `first`, not `limit 1`: under mdl 1 `limit 1` is a list of + // one, and `first` means the object in every version, so the + // output reads back to this range whatever the reader's + // header (ako/mxcli#734). + stmt += "\n first" case microflows.RangeTypeCustom: if dbSource.Range.Limit != "" { stmt += fmt.Sprintf("\n limit %s", dbSource.Range.Limit) @@ -1136,6 +1144,11 @@ func formatListOperation(ctx *ExecContext, op microflows.ListOperation, outputVa if op == nil { return fmt.Sprintf("$%s = list operation ...;", outputVar) } + if describeLanguage(ctx) >= langver.V1 { + if stmt, ok := formatListActivity(op, outputVar); ok { + return stmt + } + } switch o := op.(type) { case *microflows.HeadOperation: diff --git a/mdl/executor/cmd_microflows_format_action_test.go b/mdl/executor/cmd_microflows_format_action_test.go index 427f7337b0..792db3f71c 100644 --- a/mdl/executor/cmd_microflows_format_action_test.go +++ b/mdl/executor/cmd_microflows_format_action_test.go @@ -791,7 +791,7 @@ func TestFormatAction_Retrieve_WithLimit(t *testing.T) { }, } got := e.formatAction(action, nil, nil) - want := "retrieve $First from MyModule.Customer\n limit 1;" + want := "retrieve $First from MyModule.Customer\n first;" // the object range (#734) if got != want { t.Errorf("got %q, want %q", got, want) } @@ -868,7 +868,7 @@ func TestFormatAction_Retrieve_ReverseAssociationRequiresSimpleAllRange(t *testi } got := e.formatAction(action, nil, nil) - want := "retrieve $Domains from SampleRuntime.Domain\n where SampleRuntime.Domain_Runtime = $Runtime\n limit 1;" + want := "retrieve $Domains from SampleRuntime.Domain\n where SampleRuntime.Domain_Runtime = $Runtime\n first;" if got != want { t.Errorf("got %q, want %q", got, want) } diff --git a/mdl/executor/cmd_microflows_format_list_activity.go b/mdl/executor/cmd_microflows_format_list_activity.go new file mode 100644 index 0000000000..80f79a5c53 --- /dev/null +++ b/mdl/executor/cmd_microflows_format_list_activity.go @@ -0,0 +1,127 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "fmt" + "strings" + + "github.com/mendixlabs/mxcli/mdl/langver" + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/sdk/microflows" +) + +// describeLanguage is the MDL language version describe writes: the newest +// frozen version (the one whose header describe emits, langver.HeaderLine), +// or the version of the script the describe runs in when that is newer. +// +// While mdl 1 is a preview, a plain `describe` therefore keeps writing the +// mdl 0 forms, because its output carries no header that would make the newer +// forms mean what they say (ADR-0011). Inside an `mdl 1;` script it writes the +// mdl 1 forms, which the same script can execute back. +func describeLanguage(ctx *ExecContext) langver.Version { + v := langver.Frozen + if ctx != nil && ctx.LanguageVersion > v { + v = ctx.LanguageVersion + } + return v +} + +// formatListActivity renders a List operation activity as the statement that +// mirrors it (PROPOSAL_mdl_beta_syntax_freeze.md §4, #733). It returns false +// for an activity the statement form cannot express, which then falls back to +// the call form. +func formatListActivity(op microflows.ListOperation, outputVar string) (string, bool) { + out := "$" + outputVar + switch o := op.(type) { + case *microflows.HeadOperation: + return fmt.Sprintf("%s = head $%s;", out, o.ListVariable), true + case *microflows.TailOperation: + return fmt.Sprintf("%s = tail $%s;", out, o.ListVariable), true + case *microflows.FindOperation: + return fmt.Sprintf("%s = find $%s where %s;", out, o.ListVariable, describeExpr(o.Expression)), true + case *microflows.FilterOperation: + return fmt.Sprintf("%s = filter $%s where %s;", out, o.ListVariable, describeExpr(o.Expression)), true + case *microflows.FindByAttributeOperation: + return formatMemberListActivity("find", out, o.ListVariable, o.Attribute, o.Association, o.Expression) + case *microflows.FilterByAttributeOperation: + return formatMemberListActivity("filter", out, o.ListVariable, o.Attribute, o.Association, o.Expression) + case *microflows.SortOperation: + if len(o.Sorting) == 0 { + return "", false + } + items := make([]string, 0, len(o.Sorting)) + for _, s := range o.Sorting { + dir := "asc" + if s.Direction == microflows.SortDirectionDescending { + dir = "desc" + } + name := s.AttributeQualifiedName + if i := strings.LastIndex(name, "."); i >= 0 { + name = name[i+1:] + } + if name == "" { + return "", false + } + items = append(items, mdlIdent(name)+" "+dir) + } + return fmt.Sprintf("%s = sort $%s by %s;", out, o.ListVariable, strings.Join(items, ", ")), true + case *microflows.UnionOperation: + return fmt.Sprintf("%s = union $%s with $%s;", out, o.ListVariable1, o.ListVariable2), true + case *microflows.IntersectOperation: + return fmt.Sprintf("%s = intersect $%s with $%s;", out, o.ListVariable1, o.ListVariable2), true + case *microflows.SubtractOperation: + // Studio Pro's Subtract is the first list minus the second. + return fmt.Sprintf("%s = subtract $%s from $%s;", out, o.ListVariable2, o.ListVariable1), true + case *microflows.ContainsOperation: + return fmt.Sprintf("%s = contains $%s in $%s;", out, o.ObjectVariable, o.ListVariable), true + case *microflows.EqualsOperation: + return fmt.Sprintf("%s = equals $%s and $%s;", out, o.ListVariable1, o.ListVariable2), true + case *microflows.ListRangeOperation: + stmt := fmt.Sprintf("%s = range $%s", out, o.ListVariable) + if o.OffsetExpression != "" { + stmt += " offset " + describeExpr(o.OffsetExpression) + } + if o.LimitExpression != "" { + stmt += " limit " + describeExpr(o.LimitExpression) + } + return stmt + ";", true + } + return "", false +} + +// formatMemberListActivity renders Find / Filter by member: `by Member = value`. +func formatMemberListActivity(verb, out, list, attribute, association, value string) (string, bool) { + field := extractFieldName(attribute, association) + if value == "" { + return "", false + } + if field == "" { + return fmt.Sprintf("%s = %s $%s where %s;", out, verb, list, describeExpr(value)), true + } + return fmt.Sprintf("%s = %s $%s by %s = %s;", out, verb, list, field, describeExpr(value)), true +} + +// formatAggregateActivity renders an Aggregate list activity as the statement +// that mirrors it. fn is the function's MDL keyword and attrName the short +// name of the aggregated attribute, both already resolved by the caller. +func formatAggregateActivity(ctx *ExecContext, a *microflows.AggregateListAction, fn, attrName, outputVar string, entityNames map[model.ID]string) string { + out, list := "$"+outputVar, "$"+a.InputVariable + switch a.Function { + case microflows.AggregateFunctionCount: + return fmt.Sprintf("%s = count %s;", out, list) + case microflows.AggregateFunctionReduce: + initial := describeExpr(a.ReduceInitialValue) + if initial == "" { + initial = "empty" + } + return fmt.Sprintf("%s = reduce %s from %s as %s using %s;", out, list, initial, + formatMicroflowDataType(ctx, a.ReduceReturnType, entityNames), describeExpr(a.Expression)) + case microflows.AggregateFunctionAll, microflows.AggregateFunctionAny: + return fmt.Sprintf("%s = %s %s where %s;", out, fn, list, describeExpr(a.Expression)) + } + if a.UseExpression && a.Expression != "" { + return fmt.Sprintf("%s = %s %s of %s;", out, fn, list, describeExpr(a.Expression)) + } + return fmt.Sprintf("%s = %s %s by %s;", out, fn, list, mdlIdent(attrName)) +} diff --git a/mdl/executor/cmd_microflows_format_list_activity_test.go b/mdl/executor/cmd_microflows_format_list_activity_test.go new file mode 100644 index 0000000000..7a76cebae7 --- /dev/null +++ b/mdl/executor/cmd_microflows_format_list_activity_test.go @@ -0,0 +1,109 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "testing" + + "github.com/mendixlabs/mxcli/mdl/langver" + "github.com/mendixlabs/mxcli/mdl/visitor" + "github.com/mendixlabs/mxcli/sdk/microflows" +) + +// describe prints a List operation / Aggregate list activity as the statement +// that mirrors it when it describes under mdl 1, and keeps the call form while +// it describes under mdl 0 — which, while mdl 1 is a preview, is every describe +// outside an `mdl 1;` script (#733). +func TestDescribeListActivityUnderMdl1(t *testing.T) { + cases := []struct { + action microflows.MicroflowAction + mdl1 string + mdl0 string + wantKind string // the statement type the mdl 1 output parses back to + }{ + {listOp(&microflows.HeadOperation{ListVariable: "Orders"}, "First"), + "$First = head $Orders;", "$First = head($Orders);", "list"}, + {listOp(&microflows.TailOperation{ListVariable: "Orders"}, "Rest"), + "$Rest = tail $Orders;", "$Rest = tail($Orders);", "list"}, + {listOp(&microflows.FilterOperation{ListVariable: "Orders", Expression: "$currentObject/Total > 1000"}, "Big"), + "$Big = filter $Orders where $currentObject/Total > 1000;", "$Big = filter($Orders, $currentObject/Total > 1000);", "list"}, + {listOp(&microflows.FindOperation{ListVariable: "Orders", Expression: "$currentObject/Number < 3"}, "Late"), + "$Late = find $Orders where $currentObject/Number < 3;", "$Late = find($Orders, $currentObject/Number < 3);", "list"}, + {listOp(&microflows.FilterByAttributeOperation{ListVariable: "Orders", Attribute: "M.Order.Status", Expression: "M.Status.Open"}, "Open"), + "$Open = filter $Orders by Status = M.Status.Open;", "$Open = filter($Orders, Status = M.Status.Open);", "list"}, + {listOp(&microflows.FindByAttributeOperation{ListVariable: "Orders", Association: "M.Order_Customer", Expression: "$Customer"}, "Match"), + "$Match = find $Orders by Order_Customer = $Customer;", "$Match = find($Orders, Order_Customer = $Customer);", "list"}, + {listOp(&microflows.SortOperation{ListVariable: "Orders", Sorting: []*microflows.SortItem{ + {AttributeQualifiedName: "M.Order.Date", Direction: microflows.SortDirectionDescending}, + {AttributeQualifiedName: "M.Order.Number", Direction: microflows.SortDirectionAscending}, + }}, "Sorted"), + `$Sorted = sort $Orders by "Date" desc, Number asc;`, `$Sorted = sort($Orders, "Date" desc, Number asc);`, "list"}, + {listOp(&microflows.ListRangeOperation{ListVariable: "Orders", OffsetExpression: "20", LimitExpression: "10"}, "Page"), + "$Page = range $Orders offset 20 limit 10;", "$Page = range($Orders, 20, 10);", "list"}, + {listOp(&microflows.ListRangeOperation{ListVariable: "Orders", LimitExpression: "10"}, "Page"), + "$Page = range $Orders limit 10;", "$Page = range($Orders, 0, 10);", "list"}, + {listOp(&microflows.UnionOperation{ListVariable1: "A", ListVariable2: "B"}, "All"), + "$All = union $A with $B;", "$All = union($A, $B);", "list"}, + {listOp(&microflows.IntersectOperation{ListVariable1: "A", ListVariable2: "B"}, "Both"), + "$Both = intersect $A with $B;", "$Both = intersect($A, $B);", "list"}, + {listOp(&microflows.SubtractOperation{ListVariable1: "A", ListVariable2: "B"}, "Left"), + "$Left = subtract $B from $A;", "$Left = subtract($A, $B);", "list"}, + {listOp(&microflows.ContainsOperation{ListVariable: "Orders", ObjectVariable: "Order"}, "Has"), + "$Has = contains $Order in $Orders;", "$Has = contains($Orders, $Order);", "list"}, + {listOp(&microflows.EqualsOperation{ListVariable1: "A", ListVariable2: "B"}, "Same"), + "$Same = equals $A and $B;", "$Same = equals($A, $B);", "list"}, + {&microflows.AggregateListAction{InputVariable: "Orders", OutputVariable: "N", Function: microflows.AggregateFunctionCount}, + "$N = count $Orders;", "$N = count($Orders);", "aggregate"}, + {&microflows.AggregateListAction{InputVariable: "Orders", OutputVariable: "Total", Function: microflows.AggregateFunctionSum, AttributeQualifiedName: "M.Order.Amount"}, + "$Total = sum $Orders by Amount;", "$Total = sum($Orders.Amount);", "aggregate"}, + {&microflows.AggregateListAction{InputVariable: "Orders", OutputVariable: "Total", Function: microflows.AggregateFunctionAverage, UseExpression: true, Expression: "$currentObject/Price * 2"}, + "$Total = average $Orders of $currentObject/Price * 2;", "$Total = average($Orders, $currentObject/Price * 2);", "aggregate"}, + {&microflows.AggregateListAction{InputVariable: "Orders", OutputVariable: "Paid", Function: microflows.AggregateFunctionAll, UseExpression: true, Expression: "$currentObject/Paid"}, + "$Paid = all $Orders where $currentObject/Paid;", "$Paid = all($Orders, $currentObject/Paid);", "aggregate"}, + {&microflows.AggregateListAction{InputVariable: "Orders", OutputVariable: "Late", Function: microflows.AggregateFunctionAny, UseExpression: true, Expression: "$currentObject/Late"}, + "$Late = any $Orders where $currentObject/Late;", "$Late = any($Orders, $currentObject/Late);", "aggregate"}, + {&microflows.AggregateListAction{InputVariable: "Orders", OutputVariable: "Csv", Function: microflows.AggregateFunctionReduce, UseExpression: true, + Expression: "$currentResult + $currentObject/Name", ReduceInitialValue: "''", ReduceReturnType: &microflows.StringType{}}, + "$Csv = reduce $Orders from '' as String using $currentResult + $currentObject/Name;", + "$Csv = reduce($Orders, $currentResult + $currentObject/Name, initial: '', returns: String);", "aggregate"}, + } + for _, tc := range cases { + t.Run(tc.mdl1, func(t *testing.T) { + if got := formatAction(&ExecContext{LanguageVersion: langver.V1}, tc.action, nil, nil); got != tc.mdl1 { + t.Errorf("mdl 1 describe:\n got %s\n want %s", got, tc.mdl1) + } + // Control: outside an mdl 1 script describe keeps the call form. + for _, ctx := range []*ExecContext{nil, {LanguageVersion: langver.V0}} { + if got := formatAction(ctx, tc.action, nil, nil); got != tc.mdl0 { + t.Errorf("mdl 0 describe:\n got %s\n want %s", got, tc.mdl0) + } + } + + // The mdl 1 output parses back under the header, with nothing to warn about. + src := "mdl 1;\ncreate microflow M.A ($Orders: List of M.Order, $A: List of M.Order, $B: List of M.Order, " + + "$Order: M.Order, $Customer: M.Customer)\nbegin\n " + tc.mdl1 + "\nend;" + prog, errs := visitor.Build(src) + if len(errs) > 0 { + t.Fatalf("the mdl 1 describe output does not parse: %v", errs) + } + if len(prog.Deprecations) > 0 || len(prog.LanguageNotes) > 0 { + t.Errorf("the mdl 1 describe output warns: %v %v", prog.Deprecations, prog.LanguageNotes) + } + }) + } +} + +func listOp(op microflows.ListOperation, out string) *microflows.ListOperationAction { + return &microflows.ListOperationAction{Operation: op, OutputVariable: out} +} + +// The describe language is the newest frozen version unless the describe runs +// inside a script whose header names a newer one. +func TestDescribeLanguage(t *testing.T) { + if got := describeLanguage(nil); got != langver.Frozen { + t.Errorf("describeLanguage(nil) = %v, want the frozen %v", got, langver.Frozen) + } + if got := describeLanguage(&ExecContext{LanguageVersion: langver.V1}); got != langver.V1 { + t.Errorf("under mdl 1 = %v", got) + } +} diff --git a/mdl/executor/cmd_microflows_handles.go b/mdl/executor/cmd_microflows_handles.go index 2172169fa1..1c871e2415 100644 --- a/mdl/executor/cmd_microflows_handles.go +++ b/mdl/executor/cmd_microflows_handles.go @@ -88,9 +88,15 @@ func printedStatement(obj microflows.MicroflowObject, body []string, r elkSource return strings.Join(parts, " ") } default: - if strings.HasSuffix(line, ";") || strings.HasSuffix(line, "{") { + if strings.HasSuffix(line, ";") { return strings.Join(parts, " ") } + if strings.HasSuffix(line, "{") { + // The `{` opens the error handler block. It is not part of + // the statement: a handle is written as an alter target, and a + // target ends where a fragment's `{` begins. + return strings.TrimSpace(strings.TrimSuffix(strings.Join(parts, " "), "{")) + } } if len(parts) >= 50 { break diff --git a/mdl/executor/cmd_microflows_handles_test.go b/mdl/executor/cmd_microflows_handles_test.go index 056cdd56ec..10ae2ce01c 100644 --- a/mdl/executor/cmd_microflows_handles_test.go +++ b/mdl/executor/cmd_microflows_handles_test.go @@ -308,3 +308,18 @@ func TestDescribeWithHandles_ErrorHandlerBody(t *testing.T) { t.Errorf("with handles minus the handle lines differs from plain describe:\n%s\n---\n%s", strings.Join(stripped, "\n"), strings.Join(plain, "\n")) } } + +// A handle has to be writable as an `alter microflow` target, and a target +// ends at the `{` that opens a fragment. So the `{` describe prints after an +// activity with a custom error handler is not part of its statement. +func TestPrintedStatement_ErrorHandlerBlockOpenerIsNotPartOfTheStatement(t *testing.T) { + body := []string{" commit $Order on error {", " return false;", " };"} + obj := &microflows.ActionActivity{} + got := printedStatement(obj, body, elkSourceRange{StartLine: 0, EndLine: 2}) + if got != "commit $Order on error" { + t.Errorf("printed statement %q, want %q", got, "commit $Order on error") + } + if _, err := mfmutator.ParseTarget(got); err != nil { + t.Errorf("the handle does not parse as a target: %v", err) + } +} diff --git a/mdl/executor/cmd_microflows_retrieve_first_test.go b/mdl/executor/cmd_microflows_retrieve_first_test.go new file mode 100644 index 0000000000..d987248345 --- /dev/null +++ b/mdl/executor/cmd_microflows_retrieve_first_test.go @@ -0,0 +1,148 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/visitor" + "github.com/mendixlabs/mxcli/sdk/microflows" +) + +// `retrieve … first` vs `limit 1` (ako/mxcli#734): the writer stores the range +// the visitor resolved, and describe spells the object range `first`, which +// means the same in every language version. + +// retrieveRangeOf builds a retrieve from MDL source and returns the range the +// writer stores and the type the builder gives the output variable. +func retrieveRangeOf(t *testing.T, src string) (*microflows.Range, string) { + t.Helper() + prog, errs := visitor.Build(src) + if len(errs) > 0 { + t.Fatalf("%q: %v", src, errs) + } + var mf *ast.CreateMicroflowStmt + for _, s := range prog.Statements { + if m, ok := s.(*ast.CreateMicroflowStmt); ok { + mf = m + } + } + if mf == nil { + t.Fatalf("%q: no microflow", src) + } + r := mf.Body[0].(*ast.RetrieveStmt) + fb := &flowBuilder{varTypes: map[string]string{}} + fb.addRetrieveAction(r) + if len(fb.errors) > 0 { + t.Fatalf("%q: builder errors: %v", src, fb.errors) + } + act := fb.objects[0].(*microflows.ActionActivity).Action.(*microflows.RetrieveAction) + return act.Source.(*microflows.DatabaseRetrieveSource).Range, fb.varTypes[r.Variable] +} + +func TestRetrieveRange_WrittenPerLanguageVersion(t *testing.T) { + const body = "create microflow M.F () begin\n retrieve $O from M.Order where [Status = 'Open'] %s;\nend;" + cases := []struct { + name, src string + wantType microflows.RangeType + wantLimit string + wantObjectTyped bool + }{ + {"first, mdl 0", strings.Replace(body, "%s", "first", 1), microflows.RangeTypeFirst, "", true}, + {"first, mdl 1", "mdl 1;\n" + strings.Replace(body, "%s", "first", 1), microflows.RangeTypeFirst, "", true}, + {"limit 1, mdl 0 (alpha meaning)", strings.Replace(body, "%s", "limit 1", 1), microflows.RangeTypeFirst, "", true}, + {"limit 1, mdl 1 (a list of one)", "mdl 1;\n" + strings.Replace(body, "%s", "limit 1", 1), microflows.RangeTypeCustom, "1", false}, + {"limit 2, mdl 0", strings.Replace(body, "%s", "limit 2", 1), microflows.RangeTypeCustom, "2", false}, + // The offset makes it a Custom range; the variable type used to say + // object here while the writer stored a list. + {"limit 1 offset 3, mdl 0", strings.Replace(body, "%s", "limit 1 offset 3", 1), microflows.RangeTypeCustom, "1", false}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + rng, varType := retrieveRangeOf(t, tc.src) + if rng == nil || rng.RangeType != tc.wantType || rng.Limit != tc.wantLimit { + t.Fatalf("range = %+v, want %s with limit %q", rng, tc.wantType, tc.wantLimit) + } + if isObject := !strings.HasPrefix(varType, "List of "); isObject != tc.wantObjectTyped { + t.Errorf("output typed %q; want object=%v", varType, tc.wantObjectTyped) + } + }) + } +} + +// describe prints the object range as `first` (not `limit 1`, which is a list +// under mdl 1) and a Custom range as limit/offset, so its output reads back to +// the same range. +func TestFormatAction_Retrieve_RangeSpelling(t *testing.T) { + e := newTestExecutor() + cases := []struct { + rng *microflows.Range + want string + }{ + {&microflows.Range{RangeType: microflows.RangeTypeFirst}, "retrieve $O from M.Order\n first;"}, + {&microflows.Range{RangeType: microflows.RangeTypeCustom, Limit: "1"}, "retrieve $O from M.Order\n limit 1;"}, + {&microflows.Range{RangeType: microflows.RangeTypeCustom, Limit: "10", Offset: "20"}, "retrieve $O from M.Order\n limit 10\n offset 20;"}, + } + for _, tc := range cases { + got := e.formatAction(&microflows.RetrieveAction{ + OutputVariable: "O", + Source: &microflows.DatabaseRetrieveSource{EntityQualifiedName: "M.Order", Range: tc.rng}, + }, nil, nil) + if got != tc.want { + t.Errorf("%+v: got %q, want %q", tc.rng, got, tc.want) + } + } +} + +// What check reports for each spelling in each version: the MDL-V1-LIMIT1 +// warning on an mdl 0 `limit 1`, and MDL-RETRIEVE01 wherever the variable is an +// object used as a list. +func TestRetrieveRange_CheckPerLanguageVersion(t *testing.T) { + // A loop rather than count(): it is a list use in both versions, where the + // call form of an aggregate is not (ako/mxcli#733). + const body = `create microflow M.F () +begin + retrieve $O from M.Order where [Status = 'Open'] %s; + loop $Item in $O + begin + log info node 'F' 'item'; + end loop; +end;` + cases := []struct { + name, header, clause string + wantLimitOneWarning bool + wantRetrieve01 bool + }{ + {"mdl 0 limit 1: object, warned, and looping over it is CE0097", "", "limit 1", true, true}, + {"mdl 1 limit 1: a list of one, nothing to report", "mdl 1;\n", "limit 1", false, false}, + {"mdl 0 first: object, looping over it is CE0097", "", "first", false, true}, + {"mdl 1 first: object, looping over it is CE0097", "mdl 1;\n", "first", false, true}, + {"mdl 0 limit 2: a list", "", "limit 2", false, false}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + src := tc.header + strings.Replace(body, "%s", tc.clause, 1) + prog, errs := visitor.Build(src) + if len(errs) > 0 { + t.Fatalf("parse: %v", errs) + } + gotWarn, gotR01 := false, false + for _, v := range ValidateProgram(prog, "") { + switch v.RuleID { + case "MDL-V1-LIMIT1": + gotWarn = true + case retrieveSingleRule: + gotR01 = true + } + } + if gotWarn != tc.wantLimitOneWarning { + t.Errorf("MDL-V1-LIMIT1 reported = %v, want %v", gotWarn, tc.wantLimitOneWarning) + } + if gotR01 != tc.wantRetrieve01 { + t.Errorf("%s reported = %v, want %v", retrieveSingleRule, gotR01, tc.wantRetrieve01) + } + }) + } +} diff --git a/mdl/executor/cmd_microflows_show.go b/mdl/executor/cmd_microflows_show.go index 4ef3c8b835..f0a32b6e0e 100644 --- a/mdl/executor/cmd_microflows_show.go +++ b/mdl/executor/cmd_microflows_show.go @@ -368,8 +368,6 @@ func describeMicroflowMode(ctx *ExecContext, name ast.QualifiedName, opts descri name.Module, name.Name, strings.Join(roles, ", "))) } - lines = append(lines, "/") - // Output fmt.Fprintln(ctx.Output, strings.Join(lines, "\n")) return nil @@ -501,7 +499,6 @@ func describeNanoflow(ctx *ExecContext, name ast.QualifiedName) error { } lines = append(lines, "end;") - lines = append(lines, "/") fmt.Fprintln(ctx.Output, strings.Join(lines, "\n")) return nil @@ -717,8 +714,6 @@ func renderMicroflowMDL( flowType, name.Module, name.Name, strings.Join(roles, ", "))) } - lines = append(lines, "/") - return strings.Join(lines, "\n") } @@ -1682,7 +1677,6 @@ func describeRule(ctx *ExecContext, name ast.QualifiedName) error { } lines = append(lines, "end;") - lines = append(lines, "/") fmt.Fprintln(ctx.Output, strings.Join(lines, "\n")) return nil diff --git a/mdl/executor/cmd_misc.go b/mdl/executor/cmd_misc.go index 9c36cba2b8..5b0cf3e152 100644 --- a/mdl/executor/cmd_misc.go +++ b/mdl/executor/cmd_misc.go @@ -231,8 +231,8 @@ Code Search (requires refresh catalog full): show context of Module.Element [depth n]; -- Assemble context for LLM Security - Roles: - create module role Module.Role [description 'text']; - drop module role Module.Role; + create [or modify] module role Module.Role [description 'text']; + drop module role [if exists] Module.Role; create user role Name (Module.Role [, ...]) [manage all roles]; alter user role Name add module roles (Module.Role [, ...]); alter user role Name remove module roles (Module.Role [, ...]); @@ -271,7 +271,7 @@ Navigation: show navigation menu [Profile]; show navigation homes; describe navigation Profile; - create or replace navigation Profile + create or modify navigation Profile home page Module.Page [home page Module.Page for UserRole] [login page Module.Page] diff --git a/mdl/executor/cmd_modules.go b/mdl/executor/cmd_modules.go index 5a84ce892b..199d9646f6 100644 --- a/mdl/executor/cmd_modules.go +++ b/mdl/executor/cmd_modules.go @@ -612,7 +612,7 @@ func describeModule(ctx *ExecContext, moduleName string, withAll bool) error { } // Output basic CREATE MODULE statement - fmt.Fprintf(ctx.Output, "create module %s;\n", targetModule.Name) + fmt.Fprintf(ctx.Output, "create or modify module %s;\n", targetModule.Name) // Module roles live in the module's own Security$ModuleSecurity unit rather // than in any document, so a sweep that describes every document in a module @@ -621,7 +621,6 @@ func describeModule(ctx *ExecContext, moduleName string, withAll bool) error { describeModuleRoles(ctx, targetModule) if !withAll { - fmt.Fprintln(ctx.Output, "/") return nil } @@ -832,7 +831,6 @@ func describeModule(ctx *ExecContext, moduleName string, withAll bool) error { } } - fmt.Fprintln(ctx.Output, "/") return nil } diff --git a/mdl/executor/cmd_modules_security_test.go b/mdl/executor/cmd_modules_security_test.go index 1ee2928a9f..af2b278f5e 100644 --- a/mdl/executor/cmd_modules_security_test.go +++ b/mdl/executor/cmd_modules_security_test.go @@ -81,5 +81,5 @@ func TestDescribeModule_SurvivesUnreadableSecurity(t *testing.T) { ctx, buf := newMockCtx(t, withBackend(mb)) assertNoError(t, describeModule(ctx, "Administration", false)) - assertContainsStr(t, buf.String(), "create module Administration;") + assertContainsStr(t, buf.String(), "create or modify module Administration;") } diff --git a/mdl/executor/cmd_navigation.go b/mdl/executor/cmd_navigation.go index 1052d87b3a..02f929b008 100644 --- a/mdl/executor/cmd_navigation.go +++ b/mdl/executor/cmd_navigation.go @@ -314,7 +314,7 @@ func outputNavigationProfile(ctx *ExecContext, p *types.NavigationProfile) { fmt.Fprintf(ctx.Output, "-- Native: Yes\n") } - fmt.Fprintf(ctx.Output, "create or replace navigation %s\n", p.Name) + fmt.Fprintf(ctx.Output, "create or modify navigation %s\n", p.Name) // Home page if p.HomePage != nil { diff --git a/mdl/executor/cmd_navigation_mock_test.go b/mdl/executor/cmd_navigation_mock_test.go index a25d98d81a..806b3d77ad 100644 --- a/mdl/executor/cmd_navigation_mock_test.go +++ b/mdl/executor/cmd_navigation_mock_test.go @@ -88,7 +88,7 @@ func TestDescribeNavigation_Mock(t *testing.T) { } ctx, buf := newMockCtx(t, withBackend(mb)) assertNoError(t, describeNavigation(ctx, ast.QualifiedName{Name: "Responsive"})) - assertContainsStr(t, buf.String(), "create or replace navigation") + assertContainsStr(t, buf.String(), "create or modify navigation") } func TestDescribeNavigation_NotFound(t *testing.T) { diff --git a/mdl/executor/cmd_odata.go b/mdl/executor/cmd_odata.go index 6040e4c6a7..b4cd189314 100644 --- a/mdl/executor/cmd_odata.go +++ b/mdl/executor/cmd_odata.go @@ -134,6 +134,11 @@ func outputConsumedODataServiceMDL(ctx *ExecContext, svc *model.ConsumedODataSer outputJavadoc(ctx.Output, svc.Description) } + // describe keeps a plain `create` here, not `create or modify` (ADR-0012: carry or refuse, + // never silently drop). A `create or modify` rewrite of this type does not yet carry what + // describe cannot print, so re-running this output on an existing document would silently + // lose Studio Pro-authored content; a plain `create` refuses instead. Switch the verb only + // once the rewrite's carry is proven by the round-trip harness (see #743). fmt.Fprintf(ctx.Output, "create odata client %s.%s (\n", moduleName, svc.Name) var props []string @@ -230,8 +235,6 @@ func outputConsumedODataServiceMDL(ctx *ExecContext, svc *model.ConsumedODataSer fmt.Fprintln(ctx.Output, ");") } - fmt.Fprintln(ctx.Output, "/") - return nil } @@ -328,6 +331,7 @@ func outputPublishedODataServiceMDL(ctx *ExecContext, svc *model.PublishedODataS outputJavadoc(ctx.Output, svc.Description) } + // Plain `create` on purpose; see describeODataClient. fmt.Fprintf(ctx.Output, "create odata service %s.%s (\n", moduleName, svc.Name) var props []string @@ -362,21 +366,34 @@ func outputPublishedODataServiceMDL(ctx *ExecContext, svc *model.PublishedODataS } fmt.Fprintln(ctx.Output, strings.Join(props, ",\n")) - fmt.Fprintln(ctx.Output, ")") + // The statement ends with `;` after whichever clause comes last: the + // property list, the authentication clause or the entity block (ADR-0010 + // R11). + hasBlock := len(svc.EntityTypes) > 0 || len(svc.EntitySets) > 0 || len(svc.Microflows) > 0 + hasAuth := len(svc.AuthenticationTypes) > 0 || svc.AuthMicroflow != "" + if hasBlock || hasAuth { + fmt.Fprintln(ctx.Output, ")") + } else { + fmt.Fprintln(ctx.Output, ");") + } // Authentication types. The custom-authentication microflow is part of the // clause, not a comment beside it: emitted as a comment the output looked // complete but replayed into a service Mendix rejects with CE0333 // (mxcli-formula1 §40). + authEnd := "\n" + if !hasBlock { + authEnd = ";\n" + } if len(svc.AuthenticationTypes) > 0 { - fmt.Fprintf(ctx.Output, "authentication %s\n", odataAuthClause(svc)) + fmt.Fprintf(ctx.Output, "authentication %s%s", odataAuthClause(svc), authEnd) } else if svc.AuthMicroflow != "" { // A microflow with no type recorded still has to survive the round trip. - fmt.Fprintf(ctx.Output, "authentication microflow %s\n", svc.AuthMicroflow) + fmt.Fprintf(ctx.Output, "authentication microflow %s%s", svc.AuthMicroflow, authEnd) } // Published entities block - if len(svc.EntityTypes) > 0 || len(svc.EntitySets) > 0 || len(svc.Microflows) > 0 { + if hasBlock { fmt.Fprintln(ctx.Output, "{") // Build entity set lookup by exposed name and entity type name for merging @@ -501,7 +518,7 @@ func outputPublishedODataServiceMDL(ctx *ExecContext, svc *model.PublishedODataS printPublishedMicroflowMDL(ctx.Output, pm) } - fmt.Fprintln(ctx.Output, "}") + fmt.Fprintln(ctx.Output, "};") } // Output GRANT statements for allowed module roles @@ -511,8 +528,6 @@ func outputPublishedODataServiceMDL(ctx *ExecContext, svc *model.PublishedODataS moduleName, svc.Name, strings.Join(svc.AllowedModuleRoles, ", ")) } - fmt.Fprintln(ctx.Output, "/") - return nil } @@ -757,6 +772,7 @@ func outputExternalEntityMDL(ctx *ExecContext, entity *domainmodel.Entity, modul outputJavadoc(ctx.Output, entity.Documentation) } + // Plain `create` on purpose; see describeODataClient. fmt.Fprintf(ctx.Output, "create external entity %s.%s\n", moduleName, entity.Name) fmt.Fprintf(ctx.Output, "from odata client %s\n", entity.RemoteServiceName) fmt.Fprintln(ctx.Output, "(") @@ -781,27 +797,26 @@ func outputExternalEntityMDL(ctx *ExecContext, entity *domainmodel.Entity, modul props = append(props, fmt.Sprintf(" AllowCreateChangeLocally: %s", boolStr(entity.CreateChangeLocally))) fmt.Fprintln(ctx.Output, strings.Join(props, ",\n")) + // Output attributes. Without them the statement ends at the property list, + // and still ends with `;` (ADR-0010 R11). + if len(entity.Attributes) == 0 { + fmt.Fprintln(ctx.Output, ");") + return nil + } fmt.Fprintln(ctx.Output, ")") - - // Output attributes - if len(entity.Attributes) > 0 { - fmt.Fprintln(ctx.Output, "(") - for i, attr := range entity.Attributes { - typeName := "Unknown" - if attr.Type != nil { - typeName = attr.Type.GetTypeName() - } - comma := "," - if i == len(entity.Attributes)-1 { - comma = "" - } - fmt.Fprintf(ctx.Output, " %s: %s%s\n", attr.Name, typeName, comma) + fmt.Fprintln(ctx.Output, "(") + for i, attr := range entity.Attributes { + typeName := "Unknown" + if attr.Type != nil { + typeName = attr.Type.GetTypeName() } - fmt.Fprintln(ctx.Output, ");") + comma := "," + if i == len(entity.Attributes)-1 { + comma = "" + } + fmt.Fprintf(ctx.Output, " %s: %s%s\n", attr.Name, typeName, comma) } - - fmt.Fprintln(ctx.Output, "/") - + fmt.Fprintln(ctx.Output, ");") return nil } diff --git a/mdl/executor/cmd_odata_client_describe_quoting_test.go b/mdl/executor/cmd_odata_client_describe_quoting_test.go index abc0c1e7e6..42119659d7 100644 --- a/mdl/executor/cmd_odata_client_describe_quoting_test.go +++ b/mdl/executor/cmd_odata_client_describe_quoting_test.go @@ -35,6 +35,7 @@ func describeAndReparse(t *testing.T, stored *model.ConsumedODataService, folder t.Fatalf("describe: %v", err) } prog := parseMDL(t, out.String()) + assertTerminated(t, out.String()) // #744 for _, s := range prog.Statements { if c, ok := s.(*ast.CreateODataClientStmt); ok { return c, out.String() diff --git a/mdl/executor/cmd_odata_mock_test.go b/mdl/executor/cmd_odata_mock_test.go index dd2214b158..21364836c1 100644 --- a/mdl/executor/cmd_odata_mock_test.go +++ b/mdl/executor/cmd_odata_mock_test.go @@ -96,6 +96,7 @@ func TestDescribeODataClient_Mock(t *testing.T) { out := buf.String() assertContainsStr(t, out, "create odata client") + assertNotContainsStr(t, out, "create or modify odata client") // safe refusal until its carry is proven (ADR-0012) assertContainsStr(t, out, "MyModule.PetStoreClient") assertContainsStr(t, out, "https://example.com/$metadata") assertContainsStr(t, out, "2.0") @@ -192,6 +193,7 @@ func TestDescribeODataService_Mock(t *testing.T) { out := buf.String() assertContainsStr(t, out, "create odata service") + assertNotContainsStr(t, out, "create or modify odata service") // safe refusal until its carry is proven (ADR-0012) assertContainsStr(t, out, "MyModule.CatalogService") } diff --git a/mdl/executor/cmd_pages_describe.go b/mdl/executor/cmd_pages_describe.go index 05b519b0dd..61de300fab 100644 --- a/mdl/executor/cmd_pages_describe.go +++ b/mdl/executor/cmd_pages_describe.go @@ -190,7 +190,7 @@ func describePage(ctx *ExecContext, name ast.QualifiedName) error { fmt.Fprint(ctx.Output, " }\n") } } - fmt.Fprint(ctx.Output, "}") + fmt.Fprint(ctx.Output, "};") // Add GRANT VIEW if roles are assigned if len(foundPage.AllowedRoles) > 0 { @@ -295,10 +295,10 @@ func describeSnippet(ctx *ExecContext, name ast.QualifiedName) error { for _, w := range rawWidgets { outputWidgetMDLV3(ctx, w, 1) } - fmt.Fprint(ctx.Output, "}") + fmt.Fprint(ctx.Output, "};") } else { // A widget-less snippet still needs an (empty) body block to re-parse (#626). - fmt.Fprint(ctx.Output, " {\n}") + fmt.Fprint(ctx.Output, " {\n};") } fmt.Fprint(ctx.Output, "\n") @@ -366,14 +366,14 @@ func describeLayout(ctx *ExecContext, name ast.QualifiedName) error { if foundLayout.Class != "" { header += fmt.Sprintf(",\n class: %s", mdlQuote(foundLayout.Class)) } - fmt.Fprintf(ctx.Output, "create layout %s.%s (\n%s\n) {\n", + fmt.Fprintf(ctx.Output, "create or modify layout %s.%s (\n%s\n) {\n", modName, mdlIdent(foundLayout.Name), header) for _, w := range getLayoutWidgetsFromRaw(ctx, foundLayout.ID) { outputWidgetMDLV3(ctx, w, 1) } - fmt.Fprint(ctx.Output, "}\n\n") + fmt.Fprint(ctx.Output, "};\n\n") return nil } diff --git a/mdl/executor/cmd_published_rest.go b/mdl/executor/cmd_published_rest.go index 8d9848d5fa..59e561ce5b 100644 --- a/mdl/executor/cmd_published_rest.go +++ b/mdl/executor/cmd_published_rest.go @@ -141,9 +141,10 @@ func describePublishedRestService(ctx *ExecContext, name ast.QualifiedName) erro } fmt.Fprintln(ctx.Output, "};") } else { - fmt.Fprintln(ctx.Output, ";") + // The resource block is not optional in the grammar: a service + // with no resources still needs an empty one to re-parse (#744). + fmt.Fprintln(ctx.Output, "{\n};") } - fmt.Fprintln(ctx.Output, "/") // Emit GRANT statements for any module roles with access. if len(svc.AllowedRoles) > 0 { diff --git a/mdl/executor/cmd_published_rest_mock_test.go b/mdl/executor/cmd_published_rest_mock_test.go index 99cc241bd3..ce5801c06c 100644 --- a/mdl/executor/cmd_published_rest_mock_test.go +++ b/mdl/executor/cmd_published_rest_mock_test.go @@ -65,6 +65,7 @@ func TestDescribePublishedRestService_Mock(t *testing.T) { assertNoError(t, describePublishedRestService(ctx, ast.QualifiedName{Module: "MyModule", Name: "OrderAPI"})) out := buf.String() + assertTerminated(t, out) // #744 assertContainsStr(t, out, "create or modify published rest service") assertContainsStr(t, out, "MyModule.OrderAPI") } diff --git a/mdl/executor/cmd_rest_clients.go b/mdl/executor/cmd_rest_clients.go index 1075b6334d..5ad7f3925b 100644 --- a/mdl/executor/cmd_rest_clients.go +++ b/mdl/executor/cmd_rest_clients.go @@ -121,7 +121,7 @@ func outputConsumedRestServiceMDL(ctx *ExecContext, svc *model.ConsumedRestServi outputJavadoc(w, svc.Documentation) } - fmt.Fprintf(w, "create rest client %s.%s (\n", moduleName, svc.Name) + fmt.Fprintf(w, "create or modify rest client %s.%s (\n", moduleName, svc.Name) fmt.Fprintf(w, " BaseUrl: %s,\n", mdlQuoted(svc.BaseUrl)) if svc.Authentication == nil { fmt.Fprintln(w, " Authentication: none") diff --git a/mdl/executor/cmd_rest_clients_mock_test.go b/mdl/executor/cmd_rest_clients_mock_test.go index 3c58332e06..b297212203 100644 --- a/mdl/executor/cmd_rest_clients_mock_test.go +++ b/mdl/executor/cmd_rest_clients_mock_test.go @@ -59,7 +59,7 @@ func TestDescribeRestClient_Mock(t *testing.T) { assertNoError(t, describeRestClient(ctx, ast.QualifiedName{Module: "MyModule", Name: "WeatherAPI"})) out := buf.String() - assertContainsStr(t, out, "create rest client") + assertContainsStr(t, out, "create or modify rest client") assertContainsStr(t, out, "MyModule.WeatherAPI") } diff --git a/mdl/executor/cmd_rest_openapi_mock_test.go b/mdl/executor/cmd_rest_openapi_mock_test.go index b611cbb892..20d3120d63 100644 --- a/mdl/executor/cmd_rest_openapi_mock_test.go +++ b/mdl/executor/cmd_rest_openapi_mock_test.go @@ -289,7 +289,7 @@ func TestDescribeContractFromOpenAPI_Mock(t *testing.T) { assertNoError(t, describeContractFromOpenAPI(ctx, stmt)) out := buf.String() - assertContainsStr(t, out, "create rest client") + assertContainsStr(t, out, "create or modify rest client") assertContainsStr(t, out, "https://api.example.com/v1") assertContainsStr(t, out, "listPets") } diff --git a/mdl/executor/cmd_security.go b/mdl/executor/cmd_security.go index 371b99e39f..1d4c3ba3aa 100644 --- a/mdl/executor/cmd_security.go +++ b/mdl/executor/cmd_security.go @@ -787,7 +787,6 @@ func describeModuleRole(ctx *ExecContext, name ast.QualifiedName) error { fmt.Fprintf(ctx.Output, " description %s", mdlQuoted(mr.Description)) } fmt.Fprintln(ctx.Output, ";") - fmt.Fprintln(ctx.Output, "/") // Show which user roles include this module role qualifiedRole := modName + "." + mr.Name @@ -837,7 +836,6 @@ func describeDemoUser(ctx *ExecContext, userName string) error { fmt.Fprintf(ctx.Output, " (%s)", strings.Join(du.UserRoles, ", ")) } fmt.Fprintln(ctx.Output, ";") - fmt.Fprintln(ctx.Output, "/") return nil } } @@ -854,7 +852,7 @@ func describeUserRole(ctx *ExecContext, name ast.QualifiedName) error { for _, ur := range ps.UserRoles { if ur.Name == name.Name { - fmt.Fprintf(ctx.Output, "create user role %s", ur.Name) + fmt.Fprintf(ctx.Output, "create or modify user role %s", ur.Name) // Module roles if len(ur.ModuleRoles) > 0 { @@ -866,7 +864,6 @@ func describeUserRole(ctx *ExecContext, name ast.QualifiedName) error { } fmt.Fprintln(ctx.Output, ";") - fmt.Fprintln(ctx.Output, "/") // Show description if present if ur.Description != "" { diff --git a/mdl/executor/cmd_security_mock_test.go b/mdl/executor/cmd_security_mock_test.go index 7b50c3345f..7fe26023cb 100644 --- a/mdl/executor/cmd_security_mock_test.go +++ b/mdl/executor/cmd_security_mock_test.go @@ -143,6 +143,7 @@ func TestDescribeModuleRole_Mock(t *testing.T) { ctx, buf := newMockCtx(t, withBackend(mb), withHierarchy(h)) assertNoError(t, describeModuleRole(ctx, ast.QualifiedName{Module: "MyModule", Name: "Admin"})) assertContainsStr(t, buf.String(), "create or modify module role") + assertTerminated(t, buf.String()) // #744 } func TestDescribeUserRole_Mock(t *testing.T) { @@ -158,7 +159,8 @@ func TestDescribeUserRole_Mock(t *testing.T) { } ctx, buf := newMockCtx(t, withBackend(mb)) assertNoError(t, describeUserRole(ctx, ast.QualifiedName{Name: "Administrator"})) - assertContainsStr(t, buf.String(), "create user role") + assertContainsStr(t, buf.String(), "create or modify user role") + assertTerminated(t, buf.String()) // #744 } func TestDescribeDemoUser_Mock(t *testing.T) { @@ -178,6 +180,7 @@ func TestDescribeDemoUser_Mock(t *testing.T) { // `create or modify` so the output replays onto a project that has the // user (ako/mxcli#707). assertContainsStr(t, buf.String(), "create or modify demo user") + assertTerminated(t, buf.String()) // #744 } func TestShowModuleRoles_Mock_FilterByModule(t *testing.T) { diff --git a/mdl/executor/cmd_security_write.go b/mdl/executor/cmd_security_write.go index 0092a89a87..31249e5f58 100644 --- a/mdl/executor/cmd_security_write.go +++ b/mdl/executor/cmd_security_write.go @@ -111,6 +111,10 @@ func execDropModuleRole(ctx *ExecContext, s *ast.DropModuleRoleStmt) error { } } if !found { + if s.IfExists { + fmt.Fprintf(ctx.Output, "Module role '%s' does not exist, skipping\n", s.Name) + return nil + } return mdlerrors.NewNotFound("module role", s.Name.Module+"."+s.Name.Name) } diff --git a/mdl/executor/cmd_validationrules.go b/mdl/executor/cmd_validationrules.go index 1fcf142e1a..d41573b39c 100644 --- a/mdl/executor/cmd_validationrules.go +++ b/mdl/executor/cmd_validationrules.go @@ -165,7 +165,7 @@ func outputEntityValidationRules(ctx *ExecContext, entity *domainmodel.Entity, m } feedback := pickTextTranslation(vr.ErrorMessage, lang) - fmt.Fprintf(ctx.Output, "\ncreate validation rule for %s\n %s\n feedback '%s';\n", + fmt.Fprintf(ctx.Output, "\ncreate or modify validation rule for %s\n %s\n feedback '%s';\n", target, constraint, escapeMDLString(feedback)) } } diff --git a/mdl/executor/cmd_workflows.go b/mdl/executor/cmd_workflows.go index ecdaa3a85e..80b18d9f8f 100644 --- a/mdl/executor/cmd_workflows.go +++ b/mdl/executor/cmd_workflows.go @@ -185,6 +185,11 @@ func describeWorkflowToString(ctx *ExecContext, name ast.QualifiedName) (string, } lines = append(lines, "") + // describe keeps a plain `create` here, not `create or modify` (ADR-0012: carry or refuse, + // never silently drop). A `create or modify` rewrite of this type does not yet carry what + // describe cannot print, so re-running this output on an existing document would silently + // lose Studio Pro-authored content; a plain `create` refuses instead. Switch the verb only + // once the rewrite's carry is proven by the round-trip harness (see #743). lines = append(lines, fmt.Sprintf("create workflow %s", qualifiedName)) if clause := describeFolderClause(ctx, targetWf.ContainerID); clause != "" { lines = append(lines, " "+strings.TrimSpace(clause)) @@ -232,8 +237,7 @@ func describeWorkflowToString(ctx *ExecContext, name ast.QualifiedName) (string, } lines = append(lines, formatEventSubProcesses(targetWf.EventSubProcesses, " ")...) - lines = append(lines, "end workflow") - lines = append(lines, "/") + lines = append(lines, "end workflow;") return strings.Join(lines, "\n"), nil, nil } diff --git a/mdl/executor/cmd_workflows_mock_test.go b/mdl/executor/cmd_workflows_mock_test.go index e3de134463..5b5e6ae3cd 100644 --- a/mdl/executor/cmd_workflows_mock_test.go +++ b/mdl/executor/cmd_workflows_mock_test.go @@ -48,7 +48,9 @@ func TestDescribeWorkflow_Mock(t *testing.T) { assertNoError(t, describeWorkflow(ctx, ast.QualifiedName{Module: "Sales", Name: "ApproveOrder"})) out := buf.String() + assertTerminated(t, out) // #744 assertContainsStr(t, out, "create workflow") + assertNotContainsStr(t, out, "create or modify workflow") // safe refusal until its carry is proven (ADR-0012) assertContainsStr(t, out, "Sales.ApproveOrder") // Roundtrip: DESCRIBE output must be parseable as valid MDL (issue #478) diff --git a/mdl/executor/describe_create_or_modify_test.go b/mdl/executor/describe_create_or_modify_test.go index dc11fbd848..31fd6a67c7 100644 --- a/mdl/executor/describe_create_or_modify_test.go +++ b/mdl/executor/describe_create_or_modify_test.go @@ -8,6 +8,7 @@ import ( "github.com/mendixlabs/mxcli/mdl/ast" "github.com/mendixlabs/mxcli/mdl/backend/mock" + "github.com/mendixlabs/mxcli/mdl/types" "github.com/mendixlabs/mxcli/mdl/visitor" "github.com/mendixlabs/mxcli/model" "github.com/mendixlabs/mxcli/sdk/javaactions" @@ -17,8 +18,11 @@ import ( // DESCRIBE describes a document that exists, so its output has to re-execute // against the project it came from. A plain `create` fails there with "already // exists" — the round trip dies at the first statement instead of reporting -// what changed (ako/mxcli#705 item 5). Every other describer already emits -// `create or modify`; these three did not. +// what changed (ako/mxcli#705 item 5). #731 moved the rest of the describers +// over (JavaScript action, user role, navigation, layout, REST and OData +// clients, OData service, external entity, database connection, data +// transformer, agent-editor documents, workflow, validation rule, module); their +// mock tests assert the spelling, and the three below also parse it. // // Each test parses the output rather than grepping it: what matters is the // statement the parser builds, not how the verb is spelled. @@ -111,6 +115,72 @@ func TestDescribeModule_RolesEmitCreateOrModify(t *testing.T) { } } +func TestDescribeJavaScriptAction_EmitsCreateOrModify(t *testing.T) { + mod := mkModule("WebMod") + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ReadJavaScriptActionByNameFunc: func(string) (*types.JavaScriptAction, error) { + return &types.JavaScriptAction{ + BaseElement: model.BaseElement{ID: nextID("jsa")}, + ContainerID: mod.ID, + Name: "ShowAlert", + }, nil + }, + } + ctx, buf := newMockCtx(t, withBackend(mb)) + assertNoError(t, describeJavaScriptAction(ctx, ast.QualifiedName{Module: "WebMod", Name: "ShowAlert"})) + + stmt := firstStatement[*ast.CreateJavaScriptActionStmt](t, buf.String()) + if !stmt.CreateOrModify { + t.Errorf("describe javascript action emitted a plain create, which fails on the action it describes:\n%s", buf.String()) + } +} + +func TestDescribeUserRole_EmitsCreateOrModify(t *testing.T) { + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + GetProjectSecurityFunc: func() (*security.ProjectSecurity, error) { + return &security.ProjectSecurity{UserRoles: []*security.UserRole{ + {Name: "Administrator", ModuleRoles: []string{"MyModule.Admin"}}, + }}, nil + }, + } + ctx, buf := newMockCtx(t, withBackend(mb)) + assertNoError(t, describeUserRole(ctx, ast.QualifiedName{Name: "Administrator"})) + + stmt := firstStatement[*ast.CreateUserRoleStmt](t, buf.String()) + if !stmt.CreateOrModify { + t.Errorf("describe user role emitted a plain create, which fails on the role it describes:\n%s", buf.String()) + } +} + +// Navigation printed `create or replace`, the deprecated spelling (MDL-DEPR001): +// describe output must be canonical, so it warns on nothing. +func TestDescribeNavigation_EmitsCreateOrModify(t *testing.T) { + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + GetNavigationFunc: func() (*types.NavigationDocument, error) { + return &types.NavigationDocument{Profiles: []*types.NavigationProfile{{ + Name: "Responsive", Kind: "Responsive", HomePage: &types.NavHomePage{Page: "MyModule.Home"}, + }}}, nil + }, + } + ctx, buf := newMockCtx(t, withBackend(mb)) + assertNoError(t, describeNavigation(ctx, ast.QualifiedName{Name: "Responsive"})) + + prog, errs := visitor.Build(buf.String()) + if len(errs) > 0 { + t.Fatalf("describe navigation emitted MDL the parser rejects: %v\n%s", errs, buf.String()) + } + if len(prog.Deprecations) != 0 { + t.Errorf("describe navigation emitted a deprecated spelling %+v:\n%s", prog.Deprecations, buf.String()) + } + stmt := firstStatement[*ast.AlterNavigationStmt](t, buf.String()) + if !stmt.CreateOrModify { + t.Errorf("describe navigation is not an idempotent create:\n%s", buf.String()) + } +} + func firstStatement[T ast.Statement](t *testing.T, mdl string) T { t.Helper() prog, errs := visitor.Build(mdl) diff --git a/mdl/executor/describe_terminators_test.go b/mdl/executor/describe_terminators_test.go new file mode 100644 index 0000000000..8c7aaa5f7f --- /dev/null +++ b/mdl/executor/describe_terminators_test.go @@ -0,0 +1,133 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "bytes" + "fmt" + "testing" + + "github.com/antlr4-go/antlr/v4" + + "github.com/mendixlabs/mxcli/mdl/grammar/parser" + "github.com/mendixlabs/mxcli/mdl/visitor" + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/sdk/domainmodel" +) + +// ako/mxcli#744 — describe ends every statement with `;` and never prints a +// SQL*Plus `/` line (ADR-0010 R11/R12), so its output means the same under +// mdl 0 and mdl 1. testdata/pedapp covers the document types it contains +// (mdl/roundtrip TestPedAppDescribeIsValidMdl1); the tests here cover the rest. + +// assertTerminated fails the test when a statement of describe output does not +// end with `;`, is followed by `/`, or when the output does not parse under an +// `mdl 1;` header. +func assertTerminated(t *testing.T, out string) { + t.Helper() + for _, p := range terminatorProblems(out) { + t.Errorf("%s\n--- describe output ---\n%s", p, out) + } + if _, errs := visitor.Build("mdl 1;\n" + out); len(errs) > 0 { + t.Errorf("describe output is not valid under mdl 1: %v\n--- describe output ---\n%s", errs, out) + } +} + +// terminatorProblems reports each top-level statement that does not end with +// `;` or is followed by `/`. Microflow, nanoflow and workflow rules end in +// `SEMICOLON? SLASH?` of their own, so the terminators are read off the +// statement's last tokens rather than the statement rule's own. +func terminatorProblems(script string) []string { + lexer := parser.NewMDLLexer(antlr.NewInputStream(script)) + lexer.RemoveErrorListeners() + p := parser.NewMDLParser(antlr.NewCommonTokenStream(lexer, antlr.TokenDefaultChannel)) + p.RemoveErrorListeners() + var problems []string + for _, stmt := range p.Program().AllStatement() { + last := lastTokens(stmt, 2) + if len(last) == 0 { + continue + } + if last[0].GetTokenType() == parser.MDLParserSLASH { + problems = append(problems, fmt.Sprintf("line %d: a `/` line follows the statement", last[0].GetLine())) + last = last[1:] + } + if len(last) > 0 && last[0].GetTokenType() != parser.MDLParserSEMICOLON { + problems = append(problems, fmt.Sprintf("line %d: the statement ending at %q has no `;`", last[0].GetLine(), last[0].GetText())) + } + } + return problems +} + +// lastTokens returns up to n of the tree's last tokens, the last first. +func lastTokens(tree antlr.Tree, n int) []antlr.Token { + var out []antlr.Token + var walk func(antlr.Tree) + walk = func(t antlr.Tree) { + if len(out) == n { + return + } + if tn, ok := t.(antlr.TerminalNode); ok { + if tok := tn.GetSymbol(); tok != nil && tok.GetTokenType() != antlr.TokenEOF { + out = append(out, tok) + } + return + } + for i := t.GetChildCount() - 1; i >= 0 && len(out) < n; i-- { + walk(t.GetChild(i)) + } + } + walk(tree) + return out +} + +// The control: without it a check that never fires passes every test below. +func TestTerminatorProblems_Control(t *testing.T) { + for script, want := range map[string]int{ + "create constant M.C type String default 'x';\n": 0, + "create constant M.C type String default 'x';\n/\n": 1, + "create constant M.C type String default 'x'\n": 1, + "create microflow M.F ()\nbegin\n return;\nend;\n/\n": 1, + "create page M.P (title: 'P', layout: Atlas_Core.Atlas_Default) {\n}\n": 1, + "create page M.P (title: 'P', layout: Atlas_Core.Atlas_Default) {\n};\n": 0, + } { + if got := terminatorProblems(script); len(got) != want { + t.Errorf("%q: %d problems %v, want %d", script, len(got), got, want) + } + } +} + +// A published OData service ends at its property list, its authentication +// clause or its entity block, whichever comes last; each must carry the `;`. +func TestDescribePublishedODataService_Terminated(t *testing.T) { + for name, svc := range map[string]*model.PublishedODataService{ + "properties only": {Name: "S", Path: "odata/s/v1", Version: "1.0.0", ODataVersion: "OData4"}, + "authentication": {Name: "S", Path: "odata/s/v1", ODataVersion: "OData4", AuthenticationTypes: []string{"Basic"}}, + "block": {Name: "S", Path: "odata/s/v1", ODataVersion: "OData4", AuthenticationTypes: []string{"Basic"}, + Microflows: []*model.PublishedMicroflow{{Microflow: "Shop.Act", ExposedName: "Act"}}}, + "grant": {Name: "S", Path: "odata/s/v1", ODataVersion: "OData4", AllowedModuleRoles: []string{"Shop.User"}}, + } { + t.Run(name, func(t *testing.T) { + var out bytes.Buffer + assertNoError(t, outputPublishedODataServiceMDL(&ExecContext{Output: &out}, svc, "Shop", "")) + reparse(t, out.String()) + assertTerminated(t, out.String()) + }) + } +} + +func TestDescribeExternalEntity_Terminated(t *testing.T) { + attr := &domainmodel.Attribute{Name: "Code", Type: &domainmodel.StringAttributeType{Length: 20}} + for name, attrs := range map[string][]*domainmodel.Attribute{ + "no attributes": nil, + "attributes": {attr}, + } { + t.Run(name, func(t *testing.T) { + e := &domainmodel.Entity{Name: "Order", Source: "Rest$ODataRemoteEntitySource", RemoteServiceName: "Shop.Remote", RemoteEntitySet: "Orders", Attributes: attrs} + var out bytes.Buffer + assertNoError(t, outputExternalEntityMDL(&ExecContext{Output: &out}, e, "Shop")) + reparse(t, out.String()) + assertTerminated(t, out.String()) + }) + } +} diff --git a/mdl/executor/documentation_carry.go b/mdl/executor/documentation_carry.go index 0c154d5eff..879f5b85b8 100644 --- a/mdl/executor/documentation_carry.go +++ b/mdl/executor/documentation_carry.go @@ -2,6 +2,8 @@ package executor +import "strings" + // carriedDocumentation returns the documentation a rewrite should store. // // A statement that carried a `/** … */` comment — even an empty one — states its @@ -18,9 +20,30 @@ package executor // only in the domain-model grammar — there is no ALTER for a microflow, a queue // or a workflow — so an explicitly empty comment is the only clearing spelling // most doctypes have. +// +// A stated comment that is exactly the stored text's doc-comment normal form +// (see docCommentNormalForm) says the same thing in the only spelling MDL has, +// so the stored bytes are kept: overwriting them rewrote the document when +// describe output was run unchanged (#731). func carriedDocumentation(set bool, stated, stored string) string { if set { + if stated != stored && stated == docCommentNormalForm(stored) { + return stored + } return stated } return stored } + +// docCommentNormalForm is what a `/** … */` comment can carry of text: the +// visitor keeps only its trimmed, non-empty lines (extractDocComment), so blank +// lines, trailing whitespace and CRLF line ends have no spelling. +func docCommentNormalForm(s string) string { + var kept []string + for line := range strings.SplitSeq(strings.ReplaceAll(s, "\r\n", "\n"), "\n") { + if t := strings.TrimSpace(line); t != "" { + kept = append(kept, t) + } + } + return strings.Join(kept, "\n") +} diff --git a/mdl/executor/documentation_carry_test.go b/mdl/executor/documentation_carry_test.go new file mode 100644 index 0000000000..d0ba62b2aa --- /dev/null +++ b/mdl/executor/documentation_carry_test.go @@ -0,0 +1,37 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import "testing" + +// A doc comment cannot spell a blank line, trailing whitespace or a CRLF: the +// visitor keeps only the trimmed, non-empty lines. So describe output of +// Studio Pro prose ("line\r\n\r\nline \r\n") re-executed as `create or modify` +// states the same text in that normal form, and overwriting the stored value +// with it rewrote the document with nothing changed (#731: 7 JavaScript actions +// in PedApp). When the stated text is exactly the stored text's normal form, the +// stored bytes are kept; any real edit still wins. +func TestCarriedDocumentation_KeepsStoredWhenOnlyTheSpellingDiffers(t *testing.T) { + stored := "What does this JavaScript action do?\r\n\r\nWhen you upload a screenshot manually. \r\n\r\nReturn Type:\r\nWill return base 64 image string\r\n" + cases := []struct { + name string + set bool + stated string + want string + }{ + {"no comment keeps stored", false, "", stored}, + {"normal form keeps stored bytes", true, + "What does this JavaScript action do?\nWhen you upload a screenshot manually.\nReturn Type:\nWill return base 64 image string", + stored}, + // Controls: a real change is written, and an empty comment still clears. + {"edited text wins", true, "What does this JavaScript action do?\nSomething else.", "What does this JavaScript action do?\nSomething else."}, + {"empty comment clears", true, "", ""}, + } + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + if got := carriedDocumentation(c.set, c.stated, stored); got != c.want { + t.Errorf("carriedDocumentation = %q, want %q", got, c.want) + } + }) + } +} diff --git a/mdl/executor/issue707_describe_roundtrip_test.go b/mdl/executor/issue707_describe_roundtrip_test.go index 595ec0f6cb..ac74cae7d8 100644 --- a/mdl/executor/issue707_describe_roundtrip_test.go +++ b/mdl/executor/issue707_describe_roundtrip_test.go @@ -35,6 +35,7 @@ func reparse(t *testing.T, out string) *ast.Program { if len(errs) > 0 { t.Fatalf("describe output does not re-parse: %v\n--- output ---\n%s", errs, out) } + assertTerminated(t, out) // #744 return prog } diff --git a/mdl/executor/register_stubs.go b/mdl/executor/register_stubs.go index 130cefb935..aac52f5ba7 100644 --- a/mdl/executor/register_stubs.go +++ b/mdl/executor/register_stubs.go @@ -519,6 +519,9 @@ func registerLintHandlers(r *Registry) { } func registerAlterPageHandlers(r *Registry) { + r.Register(&ast.AlterFlowStmt{}, func(ctx *ExecContext, stmt ast.Statement) error { + return execAlterFlow(ctx, stmt.(*ast.AlterFlowStmt)) + }) r.Register(&ast.AlterPageStmt{}, func(ctx *ExecContext, stmt ast.Statement) error { return execAlterPage(ctx, stmt.(*ast.AlterPageStmt)) }) diff --git a/mdl/executor/registry_test.go b/mdl/executor/registry_test.go index 9340826a36..a59e8e7870 100644 --- a/mdl/executor/registry_test.go +++ b/mdl/executor/registry_test.go @@ -177,6 +177,7 @@ func allKnownStatements() []ast.Statement { &ast.AlterODataClientStmt{}, &ast.AlterODataServiceStmt{}, &ast.AlterPageStmt{}, + &ast.AlterFlowStmt{}, &ast.AlterPagesLayoutStmt{}, &ast.AlterPagesStylingStmt{}, &ast.AlterProjectSecurityStmt{}, diff --git a/mdl/executor/roundtrip_microflow_test.go b/mdl/executor/roundtrip_microflow_test.go index 1fced12f63..c7e4fbecb4 100644 --- a/mdl/executor/roundtrip_microflow_test.go +++ b/mdl/executor/roundtrip_microflow_test.go @@ -369,7 +369,9 @@ begin end;` assertMicroflowContains(t, env, mfName, createMDL, - []string{"retrieve", "RoundtripTest.MfTestItem", "limit 1", "return"}, + // No header, so `limit 1` is the object range, which describe + // prints as `first` (ako/mxcli#734). + []string{"retrieve", "RoundtripTest.MfTestItem", "first", "return"}, nil, ) } diff --git a/mdl/executor/stmt_summary.go b/mdl/executor/stmt_summary.go index 362f916039..bcb135d796 100644 --- a/mdl/executor/stmt_summary.go +++ b/mdl/executor/stmt_summary.go @@ -198,6 +198,9 @@ func stmtSummary(stmt ast.Statement) string { case *ast.AlterStylingStmt: return fmt.Sprintf("alter styling on %s %s widget %s", s.ContainerType, s.ContainerName, s.WidgetName) + case *ast.AlterFlowStmt: + return fmt.Sprintf("alter %s %s", s.Kind(), s.Name) + // ALTER PAGE / ALTER SNIPPET case *ast.AlterPageStmt: ct := s.ContainerType diff --git a/mdl/executor/strict_parsing_test.go b/mdl/executor/strict_parsing_test.go new file mode 100644 index 0000000000..02de497512 --- /dev/null +++ b/mdl/executor/strict_parsing_test.go @@ -0,0 +1,122 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "io" + "reflect" + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/linter" + "github.com/mendixlabs/mxcli/mdl/visitor" +) + +// The R11 strictness changes (#732), end to end through check and exec. Under +// mdl 0 each script keeps running, and `check` names the construct with its +// registered code as a warning; under mdl 1 the same script does not get past +// the parser, so exec never starts. +var strictParsingCases = []struct { + name, src, code string + refusal string // what the mdl 1 error says +}{ + {"missing semicolon", "show modules\nshow entities;", "MDL-V1-SEMI", "no terminating `;`"}, + {"slash terminator", "show modules;\n/\nshow entities;", "MDL-V1-SLASH", "`/` is not a statement terminator"}, + {"unknown property key", "show modules;\ncreate published rest service M.S (Path: 'rest/s', Verison: '1.0') { };", "MDL-V1-PROP", "did you mean 'Version'"}, + {"mis-shaped property value", "show modules;\ncreate rest client M.Api (BaseUrl: 'https://x', Authentication: none) { operation Post { Method: post, Path: '/u', Response: json from $X } };", "MDL-V1-PROPVALUE", "takes none or json as $var"}, + // A change of meaning, not a rejection: see TestStringEscapeUnderEachVersion. + {"backslash escape", "show modules;\ncreate persistent entity M.N (T: String(20) default 'C:\\temp');", "MDL-V1-ESCAPE", ""}, +} + +// countStatements registers handlers that count what exec runs. +func countStatements(e *Executor, ran *int, defaults *[]any) { + e.registry.handlers[reflect.TypeOf(&ast.ShowStmt{})] = func(*ExecContext, ast.Statement) error { + *ran++ + return nil + } + for _, s := range []ast.Statement{&ast.CreatePublishedRestServiceStmt{}, &ast.CreateRestClientStmt{}} { + e.registry.handlers[reflect.TypeOf(s)] = func(*ExecContext, ast.Statement) error { + *ran++ + return nil + } + } + e.registry.handlers[reflect.TypeOf(&ast.CreateEntityStmt{})] = func(_ *ExecContext, s ast.Statement) error { + *ran++ + *defaults = append(*defaults, s.(*ast.CreateEntityStmt).Attributes[0].DefaultValue) + return nil + } +} + +func TestStrictParsingWarnsUnderMdl0AndRunsTheScript(t *testing.T) { + for _, tc := range strictParsingCases { + t.Run(tc.name, func(t *testing.T) { + prog, errs := visitor.Build(tc.src) + if len(errs) > 0 { + t.Fatalf("mdl 0 must keep parsing it: %v", errs) + } + var warned bool + for _, v := range ValidateProgram(prog, "") { + if v.RuleID == tc.code { + warned = true + if v.Severity != linter.SeverityWarning { + t.Errorf("%s must be a warning under mdl 0, got %v", tc.code, v.Severity) + } + } + } + if !warned { + t.Errorf("check did not report %s", tc.code) + } + + e := New(io.Discard) + var ran int + var defaults []any + countStatements(e, &ran, &defaults) + if err := e.ExecuteProgram(prog); err != nil { + t.Fatal(err) + } + if ran != 2 { + t.Errorf("mdl 0: ran %d statements, want 2", ran) + } + }) + } +} + +func TestStrictParsingRefusedUnderMdl1(t *testing.T) { + for _, tc := range strictParsingCases { + if tc.refusal == "" { + continue + } + t.Run(tc.name, func(t *testing.T) { + _, errs := visitor.Build("mdl 1;\n" + tc.src) + var refused bool + for _, err := range errs { + refused = refused || strings.Contains(err.Error(), tc.refusal) + } + if !refused { + t.Fatalf("mdl 1 must refuse it with %q, got %v\n%s", tc.refusal, errs, tc.src) + } + }) + } +} + +// Under mdl 0 exec stores the backslash escape's old value (a tab); under +// mdl 1 the backslash is the character written. +func TestStringEscapeUnderEachVersion(t *testing.T) { + for header, want := range map[string]string{"": "C:\temp", "mdl 1;\n": `C:\temp`} { + prog, errs := visitor.Build(header + "create persistent entity M.N (T: String(20) default 'C:\\temp');") + if len(errs) > 0 { + t.Fatal(errs) + } + e := New(io.Discard) + var ran int + var defaults []any + countStatements(e, &ran, &defaults) + if err := e.ExecuteProgram(prog); err != nil { + t.Fatal(err) + } + if len(defaults) != 1 || defaults[0] != want { + t.Errorf("%q: exec saw default %q, want %q", header, defaults, want) + } + } +} diff --git a/mdl/executor/validate_deprecations.go b/mdl/executor/validate_deprecations.go index cd27238466..5464a3445d 100644 --- a/mdl/executor/validate_deprecations.go +++ b/mdl/executor/validate_deprecations.go @@ -33,7 +33,7 @@ func ValidateDeprecations(prog *ast.Program) []linter.Violation { Severity: linter.SeverityWarning, Message: fmt.Sprintf("line %d: `%s`%s is deprecated; write `%s` — same meaning. "+ "Refused from `mdl %d`.", d.Line, e.Old, on, e.Canonical, e.RemovedIn), - Suggestion: fmt.Sprintf("Replace `%s` with `%s`. %s", e.Rewrite.Token, e.Rewrite.Replacement, e.Note), + Suggestion: deprecationSuggestion(e), }) } return out @@ -54,3 +54,12 @@ func ApplyDeprecationPolicy(violations []linter.Violation, policy deprecation.Po } return out } + +// deprecationSuggestion says how to rewrite a deprecated spelling: the keyword +// to swap, or for a structural rewrite, the rewrite itself. +func deprecationSuggestion(e deprecation.Entry) string { + if e.Rewrite.Structural != "" { + return fmt.Sprintf("Rewrite the %s. %s", e.Rewrite.Structural, e.Note) + } + return fmt.Sprintf("Replace `%s` with `%s`. %s", e.Rewrite.Token, e.Rewrite.Replacement, e.Note) +} diff --git a/mdl/executor/validate_microflow_listop_source.go b/mdl/executor/validate_microflow_listop_source.go index 313a68d335..f984b9ea82 100644 --- a/mdl/executor/validate_microflow_listop_source.go +++ b/mdl/executor/validate_microflow_listop_source.go @@ -39,6 +39,11 @@ import ( // The control for all of them is the reporter's own workaround, which builds // the same two activities explicitly and passes at 0 errors. // +// Retired for `mdl 1;` scripts (#733): there a list operation is one statement +// per activity whose operand is a VARIABLE, so a nested call does not parse and +// the visitor refuses the call forms that could nest (MDL-V1-LIST). This rule is +// reachable only from mdl 0 scripts, and goes with them. +// // Why refuse rather than materialise an implicit variable: the refusal covers // every spelling from one rule, including the literal operand, which no amount // of materialising would fix. And it needs no project — the answer is in the diff --git a/mdl/executor/validate_microflow_retrieve_single.go b/mdl/executor/validate_microflow_retrieve_single.go index d9edf2a2c4..24d397bbc0 100644 --- a/mdl/executor/validate_microflow_retrieve_single.go +++ b/mdl/executor/validate_microflow_retrieve_single.go @@ -9,45 +9,47 @@ import ( "github.com/mendixlabs/mxcli/mdl/linter" ) -// retrieveSingleRule is the rule ID for "a LIMIT 1 retrieve used as a list". +// retrieveSingleRule is the rule ID for "an object-range retrieve used as a list". const retrieveSingleRule = "MDL-RETRIEVE01" -// checkRetrieveLimitOneAsList flags a variable that `RETRIEVE … LIMIT 1` bound to -// a single OBJECT and a later statement uses as a LIST. +// checkRetrieveLimitOneAsList flags a variable that a retrieve bound to a single +// OBJECT and a later statement uses as a LIST. // -// `limit 1` is not a one-element list here. The executor maps it to Mendix's -// "First object" range (RangeTypeFirst, cmd_microflows_builder_actions.go), which -// makes the output variable an object — and that is deliberate and documented -// (MDL_QUICK_REFERENCE.md), not something to change under anyone's feet. +// The object range is Mendix's "First object" range, written `retrieve … first` +// in every language version, and `retrieve … limit 1` in a script without the +// `mdl 1;` header, where `limit 1` keeps its alpha meaning (ako/mxcli#734). The +// visitor resolves both to RetrieveStmt.First, which is what the writer stores +// too, so the rule keys on it and never on limit text: a check that disagrees +// with the writer it describes is worse than no check. // -// What was missing is any sign of it before the build. Nothing in the MDL says -// the variable changed shape: `mxcli check --references` passed, and DESCRIBE -// re-emits `limit 1`, so the source of an object retrieve and a list retrieve are -// identical text. The first thing the author saw was CE0097 "The selected 'x' -// variable must be of type List" from mxbuild — and inside a .test.mdl file, not -// even that: the injected test simply failed to build (mendixlabs/mxcli#1103). -// -// The clause is also spelled the other way round elsewhere in the same language — -// `import from mapping … first` binds an object and `… limit 1` a one-element -// list — so reading it as a list is a reasonable mistake rather than a careless -// one. The message therefore names the working spelling instead of only refusing. -// -// Keyed on exactly the condition the writer uses (limit "1", no offset), because -// a check that disagrees with the writer it describes is worse than no check. +// Before #734 this rule was the only sign of the object: describe re-emitted +// `limit 1` for it, so an object retrieve and a list retrieve were identical text, +// and the first thing the author saw was CE0097 from mxbuild — inside a +// .test.mdl, not even that (mendixlabs/mxcli#1103). describe now prints `first` +// and an mdl 0 `limit 1` warns MDL-V1-LIMIT1, so the spelling says it; the rule +// stays for the mistake that is left, treating an object as a list, which +// check --references does not otherwise catch before the build. func (v *microflowValidator) checkRetrieveLimitOneAsList(body []ast.MicroflowStatement) { - // single holds the variables currently bound to one object by a LIMIT 1 - // retrieve. Maintained in statement order so a rebinding clears it: a name - // reused for a real list further down is not this rule's business. + // single holds the variables currently bound to one object by an + // object-range retrieve. Maintained in statement order so a rebinding + // clears it: a name reused for a real list further down is not this rule's + // business. single := map[string]bool{} forEachMicroflowStatement(body, func(s ast.MicroflowStatement) { if name, op := listUseOf(s); name != "" && single[name] { + // Measured on 11.13: a loop reports CE0100, a list activity CE0097. + rejection := fmt.Sprintf("CE0097 \"The selected '%s' variable must be of type List\"", name) + if op == "a loop" { + rejection = fmt.Sprintf("CE0100 \"'%s' is of type …, but should be of type List\"", name) + } v.addViolation(retrieveSingleRule, linter.SeverityError, - fmt.Sprintf("$%s was retrieved with LIMIT 1, which binds a single object rather than a "+ - "one-element list, so %s cannot take it — mxbuild rejects this with CE0097 "+ - "\"The selected '%s' variable must be of type List\".", name, op, name), - fmt.Sprintf("Drop the LIMIT to retrieve a list and keep %s, or keep LIMIT 1 and use "+ - "$%s as the object it already is.", op, name)) + fmt.Sprintf("$%s was retrieved as a single object (`first`, which is also what `limit 1` "+ + "means in a script without the `mdl 1;` header), not a list, so %s cannot take it — "+ + "mxbuild rejects this with %s.", name, op, rejection), + fmt.Sprintf("Retrieve a list instead (no range, or under `mdl 1;` `limit 1` for a list of "+ + "one) and keep %s, or keep the object range and use $%s as the object it already is.", + op, name)) } // Rebinding first, so a statement that both consumes and produces the @@ -55,7 +57,7 @@ func (v *microflowValidator) checkRetrieveLimitOneAsList(body []ast.MicroflowSta for _, p := range statementProducedVars(s) { delete(single, p.name) } - if r, ok := s.(*ast.RetrieveStmt); ok && r.Limit == "1" && r.Offset == "" && r.Variable != "" { + if r, ok := s.(*ast.RetrieveStmt); ok && r.First && r.StartVariable == "" && r.Variable != "" { single[r.Variable] = true } }) diff --git a/mdl/executor/validate_microflow_retrieve_single_test.go b/mdl/executor/validate_microflow_retrieve_single_test.go index e034d0165f..6a24002d76 100644 --- a/mdl/executor/validate_microflow_retrieve_single_test.go +++ b/mdl/executor/validate_microflow_retrieve_single_test.go @@ -27,6 +27,16 @@ func mfWith(body ...ast.MicroflowStatement) *ast.CreateMicroflowStmt { } } +// retrieveFirst is the object range: `first`, or `limit 1` in an mdl 0 script +// (the visitor resolves both to First, ako/mxcli#734). +func retrieveFirst(variable string) *ast.RetrieveStmt { + return &ast.RetrieveStmt{ + Variable: variable, + Source: ast.QualifiedName{Module: "Probe", Name: "Request"}, + First: true, + } +} + func retrieveLimit(variable, limit, offset string) *ast.RetrieveStmt { return &ast.RetrieveStmt{ Variable: variable, @@ -47,13 +57,13 @@ func retrieveLimit(variable, limit, offset string) *ast.RetrieveStmt { func TestRetrieveLimitOneUsedAsList(t *testing.T) { t.Run("HEAD of a LIMIT 1 retrieve is flagged", func(t *testing.T) { got := retrieveSingleViolations(t, mfWith( - retrieveLimit("reqs", "1", ""), + retrieveFirst("reqs"), &ast.ListOperationStmt{Operation: ast.ListOpHead, InputVariable: "reqs", OutputVariable: "req"}, )) if len(got) != 1 { t.Fatalf("got %d violations %q, want 1", len(got), got) } - for _, want := range []string{"$reqs", "LIMIT 1", "CE0097"} { + for _, want := range []string{"$reqs", "`first`", "CE0097"} { if !strings.Contains(got[0], want) { t.Errorf("message %q does not mention %q", got[0], want) } @@ -62,17 +72,21 @@ func TestRetrieveLimitOneUsedAsList(t *testing.T) { t.Run("looping over a LIMIT 1 retrieve is flagged", func(t *testing.T) { got := retrieveSingleViolations(t, mfWith( - retrieveLimit("reqs", "1", ""), + retrieveFirst("reqs"), &ast.LoopStmt{LoopVariable: "r", ListVariable: "reqs"}, )) if len(got) != 1 { t.Fatalf("got %d violations %q, want 1", len(got), got) } + // mxbuild 11.13 reports a loop over an object as CE0100, not CE0097. + if !strings.Contains(got[0], "CE0100") { + t.Errorf("message %q should name CE0100, what mxbuild reports for a loop", got[0]) + } }) t.Run("aggregating a LIMIT 1 retrieve is flagged", func(t *testing.T) { got := retrieveSingleViolations(t, mfWith( - retrieveLimit("reqs", "1", ""), + retrieveFirst("reqs"), &ast.AggregateListStmt{Operation: ast.AggregateCount, InputVariable: "reqs", OutputVariable: "n"}, )) if len(got) != 1 { @@ -104,13 +118,19 @@ func TestRetrieveLimitOneControls(t *testing.T) { &ast.ListOperationStmt{Operation: ast.ListOpHead, InputVariable: "reqs", OutputVariable: "req"}, ), // Using it as an object is exactly right and must stay silent. + // `limit 1` under mdl 1: the visitor leaves it a Custom range, a list + // of one, so every list use is right. + "limit 1 under mdl 1": mfWith( + retrieveLimit("reqs", "1", ""), + &ast.ListOperationStmt{Operation: ast.ListOpHead, InputVariable: "reqs", OutputVariable: "req"}, + ), "used as an object": mfWith( - retrieveLimit("req", "1", ""), + retrieveFirst("req"), &ast.MfCommitStmt{Variable: "req"}, ), // Rebound to a real list before the list use. "rebound to a list": mfWith( - retrieveLimit("reqs", "1", ""), + retrieveFirst("reqs"), &ast.CreateListStmt{Variable: "reqs", EntityType: ast.QualifiedName{Module: "Probe", Name: "Request"}}, &ast.ListOperationStmt{Operation: ast.ListOpHead, InputVariable: "reqs", OutputVariable: "req"}, ), diff --git a/mdl/executor/validate_odata_properties.go b/mdl/executor/validate_odata_properties.go index 069f86aa82..052fa0dc60 100644 --- a/mdl/executor/validate_odata_properties.go +++ b/mdl/executor/validate_odata_properties.go @@ -18,6 +18,7 @@ import ( "github.com/mendixlabs/mxcli/mdl/ast" "github.com/mendixlabs/mxcli/mdl/linter" + "github.com/mendixlabs/mxcli/mdl/suggest" ) // Known property names, in the spelling the syntax help uses. These are for the @@ -119,54 +120,8 @@ func unknownODataProps(location string, unknown, known []string) []linter.Violat } // closestProperty returns the known property a misspelling most likely meant, -// or "" when nothing is close enough to be worth guessing. Case-insensitive -// prefix/substring first, then a single edit. -func closestProperty(name string, known []string) string { - lower := strings.ToLower(name) - for _, k := range known { - lk := strings.ToLower(k) - if strings.HasPrefix(lk, lower) || strings.HasPrefix(lower, lk) || strings.Contains(lk, lower) { - return k - } - } - for _, k := range known { - if withinOneEdit(lower, strings.ToLower(k)) { - return k - } - } - return "" -} - -// withinOneEdit reports whether a and b differ by at most one insertion, -// deletion or substitution. -func withinOneEdit(a, b string) bool { - if a == b { - return true - } - if len(a) > len(b) { - a, b = b, a - } - if len(b)-len(a) > 1 { - return false - } - i, j, edits := 0, 0, 0 - for i < len(a) && j < len(b) { - if a[i] == b[j] { - i++ - j++ - continue - } - edits++ - if edits > 1 { - return false - } - if len(a) == len(b) { - i++ - } - j++ - } - return true -} +// or "" when nothing is close enough to be worth guessing. +func closestProperty(name string, known []string) string { return suggest.Closest(name, known) } // quotedConstantRef matches the text of a string literal that is really a // constant reference: `@Module.Name`. diff --git a/mdl/grammar/MDLLexer.g4 b/mdl/grammar/MDLLexer.g4 index cf037486c1..e28cccad7e 100644 --- a/mdl/grammar/MDLLexer.g4 +++ b/mdl/grammar/MDLLexer.g4 @@ -8,6 +8,70 @@ */ lexer grammar MDLLexer; +// Hand-written lexer helpers. In the Go target `@members` is package-level +// code, so these are methods on the generated lexer; the predicates below call +// them through the receiver `p` the generator names. +@lexer::members { +// StrictEscapeStream is the character stream of a script written in mdl 1 or +// later, where `''` is the only escape in a string literal and a backslash is +// an ordinary character (ADR-0010 R11). The escape rule changes where a +// literal ENDS — `'C:\'` is complete under mdl 1 and unterminated under mdl +// 0 — so it has to be decided before lexing, from the `mdl <n>;` header, and +// travel with the stream: every token keeps its stream, which is how the +// visitor reads a literal's value under the same rule it was lexed with. +type StrictEscapeStream struct{ antlr.CharStream } + +// HasStrictEscapes reports whether in is lexed under mdl 1's string rules. +func HasStrictEscapes(in antlr.CharStream) bool { + _, ok := in.(*StrictEscapeStream) + return ok +} + +// isTrailingComma reports whether the ',' just matched ends a bracketed list: +// the next significant character closes a (), {} or [] (whitespace and +// comments skipped), and the one before it neither opens a list nor is another +// comma, so `()` stays the only spelling of an empty list and `(a,,)` stays an +// error. +func (l *MDLLexer) isTrailingComma() bool { + in := l.GetInputStream() + // The predicate runs with the comma consumed: LA(1) is the character + // after it and LA(-1) the comma itself. + for i := 1; ; i++ { + switch c := in.LA(i); c { + case ' ', '\t', '\r', '\n', '\v', '\f': + case '-': + if in.LA(i+1) != '-' { + return false + } + for i++; in.LA(i+1) != '\n' && in.LA(i+1) != antlr.TokenEOF; i++ { + } + case '/': + if in.LA(i+1) != '*' { + return false + } + for i += 2; !(in.LA(i) == '*' && in.LA(i+1) == '/'); i++ { + if in.LA(i) == antlr.TokenEOF { + return false + } + } + i++ + case ')', '}', ']': + for j := -2; ; j-- { + switch in.LA(j) { + case ' ', '\t', '\r', '\n', '\v', '\f': + continue + case '(', '{', '[', ',', antlr.TokenEOF, 0: + return false + } + return true + } + default: + return false + } + } +} +} + // ============================================================================= // WHITESPACE AND COMMENTS // ============================================================================= @@ -257,6 +321,8 @@ MAXIMUM: M A X I M U M; REDUCE: R E D U C E; ANY: A N Y; INITIAL: I N I T I A L; +// reduce $L from <initial> as <type> using <expression> (#733). +USING: U S I N G; LIST: L I S T; REMOVE: R E M O V E; EQUALS_OP: E Q U A L S; @@ -872,6 +938,13 @@ DIV: D I V; // ============================================================================= SEMICOLON: ';'; +// A trailing comma is allowed in every bracketed list, under every language +// version (ADR-0010 R11): `(A: String(200),)`, `{ Method: get, }`. It is +// dropped here rather than written into each list rule, so it is one rule for +// every list, and so a list's error messages stay LL(1): with `(COMMA x)* +// COMMA?` in the parser, an unknown item after a comma is reported as "no +// viable alternative" instead of naming what the list expects. +TRAILING_COMMA: ',' {p.isTrailingComma()}? -> skip; COMMA: ','; DOT: '.'; LPAREN: '('; @@ -895,9 +968,14 @@ HASH: '#'; // Mendix token: [%TokenName%] or [%'literal'%] MENDIX_TOKEN: '[%' .*? '%]'; -// String literals (single-quoted, with escape support) +// String literals, single-quoted. `''` is an apostrophe under every language +// version. Under mdl 0 a backslash also escapes the next character (`\'`, +// `\n`, `\\`, …); from mdl 1 it is an ordinary character, as in a Mendix +// expression, so `'C:\temp'` is the path it looks like (ADR-0010 R11). Which +// rule applies is fixed by the stream the script is lexed from: see +// StrictEscapeStream. STRING_LITERAL - : '\'' ( ~['\\] | '\\' . | '\'\'' )* '\'' + : '\'' ( ~['\\] | '\'\'' | '\\' {!HasStrictEscapes(p.GetInputStream())}? . | '\\' {HasStrictEscapes(p.GetInputStream())}? )* '\'' ; // Dollar-quoted string literal (PostgreSQL style) for embedding code blocks diff --git a/mdl/grammar/MDLParser.g4 b/mdl/grammar/MDLParser.g4 index 472d389618..cb8699bdec 100644 --- a/mdl/grammar/MDLParser.g4 +++ b/mdl/grammar/MDLParser.g4 @@ -110,6 +110,9 @@ createStatement CREATE (OR (MODIFY | REPLACE /* @alias MDL-DEPR001 */))? ( createEntityStatement | createAssociationStatement + // Before createModuleStatement: `create module role M.R` is a role, and + // `create module Role;` (no qualified name follows) is still a module. + | createModuleRoleStatement | createModuleStatement | createMicroflowStatement | createJavaActionStatement @@ -173,6 +176,11 @@ alterStatement // there; a scroll-container region is addressed as `layoutContainer.top`, // because a region has no Name of its own. | ALTER alterDocumentType qualifiedName LBRACE alterOperation+ RBRACE + // The generic ALTER on a microflow or nanoflow (ADR-0012 decision 3): a + // graph splice into the stored flow. Its targets are content addresses + // (`$Var`, `'Caption'`, a statement pattern) and its fragments are + // microflow statements, so it has its own operation rule. + | ALTER (MICROFLOW | NANOFLOW) qualifiedName LBRACE alterFlowOperation* RBRACE | alterPagesLayoutStatement | alterPagesStylingStatement | ALTER WORKFLOW qualifiedName alterWorkflowAction+ SEMICOLON? @@ -312,6 +320,36 @@ alterTarget | STRING_LITERAL (AT NUMBER_LITERAL)? ; +/** + * `alter microflow` / `alter nanoflow` operations (ADR-0012 decision 3, + * ako/mxcli#736): + * + * ```mdl + * alter microflow FeedbackModule.VAL_Feedback { + * insert after $IsValidEmail { log info node 'Feedback' 'Email checked'; } + * insert before 'Email is Valid?' { … } + * replace commit $Order with { commit $Order with events; } + * drop log * node 'Debug' *; + * } + * ``` + * + * A fragment is written exactly as the same statements are in `create + * microflow`. A target is a content address, resolved by mfmutator: `$Var` + * (the activity that outputs it), `'Caption'`, or a statement pattern with `*` + * wildcards, each optionally followed by `@n`. A pattern is any run of tokens, + * so the target is taken as raw text up to the `{`, `with` or `;` that ends it; + * that is why `drop` needs its semicolon. + */ +alterFlowOperation + : INSERT (AFTER | BEFORE) alterFlowTarget LBRACE microflowBody RBRACE SEMICOLON? + | REPLACE alterFlowTarget WITH LBRACE microflowBody RBRACE SEMICOLON? + | DROP alterFlowTarget SEMICOLON + ; + +alterFlowTarget + : ~(LBRACE | RBRACE | SEMICOLON | WITH)+ + ; + // ALTER PAGES [IN <module>] SET LAYOUT = Module.Layout [MAP (...)] [WHERE LAYOUT = Module.Old] // // The bulk form is the real one: an app has one layout and many pages, so @@ -651,8 +689,7 @@ moveDocumentType // ============================================================================= securityStatement - : createModuleRoleStatement - | dropModuleRoleStatement + : dropModuleRoleStatement | alterUserRoleStatement | dropUserRoleStatement | grantEntityAccessStatement diff --git a/mdl/grammar/domains/MDLMicroflow.g4 b/mdl/grammar/domains/MDLMicroflow.g4 index f96b8fd3d2..190192da67 100644 --- a/mdl/grammar/domains/MDLMicroflow.g4 +++ b/mdl/grammar/domains/MDLMicroflow.g4 @@ -419,13 +419,21 @@ rollbackStatement : ROLLBACK VARIABLE REFRESH? ; -// RETRIEVE $ProductList FROM MfTest.Product WHERE Code = $SearchCode SORT BY Name ASC LIMIT 1; +// RETRIEVE $Product FROM MfTest.Product WHERE Code = $SearchCode SORT BY Name ASC FIRST; +// RETRIEVE $Top FROM MfTest.Product SORT BY Price DESC LIMIT 10 OFFSET 20; +// +// FIRST is Mendix's "First object" range: it binds ONE object, and Mendix gives +// it no offset. LIMIT/OFFSET is the Custom range and always binds a list. +// `LIMIT 1` without OFFSET is version-gated (ako/mxcli#734): under `mdl 1;` it is +// a list of one, without the header it keeps its alpha meaning, the object, and +// warns MDL-V1-LIMIT1. The same split as `import from mapping … first | limit n`. retrieveStatement : RETRIEVE VARIABLE FROM retrieveSource (WHERE (xpathConstraint (andOrXpath? xpathConstraint)* | expression))? (SORT_BY sortColumn (COMMA sortColumn)*)? - (LIMIT limitExpr=expression)? - (OFFSET offsetExpr=expression)? + ( FIRST + | (LIMIT limitExpr=expression)? (OFFSET offsetExpr=expression)? + ) onErrorClause? ; @@ -878,10 +886,46 @@ transformJsonStatement // ============================================================================= /** - * List operations that return a single item or a modified list. + * List operations that return a single item or a modified list: one statement + * per Studio Pro "List operation" activity (PROPOSAL_mdl_beta_syntax_freeze.md + * §4, #733). The operand is always a variable, as it is in the activity's + * dialog, so one activity cannot be nested inside another. */ listOperationStatement - : VARIABLE EQUALS listOperation + : VARIABLE EQUALS listOperationActivity + // The call form. A respelling for every operation but find and contains, + // whose call form is also the string function: the visitor version-gates + // those instead (MDL-V1-LIST), since no rewrite can know which was meant. + | VARIABLE EQUALS listOperation /* @alias MDL-DEPR003 */ + ; + +listOperationActivity + : HEAD VARIABLE // $x = head $L + | TAIL VARIABLE // $x = tail $L + | FIND VARIABLE listOperationCondition // $x = find $L by Number = 3 + | FILTER VARIABLE listOperationCondition // $x = filter $L where $currentObject/Paid + | SORT VARIABLE BY listSortItem (COMMA listSortItem)* // $x = sort $L by Date desc, Number + | UNION VARIABLE WITH VARIABLE // $x = union $A with $B + | INTERSECT VARIABLE WITH VARIABLE // $x = intersect $A with $B + | SUBTRACT VARIABLE FROM VARIABLE // $x = subtract $B from $A (A minus B) + | CONTAINS VARIABLE IN VARIABLE // $b = contains $Object in $L + | EQUALS_OP VARIABLE AND VARIABLE // $b = equals $A and $B + | RANGE VARIABLE (OFFSET expression)? (LIMIT expression)? // $x = range $L offset 20 limit 10 + ; + +// `by` picks a member (Studio Pro's Find / Filter: an attribute or association +// and the value it must have), written `Member = value`; the visitor refuses +// any other shape. `where` takes an expression over $currentObject (Find by +// expression / Filter by expression). +listOperationCondition + : BY expression + | WHERE expression + ; + +// A sort attribute may be any word, so an attribute called Count or Date needs +// no quotes here. +listSortItem + : identifierOrKeyword (ASC | DESC)? ; listOperation @@ -910,7 +954,19 @@ sortSpec * Aggregate operations on lists. */ aggregateListStatement - : VARIABLE EQUALS listAggregateOperation + : VARIABLE EQUALS aggregateListActivity + | VARIABLE EQUALS listAggregateOperation /* @alias MDL-DEPR004 */ + ; + +/** + * One Studio Pro "Aggregate list" activity. `by` aggregates an attribute and + * `of` an expression (the dialog's "Aggregate with: Attribute / Expression"). + */ +aggregateListActivity + : COUNT VARIABLE // $n = count $L + | (SUM | AVERAGE | MINIMUM | MAXIMUM) VARIABLE (BY identifierOrKeyword | OF expression) // $t = sum $L by Amount + | (ALL | ANY) VARIABLE WHERE expression // $b = all $L where $currentObject/Paid + | REDUCE VARIABLE FROM expression AS dataType USING expression // $s = reduce $L from '' as String using … ; listAggregateOperation diff --git a/mdl/grammar/domains/MDLSecurity.g4 b/mdl/grammar/domains/MDLSecurity.g4 index 840068fa26..f5a06f137d 100644 --- a/mdl/grammar/domains/MDLSecurity.g4 +++ b/mdl/grammar/domains/MDLSecurity.g4 @@ -10,15 +10,16 @@ options { tokenVocab = MDLLexer; } // SECURITY STATEMENTS // ============================================================================= -// OR MODIFY makes a security script re-runnable. Without it, re-executing the -// script that sets up roles fails on the first role that already exists, so -// role creation had to live in its own run-once file. +// A createStatement kind (MDLParser.g4), so it takes the same prefixes as every +// other document type: OR MODIFY makes a security script re-runnable (without +// it, re-executing the script that sets up roles fails on the first role that +// already exists), and a doc comment attaches to it (#731). createModuleRoleStatement - : CREATE (OR MODIFY)? MODULE ROLE qualifiedName (DESCRIPTION STRING_LITERAL)? + : MODULE ROLE qualifiedName (DESCRIPTION STRING_LITERAL)? ; dropModuleRoleStatement - : DROP MODULE ROLE qualifiedName + : DROP MODULE ROLE ifExists? qualifiedName ; createUserRoleStatement diff --git a/mdl/grammar/domains/MDLSettings.g4 b/mdl/grammar/domains/MDLSettings.g4 index 87da00016f..e09bfa3c5f 100644 --- a/mdl/grammar/domains/MDLSettings.g4 +++ b/mdl/grammar/domains/MDLSettings.g4 @@ -643,7 +643,7 @@ keyword | COUNT | SUM | AVG | MIN | MAX | DISTINCT | ALL | ASC | DESC | UNION | INTERSECT | SUBTRACT | EXISTS | CAST | COALESCE | TRIM | LENGTH | CONTAINS | MATCH - | AVERAGE | MINIMUM | MAXIMUM | REDUCE | ANY | INITIAL + | AVERAGE | MINIMUM | MAXIMUM | REDUCE | ANY | INITIAL | USING | IS_NULL | IS_NOT_NULL | NOT_NULL | HEAD | TAIL | FIND | SORT | EMPTY | LIST_OF | LIST_KW | EQUALS_OP diff --git a/mdl/grammar/keyword_coverage_test.go b/mdl/grammar/keyword_coverage_test.go index 88ee8b834e..a05f246222 100644 --- a/mdl/grammar/keyword_coverage_test.go +++ b/mdl/grammar/keyword_coverage_test.go @@ -27,7 +27,7 @@ func TestKeywordRuleCoverage(t *testing.T) { // Literals "STRING_LITERAL": true, "DOLLAR_STRING": true, "NUMBER_LITERAL": true, "MENDIX_TOKEN": true, // Punctuation - "SEMICOLON": true, "COMMA": true, "DOT": true, + "SEMICOLON": true, "COMMA": true, "TRAILING_COMMA": true, "DOT": true, "LPAREN": true, "RPAREN": true, "LBRACE": true, "RBRACE": true, "LBRACKET": true, "RBRACKET": true, diff --git a/mdl/langver/langver.go b/mdl/langver/langver.go index 2a814feac8..07ee98bc70 100644 --- a/mdl/langver/langver.go +++ b/mdl/langver/langver.go @@ -22,6 +22,8 @@ package langver import ( "fmt" "regexp" + "strconv" + "strings" ) // Version is an MDL language version, the number in `mdl <n>;`. @@ -132,3 +134,73 @@ func (c Change) Warning(v Version) string { "Start the script with `%s;` to opt in, or keep this meaning explicitly.", c.Old, v, c.Since, c.New, c.Since) } + +// ScanHeader reads the language version from the `mdl <n>;` header src starts +// with, before it is parsed. It is for the one decision that has to be made +// before lexing: under mdl 1 a backslash in a string literal is an ordinary +// character, which changes where the literal ends (ADR-0010 R11). +// +// It reads the header exactly as the grammar does — the first statement, +// after any whitespace, `--` and `/* */` comments, with the same between its +// tokens — and returns V0 when there is none. A header naming a version this +// mxcli does not know also returns V0: the parser refuses the script anyway. +func ScanHeader(src string) Version { + i := skipTrivia(src, 0) + j := i + for j < len(src) && isIdentByte(src[j]) { + j++ + } + if !strings.EqualFold(src[i:j], "mdl") { + return V0 + } + i = skipTrivia(src, j) + j = i + for j < len(src) && src[j] >= '0' && src[j] <= '9' { + j++ + } + if j == i || (j < len(src) && (isIdentByte(src[j]) || src[j] == '.')) { + return V0 + } + n, err := strconv.Atoi(src[i:j]) + if err != nil { + return V0 + } + if k := skipTrivia(src, j); k >= len(src) || src[k] != ';' { + return V0 + } + if v := Version(n); v.Known() { + return v + } + return V0 +} + +// skipTrivia returns the index of the first byte at or after i that is not +// whitespace or inside a `--` or `/* */` comment. A `/** */` doc comment is a +// token, not trivia, so it stops the scan. +func skipTrivia(src string, i int) int { + for i < len(src) { + switch { + case src[i] == ' ' || src[i] == '\t' || src[i] == '\r' || src[i] == '\n' || src[i] == '\v' || src[i] == '\f': + i++ + case strings.HasPrefix(src[i:], "--"): + if nl := strings.IndexByte(src[i:], '\n'); nl >= 0 { + i += nl + 1 + } else { + return len(src) + } + case strings.HasPrefix(src[i:], "/*") && !strings.HasPrefix(src[i:], "/**"): + end := strings.Index(src[i+2:], "*/") + if end < 0 { + return len(src) + } + i += 2 + end + 2 + default: + return i + } + } + return i +} + +func isIdentByte(c byte) bool { + return c == '_' || c >= 'a' && c <= 'z' || c >= 'A' && c <= 'Z' || c >= '0' && c <= '9' || c >= 0x80 +} diff --git a/mdl/langver/langver_test.go b/mdl/langver/langver_test.go index 869257a0c7..659bc87ba6 100644 --- a/mdl/langver/langver_test.go +++ b/mdl/langver/langver_test.go @@ -64,3 +64,25 @@ func TestIsHeaderLine(t *testing.T) { } } } + +func TestScanHeader(t *testing.T) { + for src, want := range map[string]Version{ + "mdl 1;\nshow entities;": V1, + "MDL 1 ;": V1, + " -- a note\n/* block */\nmdl\n1\n;": V1, + "mdl /* between */ 1 -- and here\n;": V1, + "show entities;": V0, + "": V0, + "mdl 1": V0, // no `;`: not a header + "mdl 1.5;": V0, + "mdl 99;": V0, // unknown: the parser refuses it + "mdlx 1;": V0, + "/** doc */ mdl 1;": V0, // a doc comment is a token + "create entity mdl.X ( A: String );\nmdl 1;": V0, // only the first statement + "mdl 0;": V0, + } { + if got := ScanHeader(src); got != want { + t.Errorf("ScanHeader(%q) = %v, want %v", src, got, want) + } + } +} diff --git a/mdl/roundtrip/allowlist_test.go b/mdl/roundtrip/allowlist_test.go index 9b61769185..019cc914bd 100644 --- a/mdl/roundtrip/allowlist_test.go +++ b/mdl/roundtrip/allowlist_test.go @@ -102,53 +102,6 @@ var knownFailures = map[string]knownFailure{ "entity NanoflowCommons.Geolocation": {laws: []law{lawGetPut}, issue: "#721", why: "access-rule member pointers and generalization flags dropped (#721 B)"}, "entity NanoflowCommons.Position": {laws: []law{lawGetPut}, issue: "#721", why: "access-rule member pointers and generalization flags dropped (#721 B)"}, - // #721 F: plain `create` for user roles and JavaScript actions. - "javascript action DataWidgets.Export_To_Excel": {laws: []law{lawExec}, issue: "#721", why: "describe prints a plain create (#721 F)"}, - "javascript action FeedbackModule.JS_GetSingleLocalStorageObjectItem": {laws: []law{lawExec}, issue: "#721", why: "describe prints a plain create (#721 F)"}, - "javascript action FeedbackModule.JS_PopulateFeedbackMetadata": {laws: []law{lawExec}, issue: "#721", why: "describe prints a plain create (#721 F)"}, - "javascript action FeedbackModule.JS_Recalculate_MendixModal_Error_PopUp_Zindex": {laws: []law{lawExec}, issue: "#721", why: "describe prints a plain create (#721 F)"}, - "javascript action FeedbackModule.JS_RevokeUploadedFileFromMemory": {laws: []law{lawExec}, issue: "#721", why: "describe prints a plain create (#721 F)"}, - "javascript action FeedbackModule.JS_SetFeedbackStorageObject": {laws: []law{lawExec}, issue: "#721", why: "describe prints a plain create (#721 F)"}, - "javascript action FeedbackModule.JS_SetSingleLocalStorageObjectItem": {laws: []law{lawExec}, issue: "#721", why: "describe prints a plain create (#721 F)"}, - "javascript action FeedbackModule.JS_ToggleFeedbackAnnotateWidget": {laws: []law{lawExec}, issue: "#721", why: "describe prints a plain create (#721 F)"}, - "javascript action FeedbackModule.JS_ToggleFeedbackScreenshotWidget": {laws: []law{lawExec}, issue: "#721", why: "describe prints a plain create (#721 F)"}, - "javascript action FeedbackModule.JS_UploadAndConvertToFileBlobURL": {laws: []law{lawExec}, issue: "#721", why: "describe prints a plain create (#721 F)"}, - "javascript action FeedbackModule.JS_isStrictMode": {laws: []law{lawExec}, issue: "#721", why: "describe prints a plain create (#721 F)"}, - "javascript action NanoflowCommons.Base64Decode": {laws: []law{lawExec}, issue: "#721", why: "describe prints a plain create (#721 F)"}, - "javascript action NanoflowCommons.Base64DecodeToImage": {laws: []law{lawExec}, issue: "#721", why: "describe prints a plain create (#721 F)"}, - "javascript action NanoflowCommons.Base64Encode": {laws: []law{lawExec}, issue: "#721", why: "describe prints a plain create (#721 F)"}, - "javascript action NanoflowCommons.CallPhoneNumber": {laws: []law{lawExec}, issue: "#721", why: "describe prints a plain create (#721 F)"}, - "javascript action NanoflowCommons.ClearCachedSessionData": {laws: []law{lawExec}, issue: "#721", why: "describe prints a plain create (#721 F)"}, - "javascript action NanoflowCommons.ClearLocalStorage": {laws: []law{lawExec}, issue: "#721", why: "describe prints a plain create (#721 F)"}, - "javascript action NanoflowCommons.GenerateUniqueID": {laws: []law{lawExec}, issue: "#721", why: "describe prints a plain create (#721 F)"}, - "javascript action NanoflowCommons.GetPlatform": {laws: []law{lawExec}, issue: "#721", why: "describe prints a plain create (#721 F)"}, - "javascript action NanoflowCommons.GetRemoteUrl": {laws: []law{lawExec}, issue: "#721", why: "describe prints a plain create (#721 F)"}, - "javascript action NanoflowCommons.GetStorageItemString": {laws: []law{lawExec}, issue: "#721", why: "describe prints a plain create (#721 F)"}, - "javascript action NanoflowCommons.GetStraightLineDistance": {laws: []law{lawExec}, issue: "#721", why: "describe prints a plain create (#721 F)"}, - "javascript action NanoflowCommons.HideProgress": {laws: []law{lawExec}, issue: "#721", why: "describe prints a plain create (#721 F)"}, - "javascript action NanoflowCommons.IsConnectedToServer": {laws: []law{lawExec}, issue: "#721", why: "describe prints a plain create (#721 F)"}, - "javascript action NanoflowCommons.NavigateTo": {laws: []law{lawExec}, issue: "#721", why: "describe prints a plain create (#721 F)"}, - "javascript action NanoflowCommons.OpenMap": {laws: []law{lawExec}, issue: "#721", why: "describe prints a plain create (#721 F)"}, - "javascript action NanoflowCommons.OpenURL": {laws: []law{lawExec}, issue: "#721", why: "describe prints a plain create (#721 F)"}, - "javascript action NanoflowCommons.Reload": {laws: []law{lawExec}, issue: "#721", why: "describe prints a plain create (#721 F)"}, - "javascript action NanoflowCommons.RemoveStorageItem": {laws: []law{lawExec}, issue: "#721", why: "describe prints a plain create (#721 F)"}, - "javascript action NanoflowCommons.RequestLocationPermission": {laws: []law{lawExec}, issue: "#721", why: "describe prints a plain create (#721 F)"}, - "javascript action NanoflowCommons.SendTextMessage": {laws: []law{lawExec}, issue: "#721", why: "describe prints a plain create (#721 F)"}, - "javascript action NanoflowCommons.SignOut": {laws: []law{lawExec}, issue: "#721", why: "describe prints a plain create (#721 F)"}, - "javascript action NanoflowCommons.StorageItemExists": {laws: []law{lawExec}, issue: "#721", why: "describe prints a plain create (#721 F)"}, - "javascript action NanoflowCommons.TimeBetween": {laws: []law{lawExec}, issue: "#721", why: "describe prints a plain create (#721 F)"}, - "javascript action NanoflowCommons.ToggleSidebar": {laws: []law{lawExec}, issue: "#721", why: "describe prints a plain create (#721 F)"}, - "javascript action NanoflowCommons.Wait": {laws: []law{lawExec}, issue: "#721", why: "describe prints a plain create (#721 F)"}, - "javascript action WebActions.FocusNext": {laws: []law{lawExec}, issue: "#721", why: "describe prints a plain create (#721 F)"}, - "javascript action WebActions.FocusPrevious": {laws: []law{lawExec}, issue: "#721", why: "describe prints a plain create (#721 F)"}, - "javascript action WebActions.ReadCookie": {laws: []law{lawExec}, issue: "#721", why: "describe prints a plain create (#721 F)"}, - "javascript action WebActions.ScrollTo": {laws: []law{lawExec}, issue: "#721", why: "describe prints a plain create (#721 F)"}, - "javascript action WebActions.SetCookie": {laws: []law{lawExec}, issue: "#721", why: "describe prints a plain create (#721 F)"}, - "javascript action WebActions.SetFavicon": {laws: []law{lawExec}, issue: "#721", why: "describe prints a plain create (#721 F)"}, - "javascript action WebActions.SetFocus": {laws: []law{lawExec}, issue: "#721", why: "describe prints a plain create (#721 F)"}, - "user role 'Administrator'": {laws: []law{lawExec}, issue: "#721", why: "describe prints a plain create (#721 F)"}, - "user role 'User'": {laws: []law{lawExec}, issue: "#721", why: "describe prints a plain create (#721 F)"}, - // #721 E: menus. "menu Atlas_Core.Phone_Menu": {laws: []law{lawGetPut}, issue: "#721", why: "nested menu items dropped (#721 E)"}, "menu Atlas_Core.Tablet_Menu": {laws: []law{lawGetPut}, issue: "#721", why: "nested menu items dropped (#721 E)"}, @@ -158,7 +111,7 @@ var knownFailures = map[string]knownFailure{ "microflow Administration.ChangePassword": {laws: []law{lawGetPut, lawPutGet}, issue: "#721", why: "whole-document rebuild: curves, merges, case values (#721 A)"}, "microflow Administration.ManageMyAccount": {laws: []law{lawGetPut, lawPutGet}, issue: "#721", why: "whole-document rebuild: curves, merges, case values (#721 A)"}, "microflow Administration.NewAccount": {laws: []law{lawGetPut}, issue: "#721", why: "whole-document rebuild: curves, merges, case values (#721 A)"}, - "microflow Administration.NewWebServiceAccount": {laws: []law{lawGetPut, lawPutGet}, issue: "#721", why: "whole-document rebuild: curves, merges, case values (#721 A)"}, + "microflow Administration.NewWebServiceAccount": {laws: []law{lawGetPut}, issue: "#721", why: "whole-document rebuild: curves, merges, case values (#721 A)"}, "microflow Administration.SaveNewAccount": {laws: []law{lawGetPut, lawPutGet}, issue: "#721", why: "whole-document rebuild: curves, merges, case values (#721 A)"}, "microflow Administration.ShowMyPasswordForm": {laws: []law{lawGetPut}, issue: "#721", why: "whole-document rebuild: curves, merges, case values (#721 A)"}, "microflow Administration.ShowPasswordForm": {laws: []law{lawGetPut}, issue: "#721", why: "whole-document rebuild: curves, merges, case values (#721 A)"}, diff --git a/mdl/roundtrip/describe_terminators_test.go b/mdl/roundtrip/describe_terminators_test.go new file mode 100644 index 0000000000..ceb279d3ac --- /dev/null +++ b/mdl/roundtrip/describe_terminators_test.go @@ -0,0 +1,156 @@ +// SPDX-License-Identifier: Apache-2.0 + +//go:build integration + +package roundtrip + +import ( + "fmt" + "sort" + "strings" + "testing" + + "github.com/antlr4-go/antlr/v4" + + "github.com/mendixlabs/mxcli/mdl/grammar/parser" + "github.com/mendixlabs/mxcli/mdl/visitor" +) + +// TestPedAppDescribeIsValidMdl1 holds describe to ADR-0010 R11/R12 (#744): +// every statement it prints ends with `;` and none is followed by a SQL*Plus +// `/` line, so the output of every document type means the same under mdl 0 +// and mdl 1. It checks the terminators on the parse tree itself, so it does not +// depend on the parser enforcing them, and then parses the output again under +// an `mdl 1;` header, which must report nothing. +func TestPedAppDescribeIsValidMdl1(t *testing.T) { + h := newHarness(t) + defer h.close() + + targets := describeTargets(h) + if len(targets) < 100 { + t.Fatalf("only %d describe targets — the enumeration is broken", len(targets)) + } + checked := 0 + for _, target := range targets { + t.Run(target, func(t *testing.T) { + out, err := h.describe(target) + if err != nil || strings.TrimSpace(out) == "" { + t.Skipf("describe %s: err=%v (judged by TestPedAppRoundTrip)", target, err) + } + if _, errs := visitor.Build(out); len(errs) > 0 { + if hasKnownLaw(target, lawParse) { + t.Skipf("does not parse under mdl 0 either (allowlisted): %v", errs[0]) + } + t.Fatalf("describe output does not parse: %v\n%s", errs[0], out) + } + checked++ + for _, problem := range terminatorProblems(out) { + t.Errorf("%s\n--- describe output ---\n%s", problem, out) + } + if _, errs := visitor.Build("mdl 1;\n" + out); len(errs) > 0 { + t.Errorf("describe output is not valid under mdl 1: %v\n--- describe output ---\n%s", errs, out) + } + }) + } + if checked == 0 { + t.Fatal("no describe output was checked — the test proves nothing") + } +} + +// TestDescribeTerminatorCheck is the control for terminatorProblems: without +// it, a check that never fires would make TestPedAppDescribeIsValidMdl1 pass +// on the output #744 was filed against. +func TestDescribeTerminatorCheck(t *testing.T) { + for script, want := range map[string]int{ + "create constant M.C type String default 'x';\n": 0, + "create constant M.C type String default 'x';\n/\n": 1, + "create constant M.C type String default 'x'\n/\n": 2, + "create constant M.C type String default 'x'\n": 1, + "create microflow M.F ()\nbegin\n return;\nend;\n": 0, + "create microflow M.F ()\nbegin\n return;\nend;\n/\n": 1, + "create page M.P (title: 'P', layout: Atlas_Core.Atlas_Default) {\n}\n": 1, + } { + if got := terminatorProblems(script); len(got) != want { + t.Errorf("%q: %d problems %v, want %d", script, len(got), got, want) + } + } +} + +// describeTargets is every document the round trip enumerates, plus each +// module (alone and with all its objects), navigation and settings. +func describeTargets(h *harness) []string { + var out []string + modules := map[string]bool{} + for _, d := range h.documents() { + out = append(out, d.target()) + if mod, _, ok := strings.Cut(d.name, "."); ok && !strings.HasPrefix(d.name, "'") { + modules[mod] = true + } + } + var mods []string + for m := range modules { + mods = append(mods, m) + } + sort.Strings(mods) + for _, m := range mods { + out = append(out, "module "+m, "module "+m+" with all") + } + return append(out, "navigation", "settings") +} + +func hasKnownLaw(key string, l law) bool { + for _, k := range knownFailures[key].laws { + if k == l { + return true + } + } + return false +} + +// terminatorProblems reports each top-level statement of script that does not +// end with `;` or is followed by `/`. It reads the terminators off the +// statement's last tokens: microflow, nanoflow and workflow bodies end in +// `SEMICOLON? SLASH?` of their own, so the statement rule's may be empty. +func terminatorProblems(script string) []string { + lexer := parser.NewMDLLexer(antlr.NewInputStream(script)) + lexer.RemoveErrorListeners() + p := parser.NewMDLParser(antlr.NewCommonTokenStream(lexer, antlr.TokenDefaultChannel)) + p.RemoveErrorListeners() + var problems []string + for _, stmt := range p.Program().AllStatement() { + last := lastTokens(stmt, 2) + if len(last) == 0 { + continue + } + if last[0].GetTokenType() == parser.MDLParserSLASH { + problems = append(problems, fmt.Sprintf("line %d: a `/` line follows the statement", last[0].GetLine())) + last = last[1:] + } + if len(last) > 0 && last[0].GetTokenType() != parser.MDLParserSEMICOLON { + problems = append(problems, fmt.Sprintf("line %d: the statement ending at %q has no `;`", last[0].GetLine(), last[0].GetText())) + } + } + return problems +} + +// lastTokens returns up to n of the tree's last tokens, the last first. +func lastTokens(tree antlr.Tree, n int) []antlr.Token { + var out []antlr.Token + var walk func(antlr.Tree) + walk = func(t antlr.Tree) { + if len(out) == n { + return + } + if tn, ok := t.(antlr.TerminalNode); ok { + if tok := tn.GetSymbol(); tok != nil && tok.GetTokenType() != antlr.TokenEOF { + out = append(out, tok) + } + return + } + for i := t.GetChildCount() - 1; i >= 0 && len(out) < n; i-- { + walk(t.GetChild(i)) + } + } + walk(tree) + return out +} diff --git a/mdl/roundtrip/doc.go b/mdl/roundtrip/doc.go index b677c50588..bc1abc5233 100644 --- a/mdl/roundtrip/doc.go +++ b/mdl/roundtrip/doc.go @@ -9,6 +9,11 @@ // nothing. // - PutGet: describing the document again returns what was described before. // +// It also holds the execute-both property test of `mxcli fmt --upgrade` +// (upgrade_property_test.go): every mdl-examples script and its upgrade run on +// two copies of the fixture and must write the same model (by default only the +// scripts it rewrites beyond the header; MXCLI_UPGRADE_ALL=1 runs all of them). +// // Every other round-trip test in the repo runs on mxcli-authored content, where // an element's GUID equals its $ID, so identity loss cannot show up there. This // fixture was authored by Studio Pro and the marketplace, so it can. diff --git a/mdl/roundtrip/list_activity_test.go b/mdl/roundtrip/list_activity_test.go new file mode 100644 index 0000000000..cb3b55fc24 --- /dev/null +++ b/mdl/roundtrip/list_activity_test.go @@ -0,0 +1,98 @@ +// SPDX-License-Identifier: Apache-2.0 + +//go:build integration + +package roundtrip + +import ( + "regexp" + "strings" + "testing" +) + +// listActivityRows is one statement per row of the §4 Microflows table of +// PROPOSAL_mdl_beta_syntax_freeze.md, written against PedApp's +// FeedbackModule.Feedback. Each is also the exact line describe must print +// back under mdl 1 (#733). +var listActivityRows = []string{ + "$Open = filter $Feedbacks by Subject = 'Open';", + "$Wide = filter $Feedbacks where $currentObject/ScreenWidth > 1000;", + "$Match = find $Feedbacks by ScreenWidth = 3;", + "$Tall = find $Feedbacks where $currentObject/ScreenHeight > 2;", + "$Sorted = sort $Feedbacks by Subject desc, ScreenWidth asc;", + "$First = head $Feedbacks;", + "$Rest = tail $Feedbacks;", + "$Page = range $Feedbacks offset 20 limit 10;", + "$All = union $Feedbacks with $Others;", + "$Both = intersect $Feedbacks with $Others;", + "$Left = subtract $Others from $Feedbacks;", + "$Has = contains $One in $Feedbacks;", + "$Same = equals $Feedbacks and $Others;", + "$N = count $Feedbacks;", + "$Total = sum $Feedbacks by ScreenWidth;", + "$Double = sum $Feedbacks of $currentObject/ScreenWidth * 2;", + "$Avg = average $Feedbacks by ScreenWidth;", + "$Min = minimum $Feedbacks by ScreenHeight;", + "$Max = maximum $Feedbacks by ScreenHeight;", + "$AllShown = all $Feedbacks where $currentObject/_showEmail;", + "$AnyShown = any $Feedbacks where $currentObject/_showEmail;", + "$Csv = reduce $Feedbacks from '' as String using $currentResult + $currentObject/Subject;", +} + +const listActivityTarget = "microflow MyFirstModule.ListActivities" + +func listActivityScript() string { + return "mdl 1;\ncreate microflow MyFirstModule.ListActivities (\n" + + " $Feedbacks: List of FeedbackModule.Feedback,\n" + + " $Others: List of FeedbackModule.Feedback,\n" + + " $One: FeedbackModule.Feedback\n)\nreturns Boolean\nbegin\n " + + strings.Join(listActivityRows, "\n ") + "\n return $Has;\nend;\n" +} + +var positionLine = regexp.MustCompile(`(?m)^\s*@(position|anchor|curve|merge)\b.*\n`) + +// Every row of the §4 table parses, executes on the Studio Pro-authored +// fixture, and reads back through describe under the header as the same +// statement; the described microflow executes back to the same description. +func TestPedAppListActivitiesUnderMdl1(t *testing.T) { + h := newHarness(t) + defer h.close() + + if err := h.exec(listActivityScript()); err != nil { + t.Fatalf("exec under mdl 1: %v\n%s", err, h.out.String()) + } + first := h.describeUnder("mdl 1;", listActivityTarget) + body := positionLine.ReplaceAllString(first, "") + for _, row := range listActivityRows { + if !strings.Contains(body, " "+row+"\n") { + t.Errorf("describe under mdl 1 does not print %q:\n%s", row, body) + } + } + + // PutGet: executing the mdl 1 description describes back to itself. + if err := h.exec("mdl 1;\n" + first); err != nil { + t.Fatalf("exec the mdl 1 description: %v\n%s", err, first) + } + if again := h.describeUnder("mdl 1;", listActivityTarget); again != first { + t.Errorf("describe -> exec -> describe changed the microflow:\n%s", lineDiff(first, again)) + } + + // Control: outside an mdl 1 script describe keeps the call form while mdl 1 + // is a preview, so the rows above are not what a plain describe prints. + plain, err := h.describe(listActivityTarget) + if err != nil { + t.Fatal(err) + } + if strings.Contains(plain, "= head $Feedbacks;") || !strings.Contains(plain, "= head($Feedbacks);") { + t.Errorf("a plain describe must keep the call form while mdl 1 is a preview:\n%s", plain) + } +} + +// describeUnder runs describe inside a script with the given header. +func (h *harness) describeUnder(header, target string) string { + h.t.Helper() + if err := h.exec(header + "\ndescribe " + target + ";"); err != nil { + h.t.Fatalf("describe %s under %q: %v", target, header, err) + } + return h.out.String() +} diff --git a/mdl/roundtrip/retrieve_range_test.go b/mdl/roundtrip/retrieve_range_test.go new file mode 100644 index 0000000000..e090932b5d --- /dev/null +++ b/mdl/roundtrip/retrieve_range_test.go @@ -0,0 +1,165 @@ +// SPDX-License-Identifier: Apache-2.0 + +//go:build integration + +package roundtrip + +import ( + "fmt" + "sort" + "strings" + "testing" + + "go.mongodb.org/mongo-driver/v2/bson" + + "github.com/mendixlabs/mxcli/mdl/executor" + "github.com/mendixlabs/mxcli/mdl/visitor" +) + +// TestPedAppRoundTrip_RetrieveRange is ako/mxcli#734 on Studio Pro-authored +// content: `retrieve … first` is the object range in every language version, +// and under `mdl 1;` `retrieve … limit 1` is a list of one. describe → exec +// must keep each range as stored, for both forms. +// +// FeedbackModule.PopulateUserAttributes holds PedApp's only retrieve with a +// range: Studio Pro's "First object" (a ConstantRange with SingleObject). The +// whole microflow does not pass GetPut yet (#721, allowlisted), so this test +// judges the range element itself rather than the unit's bytes. +func TestPedAppRoundTrip_RetrieveRange(t *testing.T) { + h := newHarness(t) + defer h.close() + + const target = "microflow FeedbackModule.PopulateUserAttributes" + const objectRange = "Microflows$ConstantRange SingleObject=true" + const listOfOne = "Microflows$CustomRange Limit=\"1\" Offset=\"\"" + + if got := h.retrieveRanges(t, "PopulateUserAttributes"); strings.Join(got, "; ") != objectRange { + t.Fatalf("fixture changed: the retrieve's stored range is %q, want the one object range %q", got, objectRange) + } + + // --- The object form ------------------------------------------------- + described, err := h.describe(target) + if err != nil { + t.Fatalf("describe: %v", err) + } + if !strings.Contains(described, "\n first;") || strings.Contains(described, "limit 1") { + t.Fatalf("describe must print the object range as `first` (never `limit 1`, a list under mdl 1):\n%s", described) + } + for _, header := range []string{"", "mdl 1;\n"} { + script := header + described + if warned := limitOneWarnings(t, script); warned != 0 { + t.Errorf("header %q: `first` warned MDL-V1-LIMIT1 %d time(s); it means the same in every version", header, warned) + } + if err := h.exec(script); err != nil { + t.Fatalf("header %q: exec describe output: %v", header, err) + } + if got := h.retrieveRanges(t, "PopulateUserAttributes"); strings.Join(got, "; ") != objectRange { + t.Errorf("header %q: object form stored as %q, want %q", header, got, objectRange) + } + if again, _ := h.describe(target); again != described { + t.Errorf("header %q: PutGet — describe changed after exec:\n%s", header, lineDiff(described, again)) + } + } + + // --- The list form, under mdl 1 ---------------------------------------- + listScript := "mdl 1;\n" + strings.Replace(described, "\n first;", "\n limit 1;", 1) + if err := h.exec(listScript); err != nil { + t.Fatalf("exec the list form under mdl 1: %v", err) + } + if got := h.retrieveRanges(t, "PopulateUserAttributes"); strings.Join(got, "; ") != listOfOne { + t.Fatalf("`limit 1` under mdl 1 stored as %q, want a list of one %q", got, listOfOne) + } + listDescribed, err := h.describe(target) + if err != nil { + t.Fatalf("describe the list form: %v", err) + } + if !strings.Contains(listDescribed, "\n limit 1;") || strings.Contains(listDescribed, "first;") { + t.Fatalf("describe must print a Custom range with limit 1 as `limit 1`:\n%s", listDescribed) + } + if err := h.exec("mdl 1;\n" + listDescribed); err != nil { + t.Fatalf("exec the list form's describe output under mdl 1: %v", err) + } + if got := h.retrieveRanges(t, "PopulateUserAttributes"); strings.Join(got, "; ") != listOfOne { + t.Errorf("list form's describe output under mdl 1 stored %q, want %q", got, listOfOne) + } + if again, _ := h.describe(target); again != listDescribed { + t.Errorf("PutGet under mdl 1 — describe changed after exec:\n%s", lineDiff(listDescribed, again)) + } + + // --- Control: the same text without the header keeps the alpha meaning -- + // and says so. Without this, a writer that ignored the version would pass + // both halves above. + if warned := limitOneWarnings(t, listDescribed); warned != 1 { + t.Errorf("headerless `limit 1` warned MDL-V1-LIMIT1 %d time(s), want 1", warned) + } + if err := h.exec(listDescribed); err != nil { + t.Fatalf("exec the list form's describe output without a header: %v", err) + } + if got := h.retrieveRanges(t, "PopulateUserAttributes"); strings.Join(got, "; ") != objectRange { + t.Errorf("headerless `limit 1` stored %q, want the alpha meaning, the object range %q", got, objectRange) + } +} + +// limitOneWarnings counts the MDL-V1-LIMIT1 warnings check reports for script. +func limitOneWarnings(t *testing.T, script string) int { + t.Helper() + prog, errs := visitor.Build(script) + if len(errs) > 0 { + t.Fatalf("parse: %v", errs[0]) + } + n := 0 + for _, v := range executor.ValidateLanguageVersion(prog) { + if v.RuleID == "MDL-V1-LIMIT1" { + n++ + } + } + return n +} + +// retrieveRanges returns the range of every retrieve in the named microflow, as +// stored: the range's $Type and the properties that decide object or list. +func (h *harness) retrieveRanges(t *testing.T, microflow string) []string { + t.Helper() + snap := h.snapshot() + var found []string + for id, b := range snap.units { + typ, name := typeAndName(b) + if typ != "Microflows$Microflow" || name != microflow { + continue + } + var doc bson.D + if err := bson.Unmarshal(b, &doc); err != nil { + t.Fatalf("decode %s: %v", id, err) + } + collectRanges(doc, &found) + sort.Strings(found) + return found + } + t.Fatalf("microflow %s not found", microflow) + return nil +} + +func collectRanges(v any, out *[]string) { + switch x := v.(type) { + case bson.D: + m := map[string]any{} + for _, e := range x { + m[e.Key] = e.Value + } + switch m["$Type"] { + case "Microflows$ConstantRange": + *out = append(*out, fmt.Sprintf("Microflows$ConstantRange SingleObject=%v", m["SingleObject"])) + case "Microflows$CustomRange": + // An absent expression reads as empty, the way Mendix treats it. + str := func(k string) string { s, _ := m[k].(string); return s } + *out = append(*out, fmt.Sprintf("Microflows$CustomRange Limit=%q Offset=%q", str("LimitExpression"), str("OffsetExpression"))) + } + for _, e := range x { + collectRanges(e.Value, out) + } + case bson.A: + for _, e := range x { + collectRanges(e, out) + } + } +} diff --git a/mdl/roundtrip/upgrade_property_test.go b/mdl/roundtrip/upgrade_property_test.go new file mode 100644 index 0000000000..0ef29a9641 --- /dev/null +++ b/mdl/roundtrip/upgrade_property_test.go @@ -0,0 +1,532 @@ +// SPDX-License-Identifier: Apache-2.0 + +//go:build integration + +package roundtrip + +import ( + "crypto/sha256" + "encoding/hex" + "errors" + "fmt" + "os" + "path/filepath" + "regexp" + "sort" + "strings" + "testing" + + "go.mongodb.org/mongo-driver/v2/bson" + + "github.com/mendixlabs/mxcli/mdl/upgrade" + "github.com/mendixlabs/mxcli/mdl/visitor" + "github.com/mendixlabs/mxcli/modelsdk/canon" + mmpr "github.com/mendixlabs/mxcli/modelsdk/mpr" +) + +// The safety property of `mxcli fmt --upgrade` (ako/mxcli#735, ADR-0011 +// decision 1): an upgraded script executes to the same model as the original. +// A wrong rewrite is a silent change of meaning — the script still parses and +// still runs, it just builds something else — so the only proof is to run both +// and compare what they wrote. +// +// For every mdl-examples script: +// +// 1. upgrade it, with the language header, which is the strongest form: every +// alias rewritten and the script run under mdl 1 (a script the header is +// refused for, HeaderBlockedError, is upgraded without it); +// 2. require zero deprecated spellings in the result; +// 3. execute the original on one copy of the PedApp fixture and the upgrade on +// another; +// 4. compare the two models unit by unit, as canonical BSON (canon.Digest, +// which numbers element $IDs by containment so freshly minted IDs do not +// count as a difference), with units matched by their place in the project +// tree rather than by unit ID, which is fresh for every created document. +// +// A script whose original does not execute cleanly on PedApp is out of scope +// and listed; the models are still compared, so a rewrite that changes how far +// a failing script gets is caught too. +// +// MXCLI_UPGRADE_EXAMPLES=<regexp> limits the run to matching script paths. +// +// By default only scripts the upgrade rewrites beyond adding the header and +// statement terminators are executed: with those as the only edits, the two +// runs execute the same statements (mdl/upgrade's examples test proves it), +// and what they compare is langver's gating rather than a rewrite. +// Executing the whole corpus takes about 15 minutes, which on its own exceeds +// what the CI integration step has left (ako/mxcli#742 timed out there). +// MXCLI_UPGRADE_ALL=1 executes every script, header-only ones included; run it +// when a langver.Change or a gated rewrite lands. +func TestUpgradeExecutesToTheSameModel(t *testing.T) { + a, b := newHarness(t), newHarness(t) + defer a.close() + defer b.close() + + scripts := upgradeExampleScripts(t) + all := os.Getenv("MXCLI_UPGRADE_ALL") != "" + var same, outOfScope, unparsed, headerOnly, keptVersion []string + for _, path := range scripts { + src, err := os.ReadFile(path) + if err != nil { + t.Fatal(err) + } + rel, _ := filepath.Rel("../..", path) + if _, errs := visitor.Build(string(src)); len(errs) > 0 { + unparsed = append(unparsed, rel) + continue + } + t.Run(rel, func(t *testing.T) { + res, err := upgrade.Upgrade(string(src), upgrade.Options{AddHeader: true}) + var hb *upgrade.HeaderBlockedError + if errors.As(err, &hb) { + // A construct with no mechanical rewrite keeps the script at + // mdl 0 (mdl/upgrade's keepsItsVersion lists which); its alias + // rewrites are still proven here. + keptVersion = append(keptVersion, rel) + res, err = upgrade.Upgrade(string(src), upgrade.Options{}) + } + if err != nil { + t.Fatalf("upgrade: %v", err) + } + prog, errs := visitor.Build(res.Source) + if len(errs) > 0 { + t.Fatalf("upgraded script does not parse: %v", errs[0]) + } + if n := len(prog.Deprecations); n > 0 { + t.Fatalf("upgraded script still has %d deprecated spelling(s), first %s at line %d", + n, prog.Deprecations[0].Code, prog.Deprecations[0].Line) + } + if !all && onlyTerminatorsAndHeader(res) { + headerOnly = append(headerOnly, rel) + return + } + + errA, errB, diff := executeBoth(t, a, b, string(src), res.Source) + if len(diff) > 0 { + t.Errorf("the upgrade changed what the script writes (rewrote %v):\n %s", + res.Rewritten, strings.Join(diff, "\n ")) + } + if (errA == nil) != (errB == nil) { + t.Errorf("original error: %v\nupgraded error: %v", errA, errB) + } + if errA != nil { + outOfScope = append(outOfScope, fmt.Sprintf("%s: %s", rel, firstLine(errA.Error()))) + return + } + if !t.Failed() { + same = append(same, rel) + } + }) + } + + t.Logf("%d scripts execute on PedApp and upgrade to the same model", len(same)) + t.Logf("%d scripts are out of scope: their original does not execute cleanly on PedApp:\n %s", + len(outOfScope), strings.Join(outOfScope, "\n ")) + if !all { + t.Logf("%d scripts only gain the header and statement terminators and were not executed "+ + "(MXCLI_UPGRADE_ALL=1 executes them)", len(headerOnly)) + } + t.Logf("%d scripts keep mdl 0 (a construct with no rewrite) and were upgraded without the header:\n %s", + len(keptVersion), strings.Join(keptVersion, "\n ")) + t.Logf("%d scripts do not parse (negative tests) and cannot be upgraded:\n %s", + len(unparsed), strings.Join(unparsed, "\n ")) + if os.Getenv("MXCLI_UPGRADE_EXAMPLES") == "" && len(same) < 50 { + // Hundreds execute today; a handful means the harness broke, and a + // property checked on nothing passes. + t.Errorf("only %d scripts executed cleanly — the harness is not exercising the property", len(same)) + } +} + +// terminatorCodes are the gated rewrites that only add a `;` or delete a `/` +// line. They never reach the AST — mdl/upgrade's examples test proves every +// script builds the same statements with and without them — so executing a +// script whose only edits they are compares langver's gating, not a rewrite. +var terminatorCodes = map[string]bool{"MDL-V1-SEMI": true, "MDL-V1-SLASH": true} + +// onlyTerminatorsAndHeader reports whether the upgrade's edits were the +// header and statement terminators alone. +func onlyTerminatorsAndHeader(res upgrade.Result) bool { + if len(res.Rewritten) > 0 { + return false + } + for code := range res.GatedRewritten { + if !terminatorCodes[code] { + return false + } + } + return true +} + +// TestUpgradeExecuteBoth_Controls are the controls CLAUDE.md requires of a +// "nothing changed" assertion. The comparison must see a real difference, and +// must not see the fresh IDs two runs of the same script mint. +func TestUpgradeExecuteBoth_Controls(t *testing.T) { + a, b := newHarness(t), newHarness(t) + defer a.close() + defer b.close() + + const script = `create module UpgradeControl; +create or modify enumeration UpgradeControl.Color (Red 'Red', Green 'Green'); +create or modify persistent entity UpgradeControl.Customer (Name: String(200), Color: Enumeration(UpgradeControl.Color)); +create or modify microflow UpgradeControl.ACT_Touch ($c: UpgradeControl.Customer) begin + change $c (Name = 'x'); + commit $c; +end; +` + // The same script twice: every created element gets a fresh $ID and GUID + // on each copy, and that must not register as a difference. + if errA, errB, diff := executeBoth(t, a, b, script, script); errA != nil || errB != nil || len(diff) > 0 { + t.Fatalf("the same script on both copies differs (errA=%v errB=%v):\n %s", errA, errB, strings.Join(diff, "\n ")) + } + + // One changed value MUST register, or every pass above is meaningless. + changed := strings.Replace(script, "'Green'", "'Groen'", 1) + _, _, diff := executeBoth(t, a, b, script, changed) + if len(diff) == 0 { + t.Fatal("a changed enumeration caption registered as the same model — the comparison cannot see a difference") + } + t.Logf("changed caption registered as: %s", strings.Join(diff, "; ")) + + // And a rewrite that changes meaning is caught. `create or replace + // translations` replaces the whole set where `or modify` merges; the + // registry exempts it, so a rewrite that ignored the exemption would write + // a different model. + const tr = `create or modify translations in Administration for nl_NL ('Save' as 'Opslaan'); +` + merged := tr + "create or modify translations in Administration for nl_NL ('Cancel' as 'Annuleren');\n" + replaced := tr + "create or replace translations in Administration for nl_NL ('Cancel' as 'Annuleren');\n" + errA, errB, diff := executeBoth(t, a, b, replaced, merged) + if errA != nil || errB != nil { + t.Fatalf("translations control did not execute: %v / %v", errA, errB) + } + if len(diff) == 0 { + t.Fatal("rewriting `create or replace translations` to `or modify` registered as the same model — " + + "the property test cannot catch a rewrite that changes meaning") + } + t.Logf("a meaning-changing rewrite registered as: %s", strings.Join(diff, "; ")) + + // A re-minted identity on a Studio Pro-authored element must register: + // maskFresh masks only values the fixture does not contain. + known := fixtureIdentities(a) + for id, raw := range a.orig.units { + var doc bson.D + if bson.Unmarshal(raw, &doc) != nil { + continue + } + hit := false + reminted := walkIdentities(doc, func(key string, v any) any { + b, isBin := v.(bson.Binary) + if key != "GUID" || !isBin || hit { + return v + } + hit = true + return bson.Binary{Subtype: b.Subtype, Data: []byte("re-minted guid!!")} + }).(bson.D) + if !hit { + continue + } + out, err := bson.Marshal(reminted) + if err != nil { + t.Fatal(err) + } + d1, err1 := canon.Digest(maskFresh(raw, known, a.dir)) + d2, err2 := canon.Digest(maskFresh(out, known, a.dir)) + if err1 != nil || err2 != nil || d1 == d2 { + t.Fatalf("a re-minted GUID in unit %s is invisible to the comparison (%v %v)", id, err1, err2) + } + return + } + t.Fatal("the fixture has no unit with a GUID — the identity control has nothing to change") +} + +// executeBoth runs one script on each harness, from the committed fixture, and +// returns each run's error and the differences between the models they wrote. +func executeBoth(t *testing.T, a, b *harness, srcA, srcB string) (errA, errB error, diff []string) { + t.Helper() + a.restore() + b.restore() + errA = a.exec(srcA) + errB = b.exec(srcB) + return errA, errB, compareModels(canonicalModel(t, a), canonicalModel(t, b)) +} + +// canonicalModel is a working copy as a map from a unit's place in the project +// tree to its canonical digest, plus the files outside the model. +// +// A unit is keyed by the chain of container names down to it and its own +// "$Type Name", not by its unit ID: a document created by the script gets a +// fresh unit ID on each copy. Units that share a key (unnamed units of one type +// in one container) are told apart by their sorted digests. +// +// Identity values minted for a created element — its StableId, its GUID where +// that is not its $ID, string ids — are fresh on each copy too, so they are +// masked by maskFresh. Only values the fixture does not contain are masked: a +// Studio Pro-authored element's GUID or StableId is compared as it is, so a +// rewrite that made the executor re-mint one still registers. +func canonicalModel(t *testing.T, h *harness) map[string]string { + t.Helper() + known := fixtureIdentities(h) + r, err := mmpr.Open(h.mpr) + if err != nil { + t.Fatalf("open working copy: %v", err) + } + defer r.Close() + units, err := r.ListUnits() + if err != nil { + t.Fatalf("list units: %v", err) + } + label := map[string]string{} + parent := map[string]string{} + digest := map[string]string{} + for _, u := range units { + raw, err := r.GetRawUnitBytes(u.ID) + if err != nil { + t.Fatalf("read unit %s: %v", u.ID, err) + } + typ, name := typeAndName(raw) + label[u.ID] = strings.TrimSpace(typ + " " + name) + parent[u.ID] = u.ContainerID + "/" + u.ContainmentName + d, err := canon.Digest(maskFresh(raw, known, h.dir)) + if err != nil { + sum := sha256.Sum256(raw) + d = "raw:" + hex.EncodeToString(sum[:]) + } + digest[u.ID] = d + } + var path func(id string, depth int) string + path = func(id string, depth int) string { + p, ok := parent[id] + if !ok || depth > 64 { + return "" + } + container, containment, _ := strings.Cut(p, "/") + if _, known := label[container]; !known || container == id { + return containment + ":" + label[id] + } + return path(container, depth+1) + " > " + containment + ":" + label[id] + } + byKey := map[string][]string{} + for id := range label { + k := path(id, 0) + byKey[k] = append(byKey[k], digest[id]) + } + out := map[string]string{} + for k, ds := range byKey { + sort.Strings(ds) + for i, d := range ds { + key := k + if len(ds) > 1 { + key = fmt.Sprintf("%s #%d", k, i+1) + } + out[key] = d + } + } + for rel, b := range h.snapshot().files { + sum := sha256.Sum256(b) + out["file "+rel] = hex.EncodeToString(sum[:]) + } + return out +} + +// fixtureIdentities is every identity-shaped value in the committed fixture: +// each 16-byte binary, and each string that reads as a UUID. +func fixtureIdentities(h *harness) map[string]bool { + if knownIdentities != nil { + return knownIdentities + } + knownIdentities = map[string]bool{} + for _, raw := range h.orig.units { + var doc bson.D + if bson.Unmarshal(raw, &doc) != nil { + continue + } + walkIdentities(doc, func(_ string, v any) any { + if s, ok := v.(string); ok { + for _, u := range uuidInString.FindAllString(s, -1) { + knownIdentities[identityKey(u)] = true + } + return v + } + knownIdentities[identityKey(v)] = true + return v + }) + } + return knownIdentities +} + +var knownIdentities map[string]bool + +// uuidInString matches the string spellings of a UUID mxcli and Mendix +// store, on their own or inside a larger text such as a JSON document: 32 hex +// digits, or the dashed 8-4-4-4-12 form. +var uuidInString = regexp.MustCompile(`(?i)\b(?:[0-9a-f]{32}|[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12})\b`) + +func identityKey(v any) string { + switch t := v.(type) { + case bson.Binary: + return "b:" + string(t.Data) + case string: + return "s:" + strings.ToLower(t) + } + return "" +} + +// maskFresh prepares a unit for canon.Digest when it may have been written by +// a script, not only by Studio Pro. +// +// Every identity value the fixture does not contain, and that is not one of +// the unit's own element $IDs (canon numbers those, references included), is +// freshly minted — a created element's GUID or StableId, a string id such as +// a business event channel name, or a UUID inside a text such as an agent +// document's JSON — and is replaced by a placeholder +// numbered by first appearance in a key-sorted walk. Two runs that minted the +// same number of identities in the same places are then equal. dir, the +// working copy's path, is replaced too: a statement that stores a local file +// URL stores a path that differs between the two copies. +func maskFresh(raw []byte, known map[string]bool, dir string) []byte { + var doc bson.D + if bson.Unmarshal(raw, &doc) != nil { + return raw + } + ids := map[string]bool{} + walkIdentities(doc, func(key string, v any) any { + if b, ok := v.(bson.Binary); ok && key == "$ID" { + ids[identityKey(b)] = true + ids[identityKey(uuidOf(b.Data))] = true // canon also numbers its string form + } + return v + }) + fresh := map[string]int{} + placeholder := func(k string) int { + n, ok := fresh[k] + if !ok { + n = len(fresh) + 1 + fresh[k] = n + } + return n + } + masked := walkIdentities(doc, func(key string, v any) any { + if str, ok := v.(string); ok { + if dir != "" { + str = strings.ReplaceAll(str, dir, "<working copy>") + } + return uuidInString.ReplaceAllStringFunc(str, func(u string) string { + k := identityKey(u) + if known[k] || ids[k] { + return u + } + return fmt.Sprintf("<fresh %d>", placeholder(k)) + }) + } + b := v.(bson.Binary) + k := identityKey(b) + if known[k] || ids[k] { + return v + } + data := make([]byte, 16) + copy(data, fmt.Sprintf("fresh%011d", placeholder(k))) + return bson.Binary{Subtype: b.Subtype, Data: data} + }).(bson.D) + out, err := bson.Marshal(masked) + if err != nil { + return raw + } + return out +} + +// uuidOf renders a 16-byte Mendix GUID blob as a UUID string, the way canon +// does (Microsoft layout: first three groups little-endian). +func uuidOf(b []byte) string { + return fmt.Sprintf("%02x%02x%02x%02x-%02x%02x-%02x%02x-%02x%02x-%02x%02x%02x%02x%02x%02x", + b[3], b[2], b[1], b[0], b[5], b[4], b[7], b[6], b[8], b[9], b[10], b[11], b[12], b[13], b[14], b[15]) +} + +// walkIdentities visits every 16-byte binary and every string in v, +// documents in key order, and returns v with each replaced by what f returns. +func walkIdentities(v any, f func(key string, v any) any) any { + return walkIdentitiesKey("", v, f) +} + +func walkIdentitiesKey(key string, v any, f func(string, any) any) any { + switch t := v.(type) { + case bson.D: + order := make([]int, len(t)) + for i := range order { + order[i] = i + } + sort.SliceStable(order, func(i, j int) bool { return t[order[i]].Key < t[order[j]].Key }) + out := make(bson.D, len(t)) + copy(out, t) + for _, i := range order { + out[i].Value = walkIdentitiesKey(t[i].Key, t[i].Value, f) + } + return out + case bson.A: + out := make(bson.A, len(t)) + for i, e := range t { + out[i] = walkIdentitiesKey(key, e, f) + } + return out + case bson.Binary: + if len(t.Data) == 16 { + return f(key, t) + } + case string: + return f(key, t) + } + return v +} + +func compareModels(a, b map[string]string) []string { + var out []string + for k, da := range a { + db, ok := b[k] + switch { + case !ok: + out = append(out, "only the original wrote "+k) + case da != db: + out = append(out, "different content: "+k) + } + } + for k := range b { + if _, ok := a[k]; !ok { + out = append(out, "only the upgrade wrote "+k) + } + } + sort.Strings(out) + return out +} + +func upgradeExampleScripts(t *testing.T) []string { + t.Helper() + var filter *regexp.Regexp + if f := os.Getenv("MXCLI_UPGRADE_EXAMPLES"); f != "" { + filter = regexp.MustCompile(f) + } + var out []string + err := filepath.Walk("../../mdl-examples", func(p string, info os.FileInfo, err error) error { + if err != nil { + return err + } + if !info.IsDir() && strings.HasSuffix(p, ".mdl") && (filter == nil || filter.MatchString(p)) { + out = append(out, p) + } + return nil + }) + if err != nil { + t.Fatal(err) + } + sort.Strings(out) + if len(out) == 0 { + t.Fatal("no mdl-examples scripts found") + } + return out +} + +func firstLine(s string) string { + s, _, _ = strings.Cut(s, "\n") + if r := []rune(s); len(r) > 160 { + return string(r[:160]) + "…" + } + return s +} diff --git a/mdl/roundtrip/viewentity_identity_test.go b/mdl/roundtrip/viewentity_identity_test.go new file mode 100644 index 0000000000..fc6fa615f4 --- /dev/null +++ b/mdl/roundtrip/viewentity_identity_test.go @@ -0,0 +1,181 @@ +// SPDX-License-Identifier: Apache-2.0 + +//go:build integration + +package roundtrip + +import ( + "bytes" + "encoding/hex" + "os" + "path/filepath" + "testing" + + "go.mongodb.org/mongo-driver/v2/bson" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/backend" + modelsdkbackend "github.com/mendixlabs/mxcli/mdl/backend/modelsdk" + "github.com/mendixlabs/mxcli/mdl/executor" + "github.com/mendixlabs/mxcli/mdl/visitor" + mmpr "github.com/mendixlabs/mxcli/modelsdk/mpr" +) + +// viewsFixtureDir holds MyFirstModule.VCar, a view entity authored in Studio +// Pro 11.14.0, so its GUIDs differ from its $IDs (testdata/testapp-views/README.md). +// PedApp has no view entity, and on one mxcli created GUID == $ID, so a +// re-minted GUID reproduces the same value and the test could not fail. +const ( + viewsFixtureDir = "../../testdata/testapp-views" + viewsFixtureMPR = "TestApp.mpr" +) + +const vcarBody = `view entity MyFirstModule.VCar ( + ModelID: Long, + Brand: String(200) +) as ( + from MyFirstModule.Car as c + select c.ModelId as ModelID + , c.Brand as Brand +); +` + +// R1 (#731): `create or replace` means `create or modify` on every document +// type, and for a view entity that is the identity-carrying rewrite rather than +// a delete and recreate. Under mdl 0 the old meaning is kept, which is the +// control: it re-mints the GUID, so the assertion can detect a lost one. +func TestViewEntityCreateOrModifyKeepsGUID(t *testing.T) { + cases := []struct { + name string + script string + keepGUID bool + }{ + {"create or modify", "create or modify " + vcarBody, true}, + {"mdl 1 create or replace", "mdl 1;\ncreate or replace " + vcarBody, true}, + // Control: mdl 0 keeps delete-and-recreate, so the GUID changes. + {"mdl 0 create or replace (control)", "create or replace " + vcarBody, false}, + } + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + mpr := copyViewsFixture(t) + before := vcarGUIDs(t, mpr) + for name, id := range before { + if id[0] == id[1] { + t.Fatalf("fixture %s has GUID == $ID; the test cannot detect a re-minted GUID", name) + } + } + + runScript(t, mpr, c.script) + + after := vcarGUIDs(t, mpr) + for name, want := range before { + got, ok := after[name] + kept := ok && got[1] == want[1] + if c.keepGUID && !kept { + t.Errorf("%s: GUID %s -> %s; the rewrite must keep the stored GUID", name, want[1], got[1]) + } + if !c.keepGUID && kept { + t.Errorf("%s: GUID kept under the control; the test cannot tell a carried GUID from a lost one", name) + } + } + }) + } +} + +// vcarGUIDs maps VCar and each of its attributes to [$ID, GUID] as hex. +func vcarGUIDs(t *testing.T, mprPath string) map[string][2]string { + t.Helper() + r, err := mmpr.Open(mprPath) + if err != nil { + t.Fatalf("open: %v", err) + } + defer r.Close() + units, err := r.ListUnits() + if err != nil { + t.Fatalf("list units: %v", err) + } + for _, u := range units { + b, err := r.GetRawUnitBytes(u.ID) + if err != nil { + t.Fatalf("read unit: %v", err) + } + if typ, _ := typeAndName(b); typ != "DomainModels$DomainModel" { + continue + } + var dm bson.D + if err := bson.Unmarshal(b, &dm); err != nil { + t.Fatalf("decode: %v", err) + } + ents, _ := field(dm, "Entities").(bson.A) + for _, e := range ents { + ent, ok := e.(bson.D) + if !ok || field(ent, "Name") != "VCar" { + continue + } + out := map[string][2]string{"entity VCar": ids(ent)} + attrs, _ := field(ent, "Attributes").(bson.A) + for _, a := range attrs { + if attr, ok := a.(bson.D); ok { + name, _ := field(attr, "Name").(string) + out["attribute "+name] = ids(attr) + } + } + return out + } + } + t.Fatal("MyFirstModule.VCar not found") + return nil +} + +func field(d bson.D, key string) any { + for _, e := range d { + if e.Key == key { + return e.Value + } + } + return nil +} + +func ids(el bson.D) [2]string { + h := func(v any) string { + if b, ok := v.(bson.Binary); ok { + return hex.EncodeToString(b.Data) + } + return "" + } + return [2]string{h(field(el, "$ID")), h(field(el, "GUID"))} +} + +func copyViewsFixture(t *testing.T) string { + t.Helper() + src, err := filepath.Abs(viewsFixtureDir) + if err != nil { + t.Fatal(err) + } + dst := t.TempDir() + if err := os.CopyFS(dst, os.DirFS(src)); err != nil { + t.Fatalf("copy fixture: %v", err) + } + return filepath.Join(dst, viewsFixtureMPR) +} + +// runScript runs a script the way `mxcli exec` does after its preflight. +func runScript(t *testing.T, mprPath, script string) { + t.Helper() + var out bytes.Buffer + exe := executor.New(&out) + exe.SetQuiet(true) + exe.SetBackendFactory(func() backend.FullBackend { return modelsdkbackend.New() }) + if err := exe.Execute(&ast.ConnectStmt{Path: mprPath}); err != nil { + t.Fatalf("connect: %v", err) + } + prog, errs := visitor.Build(script) + if len(errs) > 0 { + t.Fatalf("parse: %v", errs[0]) + } + if err := exe.ExecuteProgram(prog); err != nil { + t.Fatalf("exec: %v\n%s", err, out.String()) + } + _ = exe.Execute(&ast.DisconnectStmt{}) + _ = exe.Close() +} diff --git a/mdl/suggest/suggest.go b/mdl/suggest/suggest.go new file mode 100644 index 0000000000..35bea2bb07 --- /dev/null +++ b/mdl/suggest/suggest.go @@ -0,0 +1,94 @@ +// SPDX-License-Identifier: Apache-2.0 + +// Package suggest finds the name a misspelling most likely meant, for the +// "did you mean" part of an error about an unknown name. +package suggest + +import "strings" + +// Closest returns the known name a misspelling most likely meant, +// or "" when nothing is close enough to be worth guessing. Case-insensitive +// prefix/substring first, then a single edit, then — for a name of five +// letters or more — the nearest within two edits, counting a swap of two +// neighbouring letters as one (`Verison`, `Passwd`). +func Closest(name string, known []string) string { + lower := strings.ToLower(name) + for _, k := range known { + lk := strings.ToLower(k) + if strings.HasPrefix(lk, lower) || strings.HasPrefix(lower, lk) || strings.Contains(lk, lower) { + return k + } + } + for _, k := range known { + if WithinOneEdit(lower, strings.ToLower(k)) { + return k + } + } + if len(lower) < 5 { + return "" + } + best, bestDist := "", 3 + for _, k := range known { + if d := editDistance(lower, strings.ToLower(k)); d < bestDist { + best, bestDist = k, d + } + } + return best +} + +// editDistance is the optimal string alignment distance: insertions, +// deletions, substitutions and swaps of two neighbouring bytes. +func editDistance(a, b string) int { + d := make([][]int, len(a)+1) + for i := range d { + d[i] = make([]int, len(b)+1) + d[i][0] = i + } + for j := 0; j <= len(b); j++ { + d[0][j] = j + } + for i := 1; i <= len(a); i++ { + for j := 1; j <= len(b); j++ { + cost := 1 + if a[i-1] == b[j-1] { + cost = 0 + } + d[i][j] = min(d[i-1][j]+1, d[i][j-1]+1, d[i-1][j-1]+cost) + if i > 1 && j > 1 && a[i-1] == b[j-2] && a[i-2] == b[j-1] { + d[i][j] = min(d[i][j], d[i-2][j-2]+1) + } + } + } + return d[len(a)][len(b)] +} + +// WithinOneEdit reports whether a and b differ by at most one insertion, +// deletion or substitution. +func WithinOneEdit(a, b string) bool { + if a == b { + return true + } + if len(a) > len(b) { + a, b = b, a + } + if len(b)-len(a) > 1 { + return false + } + i, j, edits := 0, 0, 0 + for i < len(a) && j < len(b) { + if a[i] == b[j] { + i++ + j++ + continue + } + edits++ + if edits > 1 { + return false + } + if len(a) == len(b) { + i++ + } + j++ + } + return true +} diff --git a/mdl/suggest/suggest_test.go b/mdl/suggest/suggest_test.go new file mode 100644 index 0000000000..e836b629d5 --- /dev/null +++ b/mdl/suggest/suggest_test.go @@ -0,0 +1,22 @@ +// SPDX-License-Identifier: Apache-2.0 + +package suggest + +import "testing" + +func TestClosest(t *testing.T) { + known := []string{"Path", "Version", "ServiceName", "Password", "Enabled", "DisplayName"} + for in, want := range map[string]string{ + "pathh": "Path", // prefix + "Verison": "Version", // swapped neighbours + "Passwd": "Password", // two deletions + "Enabeld": "Enabled", // swapped neighbours + "DisplayNmae": "DisplayName", // swapped neighbours + "Colour": "", // nothing close + "Xyz": "", // too short to guess past one edit + } { + if got := Closest(in, known); got != want { + t.Errorf("Closest(%q) = %q, want %q", in, got, want) + } + } +} diff --git a/mdl/upgrade/examples_test.go b/mdl/upgrade/examples_test.go new file mode 100644 index 0000000000..6187a4afab --- /dev/null +++ b/mdl/upgrade/examples_test.go @@ -0,0 +1,251 @@ +// SPDX-License-Identifier: Apache-2.0 + +package upgrade + +import ( + "errors" + "os" + "path/filepath" + "reflect" + "sort" + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/formatter" + "github.com/mendixlabs/mxcli/mdl/visitor" +) + +// examplesDir is the corpus the upgrade is proven on. The execute-both half of +// the proof (same model written) is mdl/roundtrip's upgrade property test; this +// is the half that needs no project: every script upgrades to one that parses, +// records no deprecation, builds the same statements, and is a fixed point of +// `fmt --upgrade`. +const examplesDir = "../../mdl-examples" + +func exampleScripts(t *testing.T) []string { + t.Helper() + var out []string + err := filepath.Walk(examplesDir, func(p string, info os.FileInfo, err error) error { + if err != nil { + return err + } + if !info.IsDir() && strings.HasSuffix(p, ".mdl") { + out = append(out, p) + } + return nil + }) + if err != nil { + t.Fatal(err) + } + sort.Strings(out) + if len(out) < 100 { + t.Fatalf("found only %d scripts under %s — the walk is broken", len(out), examplesDir) + } + return out +} + +// fmtUpgrade is what `mxcli fmt --upgrade` writes: the upgrade rewrites and +// nothing else. It deliberately does not run formatter.Format, whose keyword +// upper-casing changes identifiers (see TestFormatterChangesMeaning). +func fmtUpgrade(src string, opts Options) (string, Result, error) { + res, err := Upgrade(src, opts) + return res.Source, res, err +} + +// TestFormatterChangesMeaning records why `fmt --upgrade` does not compose with +// the heuristic formatter: formatter.Format upper-cases every word on its +// keyword list, identifiers included (`Issue64.User` becomes `Issue64.USER`), +// so it changes what a script builds. Plan item 3.2 (lowercase canonical) +// replaces it. When this test starts failing, the formatter is safe and +// fmt --upgrade may format as well. +func TestFormatterChangesMeaning(t *testing.T) { + changed := 0 + for _, path := range exampleScripts(t) { + b, err := os.ReadFile(path) + if err != nil { + t.Fatal(err) + } + want, errs := visitor.Build(string(b)) + if len(errs) > 0 { + continue + } + got, errs := visitor.Build(formatter.Format(string(b))) + if len(errs) > 0 || !reflect.DeepEqual(want.Statements, got.Statements) { + changed++ + } + } + t.Logf("formatter.Format changes what %d example scripts build", changed) + if changed == 0 { + t.Error("formatter.Format no longer changes any example's statements: let fmt --upgrade format too, and drop this test") + } +} + +// foldModeFlags erases the one AST difference between `or replace` and +// `or modify` that means nothing: pages, snippets and layouts keep both flags, +// and every reader in the executor tests `IsModify || IsReplace` (pinned by +// mdl/visitor's TestCreateOrReplaceMatchesModifyExceptExemptKinds). +func foldModeFlags(stmts []ast.Statement) { + for _, s := range stmts { + switch n := s.(type) { + case *ast.CreatePageStmtV3: + n.IsModify, n.IsReplace = n.IsModify || n.IsReplace, false + case *ast.CreateSnippetStmtV3: + n.IsModify, n.IsReplace = n.IsModify || n.IsReplace, false + case *ast.CreateLayoutStmt: + n.IsModify, n.IsReplace = n.IsModify || n.IsReplace, false + } + } +} + +// keepsItsVersion lists the example scripts `fmt --upgrade --header` refuses, +// with the codes of the constructs that block it: each has no mechanical +// rewrite (HeaderBlockedError says why), so the script keeps mdl 0. The list +// may only shrink — a script that starts upgrading must be removed, and one +// that stops upgrading is a regression. +var keepsItsVersion = map[string][]string{ + // Nested list operations (`count(filter(…))`) and a non-variable operand: + // one activity takes a variable, so the inner call must become a statement + // of its own, which needs a variable name nobody chose. + "bug-tests/1101-nested-list-operand-dropped.fail.mdl": {"MDL-V1-LIST"}, + // `log … 'line 1\nline 2'`: under mdl 1 the line break is written into the + // literal, which makes the message a stored expression (a `{1}` template + // parameter) instead of the template text. Measured by the execute-both + // test before this entry was made. + "bug-tests/264-log-node-expression-roundtrip.mdl": {"MDL-V1-ESCAPE"}, +} + +// buildsTheSameModelNotTheSameAST lists the example scripts whose upgrade +// builds a different AST that the executor writes identically, so the AST +// comparison is skipped for them; the execute-both test in mdl/roundtrip is +// the proof. May only shrink. +var buildsTheSameModelNotTheSameAST = map[string]string{ + // `set $x = contains($Hay, $Needle)` / `find(…)` on a String: mdl 0 builds + // a List operation statement, which the flow builder turns into a Change + // variable with the string function because $Hay is declared String; under + // mdl 1 it is that Change variable directly. + "bug-tests/ledger-53-string-contains.mdl": "string contains: a List operation turned into the string function", + "bug-tests/ledger-63-string-find.mdl": "string find: a List operation turned into the string function", +} + +// foldSourceText erases the one part of an ast.SourceExpr the executor never +// writes: the parsed expression kept beside the source text for checks +// (expressionToString writes Source whenever it is set). A string escape in a +// multi-line expression changes that parsed value and nothing it writes. +func foldSourceText(v reflect.Value) { + switch v.Kind() { + case reflect.Pointer, reflect.Interface: + if v.IsNil() { + return + } + if se, ok := v.Interface().(*ast.SourceExpr); ok && se.Source != "" { + se.Expression = nil + return + } + foldSourceText(v.Elem()) + case reflect.Struct: + for i := 0; i < v.NumField(); i++ { + if v.Type().Field(i).IsExported() { + foldSourceText(v.Field(i)) + } + } + case reflect.Slice, reflect.Array: + for i := 0; i < v.Len(); i++ { + foldSourceText(v.Index(i)) + } + case reflect.Map: + for _, k := range v.MapKeys() { + foldSourceText(v.MapIndex(k)) + } + } +} + +func TestUpgrade_ExamplesKeepTheirStatements(t *testing.T) { + var parsed, upgraded, skipped int + blocked := map[string]bool{} + for _, path := range exampleScripts(t) { + b, err := os.ReadFile(path) + if err != nil { + t.Fatal(err) + } + rel := filepath.ToSlash(strings.TrimPrefix(path, examplesDir+"/")) + src := string(b) + if _, errs := visitor.Build(src); len(errs) > 0 { + skipped++ // a negative test, or a script that no longer parses: not upgradable + continue + } + parsed++ + for _, opts := range []Options{{}, {AddHeader: true}} { + out, res, err := fmtUpgrade(src, opts) + var hb *HeaderBlockedError + if opts.AddHeader && errors.As(err, &hb) { + blocked[rel] = true + allowed := map[string]bool{} + for _, c := range keepsItsVersion[rel] { + allowed[c] = true + } + for _, c := range hb.Constructs { + if !allowed[c.Code] { + t.Errorf("%s: the header is refused over %s at line %d (%s), which is not listed in keepsItsVersion", + path, c.Code, c.Line, c.Reason) + } + } + continue + } + if err != nil { + t.Errorf("%s (%+v): %v", path, opts, err) + continue + } + if res.Changed() && !opts.AddHeader { + upgraded++ + } + got, errs := visitor.Build(out) + if len(errs) > 0 { + t.Errorf("%s (%+v): upgraded output does not parse: %v", path, opts, errs[0]) + continue + } + if len(got.Deprecations) > 0 { + t.Errorf("%s (%+v): upgraded output still records %d deprecation(s), first %s at line %d", + path, opts, len(got.Deprecations), got.Deprecations[0].Code, got.Deprecations[0].Line) + } + want, _ := visitor.Build(src) + foldModeFlags(want.Statements) + foldModeFlags(got.Statements) + foldSourceText(reflect.ValueOf(want.Statements)) + foldSourceText(reflect.ValueOf(got.Statements)) + if len(want.Statements) != len(got.Statements) { + t.Errorf("%s (%+v): %d statements became %d", path, opts, len(want.Statements), len(got.Statements)) + continue + } + differs := false + for i := range want.Statements { + if !reflect.DeepEqual(want.Statements[i], got.Statements[i]) { + differs = true + if _, known := buildsTheSameModelNotTheSameAST[rel]; !known { + t.Errorf("%s (%+v): statement %d builds differently after the upgrade:\n before: %#v\n after: %#v", + path, opts, i+1, want.Statements[i], got.Statements[i]) + } + break + } + } + if _, known := buildsTheSameModelNotTheSameAST[rel]; known && opts.AddHeader && !differs { + t.Errorf("%s now builds the same AST after the upgrade: remove it from buildsTheSameModelNotTheSameAST", rel) + } + // Idempotent: fmt --upgrade over its own output changes nothing. + again, res2, err := fmtUpgrade(out, opts) + if err != nil || again != out || res2.Changed() { + t.Errorf("%s (%+v): fmt --upgrade is not idempotent (err=%v)", path, opts, err) + } + } + } + for rel := range keepsItsVersion { + if !blocked[rel] { + t.Errorf("%s now takes the header: remove it from keepsItsVersion", rel) + } + } + t.Logf("%d scripts parse, %d of them upgraded, %d do not parse and are out of scope, %d keep mdl 0", + parsed, upgraded, skipped, len(blocked)) + if upgraded == 0 { + t.Error("no example needed an upgrade — the corpus no longer exercises the rewrites") + } +} diff --git a/mdl/upgrade/gated.go b/mdl/upgrade/gated.go new file mode 100644 index 0000000000..9d58b9ff05 --- /dev/null +++ b/mdl/upgrade/gated.go @@ -0,0 +1,62 @@ +// SPDX-License-Identifier: Apache-2.0 + +package upgrade + +import ( + "errors" + + "github.com/mendixlabs/mxcli/mdl/ast" +) + +// GatedRewriter rewrites one header-gated construct so that, once the script +// carries the new header, it still means what it meant without one. +// +// note is the ast.LanguageNote the visitor recorded through Builder.gate when +// it parsed the headerless script (its Code is the langver.Change's rule ID); +// prog is that parse. The rewriter returns the source edits. An error says why +// this occurrence has no mechanical rewrite; the upgrade then reports it and +// refuses the header (HeaderBlockedError). +type GatedRewriter func(src *Source, prog *ast.Program, note ast.LanguageNote) ([]Edit, error) + +// gatedRewriters holds one rewrite per langver.Change code: a change of meaning +// or new rejection that applies only under `mdl 1` (ADR-0011 decision 2). +// +// A change lands with its rewrite here (proposal §9, "Each change lands with a +// registry entry, an fmt --upgrade rewrite where one is possible"), or in +// unrewritable with the reason it has none. TestGatedRegistryIsComplete holds +// every change the visitor gates to exactly one of the two. +var gatedRewriters = map[string]GatedRewriter{ + "MDL-V1-SEMI": visitorFix, // insert the missing `;` + "MDL-V1-SLASH": visitorFix, // delete the `/` line + "MDL-V1-ESCAPE": visitorFix, // write the string's mdl 0 value with `''` as the only escape + "MDL-V1-LIMIT1": visitorFix, // `limit 1` (an object) -> `first` + "MDL-V1-SET": visitorFix, // `$x = e` -> `set $x = e` + "MDL-V1-LIST": visitorFix, // call form -> statement form, or `set` for the string function + "MDL-V1-REPLACE02": visitorFix, // `create or replace user role` (a plain create) -> `create user role` +} + +// unrewritable lists the changes with no mechanical rewrite at all, and why. +// A script that uses one keeps its version: `fmt --upgrade --header` reports +// each use and refuses the header. This list may only shrink. +var unrewritable = map[string]string{ + "MDL-V1-PROP": "an unknown property key is ignored under mdl 0 and an error under mdl 1; " + + "which key was meant cannot be guessed, so correct or delete it by hand", + "MDL-V1-PROPVALUE": "a value its key does not take is ignored or read by its shape under mdl 0; " + + "what was meant cannot be guessed, so correct it by hand", + "MDL-V1-REPLACE01": "`create or replace view entity` drops and recreates the view entity under mdl 0, " + + "which no mdl 1 statement does: write `create or modify view entity` to keep its identity, or " + + "`drop entity` then `create view entity` to discard it", +} + +// visitorFix applies the rewrite the visitor computed from the parse tree when +// it recorded the note (mdl/visitor/visitor_upgrade_fixes.go). An occurrence +// without one carries the reason. +func visitorFix(_ *Source, _ *ast.Program, n ast.LanguageNote) ([]Edit, error) { + if n.Fix == nil { + if n.NoFix != "" { + return nil, errors.New(n.NoFix) + } + return nil, errors.New("no mechanical rewrite for this occurrence") + } + return n.Fix.Edits, nil +} diff --git a/mdl/upgrade/gated_test.go b/mdl/upgrade/gated_test.go new file mode 100644 index 0000000000..017862f61e --- /dev/null +++ b/mdl/upgrade/gated_test.go @@ -0,0 +1,205 @@ +// SPDX-License-Identifier: Apache-2.0 + +package upgrade + +import ( + "errors" + "reflect" + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/visitor" +) + +// sameStatements requires two scripts to build the same statements, with the +// folds TestUpgrade_ExamplesKeepTheirStatements applies. +func sameStatements(t *testing.T, before, after string) { + t.Helper() + a, errs := visitor.Build(before) + if len(errs) > 0 { + t.Fatalf("original does not parse: %v", errs[0]) + } + b, errs := visitor.Build(after) + if len(errs) > 0 { + t.Fatalf("upgrade does not parse: %v\n%s", errs[0], after) + } + foldModeFlags(a.Statements) + foldModeFlags(b.Statements) + foldSourceText(reflect.ValueOf(a.Statements)) + foldSourceText(reflect.ValueOf(b.Statements)) + if !reflect.DeepEqual(a.Statements, b.Statements) { + t.Errorf("the upgrade builds different statements:\n before: %#v\n after: %#v", a.Statements, b.Statements) + } +} + +const mf = "create microflow M.F ($L: List of M.E, $S: String, $o: M.E) begin\n" + +// Each header-gated construct, rewritten so that under `mdl 1;` it means what +// it meant under mdl 0. The execute-both test in mdl/roundtrip proves the same +// on the example corpus against a real project. +func TestUpgrade_GatedRewrites(t *testing.T) { + for _, c := range []struct { + name, src, want string + sameAST bool // false where the AST differs but the executor writes the same (see buildsTheSameModelNotTheSameAST) + }{ + {"missing semicolon", "create module M\ncreate module N -- note\n", + "create module M;\ncreate module N; -- note\n", true}, + {"slash line", "create module M;\n/\ncreate module N;\n / \n", + "create module M;\ncreate module N;\n", true}, + {"slash and no semicolon", "create module M\n/\n", "create module M;\n", true}, + {"slash on the statement's line", "create module M; /\n", "create module M; \n", true}, + {"flow ending in a slash", mf + " log info 'x';\nend;\n/\n", mf + " log info 'x';\nend;\n", true}, + {"escapes in a plain literal", "create enumeration M.C (A 'it\\'s\\ta\\\\b');\n", + "create enumeration M.C (A 'it''s\ta\\b');\n", true}, + {"escaped line break outside an expression", "create enumeration M.C (A 'a\\nb');\n", + "create enumeration M.C (A 'a\nb');\n", true}, + {"escape in an expression stored as written", mf + " declare $s String = 'a' +\n '\\n';\nend;\n", + mf + " declare $s String = 'a' +\n '\\n';\nend;\n", true}, + {"escaped tab in an expression", mf + " declare $s String = 'a\\tb';\nend;\n", + mf + " declare $s String = 'a\tb';\nend;\n", true}, + {"limit 1 is an object", mf + " retrieve $X from M.E where Name = 'a' limit 1;\nend;\n", + mf + " retrieve $X from M.E where Name = 'a' first;\nend;\n", true}, + {"LIMIT 1 in upper case", mf + " RETRIEVE $X FROM M.E LIMIT 1;\nend;\n", + mf + " RETRIEVE $X FROM M.E FIRST;\nend;\n", true}, + {"reassignment says set", mf + " declare $n Integer = 1;\n $n = 2;\nend;\n", + mf + " declare $n Integer = 1;\n set $n = 2;\nend;\n", true}, + {"set with a list call", mf + " set $T = head($L);\nend;\n", mf + " $T = head $L;\nend;\n", true}, + {"set with an aggregate", mf + " set $N = sum($L.Amount);\nend;\n", mf + " $N = sum $L by Amount;\nend;\n", true}, + {"find on a list, by member", mf + " $F = find($L, Name = 'x');\nend;\n", mf + " $F = find $L by Name = 'x';\nend;\n", true}, + {"find on a list, by expression", mf + " $F = FIND($L, $currentObject/Name = 'x');\nend;\n", + mf + " $F = FIND $L WHERE $currentObject/Name = 'x';\nend;\n", true}, + {"contains on a list", mf + " $b = contains($L, $o);\nend;\n", mf + " $b = contains $o in $L;\nend;\n", true}, + {"find on a retrieved list", mf + " retrieve $R from M.E;\n $F = find($R, Name = 'x');\nend;\n", + mf + " retrieve $R from M.E;\n $F = find $R by Name = 'x';\nend;\n", true}, + {"find on a String is the string function", mf + " $i = find($S, 'x');\nend;\n", + mf + " set $i = find($S, 'x');\nend;\n", false}, + {"set contains on a String", mf + " declare $b Boolean = false;\n set $b = contains($S, $S);\nend;\n", + mf + " declare $b Boolean = false;\n set $b = contains($S, $S);\nend;\n", false}, + {"create or replace user role is a plain create", "create or replace user role R (M.Admin);\n", + "create user role R (M.Admin);\n", true}, + } { + t.Run(c.name, func(t *testing.T) { + res, err := Upgrade(c.src, Options{AddHeader: true}) + if err != nil { + t.Fatal(err) + } + if want := "mdl 1;\n" + c.want; res.Source != want { + t.Fatalf("got:\n%s\nwant:\n%s", res.Source, want) + } + if len(res.GatedRewritten) == 0 { + t.Errorf("no gated rewrite recorded") + } + if c.sameAST { + sameStatements(t, c.src, res.Source) + } + again, err := Upgrade(res.Source, Options{AddHeader: true}) + if err != nil || again.Source != res.Source || again.Changed() { + t.Errorf("not idempotent: %v %q", err, again.Source) + } + }) + } +} + +// A construct with no mechanical rewrite is reported with its reason and +// blocks the header; nothing is written. +func TestUpgrade_UnrewritableBlocksTheHeader(t *testing.T) { + for _, c := range []struct{ name, src, code, reason string }{ + {"nested list operation", mf + " $n = count(filter($L, Name = 'x'));\nend;\n", "MDL-V1-LIST", "nested call"}, + {"find on a call's result", mf + " $R = call microflow M.G();\n $F = find($R, Name = 'x');\nend;\n", + "MDL-V1-LIST", "does not state"}, + {"escaped line break in an expression", mf + " declare $s String = 'a\\nb';\nend;\n", + "MDL-V1-ESCAPE", "line break"}, + {"view entity replace", "create or replace view entity M.V (Name: String(100)) as (select c.Name as Name from M.Customer as c);\n", + "MDL-V1-REPLACE01", "drops and recreates"}, + {"unknown property", "create rest client M.Api (BaseUrl: 'https://x', Authentication: none) " + + "{ operation GetUser { Method: get, pathh: '/users', Response: none } };\n", "MDL-V1-PROP", "cannot be guessed"}, + } { + t.Run(c.name, func(t *testing.T) { + res, err := Upgrade(c.src, Options{AddHeader: true}) + var hb *HeaderBlockedError + if !errors.As(err, &hb) { + t.Fatalf("header added over a construct with no rewrite: %v\n%s", err, res.Source) + } + found := false + for _, b := range hb.Constructs { + if b.Code == c.code && strings.Contains(b.Reason, c.reason) { + found = true + } + } + if !found { + t.Errorf("want %s (%q) reported, got %+v", c.code, c.reason, hb.Constructs) + } + // Without the header the script keeps its meaning, so it upgrades. + if _, err := Upgrade(c.src, Options{}); err != nil { + t.Errorf("without the header: %v", err) + } + }) + } +} + +// The call form of every list operation and aggregate becomes its statement +// form without the header (MDL-DEPR003/004 are aliases under every version), +// building the same activity. +func TestUpgrade_ListCallFormToStatementForm(t *testing.T) { + for _, c := range [][2]string{ + {"$X = head($L);", "$X = head $L;"}, + {"$X = TAIL( $L );", "$X = TAIL $L;"}, + {"$X = filter($L, Name = 'x');", "$X = filter $L by Name = 'x';"}, + {"$X = filter($L, Qty > 0 and Name != 'a');", "$X = filter $L where Qty > 0 and Name != 'a';"}, + {"$X = sort($L, \"Status\" asc, Name desc);", "$X = sort $L by \"Status\" asc, Name desc;"}, + {"$X = union($L, $L2);", "$X = union $L with $L2;"}, + {"$X = intersect($L, $L2);", "$X = intersect $L with $L2;"}, + {"$X = subtract($L, $L2);", "$X = subtract $L2 from $L;"}, + {"$X = equals($L, $L2);", "$X = equals $L and $L2;"}, + {"$X = range($L, 5);", "$X = range $L offset 5;"}, + {"$X = range($L, $o2, 10);", "$X = range $L offset $o2 limit 10;"}, + {"$X = count($L);", "$X = count $L;"}, + {"$X = sum($L.Amount);", "$X = sum $L by Amount;"}, + {"$X = average($L/Amount);", "$X = average $L by Amount;"}, + {"$X = minimum($L, $currentObject/Amount * 2);", "$X = minimum $L of $currentObject/Amount * 2;"}, + {"$X = MAXIMUM($L.Amount);", "$X = MAXIMUM $L BY Amount;"}, + {"$X = all($L, $currentObject/Paid);", "$X = all $L where $currentObject/Paid;"}, + {"$X = any($L, not($currentObject/Paid));", "$X = any $L where not($currentObject/Paid);"}, + {"$X = reduce($L, $currentResult + $currentObject/Amount, initial: 0, returns: Decimal);", + "$X = reduce $L from 0 as Decimal using $currentResult + $currentObject/Amount;"}, + } { + src := "create microflow M.F ($L: List of M.E, $L2: List of M.E, $o2: Integer) begin\n " + c[0] + "\nend;\n" + want := "create microflow M.F ($L: List of M.E, $L2: List of M.E, $o2: Integer) begin\n " + c[1] + "\nend;\n" + t.Run(c[0], func(t *testing.T) { + res, err := Upgrade(src, Options{}) + if err != nil { + t.Fatal(err) + } + if res.Source != want { + t.Fatalf("got:\n%s\nwant:\n%s", res.Source, want) + } + sameStatements(t, src, res.Source) + }) + } +} + +// Every change of meaning the visitor gates has a rewrite or is listed as +// having none, never both. The list of changes with none may only shrink. +func TestGatedRegistryIsComplete(t *testing.T) { + pinned := map[string]bool{"MDL-V1-PROP": true, "MDL-V1-PROPVALUE": true, "MDL-V1-REPLACE01": true} + known := map[string]bool{} + for _, c := range visitor.LanguageChanges() { + known[c.Code] = true + _, rw := gatedRewriters[c.Code] + _, none := unrewritable[c.Code] + if rw == none { + t.Errorf("%s: needs exactly one of a rewrite in gatedRewriters or a reason in unrewritable (rewrite=%v, reason=%v)", + c.Code, rw, none) + } + } + for code := range gatedRewriters { + if !known[code] { + t.Errorf("gatedRewriters has %s, which the visitor does not gate", code) + } + } + for code := range unrewritable { + if !pinned[code] { + t.Errorf("%s was added to unrewritable: the list may only shrink — give the change a rewrite", code) + } + } +} diff --git a/mdl/upgrade/upgrade.go b/mdl/upgrade/upgrade.go new file mode 100644 index 0000000000..9efa14016d --- /dev/null +++ b/mdl/upgrade/upgrade.go @@ -0,0 +1,350 @@ +// SPDX-License-Identifier: Apache-2.0 + +// Package upgrade rewrites an MDL script to the canonical language: the engine +// behind `mxcli fmt --upgrade` (ADR-0011 decision 1, plan item 1.3 of +// PROPOSAL_mdl_beta_syntax_freeze.md). +// +// It does two kinds of rewrite, and nothing else: +// +// - Every use of a deprecated spelling registered in mdl/deprecation becomes +// its canonical spelling. A deprecated spelling is a respelling, so the +// rewrite never changes meaning. The visitor records where each use is +// (ast.Program.Deprecations); the entry's Rewrite says what replaces it. +// - When the caller asks for the language header, every construct whose +// meaning differs under the new version (ast.Program.LanguageNotes, one per +// langver.Change) is rewritten to the spelling that keeps its OLD meaning +// under the new header, and `mdl <n>;` is added. The rewrites live in +// gatedRewriters (gated.go); the changes with none, in unrewritable. +// +// A rewrite that is more than a keyword swap — a structural deprecation such as +// a list operation's call form, or a header-gated construct — is computed from +// the parse tree by the visitor that records the construct, as an ast.Fix of +// rune-offset edits (mdl/visitor/visitor_upgrade_fixes.go). An occurrence the +// visitor cannot rewrite carries the reason instead. +// +// Anything it cannot rewrite is reported, never guessed at: a deprecated use +// with no rewrite is left in place and listed in Result.Unrewritten, and a +// header-gated construct without one blocks the header (HeaderBlockedError), +// because adding the header over it would silently change the script's meaning. +// +// The rewrite works on the source text, not on the AST, so comments, layout +// and every spelling it does not own survive byte for byte. The output is +// re-parsed before it is returned: it must parse, record no deprecation the +// upgrade knows how to rewrite, and keep every statement. +// +// The proof that a rewrite is safe is the execute-both property test in +// mdl/roundtrip (upgrade_property_test.go): each mdl-examples script and its +// upgrade run on two copies of the PedApp fixture and must write the same +// model, compared as canonical BSON per unit. +package upgrade + +import ( + "fmt" + "sort" + "strings" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/deprecation" + "github.com/mendixlabs/mxcli/mdl/langver" + "github.com/mendixlabs/mxcli/mdl/visitor" +) + +// Options says what Upgrade may do beyond the alias rewrites. +type Options struct { + // AddHeader adds the `mdl <Latest>;` header to a headerless script, after + // rewriting every construct whose meaning the header would change. While + // mdl 1 is a preview this is opt-in (`fmt --upgrade --header`); at beta it + // becomes the default (langver.Frozen is the single switch). + AddHeader bool +} + +// DefaultOptions is what `fmt --upgrade` does when no header flag is given: +// add the header only once the version it names is frozen. +func DefaultOptions() Options { return Options{AddHeader: langver.Latest <= langver.Frozen} } + +// Result is an upgraded script and what the upgrade could not do. +type Result struct { + // Source is the upgraded script. It equals the input when nothing applied. + Source string + // Rewritten counts the deprecated uses rewritten, by registry code. + Rewritten map[string]int + // GatedRewritten counts the header-gated constructs rewritten, by code. + GatedRewritten map[string]int + // HeaderAdded reports whether the language header was added. + HeaderAdded bool + // Unrewritten lists the deprecated uses left in place because their + // registry entry carries no rewrite. + Unrewritten []ast.DeprecatedSpelling +} + +// Changed reports whether the upgrade rewrote anything. +func (r Result) Changed() bool { + return r.HeaderAdded || len(r.Rewritten) > 0 || len(r.GatedRewritten) > 0 +} + +// Upgrade rewrites src to the canonical language. It returns an error when src +// does not parse, when the header was asked for but a construct under it has no +// rewrite, or when the rewritten script fails its re-parse check. +func Upgrade(src string, opts Options) (Result, error) { + return upgrader{lookup: deprecation.Lookup, gated: gatedRewriters}.upgrade(src, opts) +} + +// upgrader carries the two registries so a test can substitute them. +type upgrader struct { + lookup func(code string) (deprecation.Entry, bool) + gated map[string]GatedRewriter +} + +func (u upgrader) upgrade(src string, opts Options) (Result, error) { + return u.upgradeProg(src, opts, parse) +} + +func parse(src string) (*ast.Program, error) { + prog, errs := visitor.Build(src) + if len(errs) > 0 { + return nil, errs[0] + } + return prog, nil +} + +// upgradeProg is upgrade with the parser as a parameter, so a test can hand it +// a Program carrying constructs the visitor does not produce yet. +func (u upgrader) upgradeProg(src string, opts Options, parse func(string) (*ast.Program, error)) (Result, error) { + res := Result{Source: src, Rewritten: map[string]int{}, GatedRewritten: map[string]int{}} + prog, err := parse(src) + if err != nil { + return res, fmt.Errorf("the script does not parse, so it cannot be upgraded: %w", err) + } + text := newSource(src) + + var edits []Edit + for _, d := range prog.Deprecations { + e, ok := u.lookup(d.Code) + if !ok { + return res, fmt.Errorf("line %d: the visitor recorded %s, which is not in the deprecation registry", d.Line, d.Code) + } + switch { + case e.Rewrite.Token != "": + ed, err := tokenSwap(text, d, e.Rewrite) + if err != nil { + return res, err + } + edits = append(edits, ed) + case e.Rewrite.Structural != "" && d.Fix != nil: + // A structural rewrite is computed from the parse tree by the + // visitor that recorded the use (mdl/visitor/visitor_upgrade_fixes.go). + edits = append(edits, d.Fix.Edits...) + default: + res.Unrewritten = append(res.Unrewritten, d) + continue + } + res.Rewritten[d.Code]++ + } + + addHeader := opts.AddHeader && prog.LanguageHeaderLine == 0 + if addHeader { + var blocked []Blocked + for _, n := range prog.LanguageNotes { + rw, ok := u.gated[n.Code] + if !ok { + reason := unrewritable[n.Code] + if reason == "" { + reason = "no rewrite is registered for it" + } + blocked = append(blocked, Blocked{Line: n.Line, Code: n.Code, Reason: reason}) + continue + } + ed, err := rw(text, prog, n) + if err != nil { + blocked = append(blocked, Blocked{Line: n.Line, Code: n.Code, Reason: err.Error()}) + continue + } + edits = append(edits, ed...) + res.GatedRewritten[n.Code]++ + } + if len(blocked) > 0 { + return res, &HeaderBlockedError{From: prog.LanguageVersion, Constructs: blocked} + } + } + + out, err := text.apply(edits) + if err != nil { + return res, err + } + if addHeader { + out = langver.Latest.String() + ";\n" + out + res.HeaderAdded = true + } + if err := u.verify(prog, out, parse); err != nil { + return res, err + } + res.Source = out + return res, nil +} + +// Blocked is one construct that keeps a script from taking the header: its +// meaning would change under it, and it has no mechanical rewrite. +type Blocked struct { + Line int + Code string // the langver.Change's rule ID + Reason string // why there is no rewrite, and what to write instead +} + +// HeaderBlockedError is Upgrade's refusal to add the language header over +// constructs whose meaning it would change and that it cannot rewrite. The +// script is left at its version; nothing is written. +type HeaderBlockedError struct { + From langver.Version + Constructs []Blocked +} + +func (e *HeaderBlockedError) Error() string { + lines := make([]string, len(e.Constructs)) + for i, c := range e.Constructs { + lines[i] = fmt.Sprintf("line %d: %s: %s", c.Line, c.Code, c.Reason) + } + return fmt.Sprintf("cannot add the `%s;` header: %d construct(s) would change meaning under it "+ + "and have no mechanical rewrite, so the script is left at %s:\n %s", + langver.Latest, len(e.Constructs), e.From, strings.Join(lines, "\n ")) +} + +// verify re-parses the upgraded script. It must parse, keep every statement, +// and record no deprecated use the upgrade knows how to rewrite: a use that +// survives means a rewrite did not produce the canonical spelling, and a +// second run would rewrite again, so fmt would not be idempotent. +// +// A header-gated construct that survives is caught the same way: under the +// header the parse records no LanguageNote, but the construct must now be +// spelled so that it means what it meant before, which only the execute-both +// test can judge. +func (u upgrader) verify(orig *ast.Program, out string, parse func(string) (*ast.Program, error)) error { + prog, err := parse(out) + if err != nil { + return fmt.Errorf("the upgraded script does not parse (an upgrade rewrite is wrong): %w", err) + } + if len(prog.Statements) != len(orig.Statements) { + return fmt.Errorf("the upgraded script has %d statements, the original %d (an upgrade rewrite is wrong)", + len(prog.Statements), len(orig.Statements)) + } + for _, d := range prog.Deprecations { + if e, ok := u.lookup(d.Code); ok && (e.Rewrite.Token != "" || d.Fix != nil) { + return fmt.Errorf("line %d: the upgraded script still uses a deprecated spelling (%s); its rewrite did not produce %q", + d.Line, d.Code, e.Canonical) + } + } + return nil +} + +// Edit replaces the runes [Start, Stop) of the source with Text; Start == +// Stop inserts. Offsets count runes from the start of the script, the +// coordinates ANTLR's character stream reports. +type Edit = ast.TextEdit + +// Source is a script as runes, the coordinates the parser reports positions +// in, with the offset each line starts at. +type Source struct { + runes []rune + lineStarts []int +} + +func newSource(src string) *Source { + s := &Source{runes: []rune(src), lineStarts: []int{0}} + for i, r := range s.runes { + if r == '\n' { + s.lineStarts = append(s.lineStarts, i+1) + } + } + return s +} + +// Offset returns the rune offset of (line, col): line 1-based, col 0-based in +// runes, as ANTLR reports token positions. False when out of range. +func (s *Source) Offset(line, col int) (int, bool) { + if line < 1 || line > len(s.lineStarts) || col < 0 { + return 0, false + } + end := len(s.runes) + if line < len(s.lineStarts) { + end = s.lineStarts[line] - 1 + } + off := s.lineStarts[line-1] + col + if off > end { + return 0, false + } + return off, true +} + +// At returns the n runes at (line, col), or false when they are out of range +// or cross a line end. +func (s *Source) At(line, col, n int) (string, bool) { + off, ok := s.Offset(line, col) + if !ok || n < 0 { + return "", false + } + if end, ok := s.Offset(line, col+n); !ok || end != off+n { + return "", false + } + return string(s.runes[off : off+n]), true +} + +// apply returns the source with edits applied. Edits may not overlap; two at +// the same offset apply in the order given, an insertion before a replacement. +func (s *Source) apply(edits []Edit) (string, error) { + sorted := append([]Edit(nil), edits...) + sort.SliceStable(sorted, func(i, j int) bool { + if sorted[i].Start != sorted[j].Start { + return sorted[i].Start < sorted[j].Start + } + return sorted[i].Stop-sorted[i].Start < sorted[j].Stop-sorted[j].Start + }) + var b strings.Builder + at := 0 + for _, e := range sorted { + if e.Start < 0 || e.Stop < e.Start || e.Stop > len(s.runes) { + return "", fmt.Errorf("%s: upgrade rewrite is outside the source", s.where(e.Start)) + } + if e.Start < at { + return "", fmt.Errorf("%s: two upgrade rewrites overlap", s.where(e.Start)) + } + b.WriteString(string(s.runes[at:e.Start])) + b.WriteString(e.Text) + at = e.Stop + } + b.WriteString(string(s.runes[at:])) + return b.String(), nil +} + +// where names a rune offset as "line L column C" (column 1-based). +func (s *Source) where(off int) string { + line := sort.Search(len(s.lineStarts), func(i int) bool { return s.lineStarts[i] > off }) + if line == 0 { + return fmt.Sprintf("offset %d", off) + } + return fmt.Sprintf("line %d column %d", line, off-s.lineStarts[line-1]+1) +} + +// tokenSwap is the edit for a deprecation.Rewrite: the recorded token, which +// must be the rewrite's Token, becomes its Replacement in the same letter case. +func tokenSwap(s *Source, d ast.DeprecatedSpelling, rw deprecation.Rewrite) (Edit, error) { + n := len([]rune(rw.Token)) + got, ok := s.At(d.Line, d.Column, n) + if !ok || !strings.EqualFold(got, rw.Token) { + return Edit{}, fmt.Errorf("line %d column %d: %s is recorded here, but the source has %q where %q was expected", + d.Line, d.Column, d.Code, got, rw.Token) + } + off, _ := s.Offset(d.Line, d.Column) + return Edit{Start: off, Stop: off + n, Text: matchCase(got, rw.Replacement)}, nil +} + +// matchCase spells word in the letter case of like: all upper, capitalised, or +// lower. A script written in upper-case keywords stays upper-case. +func matchCase(like, word string) string { + switch { + case like == strings.ToUpper(like) && like != strings.ToLower(like): + return strings.ToUpper(word) + case len(like) > 1 && like[:1] == strings.ToUpper(like[:1]) && like[1:] == strings.ToLower(like[1:]): + return strings.ToUpper(word[:1]) + strings.ToLower(word[1:]) + default: + return strings.ToLower(word) + } +} diff --git a/mdl/upgrade/upgrade_test.go b/mdl/upgrade/upgrade_test.go new file mode 100644 index 0000000000..2a99fd148c --- /dev/null +++ b/mdl/upgrade/upgrade_test.go @@ -0,0 +1,228 @@ +// SPDX-License-Identifier: Apache-2.0 + +package upgrade + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/deprecation" + "github.com/mendixlabs/mxcli/mdl/langver" + "github.com/mendixlabs/mxcli/mdl/visitor" +) + +func mustUpgrade(t *testing.T, src string, opts Options) Result { + t.Helper() + res, err := Upgrade(src, opts) + if err != nil { + t.Fatalf("Upgrade(%q): %v", src, err) + } + return res +} + +// Every registry entry's own example upgrades to exactly its canonical example. +// A new entry is covered the moment it is registered. +func TestUpgrade_EveryRegistryExample(t *testing.T) { + for _, e := range deprecation.All() { + t.Run(e.Code, func(t *testing.T) { + res := mustUpgrade(t, e.Example, Options{}) + if res.Source != e.CanonicalExample { + t.Errorf("upgrade of %q\n got: %q\nwant: %q", e.Example, res.Source, e.CanonicalExample) + } + if res.Rewritten[e.Code] != 1 { + t.Errorf("Rewritten = %v, want one %s", res.Rewritten, e.Code) + } + }) + } +} + +func TestUpgrade_RewritesInPlaceKeepingCaseCommentsAndLayout(t *testing.T) { + src := "-- create or replace in a comment stays\n" + + "CREATE OR REPLACE ENUMERATION M.Color (Red 'show me'); -- trailing\n" + + " Show entities in M; show associations;\n" + + "create or modify enumeration M.Size (S 's');\n" + + "show entity M.Customer;\n" + want := "-- create or replace in a comment stays\n" + + "CREATE OR MODIFY ENUMERATION M.Color (Red 'show me'); -- trailing\n" + + " List entities in M; list associations;\n" + + "create or modify enumeration M.Size (S 's');\n" + + // `show entity X` becomes `describe`, which is not registered yet: untouched. + "show entity M.Customer;\n" + res := mustUpgrade(t, src, Options{}) + if res.Source != want { + t.Fatalf("got:\n%s\nwant:\n%s", res.Source, want) + } + if res.Rewritten[deprecation.CreateOrReplace] != 1 || res.Rewritten[deprecation.Show] != 2 { + t.Errorf("Rewritten = %v", res.Rewritten) + } +} + +// Non-ASCII text before the token on the same line: ANTLR columns count runes, +// and a byte-indexed rewrite would land in the wrong place. +func TestUpgrade_ColumnsAreRunes(t *testing.T) { + src := "/* café ☕ */ create or replace enumeration M.Color (Red 'Rood ☕');" + res := mustUpgrade(t, src, Options{}) + want := "/* café ☕ */ create or modify enumeration M.Color (Red 'Rood ☕');" + if res.Source != want { + t.Fatalf("got %q\nwant %q", res.Source, want) + } +} + +// Where `or replace` does not mean `or modify` it is not an alias, is not +// recorded, and must not be rewritten. +func TestUpgrade_LeavesNonAliasesAlone(t *testing.T) { + for _, src := range []string{ + "create or replace view entity M.V (Name: String(100)) as (select c.Name as Name from M.Customer as c);", + "create or replace translations in Administration for nl_NL ('Save' as 'Opslaan');", + "create or replace user role Clerk (M.User);", + "show version;", + } { + res := mustUpgrade(t, src, Options{}) + if res.Source != src || res.Changed() { + t.Errorf("%q was rewritten to %q", src, res.Source) + } + } +} + +func TestUpgrade_IsIdempotent(t *testing.T) { + src := "create or replace enumeration M.Color (Red 'Red');\nshow entities;\n" + for _, opts := range []Options{{}, {AddHeader: true}} { + once := mustUpgrade(t, src, opts) + twice := mustUpgrade(t, once.Source, opts) + if twice.Source != once.Source || twice.Changed() { + t.Errorf("opts %+v: second upgrade changed %q to %q", opts, once.Source, twice.Source) + } + } +} + +func TestUpgrade_HeaderOnlyWhenAsked(t *testing.T) { + src := "show entities;\n" + if res := mustUpgrade(t, src, Options{}); res.HeaderAdded || strings.HasPrefix(res.Source, "mdl") { + t.Fatalf("header added without being asked: %q", res.Source) + } + res := mustUpgrade(t, src, Options{AddHeader: true}) + if !res.HeaderAdded || res.Source != "mdl 1;\nlist entities;\n" { + t.Fatalf("got %q (HeaderAdded=%v)", res.Source, res.HeaderAdded) + } + prog, errs := visitor.Build(res.Source) + if len(errs) > 0 || prog.LanguageVersion != langver.Latest { + t.Fatalf("upgraded script is not %s: %v", langver.Latest, errs) + } + // An existing header is kept, not doubled. + again := mustUpgrade(t, res.Source, Options{AddHeader: true}) + if again.HeaderAdded || again.Source != res.Source { + t.Fatalf("header added twice: %q", again.Source) + } +} + +// While mdl 1 is a preview the header is opt-in; DefaultOptions turns it on by +// itself once langver.Frozen reaches Latest, with no edit here. +func TestDefaultOptions_FollowsFrozen(t *testing.T) { + if got, want := DefaultOptions().AddHeader, langver.Latest <= langver.Frozen; got != want { + t.Fatalf("DefaultOptions().AddHeader = %v, want %v", got, want) + } +} + +func TestUpgrade_RefusesUnparseableInput(t *testing.T) { + if _, err := Upgrade("create or replace enumeration (", Options{}); err == nil { + t.Fatal("upgraded a script that does not parse") + } +} + +// An entry without a rewrite is reported, not guessed at, and the use is left. +func TestUpgrade_EntryWithoutRewriteIsReported(t *testing.T) { + u := upgrader{ + lookup: func(code string) (deprecation.Entry, bool) { + e, ok := deprecation.Lookup(code) + if code == deprecation.Show { + e.Rewrite = deprecation.Rewrite{} + } + return e, ok + }, + gated: map[string]GatedRewriter{}, + } + src := "show entities;\ncreate or replace enumeration M.Color (Red 'Red');" + res, err := u.upgrade(src, Options{}) + if err != nil { + t.Fatal(err) + } + if !strings.HasPrefix(res.Source, "show entities;\ncreate or modify") { + t.Fatalf("got %q", res.Source) + } + if len(res.Unrewritten) != 1 || res.Unrewritten[0].Code != deprecation.Show || res.Unrewritten[0].Line != 1 { + t.Fatalf("Unrewritten = %+v, want the show on line 1", res.Unrewritten) + } +} + +// A rewrite that does not produce the canonical spelling is caught by the +// re-parse, before anything is returned. +func TestUpgrade_WrongRewriteIsCaught(t *testing.T) { + u := upgrader{ + lookup: func(code string) (deprecation.Entry, bool) { + e, ok := deprecation.Lookup(code) + e.Rewrite.Replacement = e.Rewrite.Token // rewrites to itself + return e, ok + }, + gated: map[string]GatedRewriter{}, + } + if _, err := u.upgrade("show entities;", Options{}); err == nil || !strings.Contains(err.Error(), "still uses") { + t.Fatalf("a rewrite that changes nothing was accepted: %v", err) + } + u.lookup = func(code string) (deprecation.Entry, bool) { + e, ok := deprecation.Lookup(code) + e.Rewrite.Replacement = "frobnicate" // not a statement: ANTLR drops it silently + return e, ok + } + if _, err := u.upgrade("show entities;", Options{}); err == nil || !strings.Contains(err.Error(), "rewrite is wrong") { + t.Fatalf("a rewrite to unparseable text was accepted: %v", err) + } +} + +// The header-gated hook. A construct whose meaning the header would change +// must be rewritten before the header goes on; without a rewrite the header is +// refused. The visitor has no gated construct yet, so the test drives the hook +// through a Program the way Builder.gate would fill it. +func TestUpgrade_GatedConstructs(t *testing.T) { + const code = "MDL-V1-TEST" + src := "show entities;\nshow modules;\n" + withNote := func(src string) (*ast.Program, error) { + prog, errs := visitor.Build(src) + if len(errs) > 0 { + return nil, errs[0] + } + if prog.LanguageHeaderLine == 0 { + prog.LanguageNotes = append(prog.LanguageNotes, ast.LanguageNote{Line: 2, Code: code}) + } + return prog, nil + } + + // No rewriter: the header is refused and nothing is returned. + u := upgrader{lookup: deprecation.Lookup, gated: map[string]GatedRewriter{}} + res, err := u.upgradeProg(src, Options{AddHeader: true}, withNote) + if err == nil || !strings.Contains(err.Error(), code) { + t.Fatalf("header added over a gated construct with no rewrite: %v %q", err, res.Source) + } + // Without the header nothing changes meaning, so the aliases still upgrade. + res, err = u.upgradeProg(src, Options{}, withNote) + if err != nil || res.Source != "list entities;\nlist modules;\n" { + t.Fatalf("got %q, %v", res.Source, err) + } + + // With a rewriter: its edits are applied along with the alias rewrites. + u.gated = map[string]GatedRewriter{code: func(s *Source, _ *ast.Program, n ast.LanguageNote) ([]Edit, error) { + got, _ := s.At(n.Line, 5, 7) + if got != "modules" { + t.Errorf("note does not point at the construct: %q", got) + } + off, _ := s.Offset(n.Line, 5) + return []Edit{{Start: off, Stop: off + 7, Text: "microflows"}}, nil + }} + res, err = u.upgradeProg(src, Options{AddHeader: true}, withNote) + if err != nil { + t.Fatal(err) + } + if want := "mdl 1;\nlist entities;\nlist microflows;\n"; res.Source != want || res.GatedRewritten[code] != 1 { + t.Fatalf("got %q (%v), want %q", res.Source, res.GatedRewritten, want) + } +} diff --git a/mdl/visitor/language_changes_test.go b/mdl/visitor/language_changes_test.go new file mode 100644 index 0000000000..aff2f13b94 --- /dev/null +++ b/mdl/visitor/language_changes_test.go @@ -0,0 +1,64 @@ +// SPDX-License-Identifier: Apache-2.0 + +package visitor + +import ( + "go/ast" + "go/parser" + "go/token" + "os" + "strings" + "testing" +) + +// LanguageChanges must list every langver.Change this package declares: the +// upgrade's registry test holds each listed change to a rewrite or a reason it +// has none, so a change left off the list would escape it. +func TestLanguageChangesListsEveryDeclaredChange(t *testing.T) { + listed := map[string]bool{} + for _, c := range LanguageChanges() { + listed[c.Code] = true + } + entries, err := os.ReadDir(".") + if err != nil { + t.Fatal(err) + } + declared := 0 + fset := token.NewFileSet() + for _, e := range entries { + if !strings.HasSuffix(e.Name(), ".go") || strings.HasSuffix(e.Name(), "_test.go") { + continue + } + f, err := parser.ParseFile(fset, e.Name(), nil, 0) + if err != nil { + t.Fatal(err) + } + ast.Inspect(f, func(n ast.Node) bool { + lit, ok := n.(*ast.CompositeLit) + if !ok { + return true + } + sel, ok := lit.Type.(*ast.SelectorExpr) + if !ok || sel.Sel.Name != "Change" || len(lit.Elts) == 0 { + return true + } + for _, el := range lit.Elts { + kv, ok := el.(*ast.KeyValueExpr) + if k, _ := kv.Key.(*ast.Ident); !ok || k == nil || k.Name != "Code" { + continue + } + if v, ok := kv.Value.(*ast.BasicLit); ok { + declared++ + code := strings.Trim(v.Value, `"`) + if !listed[code] { + t.Errorf("%s declares the language change %s, which LanguageChanges does not list", e.Name(), code) + } + } + } + return true + }) + } + if declared != len(listed) { + t.Errorf("%d changes declared, %d listed", declared, len(listed)) + } +} diff --git a/mdl/visitor/module_role_create_statement_test.go b/mdl/visitor/module_role_create_statement_test.go new file mode 100644 index 0000000000..abdc78eabc --- /dev/null +++ b/mdl/visitor/module_role_create_statement_test.go @@ -0,0 +1,79 @@ +// SPDX-License-Identifier: Apache-2.0 + +package visitor + +import ( + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" +) + +// `create module role` was its own securityStatement rule with only an +// `or modify` prefix, so it took none of the prefixes every other document type +// gets from createStatement (#731): `or replace` was a parse error, and a doc +// comment in front of it was not attached. It is a createStatement kind now. +func TestCreateModuleRole_IsACreateStatementKind(t *testing.T) { + for _, tc := range []struct { + src string + orModify bool + desc string + deprecated bool + }{ + {"create module role M.Admin;", false, "", false}, + {"create or modify module role M.Admin description 'Full access';", true, "Full access", false}, + {"create or replace module role M.Admin;", true, "", true}, + {"/** doc */ create or modify module role M.Admin;", true, "", false}, + } { + prog, errs := Build(tc.src) + if len(errs) > 0 { + t.Fatalf("%s: parse: %v", tc.src, errs[0]) + } + if len(prog.Statements) != 1 { + t.Fatalf("%s: got %d statements", tc.src, len(prog.Statements)) + } + s, ok := prog.Statements[0].(*ast.CreateModuleRoleStmt) + if !ok { + t.Fatalf("%s: got %T", tc.src, prog.Statements[0]) + } + if s.Name.String() != "M.Admin" || s.CreateOrModify != tc.orModify || s.Description != tc.desc { + t.Errorf("%s: got %+v", tc.src, s) + } + if got := len(prog.Deprecations) > 0; got != tc.deprecated { + t.Errorf("%s: deprecations %v", tc.src, prog.Deprecations) + } + } +} + +// `create module Role;` names a module called Role; moving module role into +// createStatement must not take that spelling away. +func TestCreateModuleNamedRole_StillAModule(t *testing.T) { + prog, errs := Build("create module Role;") + if len(errs) > 0 { + t.Fatalf("parse: %v", errs[0]) + } + if s, ok := prog.Statements[0].(*ast.CreateModuleStmt); !ok || s.Name != "Role" { + t.Fatalf("got %#v", prog.Statements[0]) + } +} + +func TestDropModuleRole_IfExistsIsParsed(t *testing.T) { + for _, tc := range []struct { + src string + want bool + }{ + {"drop module role if exists M.Admin;", true}, + {"drop module role M.Admin;", false}, + } { + prog, errs := Build(tc.src) + if len(errs) > 0 { + t.Fatalf("%s: parse: %v", tc.src, errs[0]) + } + s, ok := prog.Statements[0].(*ast.DropModuleRoleStmt) + if !ok { + t.Fatalf("%s: got %T", tc.src, prog.Statements[0]) + } + if s.IfExists != tc.want || s.Name.String() != "M.Admin" { + t.Errorf("%s: got %+v", tc.src, s) + } + } +} diff --git a/mdl/visitor/retrieve_first_test.go b/mdl/visitor/retrieve_first_test.go new file mode 100644 index 0000000000..9973065782 --- /dev/null +++ b/mdl/visitor/retrieve_first_test.go @@ -0,0 +1,119 @@ +// SPDX-License-Identifier: Apache-2.0 + +package visitor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" +) + +// `retrieve … first` binds one object and `retrieve … limit 1` a list of one +// under mdl 1; without the header `limit 1` keeps its alpha meaning (an object) +// and warns MDL-V1-LIMIT1 (ako/mxcli#734, ADR-0011). + +func retrieveOf(t *testing.T, src string) (*ast.RetrieveStmt, *ast.Program) { + t.Helper() + prog, errs := Build(src) + if len(errs) > 0 { + t.Fatalf("%q: unexpected errors: %v", src, errs) + } + var mf *ast.CreateMicroflowStmt + for _, s := range prog.Statements { + if m, ok := s.(*ast.CreateMicroflowStmt); ok { + mf = m + } + } + if mf == nil || len(mf.Body) == 0 { + t.Fatalf("%q: no microflow body", src) + } + r, ok := mf.Body[0].(*ast.RetrieveStmt) + if !ok { + t.Fatalf("%q: body[0] = %T, want *ast.RetrieveStmt", src, mf.Body[0]) + } + return r, prog +} + +func retrieveMicroflow(rangeClause string) string { + return "create microflow M.F () begin\n retrieve $O from M.Order where [Status = 'Open'] " + + rangeClause + ";\nend;" +} + +func limitOneNotes(prog *ast.Program) int { + n := 0 + for _, note := range prog.LanguageNotes { + if note.Code == "MDL-V1-LIMIT1" { + n++ + } + } + return n +} + +func TestRetrieveFirst_IsAnObjectUnderBothVersions(t *testing.T) { + for _, header := range []string{"", "mdl 1;\n"} { + r, prog := retrieveOf(t, header+retrieveMicroflow("first")) + if !r.First || r.Limit != "" || r.Offset != "" { + t.Errorf("header %q: first → First=%v Limit=%q Offset=%q; want the object range and no limit", + header, r.First, r.Limit, r.Offset) + } + if n := limitOneNotes(prog); n != 0 { + t.Errorf("header %q: `first` means the same in every version, yet %d MDL-V1-LIMIT1 note(s)", header, n) + } + } +} + +func TestRetrieveLimitOne_Mdl0IsAnObjectAndWarns(t *testing.T) { + r, prog := retrieveOf(t, retrieveMicroflow("limit 1")) + if !r.First || r.Limit != "" { + t.Errorf("mdl 0 `limit 1` → First=%v Limit=%q; want the alpha meaning, the object range", r.First, r.Limit) + } + if n := limitOneNotes(prog); n != 1 { + t.Fatalf("mdl 0 `limit 1`: got %d MDL-V1-LIMIT1 notes, want 1: %+v", n, prog.LanguageNotes) + } + if msg := prog.LanguageNotes[0].Message; !strings.Contains(msg, "first") { + t.Errorf("the warning should name the spelling that keeps the object meaning, `first`: %s", msg) + } +} + +func TestRetrieveLimitOne_Mdl1IsAListOfOne(t *testing.T) { + r, prog := retrieveOf(t, "mdl 1;\n"+retrieveMicroflow("limit 1")) + if r.First || r.Limit != "1" { + t.Errorf("mdl 1 `limit 1` → First=%v Limit=%q; want a Custom range with limit 1 (a list)", r.First, r.Limit) + } + if n := limitOneNotes(prog); n != 0 { + t.Errorf("mdl 1 `limit 1` has the new meaning, nothing to warn about; got %d note(s)", n) + } +} + +// Every other range was a list before and stays one; none of them warns. +func TestRetrieveOtherRanges_AreListsInBothVersionsWithoutWarning(t *testing.T) { + for _, header := range []string{"", "mdl 1;\n"} { + for _, clause := range []string{"limit 2", "limit 1 offset 3", "offset 3", "limit $N"} { + r, prog := retrieveOf(t, header+retrieveMicroflow(clause)) + if r.First { + t.Errorf("header %q, %q: bound as an object; every range other than the first object is a list", header, clause) + } + if n := limitOneNotes(prog); n != 0 { + t.Errorf("header %q, %q: %d MDL-V1-LIMIT1 note(s), want none", header, clause, n) + } + } + } +} + +// The object range has no offset in Mendix, so `first` does not take one. +func TestRetrieveFirst_RefusesLimitAndOffset(t *testing.T) { + for _, clause := range []string{"first offset 2", "first limit 1", "limit 1 first"} { + if _, errs := Build(retrieveMicroflow(clause)); len(errs) == 0 { + t.Errorf("%q parsed; `first` is the whole range and combines with neither limit nor offset", clause) + } + } +} + +// An association retrieve has no range in Mendix; `first` there would be dropped. +func TestRetrieveFirst_RefusedOnAnAssociation(t *testing.T) { + src := "create microflow M.F ($P: M.Parent) begin\n retrieve $C from $P/M.Child_Parent first;\nend;" + if _, errs := Build(src); len(errs) == 0 { + t.Errorf("`first` on an association retrieve was accepted") + } +} diff --git a/mdl/visitor/strict_properties_test.go b/mdl/visitor/strict_properties_test.go new file mode 100644 index 0000000000..08055bf209 --- /dev/null +++ b/mdl/visitor/strict_properties_test.go @@ -0,0 +1,225 @@ +// SPDX-License-Identifier: Apache-2.0 + +package visitor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" +) + +// A property list whose key is unknown, or whose value has a shape its key +// does not take, parses under mdl 0 as it always has — the property is +// ignored or read by its shape — and warns; under mdl 1 it is an error that +// names the key and, for a misspelling, the key meant (#732, ADR-0010 R11). +var strictPropertyCases = []struct { + name string + src string + code string + // what the mdl 1 error must say + want []string +}{ + { + "rest client operation key", + "create rest client M.Api (BaseUrl: 'https://x', Authentication: none) { operation GetUser { Method: get, pathh: '/users', Response: none } };", + "MDL-V1-PROP", []string{"unknown property 'pathh'", "REST client operation", "did you mean 'Path'"}, + }, + { + "rest client key", + "create rest client M.Api (BaseUrll: 'https://x', Authentication: none) { operation GetUser { Method: get, Path: '/u', Response: none } };", + "MDL-V1-PROP", []string{"unknown property 'BaseUrll'", "did you mean 'BaseUrl'"}, + }, + { + "rest client basic auth key", + "create rest client M.Api (BaseUrl: 'https://x', Authentication: basic (Username: 'u', Passwd: 'p')) { operation GetUser { Method: get, Path: '/u', Response: none } };", + "MDL-V1-PROP", []string{"unknown property 'Passwd'", "did you mean 'Password'"}, + }, + { + "response written as a body", + "create rest client M.Api (BaseUrl: 'https://x', Authentication: none) { operation Post { Method: post, Path: '/u', Response: json from $X } };", + "MDL-V1-PROPVALUE", []string{"'Response'", "json as $var"}, + }, + { + "body written as a response", + "create rest client M.Api (BaseUrl: 'https://x', Authentication: none) { operation Post { Method: post, Path: '/u', Body: status as $Y, Response: none } };", + "MDL-V1-PROPVALUE", []string{"'Body'", "json from $var"}, + }, + { + "rest client path as a number", + "create rest client M.Api (BaseUrl: 'https://x', Authentication: none) { operation Get { Method: get, Path: 42, Response: none } };", + "MDL-V1-PROPVALUE", []string{"'Path'", "a string"}, + }, + { + "published rest key", + "create published rest service M.S (Path: 'rest/s', Verison: '1.0') { };", + "MDL-V1-PROP", []string{"unknown property 'Verison'", "published REST service", "did you mean 'Version'"}, + }, + { + "business event service key", + "create business event service M.BE (ServiceNme: 'S', EventNamePrefix: '') { message Changed (Id: Long) publish; };", + "MDL-V1-PROP", []string{"unknown property 'ServiceNme'", "did you mean 'ServiceName'"}, + }, + { + "model key", + "create model M.GPT (Provider: MxCloudGenAI, Key: M.Key, DisplayNmae: 'GPT');", + "MDL-V1-PROP", []string{"unknown property 'DisplayNmae'", "model", "did you mean 'DisplayName'"}, + }, + { + "model provider as a string", + "create model M.GPT (Provider: 'MxCloudGenAI', Key: M.Key);", + "MDL-V1-PROPVALUE", []string{"'Provider'", "a name"}, + }, + { + "knowledge base key", + "create knowledge base M.KB (Provider: MxCloudGenAI, Key: M.Key, Enviroment: 'x');", + "MDL-V1-PROP", []string{"unknown property 'Enviroment'", "did you mean 'Environment'"}, + }, + { + "consumed mcp service key", + "create consumed mcp service M.Mcp (ProtocolVersion: v2025_03_26, Version: '1', Timeout: 30);", + "MDL-V1-PROP", []string{"unknown property 'Timeout'", "did you mean 'ConnectionTimeoutSeconds'"}, + }, + { + "agent key", + "create agent M.A (UsageType: Task, Model: M.GPT, Temprature: 0.5);", + "MDL-V1-PROP", []string{"unknown property 'Temprature'", "agent", "did you mean 'Temperature'"}, + }, + { + "agent model as a name", + "create agent M.A (UsageType: Task, Model: GPT);", + "MDL-V1-PROPVALUE", []string{"'Model'", "a qualified name"}, + }, + { + "agent tool block key", + "create agent M.A (UsageType: Task, Model: M.GPT) { tool T { Enabeld: true } };", + "MDL-V1-PROP", []string{"unknown property 'Enabeld'", "tool", "did you mean 'Enabled'"}, + }, +} + +func TestStrictPropertiesByVersion(t *testing.T) { + for _, tc := range strictPropertyCases { + t.Run(tc.name, func(t *testing.T) { + prog, errs := Build(tc.src) + if len(errs) > 0 { + t.Fatalf("mdl 0 must keep accepting it: %v", errs) + } + var notes []ast.LanguageNote + for _, n := range prog.LanguageNotes { + if strings.HasPrefix(n.Code, "MDL-V1-PROP") { + notes = append(notes, n) + } + } + if len(notes) != 1 || notes[0].Code != tc.code || notes[0].Line != 1 { + t.Fatalf("mdl 0: want one %s warning on line 1, got %+v", tc.code, notes) + } + + _, errs = Build("mdl 1;\n" + tc.src) + if len(errs) != 1 { + t.Fatalf("mdl 1: want one error, got %v", errs) + } + for _, w := range tc.want { + if !strings.Contains(errs[0].Error(), w) { + t.Errorf("mdl 1: the error does not say %q:\n%v", w, errs[0]) + } + } + if !strings.HasPrefix(errs[0].Error(), "line 2:") { + t.Errorf("mdl 1: the error must name the line: %v", errs[0]) + } + }) + } +} + +// Control: the forms describe writes, and every key each list reads, are +// accepted under mdl 1 without an error or a warning. +func TestStrictPropertiesAcceptWhatDescribeWrites(t *testing.T) { + for _, src := range []string{ + `create rest client M.Api ( + BaseUrl: 'https://x', + Authentication: basic (Username: @M.User, Password: $Pwd), + Folder: 'Integrations' +) +{ + operation GetUser { + Method: get, + Path: '/users/{id}', + Parameters: ($id: String), + Query: ($q: String), + Headers: ('Accept' = 'application/json', 'Auth' = 'Bearer ' + $Token), + Body: json from $Payload, + Timeout: 30, + Response: json as $Result + } + operation Put { Method: put, Path: '/x', Body: file from $F, Response: status as $S } + operation Tpl { Method: post, Path: '/x', Body: template '{}', Response: string as $S } + operation Map { Method: post, Path: '/x', Body: mapping M.Req { Name = name }, Response: mapping M.Res { Name = name } } + operation Del { Method: delete, Path: '/x', Response: file as $F } +};`, + "create rest client M.Open (OpenApi: 'spec.json', BaseUrl: 'https://x', Authentication: none);", + "create published rest service M.S (Path: 'rest/s', Version: '1.0', ServiceName: 'S', Folder: 'F') { };", + "create business event service M.BE (ServiceName: 'S', EventNamePrefix: '', Folder: 'F') { message Changed (Id: Long) publish; };", + "create model M.GPT (Provider: MxCloudGenAI, Key: M.Key, DisplayName: 'GPT', KeyName: 'k', KeyId: 'i', Environment: 'e', ResourceName: 'r', DeepLinkURL: 'u');", + "create knowledge base M.KB (Provider: MxCloudGenAI, Key: M.Key, ModelDisplayName: 'd', ModelName: 'n', KeyName: 'k', KeyId: 'i', Environment: 'e', DeepLinkURL: 'u');", + "create consumed mcp service M.Mcp (ProtocolVersion: 'v2025_03_26', Version: '1', ConnectionTimeoutSeconds: 30, Documentation: 'd');", + `create agent M.A ( + UsageType: Task, + Description: 'd', + Model: M.GPT, + Entity: M.Ctx, + Variables: ("Name": EntityAttribute, "Other": String), + MaxTokens: 100, + ToolChoice: Auto, + Temperature: 0.5, + TopP: 1, + SystemPrompt: $$You help.$$, + UserPrompt: 'Hi' +) +{ + mcp service M.Mcp { + Enabled: true, + Description: 'd' + } + tool T { + ToolType: Microflow, + Document: M.DoIt, + Enabled: false, + Description: 'd' + } + knowledge base KB { + Source: M.KB, + Collection: 'c', + MaxResults: 5, + Description: 'd', + Enabled: true + } +};`, + } { + prog, errs := Build("mdl 1;\n" + src) + if len(errs) > 0 { + t.Errorf("mdl 1 refused a well-formed property list: %v\n%s", errs, src) + continue + } + for _, n := range prog.LanguageNotes { + t.Errorf("mdl 1 warned on a well-formed property list: %+v", n) + } + prog, _ = Build(src) + for _, n := range prog.LanguageNotes { + if strings.HasPrefix(n.Code, "MDL-V1-PROP") { + t.Errorf("mdl 0 warned on a well-formed property list: %+v", n) + } + } + } +} + +// Under mdl 0 a mis-shaped value keeps its old reading: `Response: json from +// $X` sets the request body, as it always has. +func TestStrictPropertiesKeepTheOldReadingUnderMdl0(t *testing.T) { + prog, errs := Build(strictPropertyCases[3].src) + if len(errs) > 0 { + t.Fatal(errs) + } + op := prog.Statements[0].(*ast.CreateRestClientStmt).Operations[0] + if op.BodyType != "json" || op.BodyVariable != "$X" || op.ResponseType != "" { + t.Fatalf("mdl 0 changed what `Response: json from $X` builds: %+v", op) + } +} diff --git a/mdl/visitor/strict_terminators_test.go b/mdl/visitor/strict_terminators_test.go new file mode 100644 index 0000000000..c1010d0f34 --- /dev/null +++ b/mdl/visitor/strict_terminators_test.go @@ -0,0 +1,96 @@ +// SPDX-License-Identifier: Apache-2.0 + +package visitor + +import ( + "strings" + "testing" +) + +func notesWithCode(t *testing.T, src, code string) []string { + t.Helper() + prog, errs := Build(src) + if len(errs) > 0 { + t.Fatalf("mdl 0 must keep accepting it, got errors: %v\n%s", errs, src) + } + var out []string + for _, n := range prog.LanguageNotes { + if n.Code == code { + out = append(out, n.Message) + } + } + return out +} + +// Under mdl 0 a missing `;` keeps parsing and warns (MDL-V1-SEMI); under mdl 1 +// it is an error. A terminated script is the control: no note, no error. +func TestSemicolonRequiredUnderMdl1(t *testing.T) { + src := "show entities\nshow modules;" + if got := notesWithCode(t, src, "MDL-V1-SEMI"); len(got) != 1 { + t.Fatalf("mdl 0: want one MDL-V1-SEMI warning, for line 1, got %v", got) + } + _, errs := Build("mdl 1;\n" + src) + if len(errs) != 1 || !strings.Contains(errs[0].Error(), "no terminating `;`") { + t.Fatalf("mdl 1: want one missing-`;` error, got %v", errs) + } + if !strings.Contains(errs[0].Error(), "line 2:") { + t.Errorf("mdl 1: the error must name the statement's line, got %v", errs[0]) + } + + // Control: every statement terminated. + if got := notesWithCode(t, "show entities;\nshow modules;", "MDL-V1-SEMI"); len(got) != 0 { + t.Fatalf("mdl 0: a terminated script warned: %v", got) + } + if _, errs := Build("mdl 1;\nshow entities;\nshow modules;"); len(errs) != 0 { + t.Fatalf("mdl 1: a terminated script was refused: %v", errs) + } +} + +// A statement rule that consumes its own `;` (a java action body) is +// terminated; it must not be reported as missing one. +func TestSemicolonConsumedByTheStatementRule(t *testing.T) { + src := "mdl 1;\ncreate java action M.J () returns Boolean as $$return true;$$;\nshow modules;" + if _, errs := Build(src); len(errs) != 0 { + t.Fatalf("a statement ending in its own `;` was refused: %v", errs) + } +} + +// The SQL*Plus `/` terminator: accepted with a warning under mdl 0, an error +// under mdl 1. +func TestSlashTerminatorRejectedUnderMdl1(t *testing.T) { + src := "create persistent entity M.Note (Text: String(200))\n/\nshow modules;" + if got := notesWithCode(t, src, "MDL-V1-SLASH"); len(got) != 1 { + t.Fatalf("mdl 0: want one MDL-V1-SLASH warning, got %v", got) + } + _, errs := Build("mdl 1;\n" + src) + var slash int + for _, e := range errs { + if strings.Contains(e.Error(), "`/` is not a statement terminator") { + slash++ + } + } + if slash != 1 { + t.Fatalf("mdl 1: want one `/` error, got %v", errs) + } +} + +// The microflow, nanoflow and workflow rules end in `SEMICOLON? SLASH?` +// themselves, so their terminators are the inner rule's: `end;` followed by +// `/` is the `/` form, and `end;` alone is terminated. +func TestTerminatorsOfRulesThatEndInTheirOwn(t *testing.T) { + mf := "create microflow M.F () begin end;" + if got := notesWithCode(t, mf+"\n/\nshow modules;", "MDL-V1-SLASH"); len(got) != 1 { + t.Fatalf("mdl 0: want one MDL-V1-SLASH for a microflow's `/`, got %v", got) + } + _, errs := Build("mdl 1;\n" + mf + "\n/\nshow modules;") + if len(errs) != 1 || !strings.Contains(errs[0].Error(), "`/` is not a statement terminator") { + t.Fatalf("mdl 1: want only the `/` error for a microflow's `/`, got %v", errs) + } + if _, errs := Build("mdl 1;\n" + mf + "\nshow modules;"); len(errs) != 0 { + t.Fatalf("mdl 1: a microflow ending `end;` was refused: %v", errs) + } + _, errs = Build("mdl 1;\ncreate microflow M.F () begin end\nshow modules;") + if len(errs) != 1 || !strings.Contains(errs[0].Error(), `ending at "end"`) { + t.Fatalf("mdl 1: want a missing-`;` error naming `end`, got %v", errs) + } +} diff --git a/mdl/visitor/string_escapes_test.go b/mdl/visitor/string_escapes_test.go new file mode 100644 index 0000000000..5783a9f341 --- /dev/null +++ b/mdl/visitor/string_escapes_test.go @@ -0,0 +1,147 @@ +// SPDX-License-Identifier: Apache-2.0 + +package visitor + +import ( + "os" + "path/filepath" + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" +) + +// defaultOf builds a one-attribute entity whose default is lit and returns the +// default's value and the MDL-V1-ESCAPE notes. +func defaultOf(t *testing.T, header, lit string) (any, []ast.LanguageNote, []error) { + t.Helper() + prog, errs := Build(header + "create persistent entity M.Note (Text: String(200) default " + lit + ");") + if len(errs) > 0 { + return nil, nil, errs + } + var notes []ast.LanguageNote + for _, n := range prog.LanguageNotes { + if n.Code == "MDL-V1-ESCAPE" { + notes = append(notes, n) + } + } + e := prog.Statements[len(prog.Statements)-1].(*ast.CreateEntityStmt) + return e.Attributes[0].DefaultValue, notes, nil +} + +// A doubled apostrophe is the only escape under mdl 1; under mdl 0 a backslash still escapes +// and each literal whose value would change warns MDL-V1-ESCAPE (#732). +func TestStringEscapesByVersion(t *testing.T) { + for _, tc := range []struct { + lit string + mdl0, mdl1 string + warns bool // under mdl 0 + }{ + {`'C:\temp'`, "C:\temp", `C:\temp`, true}, + {`'a\nb'`, "a\nb", `a\nb`, true}, + {`'one\\two'`, `one\two`, `one\\two`, true}, + // Control: no backslash, or one before a character mdl 0 keeps as + // written, means the same under both, so nothing to warn about. + {`'it''s'`, "it's", "it's", false}, + {`'C:\data'`, `C:\data`, `C:\data`, false}, + } { + got, notes, errs := defaultOf(t, "", tc.lit) + if len(errs) > 0 { + t.Fatalf("mdl 0 %s: %v", tc.lit, errs) + } + if got != tc.mdl0 { + t.Errorf("mdl 0 %s: got %q, want %q", tc.lit, got, tc.mdl0) + } + if (len(notes) == 1) != tc.warns || len(notes) > 1 { + t.Errorf("mdl 0 %s: want warning=%v, got %v", tc.lit, tc.warns, notes) + } + got, notes, errs = defaultOf(t, "mdl 1;\n", tc.lit) + if len(errs) > 0 { + t.Fatalf("mdl 1 %s: %v", tc.lit, errs) + } + if got != tc.mdl1 { + t.Errorf("mdl 1 %s: got %q, want %q", tc.lit, got, tc.mdl1) + } + if len(notes) != 0 { + t.Errorf("mdl 1 %s: the new meaning must not warn: %v", tc.lit, notes) + } + } +} + +// The escape rule decides where a literal ends. Under mdl 1 `'C:\'` is a +// complete literal and `\'` does not escape the quote; under mdl 0 it is the +// other way round. +func TestStringEscapesDecideWhereALiteralEnds(t *testing.T) { + got, _, errs := defaultOf(t, "mdl 1;\n", `'C:\'`) + if len(errs) > 0 || got != `C:\` { + t.Fatalf("mdl 1: 'C:\\' must be the string C:\\, got %q %v", got, errs) + } + if _, _, errs := defaultOf(t, "", `'C:\'`); len(errs) == 0 { + t.Fatal(`mdl 0: 'C:\' must stay unterminated`) + } + + got, notes, errs := defaultOf(t, "", `'it\'s'`) + if len(errs) > 0 || got != "it's" || len(notes) != 1 { + t.Fatalf("mdl 0: 'it\\'s' must stay it's and warn, got %q %v %v", got, notes, errs) + } + _, _, errs = defaultOf(t, "mdl 1;\n", `'it\'s'`) + if len(errs) == 0 { + t.Fatal(`mdl 1: 'it\'s' must be refused — the literal ends at \'`) + } +} + +// The warning names the line of the literal, in source order among the other +// language notes. +func TestStringEscapeWarningLine(t *testing.T) { + prog, errs := Build("show modules\n;\ncreate persistent entity M.Note (\n Text: String(200) default 'a\\tb'\n);") + if len(errs) > 0 { + t.Fatal(errs) + } + var lines []int + for _, n := range prog.LanguageNotes { + if n.Code == "MDL-V1-ESCAPE" { + lines = append(lines, n.Line) + if !strings.Contains(n.Message, `'a\tb'`) { + t.Errorf("the warning must quote the literal: %s", n.Message) + } + } + } + if len(lines) != 1 || lines[0] != 4 { + t.Fatalf("want one MDL-V1-ESCAPE on line 4, got %v", lines) + } +} + +// Every string literal the visitor reads goes through unquoteStringLit, which +// knows the escape rule the literal was lexed with. A call to unquoteString on +// a token's text reads an mdl 1 literal with mdl 0's escapes, silently: the +// script parses and the value is wrong. Only the two helpers may call it. +func TestStringLiteralsAreReadUnderTheirEscapeRule(t *testing.T) { + files, err := filepath.Glob("*.go") + if err != nil { + t.Fatal(err) + } + allowed := map[string]bool{"visitor_helpers.go": true, "visitor_string_escapes.go": true} + for _, f := range files { + if strings.HasSuffix(f, "_test.go") || allowed[f] { + continue + } + src, err := os.ReadFile(f) + if err != nil { + t.Fatal(err) + } + for i, line := range strings.Split(string(src), "\n") { + if strings.Contains(line, "unquoteString(") { + t.Errorf("%s:%d: read a string literal with unquoteStringLit(node), not unquoteString(text)", f, i+1) + } + } + } +} + +// Under mdl 1 `\'` ends the literal and the parse error that follows is far +// from the cause; the hint names it. +func TestBackslashQuoteHintUnderMdl1(t *testing.T) { + _, _, errs := defaultOf(t, "mdl 1;\n", `'it\'s'`) + if len(errs) == 0 || !strings.Contains(errsText(errs), "Double the apostrophe") { + t.Fatalf("want the doubled-apostrophe hint, got %v", errs) + } +} diff --git a/mdl/visitor/trailing_comma_test.go b/mdl/visitor/trailing_comma_test.go new file mode 100644 index 0000000000..309c345df3 --- /dev/null +++ b/mdl/visitor/trailing_comma_test.go @@ -0,0 +1,74 @@ +// SPDX-License-Identifier: Apache-2.0 + +package visitor + +import ( + "reflect" + "strings" + "testing" +) + +// trailingCommaCases pairs a statement with a trailing comma in one of its +// lists (marked `,@`) against the same statement without it. ADR-0010 R11: +// trailing commas are allowed in every bracketed list, under every language +// version — it is additive, so no header is needed (#732). +// +// The `,@` marker is replaced by `,` for the trailing form and removed for the +// plain form, so the two sources differ in exactly that one comma. +var trailingCommaCases = map[string]string{ + "entity attributes": "create persistent entity M.Note (Text: String(200),@);", + "entity index": "create persistent entity M.Note (A: String(20), B: String(20)) index (A, B,@);", + "enumeration values": "create enumeration M.Color (Red 'Red', Green 'Green',@);", + "microflow parameters": "create microflow M.F ($A: String, $B: Integer,@) begin end;", + "java action parameters": "create java action M.J (A: String,@) returns Boolean as $$return true;$$;", + "call arguments": "create microflow M.F () begin call microflow M.G(A = 1, B = 2,@); end;", + "page parameters": "create page M.P (Params: { $A: M.E,@ }, Title: 'P', Layout: Atlas_Core.Atlas_Default) {};", + "rest client properties": "create rest client M.Api (BaseUrl: 'https://x', Authentication: none,@) { operation Get { Method: get, Path: '/x', Response: none,@ } };", + "rest client headers": "create rest client M.Api (BaseUrl: 'https://x', Authentication: none) { operation Get { Method: get, Path: '/x', Headers: ('A' = 'b',@), Response: none } };", + "published rest": "create published rest service M.S (Path: 'rest/s', Version: '1.0',@) { };", + "odata client": "create odata client M.C (ODataVersion: OData4, MetadataUrl: 'https://x/$metadata',@);", + "business events": "create business event service M.BE (ServiceName: 'S', EventNamePrefix: '',@) { message Changed (Id: Long,@) publish; };", + "agent model": "create model M.GPT (Provider: MxCloudGenAI, Key: M.Key,@);", + "agent": "create agent M.A (UsageType: Task, Model: M.GPT,@) { tool T { Enabled: true,@ } };", + "module role grant": "grant M.User on M.E (read *, write *,@);", + "line comment after it": "create enumeration M.Color (\n Red 'Red',\n Green 'Green',@ -- the last one\n);", + "block comment after it": "create enumeration M.Color (Red 'Red', Green 'Green',@ /* last */ );", +} + +func TestTrailingCommaInEveryList(t *testing.T) { + for name, src := range trailingCommaCases { + for _, header := range []string{"", "mdl 1;\n"} { + t.Run(name+"/"+strings.TrimSpace(header), func(t *testing.T) { + plain := header + strings.ReplaceAll(src, ",@", "") + trailing := header + strings.ReplaceAll(src, ",@", ",") + want, errs := Build(plain) + if len(errs) > 0 { + t.Fatalf("the plain form must parse: %v\n%s", errs, plain) + } + got, errs := Build(trailing) + if len(errs) > 0 { + t.Fatalf("a trailing comma must be accepted: %v\n%s", errs, trailing) + } + if !reflect.DeepEqual(got.Statements, want.Statements) { + t.Errorf("a trailing comma changed the statement:\n got %#v\n want %#v", got.Statements, want.Statements) + } + }) + } + } +} + +// A comma needs an item before it: `()` stays the only empty list, and `(,)` +// and `(a,,)` stay errors. +func TestTrailingCommaNeedsAnItem(t *testing.T) { + for _, src := range []string{ + "create persistent entity M.Note (,);", + "create enumeration M.Color (,);", + "create microflow M.F (,) begin end;", + "create enumeration M.Color (Red 'Red',,);", + "create enumeration M.Color (Red 'Red', , );", + } { + if _, errs := Build(src); len(errs) == 0 { + t.Errorf("a list holding only a comma was accepted: %s", src) + } + } +} diff --git a/mdl/visitor/visitor.go b/mdl/visitor/visitor.go index c4c4f6bb79..7682447119 100644 --- a/mdl/visitor/visitor.go +++ b/mdl/visitor/visitor.go @@ -121,6 +121,16 @@ func enhanceErrorMessage(msg, offendingLine string) string { " mdl 1;\n"+ " create entity Shop.Customer ( Name: String(200) ); (correct)", msg) } + // A `\'` in a string literal under mdl 1, where a backslash no longer + // escapes: the literal ends at the quote, and the rest of the line is read + // as MDL. Under mdl 0 the same text parses, so the hint only fires on an + // error. (#732) + if strings.Contains(offendingLine, `\'`) { + return fmt.Sprintf("%s\n\n Under mdl 1 a backslash is an ordinary character, so `\\'` ends the string\n"+ + " literal. Double the apostrophe instead, as in a Mendix expression:\n"+ + " default 'it''s' (correct)\n"+ + " default 'it\\'s' (mdl 0 only)", msg) + } // A bare `not $x` — Mendix requires `not(expr)`. The parse error surfaces // downstream (e.g. "missing THEN at '$x'"), so key off the source line, which // is unambiguous for `not $…`. (sudoku findings #3) @@ -497,6 +507,9 @@ type Builder struct { langVersion langver.Version langHeaderLine int langNotes []ast.LanguageNote + // stringLits are the string literals of the parse tree, by token index, + // for the string-escape rewrite to find the expression each is in. + stringLits map[int]antlr.TerminalNode } // NewBuilder creates a new AST builder. @@ -542,8 +555,7 @@ func build(input string, listen func(*Builder) antlr.ParseTreeListener) (*ast.Pr errListener.source = strings.Split(input, "\n") // Create lexer with custom error listener - is := antlr.NewInputStream(input) - lexer := parser.NewMDLLexer(is) + lexer := parser.NewMDLLexer(newScriptStream(input)) lexer.RemoveErrorListeners() lexer.AddErrorListener(errListener) @@ -557,6 +569,7 @@ func build(input string, listen func(*Builder) antlr.ParseTreeListener) (*ast.Pr builder := NewBuilder() tree := p.Program() antlr.ParseTreeWalkerDefault.Walk(listen(builder), tree) + builder.noteBackslashEscapes(stream.GetAllTokens()) // Combine syntax errors and builder errors allErrors := append(errListener.errors, builder.errors...) diff --git a/mdl/visitor/visitor_agenteditor.go b/mdl/visitor/visitor_agenteditor.go index 1d152eb1f5..6c3847bbc1 100644 --- a/mdl/visitor/visitor_agenteditor.go +++ b/mdl/visitor/visitor_agenteditor.go @@ -20,9 +20,10 @@ func (b *Builder) ExitCreateModelStatement(ctx *parser.CreateModelStatementConte } stmt.Documentation, stmt.DocumentationSet = findDocComment(ctx) if lit := ctx.STRING_LITERAL(); lit != nil { - stmt.Folder = unquoteString(lit.GetText()) + stmt.Folder = unquoteStringLit(lit) } + b.checkModelProperties(ctx.AllModelProperty(), &modelSchema) for _, p := range ctx.AllModelProperty() { propCtx := p.(*parser.ModelPropertyContext) // The first identifierOrKeyword is the property name; the second @@ -44,27 +45,27 @@ func (b *Builder) ExitCreateModelStatement(ctx *parser.CreateModelStatementConte } case "displayname": if lit := propCtx.STRING_LITERAL(); lit != nil { - stmt.DisplayName = unquoteString(lit.GetText()) + stmt.DisplayName = unquoteStringLit(lit) } case "keyname": if lit := propCtx.STRING_LITERAL(); lit != nil { - stmt.KeyName = unquoteString(lit.GetText()) + stmt.KeyName = unquoteStringLit(lit) } case "keyid": if lit := propCtx.STRING_LITERAL(); lit != nil { - stmt.KeyID = unquoteString(lit.GetText()) + stmt.KeyID = unquoteStringLit(lit) } case "environment": if lit := propCtx.STRING_LITERAL(); lit != nil { - stmt.Environment = unquoteString(lit.GetText()) + stmt.Environment = unquoteStringLit(lit) } case "resourcename": if lit := propCtx.STRING_LITERAL(); lit != nil { - stmt.ResourceName = unquoteString(lit.GetText()) + stmt.ResourceName = unquoteStringLit(lit) } case "deeplinkurl": if lit := propCtx.STRING_LITERAL(); lit != nil { - stmt.DeepLinkURL = unquoteString(lit.GetText()) + stmt.DeepLinkURL = unquoteStringLit(lit) } } } @@ -93,7 +94,7 @@ func parseModelProps(props []parser.IModelPropertyContext) map[string]string { if qn := pc.QualifiedName(); qn != nil { m[key] = getQualifiedNameText(qn) } else if lit := pc.STRING_LITERAL(); lit != nil { - m[key] = unquoteString(lit.GetText()) + m[key] = unquoteStringLit(lit) } else if num := pc.NUMBER_LITERAL(); num != nil { m[key] = num.GetText() } else if bl := pc.BooleanLiteral(); bl != nil { @@ -131,7 +132,7 @@ func parseVariableDefsFromProps(props []parser.IModelPropertyContext) []ast.Agen // Key is STRING_LITERAL or QUOTED_IDENTIFIER var key string if sl := vdc.STRING_LITERAL(); sl != nil { - key = unquoteString(sl.GetText()) + key = unquoteStringLit(sl) } else if qi := vdc.QUOTED_IDENTIFIER(); qi != nil { key = unquoteIdentifier(qi.GetText()) } @@ -160,9 +161,10 @@ func (b *Builder) ExitCreateConsumedMCPServiceStatement(ctx *parser.CreateConsum } stmt.OuterDocumentation, stmt.DocumentationSet = findDocComment(ctx) if lit := ctx.STRING_LITERAL(); lit != nil { - stmt.Folder = unquoteString(lit.GetText()) + stmt.Folder = unquoteStringLit(lit) } + b.checkModelProperties(ctx.AllModelProperty(), &consumedMCPServiceSchema) props := parseModelProps(ctx.AllModelProperty()) stmt.ProtocolVersion = props["protocolversion"] stmt.Version = props["version"] @@ -186,9 +188,10 @@ func (b *Builder) ExitCreateKnowledgeBaseStatement(ctx *parser.CreateKnowledgeBa } stmt.Documentation, stmt.DocumentationSet = findDocComment(ctx) if lit := ctx.STRING_LITERAL(); lit != nil { - stmt.Folder = unquoteString(lit.GetText()) + stmt.Folder = unquoteStringLit(lit) } + b.checkModelProperties(ctx.AllModelProperty(), &knowledgeBaseSchema) props := parseModelProps(ctx.AllModelProperty()) stmt.Provider = props["provider"] if k, ok := props["key"]; ok { @@ -217,9 +220,10 @@ func (b *Builder) ExitCreateAgentStatement(ctx *parser.CreateAgentStatementConte } stmt.Documentation, stmt.DocumentationSet = findDocComment(ctx) if lit := ctx.STRING_LITERAL(); lit != nil { - stmt.Folder = unquoteString(lit.GetText()) + stmt.Folder = unquoteStringLit(lit) } + b.checkModelProperties(ctx.AllModelProperty(), &agentSchema) props := parseModelProps(ctx.AllModelProperty()) stmt.UsageType = props["usagetype"] stmt.Description = props["description"] @@ -259,6 +263,7 @@ func (b *Builder) ExitCreateAgentStatement(ctx *parser.CreateAgentStatementConte bodyCtx := body.(*parser.AgentBodyContext) for _, block := range bodyCtx.AllAgentBodyBlock() { blk := block.(*parser.AgentBodyBlockContext) + b.checkAgentBodyBlock(blk) blockProps := parseModelProps(blk.AllModelProperty()) if blk.MCP() != nil && blk.SERVICE() != nil { diff --git a/mdl/visitor/visitor_alter.go b/mdl/visitor/visitor_alter.go index c878412ef3..859174a7f6 100644 --- a/mdl/visitor/visitor_alter.go +++ b/mdl/visitor/visitor_alter.go @@ -17,6 +17,10 @@ func (b *Builder) ExitAlterStatement(ctx *parser.AlterStatementContext) { b.exitAlterDocumentStatement(ctx) return } + if ctx.MICROFLOW() != nil || ctx.NANOFLOW() != nil { + b.exitAlterFlowStatement(ctx) + return + } // Handle ALTER PAGES … SET LAYOUT (the bulk repoint) if sub := ctx.AlterPagesStylingStatement(); sub != nil { diff --git a/mdl/visitor/visitor_alter_agenteditor.go b/mdl/visitor/visitor_alter_agenteditor.go index 377a59fc27..94d43273ba 100644 --- a/mdl/visitor/visitor_alter_agenteditor.go +++ b/mdl/visitor/visitor_alter_agenteditor.go @@ -76,6 +76,7 @@ func (b *Builder) applyAlterAgentAction(stmt *ast.AlterAgentStmt, ctx *parser.Al // KNOWLEDGE BASE block to the matching field on the ALTER AGENT stmt. // Mirrors the CREATE AGENT body-block parsing in ExitCreateAgentStatement. func (b *Builder) appendAgentBodyBlock(stmt *ast.AlterAgentStmt, blk *parser.AgentBodyBlockContext) { + b.checkAgentBodyBlock(blk) blockProps := parseModelProps(blk.AllModelProperty()) switch { @@ -149,7 +150,7 @@ func parseAgentEditorAlterAssignments(props []parser.IAgentEditorAlterAssignment func agentEditorAlterValueText(ctx *parser.AgentEditorAlterValueContext) string { if sl := ctx.STRING_LITERAL(); sl != nil { - return unquoteString(sl.GetText()) + return unquoteStringLit(sl) } if num := ctx.NUMBER_LITERAL(); num != nil { return num.GetText() diff --git a/mdl/visitor/visitor_alter_flow.go b/mdl/visitor/visitor_alter_flow.go new file mode 100644 index 0000000000..784848fdaf --- /dev/null +++ b/mdl/visitor/visitor_alter_flow.go @@ -0,0 +1,53 @@ +// SPDX-License-Identifier: Apache-2.0 + +package visitor + +import ( + "strings" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/grammar/parser" +) + +// exitAlterFlowStatement builds an AlterFlowStmt from +// ALTER MICROFLOW|NANOFLOW Module.Name { insert / replace / drop }. +func (b *Builder) exitAlterFlowStatement(ctx *parser.AlterStatementContext) { + stmt := &ast.AlterFlowStmt{Nanoflow: ctx.NANOFLOW() != nil} + if qn := ctx.QualifiedName(); qn != nil { + stmt.Name = buildQualifiedName(qn) + } + for _, opCtx := range ctx.AllAlterFlowOperation() { + op := opCtx.(*parser.AlterFlowOperationContext) + o := &ast.AlterFlowOperation{Target: alterFlowTargetText(op.AlterFlowTarget())} + switch { + case op.INSERT() != nil && op.BEFORE() != nil: + o.Op = ast.AlterFlowInsertBefore + case op.INSERT() != nil: + o.Op = ast.AlterFlowInsertAfter + case op.REPLACE() != nil: + o.Op = ast.AlterFlowReplace + default: + o.Op = ast.AlterFlowDrop + } + if body := op.MicroflowBody(); body != nil { + o.Body = buildMicroflowBody(body) + } + stmt.Operations = append(stmt.Operations, o) + } + b.statements = append(b.statements, stmt) +} + +// alterFlowTargetText returns a target as the author wrote it — whitespace and +// quoting included — since a statement pattern is matched token by token +// against describe's rendering, and the grammar only knows it as a run of +// arbitrary tokens. +func alterFlowTargetText(ctx parser.IAlterFlowTargetContext) string { + if ctx == nil { + return "" + } + start, stop := ctx.GetStart(), ctx.GetStop() + if start == nil || stop == nil { + return strings.TrimSpace(ctx.GetText()) + } + return strings.TrimSpace(start.GetInputStream().GetText(start.GetStart(), stop.GetStop())) +} diff --git a/mdl/visitor/visitor_alter_flow_test.go b/mdl/visitor/visitor_alter_flow_test.go new file mode 100644 index 0000000000..2391ffc992 --- /dev/null +++ b/mdl/visitor/visitor_alter_flow_test.go @@ -0,0 +1,60 @@ +// SPDX-License-Identifier: Apache-2.0 + +package visitor + +import ( + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" +) + +// alter microflow / nanoflow (ADR-0012 decision 3, ako/mxcli#736): targets are +// content addresses kept as written, fragments are microflow statements. +func TestAlterFlow_OperationsAndTargets(t *testing.T) { + prog, errs := Build(`alter microflow FeedbackModule.VAL_Feedback { + insert after $IsValidEmail { log info node 'Feedback' 'Email checked'; } + insert before 'Email is Valid?' @2 { declare $n Integer = 1; set $n = 2; } + replace log * node 'Debug' * with { log warning node 'Debug' 'x'; } + drop set $ValidFeedback = false @3; + };`) + if len(errs) > 0 { + t.Fatalf("parse errors: %v", errs) + } + stmt, ok := prog.Statements[0].(*ast.AlterFlowStmt) + if !ok { + t.Fatalf("want *ast.AlterFlowStmt, got %T", prog.Statements[0]) + } + if stmt.Nanoflow || stmt.Name.String() != "FeedbackModule.VAL_Feedback" { + t.Errorf("header: nanoflow=%v name=%s", stmt.Nanoflow, stmt.Name) + } + want := []struct { + op ast.AlterFlowOpKind + target string + body int + }{ + {ast.AlterFlowInsertAfter, "$IsValidEmail", 1}, + {ast.AlterFlowInsertBefore, "'Email is Valid?' @2", 2}, + {ast.AlterFlowReplace, "log * node 'Debug' *", 1}, + {ast.AlterFlowDrop, "set $ValidFeedback = false @3", 0}, + } + if len(stmt.Operations) != len(want) { + t.Fatalf("want %d operations, got %d", len(want), len(stmt.Operations)) + } + for i, w := range want { + got := stmt.Operations[i] + if got.Op != w.op || got.Target != w.target || len(got.Body) != w.body { + t.Errorf("op %d: got %q %q body=%d, want %q %q body=%d", i, got.Op, got.Target, len(got.Body), w.op, w.target, w.body) + } + } +} + +func TestAlterFlow_Nanoflow(t *testing.T) { + prog, errs := Build(`alter nanoflow M.NF { drop $X; }`) + if len(errs) > 0 { + t.Fatalf("parse errors: %v", errs) + } + stmt := prog.Statements[0].(*ast.AlterFlowStmt) + if !stmt.Nanoflow || stmt.Operations[0].Op != ast.AlterFlowDrop || stmt.Operations[0].Target != "$X" { + t.Errorf("got %+v %+v", stmt, stmt.Operations[0]) + } +} diff --git a/mdl/visitor/visitor_alter_page.go b/mdl/visitor/visitor_alter_page.go index cbc79b2dce..26168005fe 100644 --- a/mdl/visitor/visitor_alter_page.go +++ b/mdl/visitor/visitor_alter_page.go @@ -171,7 +171,7 @@ func (b *Builder) buildAlterPageAssignment(ctx *parser.AlterPageAssignmentContex return identifierOrKeywordText(id), buildActionV3(acCtx) } if sl := ctx.STRING_LITERAL(); sl != nil { - return unquoteString(sl.GetText()), buildActionV3(acCtx) + return unquoteStringLit(sl), buildActionV3(acCtx) } return "Action", buildActionV3(acCtx) } @@ -194,7 +194,7 @@ func (b *Builder) buildAlterPageAssignment(ctx *parser.AlterPageAssignmentContex if id := ctx.IdentifierOrKeyword(); id != nil { name = identifierOrKeywordText(id) } else if sl := ctx.STRING_LITERAL(); sl != nil { - name = unquoteString(sl.GetText()) + name = unquoteStringLit(sl) } value := buildPropertyValueV3(ctx.PropertyValueV3()) @@ -275,7 +275,7 @@ func (b *Builder) buildAlterTarget(ctx parser.IAlterTargetContext) ast.WidgetRef } } if sl := tc.STRING_LITERAL(); sl != nil { - ref.Caption = unquoteString(sl.GetText()) + ref.Caption = unquoteStringLit(sl) return ref } ids := tc.AllIdentifierOrKeyword() @@ -357,7 +357,7 @@ func (b *Builder) exitAlterPagesStylingStatement(ctx *parser.AlterPagesStylingSt } if lit := ctx.STRING_LITERAL(); lit != nil { // The WHERE value as a quoted string — a full widget id. - stmt.WidgetType = unquoteString(lit.GetText()) + stmt.WidgetType = unquoteStringLit(lit) } else if len(ids) > 0 { stmt.WidgetType = identifierOrKeywordText(ids[0]) } @@ -368,14 +368,14 @@ func (b *Builder) exitAlterPagesStylingStatement(ctx *parser.AlterPagesStylingSt if len(lits) == 0 { continue } - assignment := ast.StylingAssignment{Property: unquoteString(lits[0].GetText())} + assignment := ast.StylingAssignment{Property: unquoteStringLit(lits[0])} switch { case ac.ON() != nil: assignment.IsToggle, assignment.ToggleOn = true, true case ac.OFF() != nil: assignment.IsToggle, assignment.ToggleOn = true, false case len(lits) > 1: - assignment.Value = unquoteString(lits[1].GetText()) + assignment.Value = unquoteStringLit(lits[1]) } stmt.Assignments = append(stmt.Assignments, assignment) } diff --git a/mdl/visitor/visitor_annotation.go b/mdl/visitor/visitor_annotation.go index da63799599..6c7b7aeabe 100644 --- a/mdl/visitor/visitor_annotation.go +++ b/mdl/visitor/visitor_annotation.go @@ -27,7 +27,7 @@ func (b *Builder) ExitCreateAnnotationStatement(ctx *parser.CreateAnnotationStat // single-quoted and single-line, so a dollar-quoted block is accepted // for the caption as it is for Java and JavaScript source. if lit := propCtx.STRING_LITERAL(); lit != nil { - stmt.Caption = unquoteString(lit.GetText()) + stmt.Caption = unquoteStringLit(lit) } else if dollar := propCtx.DOLLAR_STRING(); dollar != nil { stmt.Caption = unquoteDollarString(dollar.GetText()) } diff --git a/mdl/visitor/visitor_association.go b/mdl/visitor/visitor_association.go index b0a68f6628..d059f06ad4 100644 --- a/mdl/visitor/visitor_association.go +++ b/mdl/visitor/visitor_association.go @@ -86,7 +86,7 @@ func (b *Builder) ExitCreateAssociationStatement(ctx *parser.CreateAssociationSt // COMMENT if optCtx.COMMENT() != nil && optCtx.STRING_LITERAL() != nil { - stmt.Comment = unquoteString(optCtx.STRING_LITERAL().GetText()) + stmt.Comment = unquoteStringLit(optCtx.STRING_LITERAL()) } } } @@ -283,7 +283,7 @@ func (b *Builder) ExitAlterAssociationAction(ctx *parser.AlterAssociationActionC b.statements = append(b.statements, &ast.AlterAssociationStmt{ Name: name, Operation: ast.AlterAssociationSetComment, - Comment: unquoteString(ctx.STRING_LITERAL().GetText()), + Comment: unquoteStringLit(ctx.STRING_LITERAL()), }) return } diff --git a/mdl/visitor/visitor_businessevents.go b/mdl/visitor/visitor_businessevents.go index ee4a57bd4d..89e7a98129 100644 --- a/mdl/visitor/visitor_businessevents.go +++ b/mdl/visitor/visitor_businessevents.go @@ -20,6 +20,7 @@ func (b *Builder) ExitCreateBusinessEventServiceStatement(ctx *parser.CreateBusi prop := propCtx.(*parser.OdataPropertyAssignmentContext) name := identifierOrKeywordText(prop.IdentifierOrKeyword()) value := odataAssignmentValueText(prop) + b.checkProperty(prop, &businessEventServiceSchema, name, odataPropertyShape(prop)) switch strings.ToLower(name) { case "servicename": diff --git a/mdl/visitor/visitor_connection.go b/mdl/visitor/visitor_connection.go index d3856f859e..584738b66b 100644 --- a/mdl/visitor/visitor_connection.go +++ b/mdl/visitor/visitor_connection.go @@ -17,7 +17,7 @@ func (b *Builder) ExitConnectStatement(ctx *parser.ConnectStatementContext) { // CONNECT LOCAL 'path' strings := ctx.AllSTRING_LITERAL() if len(strings) > 0 { - path := unquoteString(strings[0].GetText()) + path := unquoteStringLit(strings[0]) b.statements = append(b.statements, &ast.ConnectStmt{Path: path}) } } diff --git a/mdl/visitor/visitor_datatransformer.go b/mdl/visitor/visitor_datatransformer.go index d5d9523680..169b20ab9e 100644 --- a/mdl/visitor/visitor_datatransformer.go +++ b/mdl/visitor/visitor_datatransformer.go @@ -29,10 +29,10 @@ func (b *Builder) ExitCreateDataTransformerStatement(ctx *parser.CreateDataTrans // direct STRING_LITERALs, so an index would depend on whether a folder was // given. if tok := ctx.GetSource(); tok != nil { - stmt.SourceJSON = unquoteString(tok.GetText()) + stmt.SourceJSON = unquoteStringLit(tok) } if tok := ctx.GetFolder(); tok != nil { - stmt.Folder = unquoteString(tok.GetText()) + stmt.Folder = unquoteStringLit(tok) } // Steps @@ -48,7 +48,7 @@ func (b *Builder) ExitCreateDataTransformerStatement(ctx *parser.CreateDataTrans step.Technology = "XSLT" } if sl := sc.STRING_LITERAL(); sl != nil { - step.Expression = unquoteString(sl.GetText()) + step.Expression = unquoteStringLit(sl) } else if ds := sc.DOLLAR_STRING(); ds != nil { step.Expression = unquoteDollarString(ds.GetText()) } diff --git a/mdl/visitor/visitor_dbconnection.go b/mdl/visitor/visitor_dbconnection.go index 6864c5efbe..8de82bc511 100644 --- a/mdl/visitor/visitor_dbconnection.go +++ b/mdl/visitor/visitor_dbconnection.go @@ -17,7 +17,7 @@ func (b *Builder) ExitCreateDatabaseConnectionStatement(ctx *parser.CreateDataba Name: buildQualifiedName(ctx.QualifiedName()), } if lit := ctx.STRING_LITERAL(); lit != nil { - stmt.Folder = unquoteString(lit.GetText()) + stmt.Folder = unquoteStringLit(lit) } // Parse options @@ -25,7 +25,7 @@ func (b *Builder) ExitCreateDatabaseConnectionStatement(ctx *parser.CreateDataba opt := optCtx.(*parser.DatabaseConnectionOptionContext) if opt.TYPE() != nil && opt.STRING_LITERAL() != nil { - stmt.DatabaseType = unquoteString(opt.STRING_LITERAL().GetText()) + stmt.DatabaseType = unquoteStringLit(opt.STRING_LITERAL()) } if opt.CONNECTION() != nil && opt.STRING_TYPE() != nil { // CONNECTION STRING <value> @@ -35,17 +35,17 @@ func (b *Builder) ExitCreateDatabaseConnectionStatement(ctx *parser.CreateDataba stmt.ConnectionString = qn.String() stmt.ConnectionStringIsRef = true } else if opt.STRING_LITERAL() != nil { - stmt.ConnectionString = unquoteString(opt.STRING_LITERAL().GetText()) + stmt.ConnectionString = unquoteStringLit(opt.STRING_LITERAL()) } } if opt.HOST() != nil && opt.STRING_LITERAL() != nil { - stmt.Host = unquoteString(opt.STRING_LITERAL().GetText()) + stmt.Host = unquoteStringLit(opt.STRING_LITERAL()) } if opt.PORT() != nil && opt.NUMBER_LITERAL() != nil { stmt.Port, _ = strconv.Atoi(opt.NUMBER_LITERAL().GetText()) } if opt.DATABASE() != nil && opt.STRING_LITERAL() != nil { - stmt.Database = unquoteString(opt.STRING_LITERAL().GetText()) + stmt.Database = unquoteStringLit(opt.STRING_LITERAL()) } if opt.USERNAME() != nil { if opt.AT() != nil && opt.QualifiedName() != nil { @@ -53,7 +53,7 @@ func (b *Builder) ExitCreateDatabaseConnectionStatement(ctx *parser.CreateDataba stmt.UserName = qn.String() stmt.UserNameIsRef = true } else if opt.STRING_LITERAL() != nil { - stmt.UserName = unquoteString(opt.STRING_LITERAL().GetText()) + stmt.UserName = unquoteStringLit(opt.STRING_LITERAL()) } } if opt.PASSWORD() != nil { @@ -62,7 +62,7 @@ func (b *Builder) ExitCreateDatabaseConnectionStatement(ctx *parser.CreateDataba stmt.Password = qn.String() stmt.PasswordIsRef = true } else if opt.STRING_LITERAL() != nil { - stmt.Password = unquoteString(opt.STRING_LITERAL().GetText()) + stmt.Password = unquoteStringLit(opt.STRING_LITERAL()) } } } @@ -86,7 +86,7 @@ func (b *Builder) ExitCreateDatabaseConnectionStatement(ctx *parser.CreateDataba if ds := qc.DOLLAR_STRING(); ds != nil { q.SQL = unquoteDollarString(ds.GetText()) } else if sl := qc.STRING_LITERAL(0); sl != nil { - q.SQL = unquoteString(sl.GetText()) + q.SQL = unquoteStringLit(sl) } // RETURNS entity @@ -122,7 +122,7 @@ func (b *Builder) ExitCreateDatabaseConnectionStatement(ctx *parser.CreateDataba slIdx = defaultIdx // SQL used DOLLAR_STRING } if sl := qc.STRING_LITERAL(slIdx); sl != nil { - paramDef.DefaultValue = unquoteString(sl.GetText()) + paramDef.DefaultValue = unquoteStringLit(sl) } defaultIdx++ } else if hasNull { diff --git a/mdl/visitor/visitor_deprecations.go b/mdl/visitor/visitor_deprecations.go index 72e44bc225..491ab7a6d2 100644 --- a/mdl/visitor/visitor_deprecations.go +++ b/mdl/visitor/visitor_deprecations.go @@ -6,6 +6,7 @@ import ( "github.com/mendixlabs/mxcli/mdl/ast" "github.com/mendixlabs/mxcli/mdl/deprecation" "github.com/mendixlabs/mxcli/mdl/grammar/parser" + "github.com/mendixlabs/mxcli/mdl/langver" "github.com/antlr4-go/antlr/v4" ) @@ -29,24 +30,91 @@ func (b *Builder) recordDeprecation(code string, tok antlr.Token, subject string } // createOrReplaceIsNotAnAlias lists the create kinds (createStatementKind's -// words) where `or replace` does NOT mean `or modify`, so rewriting it would -// change the script. Measured against the visitors, and pinned by -// TestCreateOrReplaceMatchesModifyExceptExemptKinds: +// words) where `or replace` does NOT mean `or modify` under any language +// version, so rewriting it would change the script. Measured against the +// visitors, and pinned by TestCreateOrReplaceMatchesModifyExceptExemptKinds: // // - translations: `or replace` replaces the whole set, `or modify` merges. -// - userrole, demouser: the visitor reads only `or modify`; `or replace` is -// a plain create, which fails on an existing role or user. // -// `create or replace view entity` (kind "entity") drops and recreates the view -// entity, and is exempted in recordCreateOrReplace. +// Three more kinds differ only under mdl 0 and are gated below: a view entity, +// a user role and a demo user. var createOrReplaceIsNotAnAlias = map[string]bool{ "translations": true, - "userrole": true, - "demouser": true, +} + +// viewEntityReplaceIsModify is R1 for view entities (ADR-0010; #731). Under +// mdl 0, `create or replace view entity` deletes the entity and creates a new +// one: a new $ID and GUID, and the access rules and associations that pointed +// at the old one are gone with it. From mdl 1 it is `create or modify`, the +// identity-carrying rewrite, like `or replace` on every other document type. +var viewEntityReplaceIsModify = langver.Change{ + Code: "MDL-V1-REPLACE01", + Since: langver.V1, + Old: "`create or replace view entity` deletes the view entity and creates a new one (a new identity; its access rules and associations are lost)", + New: "`create or modify view entity`, which rewrites it in place and keeps its identity", +} + +// roleReplaceIsModify is R1 for user roles and demo users (#731). Under mdl 0 +// their visitors read only `or modify`, so `or replace` is a plain create that +// fails on an existing role or user. From mdl 1 it is `create or modify`. +var roleReplaceIsModify = langver.Change{ + Code: "MDL-V1-REPLACE02", + Since: langver.V1, + Old: "`create or replace` on a user role or demo user is a plain create, which fails when it already exists", + New: "`create or modify`", +} + +// replaceGate returns the language change `or replace` goes through for this +// create statement, and false when its meaning does not depend on the version. +func replaceGate(ctx *parser.CreateStatementContext) (langver.Change, bool) { + switch createStatementKind(ctx) { + case "userrole", "demouser": + return roleReplaceIsModify, true + case "entity": + if ent, ok := ctx.CreateEntityStatement().(*parser.CreateEntityStatementContext); ok && ent.VIEW() != nil { + return viewEntityReplaceIsModify, true + } + } + return langver.Change{}, false +} + +// replaceGateFix is the rewrite that keeps an mdl 0 `create or replace` on a +// gated kind meaning what it means there. +// +// For a user role or demo user that is a plain create, so `or replace` is +// dropped. A view entity's drop-and-recreate has no mdl 1 spelling: `create or +// modify` keeps the identity the old meaning discards, and a `drop` in front +// would fail where the old statement created the entity. +func replaceGateFix(ctx *parser.CreateStatementContext, c langver.Change) (*ast.Fix, string) { + if c.Code != roleReplaceIsModify.Code { + return nil, "`create or replace view entity` drops and recreates the view entity under mdl 0, which no " + + "mdl 1 statement does: write `create or modify view entity` to keep its identity, or `drop entity` " + + "then `create view entity` to discard it" + } + or, next := ctx.OR().GetSymbol(), ctx.REPLACE().GetSymbol() + stop := next.GetStop() + 1 + if is := next.GetInputStream(); stop < is.Size() && is.GetText(stop, stop) == " " { + stop++ + } + return &ast.Fix{Edits: []ast.TextEdit{{Start: or.GetStart(), Stop: stop}}}, "" +} + +// replaceMeansModify reports whether `create or replace` on createStmt builds +// `create or modify` under the script's language version. The document +// builders ask it; recordCreateOrReplace reports what it decided. +func (b *Builder) replaceMeansModify(createStmt *parser.CreateStatementContext) bool { + if createStmt == nil || createStmt.OR() == nil || createStmt.REPLACE() == nil { + return false + } + if c, gated := replaceGate(createStmt); gated { + return c.Applies(b.langVersion) + } + return true } // recordCreateOrReplace records MDL-DEPR001 for a `create or replace` whose -// meaning is exactly `create or modify`. +// meaning is exactly `create or modify`, and the language change for one whose +// meaning is kept at mdl 0. func (b *Builder) recordCreateOrReplace(ctx *parser.CreateStatementContext) { if ctx.OR() == nil || ctx.REPLACE() == nil { return @@ -55,7 +123,9 @@ func (b *Builder) recordCreateOrReplace(ctx *parser.CreateStatementContext) { if kind == "" || createOrReplaceIsNotAnAlias[kind] { return } - if ent, ok := ctx.CreateEntityStatement().(*parser.CreateEntityStatementContext); ok && ent.VIEW() != nil { + if c, gated := replaceGate(ctx); gated && !b.gate(c, ctx) { + fix, why := replaceGateFix(ctx, c) + b.fixLastNote(c.Code, fix, why) return } b.recordDeprecation(deprecation.CreateOrReplace, ctx.REPLACE().GetSymbol(), kind) diff --git a/mdl/visitor/visitor_deprecations_test.go b/mdl/visitor/visitor_deprecations_test.go index 6bb425c07b..a75e35ce08 100644 --- a/mdl/visitor/visitor_deprecations_test.go +++ b/mdl/visitor/visitor_deprecations_test.go @@ -50,6 +50,10 @@ func TestRegistryExamplesRecordTheirCode(t *testing.T) { t.Errorf("Example and CanonicalExample build different statements:\n old: %#v\n canon: %#v", old.Statements, canon.Statements) } + // A structural rewrite is proven by the AST comparison above alone. + if e.Rewrite.Structural != "" { + return + } // The rewrite is a token swap; the canonical example must be exactly it. re := regexp.MustCompile(`(?i)\b` + regexp.QuoteMeta(e.Rewrite.Token) + `\b`) if got := re.ReplaceAllString(e.Example, e.Rewrite.Replacement); got != e.CanonicalExample { @@ -67,6 +71,7 @@ var createOrReplaceCases = map[string]string{ "entity": "persistent entity M.Customer (Name: String(200));", "association": "association M.Order_Customer from M.Order to M.Customer type Reference;", "module": "module M;", + "modulerole": "module role M.Admin description 'Full access';", "microflow": "microflow M.ACT_Recalculate () begin return; end;", "javaaction": "java action M.FormatCurrency(Amount: Decimal not null) returns String as $$return \"\";$$;", "javascriptaction": "javascript action M.IsStrictMode() returns Boolean platform Web as $$return true;$$;", @@ -154,6 +159,12 @@ func createStatementKinds(t *testing.T) []string { // it is not an alias, and reporting it would tell the user to make a change // that alters their script. Both directions are asserted, so the exemption list // in the visitor cannot drift from what the visitors actually do. +// +// Which kinds are exempt depends on the language version (#731, ADR-0011): +// under mdl 1, `or replace` on a view entity, user role or demo user means +// `or modify` like everywhere else, so only translations stay exempt. Under +// mdl 0 those three keep their alpha meaning and report the language change +// that would alter it instead of the alias. func TestCreateOrReplaceMatchesModifyExceptExemptKinds(t *testing.T) { for _, kind := range createStatementKinds(t) { if _, ok := createOrReplaceCases[kind]; !ok { @@ -165,48 +176,74 @@ func TestCreateOrReplaceMatchesModifyExceptExemptKinds(t *testing.T) { for k, v := range createOrReplaceCases { cases[k] = v } - // A view entity is kind "entity" too, and its `or replace` drops and recreates. + // A view entity is kind "entity" too. cases["entity (view)"] = "view entity M.V (Name: String(100)) as (select c.Name as Name from M.Customer as c);" - for name, body := range cases { - t.Run(name, func(t *testing.T) { - rep := mustBuild(t, "create or replace "+body) - mod := mustBuild(t, "create or modify "+body) - if len(rep.Statements) != 1 || len(mod.Statements) != 1 { - t.Fatalf("want one statement each, got %d and %d — the case does not exercise the visitor", - len(rep.Statements), len(mod.Statements)) - } - foldPageModeFlags(rep.Statements) - foldPageModeFlags(mod.Statements) - same := reflect.DeepEqual(rep.Statements, mod.Statements) - exempt := createOrReplaceIsNotAnAlias[name] || name == "entity (view)" + // gatedChange names the language change a kind's `or replace` goes through; + // a kind absent here means the same thing under every version. + gatedChange := map[string]string{ + "entity (view)": viewEntityReplaceIsModify.Code, + "userrole": roleReplaceIsModify.Code, + "demouser": roleReplaceIsModify.Code, + } - switch { - case exempt && same: - t.Errorf("exempt, but `or replace` and `or modify` build the same statements: " + - "it is an alias after all — remove the exemption") - case !exempt && !same: - t.Errorf("`or replace` and `or modify` build different statements, so rewriting one "+ - "to the other changes the script — exempt this kind or fix the visitor:\n"+ - " replace: %#v\n modify: %#v", rep.Statements, mod.Statements) - } + for _, version := range []struct { + name, header string + v1 bool + }{{"mdl 0", "", false}, {"mdl 1", "mdl 1;\n", true}} { + for name, body := range cases { + t.Run(version.name+"/"+name, func(t *testing.T) { + rep := mustBuild(t, version.header+"create or replace "+strings.TrimSuffix(body, ";")+";") // every statement ends with ; (valid under mdl 0 and mdl 1) + mod := mustBuild(t, version.header+"create or modify "+strings.TrimSuffix(body, ";")+";") + if len(rep.Statements) != 1 || len(mod.Statements) != 1 { + t.Fatalf("want one statement each, got %d and %d — the case does not exercise the visitor", + len(rep.Statements), len(mod.Statements)) + } + foldPageModeFlags(rep.Statements) + foldPageModeFlags(mod.Statements) + same := reflect.DeepEqual(rep.Statements, mod.Statements) + gate, gated := gatedChange[name] + exempt := createOrReplaceIsNotAnAlias[name] || (gated && !version.v1) - got := deprecationCodes(rep) - if exempt { - if len(got) != 0 { - t.Errorf("exempt kind recorded %v", got) + switch { + case exempt && same: + t.Errorf("exempt, but `or replace` and `or modify` build the same statements: " + + "it is an alias after all — remove the exemption") + case !exempt && !same: + t.Errorf("`or replace` and `or modify` build different statements, so rewriting one "+ + "to the other changes the script — exempt this kind or fix the visitor:\n"+ + " replace: %#v\n modify: %#v", rep.Statements, mod.Statements) } - return - } - if !reflect.DeepEqual(got, []string{deprecation.CreateOrReplace}) { - t.Errorf("recorded %v, want [%s]", got, deprecation.CreateOrReplace) - } else if rep.Deprecations[0].Subject != name { - t.Errorf("subject = %q, want %q", rep.Deprecations[0].Subject, name) - } - if got := deprecationCodes(mod); len(got) != 0 { - t.Errorf("`create or modify` recorded %v", got) - } - }) + + var notes []string + for _, n := range rep.LanguageNotes { + notes = append(notes, n.Code) + } + wantNotes := []string(nil) + if gated && !version.v1 { + wantNotes = []string{gate} + } + if !reflect.DeepEqual(notes, wantNotes) { + t.Errorf("language notes %v, want %v", notes, wantNotes) + } + + got := deprecationCodes(rep) + if exempt { + if len(got) != 0 { + t.Errorf("exempt kind recorded %v", got) + } + return + } + if !reflect.DeepEqual(got, []string{deprecation.CreateOrReplace}) { + t.Errorf("recorded %v, want [%s]", got, deprecation.CreateOrReplace) + } else if want := strings.TrimSuffix(name, " (view)"); rep.Deprecations[0].Subject != want { + t.Errorf("subject = %q, want %q", rep.Deprecations[0].Subject, want) + } + if got := deprecationCodes(mod); len(got) != 0 { + t.Errorf("`create or modify` recorded %v", got) + } + }) + } } } diff --git a/mdl/visitor/visitor_entity.go b/mdl/visitor/visitor_entity.go index 41cbc4b99b..98f23d61ac 100644 --- a/mdl/visitor/visitor_entity.go +++ b/mdl/visitor/visitor_entity.go @@ -150,11 +150,12 @@ func (b *Builder) buildViewEntity(ctx *parser.CreateEntityStatementContext) { createStmt := findParentCreateStatement(ctx) if createStmt != nil { // Check for CREATE OR MODIFY / CREATE OR REPLACE + // `or replace` means `or modify` from mdl 1 on; under mdl 0 it keeps + // its delete-and-recreate meaning (viewEntityReplaceIsModify). if createStmt.OR() != nil { - if createStmt.MODIFY() != nil { + if createStmt.MODIFY() != nil || b.replaceMeansModify(createStmt) { stmt.CreateOrModify = true - } - if createStmt.REPLACE() != nil { + } else if createStmt.REPLACE() != nil { stmt.CreateOrReplace = true } } @@ -675,12 +676,12 @@ func (b *Builder) ExitAlterEntityAction(ctx *parser.AlterEntityActionContext) { case c.NOT_NULL() != nil || (c.NOT() != nil && c.NULL() != nil) || c.REQUIRED() != nil: stmt.ModifyNotNull = &true_ if c.ERROR() != nil && c.STRING_LITERAL() != nil { - stmt.ModifyNotNullError = unquoteString(c.STRING_LITERAL().GetText()) + stmt.ModifyNotNullError = unquoteStringLit(c.STRING_LITERAL()) } case c.UNIQUE() != nil: stmt.ModifyUnique = &true_ if c.ERROR() != nil && c.STRING_LITERAL() != nil { - stmt.ModifyUniqueError = unquoteString(c.STRING_LITERAL().GetText()) + stmt.ModifyUniqueError = unquoteStringLit(c.STRING_LITERAL()) } case c.DEFAULT() != nil: stmt.ModifyHasDefault = true @@ -728,7 +729,7 @@ func (b *Builder) ExitAlterEntityAction(ctx *parser.AlterEntityActionContext) { b.statements = append(b.statements, &ast.AlterEntityStmt{ Name: name, Operation: ast.AlterEntitySetDocumentation, - Documentation: unquoteString(ctx.STRING_LITERAL().GetText()), + Documentation: unquoteStringLit(ctx.STRING_LITERAL()), }) return } @@ -738,7 +739,7 @@ func (b *Builder) ExitAlterEntityAction(ctx *parser.AlterEntityActionContext) { b.statements = append(b.statements, &ast.AlterEntityStmt{ Name: name, Operation: ast.AlterEntitySetComment, - Comment: unquoteString(ctx.STRING_LITERAL().GetText()), + Comment: unquoteStringLit(ctx.STRING_LITERAL()), }) return } @@ -876,7 +877,7 @@ func (b *Builder) ExitDropStatement(ctx *parser.DropStatementContext) { if ctx.CONFIGURATION() != nil { if sl := ctx.STRING_LITERAL(); sl != nil { b.statements = append(b.statements, &ast.DropConfigurationStmt{ - Name: unquoteString(sl.GetText()), + Name: unquoteStringLit(sl), }) } return @@ -898,7 +899,7 @@ func (b *Builder) ExitDropStatement(ctx *parser.DropStatementContext) { stmt.Position = &ast.Position{X: x, Y: y} } } else if lit := ctx.STRING_LITERAL(); lit != nil { - stmt.Title = unquoteString(lit.GetText()) + stmt.Title = unquoteStringLit(lit) } b.statements = append(b.statements, stmt) return @@ -1044,7 +1045,7 @@ func (b *Builder) ExitDropStatement(ctx *parser.DropStatementContext) { Name: buildQualifiedName(names[0]), }) } else if ctx.FOLDER() != nil { - folderPath := unquoteString(ctx.STRING_LITERAL().GetText()) + folderPath := unquoteStringLit(ctx.STRING_LITERAL()) // Module can be a qualifiedName or IDENTIFIER var moduleName string if len(names) > 0 { @@ -1141,7 +1142,7 @@ func (b *Builder) ExitMoveStatement(ctx *parser.MoveStatementContext) { // Parse folder path if specified if len(ctx.AllFOLDER()) > 0 && ctx.STRING_LITERAL() != nil { - stmt.Folder = unquoteString(ctx.STRING_LITERAL().GetText()) + stmt.Folder = unquoteStringLit(ctx.STRING_LITERAL()) } // Parse target module if specified (IN Module or just Module) @@ -1168,7 +1169,7 @@ func (b *Builder) exitMoveFolderStatement(ctx *parser.MoveStatementContext, name // Parse target: either FOLDER 'path' [IN Module] or just Module if ctx.STRING_LITERAL() != nil { // MOVE FOLDER ... TO FOLDER 'path' [IN Module] - stmt.TargetFolder = unquoteString(ctx.STRING_LITERAL().GetText()) + stmt.TargetFolder = unquoteStringLit(ctx.STRING_LITERAL()) if len(names) > 1 { stmt.TargetModule = getQualifiedNameText(names[1]) } else if ctx.IDENTIFIER() != nil { diff --git a/mdl/visitor/visitor_enumeration.go b/mdl/visitor/visitor_enumeration.go index 19bace1aa1..5207027140 100644 --- a/mdl/visitor/visitor_enumeration.go +++ b/mdl/visitor/visitor_enumeration.go @@ -19,7 +19,7 @@ func (b *Builder) ExitCreateEnumerationStatement(ctx *parser.CreateEnumerationSt for _, opt := range optsCtx.AllEnumerationOption() { optCtx := opt.(*parser.EnumerationOptionContext) if optCtx.FOLDER() != nil && optCtx.STRING_LITERAL() != nil { - stmt.Folder = unquoteString(optCtx.STRING_LITERAL().GetText()) + stmt.Folder = unquoteStringLit(optCtx.STRING_LITERAL()) } } } @@ -54,7 +54,7 @@ func (b *Builder) ExitAlterEnumerationAction(ctx *parser.AlterEnumerationActionC if ctx.ADD() != nil && len(ids) >= 1 { caption := "" if ctx.STRING_LITERAL() != nil { - caption = unquoteString(ctx.STRING_LITERAL().GetText()) + caption = unquoteStringLit(ctx.STRING_LITERAL()) } b.statements = append(b.statements, &ast.AlterEnumerationStmt{ Name: name, @@ -82,7 +82,7 @@ func (b *Builder) ExitAlterEnumerationAction(ctx *parser.AlterEnumerationActionC Name: name, Operation: ast.AlterEnumModifyCaption, ValueName: ids[0].GetText(), - Caption: unquoteString(ctx.STRING_LITERAL().GetText()), + Caption: unquoteStringLit(ctx.STRING_LITERAL()), }) } } @@ -114,12 +114,12 @@ func (b *Builder) ExitCreateConstantStatement(ctx *parser.CreateConstantStatemen for _, opt := range optsCtx.AllConstantOption() { optCtx := opt.(*parser.ConstantOptionContext) if optCtx.COMMENT() != nil && optCtx.STRING_LITERAL() != nil { - stmt.Comment = unquoteString(optCtx.STRING_LITERAL().GetText()) + stmt.Comment = unquoteStringLit(optCtx.STRING_LITERAL()) } if optCtx.FOLDER() != nil && optCtx.STRING_LITERAL() != nil { - stmt.Folder = unquoteString(optCtx.STRING_LITERAL().GetText()) + stmt.Folder = unquoteStringLit(optCtx.STRING_LITERAL()) } else if optCtx.FOLDER() != nil && optCtx.STRING_LITERAL() != nil { - stmt.Folder = unquoteString(optCtx.STRING_LITERAL().GetText()) + stmt.Folder = unquoteStringLit(optCtx.STRING_LITERAL()) } else if optCtx.EXPOSED() != nil { stmt.ExposedToClient = true } diff --git a/mdl/visitor/visitor_helpers.go b/mdl/visitor/visitor_helpers.go index a7c49b09c0..4a3a52f571 100644 --- a/mdl/visitor/visitor_helpers.go +++ b/mdl/visitor/visitor_helpers.go @@ -145,7 +145,7 @@ func buildEnumValues(ctx parser.IEnumerationValueListContext, b *Builder) []ast. enumVal.Documentation = extractDocComment(docCtx.GetText()) } if ev.STRING_LITERAL() != nil { - enumVal.Caption = unquoteString(ev.STRING_LITERAL().GetText()) + enumVal.Caption = unquoteStringLit(ev.STRING_LITERAL()) } values = append(values, enumVal) } @@ -192,14 +192,14 @@ func buildAttributes(ctx parser.IAttributeDefinitionListContext, b *Builder) []a attr.NotNull = true // Extract error message if present if c.ERROR() != nil && c.STRING_LITERAL() != nil { - attr.NotNullError = unquoteString(c.STRING_LITERAL().GetText()) + attr.NotNullError = unquoteStringLit(c.STRING_LITERAL()) } } if c.UNIQUE() != nil { attr.Unique = true // Extract error message if present if c.ERROR() != nil && c.STRING_LITERAL() != nil { - attr.UniqueError = unquoteString(c.STRING_LITERAL().GetText()) + attr.UniqueError = unquoteStringLit(c.STRING_LITERAL()) } } if c.DEFAULT() != nil { @@ -215,7 +215,7 @@ func buildAttributes(ctx parser.IAttributeDefinitionListContext, b *Builder) []a attr.NotNull = true // Extract error message if present if c.ERROR() != nil && c.STRING_LITERAL() != nil { - attr.NotNullError = unquoteString(c.STRING_LITERAL().GetText()) + attr.NotNullError = unquoteStringLit(c.STRING_LITERAL()) } } if c.CALCULATED() != nil { @@ -253,13 +253,13 @@ func buildSingleAttribute(a *parser.AttributeDefinitionContext) *ast.Attribute { if c.NOT() != nil && c.NULL() != nil || c.NOT_NULL() != nil { attr.NotNull = true if c.ERROR() != nil && c.STRING_LITERAL() != nil { - attr.NotNullError = unquoteString(c.STRING_LITERAL().GetText()) + attr.NotNullError = unquoteStringLit(c.STRING_LITERAL()) } } if c.UNIQUE() != nil { attr.Unique = true if c.ERROR() != nil && c.STRING_LITERAL() != nil { - attr.UniqueError = unquoteString(c.STRING_LITERAL().GetText()) + attr.UniqueError = unquoteStringLit(c.STRING_LITERAL()) } } if c.DEFAULT() != nil { @@ -273,7 +273,7 @@ func buildSingleAttribute(a *parser.AttributeDefinitionContext) *ast.Attribute { if c.REQUIRED() != nil { attr.NotNull = true if c.ERROR() != nil && c.STRING_LITERAL() != nil { - attr.NotNullError = unquoteString(c.STRING_LITERAL().GetText()) + attr.NotNullError = unquoteStringLit(c.STRING_LITERAL()) } } if c.CALCULATED() != nil { @@ -522,7 +522,7 @@ func extractLiteralValue(ctx parser.ILiteralContext) any { // Check for different literal types if lit.STRING_LITERAL() != nil { - return unquoteString(lit.STRING_LITERAL().GetText()) + return unquoteStringLit(lit.STRING_LITERAL()) } if lit.NUMBER_LITERAL() != nil { text := lit.NUMBER_LITERAL().GetText() @@ -765,7 +765,7 @@ func buildErrorMessage(ctx parser.IErrorMessageClauseContext) string { if !ok || emc.STRING_LITERAL() == nil { return "" } - return unquoteString(emc.STRING_LITERAL().GetText()) + return unquoteStringLit(emc.STRING_LITERAL()) } // expressionSourceText is an expression as the author wrote it — whitespace kept, diff --git a/mdl/visitor/visitor_imagecollection.go b/mdl/visitor/visitor_imagecollection.go index c39f5cc72e..59c5cd4582 100644 --- a/mdl/visitor/visitor_imagecollection.go +++ b/mdl/visitor/visitor_imagecollection.go @@ -14,7 +14,7 @@ func (b *Builder) ExitCreateImageCollectionStatement(ctx *parser.CreateImageColl ExportLevel: "Hidden", } if lit := ctx.STRING_LITERAL(); lit != nil { - stmt.Folder = unquoteString(lit.GetText()) + stmt.Folder = unquoteStringLit(lit) } // Extract /** ... */ doc comment (same as other create statements) @@ -25,10 +25,10 @@ func (b *Builder) ExitCreateImageCollectionStatement(ctx *parser.CreateImageColl for _, opt := range optsCtx.AllImageCollectionOption() { optCtx := opt.(*parser.ImageCollectionOptionContext) if optCtx.EXPORT() != nil && optCtx.LEVEL() != nil && optCtx.STRING_LITERAL() != nil { - stmt.ExportLevel = unquoteString(optCtx.STRING_LITERAL().GetText()) + stmt.ExportLevel = unquoteStringLit(optCtx.STRING_LITERAL()) } if optCtx.COMMENT() != nil && optCtx.STRING_LITERAL() != nil { - stmt.Comment = unquoteString(optCtx.STRING_LITERAL().GetText()) + stmt.Comment = unquoteStringLit(optCtx.STRING_LITERAL()) } } } @@ -44,7 +44,7 @@ func (b *Builder) ExitCreateImageCollectionStatement(ctx *parser.CreateImageColl } stmt.Images = append(stmt.Images, ast.ImageItem{ Name: name, - FilePath: unquoteString(itemCtx.GetPath().GetText()), + FilePath: unquoteStringLit(itemCtx.GetPath()), }) } } diff --git a/mdl/visitor/visitor_import_export_mapping.go b/mdl/visitor/visitor_import_export_mapping.go index ea51cc9ce4..d378ceb58d 100644 --- a/mdl/visitor/visitor_import_export_mapping.go +++ b/mdl/visitor/visitor_import_export_mapping.go @@ -17,7 +17,7 @@ func (b *Builder) ExitCreateImportMappingStatement(ctx *parser.CreateImportMappi Name: buildQualifiedName(ctx.QualifiedName()), } if lit := ctx.STRING_LITERAL(); lit != nil { - stmt.Folder = unquoteString(lit.GetText()) + stmt.Folder = unquoteStringLit(lit) } // Parse WITH clause @@ -157,7 +157,7 @@ func (b *Builder) ExitCreateExportMappingStatement(ctx *parser.CreateExportMappi Name: buildQualifiedName(ctx.QualifiedName()), } if lit := ctx.STRING_LITERAL(); lit != nil { - stmt.Folder = unquoteString(lit.GetText()) + stmt.Folder = unquoteStringLit(lit) } // Parse WITH clause diff --git a/mdl/visitor/visitor_javaaction.go b/mdl/visitor/visitor_javaaction.go index a6be0a27d3..95a3749db3 100644 --- a/mdl/visitor/visitor_javaaction.go +++ b/mdl/visitor/visitor_javaaction.go @@ -18,7 +18,7 @@ func (b *Builder) ExitCreateJavaActionStatement(ctx *parser.CreateJavaActionStat stmt.Name = buildQualifiedName(qn) } if lit := ctx.STRING_LITERAL(); lit != nil { - stmt.Folder = unquoteString(lit.GetText()) + stmt.Folder = unquoteStringLit(lit) } // Get parameters @@ -71,8 +71,8 @@ func (b *Builder) ExitCreateJavaActionStatement(ctx *parser.CreateJavaActionStat // it, so removal has to be said out loud. stmt.NotExposed = true case len(allStrings) >= 2: - stmt.ExposedCaption = unquoteString(allStrings[0].GetText()) - stmt.ExposedCategory = unquoteString(allStrings[1].GetText()) + stmt.ExposedCaption = unquoteStringLit(allStrings[0]) + stmt.ExposedCategory = unquoteStringLit(allStrings[1]) stmt.ExposedBitmaps = buildExposeBitmaps(exposed.AllExposeBitmapClause()) } } @@ -123,7 +123,7 @@ func (b *Builder) ExitCreateJavaScriptActionStatement(ctx *parser.CreateJavaScri stmt.Name = buildQualifiedName(qn) } if lit := ctx.STRING_LITERAL(); lit != nil { - stmt.Folder = unquoteString(lit.GetText()) + stmt.Folder = unquoteStringLit(lit) } if paramList := ctx.JavaActionParameterList(); paramList != nil { @@ -172,8 +172,8 @@ func (b *Builder) ExitCreateJavaScriptActionStatement(ctx *parser.CreateJavaScri // it, so removal has to be said out loud. stmt.NotExposed = true case len(allStrings) >= 2: - stmt.ExposedCaption = unquoteString(allStrings[0].GetText()) - stmt.ExposedCategory = unquoteString(allStrings[1].GetText()) + stmt.ExposedCaption = unquoteStringLit(allStrings[0]) + stmt.ExposedCategory = unquoteStringLit(allStrings[1]) stmt.ExposedBitmaps = buildExposeBitmaps(exposed.AllExposeBitmapClause()) } } diff --git a/mdl/visitor/visitor_jsonstructure.go b/mdl/visitor/visitor_jsonstructure.go index 0a3f5bb453..1f789701d5 100644 --- a/mdl/visitor/visitor_jsonstructure.go +++ b/mdl/visitor/visitor_jsonstructure.go @@ -20,11 +20,11 @@ func (b *Builder) ExitCreateJsonStructureStatement(ctx *parser.CreateJsonStructu allStrings := ctx.AllSTRING_LITERAL() strIdx := 0 if ctx.FOLDER() != nil && strIdx < len(allStrings) { - stmt.Folder = unquoteString(allStrings[strIdx].GetText()) + stmt.Folder = unquoteStringLit(allStrings[strIdx]) strIdx++ } if ctx.COMMENT() != nil && strIdx < len(allStrings) { - stmt.Documentation = unquoteString(allStrings[strIdx].GetText()) + stmt.Documentation = unquoteStringLit(allStrings[strIdx]) } // Parse SNIPPET value — can be STRING_LITERAL or DOLLAR_STRING. @@ -34,7 +34,7 @@ func (b *Builder) ExitCreateJsonStructureStatement(ctx *parser.CreateJsonStructu // SNIPPET is a STRING_LITERAL; it's the last one (after COMMENT if present) idx := len(allStrings) - 1 if idx >= 0 { - stmt.JsonSnippet = unquoteString(allStrings[idx].GetText()) + stmt.JsonSnippet = unquoteStringLit(allStrings[idx]) } } @@ -48,8 +48,8 @@ func (b *Builder) ExitCreateJsonStructureStatement(ctx *parser.CreateJsonStructu if len(strings) != 2 { continue } - jsonKey := unquoteString(strings[0].GetText()) - customName := unquoteString(strings[1].GetText()) + jsonKey := unquoteStringLit(strings[0]) + customName := unquoteStringLit(strings[1]) // `item of 'key' as 'Name'` names the array's ITEM element; the bare // form names the element the key itself reaches. if mappingCtx.ITEM() != nil { diff --git a/mdl/visitor/visitor_list_activities.go b/mdl/visitor/visitor_list_activities.go new file mode 100644 index 0000000000..61f868b5a0 --- /dev/null +++ b/mdl/visitor/visitor_list_activities.go @@ -0,0 +1,402 @@ +// SPDX-License-Identifier: Apache-2.0 + +package visitor + +import ( + "fmt" + "strconv" + "strings" + + "github.com/antlr4-go/antlr/v4" + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/deprecation" + "github.com/mendixlabs/mxcli/mdl/grammar/parser" + "github.com/mendixlabs/mxcli/mdl/langver" +) + +// List operations and aggregates are statements that mirror Studio Pro's +// activities: one "List operation" or "Aggregate list" activity per statement, +// named after the operation and taking the inputs its dialog asks for +// (PROPOSAL_mdl_beta_syntax_freeze.md §4 and §5 item 3, #733): +// +// $Open = filter $Orders by Status = Shop.Status.Open; +// $N = count $Open; +// +// The operand is always a variable, so one activity cannot be nested inside +// another. The call form (`filter($Orders, …)`) keeps parsing: +// +// - for every operation except find and contains it is a respelling, a +// deprecated alias (MDL-DEPR003 / MDL-DEPR004) that builds the same AST; +// - `find(…)` and `contains(…)` are also Mendix's string functions, and which +// one a call means is guessed today from its arguments. That is not a +// respelling, so it is version-gated (listCallForm): kept and warned under +// mdl 0, refused under mdl 1; +// - so is a call after `set`, or nested in another call: under mdl 0 it is +// turned into an activity (and a nested operand is refused by MDL-LISTOP02), +// under mdl 1 `set` always assigns an expression and such a call is refused. +// +// `set` is mandatory for reassignment under mdl 1 (setIsMandatory), which is +// what removes the ambiguity: `$x = …` without it is only ever an activity. + +// listCallForm is the change of meaning for list operations and aggregates +// written as calls where the call is not a plain respelling of one activity. +var listCallForm = langver.Change{ + Code: "MDL-V1-LIST", + Since: langver.V1, + Old: "a list operation or aggregate written as a call — `find(…)` or `contains(…)`, " + + "a call after `set`, or one call nested in another — is turned into a List operation " + + "or Aggregate list activity, guessing from its arguments whether a string function was meant,", + New: "an error: a List operation or Aggregate list is one statement per activity whose operand is " + + "a variable (`$x = find $L where …;`, `$n = count $L;`), and after `set` a call is always " + + "the expression function", +} + +// setIsMandatory is the new rejection of a reassignment without `set`. +var setIsMandatory = langver.Change{ + Code: "MDL-V1-SET", + Since: langver.V1, + Old: "`$x = <expression>` without `set` changes the variable", + New: "an error: a reassignment is written `set $x = <expression>;`", +} + +// scriptLanguageVersion reads the `mdl <n>;` header of the script that tree is +// part of. The header can only be the first statement, so it is on the root. +// +// It exists for the statement builders, which are free functions without the +// Builder: they must build the AST of the version the script is written in. +// A missing or malformed header is mdl 0; a malformed one is reported by +// ExitLanguageHeader. +func scriptLanguageVersion(tree antlr.Tree) langver.Version { + for tree != nil { + if prog, ok := tree.(*parser.ProgramContext); ok { + h, ok := prog.LanguageHeader().(*parser.LanguageHeaderContext) + if !ok || h == nil || h.IDENTIFIER() == nil || h.NUMBER_LITERAL() == nil || + !strings.EqualFold(h.IDENTIFIER().GetText(), "mdl") { + return langver.V0 + } + n, err := strconv.Atoi(h.NUMBER_LITERAL().GetText()) + if v := langver.Version(n); err == nil && v.Known() { + return v + } + return langver.V0 + } + tree = tree.GetParent() + } + return langver.V0 +} + +// buildListOperationActivity fills stmt from the statement form. +func buildListOperationActivity(act *parser.ListOperationActivityContext, stmt *ast.ListOperationStmt) { + vars := act.AllVARIABLE() + varName := func(i int) string { + if i < len(vars) { + return strings.TrimPrefix(vars[i].GetText(), "$") + } + return "" + } + stmt.InputVariable = varName(0) + condition := func() { + c, ok := act.ListOperationCondition().(*parser.ListOperationConditionContext) + if !ok || c == nil { + return + } + stmt.Condition = buildSourceExpression(c.Expression()) + stmt.ByExpression = c.WHERE() != nil + } + switch { + case act.HEAD() != nil: + stmt.Operation = ast.ListOpHead + case act.TAIL() != nil: + stmt.Operation = ast.ListOpTail + case act.FIND() != nil: + stmt.Operation = ast.ListOpFind + condition() + case act.FILTER() != nil: + stmt.Operation = ast.ListOpFilter + condition() + case act.SORT() != nil: + stmt.Operation = ast.ListOpSort + for _, item := range act.AllListSortItem() { + it := item.(*parser.ListSortItemContext) + stmt.SortSpecs = append(stmt.SortSpecs, ast.SortSpec{ + Attribute: identifierOrKeywordText(it.IdentifierOrKeyword()), + Ascending: it.DESC() == nil, + }) + } + case act.UNION() != nil: + stmt.Operation, stmt.SecondVariable = ast.ListOpUnion, varName(1) + case act.INTERSECT() != nil: + stmt.Operation, stmt.SecondVariable = ast.ListOpIntersect, varName(1) + case act.SUBTRACT() != nil: + // subtract $B from $A: the result is A minus B, and A is the first list. + stmt.Operation, stmt.InputVariable, stmt.SecondVariable = ast.ListOpSubtract, varName(1), varName(0) + case act.CONTAINS() != nil: + // contains $Object in $List: the list is the activity's first operand. + stmt.Operation, stmt.InputVariable, stmt.SecondVariable = ast.ListOpContains, varName(1), varName(0) + case act.EQUALS_OP() != nil: + stmt.Operation, stmt.SecondVariable = ast.ListOpEquals, varName(1) + case act.RANGE() != nil: + stmt.Operation = ast.ListOpRange + if act.OFFSET() != nil { + stmt.OffsetExpr = buildSourceExpression(act.Expression(0)) + } + if act.LIMIT() != nil { + stmt.LimitExpr = buildSourceExpression(act.Expression(len(act.AllExpression()) - 1)) + } + } +} + +// buildAggregateListActivity fills stmt from the statement form. +func buildAggregateListActivity(act *parser.AggregateListActivityContext, stmt *ast.AggregateListStmt) { + if v := act.VARIABLE(); v != nil { + stmt.InputVariable = strings.TrimPrefix(v.GetText(), "$") + } + exprs := act.AllExpression() + switch { + case act.COUNT() != nil: + stmt.Operation = ast.AggregateCount + return + case act.SUM() != nil: + stmt.Operation = ast.AggregateSum + case act.AVERAGE() != nil: + stmt.Operation = ast.AggregateAverage + case act.MINIMUM() != nil: + stmt.Operation = ast.AggregateMinimum + case act.MAXIMUM() != nil: + stmt.Operation = ast.AggregateMaximum + case act.ALL() != nil, act.ANY() != nil: + stmt.Operation = ast.AggregateAll + if act.ANY() != nil { + stmt.Operation = ast.AggregateAny + } + stmt.IsExpression = true + if len(exprs) > 0 { + stmt.Expression = buildSourceExpression(exprs[0]) + } + return + case act.REDUCE() != nil: + // reduce $L from <initial> as <type> using <expression> + stmt.Operation = ast.AggregateReduce + stmt.IsExpression = true + if len(exprs) > 0 { + stmt.InitialValue = buildSourceExpression(exprs[0]) + } + if len(exprs) > 1 { + stmt.Expression = buildSourceExpression(exprs[1]) + } + if dt := act.DataType(); dt != nil { + t := buildDataType(dt) + stmt.ReturnType = &t + } + return + } + // sum / average / minimum / maximum: `by Attribute` or `of <expression>`. + if act.OF() != nil && len(exprs) > 0 { + stmt.IsExpression = true + stmt.Expression = buildSourceExpression(exprs[0]) + } else if id := act.IdentifierOrKeyword(); id != nil { + stmt.Attribute = identifierOrKeywordText(id) + } +} + +// unwrapSource returns the expression a SourceExpr carries. +func unwrapSource(e ast.Expression) ast.Expression { + if s, ok := e.(*ast.SourceExpr); ok { + return s.Expression + } + return e +} + +// listCallName returns the upper-cased name of a list-operation or aggregate +// call, or "" when value is not one. +func listCallName(value ast.Expression) string { + call, ok := unwrapSource(value).(*ast.FunctionCallExpr) + if !ok { + return "" + } + switch name := strings.ToUpper(call.Name); name { + case "HEAD", "TAIL", "FIND", "FILTER", "SORT", "UNION", "INTERSECT", "SUBTRACT", + "CONTAINS", "EQUALS", "RANGE", "COUNT", "SUM", "AVERAGE", "MINIMUM", "MAXIMUM": + return name + } + return "" +} + +func isStringOverload(name string) bool { return name == "FIND" || name == "CONTAINS" } + +// ExitListOperationStatement reports the call form: a deprecated alias, or for +// find/contains a version-gated construct; and checks that `by` names a member. +func (b *Builder) ExitListOperationStatement(ctx *parser.ListOperationStatementContext) { + target := "" + if v := ctx.VARIABLE(); v != nil { + target = v.GetText() + } + if op, ok := ctx.ListOperation().(*parser.ListOperationContext); ok && op != nil { + byExpression := buildListOperationStatement(ctx).ByExpression + if op.FIND() != nil || op.CONTAINS() != nil { + if b.gate(listCallForm, ctx) { + b.addError(findContainsCallError(ctx.GetStart().GetLine(), target, op.FIND() != nil)) + return + } + // Under mdl 0 the flow builder turns a String operand into the + // string function: `set` says so under mdl 1. Any other operand + // is the List operation, the statement form. + fn := strings.ToLower(op.GetStart().GetText()) + switch isString, known := operandKind(ctx, op.VARIABLE(0).GetText()); { + case !known: + b.fixLastNote(listCallForm.Code, nil, operandKindUnknown(op.VARIABLE(0).GetText(), fn)) + case isString: + b.fixLastNote(listCallForm.Code, &ast.Fix{Edits: []ast.TextEdit{insertAt(ctx.GetStart().GetStart(), "set ")}}, "") + default: + fix, why := fixOrReason(callFormFix(op, byExpression)) + b.fixLastNote(listCallForm.Code, fix, why) + } + return + } + b.recordDeprecation(deprecation.ListOperationFunctionForm, op.GetStart(), strings.ToLower(op.GetStart().GetText())) + fix, why := fixOrReason(callFormFix(op, byExpression)) + b.fixLastDeprecation(deprecation.ListOperationFunctionForm, fix, why) + return + } + act, ok := ctx.ListOperationActivity().(*parser.ListOperationActivityContext) + if !ok || act == nil { + return + } + c, ok := act.ListOperationCondition().(*parser.ListOperationConditionContext) + if !ok || c == nil || c.BY() == nil { + return + } + if !ast.IsMemberEquality(unwrapSource(buildExpression(c.Expression()))) { + b.addError(fmt.Errorf("line %d: `by` names a member and the value it must have, `by Member = value`; "+ + "`%s` is not of that shape. For any other condition write `where <expression>`, which is "+ + "Studio Pro's \"by expression\" operation and uses $currentObject", + c.GetStart().GetLine(), strings.TrimSpace(extractExpressionText(c.Expression())))) + } +} + +func findContainsCallError(line int, target string, find bool) error { + if find { + return fmt.Errorf("line %d: `%s = find(…)` is refused under mdl 1: the call is both the string function "+ + "and the List operation, one statement per activity. For the string function write "+ + "`set %s = find(…);` (the variable is declared first); for the List operation write "+ + "`%s = find $List by Member = value;` or `%s = find $List where <expression>;`", + line, target, target, target, target) + } + return fmt.Errorf("line %d: `%s = contains(…)` is refused under mdl 1: the call is both the string function "+ + "and the List operation, one statement per activity. For the string function write "+ + "`set %s = contains(…);` (the variable is declared first); for the List operation write "+ + "`%s = contains $Object in $List;`", line, target, target, target) +} + +// ExitAggregateListStatement reports the call form, a deprecated alias. +func (b *Builder) ExitAggregateListStatement(ctx *parser.AggregateListStatementContext) { + if op, ok := ctx.ListAggregateOperation().(*parser.ListAggregateOperationContext); ok && op != nil { + b.recordDeprecation(deprecation.AggregateFunctionForm, op.GetStart(), strings.ToLower(op.GetStart().GetText())) + fix, why := fixOrReason(callFormFix(op, false)) + b.fixLastDeprecation(deprecation.AggregateFunctionForm, fix, why) + } +} + +// ExitSetStatement gates the two constructs whose meaning mdl 1 changes: an +// assignment without `set`, and a list-operation or aggregate call as the value. +func (b *Builder) ExitSetStatement(ctx *parser.SetStatementContext) { + v := ctx.VARIABLE() + if v == nil || ctx.Expression() == nil { + // An attribute target (`$o/A = v`) is a respelling of `change`, not a + // reassignment; it is left to that alias. + return + } + target := v.GetText() + line := ctx.GetStart().GetLine() + value := buildExpression(ctx.Expression()) + name := listCallName(value) + valueText := strings.TrimSpace(extractExpressionText(ctx.Expression())) + + switch { + case name != "" && !(isStringOverload(name) && ctx.SET() != nil): + // A call the mdl 0 builder turns into an activity (when it does). + if call, ok := unwrapSource(value).(*ast.FunctionCallExpr); ok && + buildListOrAggregateStatement(strings.TrimPrefix(target, "$"), call) == nil { + // The string reading of find/contains: an expression either way. + break + } + if b.gate(listCallForm, ctx) { + if isStringOverload(name) { + b.addError(findContainsCallError(line, target, name == "FIND")) + return + } + b.addError(fmt.Errorf("line %d: `%s = %s` is refused under mdl 1: a list operation or aggregate is "+ + "one statement per activity, and its operand is a variable, never another call or an "+ + "expression. Write `%s = %s $List …;`, with each inner call as a statement of its own "+ + "(e.g. `$Open = filter $Orders where …;` then `$N = count $Open;`)", + line, target, valueText, target, strings.ToLower(name))) + return + } + fix, why := setCallFix(ctx, value, isStringOverload(name)) + b.fixLastNote(listCallForm.Code, fix, why) + return + case name != "" && ctx.SET() != nil: + // `set $x = find(…)` / `contains(…)`: under mdl 1 always the string + // function; under mdl 0 an activity when the arguments look like one. + if call, ok := unwrapSource(value).(*ast.FunctionCallExpr); ok && + buildListOrAggregateStatement(strings.TrimPrefix(target, "$"), call) != nil && !b.gate(listCallForm, ctx) { + fix, why := setCallFix(ctx, value, true) + b.fixLastNote(listCallForm.Code, fix, why) + } + return + } + if ctx.SET() == nil { + if b.gate(setIsMandatory, ctx) { + b.addError(fmt.Errorf("line %d: a reassignment says `set` under mdl 1: write `set %s = %s;`. "+ + "A List operation or Aggregate list activity is written without it, as its own statement "+ + "(`%s = filter $List where …;`)", line, target, valueText, target)) + return + } + b.fixLastNote(setIsMandatory.Code, &ast.Fix{Edits: []ast.TextEdit{insertAt(ctx.GetStart().GetStart(), "set ")}}, "") + } +} + +// setCallFix is the rewrite of an assignment whose value is a list-operation or +// aggregate call that mdl 0 turns into an activity (MDL-V1-LIST): the statement +// form, without `set`. For find and contains the activity is what mdl 0 builds +// only when the operand is not a String; for a String it is the string +// function, which is what `set $x = find(…)` means under mdl 1, so a script +// already written that way needs no edit. +func setCallFix(ctx *parser.SetStatementContext, value ast.Expression, overloaded bool) (*ast.Fix, string) { + op := singleListCall(ctx.Expression()) + if op == nil { + return nil, "the operand is not a variable (a nested call or an expression), and one activity takes a " + + "variable: write each inner call as a statement of its own" + } + if overloaded { + lo, ok := op.(*parser.ListOperationContext) + if !ok || lo.VARIABLE(0) == nil { + return nil, "the call has no list operand" + } + v, fn := lo.VARIABLE(0), strings.ToLower(op.GetStart().GetText()) + isString, known := operandKind(ctx, v.GetText()) + if !known { + return nil, operandKindUnknown(v.GetText(), fn) + } + if isString { + if ctx.SET() == nil { + return &ast.Fix{Edits: []ast.TextEdit{insertAt(ctx.GetStart().GetStart(), "set ")}}, "" + } + return &ast.Fix{}, "" + } + } + byExpression := false + if call, ok := unwrapSource(value).(*ast.FunctionCallExpr); ok { + if lo, ok := buildListOrAggregateStatement(strings.TrimPrefix(ctx.VARIABLE().GetText(), "$"), call).(*ast.ListOperationStmt); ok { + byExpression = lo.ByExpression + } + } + edits, why := callFormFix(op, byExpression) + if why != "" { + return nil, why + } + if set := ctx.SET(); set != nil { + edits = append(edits, ast.TextEdit{Start: set.GetSymbol().GetStart(), Stop: ctx.VARIABLE().GetSymbol().GetStart()}) + } + return &ast.Fix{Edits: edits}, "" +} diff --git a/mdl/visitor/visitor_list_activity_test.go b/mdl/visitor/visitor_list_activity_test.go new file mode 100644 index 0000000000..2d1ade9f93 --- /dev/null +++ b/mdl/visitor/visitor_list_activity_test.go @@ -0,0 +1,359 @@ +// SPDX-License-Identifier: Apache-2.0 + +package visitor + +import ( + "reflect" + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/deprecation" +) + +// List operations and aggregates are statements that mirror Studio Pro's +// activities (PROPOSAL_mdl_beta_syntax_freeze.md §4, ADR-0010 R2; #733). + +const listActivityParams = "$Orders: List of M.Order, $A: List of M.Order, $B: List of M.Order, " + + "$Order: M.Order, $Number: Integer, $S: String" + +// listActivityMicroflow wraps body in a microflow whose parameters every case +// below can name. +func listActivityMicroflow(header, body string) string { + return header + "create microflow M.F (" + listActivityParams + ")\nbegin\n " + body + "\nend;" +} + +// buildListActivity parses one statement inside a microflow and returns the +// program and the microflow's first body statement. +func buildListActivity(t *testing.T, header, body string) (*ast.Program, ast.MicroflowStatement, []error) { + t.Helper() + prog, errs := Build(listActivityMicroflow(header, body)) + if prog == nil || len(prog.Statements) == 0 { + return prog, nil, errs + } + mf, ok := prog.Statements[0].(*ast.CreateMicroflowStmt) + if !ok || len(mf.Body) == 0 { + return prog, nil, errs + } + return prog, mf.Body[0], errs +} + +func noteCodes(prog *ast.Program) []string { + var out []string + for _, n := range prog.LanguageNotes { + out = append(out, n.Code) + } + return out +} + +// Every row of the §4 table parses under both versions and builds the activity +// it names, with no deprecation and no language note. +func TestListActivityStatementsBuild(t *testing.T) { + cases := []struct { + src string + check func(t *testing.T, s ast.MicroflowStatement) + }{ + {"$Open = filter $Orders by Status = M.Status.Open;", func(t *testing.T, s ast.MicroflowStatement) { + lo := s.(*ast.ListOperationStmt) + if lo.Operation != ast.ListOpFilter || lo.InputVariable != "Orders" || lo.ByExpression || !ast.IsMemberEquality(lo.Condition) { + t.Errorf("got %#v", lo) + } + }}, + {"$Big = filter $Orders where $currentObject/Total > 1000;", func(t *testing.T, s ast.MicroflowStatement) { + lo := s.(*ast.ListOperationStmt) + if lo.Operation != ast.ListOpFilter || !lo.ByExpression || lo.Condition == nil { + t.Errorf("got %#v", lo) + } + }}, + {"$Match = find $Orders by Number = $Number;", func(t *testing.T, s ast.MicroflowStatement) { + lo := s.(*ast.ListOperationStmt) + if lo.Operation != ast.ListOpFind || lo.ByExpression || lo.OutputVariable != "Match" { + t.Errorf("got %#v", lo) + } + }}, + {"$Late = find $Orders where $currentObject/Number < 3;", func(t *testing.T, s ast.MicroflowStatement) { + lo := s.(*ast.ListOperationStmt) + if lo.Operation != ast.ListOpFind || !lo.ByExpression { + t.Errorf("got %#v", lo) + } + }}, + {"$Sorted = sort $Orders by Date desc, Number asc, Count;", func(t *testing.T, s ast.MicroflowStatement) { + lo := s.(*ast.ListOperationStmt) + want := []ast.SortSpec{{Attribute: "Date"}, {Attribute: "Number", Ascending: true}, {Attribute: "Count", Ascending: true}} + if lo.Operation != ast.ListOpSort || !reflect.DeepEqual(lo.SortSpecs, want) { + t.Errorf("got %#v", lo) + } + }}, + {"$First = head $Orders;", func(t *testing.T, s ast.MicroflowStatement) { + lo := s.(*ast.ListOperationStmt) + if lo.Operation != ast.ListOpHead || lo.InputVariable != "Orders" { + t.Errorf("got %#v", lo) + } + }}, + {"$Rest = tail $Orders;", func(t *testing.T, s ast.MicroflowStatement) { + if lo := s.(*ast.ListOperationStmt); lo.Operation != ast.ListOpTail { + t.Errorf("got %#v", lo) + } + }}, + {"$Page = range $Orders offset 20 limit 10;", func(t *testing.T, s ast.MicroflowStatement) { + lo := s.(*ast.ListOperationStmt) + if lo.Operation != ast.ListOpRange || lo.OffsetExpr == nil || lo.LimitExpr == nil { + t.Errorf("got %#v", lo) + } + }}, + {"$All = union $A with $B;", listPair(ast.ListOpUnion, "A", "B")}, + {"$Both = intersect $A with $B;", listPair(ast.ListOpIntersect, "A", "B")}, + // subtract $B from $A is A minus B: the first list is A. + {"$Left = subtract $B from $A;", listPair(ast.ListOpSubtract, "A", "B")}, + // contains $Order in $Orders: the list is the first operand. + {"$Has = contains $Order in $Orders;", listPair(ast.ListOpContains, "Orders", "Order")}, + {"$Same = equals $A and $B;", listPair(ast.ListOpEquals, "A", "B")}, + {"$N = count $Orders;", func(t *testing.T, s ast.MicroflowStatement) { + ag := s.(*ast.AggregateListStmt) + if ag.Operation != ast.AggregateCount || ag.InputVariable != "Orders" { + t.Errorf("got %#v", ag) + } + }}, + {"$Total = sum $Orders by Amount;", func(t *testing.T, s ast.MicroflowStatement) { + ag := s.(*ast.AggregateListStmt) + if ag.Operation != ast.AggregateSum || ag.Attribute != "Amount" || ag.IsExpression || ag.InputVariable != "Orders" { + t.Errorf("got %#v", ag) + } + }}, + {"$Total = sum $Orders of $currentObject/Price * $currentObject/Quantity;", func(t *testing.T, s ast.MicroflowStatement) { + ag := s.(*ast.AggregateListStmt) + if ag.Operation != ast.AggregateSum || !ag.IsExpression || ag.Expression == nil { + t.Errorf("got %#v", ag) + } + }}, + {"$Avg = average $Orders by Amount;", aggregateOp(ast.AggregateAverage)}, + {"$Min = minimum $Orders by Amount;", aggregateOp(ast.AggregateMinimum)}, + {"$Max = maximum $Orders of $currentObject/Amount;", aggregateOp(ast.AggregateMaximum)}, + {"$AllPaid = all $Orders where $currentObject/Paid;", aggregateOp(ast.AggregateAll)}, + {"$AnyLate = any $Orders where $currentObject/Late;", aggregateOp(ast.AggregateAny)}, + {"$Csv = reduce $Orders from '' as String using $currentResult + $currentObject/Name;", func(t *testing.T, s ast.MicroflowStatement) { + ag := s.(*ast.AggregateListStmt) + if ag.Operation != ast.AggregateReduce || ag.InitialValue == nil || ag.ReturnType == nil || + ag.ReturnType.Kind != ast.TypeString || ag.Expression == nil { + t.Errorf("got %#v", ag) + } + }}, + } + for _, header := range []string{"", "mdl 1;\n"} { + for _, tc := range cases { + t.Run(strings.TrimSpace(header+" "+tc.src), func(t *testing.T) { + prog, stmt, errs := buildListActivity(t, header, tc.src) + if len(errs) > 0 { + t.Fatalf("parse: %v", errs) + } + if stmt == nil { + t.Fatal("no statement built") + } + tc.check(t, stmt) + if d := deprecationCodes(prog); len(d) != 0 { + t.Errorf("canonical form recorded deprecations %v", d) + } + if n := noteCodes(prog); len(n) != 0 { + t.Errorf("canonical form recorded language notes %v", n) + } + }) + } + } +} + +func listPair(op ast.ListOperationType, first, second string) func(*testing.T, ast.MicroflowStatement) { + return func(t *testing.T, s ast.MicroflowStatement) { + lo := s.(*ast.ListOperationStmt) + if lo.Operation != op || lo.InputVariable != first || lo.SecondVariable != second { + t.Errorf("got %#v, want %v(%s, %s)", lo, op, first, second) + } + } +} + +func aggregateOp(op ast.AggregateListOperationType) func(*testing.T, ast.MicroflowStatement) { + return func(t *testing.T, s ast.MicroflowStatement) { + ag := s.(*ast.AggregateListStmt) + if ag.Operation != op || ag.InputVariable != "Orders" { + t.Errorf("got %#v", ag) + } + } +} + +// The function form is a deprecated respelling of the statement: both build the +// same AST, and only the function form records the registry code. find and +// contains are not here — they clash with the string functions (see +// TestFindContainsFunctionFormIsVersionGated). +func TestListFunctionFormIsAnAlias(t *testing.T) { + cases := []struct{ old, canon, code string }{ + {"$H = head($Orders);", "$H = head $Orders;", deprecation.ListOperationFunctionForm}, + {"$T = tail($Orders);", "$T = tail $Orders;", deprecation.ListOperationFunctionForm}, + {"$F = filter($Orders, Status = M.Status.Open);", "$F = filter $Orders by Status = M.Status.Open;", deprecation.ListOperationFunctionForm}, + {"$F = filter($Orders, $currentObject/Total > 10);", "$F = filter $Orders where $currentObject/Total > 10;", deprecation.ListOperationFunctionForm}, + {"$F = sort($Orders, OrderDate desc, Number);", "$F = sort $Orders by OrderDate desc, Number;", deprecation.ListOperationFunctionForm}, + {"$U = union($A, $B);", "$U = union $A with $B;", deprecation.ListOperationFunctionForm}, + {"$U = intersect($A, $B);", "$U = intersect $A with $B;", deprecation.ListOperationFunctionForm}, + {"$U = subtract($A, $B);", "$U = subtract $B from $A;", deprecation.ListOperationFunctionForm}, + {"$E = equals($A, $B);", "$E = equals $A and $B;", deprecation.ListOperationFunctionForm}, + {"$R = range($Orders, 20, 10);", "$R = range $Orders offset 20 limit 10;", deprecation.ListOperationFunctionForm}, + {"$N = count($Orders);", "$N = count $Orders;", deprecation.AggregateFunctionForm}, + {"$N = sum($Orders.Amount);", "$N = sum $Orders by Amount;", deprecation.AggregateFunctionForm}, + {"$N = average($Orders, $currentObject/Amount * 2);", "$N = average $Orders of $currentObject/Amount * 2;", deprecation.AggregateFunctionForm}, + {"$N = all($Orders, $currentObject/Paid);", "$N = all $Orders where $currentObject/Paid;", deprecation.AggregateFunctionForm}, + {"$N = reduce($Orders, $currentResult + 1, initial: 0, returns: Integer);", "$N = reduce $Orders from 0 as Integer using $currentResult + 1;", deprecation.AggregateFunctionForm}, + } + for _, header := range []string{"", "mdl 1;\n"} { + for _, tc := range cases { + t.Run(strings.TrimSpace(header+" "+tc.old), func(t *testing.T) { + oldProg, oldStmt, errs := buildListActivity(t, header, tc.old) + if len(errs) > 0 { + t.Fatalf("old form: %v", errs) + } + canonProg, canonStmt, errs := buildListActivity(t, header, tc.canon) + if len(errs) > 0 { + t.Fatalf("canonical form: %v", errs) + } + if !reflect.DeepEqual(oldStmt, canonStmt) { + t.Errorf("forms build different statements:\n old: %#v\n canon: %#v", oldStmt, canonStmt) + } + if got := deprecationCodes(oldProg); !reflect.DeepEqual(got, []string{tc.code}) { + t.Errorf("old form recorded %v, want [%s]", got, tc.code) + } + if got := deprecationCodes(canonProg); len(got) != 0 { + t.Errorf("canonical form recorded %v", got) + } + if n := noteCodes(oldProg); len(n) != 0 { + t.Errorf("an alias is not a change of meaning, but recorded notes %v", n) + } + }) + } + } +} + +// `$x = find(a, b)` and `$x = contains(a, b)` are either the list operation or +// the string function, decided today by what the arguments look like. Under +// mdl 0 that meaning is kept and warned; under mdl 1 the form is refused. +func TestFindContainsFunctionFormIsVersionGated(t *testing.T) { + for _, src := range []string{ + "$M = find($Orders, Number = $Number);", + "$M = contains($Orders, $Order);", + } { + t.Run("mdl 0 "+src, func(t *testing.T) { + prog, stmt, errs := buildListActivity(t, "", src) + if len(errs) > 0 { + t.Fatalf("parse: %v", errs) + } + if _, ok := stmt.(*ast.ListOperationStmt); !ok { + t.Errorf("mdl 0 must keep the list operation, got %T", stmt) + } + if got := noteCodes(prog); !reflect.DeepEqual(got, []string{listCallForm.Code}) { + t.Errorf("notes = %v, want [%s]", got, listCallForm.Code) + } + if d := deprecationCodes(prog); len(d) != 0 { + t.Errorf("find/contains are not aliases, but recorded %v", d) + } + }) + t.Run("mdl 1 "+src, func(t *testing.T) { + _, _, errs := buildListActivity(t, "mdl 1;\n", src) + if len(errs) == 0 { + t.Fatal("accepted under mdl 1") + } + if msg := errs[0].Error(); !strings.Contains(msg, "set $M =") || !strings.Contains(msg, "List operation") { + t.Errorf("the error should name both readings: %v", msg) + } + }) + } +} + +// Under mdl 1 `set` always assigns an expression, so `set $x = find(…)` is the +// string function whatever its arguments look like. +func TestSetFindIsTheStringFunctionUnderMdl1(t *testing.T) { + body := "declare $I Integer = 0;\n set $I = find($S, $S);" + prog, errs := Build(listActivityMicroflow("mdl 1;\n", body)) + if len(errs) > 0 { + t.Fatalf("parse: %v", errs) + } + stmt := prog.Statements[0].(*ast.CreateMicroflowStmt).Body[1] + if _, ok := stmt.(*ast.MfSetStmt); !ok { + t.Errorf("mdl 1: set $I = find($S, $S) built %T, want *ast.MfSetStmt", stmt) + } + + // Control: under mdl 0 the same text keeps today's list-operation reading + // and warns that its meaning differs under mdl 1. + prog, errs = Build(listActivityMicroflow("", body)) + if len(errs) > 0 { + t.Fatalf("parse: %v", errs) + } + if _, ok := prog.Statements[0].(*ast.CreateMicroflowStmt).Body[1].(*ast.ListOperationStmt); !ok { + t.Errorf("mdl 0 must keep the list operation reading") + } + if got := noteCodes(prog); !reflect.DeepEqual(got, []string{listCallForm.Code}) { + t.Errorf("notes = %v, want [%s]", got, listCallForm.Code) + } +} + +// Nesting one activity inside another cannot be written under mdl 1; under +// mdl 0 it keeps parsing (MDL-LISTOP02 refuses it at check time) and warns. +func TestNestedListOperationRefusedUnderMdl1(t *testing.T) { + for _, src := range []string{ + "$N = count(filter($Orders, Status = M.Status.Open));", + "set $N = count(filter($Orders, Status = M.Status.Open));", + "set $H = head($Orders);", + } { + t.Run("mdl 1 "+src, func(t *testing.T) { + _, _, errs := buildListActivity(t, "mdl 1;\n", src) + if len(errs) == 0 { + t.Fatal("accepted under mdl 1") + } + if msg := errs[0].Error(); !strings.Contains(msg, "one statement per activity") { + t.Errorf("error should explain the statement form: %v", msg) + } + }) + t.Run("mdl 0 "+src, func(t *testing.T) { + prog, _, errs := buildListActivity(t, "", src) + if len(errs) > 0 { + t.Fatalf("mdl 0 must keep parsing: %v", errs) + } + if got := noteCodes(prog); !reflect.DeepEqual(got, []string{listCallForm.Code}) { + t.Errorf("notes = %v, want [%s]", got, listCallForm.Code) + } + }) + } +} + +// `set` is mandatory for reassignment under mdl 1. +func TestSetIsMandatoryUnderMdl1(t *testing.T) { + body := "declare $I Integer = 0;\n $I = 5;" + _, errs := Build(listActivityMicroflow("mdl 1;\n", body)) + if len(errs) == 0 || !strings.Contains(errs[0].Error(), "set $I = 5") { + t.Fatalf("mdl 1 must refuse a reassignment without set, got %v", errs) + } + + prog, errs := Build(listActivityMicroflow("", body)) + if len(errs) > 0 { + t.Fatalf("mdl 0 must keep parsing: %v", errs) + } + if got := noteCodes(prog); !reflect.DeepEqual(got, []string{setIsMandatory.Code}) { + t.Errorf("notes = %v, want [%s]", got, setIsMandatory.Code) + } + + // Control: with `set`, neither version says anything. + for _, header := range []string{"", "mdl 1;\n"} { + prog, errs := Build(listActivityMicroflow(header, "declare $I Integer = 0;\n set $I = 5;")) + if len(errs) > 0 || len(prog.LanguageNotes) > 0 { + t.Errorf("%q: set $I = 5 gave errors %v, notes %v", header, errs, prog.LanguageNotes) + } + } + // An attribute target is `change`'s territory (plan item 3.x), not this rule. + prog, errs = Build(listActivityMicroflow("mdl 1;\n", "$Order/Number = 5;")) + if len(errs) > 0 || len(prog.LanguageNotes) > 0 { + t.Errorf("$Order/Number = 5 gave errors %v, notes %v", errs, prog.LanguageNotes) + } +} + +// `by` picks a member: anything but `Member = value` belongs after `where`. +func TestFilterByNeedsMemberEquality(t *testing.T) { + _, _, errs := buildListActivity(t, "", "$F = filter $Orders by $currentObject/Total > 3;") + if len(errs) == 0 || !strings.Contains(errs[0].Error(), "where") { + t.Fatalf("want an error pointing at `where`, got %v", errs) + } +} diff --git a/mdl/visitor/visitor_menu.go b/mdl/visitor/visitor_menu.go index 6353c5e36a..dcb14830f5 100644 --- a/mdl/visitor/visitor_menu.go +++ b/mdl/visitor/visitor_menu.go @@ -24,7 +24,7 @@ func (b *Builder) ExitCreateMenuStatement(ctx *parser.CreateMenuStatementContext // the rewrite gap of #1018, found while testing the latter. stmt.Documentation, stmt.DocumentationSet = findDocComment(ctx) if lit := ctx.STRING_LITERAL(); lit != nil { - stmt.Folder = unquoteString(lit.GetText()) + stmt.Folder = unquoteStringLit(lit) } for _, itemCtx := range ctx.AllNavMenuItemDef() { stmt.Items = append(stmt.Items, buildNavMenuItemDef(itemCtx)) diff --git a/mdl/visitor/visitor_messagedefinition.go b/mdl/visitor/visitor_messagedefinition.go index 960e385043..c10602e967 100644 --- a/mdl/visitor/visitor_messagedefinition.go +++ b/mdl/visitor/visitor_messagedefinition.go @@ -20,7 +20,7 @@ func (b *Builder) ExitCreateMessageDefinitionCollectionStatement(ctx *parser.Cre } if ctx.FOLDER() != nil { if lit := ctx.STRING_LITERAL(); lit != nil { - stmt.Folder = unquoteString(lit.GetText()) + stmt.Folder = unquoteStringLit(lit) } } for _, d := range ctx.AllMessageDefinitionDef() { @@ -84,7 +84,7 @@ func (b *Builder) buildMessageMember(c parser.IMessageMemberContext) *ast.Messag mem.Attribute = identifierOrKeywordText(ctx.IdentifierOrKeyword()) if ex, ok := ctx.MessageExample().(*parser.MessageExampleContext); ok && ex != nil { if lit := ex.STRING_LITERAL(); lit != nil { - mem.Example = unquoteString(lit.GetText()) + mem.Example = unquoteStringLit(lit) } } return mem @@ -97,7 +97,7 @@ func exposedNameOf(c parser.IMessageExposedNameContext) string { return "" } if lit := ctx.STRING_LITERAL(); lit != nil { - return unquoteString(lit.GetText()) + return unquoteStringLit(lit) } return "" } diff --git a/mdl/visitor/visitor_microflow.go b/mdl/visitor/visitor_microflow.go index 464870e172..524a9a95d6 100644 --- a/mdl/visitor/visitor_microflow.go +++ b/mdl/visitor/visitor_microflow.go @@ -32,7 +32,7 @@ func (b *Builder) ExitCreateMicroflowStatement(ctx *parser.CreateMicroflowStatem for _, opt := range optsCtx.AllMicroflowOption() { optCtx := opt.(*parser.MicroflowOptionContext) if optCtx.FOLDER() != nil && optCtx.STRING_LITERAL() != nil { - stmt.Folder = unquoteString(optCtx.STRING_LITERAL().GetText()) + stmt.Folder = unquoteStringLit(optCtx.STRING_LITERAL()) } if exposed := optCtx.MicroflowExposedClause(); exposed != nil { stmt.Expose = append(stmt.Expose, buildExposeActionClause(exposed)) @@ -86,7 +86,7 @@ func (b *Builder) ExitCreateNanoflowStatement(ctx *parser.CreateNanoflowStatemen for _, opt := range optsCtx.AllMicroflowOption() { optCtx := opt.(*parser.MicroflowOptionContext) if optCtx.FOLDER() != nil && optCtx.STRING_LITERAL() != nil { - stmt.Folder = unquoteString(optCtx.STRING_LITERAL().GetText()) + stmt.Folder = unquoteStringLit(optCtx.STRING_LITERAL()) } if exposed := optCtx.MicroflowExposedClause(); exposed != nil { stmt.Expose = append(stmt.Expose, buildExposeActionClause(exposed)) @@ -138,7 +138,7 @@ func (b *Builder) ExitCreateRuleStatement(ctx *parser.CreateRuleStatementContext for _, opt := range optsCtx.AllMicroflowOption() { optCtx := opt.(*parser.MicroflowOptionContext) if optCtx.FOLDER() != nil && optCtx.STRING_LITERAL() != nil { - stmt.Folder = unquoteString(optCtx.STRING_LITERAL().GetText()) + stmt.Folder = unquoteStringLit(optCtx.STRING_LITERAL()) } if exposed := optCtx.MicroflowExposedClause(); exposed != nil { stmt.Expose = append(stmt.Expose, buildExposeActionClause(exposed)) @@ -392,8 +392,8 @@ func buildExposeActionClause(ctx parser.IMicroflowExposedClauseContext) ast.Expo return out } if strs := ctx.AllSTRING_LITERAL(); len(strs) >= 2 { - out.Caption = unquoteString(strs[0].GetText()) - out.Category = unquoteString(strs[1].GetText()) + out.Caption = unquoteStringLit(strs[0]) + out.Category = unquoteStringLit(strs[1]) } out.Bitmaps = buildExposeBitmaps(ctx.AllExposeBitmapClause()) return out @@ -412,7 +412,7 @@ func buildExposeBitmaps(clauses []parser.IExposeBitmapClauseContext) []ast.Expos Clear: c.DROP() != nil, } if lit := c.STRING_LITERAL(); lit != nil { - b.Path = unquoteString(lit.GetText()) + b.Path = unquoteStringLit(lit) } out = append(out, b) } @@ -476,7 +476,7 @@ func applyMicroflowDocumentProperty(stmt *ast.CreateMicroflowStmt, optCtx *parse } stmt.URLSearchParameters = &names case uc.STRING_LITERAL() != nil: - url := unquoteString(uc.STRING_LITERAL().GetText()) + url := unquoteStringLit(uc.STRING_LITERAL()) stmt.URL = &url } } @@ -498,7 +498,7 @@ func applyMicroflowDocumentProperty(stmt *ast.CreateMicroflowStmt, optCtx *parse if e := cCtx.MicroflowConcurrencyError(); e != nil { eCtx := e.(*parser.MicroflowConcurrencyErrorContext) if eCtx.ERROR_MESSAGE() != nil && eCtx.STRING_LITERAL() != nil { - clause.ErrorMessage = unquoteString(eCtx.STRING_LITERAL().GetText()) + clause.ErrorMessage = unquoteStringLit(eCtx.STRING_LITERAL()) clause.ErrorMessageSet = true } if qn := eCtx.QualifiedName(); qn != nil { diff --git a/mdl/visitor/visitor_microflow_actions.go b/mdl/visitor/visitor_microflow_actions.go index 0df0fd391b..6c3559f464 100644 --- a/mdl/visitor/visitor_microflow_actions.go +++ b/mdl/visitor/visitor_microflow_actions.go @@ -543,7 +543,7 @@ func buildCallWebServiceStatement(ctx parser.ICallWebServiceStatementContext) *a if callCtx.RAW() != nil { if lit := callCtx.STRING_LITERAL(); lit != nil { - stmt.RawBSONBase64 = unquoteString(lit.GetText()) + stmt.RawBSONBase64 = unquoteStringLit(lit) } if errClause := callCtx.OnErrorClause(); errClause != nil { stmt.ErrorHandling = buildOnErrorClause(errClause) @@ -593,7 +593,7 @@ func webServiceReferenceText(ctx parser.IWebServiceReferenceContext) string { } refCtx := ctx.(*parser.WebServiceReferenceContext) if lit := refCtx.STRING_LITERAL(); lit != nil { - return unquoteString(lit.GetText()) + return unquoteStringLit(lit) } return getQualifiedNameText(refCtx.QualifiedName()) } @@ -620,7 +620,7 @@ func buildExecuteDatabaseQueryStatement(ctx parser.IExecuteDatabaseQueryStatemen // Get dynamic query if present if execCtx.DYNAMIC() != nil { if sl := execCtx.STRING_LITERAL(); sl != nil { - stmt.DynamicQuery = unquoteString(sl.GetText()) + stmt.DynamicQuery = unquoteStringLit(sl) } else if ds := execCtx.DOLLAR_STRING(); ds != nil { stmt.DynamicQuery = unquoteDollarString(ds.GetText()) } else if expr := execCtx.Expression(); expr != nil { @@ -803,7 +803,20 @@ func buildListOperationStatement(ctx parser.IListOperationStatementContext) *ast stmt.OutputVariable = strings.TrimPrefix(v.GetText(), "$") } - // Get the list operation + // The statement form: one Studio Pro activity (#733). + if act, ok := listOpCtx.ListOperationActivity().(*parser.ListOperationActivityContext); ok && act != nil { + buildListOperationActivity(act, stmt) + return stmt + } + + // The call form. For find/filter it is a respelling of `by` when the + // condition reads `Member = value` and of `where` otherwise, so it records + // which, exactly as the flow builder has always decided (ast.IsMemberEquality). + defer func() { + if stmt.Operation == ast.ListOpFind || stmt.Operation == ast.ListOpFilter { + stmt.ByExpression = !ast.IsMemberEquality(stmt.Condition) + } + }() if opCtx := listOpCtx.ListOperation(); opCtx != nil { op := opCtx.(*parser.ListOperationContext) @@ -947,7 +960,13 @@ func buildAggregateListStatement(ctx parser.IAggregateListStatementContext) *ast stmt.OutputVariable = strings.TrimPrefix(v.GetText(), "$") } - // Get the aggregate operation + // The statement form: one Studio Pro Aggregate list activity (#733). + if act, ok := aggrCtx.AggregateListActivity().(*parser.AggregateListActivityContext); ok && act != nil { + buildAggregateListActivity(act, stmt) + return stmt + } + + // The call form, a deprecated alias of the above. if opCtx := aggrCtx.ListAggregateOperation(); opCtx != nil { op := opCtx.(*parser.ListAggregateOperationContext) @@ -1493,7 +1512,7 @@ func buildRestCallStatement(ctx parser.IRestCallStatementContext) *ast.RestCallS if strLit := urlC.STRING_LITERAL(); strLit != nil { stmt.URL = &ast.LiteralExpr{ Kind: ast.LiteralString, - Value: unquoteString(strLit.GetText()), + Value: unquoteStringLit(strLit), } } else if expr := urlC.Expression(); expr != nil { stmt.URL = buildSourceExpression(expr) @@ -1516,7 +1535,7 @@ func buildRestCallStatement(ctx parser.IRestCallStatementContext) *ast.RestCallS header.Name = id.GetText() } else if strLit := hdrCtx.STRING_LITERAL(); strLit != nil { // Handle quoted header names like 'Content-Type' - header.Name = unquoteString(strLit.GetText()) + header.Name = unquoteStringLit(strLit) } if expr := hdrCtx.Expression(); expr != nil { header.Value = buildSourceExpression(expr) @@ -1564,7 +1583,7 @@ func buildRestCallStatement(ctx parser.IRestCallStatementContext) *ast.RestCallS if strLit := bodyCtx.STRING_LITERAL(); strLit != nil { body.Template = &ast.LiteralExpr{ Kind: ast.LiteralString, - Value: unquoteString(strLit.GetText()), + Value: unquoteStringLit(strLit), } } else if expr := bodyCtx.Expression(); expr != nil { body.Template = buildSourceExpression(expr) diff --git a/mdl/visitor/visitor_microflow_expression.go b/mdl/visitor/visitor_microflow_expression.go index 98563f14b1..3d32c4efde 100644 --- a/mdl/visitor/visitor_microflow_expression.go +++ b/mdl/visitor/visitor_microflow_expression.go @@ -706,7 +706,7 @@ func buildLiteralExpression(ctx parser.ILiteralContext) *ast.LiteralExpr { // String literal if str := litCtx.STRING_LITERAL(); str != nil { return &ast.LiteralExpr{ - Value: unquoteString(str.GetText()), + Value: unquoteStringLit(str), Kind: ast.LiteralString, } } diff --git a/mdl/visitor/visitor_microflow_statements.go b/mdl/visitor/visitor_microflow_statements.go index 34cca43af2..7cc4658802 100644 --- a/mdl/visitor/visitor_microflow_statements.go +++ b/mdl/visitor/visitor_microflow_statements.go @@ -598,14 +598,14 @@ func extractAnnotationValueString(ctx parser.IAnnotationValueContext) string { if lit := valCtx.Literal(); lit != nil { litCtx := lit.(*parser.LiteralContext) if litCtx.STRING_LITERAL() != nil { - return unquoteString(litCtx.STRING_LITERAL().GetText()) + return unquoteStringLit(litCtx.STRING_LITERAL()) } } // Also try expression — it might be a string literal parsed as expression if expr := valCtx.Expression(); expr != nil { text := expr.GetText() if len(text) >= 2 && text[0] == '\'' && text[len(text)-1] == '\'' { - return unquoteString(text) + return unquoteStringLit(expr) } } return "" @@ -802,8 +802,10 @@ func buildSetStatementNode(ctx parser.ISetStatementContext) ast.MicroflowStateme valueExpr = buildExpression(expr) } - // Check if the expression is a list operation or aggregate function. - if funcCall, ok := valueExpr.(*ast.FunctionCallExpr); ok { + // Check if the expression is a list operation or aggregate function. Under + // mdl 1 `set` always assigns an expression: a list-operation call there is + // refused (ExitSetStatement), and find/contains are the string functions. + if funcCall, ok := valueExpr.(*ast.FunctionCallExpr); ok && !listCallForm.Applies(scriptLanguageVersion(setCtx)) { if stmt := buildListOrAggregateStatement(targetVar, funcCall); stmt != nil { return recordUnresolvedOperands(stmt, funcCall.Arguments) } @@ -857,20 +859,24 @@ func buildListOrAggregateStatement(targetVar string, funcCall *ast.FunctionCallE // (CE0111). Ledger #63. When both arguments are plain variables the kind // is ambiguous here; the flow builder disambiguates String-typed inputs. if !isStringLiteralArg(funcCall.Arguments, 1) { + cond := getArgumentExpression(funcCall.Arguments, 1) return &ast.ListOperationStmt{ OutputVariable: targetVar, Operation: ast.ListOpFind, InputVariable: extractVariableName(funcCall.Arguments, 0), - Condition: getArgumentExpression(funcCall.Arguments, 1), + Condition: cond, + ByExpression: !ast.IsMemberEquality(cond), } } // Falls through to the default MfSetStmt (string find expression). case "FILTER": + cond := getArgumentExpression(funcCall.Arguments, 1) return &ast.ListOperationStmt{ OutputVariable: targetVar, Operation: ast.ListOpFilter, InputVariable: extractVariableName(funcCall.Arguments, 0), - Condition: getArgumentExpression(funcCall.Arguments, 1), + Condition: cond, + ByExpression: !ast.IsMemberEquality(cond), } case "SORT": stmt := &ast.ListOperationStmt{ @@ -1392,8 +1398,12 @@ func buildRetrieveStatement(ctx parser.IRetrieveStatementContext) *ast.RetrieveS } } - // Get LIMIT and OFFSET expressions - if limitExpr := retrCtx.GetLimitExpr(); limitExpr != nil { + // The range. `first` is the object range in every version; a bare + // `limit 1` is too in an mdl 0 script, and a list of one under mdl 1 + // (limitOneIsAList). Resolved here so nothing downstream reads limit text. + if retrCtx.FIRST() != nil || retrieveLimitOneIsObject(retrCtx) { + stmt.First = true + } else if limitExpr := retrCtx.GetLimitExpr(); limitExpr != nil { stmt.Limit = retrieveRangeExpressionSource(limitExpr) + retrieveLimitTrailingWhitespace(retrCtx, limitExpr) } if offsetExpr := retrCtx.GetOffsetExpr(); offsetExpr != nil { diff --git a/mdl/visitor/visitor_microflow_workflow.go b/mdl/visitor/visitor_microflow_workflow.go index 00b291954f..49d561f8a0 100644 --- a/mdl/visitor/visitor_microflow_workflow.go +++ b/mdl/visitor/visitor_microflow_workflow.go @@ -159,7 +159,7 @@ func buildSetTaskOutcomeStatement(ctx parser.ISetTaskOutcomeStatementContext) *a stmt.WorkflowTaskVariable = strings.TrimPrefix(v.GetText(), "$") } if str := c.STRING_LITERAL(); str != nil { - stmt.OutcomeValue = unquoteString(str.GetText()) + stmt.OutcomeValue = unquoteStringLit(str) } if errClause := c.OnErrorClause(); errClause != nil { stmt.ErrorHandling = buildOnErrorClause(errClause) diff --git a/mdl/visitor/visitor_module_settings.go b/mdl/visitor/visitor_module_settings.go index be8b0aeb92..6ebddac9ff 100644 --- a/mdl/visitor/visitor_module_settings.go +++ b/mdl/visitor/visitor_module_settings.go @@ -54,7 +54,7 @@ func parseJarDepAction(ctx *parser.AlterModuleJarDepActionContext) ast.JarDepAct var boolVal bool isBool := false if sl := propCtx.STRING_LITERAL(); sl != nil { - val = unquoteString(sl.GetText()) + val = unquoteStringLit(sl) } else if bl := propCtx.BooleanLiteral(); bl != nil { boolVal = strings.EqualFold(bl.GetText(), "true") isBool = true @@ -81,13 +81,13 @@ func parseJarDepAction(ctx *parser.AlterModuleJarDepActionContext) ast.JarDepAct if len(strs) == 0 { return nil } - coordinate := unquoteString(strs[0].GetText()) + coordinate := unquoteStringLit(strs[0]) if ctx.VERSION() != nil && len(strs) >= 2 { // SET JAR DEPENDENCY 'coord' VERSION 'version' return &ast.SetJarDepVersionAction{ Coordinate: coordinate, - Version: unquoteString(strs[1].GetText()), + Version: unquoteStringLit(strs[1]), } } @@ -104,7 +104,7 @@ func parseJarDepAction(ctx *parser.AlterModuleJarDepActionContext) ast.JarDepAct // SET JAR DEPENDENCY 'coord' ADD EXCLUSION 'excl' return &ast.AddJarDepExclusionAction{ Coordinate: coordinate, - Exclusion: unquoteString(strs[1].GetText()), + Exclusion: unquoteStringLit(strs[1]), } } @@ -112,7 +112,7 @@ func parseJarDepAction(ctx *parser.AlterModuleJarDepActionContext) ast.JarDepAct // SET JAR DEPENDENCY 'coord' DROP EXCLUSION 'excl' return &ast.DropJarDepExclusionAction{ Coordinate: coordinate, - Exclusion: unquoteString(strs[1].GetText()), + Exclusion: unquoteStringLit(strs[1]), } } @@ -125,7 +125,7 @@ func parseJarDepAction(ctx *parser.AlterModuleJarDepActionContext) ast.JarDepAct if len(strs) == 0 { return nil } - return &ast.DropJarDepAction{Coordinate: unquoteString(strs[0].GetText())} + return &ast.DropJarDepAction{Coordinate: unquoteStringLit(strs[0])} } return nil diff --git a/mdl/visitor/visitor_navigation.go b/mdl/visitor/visitor_navigation.go index 13a83ad7d0..b9686cd6f4 100644 --- a/mdl/visitor/visitor_navigation.go +++ b/mdl/visitor/visitor_navigation.go @@ -148,7 +148,7 @@ func buildNavSyncDef(ctx parser.INavSyncDefContext) ast.NavSyncDef { } } else if lit := mc.STRING_LITERAL(); lit != nil { // Legacy quoted form: the '' pairs are MDL escaping and come off here. - def.Constraint = unquoteString(lit.GetText()) + def.Constraint = unquoteStringLit(lit) } } return def @@ -160,7 +160,7 @@ func buildNavMenuItemDef(ctx parser.INavMenuItemDefContext) ast.NavMenuItemDef { caption := "" if sl := c.STRING_LITERAL(); sl != nil { - caption = unquoteString(sl.GetText()) + caption = unquoteStringLit(sl) } item := ast.NavMenuItemDef{Caption: caption} diff --git a/mdl/visitor/visitor_odata.go b/mdl/visitor/visitor_odata.go index 6f3a9561f6..acde9fa4e6 100644 --- a/mdl/visitor/visitor_odata.go +++ b/mdl/visitor/visitor_odata.go @@ -279,7 +279,7 @@ func (b *Builder) ExitCreateExternalEntitiesStatement(ctx *parser.CreateExternal // odataValueText extracts the string value from an OData property value context. func odataValueText(val *parser.OdataPropertyValueContext) string { if sl := val.STRING_LITERAL(); sl != nil { - return unquoteString(sl.GetText()) + return unquoteStringLit(sl) } if nl := val.NUMBER_LITERAL(); nl != nil { return nl.GetText() @@ -357,7 +357,7 @@ func parsePublishEntityBlock(ctx parser.IPublishEntityBlockContext) *ast.Publish // Optional AS 'ExposedName' if sl := block.STRING_LITERAL(); sl != nil { - entity.ExposedName = unquoteString(sl.GetText()) + entity.ExposedName = unquoteStringLit(sl) } // Parse entity-level properties (ReadMode, InsertMode, etc.) @@ -407,7 +407,7 @@ func parseODataHeaders(ctx parser.IOdataHeadersClauseContext) []ast.HeaderDef { for _, entryCtx := range clause.AllOdataHeaderEntry() { entry := entryCtx.(*parser.OdataHeaderEntryContext) - key := unquoteString(entry.STRING_LITERAL().GetText()) + key := unquoteStringLit(entry.STRING_LITERAL()) // A header value is a Mendix expression, kept as written. value, isLiteral := odataExpressionValue(entry.OdataPropertyValue(), entry.Expression()) headers = append(headers, ast.HeaderDef{Key: key, Value: value, ValueIsLiteral: isLiteral}) @@ -428,7 +428,7 @@ func parsePublishMicroflowBlock(ctx parser.IPublishMicroflowBlockContext) *ast.P Microflow: buildQualifiedName(block.QualifiedName()), } if sl := block.STRING_LITERAL(); sl != nil { - def.ExposedName = unquoteString(sl.GetText()) + def.ExposedName = unquoteStringLit(sl) } if exposeCtx := block.ExposeClause(); exposeCtx != nil { expose := exposeCtx.(*parser.ExposeClauseContext) @@ -442,7 +442,7 @@ func parsePublishMicroflowBlock(ctx parser.IPublishMicroflowBlockContext) *ast.P } p := &ast.PublishedParamDef{Name: member.IdentifierOrKeyword().GetText()} if sl := member.STRING_LITERAL(); sl != nil { - p.ExposedName = unquoteString(sl.GetText()) + p.ExposedName = unquoteStringLit(sl) } if opts := member.ExposeMemberOptions(); opts != nil { optsCtx := opts.(*parser.ExposeMemberOptionsContext) @@ -486,7 +486,7 @@ func parseExposeMembers(ctx parser.IExposeClauseContext) []*ast.PublishedMemberD // Optional AS 'ExposedName' if sl := member.STRING_LITERAL(); sl != nil { - m.ExposedName = unquoteString(sl.GetText()) + m.ExposedName = unquoteStringLit(sl) } // Optional options (Filterable, Sortable, IsPartOfKey) diff --git a/mdl/visitor/visitor_page_v3.go b/mdl/visitor/visitor_page_v3.go index 5dfddafe67..4fa63666cd 100644 --- a/mdl/visitor/visitor_page_v3.go +++ b/mdl/visitor/visitor_page_v3.go @@ -103,34 +103,34 @@ func (b *Builder) parsePageHeaderV3(ctx parser.IPageHeaderV3Context, stmt *ast.C } else if prop.TITLE() != nil { // Title: 'My Page' if str := prop.STRING_LITERAL(); str != nil { - stmt.Title = unquoteString(str.GetText()) + stmt.Title = unquoteStringLit(str) } } else if prop.LAYOUT() != nil { // Layout: Atlas_Core.Atlas_Default or 'Layout Name' if qn := prop.QualifiedName(); qn != nil { stmt.Layout = getQualifiedNameText(qn) } else if str := prop.STRING_LITERAL(); str != nil { - stmt.Layout = unquoteString(str.GetText()) + stmt.Layout = unquoteStringLit(str) } } else if prop.URL() != nil { // Url: 'my-page' if str := prop.STRING_LITERAL(); str != nil { - stmt.URL = unquoteString(str.GetText()) + stmt.URL = unquoteStringLit(str) } } else if prop.FOLDER() != nil { // Folder: 'Pages/Admin' if str := prop.STRING_LITERAL(); str != nil { - stmt.Folder = unquoteString(str.GetText()) + stmt.Folder = unquoteStringLit(str) } } else if prop.CLASS() != nil { // Class: 'my-page' — page-level CSS class (issue #714) if str := prop.STRING_LITERAL(); str != nil { - stmt.Class = unquoteString(str.GetText()) + stmt.Class = unquoteStringLit(str) } } else if prop.STYLE() != nil { // Style: 'padding: 10px' — page-level inline CSS (issue #714) if str := prop.STRING_LITERAL(); str != nil { - stmt.Style = unquoteString(str.GetText()) + stmt.Style = unquoteStringLit(str) } } else if id := prop.IDENTIFIER(); id != nil { // Generic page header property (e.g. PopupWidth: 800). Only the @@ -227,7 +227,7 @@ func (b *Builder) buildSnippetV3(ctx *parser.CreateSnippetStatementContext) *ast for _, opt := range optsCtx.AllSnippetOption() { optCtx := opt.(*parser.SnippetOptionContext) if optCtx.FOLDER() != nil && optCtx.STRING_LITERAL() != nil { - stmt.Folder = unquoteString(optCtx.STRING_LITERAL().GetText()) + stmt.Folder = unquoteStringLit(optCtx.STRING_LITERAL()) } } } @@ -263,7 +263,7 @@ func (b *Builder) parseSnippetHeaderV3(ctx parser.ISnippetHeaderV3Context, stmt } else if prop.FOLDER() != nil { // Folder: 'Snippets/Common' if str := prop.STRING_LITERAL(); str != nil { - stmt.Folder = unquoteString(str.GetText()) + stmt.Folder = unquoteStringLit(str) } } } @@ -305,7 +305,7 @@ func buildSingleVariableDeclaration(vdCtx *parser.VariableDeclarationContext) as } if str := vdCtx.STRING_LITERAL(); str != nil { - v.DefaultValue = unquoteString(str.GetText()) + v.DefaultValue = unquoteStringLit(str) } return v @@ -536,10 +536,10 @@ func buildWidgetV3(ctx parser.IWidgetV3Context, b *Builder) *ast.WidgetV3 { // Get widget type if wCtx.PLUGGABLEWIDGET() != nil { widget.Type = "pluggablewidget" - widget.Properties["WidgetType"] = unquoteString(wCtx.STRING_LITERAL().GetText()) + widget.Properties["WidgetType"] = unquoteStringLit(wCtx.STRING_LITERAL()) } else if wCtx.CUSTOMWIDGET() != nil { widget.Type = "customwidget" - widget.Properties["WidgetType"] = unquoteString(wCtx.STRING_LITERAL().GetText()) + widget.Properties["WidgetType"] = unquoteStringLit(wCtx.STRING_LITERAL()) } else if typeCtx := wCtx.WidgetTypeV3(); typeCtx != nil { widget.Type = strings.ToLower(typeCtx.GetText()) // Which alternative matched, taken from the parse tree rather than by @@ -643,7 +643,7 @@ func parseWidgetPropertyV3(ctx parser.IWidgetPropertyV3Context, widget *ast.Widg // Placeholder: 'text' (input widgets) if propCtx.PLACEHOLDER() != nil { if str := propCtx.STRING_LITERAL(); str != nil { - widget.Properties["Placeholder"] = unquoteString(str.GetText()) + widget.Properties["Placeholder"] = unquoteStringLit(str) } return } @@ -659,7 +659,7 @@ func parseWidgetPropertyV3(ctx parser.IWidgetPropertyV3Context, widget *ast.Widg // Label: ... if propCtx.LABEL() != nil { if str := propCtx.STRING_LITERAL(); str != nil { - widget.Properties["Label"] = unquoteString(str.GetText()) + widget.Properties["Label"] = unquoteStringLit(str) } return } @@ -722,7 +722,7 @@ func parseWidgetPropertyV3(ctx parser.IWidgetPropertyV3Context, widget *ast.Widg // Class: ... if propCtx.CLASS() != nil { if str := propCtx.STRING_LITERAL(); str != nil { - widget.Properties["Class"] = unquoteString(str.GetText()) + widget.Properties["Class"] = unquoteStringLit(str) } return } @@ -730,7 +730,7 @@ func parseWidgetPropertyV3(ctx parser.IWidgetPropertyV3Context, widget *ast.Widg // Style: ... if propCtx.STYLE() != nil { if str := propCtx.STRING_LITERAL(); str != nil { - widget.Properties["Style"] = unquoteString(str.GetText()) + widget.Properties["Style"] = unquoteStringLit(str) } return } @@ -1146,7 +1146,7 @@ func buildActionV3(ctx parser.IActionExprV3Context) *ast.ActionV3 { } else if actCtx.OPEN_LINK() != nil { action.Type = "openLink" if str := actCtx.STRING_LITERAL(); str != nil { - action.LinkURL = unquoteString(str.GetText()) + action.LinkURL = unquoteStringLit(str) } // A dynamic address: `open_link $currentObject/URL`. if v := actCtx.VARIABLE(); v != nil { @@ -1160,7 +1160,7 @@ func buildActionV3(ctx parser.IActionExprV3Context) *ast.ActionV3 { } else if actCtx.COMPLETE_TASK() != nil { action.Type = "completeTask" if str := actCtx.STRING_LITERAL(); str != nil { - action.OutcomeValue = unquoteString(str.GetText()) + action.OutcomeValue = unquoteStringLit(str) } } @@ -1261,7 +1261,7 @@ func buildStringExprV3(ctx parser.IStringExprV3Context) string { // String literal: 'Hello {1}' if str := strCtx.STRING_LITERAL(); str != nil { - return unquoteString(str.GetText()) + return unquoteStringLit(str) } // Attribute path: Name or Entity/Attr @@ -1592,7 +1592,7 @@ func buildPropertyValueV3(ctx parser.IPropertyValueV3Context) any { pvCtx := ctx.(*parser.PropertyValueV3Context) if str := pvCtx.STRING_LITERAL(); str != nil { - return unquoteString(str.GetText()) + return unquoteStringLit(str) } // "AttrName" — a double-quoted value, used for pluggable-widget attribute // sub-properties like chart series `staticXAttribute: "StatusValue"`. Strip @@ -1679,7 +1679,7 @@ func buildDesignPropertyEntryV3(ctx parser.IDesignPropertyEntryV3Context) *ast.D return nil } - key := unquoteString(allStrings[0].GetText()) + key := unquoteStringLit(allStrings[0]) // Compound (nested): 'Spacing': ['margin-top': 'Large', 'margin-bottom': 'Medium'] if listCtx := entryCtx.DesignPropertyListV3(); listCtx != nil { @@ -1694,7 +1694,7 @@ func buildDesignPropertyEntryV3(ctx parser.IDesignPropertyEntryV3Context) *ast.D return &ast.DesignPropertyEntryV3{Key: key, Value: "off"} } if len(allStrings) >= 2 { - return &ast.DesignPropertyEntryV3{Key: key, Value: unquoteString(allStrings[1].GetText())} + return &ast.DesignPropertyEntryV3{Key: key, Value: unquoteStringLit(allStrings[1])} } return nil @@ -2024,7 +2024,7 @@ func widgetIconName(c *parser.WidgetIconV3Context) string { return buildQualifiedName(qn).String() } if str := c.STRING_LITERAL(); str != nil { - return unquoteString(str.GetText()) + return unquoteStringLit(str) } return "" } diff --git a/mdl/visitor/visitor_query.go b/mdl/visitor/visitor_query.go index 90c0f078da..4153d14344 100644 --- a/mdl/visitor/visitor_query.go +++ b/mdl/visitor/visitor_query.go @@ -597,7 +597,7 @@ func (b *Builder) ExitShowStatement(ctx *parser.ShowStatementContext) { // document, so there is no IN clause. stmt := &ast.ShowStmt{ObjectType: ast.ShowGlyphs} if sl := ctx.STRING_LITERAL(); sl != nil { - stmt.Like = unquoteString(sl.GetText()) + stmt.Like = unquoteStringLit(sl) } b.statements = append(b.statements, stmt) } else if ctx.ICON() != nil && ctx.COLLECTION() != nil { @@ -825,7 +825,7 @@ func (b *Builder) ExitDescribeStatement(ctx *parser.DescribeStatementContext) { if n := ctx.NUMBER_LITERAL(); n != nil { stmt.Qualifier = n.GetText() } else if sl := ctx.STRING_LITERAL(); sl != nil { - stmt.Qualifier = unquoteString(sl.GetText()) + stmt.Qualifier = unquoteStringLit(sl) } if stmt.Qualifier != "" { b.statements = append(b.statements, stmt) @@ -872,7 +872,7 @@ func (b *Builder) ExitDescribeStatement(ctx *parser.DescribeStatementContext) { // Handle DESCRIBE DEMO USER 'name' (uses STRING_LITERAL) if ctx.DEMO() != nil && ctx.USER() != nil { if sl := ctx.STRING_LITERAL(); sl != nil { - userName := unquoteString(sl.GetText()) + userName := unquoteStringLit(sl) b.statements = append(b.statements, &ast.DescribeStmt{ ObjectType: ast.DescribeDemoUser, Name: ast.QualifiedName{Name: userName}, @@ -886,7 +886,7 @@ func (b *Builder) ExitDescribeStatement(ctx *parser.DescribeStatementContext) { if ctx.USER() != nil && ctx.ROLE() != nil { roleName := "" if sl := ctx.STRING_LITERAL(); sl != nil { - roleName = unquoteString(sl.GetText()) + roleName = unquoteStringLit(sl) } else if iok := ctx.IdentifierOrKeyword(0); iok != nil { roleName = identifierOrKeywordText(iok) } @@ -981,7 +981,7 @@ func (b *Builder) ExitDescribeStatement(ctx *parser.DescribeStatementContext) { // parse error is the wrong lesson (mxcli-formula1 findings #8). if ctx.CONFIGURATION() != nil { if sl := ctx.STRING_LITERAL(); sl != nil { - stmt.Qualifier = unquoteString(sl.GetText()) + stmt.Qualifier = unquoteStringLit(sl) } } b.statements = append(b.statements, stmt) @@ -1045,7 +1045,7 @@ func (b *Builder) ExitDescribeStatement(ctx *parser.DescribeStatementContext) { if ctx.WIDGET() != nil && ctx.STYLING() == nil { name := "" if lit := ctx.STRING_LITERAL(); lit != nil { - name = unquoteString(lit.GetText()) + name = unquoteStringLit(lit) } else if id := ctx.IdentifierOrKeyword(0); id != nil { name = id.GetText() } @@ -1095,7 +1095,7 @@ func (b *Builder) ExitDescribeStatement(ctx *parser.DescribeStatementContext) { } coordinate := "" if sl := ctx.STRING_LITERAL(); sl != nil { - coordinate = unquoteString(sl.GetText()) + coordinate = unquoteStringLit(sl) } b.statements = append(b.statements, &ast.DescribeStmt{ ObjectType: ast.DescribeJarDependency, @@ -1110,7 +1110,7 @@ func (b *Builder) ExitDescribeStatement(ctx *parser.DescribeStatementContext) { if ctx.CONTRACT() != nil && ctx.OPERATION() != nil && ctx.FROM() != nil && ctx.OPENAPI() != nil { if sl := ctx.STRING_LITERAL(); sl != nil { b.statements = append(b.statements, &ast.DescribeContractFromOpenAPIStmt{ - SpecPath: unquoteString(sl.GetText()), + SpecPath: unquoteStringLit(sl), }) } return @@ -1306,7 +1306,7 @@ func (b *Builder) ExitDescribeStatement(ctx *parser.DescribeStatementContext) { func (b *Builder) ExitSearchStatement(ctx *parser.SearchStatementContext) { if ctx.STRING_LITERAL() != nil { b.statements = append(b.statements, &ast.SearchStmt{ - Query: unquoteString(ctx.STRING_LITERAL().GetText()), + Query: unquoteStringLit(ctx.STRING_LITERAL()), }) } } @@ -1319,7 +1319,7 @@ func (b *Builder) ExitSearchStatement(ctx *parser.SearchStatementContext) { func (b *Builder) ExitExecuteScriptStatement(ctx *parser.ExecuteScriptStatementContext) { if ctx.STRING_LITERAL() != nil { b.statements = append(b.statements, &ast.ExecuteScriptStmt{ - Path: unquoteString(ctx.STRING_LITERAL().GetText()), + Path: unquoteStringLit(ctx.STRING_LITERAL()), }) } } @@ -1346,7 +1346,7 @@ func (b *Builder) ExitSessionSetStatement(ctx *parser.SessionSetStatementContext value = valCtx.NUMBER_LITERAL().GetText() } case valCtx.STRING_LITERAL() != nil: - value = unquoteString(valCtx.STRING_LITERAL().GetText()) + value = unquoteStringLit(valCtx.STRING_LITERAL()) case valCtx.IdentifierOrKeyword() != nil: value = valCtx.IdentifierOrKeyword().GetText() } diff --git a/mdl/visitor/visitor_queue.go b/mdl/visitor/visitor_queue.go index a90fdebb0d..46a26af7e0 100644 --- a/mdl/visitor/visitor_queue.go +++ b/mdl/visitor/visitor_queue.go @@ -17,7 +17,7 @@ func (b *Builder) ExitCreateQueueStatement(ctx *parser.CreateQueueStatementConte Documentation: findDocCommentText(ctx), } if lit := ctx.STRING_LITERAL(); lit != nil { - stmt.Folder = unquoteString(lit.GetText()) + stmt.Folder = unquoteStringLit(lit) } if createStmt := findParentCreateStatement(ctx); createStmt != nil { if createStmt.OR() != nil && (createStmt.MODIFY() != nil || createStmt.REPLACE() != nil) { @@ -63,7 +63,7 @@ func queuePropertyText(pc *parser.QueuePropertyContext) string { return n.GetText() } if s := pc.STRING_LITERAL(); s != nil { - return unquoteString(s.GetText()) + return unquoteStringLit(s) } if bl := pc.BooleanLiteral(); bl != nil { return bl.GetText() diff --git a/mdl/visitor/visitor_regularexpression.go b/mdl/visitor/visitor_regularexpression.go index 3007746c94..a02df975fa 100644 --- a/mdl/visitor/visitor_regularexpression.go +++ b/mdl/visitor/visitor_regularexpression.go @@ -17,7 +17,7 @@ func (b *Builder) ExitCreateRegularExpressionStatement(ctx *parser.CreateRegular Documentation: findDocCommentText(ctx), } if lit := ctx.STRING_LITERAL(); lit != nil { - stmt.Folder = unquoteString(lit.GetText()) + stmt.Folder = unquoteStringLit(lit) } if createStmt := findParentCreateStatement(ctx); createStmt != nil { if createStmt.OR() != nil && (createStmt.MODIFY() != nil || createStmt.REPLACE() != nil) { @@ -56,7 +56,7 @@ func (b *Builder) ExitCreateRegularExpressionStatement(ctx *parser.CreateRegular // reading index 0 would echo the key back as the value. func regularExpressionPropertyText(pc *parser.RegularExpressionPropertyContext) string { if s := pc.STRING_LITERAL(); s != nil { - return unquoteString(s.GetText()) + return unquoteStringLit(s) } if bl := pc.BooleanLiteral(); bl != nil { return bl.GetText() diff --git a/mdl/visitor/visitor_rest.go b/mdl/visitor/visitor_rest.go index 8d33a73587..459469bac1 100644 --- a/mdl/visitor/visitor_rest.go +++ b/mdl/visitor/visitor_rest.go @@ -30,19 +30,21 @@ func (b *Builder) ExitCreateRestClientStatement(ctx *parser.CreateRestClientStat if iok == nil { continue } - key := strings.ToLower(identifierOrKeywordText(iok.(*parser.IdentifierOrKeywordContext))) + rawKey := identifierOrKeywordText(iok.(*parser.IdentifierOrKeywordContext)) + b.checkProperty(pc, &restClientSchema, rawKey, restClientPropertyShape(pc)) + key := strings.ToLower(rawKey) switch key { case "baseurl": if sl := pc.STRING_LITERAL(); sl != nil { - stmt.BaseUrl = unquoteString(sl.GetText()) + stmt.BaseUrl = unquoteStringLit(sl) } case "folder": if sl := pc.STRING_LITERAL(); sl != nil { - stmt.Folder = unquoteString(sl.GetText()) + stmt.Folder = unquoteStringLit(sl) } case "openapi": if sl := pc.STRING_LITERAL(); sl != nil { - stmt.OpenApiPath = unquoteString(sl.GetText()) + stmt.OpenApiPath = unquoteStringLit(sl) } case "authentication": if pc.BASIC() != nil { @@ -52,10 +54,12 @@ func (b *Builder) ExitCreateRestClientStatement(ctx *parser.CreateRestClientStat if !spOk || sp == nil || sp.IdentifierOrKeyword() == nil { continue } - subKey := strings.ToLower(identifierOrKeywordText(sp.IdentifierOrKeyword().(*parser.IdentifierOrKeywordContext))) + rawSubKey := identifierOrKeywordText(sp.IdentifierOrKeyword().(*parser.IdentifierOrKeywordContext)) + b.checkProperty(sp, &restClientBasicAuthSchema, rawSubKey, restClientPropertyShape(sp)) + subKey := strings.ToLower(rawSubKey) var val string if sl := sp.STRING_LITERAL(); sl != nil { - val = unquoteString(sl.GetText()) + val = unquoteStringLit(sl) } else if v := sp.VARIABLE(); v != nil { // $Constant reference (legacy) — keep $ prefix val = v.GetText() @@ -85,6 +89,12 @@ func (b *Builder) ExitCreateRestClientStatement(ctx *parser.CreateRestClientStat if !ok || oc == nil { continue } + for _, p := range oc.AllRestClientOpProp() { + if pc, ok := p.(*parser.RestClientOpPropContext); ok && pc != nil && pc.IdentifierOrKeyword() != nil { + key := identifierOrKeywordText(pc.IdentifierOrKeyword().(*parser.IdentifierOrKeywordContext)) + b.checkProperty(pc, &restClientOperationSchema, key, restClientOpPropShape(pc)) + } + } opDef := parseRestClientOperation(oc) stmt.Operations = append(stmt.Operations, opDef) } @@ -109,7 +119,7 @@ func parseRestClientOperation(ctx *parser.RestClientOperationContext) *ast.RestO if iok := ctx.IdentifierOrKeyword(); iok != nil { op.Name = identifierOrKeywordText(iok) } else if sl := ctx.STRING_LITERAL(); sl != nil { - op.Name = unquoteString(sl.GetText()) + op.Name = unquoteStringLit(sl) } // Documentation @@ -170,7 +180,7 @@ func parseRestClientOpProp(ctx *parser.RestClientOpPropContext, op *ast.RestOper if ctx.TEMPLATE() != nil { if sl := ctx.STRING_LITERAL(); sl != nil { op.BodyType = "template" - op.BodyVariable = unquoteString(sl.GetText()) + op.BodyVariable = unquoteStringLit(sl) } return } @@ -243,12 +253,12 @@ func parseRestClientOpProp(ctx *parser.RestClientOpPropContext, op *ast.RestOper header := ast.RestHeaderDef{} allSL := hic.AllSTRING_LITERAL() if len(allSL) >= 1 { - header.Name = unquoteString(allSL[0].GetText()) + header.Name = unquoteStringLit(allSL[0]) } if hic.PLUS() != nil { // 'prefix' + $Variable if len(allSL) >= 2 { - header.Prefix = unquoteString(allSL[1].GetText()) + header.Prefix = unquoteStringLit(allSL[1]) } if v := hic.VARIABLE(); v != nil { header.Variable = v.GetText() @@ -256,7 +266,7 @@ func parseRestClientOpProp(ctx *parser.RestClientOpPropContext, op *ast.RestOper } else if hic.VARIABLE() != nil { header.Variable = hic.VARIABLE().GetText() } else if len(allSL) >= 2 { - header.Value = unquoteString(allSL[1].GetText()) + header.Value = unquoteStringLit(allSL[1]) } op.Headers = append(op.Headers, header) } @@ -267,7 +277,7 @@ func parseRestClientOpProp(ctx *parser.RestClientOpPropContext, op *ast.RestOper if sl := ctx.STRING_LITERAL(); sl != nil { switch key { case "path": - op.Path = unquoteString(sl.GetText()) + op.Path = unquoteStringLit(sl) } return } @@ -357,7 +367,8 @@ func (b *Builder) ExitCreatePublishedRestServiceStatement(ctx *parser.CreatePubl for _, propCtx := range ctx.AllPublishedRestProperty() { pc := propCtx.(*parser.PublishedRestPropertyContext) key := identifierOrKeywordText(pc.IdentifierOrKeyword().(*parser.IdentifierOrKeywordContext)) - val := unquoteString(pc.STRING_LITERAL().GetText()) + b.checkProperty(pc, &publishedRestSchema, key, shapeString) + val := unquoteStringLit(pc.STRING_LITERAL()) switch strings.ToLower(key) { case "path": stmt.Path = val @@ -389,7 +400,7 @@ func buildPublishedRestResourceDef(rc *parser.PublishedRestResourceContext) *ast return nil } resDef := &ast.PublishedRestResourceDef{ - Name: unquoteString(rc.STRING_LITERAL().GetText()), + Name: unquoteStringLit(rc.STRING_LITERAL()), } for _, opCtx := range rc.AllPublishedRestOperation() { @@ -405,7 +416,7 @@ func buildPublishedRestResourceDef(rc *parser.PublishedRestResourceContext) *ast if pCtx := oc.PublishedRestOpPath(); pCtx != nil { pc := pCtx.(*parser.PublishedRestOpPathContext) if pc.STRING_LITERAL() != nil { - opDef.Path = strings.Trim(unquoteString(pc.STRING_LITERAL().GetText()), "/") + opDef.Path = strings.Trim(unquoteStringLit(pc.STRING_LITERAL()), "/") } } @@ -468,7 +479,7 @@ func (b *Builder) exitAlterPublishedRestServiceStatement(ctx *parser.AlterStatem for _, asnCtx := range ac.AllPublishedRestAlterAssignment() { asn := asnCtx.(*parser.PublishedRestAlterAssignmentContext) key := identifierOrKeywordText(asn.IdentifierOrKeyword().(*parser.IdentifierOrKeywordContext)) - val := unquoteString(asn.STRING_LITERAL().GetText()) + val := unquoteStringLit(asn.STRING_LITERAL()) changes[key] = val } stmt.Actions = append(stmt.Actions, &ast.PublishedRestSetAction{Changes: changes}) @@ -487,7 +498,7 @@ func (b *Builder) exitAlterPublishedRestServiceStatement(ctx *parser.AlterStatem // DROP RESOURCE 'name' if ac.DROP() != nil && ac.RESOURCE() != nil { - name := unquoteString(ac.STRING_LITERAL().GetText()) + name := unquoteStringLit(ac.STRING_LITERAL()) stmt.Actions = append(stmt.Actions, &ast.PublishedRestDropResourceAction{Name: name}) continue } diff --git a/mdl/visitor/visitor_retrieve_range.go b/mdl/visitor/visitor_retrieve_range.go new file mode 100644 index 0000000000..40dcfc605c --- /dev/null +++ b/mdl/visitor/visitor_retrieve_range.go @@ -0,0 +1,111 @@ +// SPDX-License-Identifier: Apache-2.0 + +package visitor + +import ( + "fmt" + "strconv" + "strings" + + "github.com/antlr4-go/antlr/v4" + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/grammar/parser" + "github.com/mendixlabs/mxcli/mdl/langver" +) + +// limitOneIsAList is the change of meaning of `retrieve … limit 1` (ako/mxcli#734, +// PROPOSAL_mdl_beta_syntax_freeze.md §5 item 2). +// +// In the alpha language `limit 1` without an offset binds ONE object: the writer +// stores Mendix's "First object" range. Every SQL reader expects a list of one, +// and `import from mapping` already spells the object `first` and a list of one +// `limit 1`. Under mdl 1 `limit 1` is the Custom range, a list; the object is +// spelled `first` in both versions. +var limitOneIsAList = langver.Change{ + Code: "MDL-V1-LIMIT1", + Since: langver.V1, + Old: "`retrieve … limit 1` binds a single object (Mendix's \"First object\" range, " + + "so a loop over it is CE0100 and count() or head() CE0097); write `retrieve … first` to say so", + New: "a list of one (a Custom range), and the object is written `first`", +} + +// ExitRetrieveStatement warns on `limit 1` kept at its alpha meaning. The +// statement builder, a free function, applies the same decision through +// retrieveLimitOneIsObject; this is where the note is recorded, once. +// +// It also refuses `first` on an association retrieve: Mendix gives that source +// no range, so the keyword would be dropped on the way to the model, and +// whether the result is an object or a list follows from the association. +func (b *Builder) ExitRetrieveStatement(ctx *parser.RetrieveStatementContext) { + if ctx.FIRST() != nil && isAssociationRetrieve(ctx) { + b.addError(fmt.Errorf("line %d: `first` does not apply to a retrieve over an association: Mendix gives "+ + "that source no range, and whether it binds an object or a list follows from the association. "+ + "Drop `first`, or retrieve from the database (`retrieve $x from Module.Entity where … first`)", + ctx.GetStart().GetLine())) + return + } + if isBareLimitOne(ctx) && !b.gate(limitOneIsAList, ctx) { + // `limit 1` -> `first`: the object range, spelled the way that means + // it under every version. + limit := ctx.LIMIT().GetSymbol() + b.fixLastNote(limitOneIsAList.Code, &ast.Fix{Edits: []ast.TextEdit{ + replaceSpan(limit, ctx.GetLimitExpr().GetStop(), keywordLike(limit.GetText(), "first")), + }}, "") + } +} + +// isAssociationRetrieve reports whether the retrieve's source is `$Var/Assoc`. +func isAssociationRetrieve(ctx *parser.RetrieveStatementContext) bool { + src, ok := ctx.RetrieveSource().(*parser.RetrieveSourceContext) + return ok && src != nil && src.VARIABLE() != nil +} + +// retrieveLimitOneIsObject reports whether ctx is a `limit 1` that binds an +// object: a bare `limit 1` in a script written before mdl 1. +func retrieveLimitOneIsObject(ctx *parser.RetrieveStatementContext) bool { + return isBareLimitOne(ctx) && !limitOneIsAList.Applies(languageVersionOf(ctx)) +} + +// isBareLimitOne reports whether a retrieve's range is exactly `limit 1` with no +// offset, on a source that is not an association: the one spelling whose +// meaning depends on the language version. +// +// The limit is compared as the builder stores it (source text plus the +// whitespace it keeps before `;`), which is the condition the alpha writer has +// always used to choose the object range: `limit 1` is an object, `limit (1)` +// or `limit $One` a list. +func isBareLimitOne(ctx *parser.RetrieveStatementContext) bool { + if ctx == nil || ctx.FIRST() != nil || ctx.GetOffsetExpr() != nil || ctx.GetLimitExpr() == nil { + return false + } + if isAssociationRetrieve(ctx) { + return false // an association retrieve has no range + } + limit := ctx.GetLimitExpr() + return retrieveRangeExpressionSource(limit)+retrieveLimitTrailingWhitespace(ctx, limit) == "1" +} + +// languageVersionOf reads the `mdl <n>;` header of the script tree is part of. +// The header can only be the first statement, so it is on the root. It exists +// for the statement builders, which are free functions without the Builder and +// must still build the AST of the version the script is written in. A missing +// or malformed header is mdl 0 (ExitLanguageHeader reports a malformed one). +func languageVersionOf(tree antlr.Tree) langver.Version { + for ; tree != nil; tree = tree.GetParent() { + prog, ok := tree.(*parser.ProgramContext) + if !ok { + continue + } + h, ok := prog.LanguageHeader().(*parser.LanguageHeaderContext) + if !ok || h == nil || h.IDENTIFIER() == nil || h.NUMBER_LITERAL() == nil || + !strings.EqualFold(h.IDENTIFIER().GetText(), "mdl") { + return langver.V0 + } + n, err := strconv.Atoi(h.NUMBER_LITERAL().GetText()) + if v := langver.Version(n); err == nil && v.Known() { + return v + } + return langver.V0 + } + return langver.V0 +} diff --git a/mdl/visitor/visitor_scheduledevent.go b/mdl/visitor/visitor_scheduledevent.go index 8c945376c7..d412bc95cd 100644 --- a/mdl/visitor/visitor_scheduledevent.go +++ b/mdl/visitor/visitor_scheduledevent.go @@ -23,7 +23,7 @@ func (b *Builder) ExitCreateScheduledEventStatement(ctx *parser.CreateScheduledE Documentation: findDocCommentText(ctx), } if lit := ctx.STRING_LITERAL(); lit != nil { - stmt.Folder = unquoteString(lit.GetText()) + stmt.Folder = unquoteStringLit(lit) } if createStmt := findParentCreateStatement(ctx); createStmt != nil { if createStmt.OR() != nil && (createStmt.MODIFY() != nil || createStmt.REPLACE() != nil) { @@ -102,7 +102,7 @@ func scheduledEventPropertyText(pc *parser.ScheduledEventPropertyContext) string return n.GetText() } if s := pc.STRING_LITERAL(); s != nil { - return unquoteString(s.GetText()) + return unquoteStringLit(s) } if bl := pc.BooleanLiteral(); bl != nil { return bl.GetText() diff --git a/mdl/visitor/visitor_security.go b/mdl/visitor/visitor_security.go index 4a3b156c76..eaf0a4ccb2 100644 --- a/mdl/visitor/visitor_security.go +++ b/mdl/visitor/visitor_security.go @@ -7,29 +7,32 @@ import ( "github.com/mendixlabs/mxcli/mdl/grammar/parser" ) -// ExitCreateModuleRoleStatement handles CREATE MODULE ROLE Module.RoleName [DESCRIPTION '...'] +// ExitCreateModuleRoleStatement handles CREATE [OR MODIFY] MODULE ROLE Module.RoleName [DESCRIPTION '...'] func (b *Builder) ExitCreateModuleRoleStatement(ctx *parser.CreateModuleRoleStatementContext) { qn := ctx.QualifiedName() if qn == nil { return } stmt := &ast.CreateModuleRoleStmt{ - Name: buildQualifiedName(qn), - CreateOrModify: ctx.MODIFY() != nil, + Name: buildQualifiedName(qn), + } + if createStmt := findParentCreateStatement(ctx); createStmt != nil { + stmt.CreateOrModify = createStmt.OR() != nil && (createStmt.MODIFY() != nil || b.replaceMeansModify(createStmt)) } if ctx.DESCRIPTION() != nil { if sl := ctx.STRING_LITERAL(); sl != nil { - stmt.Description = unquoteString(sl.GetText()) + stmt.Description = unquoteStringLit(sl) } } b.statements = append(b.statements, stmt) } -// ExitDropModuleRoleStatement handles DROP MODULE ROLE Module.RoleName +// ExitDropModuleRoleStatement handles DROP MODULE ROLE [IF EXISTS] Module.RoleName func (b *Builder) ExitDropModuleRoleStatement(ctx *parser.DropModuleRoleStatementContext) { if qn := ctx.QualifiedName(); qn != nil { b.statements = append(b.statements, &ast.DropModuleRoleStmt{ - Name: buildQualifiedName(qn), + IfExists: ctx.IfExists() != nil, + Name: buildQualifiedName(qn), }) } } @@ -47,8 +50,9 @@ func (b *Builder) ExitCreateUserRoleStatement(ctx *parser.CreateUserRoleStatemen } // Check parent createStatement for OR MODIFY + // `or replace` means `or modify` from mdl 1 on (roleReplaceIsModify). if createStmt := findParentCreateStatement(ctx); createStmt != nil { - if createStmt.OR() != nil && createStmt.MODIFY() != nil { + if createStmt.OR() != nil && (createStmt.MODIFY() != nil || b.replaceMeansModify(createStmt)) { stmt.CreateOrModify = true } } @@ -91,7 +95,7 @@ func (b *Builder) ExitDropUserRoleStatement(ctx *parser.DropUserRoleStatementCon } else if sl := ctx.STRING_LITERAL(); sl != nil { // Quoted form (FINDINGS #5): DROP USER ROLE 'User' — matches DESCRIBE, which // also accepts quotes. Bare and quoted are now consistent across both. - name = unquoteString(sl.GetText()) + name = unquoteStringLit(sl) } if name != "" { b.statements = append(b.statements, &ast.DropUserRoleStmt{ @@ -129,7 +133,7 @@ func (b *Builder) ExitGrantEntityAccessStatement(ctx *parser.GrantEntityAccessSt // Parse WHERE clause if ctx.WHERE() != nil { if sl := ctx.STRING_LITERAL(); sl != nil { - stmt.XPathConstraint = unquoteString(sl.GetText()) + stmt.XPathConstraint = unquoteStringLit(sl) } } @@ -441,13 +445,14 @@ func (b *Builder) ExitCreateDemoUserStatement(ctx *parser.CreateDemoUserStatemen } stmt := &ast.CreateDemoUserStmt{ - UserName: unquoteString(sls[0].GetText()), - Password: unquoteString(sls[1].GetText()), + UserName: unquoteStringLit(sls[0]), + Password: unquoteStringLit(sls[1]), } // Check parent createStatement for OR MODIFY + // `or replace` means `or modify` from mdl 1 on (roleReplaceIsModify). if createStmt := findParentCreateStatement(ctx); createStmt != nil { - if createStmt.OR() != nil && createStmt.MODIFY() != nil { + if createStmt.OR() != nil && (createStmt.MODIFY() != nil || b.replaceMeansModify(createStmt)) { stmt.CreateOrModify = true } } @@ -469,7 +474,7 @@ func (b *Builder) ExitCreateDemoUserStatement(ctx *parser.CreateDemoUserStatemen func (b *Builder) ExitDropDemoUserStatement(ctx *parser.DropDemoUserStatementContext) { if sl := ctx.STRING_LITERAL(); sl != nil { b.statements = append(b.statements, &ast.DropDemoUserStmt{ - UserName: unquoteString(sl.GetText()), + UserName: unquoteStringLit(sl), IfExists: ctx.IfExists() != nil, }) } diff --git a/mdl/visitor/visitor_settings.go b/mdl/visitor/visitor_settings.go index a9463908bf..708e26a267 100644 --- a/mdl/visitor/visitor_settings.go +++ b/mdl/visitor/visitor_settings.go @@ -21,17 +21,17 @@ func (b *Builder) ExitAlterSettingsClause(ctx *parser.AlterSettingsClauseContext stmt.DropConstant = true allStrings := ctx.AllSTRING_LITERAL() if len(allStrings) > 0 { - stmt.ConstantId = unquoteString(allStrings[0].GetText()) + stmt.ConstantId = unquoteStringLit(allStrings[0]) } if ctx.IN() != nil && ctx.CONFIGURATION() != nil && len(allStrings) > 1 { - stmt.ConfigName = unquoteString(allStrings[1].GetText()) + stmt.ConfigName = unquoteStringLit(allStrings[1]) } } else if ctx.CONSTANT() != nil { // ALTER SETTINGS CONSTANT 'name' (VALUE 'value' | DROP) [IN CONFIGURATION 'cfg'] stmt.Section = "constant" allStrings := ctx.AllSTRING_LITERAL() if len(allStrings) > 0 { - stmt.ConstantId = unquoteString(allStrings[0].GetText()) + stmt.ConstantId = unquoteStringLit(allStrings[0]) } if ctx.DROP() != nil { stmt.DropConstant = true @@ -40,14 +40,14 @@ func (b *Builder) ExitAlterSettingsClause(ctx *parser.AlterSettingsClauseContext } // Check for IN CONFIGURATION 'name' if ctx.IN() != nil && ctx.CONFIGURATION() != nil && len(allStrings) > 1 { - stmt.ConfigName = unquoteString(allStrings[1].GetText()) + stmt.ConfigName = unquoteStringLit(allStrings[1]) } } else if ctx.CONFIGURATION() != nil { // ALTER SETTINGS CONFIGURATION 'name' Key = Value, ... stmt.Section = "configuration" allStrings := ctx.AllSTRING_LITERAL() if len(allStrings) > 0 { - stmt.ConfigName = unquoteString(allStrings[0].GetText()) + stmt.ConfigName = unquoteStringLit(allStrings[0]) } for _, assignCtx := range ctx.AllSettingsAssignment() { assign, ok := assignCtx.(*parser.SettingsAssignmentContext) @@ -75,7 +75,7 @@ func (b *Builder) ExitAlterSettingsClause(ctx *parser.AlterSettingsClauseContext stmt.ModifyGroup = ctx.MODIFY() != nil && !stmt.UpsertGroup stmt.RemoveGroup = ctx.REMOVE() != nil if all := ctx.AllSTRING_LITERAL(); len(all) > 0 { - stmt.GroupName = unquoteString(all[0].GetText()) + stmt.GroupName = unquoteStringLit(all[0]) } collectSettingsItemOptions(ctx.SettingsItemOptions(), stmt.Properties) } else if ctx.SettingsSection() != nil && (ctx.ADD() != nil || ctx.MODIFY() != nil || ctx.REMOVE() != nil) { @@ -88,7 +88,7 @@ func (b *Builder) ExitAlterSettingsClause(ctx *parser.AlterSettingsClauseContext stmt.ModifyLanguage = ctx.MODIFY() != nil && !stmt.UpsertLanguage stmt.RemoveLanguage = ctx.REMOVE() != nil if all := ctx.AllSTRING_LITERAL(); len(all) > 0 { - stmt.LanguageCode = unquoteString(all[0].GetText()) + stmt.LanguageCode = unquoteStringLit(all[0]) } collectSettingsItemOptions(ctx.SettingsItemOptions(), stmt.Properties) } else if ctx.SettingsSection() != nil { @@ -122,7 +122,7 @@ func (b *Builder) ExitCreateConfigurationStatement(ctx *parser.CreateConfigurati } if sl := ctx.STRING_LITERAL(); sl != nil { - stmt.Name = unquoteString(sl.GetText()) + stmt.Name = unquoteStringLit(sl) } // CREATE OR MODIFY / OR REPLACE, read off the shared create wrapper. @@ -178,7 +178,7 @@ func collectSettingsItemOptions(opts parser.ISettingsItemOptionsContext, into ma // settingsValueText extracts the string value from a SettingsValue context. func settingsValueText(ctx *parser.SettingsValueContext) string { if sl := ctx.STRING_LITERAL(); sl != nil { - return unquoteString(sl.GetText()) + return unquoteStringLit(sl) } if nl := ctx.NUMBER_LITERAL(); nl != nil { return nl.GetText() @@ -195,7 +195,7 @@ func settingsValueText(ctx *parser.SettingsValueContext) string { // settingsValueToInterface extracts a typed value from a SettingsValue context. func settingsValueToInterface(ctx *parser.SettingsValueContext) any { if sl := ctx.STRING_LITERAL(); sl != nil { - return unquoteString(sl.GetText()) + return unquoteStringLit(sl) } if nl := ctx.NUMBER_LITERAL(); nl != nil { if v, err := strconv.ParseInt(nl.GetText(), 10, 64); err == nil { diff --git a/mdl/visitor/visitor_sql.go b/mdl/visitor/visitor_sql.go index 160f9298d5..6fa0e98d90 100644 --- a/mdl/visitor/visitor_sql.go +++ b/mdl/visitor/visitor_sql.go @@ -42,7 +42,7 @@ func (b *Builder) ExitSqlConnect(ctx *parser.SqlConnectContext) { } b.statements = append(b.statements, &ast.SQLConnectStmt{ Driver: words[0], - DSN: unquoteString(ctx.STRING_LITERAL().GetText()), + DSN: unquoteStringLit(ctx.STRING_LITERAL()), Alias: words[1], }) } @@ -105,7 +105,7 @@ func (b *Builder) ExitImportFromQuery(ctx *parser.ImportFromQueryContext) { } var query string if sl := ctx.STRING_LITERAL(); sl != nil { - query = unquoteString(sl.GetText()) + query = unquoteStringLit(sl) } else if ds := ctx.DOLLAR_STRING(); ds != nil { s := ds.GetText() if len(s) >= 4 && strings.HasPrefix(s, "$$") && strings.HasSuffix(s, "$$") { diff --git a/mdl/visitor/visitor_strict_properties.go b/mdl/visitor/visitor_strict_properties.go new file mode 100644 index 0000000000..75c0ae8eae --- /dev/null +++ b/mdl/visitor/visitor_strict_properties.go @@ -0,0 +1,393 @@ +// SPDX-License-Identifier: Apache-2.0 + +package visitor + +import ( + "fmt" + "strings" + + "github.com/antlr4-go/antlr/v4" + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/grammar/parser" + "github.com/mendixlabs/mxcli/mdl/langver" + "github.com/mendixlabs/mxcli/mdl/suggest" +) + +// ADR-0010 R11: an unknown property key, or a value whose shape its key does +// not take, is an error. The grammar accepts any `Key: value` in these lists, +// and the visitors read the keys they know and drop the rest, so a typo such +// as `pathh: '/users'` parsed, checked and executed with the path missing. +// A value was also read by its shape rather than its key: `Response: json from +// $X` set the request body. +// +// Both are new rejections, tied to the header (ADR-0011): under mdl 0 the list +// is read as before and each property warns. The OData lists already refuse +// an unknown key (MDL-ODATA01); these are the lists that did not (#732). + +var unknownPropertyKey = langver.Change{ + Code: "MDL-V1-PROP", + Since: langver.V1, + Old: "an unknown property key is accepted and ignored", + New: "an error", +} + +var misshapedPropertyValue = langver.Change{ + Code: "MDL-V1-PROPVALUE", + Since: langver.V1, + Old: "a value its key does not take is accepted, and ignored or read by its shape", + New: "an error", +} + +// propShape is one shape a property value can have, in the words an error +// message uses for it. +type propShape string + +const ( + shapeString propShape = "a string ('…')" + shapeDollar propShape = "a $$…$$ block" + shapeName propShape = "a name" + shapeQName propShape = "a qualified name (Module.Name)" + shapeNumber propShape = "a number" + shapeBool propShape = "true or false" + shapeVarList propShape = `a variable list ("Key": String, …)` + shapeVariable propShape = "a $variable" + shapeConstant propShape = "a constant (@Module.Name)" + shapeNone propShape = "none" + shapeBasic propShape = "basic (Username: …, Password: …)" + shapeMethod propShape = "an HTTP method (get, post, put, patch, delete)" + shapeParamList propShape = "a parameter list ($name: Type, …)" + shapeHeaderList propShape = "a header list ('Name' = value, …)" + shapeTemplate propShape = "template '…'" + shapeMapping propShape = "mapping Module.Entity { … }" + shapeJSONFrom propShape = "json from $var" + shapeFileFrom propShape = "file from $var" + shapeJSONAs propShape = "json as $var" + shapeStringAs propShape = "string as $var" + shapeFileAs propShape = "file as $var" + shapeStatusAs propShape = "status as $var" + shapeOther propShape = "some other value" +) + +// propKey is one key a property list reads, and the shapes it takes. +type propKey struct { + name string // the spelling the syntax help and describe use + shapes []propShape +} + +// propSchema is the keys one property list reads. +type propSchema struct { + on string // what the list belongs to, for the message: "a REST client operation" + keys []propKey +} + +func (s *propSchema) lookup(key string) (propKey, bool) { + for _, k := range s.keys { + if strings.EqualFold(k.name, key) { + return k, true + } + } + return propKey{}, false +} + +func (s *propSchema) names() []string { + out := make([]string, len(s.keys)) + for i, k := range s.keys { + out[i] = k.name + } + return out +} + +// checkProperty reports a property whose key s does not read, or whose value +// has a shape the key does not take: an error under mdl 1, a warning under +// mdl 0. It reports nothing for a well-formed property. The caller reads the +// property as it always has either way; under mdl 1 the error stops the script. +func (b *Builder) checkProperty(at antlr.ParserRuleContext, s *propSchema, key string, shape propShape) { + k, ok := s.lookup(key) + if !ok { + detail := fmt.Sprintf("unknown property '%s' on %s", key, s.on) + if near := suggest.Closest(key, s.names()); near != "" { + detail += fmt.Sprintf(" — did you mean '%s'?", near) + } else { + detail += "." + } + detail += " Known properties: " + strings.Join(s.names(), ", ") + b.strict(unknownPropertyKey, at, detail) + return + } + for _, want := range k.shapes { + if want == shape { + return + } + } + takes := make([]string, len(k.shapes)) + for i, sh := range k.shapes { + takes[i] = string(sh) + } + b.strict(misshapedPropertyValue, at, fmt.Sprintf("property '%s' on %s takes %s, not %s", + k.name, s.on, strings.Join(takes, " or "), shape)) +} + +// strict is gate for a new rejection whose message names the construct: an +// error from c.Since on, a warning with that detail before it. +func (b *Builder) strict(c langver.Change, at antlr.ParserRuleContext, detail string) { + line := 0 + if at != nil && at.GetStart() != nil { + line = at.GetStart().GetLine() + } + if c.Applies(b.langVersion) { + b.addError(fmt.Errorf("line %d: %s", line, detail)) + return + } + b.langNotes = append(b.langNotes, ast.LanguageNote{ + Line: line, + Code: c.Code, + Message: detail + "; " + c.Warning(b.langVersion), + }) +} + +// --------------------------------------------------------------------------- +// The schemas. Each lists exactly what its visitor reads, plus what describe +// writes for it, so describe output is accepted under mdl 1. +// --------------------------------------------------------------------------- + +var ( + textShapes = []propShape{shapeString, shapeDollar} + nameShapes = []propShape{shapeName, shapeString} +) + +var restClientSchema = propSchema{on: "a REST client", keys: []propKey{ + {"BaseUrl", []propShape{shapeString}}, + {"Authentication", []propShape{shapeNone, shapeBasic}}, + {"OpenApi", []propShape{shapeString}}, + {"Folder", []propShape{shapeString}}, +}} + +var restClientBasicAuthSchema = propSchema{on: "REST client basic authentication", keys: []propKey{ + {"Username", []propShape{shapeString, shapeConstant, shapeVariable}}, + {"Password", []propShape{shapeString, shapeConstant, shapeVariable}}, +}} + +var restClientOperationSchema = propSchema{on: "a REST client operation", keys: []propKey{ + {"Method", []propShape{shapeMethod}}, + {"Path", []propShape{shapeString}}, + {"Parameters", []propShape{shapeParamList}}, + {"Query", []propShape{shapeParamList}}, + {"Headers", []propShape{shapeHeaderList}}, + {"Body", []propShape{shapeJSONFrom, shapeFileFrom, shapeTemplate, shapeMapping}}, + {"Timeout", []propShape{shapeNumber}}, + {"Response", []propShape{shapeNone, shapeJSONAs, shapeStringAs, shapeFileAs, shapeStatusAs, shapeMapping}}, +}} + +var publishedRestSchema = propSchema{on: "a published REST service", keys: []propKey{ + {"Path", []propShape{shapeString}}, + {"Version", []propShape{shapeString}}, + {"ServiceName", []propShape{shapeString}}, + {"Folder", []propShape{shapeString}}, +}} + +var businessEventServiceSchema = propSchema{on: "a business event service", keys: []propKey{ + {"ServiceName", []propShape{shapeString}}, + {"EventNamePrefix", []propShape{shapeString}}, + {"Folder", []propShape{shapeString}}, +}} + +// The model visitor reads each value by one alternative only: a Provider +// written as a string was dropped. +var modelSchema = propSchema{on: "a model", keys: []propKey{ + {"Provider", []propShape{shapeName}}, + {"Key", []propShape{shapeQName}}, + {"DisplayName", []propShape{shapeString}}, + {"KeyName", []propShape{shapeString}}, + {"KeyId", []propShape{shapeString}}, + {"Environment", []propShape{shapeString}}, + {"ResourceName", []propShape{shapeString}}, + {"DeepLinkURL", []propShape{shapeString}}, +}} + +var knowledgeBaseSchema = propSchema{on: "a knowledge base", keys: []propKey{ + {"Provider", nameShapes}, + {"Key", []propShape{shapeQName}}, + {"ModelDisplayName", []propShape{shapeString}}, + {"ModelName", []propShape{shapeString}}, + {"KeyName", []propShape{shapeString}}, + {"KeyId", []propShape{shapeString}}, + {"Environment", []propShape{shapeString}}, + {"DeepLinkURL", []propShape{shapeString}}, +}} + +var consumedMCPServiceSchema = propSchema{on: "a consumed MCP service", keys: []propKey{ + {"ProtocolVersion", nameShapes}, + {"Version", []propShape{shapeString}}, + {"ConnectionTimeoutSeconds", []propShape{shapeNumber}}, + {"Documentation", textShapes}, +}} + +var agentSchema = propSchema{on: "an agent", keys: []propKey{ + {"UsageType", nameShapes}, + {"Description", textShapes}, + {"Model", []propShape{shapeQName}}, + {"Entity", []propShape{shapeQName}}, + {"Variables", []propShape{shapeVarList}}, + {"MaxTokens", []propShape{shapeNumber}}, + {"ToolChoice", nameShapes}, + {"Temperature", []propShape{shapeNumber}}, + {"TopP", []propShape{shapeNumber}}, + {"SystemPrompt", textShapes}, + {"UserPrompt", textShapes}, +}} + +var agentMCPServiceSchema = propSchema{on: "an agent's mcp service block", keys: []propKey{ + {"Enabled", []propShape{shapeBool}}, + {"Description", textShapes}, +}} + +// ToolType and Document are what describe writes for a tool; create does not +// read them yet, the tool's kind and target coming from the block itself. +var agentToolSchema = propSchema{on: "an agent's tool block", keys: []propKey{ + {"Enabled", []propShape{shapeBool}}, + {"Description", textShapes}, + {"ToolType", []propShape{shapeName}}, + {"Document", []propShape{shapeQName}}, +}} + +var agentKnowledgeBaseSchema = propSchema{on: "an agent's knowledge base block", keys: []propKey{ + {"Source", []propShape{shapeQName}}, + {"Collection", []propShape{shapeString}}, + {"Description", textShapes}, + {"MaxResults", []propShape{shapeNumber}}, + {"Enabled", []propShape{shapeBool}}, +}} + +// --------------------------------------------------------------------------- +// Value shapes, read off the parse tree. +// --------------------------------------------------------------------------- + +// modelPropertyShape is the shape of a `Key: value` in an agent-editor list. +func modelPropertyShape(pc *parser.ModelPropertyContext) propShape { + idents := pc.AllIdentifierOrKeyword() + switch { + case pc.QualifiedName() != nil: + return shapeQName + case pc.STRING_LITERAL() != nil: + return shapeString + case pc.DOLLAR_STRING() != nil: + return shapeDollar + case pc.NUMBER_LITERAL() != nil: + return shapeNumber + case pc.BooleanLiteral() != nil: + return shapeBool + case pc.VariableDefList() != nil: + return shapeVarList + case len(idents) > 1: + // `Enabled: true` can reach the name alternative, and reads as true. + if t := strings.ToLower(idents[1].GetText()); t == "true" || t == "false" { + return shapeBool + } + return shapeName + } + return shapeOther +} + +// checkModelProperties checks every property of an agent-editor list. +func (b *Builder) checkModelProperties(props []parser.IModelPropertyContext, s *propSchema) { + for _, p := range props { + pc, ok := p.(*parser.ModelPropertyContext) + if !ok || pc == nil { + continue + } + idents := pc.AllIdentifierOrKeyword() + if len(idents) == 0 { + continue + } + b.checkProperty(pc, s, idents[0].GetText(), modelPropertyShape(pc)) + } +} + +// checkAgentBodyBlock checks a TOOL, MCP SERVICE or KNOWLEDGE BASE block of +// an agent, in create agent and in alter agent … add. +func (b *Builder) checkAgentBodyBlock(blk *parser.AgentBodyBlockContext) { + switch { + case blk.MCP() != nil && blk.SERVICE() != nil: + b.checkModelProperties(blk.AllModelProperty(), &agentMCPServiceSchema) + case blk.KNOWLEDGE() != nil && blk.BASE() != nil: + b.checkModelProperties(blk.AllModelProperty(), &agentKnowledgeBaseSchema) + case blk.TOOL() != nil: + b.checkModelProperties(blk.AllModelProperty(), &agentToolSchema) + } +} + +// restClientPropertyShape is the shape of a REST client's `Key: value`. +func restClientPropertyShape(pc *parser.RestClientPropertyContext) propShape { + switch { + case pc.BASIC() != nil: + return shapeBasic + case pc.NONE() != nil: + return shapeNone + case pc.STRING_LITERAL() != nil: + return shapeString + case pc.VARIABLE() != nil: + return shapeVariable + case pc.AT() != nil: + return shapeConstant + } + return shapeOther +} + +// restClientOpPropShape is the shape of a REST client operation's `Key: value`. +func restClientOpPropShape(pc *parser.RestClientOpPropContext) propShape { + switch { + case pc.RestHttpMethod() != nil: + return shapeMethod + case pc.MAPPING() != nil: + return shapeMapping + case pc.TEMPLATE() != nil: + return shapeTemplate + case pc.VARIABLE() != nil: + kind := "json" + switch { + case pc.FILE_KW() != nil: + kind = "file" + case pc.STRING_TYPE() != nil: + kind = "string" + case pc.STATUS() != nil: + kind = "status" + } + if pc.FROM() != nil { + return propShape(kind + " from $var") + } + return propShape(kind + " as $var") + case pc.NONE() != nil: + return shapeNone + case len(pc.AllRestClientParamItem()) > 0: + return shapeParamList + case len(pc.AllRestClientHeaderItem()) > 0: + return shapeHeaderList + case pc.STRING_LITERAL() != nil: + return shapeString + case pc.NUMBER_LITERAL() != nil: + return shapeNumber + } + return shapeOther +} + +// odataPropertyShape is the shape of a `Key: value` in an OData-style list, +// which the business event service shares. +func odataPropertyShape(pc *parser.OdataPropertyAssignmentContext) propShape { + v, ok := pc.OdataPropertyValue().(*parser.OdataPropertyValueContext) + if !ok || v == nil { + return shapeOther + } + switch { + case v.STRING_LITERAL() != nil: + return shapeString + case v.NUMBER_LITERAL() != nil: + return shapeNumber + case v.TRUE() != nil || v.FALSE() != nil: + return shapeBool + case v.AT() != nil: + return shapeConstant + case v.QualifiedName() != nil && v.MICROFLOW() == nil: + return shapeQName + } + return shapeOther +} diff --git a/mdl/visitor/visitor_strict_terminators.go b/mdl/visitor/visitor_strict_terminators.go new file mode 100644 index 0000000000..f5cf67455f --- /dev/null +++ b/mdl/visitor/visitor_strict_terminators.go @@ -0,0 +1,87 @@ +// SPDX-License-Identifier: Apache-2.0 + +package visitor + +import ( + "fmt" + + "github.com/antlr4-go/antlr/v4" + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/grammar/parser" + "github.com/mendixlabs/mxcli/mdl/langver" +) + +// ADR-0010 R11: `;` terminates every statement, and the SQL*Plus `/` line is +// not a terminator. The grammar keeps both optional (`SEMICOLON? SLASH?`) so a +// headerless script parses exactly as before; the version decides here. + +// semicolonRequired is the new rejection of a statement without `;`. +var semicolonRequired = langver.Change{ + Code: "MDL-V1-SEMI", + Since: langver.V1, + Old: "a statement without a terminating `;` is accepted", + New: "an error: every statement ends with `;`", +} + +// slashIsNotATerminator is the new rejection of the SQL*Plus `/` line. +var slashIsNotATerminator = langver.Change{ + Code: "MDL-V1-SLASH", + Since: langver.V1, + Old: "a `/` after a statement is accepted as a terminator (SQL*Plus style)", + New: "an error: `;` is the only statement terminator", +} + +// ExitStatement applies R11's terminator rules to one top-level statement. +// +// The terminators are read off the statement's last tokens rather than its +// own SEMICOLON and SLASH: the microflow, nanoflow and workflow rules end in +// `SEMICOLON? SLASH?` themselves, and `create java action … as $$…$$;` in +// `SEMICOLON?`, so either can belong to the inner rule. +func (b *Builder) ExitStatement(ctx *parser.StatementContext) { + last := lastTerminals(ctx, 2) + if len(last) == 0 { + return + } + var slash antlr.Token + if last[0].GetTokenType() == parser.MDLParserSLASH { + slash, last = last[0], last[1:] + } + if len(last) > 0 && last[0].GetTokenType() != parser.MDLParserSEMICOLON { + if b.gate(semicolonRequired, ctx) { + b.addError(fmt.Errorf("line %d: the statement ending at %q has no terminating `;`: "+ + "under %s every statement ends with `;`", last[0].GetLine(), last[0].GetText(), b.langVersion)) + } else { + b.fixLastNote(semicolonRequired.Code, &ast.Fix{Edits: []ast.TextEdit{insertAt(last[0].GetStop()+1, ";")}}, "") + } + } + if slash != nil { + if b.gate(slashIsNotATerminator, ctx) { + b.addError(fmt.Errorf("line %d: `/` is not a statement terminator under %s; end the statement "+ + "with `;` and delete the `/` line", slash.GetLine(), b.langVersion)) + } else { + b.fixLastNote(slashIsNotATerminator.Code, &ast.Fix{Edits: []ast.TextEdit{slashLineFix(slash)}}, "") + } + } +} + +// lastTerminals returns up to n of the tree's last tokens, the last first. +func lastTerminals(tree antlr.Tree, n int) []antlr.Token { + var out []antlr.Token + var walk func(antlr.Tree) + walk = func(t antlr.Tree) { + if len(out) == n { + return + } + if tn, ok := t.(antlr.TerminalNode); ok { + if tok := tn.GetSymbol(); tok != nil && tok.GetTokenType() != antlr.TokenEOF { + out = append(out, tok) + } + return + } + for i := t.GetChildCount() - 1; i >= 0 && len(out) < n; i-- { + walk(t.GetChild(i)) + } + } + walk(tree) + return out +} diff --git a/mdl/visitor/visitor_string_escapes.go b/mdl/visitor/visitor_string_escapes.go new file mode 100644 index 0000000000..ceb08eb24a --- /dev/null +++ b/mdl/visitor/visitor_string_escapes.go @@ -0,0 +1,161 @@ +// SPDX-License-Identifier: Apache-2.0 + +package visitor + +import ( + "sort" + "strings" + + "github.com/antlr4-go/antlr/v4" + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/grammar/parser" + "github.com/mendixlabs/mxcli/mdl/langver" +) + +// backslashIsLiteral is ADR-0010 R11's string rule: a doubled apostrophe is the only escape in +// a string literal. Under mdl 0 a backslash also escapes — `\n` is a newline, +// `\'` an apostrophe, `\\` one backslash — which contradicts Mendix's own +// expressions and makes `'C:\temp'` a tab. It changes what text means, so it is +// tied to the header (#732). +var backslashIsLiteral = langver.Change{ + Code: "MDL-V1-ESCAPE", + Since: langver.V1, + Old: "a backslash in a string literal is an escape (`\\n` is a newline, `\\t` a tab, `\\'` an apostrophe, `\\\\` one backslash)", + New: "an ordinary character, as in a Mendix expression; `''` is the only escape", +} + +// newScriptStream is the character stream a script is lexed from. Whether a +// backslash escapes decides where a string literal ends, so the rule is fixed +// here, from the header, before the first token (see StrictEscapeStream). +func newScriptStream(input string) antlr.CharStream { + var is antlr.CharStream = antlr.NewInputStream(input) + if backslashIsLiteral.Applies(langver.ScanHeader(input)) { + is = &parser.StrictEscapeStream{CharStream: is} + } + return is +} + +// unquoteStringLit is the value of a STRING_LITERAL (a terminal node, or a +// rule whose text is one), read under the escape rule it was lexed with. +func unquoteStringLit(n interface{ GetText() string }) string { + text := n.GetText() + if !lexedWithStrictEscapes(n) { + return unquoteString(text) + } + if len(text) >= 2 && text[0] == '\'' && text[len(text)-1] == '\'' { + text = text[1 : len(text)-1] + } + return strings.ReplaceAll(text, "''", "'") +} + +func lexedWithStrictEscapes(n any) bool { + var tok antlr.Token + switch x := n.(type) { + case antlr.TerminalNode: + tok = x.GetSymbol() + case antlr.ParserRuleContext: + tok = x.GetStart() + } + return tok != nil && parser.HasStrictEscapes(tok.GetInputStream()) +} + +// noteBackslashEscapes records MDL-V1-ESCAPE for every string literal of an +// mdl 0 script whose value would differ under mdl 1: one holding an escape +// unquoteString interprets. A backslash before any other character is kept +// as written under both, so it is not reported. +func (b *Builder) noteBackslashEscapes(tokens []antlr.Token) { + if backslashIsLiteral.Applies(b.langVersion) { + return + } + added := false + for _, t := range tokens { + if t.GetTokenType() != parser.MDLLexerSTRING_LITERAL || !hasInterpretedEscape(t.GetText()) { + continue + } + b.langNotes = append(b.langNotes, ast.LanguageNote{ + Line: t.GetLine(), + Code: backslashIsLiteral.Code, + Message: "the string " + t.GetText() + ": " + backslashIsLiteral.Warning(b.langVersion), + }) + b.langNotes[len(b.langNotes)-1].Fix, b.langNotes[len(b.langNotes)-1].NoFix = b.escapeFix(t) + added = true + } + if added { + sort.SliceStable(b.langNotes, func(i, j int) bool { return b.langNotes[i].Line < b.langNotes[j].Line }) + } +} + +// hasInterpretedEscape reports whether an mdl 0 string literal's text holds a +// backslash escape unquoteString turns into something else. +func hasInterpretedEscape(lit string) bool { + for i := 0; i+1 < len(lit); i++ { + if lit[i] != '\\' { + continue + } + switch lit[i+1] { + case 'n', 'r', 't', '\\', '\'': + return true + } + } + return false +} + +// VisitTerminal collects the string literals of the tree for escapeFix. +func (b *Builder) VisitTerminal(node antlr.TerminalNode) { + if t := node.GetSymbol(); t != nil && t.GetTokenType() == parser.MDLLexerSTRING_LITERAL { + if b.stringLits == nil { + b.stringLits = map[int]antlr.TerminalNode{} + } + b.stringLits[t.GetTokenIndex()] = node + } +} + +// escapeFix is the rewrite that keeps an mdl 0 string literal's meaning under +// mdl 1, where a backslash is an ordinary character. +// +// What the literal means under mdl 0 depends on where it is. On its own (a +// name, a caption) and in an expression the builder re-renders, it is its +// unescaped value, so the rewrite writes that value with a doubled apostrophe as the only +// escape. In an expression the builder stores as written — one that spans +// lines (shouldPreserveExpressionSource) — mdl 0 already passes the backslash +// through to Mendix, exactly as mdl 1 does, so nothing changes. +// +// An escaped line break in a re-rendered expression has no rewrite: writing the +// break into the source makes the builder store the expression as written, +// which is not always what it wrote before. A log message is the measured case +// (mdl-examples/bug-tests/264-log-node-expression-roundtrip.mdl): a lone +// literal is the message template, the stored expression a `{1}` parameter. +func (b *Builder) escapeFix(t antlr.Token) (*ast.Fix, string) { + requoted := requoteForV1(t.GetText()) + rewrite := &ast.Fix{Edits: []ast.TextEdit{replaceSpan(t, t, requoted)}} + node := b.stringLits[t.GetTokenIndex()] + if node == nil { + return rewrite, "" + } + var top antlr.ParserRuleContext + for p := node.GetParent(); p != nil; p = p.GetParent() { + if e, ok := p.(*parser.ExpressionContext); ok { + top = e + } + } + if top == nil { + return rewrite, "" + } + source := strings.TrimSpace(extractExpressionText(top)) + if shouldPreserveExpressionSource(source) { + return &ast.Fix{}, "" + } + if strings.ContainsAny(requoted, "\r\n") { + return nil, "under mdl 1 the line break is written into the string itself, which makes the expression " + + "one that is stored as written rather than re-rendered, and that can change what it builds " + + "(a log message becomes a `{1}` parameter); rewrite it by hand" + } + return rewrite, "" +} + +// requoteForV1 writes an mdl 0 string literal so that it has the same value +// under mdl 1, where a backslash is an ordinary character and a doubled apostrophe the only +// escape. +func requoteForV1(lit string) string { + return "'" + strings.ReplaceAll(unquoteString(lit), "'", "''") + "'" +} diff --git a/mdl/visitor/visitor_styling.go b/mdl/visitor/visitor_styling.go index d667e075bb..54c25be953 100644 --- a/mdl/visitor/visitor_styling.go +++ b/mdl/visitor/visitor_styling.go @@ -58,21 +58,21 @@ func parseStylingAssignment(ctx *parser.AlterStylingAssignmentContext) ast.Styli assignment.Property = "Class" assignment.IsCSS = true if sl := ctx.STRING_LITERAL(0); sl != nil { - assignment.Value = unquoteString(sl.GetText()) + assignment.Value = unquoteStringLit(sl) } } else if ctx.STYLE() != nil { // STYLE = 'value' (CSS appearance, not a design property) assignment.Property = "Style" assignment.IsCSS = true if sl := ctx.STRING_LITERAL(0); sl != nil { - assignment.Value = unquoteString(sl.GetText()) + assignment.Value = unquoteStringLit(sl) } } else { // STRING_LITERAL = STRING_LITERAL | ON | OFF (design property; the key // may itself be 'Class' or 'Style' — that is a design property, not CSS) literals := ctx.AllSTRING_LITERAL() if len(literals) > 0 { - assignment.Property = unquoteString(literals[0].GetText()) + assignment.Property = unquoteStringLit(literals[0]) } if ctx.ON() != nil { assignment.IsToggle = true @@ -81,7 +81,7 @@ func parseStylingAssignment(ctx *parser.AlterStylingAssignmentContext) ast.Styli assignment.IsToggle = true assignment.ToggleOn = false } else if len(literals) > 1 { - assignment.Value = unquoteString(literals[1].GetText()) + assignment.Value = unquoteStringLit(literals[1]) } } diff --git a/mdl/visitor/visitor_test.go b/mdl/visitor/visitor_test.go index 9ffbcaab76..273fbeadea 100644 --- a/mdl/visitor/visitor_test.go +++ b/mdl/visitor/visitor_test.go @@ -1041,8 +1041,11 @@ END;` t.Fatal("Expected to find RETRIEVE statement") } - if retrieveStmt.Limit != "1" { - t.Errorf("Expected Limit '1', got %q", retrieveStmt.Limit) + // Without the `mdl 1;` header `limit 1` keeps its alpha meaning, the object + // range (ako/mxcli#734); retrieve_first_test.go covers both versions. + if !retrieveStmt.First || retrieveStmt.Limit != "" { + t.Errorf("Expected the object range (First) for an mdl 0 `limit 1`, got First=%v Limit=%q", + retrieveStmt.First, retrieveStmt.Limit) } t.Log("RETRIEVE with LIMIT parsed correctly") diff --git a/mdl/visitor/visitor_translations.go b/mdl/visitor/visitor_translations.go index 0f7c714b74..6271d4507d 100644 --- a/mdl/visitor/visitor_translations.go +++ b/mdl/visitor/visitor_translations.go @@ -47,8 +47,8 @@ func (b *Builder) ExitCreateTranslationsStatement(ctx *parser.CreateTranslations continue } stmt.Entries = append(stmt.Entries, ast.TranslationEntry{ - Source: unquoteString(lits[0].GetText()), - Target: unquoteString(lits[1].GetText()), + Source: unquoteStringLit(lits[0]), + Target: unquoteStringLit(lits[1]), }) } diff --git a/mdl/visitor/visitor_upgrade_fixes.go b/mdl/visitor/visitor_upgrade_fixes.go new file mode 100644 index 0000000000..1f590647c1 --- /dev/null +++ b/mdl/visitor/visitor_upgrade_fixes.go @@ -0,0 +1,399 @@ +// SPDX-License-Identifier: Apache-2.0 + +package visitor + +import ( + "fmt" + "regexp" + "strings" + + "github.com/antlr4-go/antlr/v4" + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/grammar/parser" + "github.com/mendixlabs/mxcli/mdl/langver" +) + +// The mechanical rewrites `mxcli fmt --upgrade` applies (mdl/upgrade) are +// computed here, where the construct is recorded, because the parse tree is the +// only place its tokens and their positions are still at hand. Each rewrite is +// an ast.Fix on the recorded ast.DeprecatedSpelling or ast.LanguageNote; a +// construct with no mechanical rewrite gets a NoFix reason instead, and the +// upgrade reports it rather than guessing. +// +// A Fix edits as little as it can — the tokens that differ between the two +// spellings — so comments, layout and nested constructs survive, and a nested +// fix (a string escape inside a list operation's condition) never overlaps it. + +// LanguageChanges returns every change of meaning the visitor gates on the +// language header. The upgrade's registry test requires each to have a +// rewrite or to be listed as having none. +func LanguageChanges() []langver.Change { + return []langver.Change{ + semicolonRequired, slashIsNotATerminator, backslashIsLiteral, limitOneIsAList, + listCallForm, setIsMandatory, viewEntityReplaceIsModify, roleReplaceIsModify, + unknownPropertyKey, misshapedPropertyValue, + } +} + +// fixLastNote attaches a rewrite, or the reason there is none, to the note +// gate just recorded for code. +func (b *Builder) fixLastNote(code string, fix *ast.Fix, noFix string) { + if n := len(b.langNotes); n > 0 && b.langNotes[n-1].Code == code { + b.langNotes[n-1].Fix, b.langNotes[n-1].NoFix = fix, noFix + } +} + +// fixLastDeprecation attaches a structural rewrite, or the reason there is +// none, to the deprecated use just recorded for code. +func (b *Builder) fixLastDeprecation(code string, fix *ast.Fix, noFix string) { + if n := len(b.deprecations); n > 0 && b.deprecations[n-1].Code == code { + b.deprecations[n-1].Fix, b.deprecations[n-1].NoFix = fix, noFix + } +} + +// fixOrReason turns an edit list and the reason there is none into the pair +// fixLastNote and fixLastDeprecation take. +func fixOrReason(edits []ast.TextEdit, reason string) (*ast.Fix, string) { + if reason != "" { + return nil, reason + } + return &ast.Fix{Edits: edits}, "" +} + +// insertAt inserts text at rune offset pos. +func insertAt(pos int, text string) ast.TextEdit { + return ast.TextEdit{Start: pos, Stop: pos, Text: text} +} + +// replaceSpan replaces from the start of first to the end of last. +func replaceSpan(first, last antlr.Token, text string) ast.TextEdit { + return ast.TextEdit{Start: first.GetStart(), Stop: last.GetStop() + 1, Text: text} +} + +// replaceGap replaces the runes after aStop and before bStart. +func replaceGap(aStop, bStart int, text string) ast.TextEdit { + return ast.TextEdit{Start: aStop + 1, Stop: bStart, Text: text} +} + +// keywordLike spells word in the case of like: upper-case when like is, else +// lower-case (the canonical case, ADR-0010). +func keywordLike(like, word string) string { + if like == strings.ToUpper(like) && like != strings.ToLower(like) { + return strings.ToUpper(word) + } + return strings.ToLower(word) +} + +// nodeSpan is the rune span of a terminal or a rule context, inclusive. +func nodeSpan(n antlr.Tree) (start, stop int) { + switch x := n.(type) { + case antlr.TerminalNode: + t := x.GetSymbol() + return t.GetStart(), t.GetStop() + case antlr.ParserRuleContext: + if x.GetStart() != nil && x.GetStop() != nil { + return x.GetStart().GetStart(), x.GetStop().GetStop() + } + } + return -1, -1 +} + +// nodeText is the source text of a node, as written. +func nodeText(n antlr.Tree) string { + switch x := n.(type) { + case antlr.TerminalNode: + return x.GetText() + case antlr.ParserRuleContext: + return extractOriginalText(x) + } + return "" +} + +// holdsInterpretedEscape reports whether a subtree has an mdl 0 string literal +// whose value differs under mdl 1. Text a rewrite moves must not hold one, or +// the string-escape rewrite would edit text that is no longer there. +func holdsInterpretedEscape(n antlr.Tree) bool { + if tn, ok := n.(antlr.TerminalNode); ok { + t := tn.GetSymbol() + return t.GetTokenType() == parser.MDLLexerSTRING_LITERAL && hasInterpretedEscape(t.GetText()) && + !parser.HasStrictEscapes(t.GetInputStream()) + } + for i := 0; i < n.GetChildCount(); i++ { + if holdsInterpretedEscape(n.GetChild(i)) { + return true + } + } + return false +} + +// --------------------------------------------------------------------------- +// List operations and aggregates: call form -> statement form (MDL-DEPR003, +// MDL-DEPR004, MDL-V1-LIST). +// --------------------------------------------------------------------------- + +// simpleAttribute is an attribute name the statement form's `by` takes as is. +var simpleAttribute = regexp.MustCompile(`^[A-Za-z_][A-Za-z0-9_]*$`) + +// callFormFix rewrites a list-operation or aggregate call (a +// ListOperationContext or ListAggregateOperationContext) to the statement form +// that builds the same activity. byExpression says, for find and filter, which +// condition the call form built: `where` (an expression) or `by` (a member). +// It returns the edits, or the reason there are none. +func callFormFix(op antlr.ParserRuleContext, byExpression bool) ([]ast.TextEdit, string) { + n := op.GetChildCount() + if n < 4 { + return nil, "the call is incomplete" + } + kwNode, ok := op.GetChild(0).(antlr.TerminalNode) + if !ok { + return nil, "the call has no operation keyword" + } + rpNode, ok := op.GetChild(n - 1).(antlr.TerminalNode) + if !ok || rpNode.GetSymbol().GetTokenType() != parser.MDLParserRPAREN { + return nil, "the call is incomplete" + } + kw, rp := kwNode.GetSymbol(), rpNode.GetSymbol() + var args []antlr.Tree + for i := 2; i < n-1; i++ { + c := op.GetChild(i) + if tn, ok := c.(antlr.TerminalNode); ok && tn.GetSymbol().GetTokenType() == parser.MDLParserCOMMA { + continue + } + args = append(args, c) + } + if len(args) == 0 { + return nil, "the call has no operand" + } + word := func(w string) string { return keywordLike(kw.GetText(), w) } + + // infix keeps every argument where it is and replaces only the + // punctuation: `op(` -> `op `, each `,` -> its keyword, `)` -> nothing. + infix := func(seps ...string) ([]ast.TextEdit, string) { + if len(seps) != len(args)-1 { + return nil, "the call has an unexpected number of arguments" + } + first, _ := nodeSpan(args[0]) + edits := []ast.TextEdit{replaceGap(kw.GetStop(), first, " ")} + for i, sep := range seps { + _, prevStop := nodeSpan(args[i]) + nextStart, _ := nodeSpan(args[i+1]) + edits = append(edits, replaceGap(prevStop, nextStart, " "+word(sep)+" ")) + } + _, lastStop := nodeSpan(args[len(args)-1]) + return append(edits, replaceGap(lastStop, rp.GetStop()+1, "")), "" + } + // whole replaces everything after the keyword, for the forms that reorder + // their operands. + whole := func(text string) ([]ast.TextEdit, string) { + for _, a := range args { + if holdsInterpretedEscape(a) { + return nil, "its operands move, and hold a string with a backslash escape" + } + } + return []ast.TextEdit{{Start: kw.GetStop() + 1, Stop: rp.GetStop() + 1, Text: text}}, "" + } + + switch kw.GetTokenType() { + case parser.MDLParserHEAD, parser.MDLParserTAIL, parser.MDLParserCOUNT: + return infix() + case parser.MDLParserFIND, parser.MDLParserFILTER: + if byExpression { + return infix("where") + } + return infix("by") + case parser.MDLParserSORT: + return infix("by") + case parser.MDLParserUNION, parser.MDLParserINTERSECT: + return infix("with") + case parser.MDLParserEQUALS_OP: + return infix("and") + case parser.MDLParserRANGE: + if len(args) > 3 { + return nil, "the call has an unexpected number of arguments" + } + return infix([]string{"offset", "limit"}[:len(args)-1]...) + case parser.MDLParserSUBTRACT: + // subtract($A, $B) is A minus B, written `subtract $B from $A`. + if len(args) != 2 { + return nil, "the call has an unexpected number of arguments" + } + return whole(" " + nodeText(args[1]) + " " + word("from") + " " + nodeText(args[0])) + case parser.MDLParserCONTAINS: + // contains($List, $Object) is written `contains $Object in $List`. + if len(args) != 2 { + return nil, "the call has an unexpected number of arguments" + } + return whole(" " + nodeText(args[1]) + " " + word("in") + " " + nodeText(args[0])) + case parser.MDLParserSUM, parser.MDLParserAVERAGE, parser.MDLParserMINIMUM, parser.MDLParserMAXIMUM: + if path, ok := args[0].(*parser.AttributePathContext); ok && len(args) == 1 { + // sum($L.Attr) is `sum $L by Attr`: the builder keeps the last + // segment of the path as the attribute. + v, attr := parseAttributePath(path.GetText()) + if !simpleAttribute.MatchString(attr) { + return nil, fmt.Sprintf("the attribute `%s` is not a plain name, which is all the statement form's `by` takes", attr) + } + return whole(" $" + v + " " + word("by") + " " + attr) + } + return infix("of") + case parser.MDLParserALL, parser.MDLParserANY: + return infix("where") + case parser.MDLParserREDUCE: + // reduce($L, expr, initial: seed, returns: T) is + // `reduce $L from seed as T using expr`: the fold body stays in place, + // the seed and the type move in front of it. + opts, ok := args[len(args)-1].(*parser.ReduceFoldOptionsContext) + if !ok || len(args) != 3 || opts.Expression() == nil || opts.DataType() == nil { + return nil, "the call has an unexpected shape" + } + if holdsInterpretedEscape(opts) { + return nil, "its seed moves, and holds a string with a backslash escape" + } + bodyStart, bodyStop := nodeSpan(args[1]) + return []ast.TextEdit{ + {Start: kw.GetStop() + 1, Stop: bodyStart, Text: " " + nodeText(args[0]) + " " + word("from") + " " + + extractOriginalText(opts.Expression()) + " " + word("as") + " " + extractOriginalText(opts.DataType()) + + " " + word("using") + " "}, + replaceGap(bodyStop, rp.GetStop()+1, ""), + }, "" + } + return nil, "there is no statement form for `" + kw.GetText() + "`" +} + +// singleListCall returns the list-operation or aggregate call an expression +// consists of, or nil when it is anything else: a nested call, an operator, or +// a call the call-form grammar does not match. +func singleListCall(e antlr.Tree) antlr.ParserRuleContext { + for e != nil { + switch x := e.(type) { + case *parser.ListOperationContext: + return x + case *parser.ListAggregateOperationContext: + return x + } + if e.GetChildCount() != 1 { + return nil + } + e = e.GetChild(0) + } + return nil +} + +// operandKind says whether a find/contains operand is a String (the call is +// the string function) or not (the List operation), as the mdl 0 flow builder +// decides it from the variable's declared type. known is false when the script +// does not say, e.g. the variable holds a call's result, whose type only the +// project knows. +func operandKind(at antlr.Tree, variable string) (isString, known bool) { + name := strings.TrimPrefix(variable, "$") + var flow antlr.Tree + for t := at; t != nil && flow == nil; t = t.GetParent() { + switch t.(type) { + case *parser.CreateMicroflowStatementContext, *parser.CreateNanoflowStatementContext, *parser.CreateRuleStatementContext: + flow = t + } + } + if flow == nil { + return false, false + } + // A parameter's or declare's type is authoritative: a variable keeps its type. + var declared []bool + var otherDefs, unknownDefs int + var walk func(antlr.Tree) + walk = func(t antlr.Tree) { + switch x := t.(type) { + case antlr.TerminalNode: + return + case *parser.MicroflowParameterContext: + pname := "" + if pn := x.ParameterName(); pn != nil { + pname = parameterNameText(pn) + } else if v := x.VARIABLE(); v != nil { + pname = strings.TrimPrefix(v.GetText(), "$") + } + if pname == name && x.DataType() != nil { + declared = append(declared, buildMicroflowDataType(x.DataType()).Kind == ast.TypeString) + } + return + case *parser.DeclareStatementContext: + if v := x.VARIABLE(); v != nil && v.GetText() == variable && x.DataType() != nil { + declared = append(declared, buildMicroflowDataType(x.DataType()).Kind == ast.TypeString) + } + case *parser.RetrieveStatementContext: + if v := x.VARIABLE(); v != nil && v.GetText() == variable { + otherDefs++ + } + case *parser.CreateListStatementContext: + if v := x.VARIABLE(); v != nil && v.GetText() == variable { + otherDefs++ + } + case *parser.ListOperationStatementContext: + if v := x.VARIABLE(); v != nil && v.GetText() == variable { + otherDefs++ + } + case *parser.LoopStatementContext: + if v := x.VARIABLE(0); v != nil && v.GetText() == variable { + otherDefs++ + } + default: + // Any other `$name = …` gives it a type the script may not state + // (a call's result, a REST response, an aggregate). A set only + // reassigns. + if prc, ok := t.(antlr.ParserRuleContext); ok && prc.GetChildCount() >= 2 { + if _, isSet := prc.(*parser.SetStatementContext); !isSet { + v, ok1 := prc.GetChild(0).(antlr.TerminalNode) + eq, ok2 := prc.GetChild(1).(antlr.TerminalNode) + if ok1 && ok2 && v.GetText() == variable && eq.GetSymbol().GetTokenType() == parser.MDLParserEQUALS { + unknownDefs++ + } + } + } + } + for i := 0; i < t.GetChildCount(); i++ { + walk(t.GetChild(i)) + } + } + walk(flow) + if len(declared) > 0 { + for _, d := range declared[1:] { + if d != declared[0] { + return false, false + } + } + return declared[0], true + } + return false, otherDefs > 0 && unknownDefs == 0 +} + +func operandKindUnknown(variable, fn string) string { + return fmt.Sprintf("whether %s is a String (the string function %s) or a list (the List operation) "+ + "depends on a type the script does not state; write `set $x = %s(…);` or `$x = %s %s …;` by hand", + variable, fn, fn, fn, variable) +} + +// --------------------------------------------------------------------------- +// Terminators and strings. +// --------------------------------------------------------------------------- + +// slashLineFix deletes a `/` terminator: its whole line when the `/` is alone +// on it, else the token. +func slashLineFix(slash antlr.Token) ast.TextEdit { + is := slash.GetInputStream() + start, stop := slash.GetStart(), slash.GetStop()+1 + size := is.Size() + at := func(i int) string { return is.GetText(i, i) } + ls := start + for ls > 0 && (at(ls-1) == " " || at(ls-1) == "\t") { + ls-- + } + le := stop + for le < size && (at(le) == " " || at(le) == "\t" || at(le) == "\r") { + le++ + } + if (ls == 0 || at(ls-1) == "\n") && (le == size || at(le) == "\n") { + if le < size { + le++ // and its newline + } + return ast.TextEdit{Start: ls, Stop: le} + } + return ast.TextEdit{Start: start, Stop: stop} +} diff --git a/mdl/visitor/visitor_validationrule.go b/mdl/visitor/visitor_validationrule.go index 54d98790df..0a34b03a5f 100644 --- a/mdl/visitor/visitor_validationrule.go +++ b/mdl/visitor/visitor_validationrule.go @@ -14,7 +14,7 @@ func (b *Builder) ExitCreateValidationRuleStatement(ctx *parser.CreateValidation Attribute: buildQualifiedName(ctx.QualifiedName()), } if s := ctx.STRING_LITERAL(); s != nil { - stmt.Feedback = unquoteString(s.GetText()) + stmt.Feedback = unquoteStringLit(s) } constraint, ok := ctx.ValidationRuleConstraint().(*parser.ValidationRuleConstraintContext) diff --git a/mdl/visitor/visitor_widgets.go b/mdl/visitor/visitor_widgets.go index 160051ef4d..15e0940f2e 100644 --- a/mdl/visitor/visitor_widgets.go +++ b/mdl/visitor/visitor_widgets.go @@ -32,7 +32,7 @@ func parseWidgetConditions(conditions []parser.IWidgetConditionContext) []ast.Wi // Get value if cond.STRING_LITERAL() != nil { - filter.Value = unquoteString(cond.STRING_LITERAL().GetText()) + filter.Value = unquoteStringLit(cond.STRING_LITERAL()) } filters = append(filters, filter) @@ -53,7 +53,7 @@ func (b *Builder) ExitUpdateWidgetsStatement(ctx *parser.UpdateWidgetsStatementC // Get property path (the first STRING_LITERAL) if assignCtx.STRING_LITERAL() != nil { - assignment.PropertyPath = unquoteString(assignCtx.STRING_LITERAL().GetText()) + assignment.PropertyPath = unquoteStringLit(assignCtx.STRING_LITERAL()) } // Get value from widgetPropertyValue @@ -85,7 +85,7 @@ func (b *Builder) ExitUpdateWidgetsStatement(ctx *parser.UpdateWidgetsStatementC // parseWidgetPropertyValue parses a widget property value (string, number, bool, null). func parseWidgetPropertyValue(ctx parser.IWidgetPropertyValueContext) any { if ctx.STRING_LITERAL() != nil { - return unquoteString(ctx.STRING_LITERAL().GetText()) + return unquoteStringLit(ctx.STRING_LITERAL()) } if ctx.NUMBER_LITERAL() != nil { numStr := ctx.NUMBER_LITERAL().GetText() diff --git a/mdl/visitor/visitor_workflow.go b/mdl/visitor/visitor_workflow.go index 8d688f3981..f236c3397e 100644 --- a/mdl/visitor/visitor_workflow.go +++ b/mdl/visitor/visitor_workflow.go @@ -38,7 +38,7 @@ func (b *Builder) ExitCreateWorkflowStatement(ctx *parser.CreateWorkflowStatemen switch { case hc.FOLDER() != nil: if tok := hc.GetFolder(); tok != nil { - stmt.Folder = unquoteString(tok.GetText()) + stmt.Folder = unquoteStringLit(tok) } case hc.PARAMETER() != nil: if v := hc.VARIABLE(); v != nil { @@ -49,11 +49,11 @@ func (b *Builder) ExitCreateWorkflowStatement(ctx *parser.CreateWorkflowStatemen } case hc.DISPLAY() != nil: if tok := hc.GetDisplay(); tok != nil { - stmt.DisplayName = unquoteString(tok.GetText()) + stmt.DisplayName = unquoteStringLit(tok) } case hc.DESCRIPTION() != nil: if tok := hc.GetDescription(); tok != nil { - stmt.Description = unquoteString(tok.GetText()) + stmt.Description = unquoteStringLit(tok) } case hc.EXPORT() != nil && hc.LEVEL() != nil: // HIDDEN_KW is read alongside IDENTIFIER because `Hidden` was an @@ -75,7 +75,7 @@ func (b *Builder) ExitCreateWorkflowStatement(ctx *parser.CreateWorkflowStatemen } case hc.DUE() != nil && hc.DATE_TYPE() != nil: if tok := hc.GetDueDate(); tok != nil { - stmt.DueDate = unquoteString(tok.GetText()) + stmt.DueDate = unquoteStringLit(tok) } case hc.WorkflowEventHandlerClause() != nil: h, ok := hc.WorkflowEventHandlerClause().(*parser.WorkflowEventHandlerClauseContext) @@ -90,7 +90,7 @@ func (b *Builder) ExitCreateWorkflowStatement(ctx *parser.CreateWorkflowStatemen node.EventTypes = append(node.EventTypes, id.GetText()) } if str := h.STRING_LITERAL(); str != nil { - node.Description = unquoteString(str.GetText()) + node.Description = unquoteStringLit(str) } stmt.EventHandlers = append(stmt.EventHandlers, node) } @@ -209,7 +209,7 @@ func buildAlterWorkflowAction(ctx *parser.AlterWorkflowActionContext) ast.AlterW return nil } ref, atPos := parseAlterActivityRef(ctx.AlterActivityRef().(*parser.AlterActivityRefContext)) - outcomeName := unquoteString(ctx.STRING_LITERAL().GetText()) + outcomeName := unquoteStringLit(ctx.STRING_LITERAL()) var activities []ast.WorkflowActivityNode if body := ctx.WorkflowBody(); body != nil { activities = buildWorkflowBody(body) @@ -245,7 +245,7 @@ func buildAlterWorkflowAction(ctx *parser.AlterWorkflowActionContext) ast.AlterW return nil } ref, atPos := parseAlterActivityRef(ctx.AlterActivityRef().(*parser.AlterActivityRefContext)) - outcomeName := unquoteString(ctx.STRING_LITERAL().GetText()) + outcomeName := unquoteStringLit(ctx.STRING_LITERAL()) return &ast.DropOutcomeOp{ OutcomeName: outcomeName, ActivityRef: ref, @@ -259,7 +259,7 @@ func buildAlterWorkflowAction(ctx *parser.AlterWorkflowActionContext) ast.AlterW return nil } ref, atPos := parseAlterActivityRef(ctx.AlterActivityRef().(*parser.AlterActivityRefContext)) - pathCaption := unquoteString(ctx.STRING_LITERAL().GetText()) + pathCaption := unquoteStringLit(ctx.STRING_LITERAL()) return &ast.DropPathOp{ PathCaption: pathCaption, ActivityRef: ref, @@ -301,7 +301,7 @@ func buildAlterWorkflowAction(ctx *parser.AlterWorkflowActionContext) ast.AlterW return nil } ref, atPos := parseAlterActivityRef(ctx.AlterActivityRef().(*parser.AlterActivityRefContext)) - condition := unquoteString(ctx.STRING_LITERAL().GetText()) + condition := unquoteStringLit(ctx.STRING_LITERAL()) var activities []ast.WorkflowActivityNode if body := ctx.WorkflowBody(); body != nil { activities = buildWorkflowBody(body) @@ -320,7 +320,7 @@ func buildAlterWorkflowAction(ctx *parser.AlterWorkflowActionContext) ast.AlterW return nil } ref, atPos := parseAlterActivityRef(ctx.AlterActivityRef().(*parser.AlterActivityRefContext)) - branchName := unquoteString(ctx.STRING_LITERAL().GetText()) + branchName := unquoteStringLit(ctx.STRING_LITERAL()) return &ast.DropBranchOp{ BranchName: branchName, ActivityRef: ref, @@ -337,10 +337,10 @@ func buildWorkflowSetPropertyOp(ctx *parser.WorkflowSetPropertyContext) *ast.Set if ctx.DISPLAY() != nil { op.Property = "display" - op.Value = unquoteString(ctx.STRING_LITERAL().GetText()) + op.Value = unquoteStringLit(ctx.STRING_LITERAL()) } else if ctx.DESCRIPTION() != nil { op.Property = "description" - op.Value = unquoteString(ctx.STRING_LITERAL().GetText()) + op.Value = unquoteStringLit(ctx.STRING_LITERAL()) } else if ctx.EXPORT() != nil { op.Property = "export_level" // HIDDEN_KW alongside IDENTIFIER — see the CREATE side above. @@ -354,7 +354,7 @@ func buildWorkflowSetPropertyOp(ctx *parser.WorkflowSetPropertyContext) *ast.Set } } else if ctx.DUE() != nil { op.Property = "due_date" - op.Value = unquoteString(ctx.STRING_LITERAL().GetText()) + op.Value = unquoteStringLit(ctx.STRING_LITERAL()) } else if ctx.OVERVIEW() != nil { op.Property = "overview_page" if qn := ctx.QualifiedName(); qn != nil { @@ -385,7 +385,7 @@ func buildActivitySetPropertyOp(ctx *parser.ActivitySetPropertyContext, ref stri } } else if ctx.DESCRIPTION() != nil { op.Property = "description" - op.Value = unquoteString(ctx.STRING_LITERAL().GetText()) + op.Value = unquoteStringLit(ctx.STRING_LITERAL()) } else if ctx.TARGETING() != nil && ctx.MICROFLOW() != nil { op.Property = "targeting_microflow" if qn := ctx.QualifiedName(); qn != nil { @@ -393,10 +393,10 @@ func buildActivitySetPropertyOp(ctx *parser.ActivitySetPropertyContext, ref stri } } else if ctx.TARGETING() != nil && ctx.XPATH() != nil { op.Property = "targeting_xpath" - op.Value = unquoteString(ctx.STRING_LITERAL().GetText()) + op.Value = unquoteStringLit(ctx.STRING_LITERAL()) } else if ctx.DUE() != nil { op.Property = "due_date" - op.Value = unquoteString(ctx.STRING_LITERAL().GetText()) + op.Value = unquoteStringLit(ctx.STRING_LITERAL()) } return op @@ -411,7 +411,7 @@ func parseAlterActivityRef(ctx *parser.AlterActivityRefContext) (string, int) { if iok := ctx.IdentifierOrKeyword(); iok != nil { name = identifierOrKeywordText(iok) } else if ctx.STRING_LITERAL() != nil { - name = unquoteString(ctx.STRING_LITERAL().GetText()) + name = unquoteStringLit(ctx.STRING_LITERAL()) } atPos := 0 @@ -468,7 +468,7 @@ func buildWorkflowStatements(children []antlr.Tree) []ast.WorkflowActivityNode { func buildWorkflowEnd(ctx *parser.WorkflowEndStmtContext) *ast.WorkflowEndNode { node := &ast.WorkflowEndNode{} if ctx.COMMENT() != nil && ctx.STRING_LITERAL() != nil { - node.Caption = unquoteString(ctx.STRING_LITERAL().GetText()) + node.Caption = unquoteStringLit(ctx.STRING_LITERAL()) } return node } @@ -540,7 +540,7 @@ func buildWorkflowUserTask(ctx parser.IWorkflowUserTaskStmtContext) *ast.Workflo // The caption is the statement's own string literal; every other string now // belongs to a clause. if caption := utCtx.STRING_LITERAL(); caption != nil { - node.Caption = unquoteString(caption.GetText()) + node.Caption = unquoteStringLit(caption) } for _, clause := range utCtx.AllWorkflowUserTaskClause() { @@ -601,7 +601,7 @@ func applyWorkflowUserTaskClause(node *ast.WorkflowUserTaskNode, clause parser.I node.Targeting.Kind = "xpath" } if str := c.STRING_LITERAL(); str != nil { - node.Targeting.XPath = unquoteString(str.GetText()) + node.Targeting.XPath = unquoteStringLit(str) } case c.ON() != nil && c.CREATED() != nil: if qn := c.QualifiedName(); qn != nil { @@ -613,11 +613,11 @@ func applyWorkflowUserTaskClause(node *ast.WorkflowUserTaskNode, clause parser.I } case c.DUE() != nil && c.DATE_TYPE() != nil: if str := c.STRING_LITERAL(); str != nil { - node.DueDate = unquoteString(str.GetText()) + node.DueDate = unquoteStringLit(str) } case c.DESCRIPTION() != nil: if str := c.STRING_LITERAL(); str != nil { - node.TaskDescription = unquoteString(str.GetText()) + node.TaskDescription = unquoteStringLit(str) } case c.OUTCOMES() != nil: for _, outcomeCtx := range c.AllWorkflowUserTaskOutcome() { @@ -634,7 +634,7 @@ func applyWorkflowUserTaskClause(node *ast.WorkflowUserTaskNode, clause parser.I func buildWorkflowUserTaskOutcome(ctx parser.IWorkflowUserTaskOutcomeContext) ast.WorkflowUserTaskOutcomeNode { outCtx := ctx.(*parser.WorkflowUserTaskOutcomeContext) outcome := ast.WorkflowUserTaskOutcomeNode{ - Caption: unquoteString(outCtx.STRING_LITERAL().GetText()), + Caption: unquoteStringLit(outCtx.STRING_LITERAL()), } if body := outCtx.WorkflowBody(); body != nil { outcome.Activities = buildWorkflowBody(body) @@ -668,7 +668,7 @@ func buildWorkflowCallMicroflow(ctx parser.IWorkflowCallMicroflowStmtContext) *a } if cmCtx.COMMENT() != nil && cmCtx.STRING_LITERAL() != nil { - node.Caption = unquoteString(cmCtx.STRING_LITERAL().GetText()) + node.Caption = unquoteStringLit(cmCtx.STRING_LITERAL()) } for _, outcomeCtx := range cmCtx.AllWorkflowConditionOutcome() { @@ -724,7 +724,7 @@ func buildWorkflowParameterMappings(ctxs []parser.IWorkflowParameterMappingConte } out = append(out, ast.WorkflowParameterMappingNode{ Parameter: bareWorkflowParameterName(pmCtx2.QualifiedName().GetText()), - Expression: unquoteString(pmCtx2.STRING_LITERAL().GetText()), + Expression: unquoteStringLit(pmCtx2.STRING_LITERAL()), }) } return out @@ -739,7 +739,7 @@ func buildWorkflowCallWorkflow(ctx parser.IWorkflowCallWorkflowStmtContext) *ast } if cwCtx.COMMENT() != nil && cwCtx.STRING_LITERAL() != nil { - node.Caption = unquoteString(cwCtx.STRING_LITERAL().GetText()) + node.Caption = unquoteStringLit(cwCtx.STRING_LITERAL()) } // Parameter mappings @@ -761,16 +761,16 @@ func buildWorkflowDecision(ctx parser.IWorkflowDecisionStmtContext) *ast.Workflo // First STRING_LITERAL is the expression (if present and COMMENT is not present or expression comes first) if len(allStrings) > 0 && dCtx.COMMENT() == nil { // All strings are expression - node.Expression = unquoteString(allStrings[0].GetText()) + node.Expression = unquoteStringLit(allStrings[0]) stringIdx = 1 } else if len(allStrings) > 0 && dCtx.COMMENT() != nil { // Distinguish expression from comment if len(allStrings) >= 2 { - node.Expression = unquoteString(allStrings[0].GetText()) - node.Caption = unquoteString(allStrings[1].GetText()) + node.Expression = unquoteStringLit(allStrings[0]) + node.Caption = unquoteStringLit(allStrings[1]) } else { // Only one string with COMMENT - it's the caption - node.Caption = unquoteString(allStrings[0].GetText()) + node.Caption = unquoteStringLit(allStrings[0]) } stringIdx = len(allStrings) } @@ -796,7 +796,7 @@ func buildWorkflowConditionOutcome(ctx parser.IWorkflowConditionOutcomeContext) } else if coCtx.DEFAULT() != nil { outcome.Value = "Default" } else if coCtx.STRING_LITERAL() != nil { - outcome.Value = unquoteString(coCtx.STRING_LITERAL().GetText()) + outcome.Value = unquoteStringLit(coCtx.STRING_LITERAL()) } if body := coCtx.WorkflowBody(); body != nil { @@ -814,7 +814,7 @@ func buildWorkflowParallelSplit(ctx parser.IWorkflowParallelSplitStmtContext) *a } if psCtx.COMMENT() != nil && psCtx.STRING_LITERAL() != nil { - node.Caption = unquoteString(psCtx.STRING_LITERAL().GetText()) + node.Caption = unquoteStringLit(psCtx.STRING_LITERAL()) } for _, pathCtx := range psCtx.AllWorkflowParallelPath() { @@ -857,7 +857,7 @@ func buildWorkflowJumpTo(ctx parser.IWorkflowJumpToStmtContext) *ast.WorkflowJum } if jtCtx.COMMENT() != nil && jtCtx.STRING_LITERAL() != nil { - node.Caption = unquoteString(jtCtx.STRING_LITERAL().GetText()) + node.Caption = unquoteStringLit(jtCtx.STRING_LITERAL()) } return node @@ -872,12 +872,12 @@ func buildWorkflowWaitForTimer(ctx parser.IWorkflowWaitForTimerStmtContext) *ast allStrings := wtCtx.AllSTRING_LITERAL() if len(allStrings) > 0 && wtCtx.COMMENT() == nil { - node.DelayExpression = unquoteString(allStrings[0].GetText()) + node.DelayExpression = unquoteStringLit(allStrings[0]) } else if len(allStrings) >= 2 && wtCtx.COMMENT() != nil { - node.DelayExpression = unquoteString(allStrings[0].GetText()) - node.Caption = unquoteString(allStrings[1].GetText()) + node.DelayExpression = unquoteStringLit(allStrings[0]) + node.Caption = unquoteStringLit(allStrings[1]) } else if len(allStrings) == 1 && wtCtx.COMMENT() != nil { - node.Caption = unquoteString(allStrings[0].GetText()) + node.Caption = unquoteStringLit(allStrings[0]) } return node @@ -891,7 +891,7 @@ func buildWorkflowWaitForNotification(ctx parser.IWorkflowWaitForNotificationStm } if wnCtx.COMMENT() != nil && wnCtx.STRING_LITERAL() != nil { - node.Caption = unquoteString(wnCtx.STRING_LITERAL().GetText()) + node.Caption = unquoteStringLit(wnCtx.STRING_LITERAL()) } // BoundaryEvents (Issue #7) @@ -922,9 +922,9 @@ func buildBoundaryEventNode(beCtx parser.IWorkflowBoundaryEventClauseContext) as // A timer's string is its delay; a notification event's is its caption. if s := beCtx2.STRING_LITERAL(); s != nil { if notification { - be.Caption = unquoteString(s.GetText()) + be.Caption = unquoteStringLit(s) } else { - be.Delay = unquoteString(s.GetText()) + be.Delay = unquoteStringLit(s) } } if notification { @@ -941,7 +941,7 @@ func buildWorkflowNotification(ctx parser.IWorkflowNotificationStmtContext) *ast c := ctx.(*parser.WorkflowNotificationStmtContext) node := &ast.WorkflowNotificationNode{Name: workflowActivityNameText(c.WorkflowActivityName())} if c.COMMENT() != nil && c.STRING_LITERAL() != nil { - node.Caption = unquoteString(c.STRING_LITERAL().GetText()) + node.Caption = unquoteStringLit(c.STRING_LITERAL()) } return node } @@ -956,7 +956,7 @@ func buildWorkflowEventSubProcess(ctx parser.IWorkflowEventSubProcessContext) as Interrupting: c.NON() == nil, } if s := c.STRING_LITERAL(); s != nil { - node.Caption = unquoteString(s.GetText()) + node.Caption = unquoteStringLit(s) } if t, ok := c.WorkflowEventSubProcessTrigger().(*parser.WorkflowEventSubProcessTriggerContext); ok && t != nil { node.Timer = t.TIMER() != nil @@ -965,13 +965,13 @@ func buildWorkflowEventSubProcess(ctx parser.IWorkflowEventSubProcessContext) as switch { case node.Timer: if len(strs) > 0 { - node.FirstExecutionTime = unquoteString(strs[0].GetText()) + node.FirstExecutionTime = unquoteStringLit(strs[0]) } if t.COMMENT() != nil && len(strs) > 1 { - node.StartCaption = unquoteString(strs[1].GetText()) + node.StartCaption = unquoteStringLit(strs[1]) } case len(strs) > 0: - node.StartCaption = unquoteString(strs[0].GetText()) + node.StartCaption = unquoteStringLit(strs[0]) } } if body := c.WorkflowBody(); body != nil { @@ -985,7 +985,7 @@ func buildWorkflowAnnotation(ctx parser.IWorkflowAnnotationStmtContext) *ast.Wor annCtx := ctx.(*parser.WorkflowAnnotationStmtContext) node := &ast.WorkflowAnnotationActivityNode{} if annCtx.STRING_LITERAL() != nil { - node.Text = unquoteString(annCtx.STRING_LITERAL().GetText()) + node.Text = unquoteStringLit(annCtx.STRING_LITERAL()) } return node } @@ -1040,7 +1040,7 @@ func buildWorkflowCompletionRule(ctx *parser.WorkflowCompletionClauseContext) *a case ctx.VETO() != nil: r.Rule = "veto" if s := ctx.STRING_LITERAL(); s != nil { - r.Veto = unquoteString(s.GetText()) + r.Veto = unquoteStringLit(s) } case ctx.MICROFLOW() != nil: r.Rule = "microflow" @@ -1049,7 +1049,7 @@ func buildWorkflowCompletionRule(ctx *parser.WorkflowCompletionClauseContext) *a } } if fb, ok := ctx.WorkflowFallbackClause().(*parser.WorkflowFallbackClauseContext); ok && fb != nil && fb.STRING_LITERAL() != nil { - r.Fallback = unquoteString(fb.STRING_LITERAL().GetText()) + r.Fallback = unquoteStringLit(fb.STRING_LITERAL()) r.HasFallback = true } return r diff --git a/mdl/visitor/visitor_xpath.go b/mdl/visitor/visitor_xpath.go index c31972ea2b..6d11448f51 100644 --- a/mdl/visitor/visitor_xpath.go +++ b/mdl/visitor/visitor_xpath.go @@ -205,7 +205,7 @@ func buildXPathStepValue(ctx parser.IXpathStepValueContext) ast.Expression { return &ast.VariableExpr{Name: strings.TrimPrefix(v.GetText(), "$")} } if sl := svCtx.STRING_LITERAL(); sl != nil { - return &ast.LiteralExpr{Value: unquoteString(sl.GetText()), Kind: ast.LiteralString} + return &ast.LiteralExpr{Value: unquoteStringLit(sl), Kind: ast.LiteralString} } if nl := svCtx.NUMBER_LITERAL(); nl != nil { text := nl.GetText() diff --git a/modelsdk/canon/identity.go b/modelsdk/canon/identity.go index 8c9143d9ba..8a8cb0c960 100644 --- a/modelsdk/canon/identity.go +++ b/modelsdk/canon/identity.go @@ -36,6 +36,23 @@ type Option func(*reconcileOpts) type reconcileOpts struct { contentsOwnTranslations bool contentsOwnStorageGUIDs bool + contentsOwnElementIDs bool +} + +// ContentsOwnElementIDs tells Reconcile that the write is a PATCH of the stored +// document, not a rebuild: every element that survived kept its stored $ID, and +// every new element carries a fresh one on purpose. So the structural transplant +// must not run. +// +// The transplant exists for rebuilds, whose elements all arrive with random +// $IDs and are paired back by type and position. On a patch that pairing is not +// a no-op but a hazard: drop one sequence flow and every flow after it pairs with +// its predecessor, taking that flow's $ID — identities move onto other nodes, +// which is what ADR-0012 measured the microflow rebuild doing (51 of 161). The +// graph splice of `alter microflow` (mfmutator) is the caller; it has its own +// guard that no reference is left dangling. Elision still applies. +func ContentsOwnElementIDs() Option { + return func(o *reconcileOpts) { o.contentsOwnElementIDs = true } } // ContentsOwnTranslations tells Reconcile that the write already accounts for @@ -114,7 +131,9 @@ func Reconcile(contents, stored []byte, opts ...Option) (out []byte, unchanged b // version control as a whole-document replacement. TransplantIDs puts the // stored ids back on the elements that still correspond, rewriting every // reference with them. - contents = TransplantIDs(contents, stored) + if !o.contentsOwnElementIDs { + contents = TransplantIDs(contents, stored) + } // And the nested identity property the transplant does not cover: every // Workflows$* element carries a PersistentId that both engines re-mint on diff --git a/modelsdk/canon/patch_ids_test.go b/modelsdk/canon/patch_ids_test.go new file mode 100644 index 0000000000..cae29e2f34 --- /dev/null +++ b/modelsdk/canon/patch_ids_test.go @@ -0,0 +1,56 @@ +// SPDX-License-Identifier: Apache-2.0 + +package canon + +import ( + "bytes" + "testing" + + "go.mongodb.org/mongo-driver/v2/bson" +) + +// seqFlows builds a flow document whose Flows list holds one sequence flow per +// id in ids, each from origin i to origin i+1. Every flow has the same $Type +// and shape, which is exactly what makes the structural pairing positional. +func seqFlows(t *testing.T, ids ...byte) []byte { + t.Helper() + flows := bson.A{int32(3)} + for i, id := range ids { + flows = append(flows, bson.D{ + {Key: "$Type", Value: "Microflows$SequenceFlow"}, + {Key: "$ID", Value: bin(id)}, + {Key: "OriginPointer", Value: bin(byte(100 + i))}, + {Key: "DestinationPointer", Value: bin(byte(101 + i))}, + }) + } + return marshal(t, bson.D{ + {Key: "$Type", Value: "Microflows$Microflow"}, + {Key: "$ID", Value: bin(1)}, + {Key: "Flows", Value: flows}, + }) +} + +// A patch of the stored document — an `alter microflow` drop — keeps every +// surviving element's $ID already. The structural transplant must not then +// re-pair those elements: with one flow gone, the flows after it pair +// positionally with their predecessors and each takes its neighbour's $ID, +// moving identities onto other nodes (ADR-0012's "51 of 161 $IDs changed"). +func TestReconcile_PatchOwnsElementIDs(t *testing.T) { + stored := seqFlows(t, 10, 20, 30, 40) + patched := seqFlows(t, 10, 30, 40) // flow 20 dropped; 30 and 40 keep their $IDs + + // Control: the rebuild path pairs by position and moves $IDs, which is + // the defect the option exists for. + rebuilt, _ := Reconcile(patched, stored) + if ids := idSet(t, rebuilt); ids[blobToUUID(bin(40).Data)] { + t.Fatalf("control: expected the transplant to move $ID 40 onto another flow; the test would prove nothing") + } + + out, unchanged := Reconcile(patched, stored, ContentsOwnElementIDs()) + if unchanged { + t.Fatal("a patch that dropped a flow reported no change") + } + if !bytes.Equal(out, patched) { + t.Errorf("a patch that owns its element $IDs was rewritten by Reconcile") + } +} diff --git a/modelsdk/mpr/writer_core.go b/modelsdk/mpr/writer_core.go index 9d9e0b23cb..ec92cb7b83 100644 --- a/modelsdk/mpr/writer_core.go +++ b/modelsdk/mpr/writer_core.go @@ -828,6 +828,17 @@ func (w *Writer) UpdateRawUnitOwningTranslations(unitID string, contents []byte) return w.updateUnit(unitID, contents, canon.ContentsOwnTranslations()) } +// UpdateRawUnitPatch is UpdateRawUnit for a write that PATCHED the stored bytes +// in place rather than rebuilding them — the graph splice of `alter microflow`. +// Such a write already carries every stored $ID and translation it means to +// keep, so neither is carried back: re-pairing element $IDs structurally would +// move identities onto other elements after a drop (canon.ContentsOwnElementIDs), +// and carrying translations would undo a deliberate removal. Elision and the +// storage-GUID guard apply as for every write. +func (w *Writer) UpdateRawUnitPatch(unitID string, contents []byte) error { + return w.updateUnit(unitID, contents, canon.ContentsOwnElementIDs(), canon.ContentsOwnTranslations()) +} + // UpdateRawUnitOwningStorageGUIDs is UpdateRawUnit for a write that deliberately // transplants storage GUIDs onto elements that keep their $ID — the marketplace // module update, which carries a module's existing GUIDs onto the documents diff --git a/testdata/testapp-views/README.md b/testdata/testapp-views/README.md new file mode 100644 index 0000000000..c35eb924c6 --- /dev/null +++ b/testdata/testapp-views/README.md @@ -0,0 +1,28 @@ +# testapp-views: a Studio Pro-authored view entity + +A trimmed copy of `ako/TestApp`, a Mendix **11.14.0** app (MPR v2) authored in +Studio Pro. It exists for one thing PedApp (`testdata/pedapp`) does not have: a +**view entity mxcli did not write**, `MyFirstModule.VCar`, over the persistent +entity `MyFirstModule.Car`. + +On an element mxcli created, `GUID == $ID` from birth, so a test that re-mints +the GUID from the `$ID` reproduces the same value and cannot fail (CLAUDE.md, "A +`GUID` Is the Database's Identity"). VCar and its attributes have `GUID != $ID`. +It is used by `mdl/roundtrip/viewentity_identity_test.go` (#731). + +## What was kept, and what was left out + +- **Kept verbatim:** every unit file that belongs to the project root or to + `MyFirstModule` (its domain model, security, pages, microflows, the view-entity + source document, and the project-level security, navigation and settings). + Do not edit them. +- **Left out:** the units of every other module (Administration, Atlas_*, + WorkflowCommons, …) with their rows in `TestApp.mpr`'s `Unit` table, the + 17 MB `Projects$ModuleGuidMapping` unit, and `javasource/`, + `javascriptsource/`, `theme/`, `themesource/`, `widgets/`. mxcli opens and + describes the result; it is not meant to build with mxbuild. + +## Provenance + +Copied from the `ako/TestApp` clone on 2026-09-27; the trim removed rows from the +`Unit` table and the matching files under `mprcontents/`, and nothing else. diff --git a/testdata/testapp-views/TestApp.mpr b/testdata/testapp-views/TestApp.mpr new file mode 100644 index 0000000000..9cef8019c5 Binary files /dev/null and b/testdata/testapp-views/TestApp.mpr differ diff --git a/testdata/testapp-views/mprcontents/16/74/16747dc6-b6b7-42ae-aabf-255dca2aeeaf.mxunit b/testdata/testapp-views/mprcontents/16/74/16747dc6-b6b7-42ae-aabf-255dca2aeeaf.mxunit new file mode 100644 index 0000000000..bd87fe1682 Binary files /dev/null and b/testdata/testapp-views/mprcontents/16/74/16747dc6-b6b7-42ae-aabf-255dca2aeeaf.mxunit differ diff --git a/testdata/testapp-views/mprcontents/1f/cc/1fcc2823-c303-5a27-b6c7-78b04c96c88f.mxunit b/testdata/testapp-views/mprcontents/1f/cc/1fcc2823-c303-5a27-b6c7-78b04c96c88f.mxunit new file mode 100644 index 0000000000..f533e24e35 Binary files /dev/null and b/testdata/testapp-views/mprcontents/1f/cc/1fcc2823-c303-5a27-b6c7-78b04c96c88f.mxunit differ diff --git a/testdata/testapp-views/mprcontents/25/d3/25d3e460-c53d-419e-a2a9-e1428f40b14f.mxunit b/testdata/testapp-views/mprcontents/25/d3/25d3e460-c53d-419e-a2a9-e1428f40b14f.mxunit new file mode 100644 index 0000000000..961a4fca96 Binary files /dev/null and b/testdata/testapp-views/mprcontents/25/d3/25d3e460-c53d-419e-a2a9-e1428f40b14f.mxunit differ diff --git a/testdata/testapp-views/mprcontents/34/f9/34f92135-5c65-40c1-9af9-166186ecf501.mxunit b/testdata/testapp-views/mprcontents/34/f9/34f92135-5c65-40c1-9af9-166186ecf501.mxunit new file mode 100644 index 0000000000..f1536c7890 Binary files /dev/null and b/testdata/testapp-views/mprcontents/34/f9/34f92135-5c65-40c1-9af9-166186ecf501.mxunit differ diff --git a/testdata/testapp-views/mprcontents/3f/64/3f64baa2-b85c-4999-9a3c-3cc38382eb54.mxunit b/testdata/testapp-views/mprcontents/3f/64/3f64baa2-b85c-4999-9a3c-3cc38382eb54.mxunit new file mode 100644 index 0000000000..7901692736 Binary files /dev/null and b/testdata/testapp-views/mprcontents/3f/64/3f64baa2-b85c-4999-9a3c-3cc38382eb54.mxunit differ diff --git a/testdata/testapp-views/mprcontents/4a/84/4a8475b3-14cb-4100-8963-58f2fd3a0be8.mxunit b/testdata/testapp-views/mprcontents/4a/84/4a8475b3-14cb-4100-8963-58f2fd3a0be8.mxunit new file mode 100644 index 0000000000..aa30e8a9fd Binary files /dev/null and b/testdata/testapp-views/mprcontents/4a/84/4a8475b3-14cb-4100-8963-58f2fd3a0be8.mxunit differ diff --git a/testdata/testapp-views/mprcontents/4e/70/4e70063d-53e1-477a-ab92-6b763330c293.mxunit b/testdata/testapp-views/mprcontents/4e/70/4e70063d-53e1-477a-ab92-6b763330c293.mxunit new file mode 100644 index 0000000000..4df5938f1d Binary files /dev/null and b/testdata/testapp-views/mprcontents/4e/70/4e70063d-53e1-477a-ab92-6b763330c293.mxunit differ diff --git a/testdata/testapp-views/mprcontents/55/86/5586af73-672c-4a18-9ce4-8a9c0f092ef3.mxunit b/testdata/testapp-views/mprcontents/55/86/5586af73-672c-4a18-9ce4-8a9c0f092ef3.mxunit new file mode 100644 index 0000000000..14bb9614ac Binary files /dev/null and b/testdata/testapp-views/mprcontents/55/86/5586af73-672c-4a18-9ce4-8a9c0f092ef3.mxunit differ diff --git a/testdata/testapp-views/mprcontents/57/d2/57d2ecb5-6c2f-4ffa-a0b3-a37a40da3adf.mxunit b/testdata/testapp-views/mprcontents/57/d2/57d2ecb5-6c2f-4ffa-a0b3-a37a40da3adf.mxunit new file mode 100644 index 0000000000..d5b42c0e0e Binary files /dev/null and b/testdata/testapp-views/mprcontents/57/d2/57d2ecb5-6c2f-4ffa-a0b3-a37a40da3adf.mxunit differ diff --git a/testdata/testapp-views/mprcontents/5b/27/5b276f2e-eed5-4fa9-903b-3483792cbea7.mxunit b/testdata/testapp-views/mprcontents/5b/27/5b276f2e-eed5-4fa9-903b-3483792cbea7.mxunit new file mode 100644 index 0000000000..de16901ec7 Binary files /dev/null and b/testdata/testapp-views/mprcontents/5b/27/5b276f2e-eed5-4fa9-903b-3483792cbea7.mxunit differ diff --git a/testdata/testapp-views/mprcontents/5c/68/5c6805fc-cadf-4d72-a5ef-f25da7cf4c97.mxunit b/testdata/testapp-views/mprcontents/5c/68/5c6805fc-cadf-4d72-a5ef-f25da7cf4c97.mxunit new file mode 100644 index 0000000000..d67cb634fd Binary files /dev/null and b/testdata/testapp-views/mprcontents/5c/68/5c6805fc-cadf-4d72-a5ef-f25da7cf4c97.mxunit differ diff --git a/testdata/testapp-views/mprcontents/65/21/652106e9-3f38-46c7-bcc4-2b4e4ad39160.mxunit b/testdata/testapp-views/mprcontents/65/21/652106e9-3f38-46c7-bcc4-2b4e4ad39160.mxunit new file mode 100644 index 0000000000..8d2c8f7b26 Binary files /dev/null and b/testdata/testapp-views/mprcontents/65/21/652106e9-3f38-46c7-bcc4-2b4e4ad39160.mxunit differ diff --git a/testdata/testapp-views/mprcontents/6c/1c/6c1ceba7-34e0-4d4c-b6ae-c54b487a3562.mxunit b/testdata/testapp-views/mprcontents/6c/1c/6c1ceba7-34e0-4d4c-b6ae-c54b487a3562.mxunit new file mode 100644 index 0000000000..286cd56d50 Binary files /dev/null and b/testdata/testapp-views/mprcontents/6c/1c/6c1ceba7-34e0-4d4c-b6ae-c54b487a3562.mxunit differ diff --git a/testdata/testapp-views/mprcontents/6e/16/6e16f1f7-1207-4d70-8847-c369a7dd360a.mxunit b/testdata/testapp-views/mprcontents/6e/16/6e16f1f7-1207-4d70-8847-c369a7dd360a.mxunit new file mode 100644 index 0000000000..7bfc1e2c26 Binary files /dev/null and b/testdata/testapp-views/mprcontents/6e/16/6e16f1f7-1207-4d70-8847-c369a7dd360a.mxunit differ diff --git a/testdata/testapp-views/mprcontents/78/44/7844f052-e1ef-4335-a833-4fe39c4b63a5.mxunit b/testdata/testapp-views/mprcontents/78/44/7844f052-e1ef-4335-a833-4fe39c4b63a5.mxunit new file mode 100644 index 0000000000..238b75cc94 Binary files /dev/null and b/testdata/testapp-views/mprcontents/78/44/7844f052-e1ef-4335-a833-4fe39c4b63a5.mxunit differ diff --git a/testdata/testapp-views/mprcontents/7d/06/7d06ff15-6fef-4887-a15a-4230d851a87d.mxunit b/testdata/testapp-views/mprcontents/7d/06/7d06ff15-6fef-4887-a15a-4230d851a87d.mxunit new file mode 100644 index 0000000000..46a67db273 Binary files /dev/null and b/testdata/testapp-views/mprcontents/7d/06/7d06ff15-6fef-4887-a15a-4230d851a87d.mxunit differ diff --git a/testdata/testapp-views/mprcontents/84/77/84776610-13b2-48cb-a265-d5e984d63129.mxunit b/testdata/testapp-views/mprcontents/84/77/84776610-13b2-48cb-a265-d5e984d63129.mxunit new file mode 100644 index 0000000000..2f32df0d60 Binary files /dev/null and b/testdata/testapp-views/mprcontents/84/77/84776610-13b2-48cb-a265-d5e984d63129.mxunit differ diff --git a/testdata/testapp-views/mprcontents/b6/29/b62986ab-1ec9-5c84-9d33-1f0829ffce9b.mxunit b/testdata/testapp-views/mprcontents/b6/29/b62986ab-1ec9-5c84-9d33-1f0829ffce9b.mxunit new file mode 100644 index 0000000000..573b766ade Binary files /dev/null and b/testdata/testapp-views/mprcontents/b6/29/b62986ab-1ec9-5c84-9d33-1f0829ffce9b.mxunit differ diff --git a/testdata/testapp-views/mprcontents/bd/6f/bd6f65bd-2393-539d-9388-4684a54d0646.mxunit b/testdata/testapp-views/mprcontents/bd/6f/bd6f65bd-2393-539d-9388-4684a54d0646.mxunit new file mode 100644 index 0000000000..bda52ddcb1 Binary files /dev/null and b/testdata/testapp-views/mprcontents/bd/6f/bd6f65bd-2393-539d-9388-4684a54d0646.mxunit differ diff --git a/testdata/testapp-views/mprcontents/d1/21/d121ee37-f102-56ae-b644-160f1d340296.mxunit b/testdata/testapp-views/mprcontents/d1/21/d121ee37-f102-56ae-b644-160f1d340296.mxunit new file mode 100644 index 0000000000..3fb28c32ae Binary files /dev/null and b/testdata/testapp-views/mprcontents/d1/21/d121ee37-f102-56ae-b644-160f1d340296.mxunit differ diff --git a/testdata/testapp-views/mprcontents/e1/e8/e1e8b130-26ba-43bd-b2e7-0f6b61a61008.mxunit b/testdata/testapp-views/mprcontents/e1/e8/e1e8b130-26ba-43bd-b2e7-0f6b61a61008.mxunit new file mode 100644 index 0000000000..0b4af26710 Binary files /dev/null and b/testdata/testapp-views/mprcontents/e1/e8/e1e8b130-26ba-43bd-b2e7-0f6b61a61008.mxunit differ diff --git a/testdata/testapp-views/mprcontents/f9/28/f928bb10-1832-49c8-9415-7e24495ec970.mxunit b/testdata/testapp-views/mprcontents/f9/28/f928bb10-1832-49c8-9415-7e24495ec970.mxunit new file mode 100644 index 0000000000..a93955a295 Binary files /dev/null and b/testdata/testapp-views/mprcontents/f9/28/f928bb10-1832-49c8-9415-7e24495ec970.mxunit differ diff --git a/testdata/testapp-views/mprcontents/mprname b/testdata/testapp-views/mprcontents/mprname new file mode 100644 index 0000000000..6387602eaa --- /dev/null +++ b/testdata/testapp-views/mprcontents/mprname @@ -0,0 +1 @@ +TestApp.mpr \ No newline at end of file