From f725ce9759afe18bd72e2e473ae94eb8cf73bd8c Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 26 Sep 2026 08:01:27 +0000 Subject: [PATCH 01/15] docs(lint): document every Starlark builtin and struct field; test the whole surface The write-lint-rules skill is what Starlark rules are written from, and it had drifted from the API: - seven query functions were missing (java_actions, documents, documentable_elements, navigation_targets, queues, module_cycles, database_connections), four of them used by shipped rules; - get_option() and struct() were missing from the helper table; - scheduled_event lacked repeat, on_overlap and time_zone, and project_security lacked anonymous_user_role. Document all of them, and add a test that holds the skill to the API as registered in code: - every name in buildPredeclared() must appear as `name(` in the skill; - every struct built with starlarkstruct.FromStringDict (found by parsing the package source, including dicts built in a local variable) must have a table listing exactly its fields. Structs documented another way are named in explicit maps: violation and location as helper parameters, cycle and module_cycle as inline struct{...} rows, and entity_permission in the shared permission table. Follow-up to mendixlabs/mxcli#1178, whose test covered only the entity and microflow tables. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011AvRv9GAQJbrHgrgMmfsBM --- .../skills/fix-issue/findings/cmd-mxcli.jsonl | 1 + .../skills/mendix/write-lint-rules/SKILL.md | 89 +++++ mdl/linter/starlark_skill_coverage_test.go | 327 ++++++++++++++++++ 3 files changed, 417 insertions(+) create mode 100644 mdl/linter/starlark_skill_coverage_test.go diff --git a/.claude/skills/fix-issue/findings/cmd-mxcli.jsonl b/.claude/skills/fix-issue/findings/cmd-mxcli.jsonl index c26598c57..6b1439dcf 100644 --- a/.claude/skills/fix-issue/findings/cmd-mxcli.jsonl +++ b/.claude/skills/fix-issue/findings/cmd-mxcli.jsonl @@ -128,3 +128,4 @@ {"area": "cmd/mxcli/test", "date": "2026-09-23", "symptom": "Windows: `mxcli test tests/ -p MyApp.mpr --local` fails with `local runtime: starting mxbuild serve: mxbuild --serve did not become ready` after caching a Linux ELF in %USERPROFILE%\\.mxcli\\mxbuild, and there is \"no flag, environment variable, or mechanism to redirect mxcli to the Windows mxbuild.exe already present in the Studio Pro installation\"", "cause": "Two layers. The platform part (downloading/exec'ing the Linux binary, serving from the cache instead of the resolved binary) was already fixed by #916 and #1122, both after the reporter's v0.21.0. What remained on main: `test` never registered `--mxbuild-path` — `run` gained it in #1125, but `test --local` boots through the same `ResolveMxBuildForLocal` and prints the same 'pass --mxbuild-path' guidance while answering `unknown flag`. `RunOptions` had no field and `localAppOptions` never set `LocalAppOptions.MxBuildPath`, though StartLocalApp honoured it. No env override existed anywhere", "file": "`cmd/mxcli/main.go` + `cmd_test_run.go` (flag), `testrunner/runner.go` + `localapp_options.go` (plumbing), `docker/mxbuild_platform.go` (`MxBuildPathEnv`, read in `resolveMxBuildForLocalOn` after the flag)", "insight": "**The guard for #1125 asserted its invariant against one command** — `TestErrorGuidanceNamesAFlagThatExists` checked only `runCmd`, while the guidance it polices is emitted by a resolver two commands share. When a test pins 'the advertised flag exists', enumerate the callers of the code that ADVERTISES it, not the command the report named; it now iterates `run` and `test`. **Before fixing a platform report, date it against the fixes**: the reporter's first two suggestions ('download platform-correct binary', 'auto-discover Studio Pro') were already on main, and re-implementing them would have been churn — only the override was missing. Put the env var in the resolver, not the CLI, so `run --local` and `test --local` both get it from one line; a flag-level env read would have been one more per-command copy to drift. Control: the env test runs as goos=windows with an unmatched version, so without the override it fails fast with the 'Linux binary cannot run natively on windows' refusal instead of hitting the CDN; removing only the `MxBuildPath:` line in localAppOptions fails the plumbing test for both runners. **Unverified**: no Windows host; code-level with OS-injected tests", "refs": ["mendixlabs/mxcli#1086", "mendixlabs/mxcli#1125", "mendixlabs/mxcli#916", "mendixlabs/mxcli#1122"]} {"area": "cmd/mxcli", "date": "2026-09-25", "symptom": "Shipped Starlark rules CONV009, CONV010, QUAL001, QUAL003, QUAL004 and CUSTOM002 report nanoflows and rules as microflows: \"Microflow 'NF_Foo' has 30 activities\", and documentType \"Microflow\" in the JSON and report output.", "cause": "microflows() yields all three flow flavours (one catalog table, MicroflowType MICROFLOW/NANOFLOW/RULE), and each rule hardcoded document_type=\"Microflow\" and a \"Microflow '...'\" message. The Go rules had already been fixed with Microflow.DocumentNoun() after MPR002 called a rule a microflow, but that method was never exposed to Starlark, so every Starlark rule had to re-derive it and none did.", "file": "mdl/linter/starlark.go, .claude/lint-rules/{conv009_max_microflow_objects,conv010_act_microflow_content,example_microflow,long_microflows,mccabe_complexity,orphaned_elements}.star, mdl/linter/starlark_flow_noun_test.go", "insight": "When a Go-side fix lives in a method, check whether the Starlark projection can reach it; a fix Starlark cannot call is a fix for half the rules. Exposed as document_noun / document_noun_title on the microflow struct, and the test runs every shipped rule over a fixture with one flow per flavour, requiring a finding on each (else vacuous) plus a guard that any new `for x in microflows()` rule joins the list. Skip the wrong turn of excluding nanoflows/rules from QUAL004: rule calls from decisions ARE emitted as 'call' refs (builder_references.go collectRuleCalls), so only the label was wrong.", "refs": "mendixlabs/mxcli#1178"} {"area": "cmd/mxcli", "date": "2026-09-25", "symptom": "Starlark rule API docs name microflow_type values the linter never returns: the write-lint-rules skill said \"microflow\"/\"nanoflow\", example_microflow.star (copied into every project by mxcli init) said \"Microflow\"/\"Nanoflow\", only mccabe_complexity.star had \"MICROFLOW\"/\"NANOFLOW\", and none listed \"RULE\". The skill's entity table also omitted has_created_date/has_changed_date/has_owner/has_changed_by.", "cause": "LintContext.Microflows() passes the catalog's MicroflowType through raw (MICROFLOW/NANOFLOW/RULE) while Entities() CASE-normalizes EntityType to TitleCase, so the two adjacent iterators have opposite conventions and each doc author guessed. Nothing tied the documented literals to what the API emits, so two passes over the same skill file (59db6e7b, #1165) fixed instances and left this one.", "file": ".claude/skills/mendix/write-lint-rules/SKILL.md, .claude/lint-rules/example_microflow.star, .claude/lint-rules/mccabe_complexity.star, mdl/linter/starlark_documented_values_test.go", "insight": "Fix the class, not the row: observe the emitted field names and enum values by running a Starlark rule (dir(e), e.entity_type) over a fixture holding every stored kind, then assert the skill tables, the shipped rules' `# .field - ...` headers, and every `.entity_type/.microflow_type ==` comparison in shipped rules against that observed set. Equality, not subset, for the docs -- an undocumented value (RULE) is a flavour a rule silently mistreats. Normalizing MicroflowType instead would have broken every user rule already comparing \"MICROFLOW\" correctly.", "refs": "mendixlabs/mxcli#1178, mendixlabs/mxcli#1164"} +{"area": "cmd/mxcli", "date": "2026-09-26", "symptom": "The write-lint-rules skill, which Starlark rules are written from, omitted seven query functions (java_actions, documents, documentable_elements, navigation_targets, queues, module_cycles, database_connections) plus get_option() and struct(), and fields on scheduled_event (repeat, on_overlap, time_zone) and project_security (anonymous_user_role). Four of the missing functions are used by shipped rules.", "cause": "Each builtin and field was added in code with its own PR and a docs-site paragraph at most; nothing compared the skill to buildPredeclared() or to the struct keys, so every API addition drifted from the skill by default. #1178's test pinned only the entity and microflow tables.", "file": ".claude/skills/mendix/write-lint-rules/SKILL.md, mdl/linter/starlark_skill_coverage_test.go", "insight": "Cover the whole surface at once rather than the table that was reported: an internal test ranges over (&StarlarkRule{}).buildPredeclared() for builtins and go/ast-parses every starlarkstruct.FromStringDict(starlark.String(name), dict) call in the package for struct fields (literal dicts and locally-built ones like ppDict), then requires each struct's skill table to match exactly. Structs documented elsewhere are named in explicit maps (helper params, inline struct{...} in a function row, shared permission table) so an exemption is a visible decision. The quick one-off Python diff over-reported (nested password_policy, permissions_for's entity_name) -- the test encodes those layouts instead of guessing.", "refs": "mendixlabs/mxcli#1178"} diff --git a/.claude/skills/mendix/write-lint-rules/SKILL.md b/.claude/skills/mendix/write-lint-rules/SKILL.md index 1b49ed80d..4f2457759 100644 --- a/.claude/skills/mendix/write-lint-rules/SKILL.md +++ b/.claude/skills/mendix/write-lint-rules/SKILL.md @@ -57,6 +57,12 @@ silently return empty results (issue #721). | `widgets()` | list of widget | All non-system widgets | | `snippets()` | list of snippet | All non-system snippets | | `scheduled_events()` | list of scheduled_event | All non-system scheduled events (requires MPR reader) | +| `queues()` | list of queue | All non-system task queues | +| `java_actions()` | list of java_action | All non-system, non-marketplace Java actions, each carrying its parameters | +| `database_connections()` | list of database_connection | All non-system external database connections | +| `documents()` | list of document | Every App Explorer document outside System and Marketplace modules, as one uniform projection with `folder` — for rules about where a document *lives* | +| `documentable_elements()` | list of documentable | Every element that can carry documentation, across all document types, with its `description` — for documentation sweeps. Leaves out microflows and Java actions; use `microflows()` / `java_actions()` for those | +| `navigation_targets()` | list of navigation_target | Every page a navigation profile routes to: profile home pages, role home pages and menu items. Login and not-found pages are excluded | | `rest_clients()` | list of rest_client | Consumed REST service documents (excluding platform modules) | | `rest_operations()` | list of rest_operation | Operations on consumed REST services, including their `timeout` | | `attributes_for(entity_qualified_name)` | list of attribute | Attributes for a specific entity | @@ -84,6 +90,7 @@ not fail). In a session, run `refresh catalog communities` before `lint`. | `layer_of(asset)` | int or None | Topological layer sequence number (no opinion on ordering) | | `community_of(asset)` | struct{id, label} or None | The asset's detected community (bounded context) | | `cycles()` | list of struct{id, size, members} | Dependency cycles (SCCs > 1 node) | +| `module_cycles()` | list of struct{id, size, members} | Module-level dependency cycles over every reference kind; `members` are module names. Use this, not `cycles()`, for "no circular module dependencies" — modules can reference each other through documents that form no asset-level cycle | | `module_dependencies()` | list of struct{source_module, target_module, ref_kind, edges} | Directed module→module edges | | `centrality(asset)` | struct{in, out, total, pagerank, betweenness} or None | Centrality of an asset | | `god_nodes(metric="degree"\|"pagerank"\|"betweenness", min=N)` | list of struct{asset, object_type, module_name, degree, pagerank, betweenness} | High-centrality assets above a threshold | @@ -255,8 +262,87 @@ def check(): | `module_name` | string | `"MyModule"` | | `microflow_name` | string | `"MyModule.MF_NightlyCleanup"` — resolved from catalog; raw UUID when catalog not built | | `interval_seconds` | int | `86400` — `0` for unrecognised interval type | +| `repeat` | string | Schedule variant: `"Minute"`, `"Hour"`, `"Day"`, `"Week"`, `"MonthDate"`, `"MonthWeekday"`, `"YearDate"` or `"YearWeekday"`; `""` when the event has no schedule | +| `on_overlap` | string | `"DelayNext"` or `"SkipNext"` — what happens when a run is still going at the next start | +| `time_zone` | string | Time zone the schedule is evaluated in | | `enabled` | bool | `True` if the event is active | +### queue +| Property | Type | Example | +|----------|------|---------| +| `name` | string | `"ImportQueue"` | +| `qualified_name` | string | `"Sales.ImportQueue"` | +| `module_name` | string | `"Sales"` | +| `parallelism` | string | `"3"` — an **expression**, stored as a string; do not assume it parses as an integer | +| `cluster_wide` | bool | `True` if parallelism applies across the cluster rather than per node | + +### java_action +| Property | Type | Example | +|----------|------|---------| +| `id` | string | Document UUID | +| `name` | string | `"JA_ParseJson"` | +| `qualified_name` | string | `"Sales.JA_ParseJson"` | +| `module_name` | string | `"Sales"` | +| `folder` | string | Folder path within module | +| `documentation` | string | Documentation text | +| `description` | string | Same as `documentation`, so a rule sweeping mixed document kinds can read one field name | +| `export_level` | string | `"Hidden"` or `"API"` | +| `return_type` | string | Return type | +| `parameter_count` | int | Number of parameters | +| `parameters` | list of java_action_parameter | The action's parameters, in order | + +#### java_action_parameter (nested in java_action) +| Property | Type | Example | +|----------|------|---------| +| `name` | string | `"InputString"` | +| `description` | string | Parameter documentation | +| `parameter_type` | string | Parameter type | +| `is_required` | bool | `True` if the parameter is required | + +### database_connection +| Property | Type | Example | +|----------|------|---------| +| `id` | string | Document UUID | +| `name` | string | `"LegacyDB"` | +| `qualified_name` | string | `"Integration.LegacyDB"` | +| `module_name` | string | `"Integration"` | +| `folder` | string | Folder path within module | +| `database_type` | string | Database engine of the connection | +| `query_count` | int | Number of queries defined on the connection | + +### document +Returned by `documents()`. + +| Property | Type | Example | +|----------|------|---------| +| `kind` | string | Catalog object type, upper-case: `"MICROFLOW"`, `"PAGE"`, `"WORKFLOW"`, … | +| `name` | string | `"Customer_Overview"` | +| `qualified_name` | string | `"Sales.Customer_Overview"` | +| `module_name` | string | `"Sales"` | +| `folder` | string | Folder path within module; `""` means directly in the module root | + +### documentable +Returned by `documentable_elements()`. + +| Property | Type | Example | +|----------|------|---------| +| `kind` | string | Mendix term, TitleCase: `"Page"`, `"Enumeration"`, `"Workflow"`, … | +| `name` | string | `"OrderStatus"` | +| `qualified_name` | string | `"Sales.OrderStatus"` | +| `module_name` | string | `"Sales"` | +| `description` | string | Documentation text, whichever of the element's Documentation/Description properties holds it | + +### navigation_target +Returned by `navigation_targets()`. + +| Property | Type | Example | +|----------|------|---------| +| `profile` | string | Navigation profile: `"Responsive"`, `"Phone"`, `"Tablet"`, … | +| `kind` | string | `"home"`, `"role_home"` or `"menu"` | +| `role` | string | User role, for a `"role_home"` target; `""` otherwise | +| `caption` | string | Menu item caption, for a `"menu"` target; `""` otherwise | +| `page` | string | Qualified name of the target page | + ### xpath_expression Returned by `xpath_expressions()`. Each row represents one XPath constraint used in a retrieve action, access rule, or widget data source. @@ -449,6 +535,7 @@ Returned by `project_security()`. Returns `none` if no MPR reader is available. | `enable_guest_access` | bool | Whether anonymous/guest access is enabled | | `check_security` | bool | Whether security checking is active | | `strict_mode` | bool | Strict security mode | +| `anonymous_user_role` | string | Name of the project's guest user role, the role anonymous users get. Read `enable_guest_access` too: the role name can stay set while guest access is off | | `password_policy` | struct | Nested password policy settings | #### password_policy (nested in project_security) @@ -469,6 +556,8 @@ Returned by `project_security()`. Returns `none` if no MPR reader is available. | `is_pascal_case(s)` | Returns True if string is PascalCase | | `is_camel_case(s)` | Returns True if string is camelCase | | `matches(s, pattern)` | Returns True if string matches regex | +| `get_option(key, default?)` | The rule's option `key` from the `options:` block under its rule ID in `.claude/lint-config.yaml`, or `default` (`None` if omitted) when unset | +| `struct(**kwargs)` | Build an ad-hoc struct, e.g. `struct(name="x", count=1)`, to group values inside a rule | ## Common Patterns diff --git a/mdl/linter/starlark_skill_coverage_test.go b/mdl/linter/starlark_skill_coverage_test.go new file mode 100644 index 000000000..8e741d473 --- /dev/null +++ b/mdl/linter/starlark_skill_coverage_test.go @@ -0,0 +1,327 @@ +// SPDX-License-Identifier: Apache-2.0 + +package linter + +import ( + "go/ast" + "go/parser" + "go/token" + "os" + "regexp" + "slices" + "sort" + "strconv" + "strings" + "testing" +) + +// The write-lint-rules skill is what a Starlark rule gets written from, and it +// has drifted from the API every time the API moved: fields added and never +// documented (the entity audit members, scheduled_event's schedule fields), +// whole query functions missing (java_actions(), queues(), ...), and literals +// the linter never emits (#1164, #1178). A field or function missing from the +// skill is one no rule author will find; a documented one that does not exist +// is a rule that fails at load time or, worse, reads a default. +// +// These tests hold the skill to the API as registered in code: every builtin in +// buildPredeclared, and every struct any builtin returns, with exactly its +// fields. starlark_documented_values_test.go pins the enum literals. + +const coverageSkillPath = "../../.claude/skills/mendix/write-lint-rules/SKILL.md" + +func TestLintSkillDocumentsEveryBuiltin(t *testing.T) { + skill := readCoverageSkill(t) + var names []string + for name := range (&StarlarkRule{}).buildPredeclared() { + names = append(names, name) + } + sort.Strings(names) + if len(names) < 10 { + t.Fatalf("buildPredeclared returned %d names -- the enumeration is broken", len(names)) + } + for _, name := range names { + if !strings.Contains(skill, "`"+name+"(") { + t.Errorf("builtin %s() is not documented in the write-lint-rules skill", name) + } + } +} + +// Structs documented somewhere other than their own "### name" table. +var ( + // Built by the rule author; the fields are the helper's parameters. + helperStructs = map[string]string{"violation": "violation", "location": "location"} + // Graph facts, documented inline as struct{...} in the function table row. + inlineStructs = map[string]string{"cycle": "cycles", "module_cycle": "module_cycles"} + // One table documents both: permissions_for() adds entity_name and drops + // element_type/element_name, and the table says which rows are which. + sharedSections = map[string]string{"entity_permission": "permission"} +) + +func TestLintSkillDocumentsEveryStructField(t *testing.T) { + skill := readCoverageSkill(t) + api := starlarkStructFields(t) + for _, want := range []string{"entity", "microflow", "project_security", "password_policy", "expr"} { + if len(api[want]) == 0 { + t.Fatalf("found no fields for struct %q in the package source -- the scan is broken", want) + } + } + sections := skillSections(skill) + + shared := map[string][]string{} + for _, name := range sortedKeys(api) { + fields := api[name] + switch { + case helperStructs[name] != "": + assertFieldSet(t, name+" (parameters of "+name+"())", helperParams(t, skill, helperStructs[name]), fields) + case inlineStructs[name] != "": + assertFieldSet(t, name+" (inline in "+inlineStructs[name]+"())", inlineStructFields(t, skill, inlineStructs[name]), fields) + case name == "expr": + // One row per node kind; the fields are the backticked names in the + // "Additional fields" column, plus the kind every node carries. + doc := []string{"kind"} + for _, row := range sections["expr"] { + doc = append(doc, typedFields(row.rest)...) + } + assertFieldSet(t, "expr", uniq(doc), fields) + default: + section := name + if s, ok := sharedSections[name]; ok { + section = s + } + if _, ok := sections[section]; !ok { + t.Errorf("struct %s has no ### %s table in the skill; fields: %v", name, section, fields) + continue + } + shared[section] = uniq(append(shared[section], fields...)) + } + } + for section, fields := range shared { + var doc []string + for _, row := range sections[section] { + doc = append(doc, row.name) + } + assertFieldSet(t, "### "+section+" table", uniq(doc), fields) + } +} + +// starlarkStructFields returns, for every struct type name the package builds +// with starlarkstruct.FromStringDict(starlark.String("name"), dict), the keys of +// dict -- a StringDict literal, or a local variable assigned one and then +// indexed. Dynamic names (rowToStruct) are skipped: their keys are SQL columns +// and are documented inline in the graph-function table. +func starlarkStructFields(t *testing.T) map[string][]string { + t.Helper() + fset := token.NewFileSet() + pkgs, err := parser.ParseDir(fset, ".", func(fi os.FileInfo) bool { + return !strings.HasSuffix(fi.Name(), "_test.go") + }, 0) + if err != nil { + t.Fatal(err) + } + out := map[string][]string{} + for _, f := range pkgs["linter"].Files { + for _, decl := range f.Decls { + fn, ok := decl.(*ast.FuncDecl) + if !ok || fn.Body == nil { + continue + } + locals := localDictKeys(fn.Body) + ast.Inspect(fn.Body, func(n ast.Node) bool { + call, ok := n.(*ast.CallExpr) + if !ok || !isSelector(call.Fun, "starlarkstruct", "FromStringDict") || len(call.Args) != 2 { + return true + } + name, ok := starlarkStringLiteral(call.Args[0]) + if !ok { + return true + } + var keys []string + switch d := call.Args[1].(type) { + case *ast.CompositeLit: + keys = literalKeys(d) + case *ast.Ident: + keys = locals[d.Name] + } + out[name] = uniq(append(out[name], keys...)) + return true + }) + } + } + return out +} + +func localDictKeys(body *ast.BlockStmt) map[string][]string { + out := map[string][]string{} + ast.Inspect(body, func(n ast.Node) bool { + as, ok := n.(*ast.AssignStmt) + if !ok || len(as.Lhs) != 1 || len(as.Rhs) != 1 { + return true + } + switch l := as.Lhs[0].(type) { + case *ast.Ident: + if lit, ok := as.Rhs[0].(*ast.CompositeLit); ok && isSelector(lit.Type, "starlark", "StringDict") { + out[l.Name] = append(out[l.Name], literalKeys(lit)...) + } + case *ast.IndexExpr: + id, ok := l.X.(*ast.Ident) + key, ok2 := l.Index.(*ast.BasicLit) + if ok && ok2 && key.Kind == token.STRING { + k, _ := strconv.Unquote(key.Value) + out[id.Name] = append(out[id.Name], k) + } + } + return true + }) + return out +} + +func literalKeys(lit *ast.CompositeLit) []string { + var keys []string + for _, e := range lit.Elts { + if kv, ok := e.(*ast.KeyValueExpr); ok { + if b, ok := kv.Key.(*ast.BasicLit); ok && b.Kind == token.STRING { + k, _ := strconv.Unquote(b.Value) + keys = append(keys, k) + } + } + } + return keys +} + +func starlarkStringLiteral(e ast.Expr) (string, bool) { + call, ok := e.(*ast.CallExpr) + if !ok || !isSelector(call.Fun, "starlark", "String") || len(call.Args) != 1 { + return "", false + } + b, ok := call.Args[0].(*ast.BasicLit) + if !ok || b.Kind != token.STRING { + return "", false + } + s, err := strconv.Unquote(b.Value) + return s, err == nil +} + +func isSelector(e ast.Expr, pkg, name string) bool { + sel, ok := e.(*ast.SelectorExpr) + if !ok || sel.Sel.Name != name { + return false + } + id, ok := sel.X.(*ast.Ident) + return ok && id.Name == pkg +} + +type skillRow struct{ name, rest string } + +// skillSections maps each "### name" / "#### name ..." heading to the table +// rows under it whose first cell is a backticked identifier. +func skillSections(skill string) map[string][]skillRow { + heading := regexp.MustCompile(`^#{3,4} (\w+)`) + row := regexp.MustCompile("^\\|\\s*`\"?(\\w+)\"?`\\s*\\|(.*)$") + out := map[string][]skillRow{} + current := "" + for _, line := range strings.Split(skill, "\n") { + if strings.HasPrefix(line, "#") { + current = "" + if m := heading.FindStringSubmatch(line); m != nil { + current = m[1] + out[current] = nil + } + continue + } + if m := row.FindStringSubmatch(line); m != nil && current != "" { + out[current] = append(out[current], skillRow{m[1], m[2]}) + } + } + return out +} + +// helperParams returns the parameter names in the helper table's +// `name(a, b, c?)` signature. +func helperParams(t *testing.T, skill, fn string) []string { + t.Helper() + m := regexp.MustCompile("`" + fn + `\(([^)]*)\)` + "`").FindStringSubmatch(skill) + if m == nil { + t.Errorf("skill has no `%s(...)` signature", fn) + return nil + } + var out []string + for _, p := range strings.Split(m[1], ",") { + out = append(out, strings.TrimSuffix(strings.TrimSpace(p), "?")) + } + return out +} + +// inlineStructFields returns the names in `struct{a, b, c}` on the function +// table row for fn(). +func inlineStructFields(t *testing.T, skill, fn string) []string { + t.Helper() + for _, line := range strings.Split(skill, "\n") { + if !strings.HasPrefix(line, "| `"+fn+"(") { + continue + } + m := regexp.MustCompile(`struct\{([^}]*)\}`).FindStringSubmatch(line) + if m == nil { + t.Errorf("%s() row documents no struct{...}: %s", fn, line) + return nil + } + var out []string + for _, f := range strings.Split(m[1], ",") { + out = append(out, strings.TrimSpace(f)) + } + return out + } + t.Errorf("skill has no function-table row for %s()", fn) + return nil +} + +// typedFields returns the names in "`name` (type)" pairs, skipping backticked +// operators and keywords in the description column. +func typedFields(s string) []string { + var out []string + for _, m := range regexp.MustCompile("`(\\w+)` \\(").FindAllStringSubmatch(s, -1) { + out = append(out, m[1]) + } + return out +} + +func assertFieldSet(t *testing.T, what string, doc, api []string) { + t.Helper() + var missing, phantom []string + for _, f := range api { + if !slices.Contains(doc, f) { + missing = append(missing, f) + } + } + for _, f := range doc { + if !slices.Contains(api, f) { + phantom = append(phantom, f) + } + } + if len(missing) > 0 || len(phantom) > 0 { + t.Errorf("%s: undocumented %v, documented but not in the API %v", what, missing, phantom) + } +} + +func readCoverageSkill(t *testing.T) string { + t.Helper() + b, err := os.ReadFile(coverageSkillPath) + if err != nil { + t.Fatal(err) + } + return string(b) +} + +func uniq(s []string) []string { + out := slices.Clone(s) + sort.Strings(out) + return slices.Compact(out) +} + +func sortedKeys(m map[string][]string) []string { + keys := make([]string, 0, len(m)) + for k := range m { + keys = append(keys, k) + } + sort.Strings(keys) + return keys +} From a8f136acc53bf8a46f3402aae65830a5c9924e92 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 26 Sep 2026 08:25:29 +0000 Subject: [PATCH 02/15] fix(alter): write a pluggable property to the field its kind declares (#1201) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `ALTER PAGE … SET ImageUrl = … ON img1` printed "Altered page" and changed nothing. setPluggableWidgetPropertyMut wrote every value to PrimitiveValue; the pluggable Image's imageUrl is a TextTemplate, which DESCRIBE, mx check and the runtime read instead. The setter now reads the property's declared ValueType.Type and dispatches through columnValueField, the same schema dispatch the DataGrid 2 column setter already uses: TextTemplate updates the template text, Expression writes Expression, primitives keep PrimitiveValue. Kinds a plain value cannot express (Action, DataSource, Image, Icon, Attribute, Widgets, …) are refused instead of reported as success, as is a list value (#750) and a null (hidden, #574) text template. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01H79ZLyf5E4LuV48TmbDdq9 --- .../fix-issue/findings/mdl-backend.jsonl | 1 + ...-alter-page-set-pluggable-texttemplate.mdl | 24 +++ mdl/backend/pagemutator/mutator.go | 101 +++++++++-- .../pluggable_property_kind_test.go | 169 ++++++++++++++++++ 4 files changed, 278 insertions(+), 17 deletions(-) create mode 100644 mdl-examples/bug-tests/1201-alter-page-set-pluggable-texttemplate.mdl create mode 100644 mdl/backend/pagemutator/pluggable_property_kind_test.go diff --git a/.claude/skills/fix-issue/findings/mdl-backend.jsonl b/.claude/skills/fix-issue/findings/mdl-backend.jsonl index ed2d0cbed..92c3e92bc 100644 --- a/.claude/skills/fix-issue/findings/mdl-backend.jsonl +++ b/.claude/skills/fix-issue/findings/mdl-backend.jsonl @@ -133,3 +133,4 @@ {"area": "mdl/backend", "date": "2026-09-23", "symptom": "`CREATE OR MODIFY PERSISTENT ENTITY` was REFUSED by the #1119 storage-GUID guard on a doctype script that had been passing for months: `failed to update entity: refusing to write unit d82b0484-…: 1 element(s) kept their $ID but would be written with a different GUID — dff2ced1-… (DomainModels$Attribute): stored 4b52b36b-…, would write dff2ced1-…`. Two independent defects wore that one message.", "cause": "(1) A REAL data loss the guard caught: `mergeDeclaredOntoStoredEntity` sets `merged.Attributes = declared.Attributes` and `merged.Indexes = declared.Indexes` — the lists the STATEMENT declares, built from text by the visitor and carrying no element ID — and `carryChildIdentity` keyed entirely on that ID, reading an empty one as 'a genuinely new member, so a fresh GUID is right'. Every attribute of a re-declared entity was therefore re-minted, i.e. #1119 through a second executor path. (2) A FALSE POSITIVE in the guard itself: `canon.TransplantIDs` pairs STRUCTURALLY ($Type + shape, LCS-anchored), so on a statement that drops six differently-named attributes and adds one, it paired the NEW attribute with a REMOVED one and handed it that stored `$ID`; the codec had written `GUID = $ID` and the transplant substitutes over every 16-byte binary, so the GUID followed. The guard's premise — written into its own doc comment as 'unambiguously' — that a shared `$ID` after the transplant means the same element, is false.", "file": "`mdl/backend/modelsdk/domainmodel_child_identity.go` (`carryAttributeIdentity`, `carryIndexIdentity`), `modelsdk/canon/storageguid.go` (`sameMember`, `elementGUIDs`)", "insight": "ONE ERROR MESSAGE, TWO DEFECTS, AND FIXING EITHER ALONE LEAVES IT RED — which is why the first fix (the name fallback) changed nothing and the SAME element and GUIDs came back byte-for-byte. That repetition was the signal: an identical failure after a real fix means the reproduction is exercising a different code path than the one reasoned about. What settled it was describing the actual subject: the marketplace `PublishedBusinessEvent` has six attributes and NONE is named `EventId`, so there was no member to carry — the pairing itself was spurious. Reproduce against the real stored document before believing any theory about which elements correspond. METHOD that made this cheap: the CI failure reproduced locally in 0.5s as a backend unit test (strip the IDs off a fixture entity's attributes, call UpdateEntity) versus 26s for the integration subtest, but ONLY the integration subtest could have found the second defect, because the false pairing needs a real drop-six-add-one document. Run both. TRADE-OFF worth restating: the guard now pairs on `$ID` + `$Type` + `Name`, which loses one arm — a RENAME that re-mints a GUID is no longer refused, since the name is what changed — and that arm is covered directly by the carry tests where it is decidable. A backstop that refuses correct writes is worse than a backstop with a hole: the first makes documented statements unusable, and this one already had. Also: feeding a deliberately-approximate pairing to a guard promotes its error rate into refusals. TransplantIDs' correctness bar is low ON PURPOSE (a wrong match only makes a diff bigger); anything that reads its output as identity has to add its own test of identity.", "refs": ["mendixlabs/mxcli#1119", "mendixlabs/mxcli#1169", "ako/mxcli#643"], "ce": []} {"area":"mdl/backend","date":"2026-09-25","symptom":"A data view with `DataSource: nanoflow Module.NF` (e.g. describe → exec of Feedback v4.0.2's FeedbackModule.ShareFeedback) passes `mxcli check` and exec, then mxbuild 11.13.0 reports CE2633 \"No nanoflow configured for the data source of this data view\". Same result through `alter page … set DataSource = nanoflow X on dv`","cause":"Both writers nested the name in a `Forms$NanoflowSettings` child (ParameterMappings marker 3) by analogy with `Forms$MicroflowSource`, which really does nest `Forms$MicroflowSettings`. Studio Pro's `Forms$NanoflowSource` is FLAT: ForceFullObjects, Nanoflow, ParameterMappings (marker 2) directly on the source — exactly gen's shape. mxbuild found no Nanoflow key. The raw nested builder also never read d.ParameterMappings, so a parameterized source nanoflow lost its arguments. On the read side, the ALTER PAGE flow-context lookup (`flowFromDataSourceDoc`) and describe's argument reader (`flowSourceArgs`) only knew the nested shape, so Studio Pro-authored nanoflow sources yielded no entity context / no arguments","file":"`mdl/backend/modelsdk/widget_write_legacy_gaps.go` (`nanoflowSourceToGen` via gen + `Forms$NanoflowSource` TypeDefaults in `widget_write.go`), `mdl/backend/pagemutator/mutator.go` (`serializeDataSourceBson`, `flowFromDataSourceDoc`), `mdl/executor/cmd_pages_describe_datasource.go` (`flowSourceArgs`)","insight":"**The code comment asserted the wrong shape as a measured fact** (\"Studio Pro nests it in a Forms$NanoflowSettings child … Legacy's shape is the one with a working project behind it\") and a unit test pinned it — both were parity-with-legacy, never measured. `Forms$NanoflowSettings` is not a type in modelsdk/gen or generated/metamodel: **when gen and a hand-rolled builder disagree about a type's shape, grep gen for the type the builder invents before trusting the builder**. What settled it in one step: a 60-line scanner that `bson.Unmarshal`s every mprcontents unit and prints the key-set (with list markers) of each `$Type` instance — 5 of 5 flat nanoflow sources, and 4 of 4 microflow sources nested as gen says, so the microflow path needed nothing. **Enumerate every writer of the type, not just the reported one**: the ALTER PAGE setter had its own copy of the same wrong literal, and the read-side lookups keyed on the wrong shape meant Studio Pro pages were the ones silently mis-read. Readers keep the nested fallback for pages written before the fix. Verified: exec + `mx check` 11.13.0 CE2633 → 0 errors for CREATE PAGE (ShareFeedback round trip, repro script) and ALTER PAGE; ShareFeedback's dataView5 DataSource ndsl now matches Studio Pro exactly. Control: implementation reverted → the 5 new tests fail with the nested key set. Repro `mdl-examples/bug-tests/dataview-nanoflow-source-ce2633.mdl`","refs":[],"ce":["CE2633"]} {"area": "mdl/backend", "date": "2026-09-25", "symptom": "`returns list of pEntity` for a declared type parameter produced mx check CE1613 \"The selected entity '.pEntity' no longer exists.\" on the action and on a `list of pEntity` parameter; a Studio Pro \"List of \" read back as a bare `List` (DESCRIBE and catalog) and a rewrite of it would serialize a list of an unnamed entity.", "cause": "types.ListType carried only Entity. The reader handled only a ConcreteEntityType list element, the writer always emitted one, and CREATE sent `list of T` down the entity path (Module \"\" + \".\" + T).", "file": "mdl/types/javaaction_types.go (ListType.TypeParameterID), mdl/backend/modelsdk/java_read.go (listTypeFromGen), java_write.go (codeActionListTypeToGen), javascript_read.go, mdl/executor/cmd_javaactions.go (listOfTypeParameter)", "insight": "The Model SDK is the arbiter for which element a slot accepts: `ParameterizedEntityType.createInListTypeUnderParameter` (metamodel 7.21.0+) settles that a list element may be a type parameter, so no version gate. No fixture had a Studio Pro-authored instance, so the evidence is mx check on 11.6.6: previous build CE1613 x2, fixed build 0 errors, clean baseline 0. The JavaScript writer reuses the Java converter, so one write fix covers both; the JS reader is separate raw-map code and needed its own case.", "refs": ["mendixlabs/mxcli#1183"]} +{"area": "mdl/backend", "date": "2026-09-26", "symptom": "`ALTER PAGE … SET ImageUrl = '…' ON img1` (pluggable Image) prints \"Altered page\" and changes nothing: DESCRIBE still shows the old URL. Same silent success for SET on any pluggable widget property of kind Expression, Image, Icon, Action, DataSource, Attribute or Widgets", "cause": "setPluggableWidgetPropertyMut (mdl/backend/pagemutator/mutator.go) wrote every value to Value.PrimitiveValue. imageUrl is a TextTemplate; readers take Value.TextTemplate. The DataGrid 2 column setter had the identical defect fixed on 2026-08-18 (columnValueField) — the widget-level setter sitting next to it was never given the same schema dispatch.", "file": "mdl/backend/pagemutator/mutator.go (setPluggableWidgetPropertyMut, buildPropKindMap), test pluggable_property_kind_test.go, example mdl-examples/bug-tests/1201-alter-page-set-pluggable-texttemplate.mdl", "insight": "When a fix lands on one setter for 'write the field the schema declares', grep the sibling setters in the same file for the same always-PrimitiveValue write — a WidgetValue carries every variant field, so the wrong write never errors and every signal stays green. The live control is cheap without mxbuild: copy testdata/expr-checker, create an image with URL A, ALTER to B, describe (pre-fix binary shows A). A null TextTemplate means the slot is hidden (#574); ALTER does not re-run visibility, so refuse rather than build an envelope there. Not fixed here: a pluggable boolean SET stores \"yes\"/\"no\" — check what CREATE stores before changing it.", "refs": ["mendixlabs/mxcli#1201", "mendixlabs/mxcli#1069", "mendixlabs/mxcli#750", "mendixlabs/mxcli#574"], "rules": []} diff --git a/mdl-examples/bug-tests/1201-alter-page-set-pluggable-texttemplate.mdl b/mdl-examples/bug-tests/1201-alter-page-set-pluggable-texttemplate.mdl new file mode 100644 index 000000000..f7435e47a --- /dev/null +++ b/mdl-examples/bug-tests/1201-alter-page-set-pluggable-texttemplate.mdl @@ -0,0 +1,24 @@ +-- #1201: `ALTER PAGE … SET ImageUrl = … ON ` printed "Altered page" +-- and changed nothing. +-- +-- The pluggable Image widget's `imageUrl` is a TextTemplate-kind property. The +-- ALTER setter wrote every pluggable value to PrimitiveValue, a field DESCRIBE, +-- mx check and the runtime never read for that kind, so the page kept the old +-- URL. It now writes the field the widget schema declares — the template text +-- for TextTemplate, Expression for Expression, PrimitiveValue for primitives — +-- and refuses kinds a plain value cannot express (Action, DataSource, Image, …) +-- instead of reporting success. +-- +-- Expected: the describe below shows 'https://b.example/y.png'. + +create module IMod; + +create page IMod.P1 ( title: 'P1', layout: Atlas_Core.Atlas_Default ) { + image img1 (ImageType: imageUrl, ImageUrl: 'https://a.example/x.png') +}; + +alter page IMod.P1 { + set ImageUrl = 'https://b.example/y.png' on img1 +}; + +describe page IMod.P1; diff --git a/mdl/backend/pagemutator/mutator.go b/mdl/backend/pagemutator/mutator.go index 32eaa2bbc..a458d783f 100644 --- a/mdl/backend/pagemutator/mutator.go +++ b/mdl/backend/pagemutator/mutator.go @@ -1798,6 +1798,31 @@ func buildPropKeyMap(widgetDoc bson.D) map[string]string { return m } +// buildPropKindMap builds a TypePointer ID -> declared value kind map +// (PropertyTypes[].ValueType.Type) for a widget's top-level properties. Kept +// apart from buildPropKeyMap, which has many callers that want only the key. +func buildPropKindMap(widgetDoc bson.D) map[string]string { + m := make(map[string]string) + objType := bsonnav.DGetDoc(bsonnav.DGetDoc(widgetDoc, "Type"), "ObjectType") + if objType == nil { + return m + } + for _, pt := range bsonnav.DGetArrayElements(bsonnav.DGet(objType, "PropertyTypes")) { + ptDoc, ok := pt.(bson.D) + if !ok { + continue + } + id := bsonnav.ExtractBinaryIDFromDoc(bsonnav.DGet(ptDoc, "$ID")) + if id == "" { + continue + } + if kind := bsonnav.DGetString(bsonnav.DGetDoc(ptDoc, "ValueType"), "Type"); kind != "" { + m[id] = kind + } + } + return m +} + // buildColumnPropKeyMap builds a TypePointer ID -> PropertyKey map for column properties. func buildColumnPropKeyMap(widgetDoc bson.D, columnsTypePointerID string) (map[string]string, map[string]string) { m := make(map[string]string) @@ -2329,6 +2354,9 @@ func settableColumnProperties(propKeyMap map[string]string) string { // Studio Pro does not read, did not survive a DESCRIBE round trip, and left // `mx check` at 0 errors. // +// Shared by setPluggableWidgetPropertyMut, which had the same always- +// PrimitiveValue bug for a widget's own properties (mendixlabs/mxcli#1201). +// // The second return reports whether the kind is settable at all. Attribute, // datasource, action and widget-valued properties need a structured value, not a // string, so ALTER refuses them rather than writing a plausible-looking wrong one. @@ -2960,6 +2988,13 @@ func setPluggableWidgetPropertyMut(widget bson.D, propName string, value any) er // now one function — a resolver that disagrees with itself is the failure // this whole area keeps producing. propTypeKeyMap := buildPropKeyMap(widget) + propKindMap := buildPropKindMap(widget) + + // No pluggable property takes a list. A bracketed value arrives as a + // []string, and %v fused its tokens into one string written as success. + if _, isList := value.([]string); isList { + return errExpressionNotAString(propName, value) + } props := bsonnav.DGetArrayElements(bsonnav.DGet(obj, "Properties")) for _, prop := range props { @@ -2972,26 +3007,58 @@ func setPluggableWidgetPropertyMut(widget bson.D, propName string, value any) er if propKey == "" || !strings.EqualFold(propKey, propName) { continue } - if valDoc := bsonnav.DGetDoc(propDoc, "Value"); valDoc != nil { - switch v := value.(type) { - case string: - bsonnav.DSet(valDoc, "PrimitiveValue", v) - case bool: - if v { - bsonnav.DSet(valDoc, "PrimitiveValue", "yes") - } else { - bsonnav.DSet(valDoc, "PrimitiveValue", "no") - } - case int: - bsonnav.DSet(valDoc, "PrimitiveValue", fmt.Sprintf("%d", v)) - case float64: - bsonnav.DSet(valDoc, "PrimitiveValue", fmt.Sprintf("%g", v)) - default: - bsonnav.DSet(valDoc, "PrimitiveValue", fmt.Sprintf("%v", v)) + valDoc := bsonnav.DGetDoc(propDoc, "Value") + if valDoc == nil { + return fmt.Errorf("property %q has no Value map", propName) + } + // Which field the value belongs in is the schema's to say, exactly as + // for a DataGrid 2 column (columnValueField). Writing PrimitiveValue for + // every kind made `SET ImageUrl` on an image report "Altered page" and + // change nothing: imageUrl is a TextTemplate, and DESCRIBE, mx check and + // the runtime all read the template (mendixlabs/mxcli#1201). An empty + // kind — a document with no ValueType — stays primitive, as before. + kind := propKindMap[typePointerID] + field, settable := columnValueField(kind) + if !settable { + return fmt.Errorf( + "pluggable property %q holds a value of kind %s, which ALTER cannot set from a plain value — "+ + "rewrite the widget with CREATE OR REPLACE PAGE (or ALTER PAGE REPLACE) instead", + propName, kind) + } + switch field { + case "TextTemplate": + // A null template is how #574 stores one its condition hides. ALTER + // does not re-run visibility, so building one here would put text in + // a pruned slot — refused, not created. + textTemplate := bsonnav.DGetDoc(valDoc, "TextTemplate") + if textTemplate == nil || !updateClientTemplateText(textTemplate, fmt.Sprintf("%v", value)) { + return fmt.Errorf( + "pluggable property %q has no text template to update — it is hidden by the widget's "+ + "current configuration; set the property that enables it with CREATE OR REPLACE PAGE", + propName) } return nil + case "Expression": + bsonnav.DSet(valDoc, "Expression", fmt.Sprintf("%v", value)) + return nil + } + switch v := value.(type) { + case string: + bsonnav.DSet(valDoc, "PrimitiveValue", v) + case bool: + if v { + bsonnav.DSet(valDoc, "PrimitiveValue", "yes") + } else { + bsonnav.DSet(valDoc, "PrimitiveValue", "no") + } + case int: + bsonnav.DSet(valDoc, "PrimitiveValue", fmt.Sprintf("%d", v)) + case float64: + bsonnav.DSet(valDoc, "PrimitiveValue", fmt.Sprintf("%g", v)) + default: + bsonnav.DSet(valDoc, "PrimitiveValue", fmt.Sprintf("%v", v)) } - return fmt.Errorf("property %q has no Value map", propName) + return nil } return fmt.Errorf("pluggable property %q not found", propName) } diff --git a/mdl/backend/pagemutator/pluggable_property_kind_test.go b/mdl/backend/pagemutator/pluggable_property_kind_test.go new file mode 100644 index 000000000..9f7adb1a4 --- /dev/null +++ b/mdl/backend/pagemutator/pluggable_property_kind_test.go @@ -0,0 +1,169 @@ +// SPDX-License-Identifier: Apache-2.0 + +package pagemutator + +import ( + "strings" + "testing" + + "go.mongodb.org/mongo-driver/bson" + "go.mongodb.org/mongo-driver/bson/primitive" + + "github.com/mendixlabs/mxcli/mdl/backend/bsonnav" +) + +// kindedPluggableWidget builds a CustomWidget with one property keyed propKey, +// whose PropertyType declares ValueType.Type = kind, and whose stored Value +// carries every variant field at once — as a real WidgetValue does. The +// TextTemplate holds oldText when withTemplate is set, and is null otherwise. +func kindedPluggableWidget(propKey, kind, oldText string, withTemplate bool) bson.D { + typeID := primitive.Binary{Subtype: 0x04, Data: []byte{ + 0x21, 0x22, 0x23, 0x24, 0x25, 0x26, 0x27, 0x28, + 0x29, 0x2a, 0x2b, 0x2c, 0x2d, 0x2e, 0x2f, 0x30, + }} + var tmpl any + if withTemplate { + tmpl = bson.D{ + {Key: "$Type", Value: "Forms$ClientTemplate"}, + {Key: "Parameters", Value: bson.A{int32(2)}}, + {Key: "Template", Value: bson.D{ + {Key: "$Type", Value: "Texts$Text"}, + {Key: "Items", Value: bson.A{int32(3), bson.D{ + {Key: "$Type", Value: "Texts$Translation"}, + {Key: "LanguageCode", Value: "en_US"}, + {Key: "Text", Value: oldText}, + }}}, + }}, + } + } + return bson.D{ + {Key: "$Type", Value: "CustomWidgets$CustomWidget"}, + {Key: "Name", Value: "img1"}, + {Key: "Type", Value: bson.D{ + {Key: "$Type", Value: "CustomWidgets$CustomWidgetType"}, + {Key: "ObjectType", Value: bson.D{ + {Key: "PropertyTypes", Value: bson.A{int32(2), bson.D{ + {Key: "$ID", Value: typeID}, + {Key: "PropertyKey", Value: propKey}, + {Key: "ValueType", Value: bson.D{{Key: "Type", Value: kind}}}, + }}}, + }}, + }}, + {Key: "Object", Value: bson.D{ + {Key: "Properties", Value: bson.A{int32(2), bson.D{ + {Key: "TypePointer", Value: typeID}, + {Key: "Value", Value: bson.D{ + {Key: "Expression", Value: ""}, + {Key: "PrimitiveValue", Value: ""}, + {Key: "TextTemplate", Value: tmpl}, + }}, + }}}, + }}, + } +} + +func kindedValue(t *testing.T, w bson.D) bson.D { + t.Helper() + props := bsonnav.DGetArrayElements(bsonnav.DGet(bsonnav.DGetDoc(w, "Object"), "Properties")) + return bsonnav.DGetDoc(props[0].(bson.D), "Value") +} + +func templateText(t *testing.T, valDoc bson.D) string { + t.Helper() + tmpl := bsonnav.DGetDoc(valDoc, "TextTemplate") + if tmpl == nil { + return "" + } + items := bsonnav.DGetArrayElements(bsonnav.DGet(bsonnav.DGetDoc(tmpl, "Template"), "Items")) + for _, it := range items { + if d, ok := it.(bson.D); ok && bsonnav.DGetString(d, "$Type") == "Texts$Translation" { + return bsonnav.DGetString(d, "Text") + } + } + return "" +} + +// mendixlabs/mxcli#1201: `ALTER PAGE … SET ImageUrl = … ON img1` printed +// "Altered page" and changed nothing. The pluggable Image widget's `imageUrl` +// is a TextTemplate-kind property; the setter wrote PrimitiveValue, which +// DESCRIBE, mx check and the runtime never read. +func TestSetPluggableProperty_TextTemplateKindWritesTheTemplate(t *testing.T) { + w := kindedPluggableWidget("imageUrl", "TextTemplate", "https://a.example/x.png", true) + if err := setPluggableWidgetPropertyMut(w, "ImageUrl", "https://b.example/y.png"); err != nil { + t.Fatalf("set: %v", err) + } + val := kindedValue(t, w) + if got := templateText(t, val); got != "https://b.example/y.png" { + t.Errorf("TextTemplate text = %q, want the new URL", got) + } + if pv := bsonnav.DGetString(val, "PrimitiveValue"); pv != "" { + t.Errorf("value also written to PrimitiveValue (%q), a field nothing reads for this kind", pv) + } +} + +// A null TextTemplate is how a template stores while its condition hides it +// (#574). ALTER does not re-run visibility, so writing there would put text in +// a pruned slot; it is refused rather than reported as success. +func TestSetPluggableProperty_NullTextTemplateIsRefused(t *testing.T) { + w := kindedPluggableWidget("imageUrl", "TextTemplate", "", false) + err := setPluggableWidgetPropertyMut(w, "ImageUrl", "https://b.example/y.png") + if err == nil { + t.Fatal("expected an error for a null (hidden) text template, got success") + } + if tmpl := bsonnav.DGet(kindedValue(t, w), "TextTemplate"); tmpl != nil { + t.Errorf("text template was created in a hidden slot: %v", tmpl) + } +} + +func TestSetPluggableProperty_ExpressionKindWritesExpression(t *testing.T) { + w := kindedPluggableWidget("visibleExpr", "Expression", "", false) + if err := setPluggableWidgetPropertyMut(w, "visibleExpr", "$currentObject/Active"); err != nil { + t.Fatalf("set: %v", err) + } + val := kindedValue(t, w) + if got := bsonnav.DGetString(val, "Expression"); got != "$currentObject/Active" { + t.Errorf("Expression = %q", got) + } + if pv := bsonnav.DGetString(val, "PrimitiveValue"); pv != "" { + t.Errorf("value written to PrimitiveValue (%q) for an Expression-kind property", pv) + } +} + +// Structured kinds cannot be expressed as a plain SET value. Before, they took +// a string in PrimitiveValue and reported success. +func TestSetPluggableProperty_StructuredKindsAreRefused(t *testing.T) { + for _, kind := range []string{"Image", "Icon", "Action", "DataSource", "Attribute", "Widgets"} { + w := kindedPluggableWidget("prop", kind, "", false) + err := setPluggableWidgetPropertyMut(w, "prop", "x") + if err == nil { + t.Errorf("%s: expected a refusal, got success", kind) + continue + } + if !strings.Contains(err.Error(), kind) { + t.Errorf("%s: error should name the kind, got: %v", kind, err) + } + if pv := bsonnav.DGetString(kindedValue(t, w), "PrimitiveValue"); pv != "" { + t.Errorf("%s: wrote PrimitiveValue %q despite refusing", kind, pv) + } + } +} + +// Primitives keep landing in PrimitiveValue. +func TestSetPluggableProperty_PrimitiveKindUnchanged(t *testing.T) { + w := kindedPluggableWidget("widthUnit", "Enumeration", "", false) + if err := setPluggableWidgetPropertyMut(w, "widthUnit", "pixels"); err != nil { + t.Fatalf("set: %v", err) + } + if pv := bsonnav.DGetString(kindedValue(t, w), "PrimitiveValue"); pv != "pixels" { + t.Errorf("PrimitiveValue = %q, want pixels", pv) + } +} + +// A bracketed value arrives as []string and %v fused its tokens into one +// string written as success (#750). No pluggable property takes a list. +func TestSetPluggableProperty_ListValueIsRefused(t *testing.T) { + w := kindedPluggableWidget("visibleExpr", "Expression", "", false) + if err := setPluggableWidgetPropertyMut(w, "visibleExpr", []string{"if", "$x", "then"}); err == nil { + t.Fatal("expected a list value to be refused") + } +} From 22d67b14cc4028e6f4c1a6e2ad8a109337f7c8e9 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 26 Sep 2026 09:37:23 +0000 Subject: [PATCH 03/15] fix(widgets): carry an action's from the .mpk (#1200) A pluggable widget built from its .mpk wrote every action property's ValueType with an empty ActionVariables list, so the stored widget Type disagreed with its package and mx check reported CE0463 on every page carrying it, even with no action configured (Signature 2.1.0, Calendar 2.6.0 on Mendix 11.12.2). The .mpk parser now reads (top-level and nested object-list properties), and the generator writes them as CustomWidgets$WidgetActionVariable entries (Caption, Key, Type), the shape Studio Pro stores in the embedded Combobox template. Reconcile brings an embedded template's list in line with the installed package, rewriting it only when it disagrees so an agreeing template keeps its entries and $IDs. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01H79ZLyf5E4LuV48TmbDdq9 --- .../skills/fix-issue/findings/modelsdk.jsonl | 1 + .../bug-tests/1200-mpk-action-variables.mdl | 24 +++ modelsdk/widgets/actionvariables_test.go | 187 ++++++++++++++++++ modelsdk/widgets/augment.go | 47 ++++- modelsdk/widgets/mpk/actionvariables_test.go | 103 ++++++++++ modelsdk/widgets/mpk/mpk.go | 40 +++- 6 files changed, 400 insertions(+), 2 deletions(-) create mode 100644 mdl-examples/bug-tests/1200-mpk-action-variables.mdl create mode 100644 modelsdk/widgets/actionvariables_test.go create mode 100644 modelsdk/widgets/mpk/actionvariables_test.go diff --git a/.claude/skills/fix-issue/findings/modelsdk.jsonl b/.claude/skills/fix-issue/findings/modelsdk.jsonl index 62b23166c..6d2e812d0 100644 --- a/.claude/skills/fix-issue/findings/modelsdk.jsonl +++ b/.claude/skills/fix-issue/findings/modelsdk.jsonl @@ -22,3 +22,4 @@ {"area": "modelsdk/canon", "date": "2026-09-23", "symptom": "The storage-GUID write guard (`canon.StorageGUIDChanges`) stopped refusing the MOVE ENTITY data loss it had exposed (ako/mxcli#503). With MoveEntity's carries removed, moving an association's TO side re-minted the in-place converted cross-association's GUID and the write went through silently, where the issue records a refusal.", "cause": "`sameMember` (added in 86927852 to stop the guard refusing transplant mis-pairings) required an equal `$Type` as well as an equal `Name`. MoveEntity converts `DomainModels$Association` to `DomainModels$CrossAssociation` IN PLACE, keeping `$ID` and `Name`, so the type clause made the guard skip the pair. The type clause excluded nothing the transplant can produce: `pairDoc` stops at a `$Type` mismatch (TestTransplantIgnoresMismatchedTypes).", "file": "`modelsdk/canon/storageguid.go` (`sameMember`, the note above `GUIDChange`)", "insight": "Before adding a clause to an identity test that sits on an approximate pairing, ask what error of THAT pairing the clause excludes. The transplant only mis-pairs same-type, different-name elements, so `$Type` excluded none of its errors. Its only effect was to exclude the one writer that keeps an `$ID` across a type change deliberately. Rule now: Name when both sides have one; `$Type` only when neither does; a pair with a name on one side only is not a match. How the gap was found: stub the three MoveEntity carries on main and run TestIssue503. The child-side case returned no error where the issue quotes a refusal. That mismatch between the recorded refusal and the observed silence was the tell. A guard's quiet is not evidence of a clean write, so a guard's comment must list every hole it leaves; this one listed only renames. Controls: (1) the new canon test fails on the old `sameMember` with `got 0 change(s)`; (2) with the MoveEntity carries stubbed the child-side move is refused again with the issue's exact message, and the parent-side move still goes through, because the moved element changes unit and pairs with nothing (a documented hole); (3) the 86927852 false positive does not return: `marketplace install --file mx-modules/BusinessEvents_3.12.0.mpk` into a copy of testdata/expr-checker, then `create or modify persistent entity BusinessEvents.PublishedBusinessEvent (EventId: long)` is accepted, while a build with an `$ID`-only rule refuses it (EventId paired with a removed attribute). The existing table case `DifferentType_NotAChange` pinned the wrong decision with the justification 'nothing authors this today', which was false the day it was written. Grep for the writers (`SetID(x.ID())` next to `New()`) before claiming nothing authors a shape.", "refs": ["ako/mxcli#503", "mendixlabs/mxcli#1119"]} {"area":"modelsdk/codec","date":"2026-09-25","symptom":"A compound design property (Atlas `Spacing` → `margin-bottom`, or a multiSelect toggle group) writes its `Forms$CompoundDesignPropertyValue.Properties` list with BSON array marker 3 where Studio Pro writes 2. `check`, `exec` and `mx check` all pass; a describe → exec round trip of FeedbackModule.ShareFeedback (Feedback v4.0.2, 11.13.0) turned every nested marker-2 list into 3","cause":"The codec picks a PartList's marker from the CHILD element's `$Type` only (`partListMarker` → `lookupListMarker`). The nested list and the enclosing `Forms$Appearance.DesignProperties` list (marker 3) both hold `Forms$DesignPropertyValue`, so no `RegisterListMarker` on the child type could tell them apart and both fell to the default 3","file":"`modelsdk/codec/defaults.go` (`RegisterPropertyListMarker`), `modelsdk/codec/encoder.go` (`propertyListMarker`), `mdl/backend/modelsdk/widget_write.go` (init)","insight":"When one child `$Type` sits in two lists with different markers, the marker belongs to the owner+key, not the child: `RegisterPropertyListMarker(owner, key, m)` is consulted first, for an empty list too and in the selective-rebuild path. Establish the marker by counting Studio Pro-authored BSON before changing anything: walking every mxunit gave Compound.Properties 373/373 marker 2 and Appearance.DesignProperties 1821/1821 marker 3 across pages, layouts, building blocks and page templates. Count per (owner $Type, key) — a flat grep of `Properties [marker=2]` in ndsl also matches unrelated lists. Pages, snippets and layouts share `newAppearance`, so one registration covers all. Test `TestAppearanceCompoundDesignPropertyMarkers`; bug-test `mdl-examples/bug-tests/compound-design-property-marker.mdl`. Same class, not fixed: the selective-rebuild branch of `encodeEntry` still hard-codes 3 for lists with no owner registration, ignoring a child-type `RegisterListMarker`","refs":["#668"]} {"area":"modelsdk/mpr","date":"2026-09-25","symptom":"`alter page FeedbackModule.ShareFeedback_Logo { insert after textBox1 { image zzImg (ImageType: imageUrl, ImageUrl: '{1}', ImageUrlParams: [{1} = ImageB64]) } }` (data view over a nanoflow the project lacks, 11.13.0) reported \"Altered page\"; `mxcli docker check` then could not LOAD the project: ArgumentNullException setting 'Attribute' of an Attribute in a Page","cause":"The bare-AttributeRef refusal (#678) lived in encodePage/encodeSnippet only. ALTER PAGE patches the stored BSON in pagemutator and saves via UpdateRawUnit, never passing the encoder; with no entity in scope the pluggable-widget template-parameter builder (widgetobj) writes the name as given, so DomainModels$AttributeRef{Attribute:\"ImageB64\"} reached disk","file":"modelsdk/canon/attributeref.go; modelsdk/mpr/writer_core.go (updateUnit, insertUnit)","insight":"A guard placed in one encoder covers one write path; the page family has at least four (encodePage/Snippet, pagemutator Save, widget sync apply, layout/template raw writes). Put an unloadable-shape refusal at the writer beside DuplicateElementIDError, as that one already argued. Measured before refusing stored refs too: 73 of 73 AttributeRefs across all 374 units of a stock 11.13 project are qualified (71 page, 1 snippet, 1 page template, none elsewhere) — a stored bare one cannot have come from Studio Pro, so refusing ALL bare refs (not only new ones) blocks nothing legitimate. The textbox path does NOT reproduce it: attributeRefToGen nulls a bare name (a silent binding drop instead); the pluggable/column template builders are the ones that write it verbatim. The test goes through the real mutator + writer on the expr-checker fixture (InsertColumns with a bare CaptionParams ref).","refs":["#678"]} +{"area": "modelsdk/widgets", "date": "2026-09-26", "symptom": "CE0463 \"The definition of this widget has changed\" on every page carrying a pluggable widget built from its .mpk whose action properties declare `` (Signature 2.1.0, Calendar 2.6.0 on 11.12.2) — even with no action configured. `mx update-widgets` clears it", "cause": "The .mpk parser had no field for `` (modelsdk/widgets/mpk/mpk.go xmlProperty/PropertyDef), and createDefaultValueType hardcoded `ActionVariables: [2]`; reconcileValueTypesFromMPK never touched the list. The typed gen class (CustomWidgets$WidgetActionVariable) existed but the map-based template pipeline never fed it", "file": "modelsdk/widgets/mpk/mpk.go (ActionVariable, toActionVariables), modelsdk/widgets/augment.go (buildActionVariablesArray, actionVariablesMatch, createDefaultValueType, reconcileValueTypesFromMPK); tests actionvariables_test.go in both packages; example mdl-examples/bug-tests/1200-mpk-action-variables.mdl", "insight": "Third instance of the same shape after #716 (onChange) and #956 (defaultType): a widget.xml attribute/element that is part of the DEFINITION, never parsed, written as its empty default. Cheapest audit: diff every key Studio Pro stores on a WidgetValueType in the embedded templates against what createDefaultValueType derives from the .mpk — the embedded combobox.json already held the correct ActionVariables entry, i.e. the oracle was in the repo. `sdk/widgets/augment.go` has no importers; the live BSON path is modelsdk/widgets. When reconciling a list from the .mpk, rewrite only on disagreement, or an agreeing template's entry $IDs churn — the Combobox augment test is the no-change control (it fails if the rewrite is unconditional).", "refs": ["mendixlabs/mxcli#1200", "mendixlabs/mxcli#956", "#716"], "ce": ["CE0463"]} diff --git a/mdl-examples/bug-tests/1200-mpk-action-variables.mdl b/mdl-examples/bug-tests/1200-mpk-action-variables.mdl new file mode 100644 index 000000000..f218f9e27 --- /dev/null +++ b/mdl-examples/bug-tests/1200-mpk-action-variables.mdl @@ -0,0 +1,24 @@ +-- #1200: a pluggable widget built from its .mpk (no embedded template) lost the +-- `` its action properties declare. The action's ValueType +-- was written with an empty ActionVariables list, so the stored widget Type +-- disagreed with the package and `mx check` reported CE0463 — even with no +-- action configured, because the variables belong to the widget DEFINITION. +-- +-- Reported on Mendix 11.12.2 with Signature 2.1.0 and Calendar 2.6.0. This +-- script needs Signature 2.1.0 in the project's widgets/ folder; mxcli now +-- writes each declared variable as a CustomWidgets$WidgetActionVariable +-- (Key, Type, Caption), the shape Studio Pro stores. +-- +-- Expected: `mxcli docker check --no-update-widgets` reports no CE0463 on +-- Sig.Sign_Edit. (`mx update-widgets` would hide the defect by repairing it.) + +create module Sig; +create persistent entity Sig.Request ( SignatureData: string(unlimited) ); + +create or replace page Sig.Sign_Edit (Title: 'Sign', Layout: Atlas_Core.Atlas_Default, + Params: { $Request: Sig.Request }) +{ + DATAVIEW dv (DataSource: $Request) { + pluggablewidget 'com.mendix.widget.web.signature.Signature' sig1 () + } +} diff --git a/modelsdk/widgets/actionvariables_test.go b/modelsdk/widgets/actionvariables_test.go new file mode 100644 index 000000000..7074131f1 --- /dev/null +++ b/modelsdk/widgets/actionvariables_test.go @@ -0,0 +1,187 @@ +// SPDX-License-Identifier: Apache-2.0 + +package widgets + +import ( + "encoding/json" + "os" + "strings" + "testing" + + "github.com/mendixlabs/mxcli/modelsdk/widgets/mpk" +) + +// actionVariablesOf returns a ValueType's ActionVariables entries, without the +// leading list marker, failing when the marker is not the 2 Studio Pro writes. +func actionVariablesOf(t *testing.T, vt map[string]any) []map[string]any { + t.Helper() + arr, ok := vt["ActionVariables"].([]any) + if !ok || len(arr) == 0 || arr[0] != float64(2) { + t.Fatalf("ActionVariables = %#v, want a list led by marker 2", vt["ActionVariables"]) + } + var out []map[string]any + for _, e := range arr[1:] { + out = append(out, e.(map[string]any)) + } + return out +} + +// mendixlabs/mxcli#1200: an action's declared variables were written as an +// empty list, so the widget Type disagreed with its package and mx check +// reported CE0463 — even with no action configured. The shape is the one in +// Studio Pro's own Combobox template (templates/mendix-11.6/combobox.json): +// CustomWidgets$WidgetActionVariable with Caption, Key and Type. +func TestValueTypeCarriesActionVariablesFromTheWidgetXML(t *testing.T) { + p := mpk.PropertyDef{ + Key: "onChangeFilterInputEvent", + Type: "action", + ActionVariables: []mpk.ActionVariable{ + {Key: "filterInput", Type: "String", Caption: "Filter Input"}, + }, + } + got := actionVariablesOf(t, createDefaultValueType("vt-1", "Action", p)) + if len(got) != 1 { + t.Fatalf("ActionVariables entries = %d, want 1: %#v", len(got), got) + } + e := got[0] + for k, want := range map[string]any{ + "$Type": "CustomWidgets$WidgetActionVariable", + "Key": "filterInput", + "Type": "String", + "Caption": "Filter Input", + } { + if e[k] != want { + t.Errorf("entry %s = %v, want %v", k, e[k], want) + } + } + if id, _ := e["$ID"].(string); id == "" { + t.Error("entry has no $ID") + } + + // A property declaring none keeps the empty list. + if n := len(actionVariablesOf(t, createDefaultValueType("vt-2", "Action", mpk.PropertyDef{Key: "a", Type: "action"}))); n != 0 { + t.Errorf("no variables declared, got %d entries", n) + } +} + +// A widget WITH an embedded template goes through reconcile rather than +// generation. A stale empty list there — or one from an older widget version — +// must be brought in line with the installed package. +func TestReconcileFillsActionVariables(t *testing.T) { + tmpl := &WidgetTemplate{ + Type: map[string]any{"ObjectType": map[string]any{"PropertyTypes": []any{float64(2), + map[string]any{"$Type": "CustomWidgets$WidgetPropertyType", "$ID": "pt1", "PropertyKey": "onSign", + "ValueType": map[string]any{"$Type": "CustomWidgets$WidgetValueType", "Type": "Action", + "ActionVariables": []any{float64(2)}}}, + }}}, + } + byKey := map[string]mpk.PropertyDef{"onSign": {Key: "onSign", Type: "action", + ActionVariables: []mpk.ActionVariable{{Key: "signature", Type: "String", Caption: "Signature"}}}} + reconcileValueTypesFromMPK(tmpl, byKey) + + vt := tmpl.Type["ObjectType"].(map[string]any)["PropertyTypes"].([]any)[1].(map[string]any)["ValueType"].(map[string]any) + got := actionVariablesOf(t, vt) + if len(got) != 1 || got[0]["Key"] != "signature" { + t.Errorf("reconciled ActionVariables = %#v, want the package's one variable", got) + } +} + +// End to end on the real Combobox package, generated WITHOUT its embedded +// template — the path a widget like Signature or Calendar always takes. +func TestGenerateFromMPK_CarriesComboboxActionVariable(t *testing.T) { + const path = "../../testdata/expr-checker/widgets/com.mendix.widget.web.Combobox.mpk" + if _, err := os.Stat(path); err != nil { + t.Skipf("fixture not present: %v", err) + } + def, err := mpk.ParseMPKForWidget(path, "com.mendix.widget.web.combobox.Combobox") + if err != nil || def == nil { + t.Fatalf("parse: %v", err) + } + tmpl := GenerateFromMPK(def) + var found map[string]any + var walk func(any) + walk = func(n any) { + switch v := n.(type) { + case map[string]any: + if v["PropertyKey"] == "onChangeFilterInputEvent" { + found, _ = v["ValueType"].(map[string]any) + } + for _, x := range v { + walk(x) + } + case []any: + for _, x := range v { + walk(x) + } + } + } + walk(tmpl.Type) + if found == nil { + t.Fatal("onChangeFilterInputEvent not generated") + } + got := actionVariablesOf(t, found) + if len(got) != 1 || got[0]["Key"] != "filterInput" || got[0]["Type"] != "String" || got[0]["Caption"] != "Filter Input" { + t.Errorf("generated ActionVariables = %#v, want [{filterInput String \"Filter Input\"}]", got) + } +} + +// Control: Studio Pro's own Combobox template already carries the package's +// action variables. Augmenting it with the matching .mpk must leave every +// ActionVariables list exactly as it was — same entries, same $IDs — or the fix +// would churn a template that was never wrong. +func TestAugment_AgreeingTemplateKeepsItsActionVariables(t *testing.T) { + const path = "../../testdata/expr-checker/widgets/com.mendix.widget.web.Combobox.mpk" + if _, err := os.Stat(path); err != nil { + t.Skipf("fixture not present: %v", err) + } + const id = "com.mendix.widget.web.combobox.Combobox" + def, err := mpk.ParseMPKForWidget(path, id) + if err != nil || def == nil { + t.Fatalf("parse: %v", err) + } + cached, err := GetTemplate(id) + if err != nil || cached == nil { + t.Fatalf("embedded template: %v", err) + } + tmpl, err := deepCloneTemplate(cached) + if err != nil { + t.Fatal(err) + } + collect := func(tm *WidgetTemplate) map[string]string { + out := map[string]string{} + var walk func(any) + walk = func(n any) { + switch v := n.(type) { + case map[string]any: + if key, _ := v["PropertyKey"].(string); key != "" { + if vt, ok := v["ValueType"].(map[string]any); ok { + b, _ := json.Marshal(vt["ActionVariables"]) + out[key] = string(b) + } + } + for _, x := range v { + walk(x) + } + case []any: + for _, x := range v { + walk(x) + } + } + } + walk(tm.Type) + return out + } + before := collect(tmpl) + if !strings.Contains(before["onChangeFilterInputEvent"], "filterInput") { + t.Fatalf("precondition: embedded template should carry filterInput, has %s", before["onChangeFilterInputEvent"]) + } + if err := AugmentTemplate(tmpl, def); err != nil { + t.Fatalf("augment: %v", err) + } + after := collect(tmpl) + for key, b := range before { + if after[key] != b { + t.Errorf("%s ActionVariables changed:\n before %s\n after %s", key, b, after[key]) + } + } +} diff --git a/modelsdk/widgets/augment.go b/modelsdk/widgets/augment.go index 3103f6c6a..c3017c008 100644 --- a/modelsdk/widgets/augment.go +++ b/modelsdk/widgets/augment.go @@ -429,6 +429,14 @@ func reconcileValueTypesFromMPK(tmpl *WidgetTemplate, byKey map[string]mpk.Prope if len(pd.Translations) > 0 { vt["Translations"] = buildTranslationsArray(pd.Translations) } + // The package's action variables. A template from an older + // widget version (or one cloned without them) otherwise keeps + // a list that disagrees with the installed definition — CE0463 + // (#1200). Rewritten only when it differs, so a template that + // already agrees keeps its entries as they are. + if !actionVariablesMatch(vt["ActionVariables"], pd.ActionVariables) { + vt["ActionVariables"] = buildActionVariablesArray(pd.ActionVariables) + } } } } @@ -533,6 +541,43 @@ func buildTranslationsArray(trans []mpk.Translation) []any { return arr } +// buildActionVariablesArray builds a ValueType.ActionVariables list (leading +// Mendix array marker 2 followed by CustomWidgets$WidgetActionVariable entries) +// from a .mpk action property's declared — the shape Studio +// Pro stores (templates/mendix-11.6/combobox.json, onChangeFilterInputEvent). +// Caption is a plain string, not a Texts$Text. Placeholder $IDs are remapped by +// the loader's ID phase. +func buildActionVariablesArray(vars []mpk.ActionVariable) []any { + arr := []any{float64(2)} + for _, v := range vars { + arr = append(arr, map[string]any{ + "$ID": placeholderID(), + "$Type": "CustomWidgets$WidgetActionVariable", + "Caption": v.Caption, + "Key": v.Key, + "Type": v.Type, + }) + } + return arr +} + +// actionVariablesMatch reports whether a stored ActionVariables list already +// declares exactly vars, in order — so reconcile leaves a template that agrees +// with its package byte-for-byte instead of re-minting its entries' $IDs. +func actionVariablesMatch(stored any, vars []mpk.ActionVariable) bool { + arr, ok := stored.([]any) + if !ok || len(arr) == 0 || len(arr)-1 != len(vars) { + return false + } + for i, v := range vars { + e, ok := arr[i+1].(map[string]any) + if !ok || e["Key"] != v.Key || e["Type"] != v.Type || e["Caption"] != v.Caption { + return false + } + } + return true +} + // buildAllowedTypesArray builds a ValueType.AllowedTypes list (leading Mendix array // marker 1 followed by the allowed Mendix type names) from a .mpk property's declared // attributeTypes. Mirrors the construction in createDefaultValueType. @@ -778,7 +823,7 @@ func createDefaultValueType(vtID string, bsonType string, p mpk.PropertyDef) map vt := map[string]any{ "$ID": vtID, "$Type": "CustomWidgets$WidgetValueType", - "ActionVariables": []any{float64(2)}, + "ActionVariables": buildActionVariablesArray(p.ActionVariables), "AllowNonPersistableEntities": false, "AllowedTypes": allowedTypes, "AssociationTypes": []any{float64(1)}, diff --git a/modelsdk/widgets/mpk/actionvariables_test.go b/modelsdk/widgets/mpk/actionvariables_test.go new file mode 100644 index 000000000..950bb6017 --- /dev/null +++ b/modelsdk/widgets/mpk/actionvariables_test.go @@ -0,0 +1,103 @@ +// SPDX-License-Identifier: Apache-2.0 + +package mpk + +import ( + "os" + "testing" +) + +// An action property may declare the variables its action receives +// (``). They are part of the widget DEFINITION — Studio Pro +// stores them on the ValueType even when no action is configured — so dropping +// them at parse time left nothing for the generator to write, and every page +// carrying such a widget was CE0463 (mendixlabs/mxcli#1200: Signature 2.1.0, +// Calendar 2.6.0). +func TestParse_ReadsActionVariables(t *testing.T) { + const widgetXML = ` + + W + + + + On sign + + + + + + + Plain + + + Items + + + + On pick + + + + + + + + + +` + + def := parseWidgetXML(t, widgetXML) + byKey := map[string]PropertyDef{} + for _, p := range def.Properties { + byKey[p.Key] = p + } + + got := byKey["onSign"].ActionVariables + want := []ActionVariable{ + {Key: "signature", Type: "String", Caption: "Signature"}, + {Key: "signedAt", Type: "DateTime", Caption: "Signed at"}, + } + if len(got) != len(want) { + t.Fatalf("onSign ActionVariables = %+v, want %+v", got, want) + } + for i := range want { + if got[i] != want[i] { + t.Errorf("onSign ActionVariables[%d] = %+v, want %+v", i, got[i], want[i]) + } + } + if av := byKey["plainAction"].ActionVariables; len(av) != 0 { + t.Errorf("plainAction declares none, parsed %+v", av) + } + + var nested []ActionVariable + for _, c := range byKey["items"].Children { + if c.Key == "onPick" { + nested = c.ActionVariables + } + } + if len(nested) != 1 || nested[0] != (ActionVariable{Key: "index", Type: "Integer", Caption: "Index"}) { + t.Errorf("nested onPick ActionVariables = %+v, want one {index Integer Index}", nested) + } +} + +// The real package: Combobox 2.5.0's onChangeFilterInputEvent declares one. +func TestParse_ComboboxFixtureActionVariable(t *testing.T) { + const path = "../../../testdata/expr-checker/widgets/com.mendix.widget.web.Combobox.mpk" + if _, err := os.Stat(path); err != nil { + t.Skipf("fixture not present: %v", err) + } + def, err := ParseMPKForWidget(path, "com.mendix.widget.web.combobox.Combobox") + if err != nil || def == nil { + t.Fatalf("parse: %v", err) + } + for _, p := range def.Properties { + if p.Key == "onChangeFilterInputEvent" { + want := ActionVariable{Key: "filterInput", Type: "String", Caption: "Filter Input"} + if len(p.ActionVariables) != 1 || p.ActionVariables[0] != want { + t.Errorf("onChangeFilterInputEvent ActionVariables = %+v, want [%+v]", p.ActionVariables, want) + } + return + } + } + t.Fatal("onChangeFilterInputEvent not parsed") +} diff --git a/modelsdk/widgets/mpk/mpk.go b/modelsdk/widgets/mpk/mpk.go index f9d68d242..f6327431a 100644 --- a/modelsdk/widgets/mpk/mpk.go +++ b/modelsdk/widgets/mpk/mpk.go @@ -47,7 +47,20 @@ type PropertyDef struct { AllowedTypes []string // for attribute properties: Mendix type names ("String", "Decimal", etc.) EnumValues []EnumValue // for enumeration properties: the declared options (key + caption) Translations []Translation // widget-shipped caption/template translations () - Children []PropertyDef // nested properties for object-type properties + // ActionVariables are the variables an action property passes to the flow + // it calls (``). Part of the widget DEFINITION — Studio Pro + // stores them on the ValueType whether or not an action is configured — so + // writing an empty list where the package declares some is CE0463 + // (mendixlabs/mxcli#1200, Signature 2.1.0, Calendar 2.6.0). + ActionVariables []ActionVariable + Children []PropertyDef // nested properties for object-type properties +} + +// ActionVariable is one `` of an action property. +type ActionVariable struct { + Key string + Type string // Mendix type name as the XML spells it: "String", "DateTime", … + Caption string } // EnumValue is one option of an enumeration-typed widget property. @@ -223,10 +236,19 @@ type xmlProperty struct { SelectionTypes []xmlSelectionType `xml:"selectionTypes>selectionType"` ReturnType xmlReturnType `xml:"returnType"` Translations []xmlTranslation `xml:"translations>translation"` + // ActionVariables of an action property (). + ActionVariables []xmlActionVariable `xml:"actionVariables>actionVariable"` // Nested properties for object type NestedProps []xmlPropGroup `xml:"properties>propertyGroup"` } +// xmlActionVariable represents . +type xmlActionVariable struct { + Key string `xml:"key,attr"` + Caption string `xml:"caption,attr"` + Type string `xml:"type,attr"` +} + // xmlSelectionType represents on a selection property. type xmlSelectionType struct { Name string `xml:"name,attr"` @@ -415,6 +437,7 @@ func walkPropertyGroup(pg xmlPropGroup, parentCategory string, def *WidgetDefini AllowedTypes: allowedTypes, EnumValues: enumValues, Translations: toTranslations(p.Translations), + ActionVariables: toActionVariables(p.ActionVariables), } // Parse nested properties for object-type properties @@ -489,6 +512,7 @@ func collectNestedProperties(pg xmlPropGroup, parent *PropertyDef, parentCategor AllowedTypes: allowedTypes, EnumValues: enumValues, Translations: toTranslations(p.Translations), + ActionVariables: toActionVariables(p.ActionVariables), } // Nested object-type properties can themselves contain object lists. if p.Type == "object" && len(p.NestedProps) > 0 { @@ -535,6 +559,20 @@ func toTranslations(xts []xmlTranslation) []Translation { return out } +func toActionVariables(xavs []xmlActionVariable) []ActionVariable { + if len(xavs) == 0 { + return nil + } + out := make([]ActionVariable, 0, len(xavs)) + for _, x := range xavs { + if x.Key == "" { + continue + } + out = append(out, ActionVariable{Key: x.Key, Type: x.Type, Caption: x.Caption}) + } + return out +} + // FindMPK looks in the project's widgets/ directory for an .mpk matching the widgetID. // Returns the path to the .mpk file, or empty string if not found. func FindMPK(projectDir string, widgetID string) (string, error) { From 9e9d25691c97af9cbc928e77a702383306ff0a98 Mon Sep 17 00:00:00 2001 From: Ako Date: Sat, 26 Sep 2026 13:00:07 +0000 Subject: [PATCH 04/15] docs: propose MDL beta syntax freeze and brownfield editing plan Critique of the whole MDL language against ADR-0003 before the alpha to beta transition, with verified before/after examples, twelve consolidation rules, the changes that cannot be bridged by an alias, a two-mode (MDL-first / data-first) editing model with content-addressed ALTER MICROFLOW, and a phased implementation plan. Co-Authored-By: Claude Opus 5.5 --- .../PROPOSAL_mdl_beta_syntax_freeze.md | 1206 +++++++++++++++++ 1 file changed, 1206 insertions(+) create mode 100644 docs/11-proposals/PROPOSAL_mdl_beta_syntax_freeze.md diff --git a/docs/11-proposals/PROPOSAL_mdl_beta_syntax_freeze.md b/docs/11-proposals/PROPOSAL_mdl_beta_syntax_freeze.md new file mode 100644 index 000000000..04d13cdf2 --- /dev/null +++ b/docs/11-proposals/PROPOSAL_mdl_beta_syntax_freeze.md @@ -0,0 +1,1206 @@ +--- +title: MDL Language Critique and Beta Syntax Freeze +status: draft +date: 2026-09-26 +--- + +# Proposal: MDL Language Critique and Beta Syntax Freeze + +**Status:** Draft +**Date:** 2026-09-26 +**Related:** [ADR-0003](../13-decisions/0003-mdl-is-sql-shaped.md), [design-mdl-syntax skill](../../.claude/skills/design-mdl-syntax.md), [PROPOSAL_workflow_microflow_syntax_alignment.md](PROPOSAL_workflow_microflow_syntax_alignment.md) + +## Summary + +Once MDL leaves alpha, breaking syntax changes get expensive. This document reviews the whole language against its own stated principles (ADR-0003 and the `design-mdl-syntax` skill) and lists what should change before beta. + +**Verdict.** The foundation is sound: SQL-shaped verbs, `Module.Name` everywhere, `/** */` documentation, `from … to …` associations and `extends`. The language does not need redesigning. It needs **consolidating**, for two reasons: + +1. Each document type grew its own local conventions: how arguments are bound, how children nest, where metadata lives, how blocks close. Knowing one area of MDL does not teach you the next. +2. Almost every construct has picked up aliases. `delete behavior` alone has 16 surface spellings for 3 values. Every alias still in the grammar at beta becomes a permanent compatibility obligation. + +Most of the findings collapse into **twelve rules** (§3). Adopt the rules, have `describe` emit only the canonical form, and leave the old forms parsing with deprecation warnings. That fixes most of this cheaply. A short list of changes **cannot** be bridged by aliases because they change the meaning of text that already parses (§5). Those are the true must-do-before-beta items. + +The review also found **silent-loss bugs**: input that parses and is then dropped or mis-stored. They are listed first (§2) because they matter more than any syntax question. + +**Brownfield editing (§8).** Most agent work will be on large Studio Pro apps that have no MDL source. For that work, the decisive questions are whether MDL can *read, locate and patch* reliably, not whether it reads well as source. Measured on two real apps: +- Surgical `alter` preserves stored state. +- Round-tripping `describe` output through `create or modify` silently lost data in 7 of 12 cases. +- Microflows, the most-edited type, have no `alter` at all. + +§8 argues that MDL needs two first-class modes that share one syntax: +- **MDL-first:** declarative `create or replace` scripts as the source of truth. This is the efficient mode for new apps. +- **Data-first:** content-addressed `alter` patches that splice into the stored graph. This is the efficient mode for changes to existing Studio Pro documents. + +Drift detection chooses between them safely, and two round-trip laws make moving between them lossless. + +### Method + +Six parallel reviews covered: +- the domain model; +- microflows and nanoflows; +- pages, snippets, layouts and navigation; +- security, workflows and settings; +- integration and agents; +- cross-cutting grammar and lexer. + +Every "parses" or "rejected" claim was run through `mxcli check`. Round-trip claims come from `exec` followed by `describe` on a scratch copy of a real project. Grammar references are to `mdl/grammar/` at commit `5dc51ceb`. + +--- + +## 1. What is good and should stay + +- The statement shape `create [or modify] Module.Name (props) body`, plus `alter`/`drop`/`describe`/`list`. +- Qualified names everywhere, and no `use module`. +- `/** */` doc comments as documentation, and `--` / `/* */` as comments. `//` is rejected, which keeps the comment syntax clean. +- `from Child to Parent` on associations, `on delete cascade|restrict|set null`, `extends`, `rename … to`, `move … to`. +- `alter page … set … on widget`, `insert after`, `replace … with`. +- The widget form `widget name (props) { children }`. +- `if … then … elsif … else … end if`, `loop $x in $L begin … end loop`, `retrieve $x from … where [xpath]`. +- `$$ … $$` for raw foreign content. + +## 2. Bugs found during the review: fix regardless of syntax decisions + +| # | Symptom | Where | Consequence | +|---|---|---|---| +| 1 | *(Withdrawn. The review claimed `create or replace entity` deletes and recreates the entity. It does not: the visitor folds `REPLACE` into `MODIFY` for persistent entities precisely to keep the GUID (`visitor_entity.go:36`). Delete-and-recreate happens only for **view** entities (`cmd_entities.go:913`), which have no stored rows. See R1 for the naming question that remains.)* | | | +| 2 | `throw ` parses, but no visitor handles it | `MDLMicroflow.g4:280,724`; no Go reference | The statement disappears silently. `raise error` is the working form. | +| 3 | `float` and `currency` attribute types become `String` | `mdl/visitor/visitor_helpers.go:420` (falls through) | The wrong type is stored with no warning. `date` silently becomes `DateTime`. | +| 4 | The parenthesised association form `association X (from … to …, type: …, storage: …)` drops all options | `MDLDomainModel.g4:174-177`; the visitor reads only `AssociationOptions` | A `ReferenceSet` is stored as a `Reference`. | +| 5 | `$x = find($S, 'abc')` is either a string function or a list FIND, depending on whether `$x` was declared earlier | `setStatement` (optional `set`) vs `listOperationStatement` | The same text means two different things. | +| 6 | Unknown property keys are accepted and ignored (`Pathh:`, `Temprature:`, `Versoin:`) on the REST client, agent, model, published REST and business events | `visitor_rest.go:33-75`, `MDLAgent.g4:23-31` | Typos are silently discarded. Only the OData client refuses them (MDL-ODATA01). | +| 7 | Some statements parse but are always refused: `grant execute on workflow`, `case … else`, `declare … list of … = empty`, and widgets `text`/`statictext`/`legacydatagrid` | `MDLSecurity.g4:79-85`, `MDLMicroflow.g4:315`, `cmd_pages_builder_v3_widgets.go:728` | Grammar surface with no meaning. Remove it now while that costs nothing. | +| 8 | Describe output that does not re-parse or loses data: string defaults containing `'`; module-role, OData, published-REST and REST-client strings; agent MCP blocks missing a comma; REST header expressions re-emitted as literals; demo-user `password '***'`; user-role description; workflow captions and annotations emitted as `--` comments | `cmd_entities_describe.go:321`, `cmd_security.go:787,826,865`, `cmd_agenteditor_agents.go:179`, `cmd_rest_clients.go:180`, `cmd_workflows.go:263,832` | Describe does not round-trip. | +| 9 | Configuration describe prints `DatabasePassword = ''` | `cmd_settings.go:976` | Credential leak into PR diffs. | +| 10 | Image-collection describe emits `/tmp/mxcli-preview/...` file paths | `cmd_imagecollections.go:158` | Output is neither portable nor reviewable. | +| 11 | Items that are accepted and then dropped: enumeration-value doc comments, index names, and the declared name on `AutoOwner`-style attributes | `MDLDomainModel.g4:151,385` | The source says something the model does not store. | + +## 3. The twelve rules + +Each rule is stated so it can be added to the `design-mdl-syntax` skill as a checklist item. The findings each rule resolves follow it. + +**About the examples.** +- Every **Before** block was run through `mxcli check` at `5dc51ceb` and parses today. The `describe` output in R12 is real output from a scratch project. +- Every **After** block is *proposed* syntax and does not parse yet. Where §7 leaves a choice open, the After blocks use the recommended option: `or replace`, `Param = expr` for arguments, and `set ( Key: value )` for alter. + +### R1. One idempotent create, with one defined meaning + +Today `create or modify` and `create or replace` are **one operation with two names**: +- The AST folds the two into one flag for every document type. +- The entity visitor folds `REPLACE` into `MODIFY` explicitly, to keep the GUID. +- The only difference is view entities, where `replace` deletes and recreates. That is safe only because they store no rows. + +The semantics are also already declarative, not a merge: +- The statement is the **whole** definition. +- Members matched **by name** keep their stored `$ID`/GUID. +- Members the statement omits are dropped, and MDL087 reports them. + +That settles renames. A declarative definition cannot tell a rename from a drop plus an add, so under it a rename destroys the column. **Renames are therefore only expressible as `alter entity … rename attribute A to B` or `rename entity … to …`**, which carry the identity across. The same holds for any document whose members have identity. + +Rule: +- One keyword, with the semantics above written into the skill and the ADR: "full definition; name-keyed identity carry; omitted members dropped and reported; renames need `alter`/`rename`". +- The keyword's name is an open decision (§7): + - `or replace` describes the full-definition behaviour honestly and matches SQL's `CREATE OR REPLACE VIEW`. + - `or modify` is what `describe` emits today, but it reads like a merge. That is exactly the misreading that makes an omitted attribute a surprise. +- Whichever is chosen, the other becomes a deprecated alias. +- Make view entities follow the same identity-carrying rewrite instead of delete and recreate, so the keyword means one thing on every type. +- Keep `if not exists` as a separate, genuinely different operation (leave an existing element untouched). Extend it to every type, not just entities and associations. +- Move `create module role` into `createStatement` (`MDLSecurity.g4:17`), so it gets the same prefix as every other type. +- Every `describe` emits the chosen keyword. Today association, layout, user role, demo user, REST client, OData, agent and database connection emit a plain `create`, and navigation emits `create or replace`. + +**Before** — two keywords for one operation, and a rename that looks harmless: + +```mdl +create or modify persistent entity Shop.Order (Number: integer, Note: string(200)); +create or replace persistent entity Shop.Order (Number: integer, Note: string(200)); -- identical effect + +-- Editing the name in the definition is NOT a rename: Note is dropped (with its data), +-- and an empty Remarks column is added. MDL087 warns; nothing refuses. +create or modify persistent entity Shop.Order (Number: integer, Remarks: string(200)); +``` + +**After** — one keyword that says what it does, and renames are a separate statement: + +```mdl +create or replace persistent entity Shop.Order ( + Number: integer, + Note: string(200), +); + +alter entity Shop.Order rename attribute Note to Remarks; -- keeps the column and its data +create persistent entity if not exists Shop.Audit (At: datetime); -- leaves an existing entity alone +``` + +### R2. Three bracket kinds, three meanings + +| Bracket | Meaning | Used for | +|---|---|---| +| `( Key: value, … )` | the properties of *this* element | every document header and every child's properties | +| `{ … }` | declarative children | page widgets, service operations and resources, mapping trees, workflow activities, agent tools, menu items | +| `begin … end <keyword>` | imperative flow | microflow bodies, `if`, `loop`, `while`, error handlers | + +Every child inside `{ }` has the shape `<kind> <Name> ( Key: value, ) [ { grandchildren } ]`. A child always ends in `)` or `}`, so, as with page widgets today, children need no separator. + +What would change: +- REST client operations and agent attachments put properties inside `{}`. They become `operation GetUser ( Method: get, Path: '/u/{id}' )`. +- Image collections put children in `()`. They become `{ image Logo ( File: 'x.png' ) }`. +- The database connection is the only declarative document using `begin…end`, and it has no property list. It becomes `( Type: postgresql, ConnectionString: @M.Url, … ) { query Q ( Sql: $$…$$, Returns: M.E ) }`. +- Message definitions use `()` for member trees. They switch to `{}`, the same as mappings. +- Menus and navigation use `( item; item; )`. They switch to `{ }`, the same as widgets. +- Maps written as `Params: { $T: M.E }`, `DesignProperties: ['k': 'v']` or `ContentParams: [{1} = expr]` all become `( key: value, )`. +- `on error { … }` is the only brace block inside a microflow. It becomes `on error [without rollback] begin … end error;`. +- `while` makes both `begin` and `end while` optional. Make both required, the same as `loop`. +- The `split type` body becomes mandatory. + +**Before** — properties in braces, children in parentheses, menu items separated by `;`, braces inside a microflow: + +```mdl +create rest client Shop.Api ( + BaseUrl: 'https://api.example.com', + authentication: none +) +{ + operation GetUser { + method: get, + path: '/users/{id}', + headers: ('Accept' = 'application/json'), + response: none + } +}; + +create image collection Shop.Icons ( + image Logo from file 'assets/logo.png' +); + +create or replace navigation Responsive + home page Shop.Home + menu ( + menu item 'Home' page Shop.Home; + menu 'Admin' ( + menu item 'Users' page Administration.Account_Overview; + ); + ); + +commit $Order on error without rollback { + log warning 'Save failed'; +}; +``` + +**After** — `()` properties, `{}` children, `begin … end` flow: + +```mdl +create consumed rest service Shop.Api ( + BaseUrl: 'https://api.example.com', + Authentication: none, +) { + operation GetUser ( + Method: get, + Path: '/users/{id}', + Headers: ( 'Accept': 'application/json' ), + Response: none, + ) +}; + +create image collection Shop.Icons { + image Logo ( File: 'assets/logo.png' ) +}; + +create or replace navigation Responsive ( + HomePage: Shop.Home, +) { + menu item 'Home' ( OnClick: show page Shop.Home ) + menu 'Admin' { + menu item 'Users' ( OnClick: show page Administration.Account_Overview ) + } +}; + +commit $Order on error without rollback begin + log warning 'Save failed'; +end error; +``` + +### R3. `:` vs `=` + +- **`:` sets a model property.** This covers property lists, including `alter … set ( Key: value )`. +- **`=` binds a runtime value.** This covers call arguments, `change $o (Attr = v)`, `set $x = …`, text-template parameters and mapping sides. +- Today the same key uses `:` in `create` and `=` in `alter`, for example page `set Caption = 'x'`, `alter odata … set Version = '1'` and `alter settings` `k = v`. +- Make `alter <type> X set ( Key: value, … )` accept exactly the keys and value grammar of `create`. That way, turning a describe fragment into an `alter` is a copy and paste. +- Also: + - Remove the optional colons, one way or the other. Clauses outside a property list take no colon (`type reference`, not `type: reference`). An attribute definition is always `Name: Type`, including `modify attribute A: T`. + - Remove `set allow_create_change_locally = true`, the only snake_case `=` alter action. + +**Before** — `create` uses `:`, but the same keys take `=` in `alter`, in settings and inside some parenthesised lists: + +```mdl +alter page Shop.Product_Overview { + set caption = 'Name' on dgProducts.Name; + set (caption = 'Cost', alignment = right) on dgProducts.Price +}; + +alter settings model AfterStartupMicroflow = 'Shop.ASU_Startup'; + +alter module Ops add jar dependency (group = 'org.postgresql', artifact = 'postgresql', version = '42.7.4', included = true); +``` + +**After** — a model property is always `Key: value` in a `( … )` list, whether it is set by `create` or `alter`: + +```mdl +alter page Shop.Product_Overview { + set ( Caption: 'Name' ) on dgProducts.Name; + set ( Caption: 'Cost', Alignment: right ) on dgProducts.Price; +}; + +alter settings runtime ( AfterStartupMicroflow: Shop.ASU_Startup ); + +alter module Ops add jar dependency ( + Group: 'org.postgresql', + Artifact: 'postgresql', + Version: '42.7.4', + Included: true, +); +``` + +### R4. One argument-binding form everywhere + +The same idea is spelled five ways today: +- `call microflow M.F($p = e)`, also accepting `p = e`; +- `show page M.P($P = e)` or `P: e`; +- page actions `(P: v)`; +- workflows `with (p = '…')`; +- `send rest request` accepts only `$p = e`. + +Canonical form, following R3: `Name = expression`, with no `$` on the parameter name. + +Text-template parameters get one form too: `with ({1} = e)`. It replaces `objects [e, …]` in `show message` and validation feedback, and the older `parameters [...]`. + +**Before** — five ways to pass a value to a parameter: + +```mdl +call microflow Shop.ACT_Approve($Order = $Order, Force = false); -- `$` optional +show page Shop.Order_Edit(Order: $Order); -- `:` here, `$Order =` also accepted +actionbutton btnApprove (Caption: 'Approve', Action: microflow Shop.ACT_Approve(Order: $currentObject)) +call microflow Shop.ACT_Process with (Order = '$WorkflowContext'); -- in a workflow: the expression is a string +show message 'Order {1} approved' type information objects [$Order/Number]; +log info node 'Shop' 'Order {1} approved' with ({1} = $Order/Number); +``` + +**After** — one form, whatever is being called and wherever the call appears: + +```mdl +call microflow Shop.ACT_Approve(Order = $Order, Force = false); +show page Shop.Order_Edit(Order = $Order); +actionbutton btnApprove (Caption: 'Approve', OnClick: call microflow Shop.ACT_Approve(Order = $currentObject)) +call microflow Shop.ACT_Process(Order = $WorkflowContext); -- in a workflow +show message 'Order {1} approved' type information with ({1} = $Order/Number); +log info node 'Shop' 'Order {1} approved' with ({1} = $Order/Number); +``` + +### R5. Expressions are bare, and XPath is in `[ ]` + +- XPath is always bracketed: `where [Active = true()]`. Today entity-access `where '…'` and workflow `targeting xpath '…'` are quoted strings. Those compose into runs of six quote characters (#750); navigation already moved to brackets for exactly that reason. +- Client and microflow expressions are always bare, never in a string and never in brackets. This affects: + - workflow `decision '$Ctx/Total > 1000'`, timers and due dates; + - page `Visible: [expr]` and `Editable: [expr]` (brackets used for a non-XPath expression, `MDLPage.g4:198-206`); + - page variable defaults `Boolean = 'true'`. +- Microflow `where` accepts both a bracketed and a bare expression (`MDLMicroflow.g4:425`). Only the bracketed form stays. +- There are four ways to refer to a constant: `@Module.Const`, legacy `$Const`, bare `Key: M.C`, and `CONSTANT 'M.C'` in settings. Choose one; the recommendation is `@Module.Const`, which is already the most common. + +**Before** — XPath inside string literals with doubled quotes, and expressions in strings or in brackets: + +```mdl +grant Shop.User on Shop.Order (read *, write *) where '[Status = ''Open'']'; + +user task ReviewOrder 'Review the order' + targeting xpath '[Status = ''Draft'']' + ... +decision '$WorkflowContext/Total > 1000' + ... + +textbox txtNote (Attribute: Note, Visible: [$currentObject/Status = 'Open']) +``` + +**After** — XPath in brackets, expressions bare, no quotes to double: + +```mdl +grant read *, write * on entity Shop.Order to Shop.User where [Status = 'Open']; + +user task ReviewOrder caption 'Review the order' + targeting users xpath [Status = 'Draft'] + ... +decision $WorkflowContext/Total > 1000 + ... + +textbox txtNote (Attribute: Note, Visible: $currentObject/Status = 'Open') +``` + +### R6. Verb inventory + +| Verb | Meaning | Changes | +|---|---|---| +| `list <plural>` | enumerate elements | Retire `show <plural>`, which today leads 72 of the syntax lines versus 15 for `list`. Add plurals for `list image collection` / `icon collection` / `message definition collection`. | +| `describe <type> Name` | one element | Remove `show entity|association|page X` (`MDLCatalog.g4:62-64`). | +| `show <state>` | non-element state (status, version, access, callers, impact) | Stop `showOrList` producing forms like `list version` and `list catalog status`. | +| `create` / `alter` / `drop` | as in SQL | `alter` children use `add`/`drop`. Replace `remove` (user role), `modify`/`add or modify` (settings), `define fragment`, `update security` and `update widgets`. Add the missing `drop`s: database connection, validation rule, external entity. | + +Also: +- `column` survives as a synonym for `attribute` in four `alter entity` actions. Remove it. +- `rest call get …` breaks the `call <kind>` pattern. Use `call rest service …`, which is also Studio Pro's name for the activity, and fold `send rest request` into it where the semantics allow. + +**Before:** + +```mdl +show entities in Shop; +show entity Shop.Order; +alter user role AppAdmin remove module roles (Shop.Admin); +alter entity Shop.Order add column Note: string(200); +alter styling on page Shop.Home widget ctnHeader set 'Full width' = on; +$Html = rest call get 'https://example.com' header Accept = 'text/html' timeout 300 returns string; +``` + +**After:** + +```mdl +list entities in Shop; +describe entity Shop.Order; +alter user role AppAdmin drop module roles (Shop.Admin); +alter entity Shop.Order add attribute Note: string(200); +alter page Shop.Home { set ( DesignProperties: ( 'Full width': true ) ) on ctnHeader; }; +$Html = call rest service get 'https://example.com' ( + Headers: ( 'Accept': 'text/html' ), + Timeout: 300, +) returns string; +``` + +### R7. The language vs the session + +The line between MDL and session commands: +- **MDL** is what makes sense in a checked-in `.mdl` file applied to a model. +- **Session commands** are anything that needs a session or an environment: `connect`, `disconnect`, `use`, `set format = …`, `status`, `check`, `build`, `lint`, `debug`, `execute script`, `help`, `introspect`. + +Those session commands live in `utilityStatement` (`MDLSettings.g4:93-112`) today. Move them out as REPL meta-commands, and have `exec` refuse them inside scripts. + +Two concrete gains: +- `set` stops having three meanings. +- `helpStatement: IDENTIFIER …` stops swallowing typos. Today `craete entity …` reports its error at the `(`, not at the misspelt word. + +**Before** — this script passes the syntax check. The only error is on line 3, at column 35 (the `(`), not at the typo: + +```mdl +connect local 'app.mpr'; +set format = json; +craete persistent entity Shop.Note (Text: string(200)); +``` + +**After** — a `.mdl` file holds model statements only. The session is set up outside the script (`mxcli exec script.mdl -p app.mpr --json`, or at the REPL). The typo is reported where it is: `unknown statement 'craete' — did you mean 'create'?`. + +### R8. Words, not SCREAMING_SNAKE; one spelling per keyword + +- Page actions use snake-case tokens: `SHOW_PAGE`, `CLOSE_PAGE`, `SAVE_CHANGES`, `CALL_MICROFLOW`, `CREATE_OBJECT`, `DELETE_OBJECT`, `OPEN_LINK`, `SIGN_OUT`, `COMPLETE_TASK`. Replace them with the microflow words: `show page`, `close page`, `save changes`, `call microflow`, `sign out`. Menus use the same vocabulary. +- Several keywords have more than one spelling, and each needs a single form: + - `DELETE_AND_REFERENCES` has three lexer spellings. + - `REFERENCE_SET`, `DELETE_BEHAVIOR` and `ALLOW_CREATE_CHANGE_LOCALLY` take optional underscores. + - `ELSEIF` exists alongside `ELSIF`. + - `returns none` sits next to `returns nothing`. + - `NOT_NULL` sits next to `NOT NULL`. +- Lowercase keywords are canonical. `describe`, `fmt` and the `mxcli syntax` examples should all agree; the examples currently use uppercase 528 times and lowercase 100 times. Today `fmt` upper-cases property keys that happen to be keywords (`FOLDER:`, `Enabled: TRUE`). Property keys must be case-preserved identifiers, never keywords. +- The same "error message" concept has three keywords: `not null error '…'`, validation-rule `feedback '…'` and association `error_message '…'`. All become `error message '…'`. + +**Before:** + +```mdl +actionbutton btnNew (Caption: 'New', Action: create_object Shop.Order then show_page Shop.Order_Edit) +actionbutton btnSave (Caption: 'Save', Action: save_changes) +actionbutton btnOut (Caption: 'Sign out', Action: sign_out) + +create association Shop.Order_Customer from Shop.Order to Shop.Customer + type reference delete_behavior prevent error_message 'Customer still has orders'; +``` + +**After** — the same words a microflow uses, and one spelling for "error message": + +```mdl +actionbutton btnNew (Caption: 'New', OnClick: create object Shop.Order then show page Shop.Order_Edit) +actionbutton btnSave (Caption: 'Save', OnClick: save changes) +actionbutton btnOut (Caption: 'Sign out', OnClick: sign out) + +create association Shop.Order_Customer from Shop.Order to Shop.Customer + type reference + on delete restrict error message 'Customer still has orders'; +``` + +### R9. Where document metadata lives + +| Metadata | Canonical form | Remove | +|---|---|---| +| Documentation | `/** … */` doc comment only | the `comment '…'` clause on constant, association, JSON structure, image collection and workflow; `Documentation:`; `set comment` | +| Folder | `folder '…'` clause after the name | `Folder:` property on REST client, OData, published REST and page headers; `@folder`; double folder on snippets | +| Canvas layout | `@position`, `@anchor`, `@curve` annotations only | `set position` on alter stays as the alter form. Also: `@Position` vs `@position` casing, and the `Position: (x,y)` property | +| Export level, security flags | clauses (`export level api`) | the `@applyentityaccess` / `@excluded` annotations, which become clauses | + +**Before** — documentation as a clause, folder as a property on one document and a clause on another: + +```mdl +create constant Shop.ApiUrl type string default 'https://api.example.com' comment 'Base URL of the API' exposed to client; + +create rest client Shop.Api ( + BaseUrl: 'https://api.example.com', + Folder: 'Integration', + authentication: none +) { }; + +create microflow Shop.Helper () folder 'Integration' +begin +end; +``` + +**After** — documentation is always a doc comment, folder is always a clause, everything else is a property: + +```mdl +/** Base URL of the API */ +create constant Shop.ApiUrl folder 'Integration' ( + Type: string, + DefaultValue: 'https://api.example.com', + ExposedToClient: true, +); + +create consumed rest service Shop.Api folder 'Integration' ( + BaseUrl: 'https://api.example.com', + Authentication: none, +) { }; + +create microflow Shop.Helper () folder 'Integration' +begin +end; +``` + +In workflows, `comment 'x'` sets the **caption** today. That misled the draft alignment proposal, which assumed `comment` was the annotation. Rename it to `caption '…'`. + +Workflow activities should all be `<kind> [Name] … [caption '…']`, with the name always optional and derived one way. Today the name is positional-and-required for user tasks, positional-and-optional elsewhere, and `as name` for calls. + +### R10. Document type names follow Studio Pro + +| Today | Canonical | +|---|---| +| `rest client` | `consumed rest service` | +| `odata client` / `odata service` | `consumed odata service` / `published odata service` | +| `business event service` (covers both directions) | `consumed` / `published business event service` | +| `queue` | `task queue` | +| `database connection` | `external database connection` (or keep; less important) | +| `model` (agents) | `ai model`. `model` is too generic and collides with `alter settings model`. | +| `alter project security` | `alter app security ( Level: production, … )` | +| `alter settings model` | the Studio Pro tab names: `runtime`, … | +| `create json structure … snippet '…'` | `… sample $$…$$`. `snippet` is already a page document type. | + +Also reserve `consumed web service`, `published web service` and `xml schema` now, even though they are not implemented. + +**Before** → **After:** + +```mdl +create rest client Shop.Api (…) -- becomes: create consumed rest service Shop.Api (…) +create odata client Shop.Crm (…) -- becomes: create consumed odata service Shop.Crm (…) +create odata service Shop.Api (…) -- becomes: create published odata service Shop.Api (…) +show project security; -- becomes: show app security; +alter settings model … -- becomes: alter settings runtime ( … ) +``` + +### R11. Strictness: a typo or a meaningless form is an error, not a no-op + +- Unknown or mis-shaped property keys are errors (§2 #6). Tightening this after beta would break scripts that were silently wrong, so it has to happen before. +- The value's shape must not decide its meaning. `Response: json from $X` vs `Body: status as $Y` is decided by `from`/`as`, not by the key. +- `;` is required on top-level statements (`MDLParser.g4:39` makes it optional). Drop the SQL*Plus `/` terminator: it adds a noise line to every describe hunk and the ADR does not mention it. +- Trailing commas are allowed in **every** list. ADR-0003 promises them, yet they are rejected in entity attributes, enumeration values, parameters and every integration list. Use one shared list rule. +- `''` is the only string escape. `STRING_LITERAL` also accepts `\'` (`MDLLexer.g4:895`), which contradicts Mendix and makes `'C:\temp'` ambiguous. +- Delete the grammar branches that can never succeed (§2 #7). + +**Before** — all of this passes `mxcli check` today. The missing `;`, the `/`, the backslash escape and the misspelt `pathh` key are all accepted; the misspelt key is then silently ignored: + +```mdl +create persistent entity Shop.Note (Text: string(200)) +/ +create persistent entity Shop.Note2 (Text: string default 'it\'s') +create rest client Shop.Api3 ( + BaseUrl: 'https://api.example.com', + authentication: none +) +{ + operation GetUser { + method: get, + pathh: '/users', + response: none + } +}; +``` + +**After:** + +```mdl +create persistent entity Shop.Note (Text: string(200),); -- `;` required, trailing comma allowed +create persistent entity Shop.Note2 (Text: string default 'it''s'); -- `''` is the only escape +create consumed rest service Shop.Api3 ( + BaseUrl: 'https://api.example.com', + Authentication: none, +) { + operation GetUser ( Method: get, Pathh: '/users', Response: none ) + -- ^ error: unknown property 'Pathh' — did you mean 'Path'? +}; +``` +- Ban bare `IDENTIFIER` in parser rules; use `identifierOrKeyword`, checked by a test like `keyword_coverage_test.go`. Today whether a keyword can be used as a name depends on where it appears. + +### R12. `describe` emits the canonical form and nothing else + +`describe` output is what reviewers read in PRs, so it defines the language in practice. + +- Emit only canonical forms, which makes `describe` the reference implementation of these rules. +- Omit defaults, because default noise inflates the output: + - associations always print `owner Default storage column on delete set null`; + - `log 'x'` comes back as `log info node 'Application' 'x'`; + - `sort by` is fully qualified; + - `commit … with events`. +- Omit **derived** layout. A microflow authored with no `@position` comes back with `@position` on every statement, about three layout lines per logic line. Re-executing that output pins the auto-layout. Extend the derived-vs-authored rule already used for `@start` to `@position`, `@anchor` and `@curve`. +- Do not invent names Mendix does not store. Layout-grid rows and columns and DataGrid 2 columns get synthetic names (`row1`, `col3`, or two columns both called `Name`). Those names churn when a column is inserted and make `alter … on dg.Name` ambiguous. Make the widget name optional in the grammar, and address grid columns explicitly. +- Fold structured control flow back into its source shape. Today `case … when A, B then` and fall-through error handlers come back as `join sharedN; … merge sharedN;` goto labels, and a lone-`if` else comes back nested instead of as `elsif`. +- Put long data sources on multiple lines (`where`, `sort by` and each XPath term on its own line). + +**Before** — real `describe` output for an association written as `create association Crit.Order_Customer from Crit.Order to Crit.Customer;` and a microflow written with **no** layout annotations: + +```mdl +create association Crit.Order_Customer +from Crit.Order to Crit.Customer +type Reference +owner Default +storage column +on delete set null; +/ +create or modify microflow Crit.CountOpen ( + $Orders: List of Crit.Order +) +returns Integer +begin + @position(360, 200) + $Open = filter($Orders, Status = 'Open'); + @position(520, 200) + $N = count($Open); + @position(680, 200) + @merge(970, 200) + @caption '$N > 10' + if $N > 10 then + @position(850, 300) + log warning node 'Crit' 'Many open orders'; + end if; + @position(1090, 200) + return $N; +end; +/ +``` + +**After** — defaults, derived layout and a caption that repeats the condition are all omitted. What is left is what the author wrote: + +```mdl +create or replace association Crit.Order_Customer + from Crit.Order to Crit.Customer; + +create or replace microflow Crit.CountOpen ( + $Orders: List of Crit.Order +) +returns Integer +begin + $Open = filter $Orders where Status = 'Open'; + $N = count $Open; + if $N > 10 then + log warning node 'Crit' 'Many open orders'; + end if; + return $N; +end; +``` + +## 4. Area-specific findings not covered by the rules + +### Domain model +- **Aliases to retire:** + - `generalization` → `extends`; + - bare `entity` → `persistent entity`, which is what `describe` emits; + - `enum M.E` / `Enumeration(M.E)` / bare `M.E` → pick one, the bare form being ambiguous with an entity reference; + - `required` → `not null`; + - `delete_behavior` (16 spellings) → `on delete …`; + - `create index X on E` → `alter entity … add index (…)`; + - the optional `by` in `calculated by`; + - the optional parentheses on a view entity's `as ( … )`; + - the enumeration caption, where `Open 'x'` and `Open caption 'x'` both work in create but alter requires `caption`. +- **Validation rules.** Required and unique are attribute constraints, but regex and range need a separate `create validation rule for …` that has no `drop` or `alter`. Make all four inline constraints: `Age: Integer range 0 to 150 error message '…'`. +- **`alter` separators.** Optional commas on `alter entity`, none on `alter association` or `alter enumeration`. Use commas everywhere. +- **`alter enumeration` value names.** They accept only `IDENTIFIER`, so `add value "Select"` fails. +- **`drop event handler on before commit`.** It can't tell apart several handlers on the same moment. Add `call M.Flow`. +- **Grammar and docs disagree.** The skill shows `rename Code as ProductCode`, but the grammar has `rename attribute Code to ProductCode`. Keep `to`. + +**Before** — three enumeration spellings, two "required" spellings, `generalization`, and validation rules as a separate statement with no `drop`: + +```mdl +create persistent entity Shop.Customer generalization Administration.Account ( + Name: string(100) required, + Status: enum Shop.Status, + Tier: Enumeration(Shop.Tier), + Region: Shop.Region +); +create validation rule for Shop.Customer.Email + regex Shop.EmailPattern + feedback 'Enter a valid email address'; +``` + +**After** — one spelling each, and every validation lives on its attribute: + +```mdl +create or replace persistent entity Shop.Customer extends Administration.Account ( + Name: string(100) not null, + Status: enum Shop.Status, + Tier: enum Shop.Tier, + Region: enum Shop.Region, + Email: string(200) matches Shop.EmailPattern error message 'Enter a valid email address', +); +``` + +### Microflows +- **`$x = …` ambiguity (§2 #5).** + - Make `set` mandatory for reassignment. `describe` already always emits it. + - Make list operations and aggregates **keyword statements over a variable**: `$A = filter $Orders where …;`, `$n = count $A;`, `$S = sort $L by Date desc;`, `$P = range $L offset $o limit $n;`. + - Nesting then no longer parses. The nesting trap in CLAUDE.md idiom 3 disappears at the grammar level instead of being caught by MDL-LISTOP02, and the `find`/`contains` collision with the string functions goes away. + - This also fixes the `RANGE($L, offset, amount)` vs `limit … offset` order inversion. +- **`retrieve … limit 1`** — see §5. +- **Aliases to retire:** + - `$o/A = v` and `set $o/A = v` → `change $o (A = v)`; + - `/` vs `.` in attribute paths → `/`; + - `SUM($L.attr)` vs `SUM($L, expr)`; + - the `case`/`else` spellings inside `split type`. +- **`@anchor(false: (from: top, to: bottom))`** → `@anchor(branch: false, from: top, to: bottom)`. `@merge(x,y)` collides with the `merge` statement; rename it to `@endmerge`. +- **Keywords that are common attribute names.** `Count`, `Sum`, `Range`, `Filter`, `Sort` and `Head` are keywords, and `sortSpec` accepts only `IDENTIFIER`, so an attribute named `Count` must be quoted. The keyword-statement form above lets these fall back to identifiers. + +**Before** — the same `$v = find(…)` shape means two things, list operations look like nestable functions, `limit 1` returns an object, and an attribute can be changed three ways: + +```mdl +declare $x integer = 0; +$x = find($S, 'abc'); -- $x was declared: the STRING function find() +$y = find($Orders, Number = 42); -- $y was not: the LIST operation FIND + +$Approved = filter($Orders, Status = Shop.Status.Approved); +$Count = count($Approved); -- count(filter(…)) also parses (MDL-LISTOP02 catches it) +$Total = sum($Approved.Amount); -- `.` here, `/` everywhere else +retrieve $First from Shop.Order where [Status = 'Open'] limit 1; -- binds an OBJECT, not a list +$First/Note = 'first'; -- also: set $First/Note = …; change $First (Note = …) +while $Count > 0 + set $Count = $Count - 1; +end while; -- begin and end while both optional +``` + +**After:** + +```mdl +declare $x integer = 0; +set $x = find($S, 'abc'); -- assignment always says `set`; find() is the string function +$y = find $Orders where Number = 42; -- a list operation is a statement, never a function call + +$Approved = filter $Orders where Status = Shop.Status.Approved; +$Count = count $Approved; -- count filter … cannot parse: the operand must be a variable +$Total = sum $Approved by Amount; +retrieve $First from Shop.Order where [Status = 'Open'] first; -- an object +retrieve $Top from Shop.Order where [Status = 'Open'] limit 1; -- a list of one +change $First (Note = 'first'); +while $Count > 0 begin + set $Count = $Count - 1; +end while; +``` + +### Pages +- **Widget-type keyword explosion.** `widgetTypeV3` has about 60 tokens, including object-list item words (`MARKER`, `SERIES`, `SCALECOLOR`, …) that then have to be quoted as names. + - Make a widget type an identifier resolved through the widget registry. + - Keep lexer tokens for structural words only, with `pluggablewidget '<id>' name` as the single escape hatch. + - Delete the aliases: `container`/`customcontainer`, `button`/`actionbutton`, `pluggablewidget`/`customwidget`. + - Remove the silent `image` fallback to a static image. + - This is the most important change for keeping MDL stable across widget versions. +- **Property-key convention.** + - A small fixed set of PascalCase common keys: `DataSource`, `Attribute`, `Label`, `Caption`, `OnClick`, `OnChange`, `Visible`, `Editable`, `Class`, `Style`, `DynamicClasses`, `DesignProperties`. + - For every other property, the pluggable widget's XML key verbatim. That key is stable across versions; Studio Pro captions are not. + - "On click" should have one key, `OnClick`. Today describe rewrites it to `Action:`. + - Booleans are `true`/`false` only, not `on/off` or `yes/no`. Enum values are unquoted. +- **Five ways to set styling.** Inline properties, `alter page set`, `alter styling`, `alter pages … where widgettype =` and `update widgets … where WidgetType like`. Collapse them to inline properties plus `alter page`, and one bulk form with one `where` language. +- **Data sources.** + - Make `database from M.E` require `from`, and require `and` between XPath terms; describe currently emits juxtaposed `[a] [b]`. + - Use one association-path form. + - Reference `Layout:`, `Icon:` and `Image:` by bare qualified name, not a string. +- **`define fragment`.** It is unqualified and session-scoped, which breaks the self-contained-statement rule. Either make it `create fragment Module.F` or remove it. + +**Before** → **After**, a data source: + +```mdl +listview lv (DataSource: database Shop.Order where [Status = 'Open'] [Total > 100]) { } +``` + +```mdl +listview lv ( + DataSource: database from Shop.Order + where [Status = 'Open'] + and [Total > 100], +) { } +``` + +### Security, workflows, settings +- **Entity `grant`.** It is the only grant with reversed word order: `grant M.Role on M.E (read, write)`. Change it to `grant read *, write (Email), create on entity M.E to M.Role where [xpath];`. The skill's own example `grant read on Shop.Product to Shop.User` does not parse today. The two forms can both parse, because they start differently. +- **Workflow decision outcomes** use `->`, the only symbol operator in MDL, which the ADR bans explicitly. User-task outcomes already have no arrow. Use `outcomes true { … } false { … }`. +- **`alter settings` mixes three syntaxes.** + - It has bare `k = v` lists, `(k: v)` items, and the verbs `add` / `modify` / `add or modify` / `remove`. It also duplicates `create or modify configuration`, and constant overrides can be dropped two ways. + - Proposed forms: + - `alter settings runtime ( AfterStartupMicroflow: M.F, … );` + - `create or modify configuration 'Default' ( … );` + - `alter configuration 'Default' set constant M.C = '…';` +- **Role, constant and demo-user headers use clauses, not property lists.** + - `create user role N (Mod.R) manage all roles`, with a positional role list; `create user role N;` with no roles fails. + - `create demo user 'u' password 'p' entity E (roles)`. + - `create constant M.C type string default 'x' exposed to client`. + - Convert them to `( Key: value )`, as scheduled events already are. +- **`alter workflow` vocabulary doesn't match create.** + - `insert condition` vs `outcomes`. + - `drop path 'Path 3'` vs `path 3`. + - The `@1` ordinal reuses the annotation sigil. +- **Guest-access doc bug.** The page shows `grant Anonymous on …`, a user role, which is always refused (MDL-GRANT02). + +**Before** — a workflow and settings as they are written today: + +```mdl +create workflow Shop.OrderApproval + parameter $WorkflowContext: Shop.OrderContext + display 'Order Approval' +begin + user task ReviewOrder 'Review the order' + page Shop.TaskPage + targeting xpath '[Status = ''Draft'']' + outcomes + 'Approve' { call microflow Shop.ACT_Process with (Order = '$WorkflowContext'); } + 'Reject' { } + ; + decision '$WorkflowContext/Total > 1000' comment 'Large order?' -- `comment` sets the CAPTION + outcomes + true -> { call microflow Shop.ACT_Escalate; } + false -> { } + ; +end workflow; + +alter settings configuration 'Default' + DatabaseType = 'PostgreSql', + HttpPortNumber = 8080; +alter settings constant 'Shop.ApiUrl' value 'https://test.example.com' in configuration 'Default'; +``` + +**After** — no arrows, no expressions in strings, `caption` says what it sets, and settings use property lists: + +```mdl +create or replace workflow Shop.OrderApproval + parameter $WorkflowContext: Shop.OrderContext + display 'Order Approval' +begin + user task ReviewOrder caption 'Review the order' + page Shop.TaskPage + targeting users xpath [Status = 'Draft'] + outcomes + 'Approve' { call microflow Shop.ACT_Process(Order = $WorkflowContext); } + 'Reject' { } + ; + decision $WorkflowContext/Total > 1000 caption 'Large order?' + outcomes + true { call microflow Shop.ACT_Escalate; } + false { } + ; +end workflow; + +create or replace configuration 'Default' ( + DatabaseType: postgresql, + HttpPortNumber: 8080, +); +alter configuration 'Default' set constant Shop.ApiUrl = 'https://test.example.com'; +``` + +### Integration and agents +- **Three mapping dialects.** + - Import is `Attr = json`. + - Export uses `as` for objects and `json = Attr` for values. + - The inline REST `Body: mapping` spells export objects with `=`. + - Message definitions use `()` and `as 'String'`. + - Adopt one rule, "the left side is the side being written", and make inline mappings reuse the standalone mapping grammar verbatim. + + **Before** — attribute lines already follow the rule; nested objects do not (`= json` on import, but `as json` on export): + + ```mdl + create import mapping Shop.IMM_Order with json structure Shop.JSON_Order { + create Shop.OrderResponse { + OrderId = orderId, + create Shop.CustomerInfo_OrderResponse/Shop.CustomerInfo = customer { + Name = name + } + } + }; + create export mapping Shop.EMM_Order with json structure Shop.JSON_Order { + Shop.OrderResponse { + orderId = OrderId, + Shop.CustomerInfo_OrderResponse/Shop.CustomerInfo as customer { + name = Name + } + } + }; + ``` + + **After** — the written side is always on the left, for objects as well as values: + + ```mdl + create import mapping Shop.IMM_Order with json structure Shop.JSON_Order { + create Shop.OrderResponse { + OrderId = orderId, + create Shop.CustomerInfo_OrderResponse/Shop.CustomerInfo = customer { + Name = name, + }, + } + }; + create export mapping Shop.EMM_Order with json structure Shop.JSON_Order { + Shop.OrderResponse { + orderId = OrderId, + customer = Shop.CustomerInfo_OrderResponse/Shop.CustomerInfo { + name = Name, + }, + } + }; + ``` +- **HTTP concepts spelled three or four ways.** Headers, basic auth, constants and timeout. + - `Headers: ( 'Name': expr, )`. + - `Authentication: basic ( Username: …, Password: … )` for the REST client, the OData client, the database connection and `call rest service`. + - `Timeout:` in seconds everywhere. +- **`OpenAPI:` as a property** is a generation directive, not stored state, so it can't round-trip. Use `create consumed rest service X from openapi '…' ( … )`, the same shape as `create external entities from …`. +- **Foreign content.** Accept `$$…$$` wherever the content is foreign; `Body: template '…'` can't be multi-line today. +- **Agent attachments come in three shapes.** Use one: `<kind> <LocalName> ( Source: M.Doc, … )`. +- **Enum-like values.** They are quoted in some places (`type 'MSSQL'`, `export level 'Public'`) and bare in others. Make them bare. OData accepts both `Yes/No` and `true/false`; keep `true/false` only. + +## 5. Changes that cannot be bridged by an alias + +These change the meaning of text that parses today, or they reject text that is accepted today. They must land before beta or never. + +1. **Pick one name for the idempotent create (R1).** Only the view-entity path gives the two names different behaviour, so aliasing the loser is safe only once view entities also use the identity-carrying rewrite. +2. **`retrieve … limit 1`.** + - Today it binds an **object** (the Mendix "first" range), so to any SQL reader it means something different from what it does. + - Import mappings already solved this with `first | limit n`, and their own grammar comment argues that a list of one and an object "cannot share syntax". + - Adopt `retrieve $x from … where … first;` for an object, so that `limit 1` means a list of one. + - Sequence: in the release before beta, warn on a bare `limit 1` and have `describe` emit `first`. At beta, flip the meaning. +3. **Make `set` mandatory, and turn list operations into keyword statements**, removing the `$x = find(...)` ambiguity. +4. **Unknown property keys become errors.** +5. **`;` becomes required**, and `/` is removed. +6. **`\` is no longer an escape character** in string literals. +7. **Remove the dead grammar**: `throw`, `grant … on workflow`, `case … else`, `text`/`statictext`, and the parenthesised association form. +8. **Remove `float`, `currency` and `date`** as attribute types. They are accepted and mis-stored today. +9. **Omit synthetic widget names and derived layout** from describe. This changes describe output, which people have already committed to repos. + +Everything else in this document can land after an alias period. + +## 6. Migration mechanism + +The language has no version marker today. `version-aware-mdl.md` proposes `set version '10.18'`, but that is the **Mendix target version**, a different axis. Proposal: + +1. **Deprecated aliases keep parsing** and emit a warning with a stable code (`MDL-DEPR001`, …) that names the canonical form. `describe` never emits a deprecated form. +2. **`mxcli fmt --upgrade`** rewrites every deprecated form to its canonical form. This is mechanical for every alias in §3–§4, and it is what makes consolidation cheap for users. +3. **An optional language header**, `mdl 1;`, as the first statement. + - `describe` and `fmt` emit it. + - With no header, a script gets the latest language version. + - After beta, a change that is not backwards compatible bumps the number, and the visitor gates removed aliases on it: under `mdl 1` they warn, under `mdl 2` they are refused. + - It is independent of the Mendix target version. +4. **Skills, `mxcli syntax` and the quick reference** are regenerated from, or checked against, describe output. They should not be hand-maintained. The skill examples already disagree with the grammar in several places, such as `DELETE_BEHAVIOR PREVENT` and `rename … as`. + +### Suggested order + +See the implementation plan in §9, which supersedes the short list that was here. + +## 7. Open decisions for the maintainer + +1. **R1, the idempotent-create keyword.** `create or replace` (honest about full-definition semantics) or `create or modify` (what describe emits today)? Renames stay in `alter`/`rename` either way. +2. **R3/R4, argument binding.** Should it be `Param = expr` (proposed: `=` binds runtime values, `:` sets model properties) or `Param: expr`? The reviews split on this. `=` aligns calls with `change`, `set` and the existing microflow describe output. `:` aligns with today's page describe output. Pick one; do not keep both. +3. **Alter form.** Should it be `alter X set ( Key: value )` (proposed, the same list as create) or SQL `alter X set Key = value`? The latter is more SQL-like, but it makes the same key take two separators. +4. **List operations as keyword statements.** This is the largest microflow change. The alternative is to keep the function form, forbid nesting in the grammar, and rename the `find`/`contains` list operations so they no longer collide with the string functions. +5. **Retire `show` entirely?** Or keep it for non-element state, as proposed? +6. **The `mdl 1;` header.** Adopt it now, or rely on aliases plus `fmt --upgrade` until the first post-beta break? + +## 8. Two ways of working: MDL-first and data-first + +Most agent work will happen on large existing apps authored in Studio Pro. There, MDL is a *view* over a stored model, not the source of truth. §3–§7 judge MDL as a language for writing. This section judges it as a language for **reading, locating and patching**. + +Three measurements were taken on two real apps: PedApp (Mendix 11.13) and Evora Factory Management (Mendix 10.24; 42 modules, 1765 microflows, 212 pages). + +### 8.1 Findings + +**Reading mostly works, but the answers it gets wrong are dangerous.** + +What works: +- `structure -m Module` is a good signature view. +- `select … from CATALOG.*` is cheap once you know the schema. +- `search --format names` is excellent. + +Orienting on a realistic task cost about 5–7k tokens. But: +- **`impact`/`refs` has no attribute, enumeration-value, `call workflow`, widget→association or mapping→entity edges.** So `impact Module.Entity.Status` answers "not referenced" for a used attribute. Only a source search (`select … from CATALOG.SOURCE where source match …`) found the real usages. That needs an 8-minute `refresh catalog full source`, and without it `search` returns empty results with no warning. +- **Too many overlapping read commands:** `show`/`describe`, `refs`/`impact`/`context`, `search`/`select … SOURCE`, `show modules`/`structure`. The overlap costs wrong turns more than tokens. +- **No partial reads.** There is no way to describe one widget or one microflow branch; `describe fragment from page … widget` can never succeed (bug). `--json` is not pure JSON on most commands. +- **Layout noise is 30–57% of microflow `describe` output**, depending on the flow. A 123-widget page is about 6k tokens. + +**Modifying is safe only through `alter`, and the most-edited document type has no `alter`.** +- **The `alter` control preserved everything.** `alter page … set Caption` changed exactly one en_US string and byte-preserved every other translation and text. +- **The describe → edit → `create or modify` path lost data in 7 of the 12 Studio Pro documents it actually wrote, even with no edit at all:** + - association storage Table→Column, which is a schema change, and `mxcli diff` reported "no changes"; + - page translations (113→102), and an empty English caption filled from the Dutch one; + - nanoflow annotation links; + - export levels on nanoflows and Java actions (the Java action became Public); + - snippet `Type`. +- **There is no `alter microflow` or `alter nanoflow`.** A one-line insert into a 16-activity Studio Pro microflow required: + - re-emitting all 107 lines; + - deleting 5 merges and resetting 11 of 15 curves; + - changing 51 of 161 element `$ID`s, with some reassigned to *different* nodes; + - placing the new activity on top of an existing one. + + The root cause is the architecture: `UpdateMicroflow` rebuilds the whole document from the AST and re-pairs IDs afterwards by type and position (`modelsdk/canon/transplant.go`). No per-statement fix can make that a fixed point. +- **`mxcli diff` is not a dry run.** It renders both sides as text for five create statements only. It lists every `alter` as "not compared" and gives false results in both directions. + +### 8.2 Two modes: MDL-first and data-first + +Neither mode is better than the other in general. Each is the efficient one for a different situation, and MDL has to support both as first-class. + +| | **MDL-first (declarative)** | **Data-first (patch)** | +|---|---|---| +| Source of truth | the `.mdl` scripts | the stored model (`.mpr`) | +| Typical use | new apps, new modules, generated scaffolding, documents mxcli owns | existing Studio Pro apps, documents people also edit in Studio Pro, marketplace modules | +| Main statement | `create or replace <type> X ( … ) { … }`, the whole definition (R1) | `alter <type> X { insert … / replace … / set … / drop … }` (§8.3) | +| What the author must know | only the intended end state; no read needed | the current state, well enough to address the target | +| Token cost of a change | proportional to the **document** | proportional to the **change** | +| Reviewability | the script *is* the design; a PR diff shows the new definition | the patch *is* the intent; a PR diff shows exactly what was changed and where | +| Identity | carried by name, so renames need `alter`/`rename` (R1) | untouched elements are byte-identical by construction | +| Risk | drops whatever the statement does not say, including content MDL cannot express | an ambiguous or stale address; needs strict matching (§8.3) | + +**For a new app, declarative is clearly more efficient.** There is nothing to read, nothing to address and nothing to preserve. Each element is written once, in its final shape, in the order a reader wants to see it. A patch-only style would force an agent to create empty shells and then fill them in with a sequence of edits: more tokens, more statements, and a script that describes a history rather than a design. Declarative scripts are also what an LLM generates most reliably, since one example generalises (ADR-0003). + +**For a small change to a large existing document, patching is clearly more efficient and safer.** It costs O(change) instead of O(document), and it cannot disturb what it does not mention (§8.1). + +**The crossover** depends on the change and on who owns the document: +- A change that rewrites most of a document is cheaper declaratively, even on an existing app, *provided* the document round-trips (§8.4). +- A change that touches a small part is cheaper as a patch, even on an app built MDL-first. + +**The modes are phases of one app's life, not a choice made once.** An app typically starts MDL-first. Then it is opened in Studio Pro, and from that point the stored model can drift away from the scripts. The practical question per document is therefore *"is the stored document still exactly what my MDL last produced?"*: +- **Yes:** declarative replace is safe and cheapest. +- **No** (someone edited it in Studio Pro, or it was never MDL): patch, or re-adopt it as MDL source first by running `describe`. Re-adopting is safe only for types whose round trip is proven. + +That question can be answered mechanically, the way Terraform detects drift: +- Record a fingerprint of each document's canonical BSON when mxcli writes it. +- `create or replace` compares the stored document against that fingerprint. +- **Match:** the replace proceeds. +- **Drift:** the replace is refused with "changed outside MDL since the last apply; use `alter`, or `describe` to re-adopt, or `--force`". + +This makes the choice between modes visible and safe, instead of something the agent has to remember. + +**Language consequences.** The two modes must share one syntax, so that moving between them is free: +- A fragment inside `alter … { … }` is written exactly as in `create`. +- `describe` output is valid declarative source. +- An agent should never have to learn a second language to switch modes, and a reviewer should see the same constructs either way. + +This is another reason for R2's uniform node shape and R12's canonical describe. + +### 8.3 The model is data: edits are tree patches + +MDL describes stored data, so it can be manipulated the way Lisp manipulates code: the text is a structure, and edits are structural operations on it. Four consequences follow. + +1. **One generic `alter` for every document type.** Rule R2 gives every child the shape `<kind> [Name] ( props ) { children }`. That rule is what makes a single patch grammar possible, replacing about 15 bespoke `alter` forms: + + ```mdl + alter <type> Module.Name { + set ( Key: value ) on <target>; + insert before|after|into <target> { <fragment> } + replace <target> with { <fragment> } + drop <target>; + } + ``` + + Fragments are written in exactly the syntax `create` uses; `alter page` already works this way. This generic form is also the strongest argument for settling R2 before beta. + +2. **Content addressing where there are no names.** Pages address widgets by name. Microflow activities have no names, so they are addressed by what they are, in this order of preference: + 1. by output variable: `after $Lines`; + 2. by caption: `before 'Email is valid?'`; + 3. by statement pattern with wildcards: `after commit $Order`, `drop log * node 'Debug' *`. + + A pattern that matches more than one activity is an error that lists the matches with an ordinal (`@2`); it is never a guess. `describe` can mark targets so an agent sees which handle to use. + + ```mdl + alter microflow Shop.ACT_CancelOrder { + insert after commit $Order { + call microflow Shop.SUB_EmailCustomer(Order = $Order); + } + replace retrieve $Lines with { + retrieve $Lines from Shop.OrderLine where [Shop.OrderLine_Order = $Order]; + } + } + ``` + +3. **A patch splices into the stored graph; it does not rebuild it.** The executor rewires sequence flows around the target and places only the new nodes, shifting downstream nodes rather than overlapping them. Every untouched activity, flow, curve and merge stays byte-identical. The microflow is a graph, not a tree: + - Tree edits apply to its structured regions (`if`, `loop`, error handlers). + - Unstructured regions are addressed through the `join`/`merge` labels that `describe` already prints. + - An inserted fragment is checked in the scope of its insertion point, for macro hygiene: a variable it declares must not collide with one declared further down. + +4. **Bulk patches are "macros":** a query plus a patch. + + ```mdl + alter microflows in Shop where contains (commit $Order) { + insert after commit $Order { call microflow Shop.SUB_Audit(Order = $Order); } + } + ``` + +### 8.4 The two round-trip laws + +Code-as-data only works if printing and reading are inverse operations. Formally, `describe` and `create or …` form a *lens* over the stored model, and they must obey two laws: + +- **GetPut:** executing `describe X` unchanged writes nothing. ADR-0008's write elision already covers this at the storage level, but today it is violated in 7 of 12 cases (§8.1). +- **PutGet:** executing a statement, then describing the result, returns what was written. This law is R12: no invented names, no derived layout, no defaults. + +Content MDL cannot express must never be silently dropped. It needs one of two things: +- **An opaque passthrough placeholder** that `describe` emits and a replace carries through, e.g. `preserved activity 'a1b2…';`. +- **A refusal:** "this rebuild would drop 3 translations; use `alter`". + +Silent loss is never acceptable. + +### 8.5 Recommendations (in order) + +1. **Guidance now: choose the mode by who owns the document (§8.2).** + - New apps and modules: write declarative MDL. + - Existing Studio Pro documents: change them with `alter`. + - describe → replace only for documents whose stored state is still what MDL produced, and never on a Studio Pro-authored document of a type without a proven round trip. +2. **Drift detection** on `create or replace`: a per-document fingerprint recorded at write time, refusing the replace on drift (§8.2). Until it exists, the guidance in step 1 is the only guard. +3. **A CI round-trip test** on a Studio Pro fixture: describe → exec → canonical BSON compared per unit, covering every document type. It would have caught every loss in §8.1. Fix those losses, or turn them into refusals. +4. **`alter microflow` / `alter nanoflow`** with content addressing and graph splicing (§8.3). This is the single largest gap for brownfield work. +5. **A real dry run.** Execute on an in-memory copy and diff canonical BSON per unit. The machinery exists in `canon.Reconcile`. Until then, `diff` saying "no changes" is not evidence of no change. +6. **Read side:** + - Complete the refs graph (attributes, enum values, `call workflow`, widget bindings, mappings) and add a typed `usages <Entity.Attr | Enum.Value>`. + - Add partial and outline reads: `describe … brief`, `describe microflow … without layout`, `describe page X widget Y`, `describe page X outline`. + - Make `--json` strict and uniform: stdout is JSON only. + - Make the source index incremental, or say it is missing. + - Collapse the overlapping read commands along the R6 verb table. +7. **Settle R2 and R12 before beta.** Both are preconditions for the generic `alter`, not just style rules. + +## 9. Implementation plan + +The plan has six phases: +- Phases 0 and 1 are foundations: a safety net, and the machinery that makes syntax changes cheap. +- Phase 2 is the **beta gate**: the changes that cannot be bridged by an alias. +- Phases 3–5 can continue past beta, because every change in them keeps the old form parsing. +- Phase 4 (data-first editing) depends only on phases 0 and 1, so it can run in parallel with phases 2 and 3. + +Every item follows the repo's working rules (CLAUDE.md): +- One concern per PR. +- The test is written first. +- The fix is proven by reverting it. +- Any "nothing changed" assertion comes with a control. + +Sizes are rough: **S** is about 1 PR, **M** is 2–4 PRs, **L** is a series. + +``` +Phase 0 safety net + bugs ──┬──> Phase 1 decisions + deprecation machinery ──> Phase 2 non-bridgeable (BETA GATE) ──> Phase 3 aliases, area by area + └──> Phase 5 read side (any time) + Phase 1 ──> Phase 4 data-first editing (alter microflow, drift, dry run) +``` + +### Phase 0: safety net and bugs (no syntax change) + +| # | Item | Size | Where | Done when | +|---|---|---|---|---| +| 0.1 | **Round-trip harness.** For every document in a Studio Pro-authored fixture: `describe` → `exec` → canonical BSON compared per unit. Known failures go in an allowlist that may only shrink. | M | integration test (`-tags integration`); compare with `canon` | runs in CI; the allowlist equals the §8.1 losses | +| 0.2 | **Fixture.** A committed Studio Pro-authored project with microflows, nanoflows, pages, snippets, translations, table-stored associations and a workflow. PedApp covers most of these, but its licence must be checked; add a workflow and REST documents. Evora (`mx-test-projects/`) serves as a large, uncommitted read-side benchmark. | S | `testdata/` | fixture in the repo | +| 0.3 | **Silent drops from §2:** `throw`; `float`/`currency`/`date`; the parenthesised association form; enumeration-value doc comments and index names; describe output that doesn't re-parse (#8); the plaintext password (#9); `/tmp` image paths (#10). | M | visitor, executor describe | each has a failing test first; §2 closed | +| 0.4 | **Round-trip losses from §8.1:** association storage, page translations, nanoflow annotations and export level, Java action export level, snippet `Type`, describe emitting a plain `create`. Tracked as separate tasks. | M | executor create paths | removed from the 0.1 allowlist | +| 0.5 | **Read-side wrong answers from §8.1:** `describe fragment`, `structure` counts, silent empty `search`, strict `--json`. Tracked as a task. | S | executor | tests | +| 0.6 | **Interim agent guidance:** add §8.2 "choose the mode by owner" and "change existing documents with `alter`" to `write-microflows`, `alter-page` and the other doctype skills; run `make sync-skills`. | S | `.claude/skills/mendix/` | skills merged | + +### Phase 1: decisions and infrastructure + +| # | Item | Size | Where | Done when | +|---|---|---|---|---| +| 1.1 | **Decide §7** and record the rules. Write an ADR ("MDL canonical syntax: R1–R12, two modes") that extends ADR-0003, and turn R1–R12 into checklist items in `design-mdl-syntax.md`. Mark the v1/v2 syntax proposals **rejected**; fold the workflow alignment proposal into R4/R9. | S | `docs/13-decisions/`, skill | ADR accepted | +| 1.2 | **Deprecation registry.** Generalise the existing MDL065 pattern, where the AST records which spelling the source used (`ast_microflow.go:273`). Add a single table of `{code MDL-DEPRnnn, old form, canonical form, removed in}`. `check` and `exec` warn; a `--deprecations=error` flag fails instead. | M | `mdl/ast`, `mdl/linter`, new `deprecations.go` | a test fails if a grammar alternative marked as an alias has no registry entry | +| 1.3 | **`mxcli fmt --upgrade`.** Each registry entry carries a rewrite, built on `mdl/formatter` / `cmd_fmt.go`. The output must parse with zero deprecation warnings and give an identical AST. | M | `mdl/formatter` | a property test over all of `mdl-examples/`: upgrade, then zero warnings, then AST equal | +| 1.4 | **Canonical-form conformance gate.** Everything in `mxcli syntax`, the skills, `docs-site/` and `mdl-examples/` is parsed with `--deprecations=error`, starting with an allowlist. This stops the docs teaching old forms, a gap measured in §3. | S | `make lint` target | allowlist only shrinks | +| 1.5 | **Language header `mdl 1;`** (if decided). It is parsed and emitted by `describe`/`fmt`, and gates alias removal after beta. | S | grammar, visitor | round-trips | +| 1.6 | **Grammar hygiene** that makes later phases cheaper: one shared trailing-comma list rule; ban bare `IDENTIFIER` in parser rules with a test modelled on `keyword_coverage_test.go`; move session commands out of `utilityStatement` into the REPL (R7). | M | `mdl/grammar` | tests; `make grammar` clean | + +### Phase 2: the beta gate (§5, cannot be aliased) + +Each change breaks existing text, so each one lands with a registry entry, an `fmt --upgrade` rewrite where one is possible, and release notes. Where a warning period is possible, it lasts one release before the break. + +| # | Item | Size | Warning period | Done when | +|---|---|---|---|---| +| 2.1 | **R1: one idempotent-create keyword.** View entities get an identity-carrying rewrite instead of delete-and-recreate; `if not exists` works on every type. | M | alias the losing keyword (safe once view entities carry identity) | GUID preserved on a view-entity replace (a test with a GUID != `$ID` control) | +| 2.2 | **Strictness:** unknown or mis-shaped property keys are errors (REST, agents and business events first, then everywhere); `;` required; `/` removed; `\` is no longer an escape. | M | `;` and `/` warn for one release; unknown keys and `\` break immediately | parse tests | +| 2.3 | **Dead grammar removed:** `grant … on workflow`, `case … else`, `text`/`statictext`/`legacydatagrid`, `throw` (if not fixed in 0.3). | S | none (they never worked) | parse errors with a hint | +| 2.4 | **Microflow assignment and list operations.** `set` becomes mandatory. List operations and aggregates become keyword statements (`$A = filter $L where …`, `$n = count $A`), so nesting no longer parses. The `find`/`contains` ambiguity disappears; MDL-LISTOP02 becomes unreachable and is retired. | L | old function form warns (except `find`/`contains`, which can't) | `write-microflows` skill and examples migrated by `fmt --upgrade` | +| 2.5 | **`retrieve … first` vs `limit 1`.** In release N, `describe` emits `first` and a bare `limit 1` warns. In release N+1, the beta, `limit 1` means a list. | M | one release | MDL-RETRIEVE01 retired | +| 2.6 | **Canonical `describe` (R12), one area per PR:** omit derived layout (extend the `@start` derived-vs-authored rule to `@position`/`@curve`/`@anchor`); omit defaults; no synthetic widget names (make the name optional in `widgetV3`, address grid columns explicitly); fold `join`/`merge` back into `case` and fall-through handlers; emit `elsif`. | L | none (output change) | the harness in 0.1 stays green; PutGet holds per area | + +**Beta gate:** +- Phases 0, 1 and 2 are complete. +- Phase 3's canonical forms are *decided and in the grammar*. Aliases may remain. +- Phase 4.2 has at least insert, replace and drop, if brownfield agent use is a beta goal. + +### Phase 3: consistency through aliases (R2–R10, §4) + +Every item follows the same recipe: +1. Add the canonical form to the grammar. +2. Register the old form as a deprecation (1.2) with an `fmt --upgrade` rewrite (1.3). +3. Switch `describe` to the canonical form. +4. Migrate examples, skills and `mxcli syntax` with `fmt --upgrade`. +5. Shrink the 1.4 allowlist. + +Order, by how many existing scripts each item touches: + +| # | Rule | Size | +|---|---|---| +| 3.1 | R3/R4: `:` vs `=`; one argument-binding form; `with ({1} = …)` for text templates | L | +| 3.2 | R8: page actions as words (`show page`, `save changes`); one spelling per keyword; `error message`; lowercase canonical (fixes `fmt` upper-casing keys) | M | +| 3.3 | R5: XPath in `[ ]`, bare expressions (grants, workflow targeting and decisions, `Visible`/`Editable`); one constant reference | M | +| 3.4 | R2: brackets (REST client, agents, image collection, database connection, message definitions, navigation/menus, `on error begin … end error`, `while`) | L | +| 3.5 | R6/R7: `list`/`describe`/`show` split; `drop` instead of `remove`; missing `drop`s; `call rest service` | M | +| 3.6 | R9/R10: metadata placement (doc comment, `folder` clause); Studio Pro document names (`consumed rest service`, …); property lists for role, constant, demo user and settings headers | M | +| 3.7 | §4 area items: domain-model aliases and inline validation rules; widget types resolved through the registry and the property-key convention; unified mapping grammar; HTTP concepts; workflow `caption`/`->`/`alter workflow` vocabulary | L | + +### Phase 4: data-first editing (§8) + +| # | Item | Size | Where | Done when | +|---|---|---|---|---| +| 4.1 | **Generic `alter <type> X { set / insert / replace / drop }`.** One grammar rule plus a per-doctype *target resolver* interface. `alter page`/`snippet`/`layout` (`pagemutator`) and `alter workflow` (`wfmutator`) are ported onto it first, with their old forms as aliases. | M | grammar, `mdl/backend` | existing alter tests pass through the new path | +| 4.2 | **`alter microflow` / `alter nanoflow`**, built as an `mfmutator` alongside `pagemutator` and `wfmutator`: | L | `mdl/backend/modelsdk`, `mdl/microflowgraph` | see acceptance below | +| | a. **Target resolver.** Address by output `$var`, by caption, or by statement pattern with `*` wildcards; `@n` disambiguates, and ambiguity is an error listing the matches. Add `describe … with handles` to show the addresses. | | | | +| | b. **Graph splice** on the *stored* object collection. Rewire the incoming and outgoing sequence flows around the target; build only the fragment's objects; never call the whole-document `UpdateMicroflow` rebuild. The write goes through `canon.Reconcile` (CLAUDE.md rule 2). | | | | +| | c. **Placement.** Put the new node on the flow's midpoint and shift downstream nodes; never overlap. | | | | +| | d. **Hygiene.** Check the fragment in the scope of the insertion point; a variable collision is an error. | | | | +| | e. **Operations,** in order: `insert after`/`before` → `replace` → `drop` → `set` (expression, caption, `on error`) → `add`/`drop parameter`. | | | | +| | f. **Both backends:** the modelsdk engine and `--mcp` (the Studio Pro MCP backend), or an explicit "not supported by this backend" error. | | | | +| 4.3 | **Drift detection.** Every write path records a per-unit canonical-BSON fingerprint through `canon.Reconcile`. `create or replace` refuses on drift, with `--force` to override. Open question: where the fingerprints live (a sidecar `.mxcli/state` file vs committed with the scripts). | M | `modelsdk/canon`, executor | a Studio Pro edit between two applies is detected; a control with no edit is not | +| 4.4 | **A real dry run:** `exec --dry-run`, replacing today's `diff`. Execute on an in-memory copy, diff canonical BSON per unit, and render changed units as a `describe` diff. Covers `alter` and every document type. | M | executor, `canon` | the §8.1 false negative (association storage) and false positives disappear | +| 4.5 | **Opaque passthrough or refusal** for content MDL cannot express, per document type as the 0.1 harness finds it: `preserved <kind> '<id>'` in `describe` output, carried by replace. | M | per doctype | no silent loss remains in the harness | +| 4.6 | **Bulk patches:** `alter microflows|pages in M where contains (<pattern>) { … }`, building on 4.2a. `alter pages … where` and `update widgets` are folded into it as aliases. | M | grammar, executor | — | + +**Acceptance for 4.2** on the Studio Pro fixture's `VAL_Feedback`: +- Insert one `log` after `$IsValidEmail`. +- Only the new activity, the two rewired flows and any shifted positions differ in canonical BSON. Every other element's `$ID`, curve and merge is byte-identical. +- Control: an empty `alter` changes nothing. +- `mx check` is unchanged from the baseline, and Studio Pro opens the project. +- An MDL script of at most 5 lines replaces today's 107. + +### Phase 5: read side (§8.1; any time) + +| # | Item | Size | +|---|---|---| +| 5.1 | **Complete the refs graph:** attribute and enumeration-value targets (expressions, XPath, widget bindings, `ContentParams`, change/create members, mappings); `call workflow` edges; widget→association; mapping, OData and REST→entity. `impact` groups by element and has a kind column. | L | +| 5.2 | **`usages <Entity.Attr \| Enum.Value>`**, and typed `callers`/`callees`. | S | +| 5.3 | **Partial and outline reads:** `describe … brief` (signatures only), `describe microflow … without layout`, `describe page X widget Y`, `describe page X outline`. | M | +| 5.4 | **Strict, uniform `--json`:** JSON only on stdout, status on stderr, one `{kind, qualifiedName, …}` shape. | M | +| 5.5 | **Incremental source index,** or a warning when it is missing. Uniform catalog column names (`QualifiedName` everywhere); hide snapshot columns from `select *`. | M | +| 5.6 | **Collapse overlapping read commands** per R6. Remove `show <element>` in favour of `describe`; merge `refs`/`impact`/`context`; document `search` as sugar over `CATALOG.SOURCE`. | M | + +### Risks + +- **`describe` output changes (2.6, Phase 3) churn MDL that users have committed.** Mitigation: ship them together in as few releases as possible before beta, with `fmt --upgrade` and a changelog entry per change. +- **Grammar changes ripple into generated artefacts:** LSP completions (`lsp_completions_gen.go`), `keyword_coverage_test.go`, the VS Code extension and the embedded skills. Each grammar PR runs `make build`, which regenerates them, and `make sync-skills`. +- **The graph splice (4.2b) is the riskiest code.** It must never rewrite an `$ID` without rewriting every reference to it (CLAUDE.md rule 1), and its tests must use a Studio Pro-authored flow, because an mxcli-created flow cannot show identity loss (GUID == `$ID`). +- **The harness fixture's licence.** If PedApp can't be committed, build a fixture in Studio Pro specifically for this purpose. +- **Scope creep in Phase 3.** Hold each rule to its recipe; anything beyond renaming belongs in its own proposal. From 52fdfd47dedd06e36a2a2494c5ae7508eef68d52 Mon Sep 17 00:00:00 2001 From: Ako <andrej@koelewijn.net> Date: Sat, 26 Sep 2026 13:23:28 +0000 Subject: [PATCH 05/15] docs: record MDL beta decisions in the syntax proposal - create or modify is the one idempotent create, with minimal-change semantics: an unchanged definition writes nothing; implemented as diff-then-patch on the same splice engine as alter. - show is dropped in favour of list (and describe for single things). - microflow list operations mirror Studio Pro's List operation and Aggregate list activities, one statement per activity. - PedApp may be committed as the round-trip fixture. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --- .../PROPOSAL_mdl_beta_syntax_freeze.md | 124 +++++++++++------- 1 file changed, 80 insertions(+), 44 deletions(-) diff --git a/docs/11-proposals/PROPOSAL_mdl_beta_syntax_freeze.md b/docs/11-proposals/PROPOSAL_mdl_beta_syntax_freeze.md index 04d13cdf2..4f539d4eb 100644 --- a/docs/11-proposals/PROPOSAL_mdl_beta_syntax_freeze.md +++ b/docs/11-proposals/PROPOSAL_mdl_beta_syntax_freeze.md @@ -29,7 +29,7 @@ The review also found **silent-loss bugs**: input that parses and is then droppe - Microflows, the most-edited type, have no `alter` at all. §8 argues that MDL needs two first-class modes that share one syntax: -- **MDL-first:** declarative `create or replace` scripts as the source of truth. This is the efficient mode for new apps. +- **MDL-first:** declarative `create or modify` scripts as the source of truth. This is the efficient mode for new apps. - **Data-first:** content-addressed `alter` patches that splice into the stored graph. This is the efficient mode for changes to existing Studio Pro documents. Drift detection chooses between them safely, and two round-trip laws make moving between them lossless. @@ -81,7 +81,7 @@ Each rule is stated so it can be added to the `design-mdl-syntax` skill as a che **About the examples.** - Every **Before** block was run through `mxcli check` at `5dc51ceb` and parses today. The `describe` output in R12 is real output from a scratch project. -- Every **After** block is *proposed* syntax and does not parse yet. Where §7 leaves a choice open, the After blocks use the recommended option: `or replace`, `Param = expr` for arguments, and `set ( Key: value )` for alter. +- Every **After** block is *proposed* syntax and does not parse yet. They follow the decisions recorded in §7. Where §7 still leaves a choice open, they use the recommended option: `Param = expr` for arguments, and `set ( Key: value )` for alter. ### R1. One idempotent create, with one defined meaning @@ -97,16 +97,14 @@ The semantics are also already declarative, not a merge: That settles renames. A declarative definition cannot tell a rename from a drop plus an add, so under it a rename destroys the column. **Renames are therefore only expressible as `alter entity … rename attribute A to B` or `rename entity … to …`**, which carry the identity across. The same holds for any document whose members have identity. -Rule: -- One keyword, with the semantics above written into the skill and the ADR: "full definition; name-keyed identity carry; omitted members dropped and reported; renames need `alter`/`rename`". -- The keyword's name is an open decision (§7): - - `or replace` describes the full-definition behaviour honestly and matches SQL's `CREATE OR REPLACE VIEW`. - - `or modify` is what `describe` emits today, but it reads like a merge. That is exactly the misreading that makes an omitted attribute a surprise. -- Whichever is chosen, the other becomes a deprecated alias. +Rule (**decided**, §7): +- **`create or modify` is the one keyword.** It reads like the author's intent: *make the stored document match this definition*. `create or replace` becomes a deprecated alias. +- **Minimal change is part of the meaning.** `modify` changes only what differs, so re-running a statement whose definition already matches the stored document **writes nothing**, and Studio Pro shows no change. That covers byte-identical units, no `$ID` churn and no version-control noise. This is the GetPut law of §8.4, promoted from a goal to the defining property of the keyword. It is **not** true today: an unchanged microflow round trip rewrote the unit and moved 51 of 161 element `$ID`s (§8.1). The implementation consequence is in §8.3 and plan item 4.2. +- The other semantics, written into the skill and the ADR: the statement is the full definition; members are matched by name and keep their identity; omitted members are dropped and reported (MDL087); renames need `alter`/`rename`. - Make view entities follow the same identity-carrying rewrite instead of delete and recreate, so the keyword means one thing on every type. - Keep `if not exists` as a separate, genuinely different operation (leave an existing element untouched). Extend it to every type, not just entities and associations. - Move `create module role` into `createStatement` (`MDLSecurity.g4:17`), so it gets the same prefix as every other type. -- Every `describe` emits the chosen keyword. Today association, layout, user role, demo user, REST client, OData, agent and database connection emit a plain `create`, and navigation emits `create or replace`. +- Every `describe` emits `create or modify`. Today association, layout, user role, demo user, REST client, OData, agent and database connection emit a plain `create`, and navigation emits `create or replace`. **Before** — two keywords for one operation, and a rename that looks harmless: @@ -119,10 +117,10 @@ create or replace persistent entity Shop.Order (Number: integer, Note: string(20 create or modify persistent entity Shop.Order (Number: integer, Remarks: string(200)); ``` -**After** — one keyword that says what it does, and renames are a separate statement: +**After** — one keyword that says what it does, a no-op when nothing differs, and renames as a separate statement: ```mdl -create or replace persistent entity Shop.Order ( +create or modify persistent entity Shop.Order ( Number: integer, Note: string(200), ); @@ -205,7 +203,7 @@ create image collection Shop.Icons { image Logo ( File: 'assets/logo.png' ) }; -create or replace navigation Responsive ( +create or modify navigation Responsive ( HomePage: Shop.Home, ) { menu item 'Home' ( OnClick: show page Shop.Home ) @@ -337,9 +335,9 @@ textbox txtNote (Attribute: Note, Visible: $currentObject/Status = 'Open') | Verb | Meaning | Changes | |---|---|---| -| `list <plural>` | enumerate elements | Retire `show <plural>`, which today leads 72 of the syntax lines versus 15 for `list`. Add plurals for `list image collection` / `icon collection` / `message definition collection`. | +| `list …` | enumerate elements, and relationship queries | Replace `show <plural>`, which today leads 72 of the syntax lines versus 15 for `list`. Add plurals for `list image collection` / `icon collection` / `message definition collection`. | | `describe <type> Name` | one element | Remove `show entity|association|page X` (`MDLCatalog.g4:62-64`). | -| `show <state>` | non-element state (status, version, access, callers, impact) | Stop `showOrList` producing forms like `list version` and `list catalog status`. | +| ~~`show`~~ | **dropped** (decided, §7) | Every one of the 88 `show` forms maps to `list`, to `describe`, or to a session command:<br>• Plurals and relationship queries become `list`: `list callers of X`, `list callees of X`, `list references to X`, `list impact of X`, `list access on E`, `list widgets …`, `list design properties …`, `list catalog tables`.<br>• Single things become `describe`: `describe entity X`, `describe navigation`, `describe app security`, `describe security matrix`, `describe structure [in M]`, `describe context of X`.<br>• Session state (`version`, `catalog status`) becomes a REPL command (R7).<br>`show` stays as a deprecated alias for one release. | | `create` / `alter` / `drop` | as in SQL | `alter` children use `add`/`drop`. Replace `remove` (user role), `modify`/`add or modify` (settings), `define fragment`, `update security` and `update widgets`. Add the missing `drop`s: database connection, validation rule, external entity. | Also: @@ -596,15 +594,15 @@ end; **After** — defaults, derived layout and a caption that repeats the condition are all omitted. What is left is what the author wrote: ```mdl -create or replace association Crit.Order_Customer +create or modify association Crit.Order_Customer from Crit.Order to Crit.Customer; -create or replace microflow Crit.CountOpen ( +create or modify microflow Crit.CountOpen ( $Orders: List of Crit.Order ) returns Integer begin - $Open = filter $Orders where Status = 'Open'; + $Open = filter $Orders by Status = 'Open'; $N = count $Open; if $N > 10 then log warning node 'Crit' 'Many open orders'; @@ -649,7 +647,7 @@ create validation rule for Shop.Customer.Email **After** — one spelling each, and every validation lives on its attribute: ```mdl -create or replace persistent entity Shop.Customer extends Administration.Account ( +create or modify persistent entity Shop.Customer extends Administration.Account ( Name: string(100) not null, Status: enum Shop.Status, Tier: enum Shop.Tier, @@ -661,9 +659,35 @@ create or replace persistent entity Shop.Customer extends Administration.Account ### Microflows - **`$x = …` ambiguity (§2 #5).** - Make `set` mandatory for reassignment. `describe` already always emits it. - - Make list operations and aggregates **keyword statements over a variable**: `$A = filter $Orders where …;`, `$n = count $A;`, `$S = sort $L by Date desc;`, `$P = range $L offset $o limit $n;`. + - Make list operations and aggregates **statements that mirror Studio Pro's activities** (decided, §7; see *List operations mirror the diagram* below). - Nesting then no longer parses. The nesting trap in CLAUDE.md idiom 3 disappears at the grammar level instead of being caught by MDL-LISTOP02, and the `find`/`contains` collision with the string functions goes away. - This also fixes the `RANGE($L, offset, amount)` vs `limit … offset` order inversion. +- **List operations mirror the diagram** (decided, §7). A developer who knows the microflow editor should recognise each statement as the activity they would drag in. + + The rules: + - **One statement per activity.** Each statement corresponds to one *List operation* or *Aggregate list* activity. + - **The keyword is the operation's name.** These are the operations the Mendix metamodel defines for those activities (`Microflows$Filter`, `FilterByExpression`, `Find`, `FindByExpression`, `Sort`, `Head`, `Tail`, `ListRange`, `Union`, `Intersect`, `Subtract`, `Contains`, `ListEquals`, and aggregate functions `Sum`, `Average`, `Count`, `Minimum`, `Maximum`, `All`, `Any`, `Reduce`). + - **The inputs are the ones the activity's dialog asks for.** The operand is always a variable, because the dialog selects a variable. So nesting cannot be written, just as it cannot be drawn. + - **`by` picks a member** (the dialog's attribute or association selector). **`where` / `of` take an expression** (the "… by expression" variants). + - `describe` prints the same form. `@caption` appears only when the caption was customised. + + | Studio Pro activity: operation | MDL statement | + |---|---| + | List operation: Filter | `$Open = filter $Orders by Status = Shop.Status.Open;` | + | List operation: Filter by expression | `$Big = filter $Orders where $currentObject/Total > 1000;` | + | List operation: Find | `$Order = find $Orders by Number = $Number;` | + | List operation: Find by expression | `$Late = find $Orders where $currentObject/DueDate < [%CurrentDateTime%];` | + | List operation: Sort | `$Sorted = sort $Orders by Date desc, Number asc;` | + | List operation: Head / Tail | `$First = head $Orders;` / `$Rest = tail $Orders;` | + | List operation: Range | `$Page = range $Orders offset 20 limit 10;` | + | List operation: Union / Intersect / Subtract | `$All = union $A with $B;` / `$Both = intersect $A with $B;` / `$Left = subtract $B from $A;` | + | List operation: Contains / Equals | `$Has = contains $Order in $Orders;` / `$Same = equals $A and $B;` | + | Aggregate list: Count | `$N = count $Orders;` | + | Aggregate list: Sum / Average / Minimum / Maximum | `$Total = sum $Orders by Amount;` or, by expression, `$Total = sum $Orders of $currentObject/Price * $currentObject/Quantity;` | + | Aggregate list: All / Any | `$AllPaid = all $Orders where $currentObject/Paid;` / `$AnyLate = any $Orders where …;` | + | Aggregate list: Reduce | `$Csv = reduce $Orders from '' as String using …;` | + + Before the grammar is fixed, check the keywords and connecting words (`by`, `where`, `of`, `with`, `from`, `in`) against the labels Studio Pro's dialogs actually show for the Mendix versions mxcli supports. The operation set above comes from the metamodel. The label wording is not verified. - **`retrieve … limit 1`** — see §5. - **Aliases to retire:** - `$o/A = v` and `set $o/A = v` → `change $o (A = v)`; @@ -695,9 +719,9 @@ end while; -- begin and end while both optional ```mdl declare $x integer = 0; set $x = find($S, 'abc'); -- assignment always says `set`; find() is the string function -$y = find $Orders where Number = 42; -- a list operation is a statement, never a function call +$y = find $Orders by Number = 42; -- a list operation is a statement, never a function call -$Approved = filter $Orders where Status = Shop.Status.Approved; +$Approved = filter $Orders by Status = Shop.Status.Approved; $Count = count $Approved; -- count filter … cannot parse: the operand must be a variable $Total = sum $Approved by Amount; retrieve $First from Shop.Order where [Status = 'Open'] first; -- an object @@ -791,7 +815,7 @@ alter settings constant 'Shop.ApiUrl' value 'https://test.example.com' in config **After** — no arrows, no expressions in strings, `caption` says what it sets, and settings use property lists: ```mdl -create or replace workflow Shop.OrderApproval +create or modify workflow Shop.OrderApproval parameter $WorkflowContext: Shop.OrderContext display 'Order Approval' begin @@ -809,7 +833,7 @@ begin ; end workflow; -create or replace configuration 'Default' ( +create or modify configuration 'Default' ( DatabaseType: postgresql, HttpPortNumber: 8080, ); @@ -878,13 +902,13 @@ alter configuration 'Default' set constant Shop.ApiUrl = 'https://test.example.c These change the meaning of text that parses today, or they reject text that is accepted today. They must land before beta or never. -1. **Pick one name for the idempotent create (R1).** Only the view-entity path gives the two names different behaviour, so aliasing the loser is safe only once view entities also use the identity-carrying rewrite. +1. **`create or modify` becomes the one idempotent create, with minimal-change semantics (R1).** `or replace` becomes an alias. Only the view-entity path gives the two names different behaviour, so aliasing is safe only once view entities also use the identity-carrying rewrite. The no-op guarantee changes behaviour for every document type that is rewritten today when nothing changed. 2. **`retrieve … limit 1`.** - Today it binds an **object** (the Mendix "first" range), so to any SQL reader it means something different from what it does. - Import mappings already solved this with `first | limit n`, and their own grammar comment argues that a list of one and an object "cannot share syntax". - Adopt `retrieve $x from … where … first;` for an object, so that `limit 1` means a list of one. - Sequence: in the release before beta, warn on a bare `limit 1` and have `describe` emit `first`. At beta, flip the meaning. -3. **Make `set` mandatory, and turn list operations into keyword statements**, removing the `$x = find(...)` ambiguity. +3. **Make `set` mandatory, and turn list operations into statements that mirror Studio Pro's activities** (§4, Microflows), removing the `$x = find(...)` ambiguity. 4. **Unknown property keys become errors.** 5. **`;` becomes required**, and `/` is removed. 6. **`\` is no longer an escape character** in string literals. @@ -911,14 +935,23 @@ The language has no version marker today. `version-aware-mdl.md` proposes `set v See the implementation plan in §9, which supersedes the short list that was here. -## 7. Open decisions for the maintainer +## 7. Decisions + +### Decided (2026-09-26) + +1. **R1, the idempotent-create keyword: `create or modify`.** It reads like the intent of the command. `modify` changes only what differs: re-running an unchanged definition writes nothing, and Studio Pro shows no change. `create or replace` becomes a deprecated alias. +2. **`show` is dropped in favour of `list`** (R6). +3. **Microflow list operations must feel intuitive to developers who work with microflow diagrams in Studio Pro.** Each statement mirrors one activity: one **List operation** or **Aggregate list** activity per statement, named after the operation Studio Pro offers, and taking the inputs its dialog asks for. Because a diagram cannot nest one activity inside another, the statements cannot nest either (§4, Microflows). +4. **PedApp may be committed** as the Studio Pro-authored round-trip fixture (plan item 0.2). -1. **R1, the idempotent-create keyword.** `create or replace` (honest about full-definition semantics) or `create or modify` (what describe emits today)? Renames stay in `alter`/`rename` either way. -2. **R3/R4, argument binding.** Should it be `Param = expr` (proposed: `=` binds runtime values, `:` sets model properties) or `Param: expr`? The reviews split on this. `=` aligns calls with `change`, `set` and the existing microflow describe output. `:` aligns with today's page describe output. Pick one; do not keep both. -3. **Alter form.** Should it be `alter X set ( Key: value )` (proposed, the same list as create) or SQL `alter X set Key = value`? The latter is more SQL-like, but it makes the same key take two separators. -4. **List operations as keyword statements.** This is the largest microflow change. The alternative is to keep the function form, forbid nesting in the grammar, and rename the `find`/`contains` list operations so they no longer collide with the string functions. -5. **Retire `show` entirely?** Or keep it for non-element state, as proposed? -6. **The `mdl 1;` header.** Adopt it now, or rely on aliases plus `fmt --upgrade` until the first post-beta break? +### Still open + +1. **R3/R4, argument binding.** Should it be `Param = expr` (proposed: `=` binds runtime values, `:` sets model properties) or `Param: expr`? The reviews split on this. `=` aligns calls with `change`, `set` and the existing microflow describe output; `:` aligns with today's page describe output. Pick one; do not keep both. +2. **Alter form.** Should it be `alter X set ( Key: value )` (proposed, the same list as create) or SQL `alter X set Key = value`? The latter is more SQL-like, but it makes the same key take two separators. +3. **The `mdl 1;` header.** Adopt it now, or rely on aliases plus `fmt --upgrade` until the first post-beta break? +4. **Release cadence up to beta.** The `limit 1` flip and the required `;` each need one release of warnings, so beta has to be at least two releases away. +5. **Is brownfield agent editing a beta goal?** If it is, `alter microflow` (plan item 4.2) joins the beta gate. It is the largest single item in the plan. +6. **Where drift fingerprints are stored** (plan item 4.3): a local `.mxcli/state` file, or committed next to the scripts. This can wait until Phase 4. ## 8. Two ways of working: MDL-first and data-first @@ -966,7 +999,7 @@ Neither mode is better than the other in general. Each is the efficient one for |---|---|---| | Source of truth | the `.mdl` scripts | the stored model (`.mpr`) | | Typical use | new apps, new modules, generated scaffolding, documents mxcli owns | existing Studio Pro apps, documents people also edit in Studio Pro, marketplace modules | -| Main statement | `create or replace <type> X ( … ) { … }`, the whole definition (R1) | `alter <type> X { insert … / replace … / set … / drop … }` (§8.3) | +| Main statement | `create or modify <type> X ( … ) { … }`, the whole definition (R1) | `alter <type> X { insert … / replace … / set … / drop … }` (§8.3) | | What the author must know | only the intended end state; no read needed | the current state, well enough to address the target | | Token cost of a change | proportional to the **document** | proportional to the **change** | | Reviewability | the script *is* the design; a PR diff shows the new definition | the patch *is* the intent; a PR diff shows exactly what was changed and where | @@ -987,7 +1020,7 @@ Neither mode is better than the other in general. Each is the efficient one for That question can be answered mechanically, the way Terraform detects drift: - Record a fingerprint of each document's canonical BSON when mxcli writes it. -- `create or replace` compares the stored document against that fingerprint. +- `create or modify` compares the stored document against that fingerprint. - **Match:** the replace proceeds. - **Drift:** the replace is refused with "changed outside MDL since the last apply; use `alter`, or `describe` to re-adopt, or `--force`". @@ -1002,7 +1035,7 @@ This is another reason for R2's uniform node shape and R12's canonical describe. ### 8.3 The model is data: edits are tree patches -MDL describes stored data, so it can be manipulated the way Lisp manipulates code: the text is a structure, and edits are structural operations on it. Four consequences follow. +MDL describes stored data, so it can be manipulated the way Lisp manipulates code: the text is a structure, and edits are structural operations on it. Five consequences follow. 1. **One generic `alter` for every document type.** Rule R2 gives every child the shape `<kind> [Name] ( props ) { children }`. That rule is what makes a single patch grammar possible, replacing about 15 bespoke `alter` forms: @@ -1040,7 +1073,9 @@ MDL describes stored data, so it can be manipulated the way Lisp manipulates cod - Unstructured regions are addressed through the `join`/`merge` labels that `describe` already prints. - An inserted fragment is checked in the scope of its insertion point, for macro hygiene: a variable it declares must not collide with one declared further down. -4. **Bulk patches are "macros":** a query plus a patch. +4. **Declarative `create or modify` is a diff that produces a patch.** Given the decided R1 semantics (change only what differs), `create or modify` is implemented as: compare the declared definition with the stored document, derive the minimal patch, and apply it with the same splice engine `alter` uses. An empty patch means no write. The two modes of §8.2 therefore share one engine, and MDL-first scripts gain the same no-churn guarantee as data-first patches. This is the reconciliation model of Terraform's plan/apply. + +5. **Bulk patches are "macros":** a query plus a patch. ```mdl alter microflows in Shop where contains (commit $Order) { @@ -1067,7 +1102,7 @@ Silent loss is never acceptable. - New apps and modules: write declarative MDL. - Existing Studio Pro documents: change them with `alter`. - describe → replace only for documents whose stored state is still what MDL produced, and never on a Studio Pro-authored document of a type without a proven round trip. -2. **Drift detection** on `create or replace`: a per-document fingerprint recorded at write time, refusing the replace on drift (§8.2). Until it exists, the guidance in step 1 is the only guard. +2. **Drift detection** on `create or modify`: a per-document fingerprint recorded at write time, refusing the replace on drift (§8.2). Until it exists, the guidance in step 1 is the only guard. 3. **A CI round-trip test** on a Studio Pro fixture: describe → exec → canonical BSON compared per unit, covering every document type. It would have caught every loss in §8.1. Fix those losses, or turn them into refusals. 4. **`alter microflow` / `alter nanoflow`** with content addressing and graph splicing (§8.3). This is the single largest gap for brownfield work. 5. **A real dry run.** Execute on an in-memory copy and diff canonical BSON per unit. The machinery exists in `canon.Reconcile`. Until then, `diff` saying "no changes" is not evidence of no change. @@ -1106,7 +1141,7 @@ Phase 0 safety net + bugs ──┬──> Phase 1 decisions + deprecation mac | # | Item | Size | Where | Done when | |---|---|---|---|---| | 0.1 | **Round-trip harness.** For every document in a Studio Pro-authored fixture: `describe` → `exec` → canonical BSON compared per unit. Known failures go in an allowlist that may only shrink. | M | integration test (`-tags integration`); compare with `canon` | runs in CI; the allowlist equals the §8.1 losses | -| 0.2 | **Fixture.** A committed Studio Pro-authored project with microflows, nanoflows, pages, snippets, translations, table-stored associations and a workflow. PedApp covers most of these, but its licence must be checked; add a workflow and REST documents. Evora (`mx-test-projects/`) serves as a large, uncommitted read-side benchmark. | S | `testdata/` | fixture in the repo | +| 0.2 | **Fixture.** Commit PedApp, a Studio Pro-authored project (cleared for use, §7), which has microflows, nanoflows, pages, snippets, translations and table-stored associations. Add a workflow and REST documents in Studio Pro. Evora (`mx-test-projects/`) serves as a large, uncommitted read-side benchmark. | S | `testdata/` | fixture in the repo | | 0.3 | **Silent drops from §2:** `throw`; `float`/`currency`/`date`; the parenthesised association form; enumeration-value doc comments and index names; describe output that doesn't re-parse (#8); the plaintext password (#9); `/tmp` image paths (#10). | M | visitor, executor describe | each has a failing test first; §2 closed | | 0.4 | **Round-trip losses from §8.1:** association storage, page translations, nanoflow annotations and export level, Java action export level, snippet `Type`, describe emitting a plain `create`. Tracked as separate tasks. | M | executor create paths | removed from the 0.1 allowlist | | 0.5 | **Read-side wrong answers from §8.1:** `describe fragment`, `structure` counts, silent empty `search`, strict `--json`. Tracked as a task. | S | executor | tests | @@ -1116,7 +1151,7 @@ Phase 0 safety net + bugs ──┬──> Phase 1 decisions + deprecation mac | # | Item | Size | Where | Done when | |---|---|---|---|---| -| 1.1 | **Decide §7** and record the rules. Write an ADR ("MDL canonical syntax: R1–R12, two modes") that extends ADR-0003, and turn R1–R12 into checklist items in `design-mdl-syntax.md`. Mark the v1/v2 syntax proposals **rejected**; fold the workflow alignment proposal into R4/R9. | S | `docs/13-decisions/`, skill | ADR accepted | +| 1.1 | **Resolve the remaining open questions in §7** and record all decisions. Write an ADR ("MDL canonical syntax: R1–R12, two modes") that extends ADR-0003, and turn R1–R12 into checklist items in `design-mdl-syntax.md`. Mark the v1/v2 syntax proposals **rejected**; fold the workflow alignment proposal into R4/R9. | S | `docs/13-decisions/`, skill | ADR accepted | | 1.2 | **Deprecation registry.** Generalise the existing MDL065 pattern, where the AST records which spelling the source used (`ast_microflow.go:273`). Add a single table of `{code MDL-DEPRnnn, old form, canonical form, removed in}`. `check` and `exec` warn; a `--deprecations=error` flag fails instead. | M | `mdl/ast`, `mdl/linter`, new `deprecations.go` | a test fails if a grammar alternative marked as an alias has no registry entry | | 1.3 | **`mxcli fmt --upgrade`.** Each registry entry carries a rewrite, built on `mdl/formatter` / `cmd_fmt.go`. The output must parse with zero deprecation warnings and give an identical AST. | M | `mdl/formatter` | a property test over all of `mdl-examples/`: upgrade, then zero warnings, then AST equal | | 1.4 | **Canonical-form conformance gate.** Everything in `mxcli syntax`, the skills, `docs-site/` and `mdl-examples/` is parsed with `--deprecations=error`, starting with an allowlist. This stops the docs teaching old forms, a gap measured in §3. | S | `make lint` target | allowlist only shrinks | @@ -1129,10 +1164,10 @@ Each change breaks existing text, so each one lands with a registry entry, an `f | # | Item | Size | Warning period | Done when | |---|---|---|---|---| -| 2.1 | **R1: one idempotent-create keyword.** View entities get an identity-carrying rewrite instead of delete-and-recreate; `if not exists` works on every type. | M | alias the losing keyword (safe once view entities carry identity) | GUID preserved on a view-entity replace (a test with a GUID != `$ID` control) | +| 2.1 | **R1: `create or modify` is the keyword; `or replace` becomes an alias.** View entities get an identity-carrying rewrite instead of delete-and-recreate; `if not exists` works on every type. The no-op guarantee (an unchanged definition writes nothing) is enforced by the 0.1 harness for every type except microflows and nanoflows, which need 4.2. | M | `or replace` warns | GUID preserved on a view-entity replace (a test with a GUID != `$ID` control); re-running unchanged `describe` output writes no unit | | 2.2 | **Strictness:** unknown or mis-shaped property keys are errors (REST, agents and business events first, then everywhere); `;` required; `/` removed; `\` is no longer an escape. | M | `;` and `/` warn for one release; unknown keys and `\` break immediately | parse tests | | 2.3 | **Dead grammar removed:** `grant … on workflow`, `case … else`, `text`/`statictext`/`legacydatagrid`, `throw` (if not fixed in 0.3). | S | none (they never worked) | parse errors with a hint | -| 2.4 | **Microflow assignment and list operations.** `set` becomes mandatory. List operations and aggregates become keyword statements (`$A = filter $L where …`, `$n = count $A`), so nesting no longer parses. The `find`/`contains` ambiguity disappears; MDL-LISTOP02 becomes unreachable and is retired. | L | old function form warns (except `find`/`contains`, which can't) | `write-microflows` skill and examples migrated by `fmt --upgrade` | +| 2.4 | **Microflow assignment and list operations.** `set` becomes mandatory. List operations and aggregates become one statement per Studio Pro activity (`$A = filter $L by …`, `$B = filter $L where …`, `$n = count $A`; the full table is in §4), so nesting no longer parses. The `find`/`contains` ambiguity disappears; MDL-LISTOP02 becomes unreachable and is retired. The first PR checks the keywords against Studio Pro's dialog labels. | L | old function form warns (except `find`/`contains`, which can't) | `write-microflows` skill and examples migrated by `fmt --upgrade` | | 2.5 | **`retrieve … first` vs `limit 1`.** In release N, `describe` emits `first` and a bare `limit 1` warns. In release N+1, the beta, `limit 1` means a list. | M | one release | MDL-RETRIEVE01 retired | | 2.6 | **Canonical `describe` (R12), one area per PR:** omit derived layout (extend the `@start` derived-vs-authored rule to `@position`/`@curve`/`@anchor`); omit defaults; no synthetic widget names (make the name optional in `widgetV3`, address grid columns explicitly); fold `join`/`merge` back into `case` and fall-through handlers; emit `elsif`. | L | none (output change) | the harness in 0.1 stays green; PutGet holds per area | @@ -1167,14 +1202,15 @@ Order, by how many existing scripts each item touches: | # | Item | Size | Where | Done when | |---|---|---|---|---| | 4.1 | **Generic `alter <type> X { set / insert / replace / drop }`.** One grammar rule plus a per-doctype *target resolver* interface. `alter page`/`snippet`/`layout` (`pagemutator`) and `alter workflow` (`wfmutator`) are ported onto it first, with their old forms as aliases. | M | grammar, `mdl/backend` | existing alter tests pass through the new path | -| 4.2 | **`alter microflow` / `alter nanoflow`**, built as an `mfmutator` alongside `pagemutator` and `wfmutator`: | L | `mdl/backend/modelsdk`, `mdl/microflowgraph` | see acceptance below | +| 4.2 | **`alter microflow` / `alter nanoflow`**, built as an `mfmutator` alongside `pagemutator` and `wfmutator`. The same engine then implements `create or modify` for microflows (step g), which is what makes the R1 no-op guarantee hold for them: | L | `mdl/backend/modelsdk`, `mdl/microflowgraph` | see acceptance below | | | a. **Target resolver.** Address by output `$var`, by caption, or by statement pattern with `*` wildcards; `@n` disambiguates, and ambiguity is an error listing the matches. Add `describe … with handles` to show the addresses. | | | | | | b. **Graph splice** on the *stored* object collection. Rewire the incoming and outgoing sequence flows around the target; build only the fragment's objects; never call the whole-document `UpdateMicroflow` rebuild. The write goes through `canon.Reconcile` (CLAUDE.md rule 2). | | | | | | c. **Placement.** Put the new node on the flow's midpoint and shift downstream nodes; never overlap. | | | | | | d. **Hygiene.** Check the fragment in the scope of the insertion point; a variable collision is an error. | | | | | | e. **Operations,** in order: `insert after`/`before` → `replace` → `drop` → `set` (expression, caption, `on error`) → `add`/`drop parameter`. | | | | | | f. **Both backends:** the modelsdk engine and `--mcp` (the Studio Pro MCP backend), or an explicit "not supported by this backend" error. | | | | -| 4.3 | **Drift detection.** Every write path records a per-unit canonical-BSON fingerprint through `canon.Reconcile`. `create or replace` refuses on drift, with `--force` to override. Open question: where the fingerprints live (a sidecar `.mxcli/state` file vs committed with the scripts). | M | `modelsdk/canon`, executor | a Studio Pro edit between two applies is detected; a control with no edit is not | +| | g. **Declarative `create or modify` as diff-then-patch:** match the declared flow against the stored one (by statement signature and output variable, as in 4.2a), derive the minimal set of insert/replace/drop operations, and apply them with the splice from 4.2b. An unchanged definition yields an empty patch and no write. This replaces `UpdateMicroflow`'s whole-document rebuild. | | | | +| 4.3 | **Drift detection.** Every write path records a per-unit canonical-BSON fingerprint through `canon.Reconcile`. `create or modify` refuses on drift, with `--force` to override. Open question: where the fingerprints live (a sidecar `.mxcli/state` file vs committed with the scripts). | M | `modelsdk/canon`, executor | a Studio Pro edit between two applies is detected; a control with no edit is not | | 4.4 | **A real dry run:** `exec --dry-run`, replacing today's `diff`. Execute on an in-memory copy, diff canonical BSON per unit, and render changed units as a `describe` diff. Covers `alter` and every document type. | M | executor, `canon` | the §8.1 false negative (association storage) and false positives disappear | | 4.5 | **Opaque passthrough or refusal** for content MDL cannot express, per document type as the 0.1 harness finds it: `preserved <kind> '<id>'` in `describe` output, carried by replace. | M | per doctype | no silent loss remains in the harness | | 4.6 | **Bulk patches:** `alter microflows|pages in M where contains (<pattern>) { … }`, building on 4.2a. `alter pages … where` and `update widgets` are folded into it as aliases. | M | grammar, executor | — | @@ -1185,6 +1221,7 @@ Order, by how many existing scripts each item touches: - Control: an empty `alter` changes nothing. - `mx check` is unchanged from the baseline, and Studio Pro opens the project. - An MDL script of at most 5 lines replaces today's 107. +- Re-running the unchanged `describe` output as `create or modify` writes nothing (GetPut), with an edited-flow control that does write. ### Phase 5: read side (§8.1; any time) @@ -1202,5 +1239,4 @@ Order, by how many existing scripts each item touches: - **`describe` output changes (2.6, Phase 3) churn MDL that users have committed.** Mitigation: ship them together in as few releases as possible before beta, with `fmt --upgrade` and a changelog entry per change. - **Grammar changes ripple into generated artefacts:** LSP completions (`lsp_completions_gen.go`), `keyword_coverage_test.go`, the VS Code extension and the embedded skills. Each grammar PR runs `make build`, which regenerates them, and `make sync-skills`. - **The graph splice (4.2b) is the riskiest code.** It must never rewrite an `$ID` without rewriting every reference to it (CLAUDE.md rule 1), and its tests must use a Studio Pro-authored flow, because an mxcli-created flow cannot show identity loss (GUID == `$ID`). -- **The harness fixture's licence.** If PedApp can't be committed, build a fixture in Studio Pro specifically for this purpose. - **Scope creep in Phase 3.** Hold each rule to its recipe; anything beyond renaming belongs in its own proposal. From 9e3b2957df5b0a5be65ef126aff4e4fd94d4b6a1 Mon Sep 17 00:00:00 2001 From: Ako <andrej@koelewijn.net> Date: Sat, 26 Sep 2026 13:41:35 +0000 Subject: [PATCH 06/15] docs: separate element describe from definition describe in MDL proposal Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --- .../PROPOSAL_mdl_beta_syntax_freeze.md | 20 +++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-) diff --git a/docs/11-proposals/PROPOSAL_mdl_beta_syntax_freeze.md b/docs/11-proposals/PROPOSAL_mdl_beta_syntax_freeze.md index 4f539d4eb..23d3e3789 100644 --- a/docs/11-proposals/PROPOSAL_mdl_beta_syntax_freeze.md +++ b/docs/11-proposals/PROPOSAL_mdl_beta_syntax_freeze.md @@ -340,6 +340,22 @@ textbox txtNote (Attribute: Note, Visible: $currentObject/Status = 'Open') | ~~`show`~~ | **dropped** (decided, §7) | Every one of the 88 `show` forms maps to `list`, to `describe`, or to a session command:<br>• Plurals and relationship queries become `list`: `list callers of X`, `list callees of X`, `list references to X`, `list impact of X`, `list access on E`, `list widgets …`, `list design properties …`, `list catalog tables`.<br>• Single things become `describe`: `describe entity X`, `describe navigation`, `describe app security`, `describe security matrix`, `describe structure [in M]`, `describe context of X`.<br>• Session state (`version`, `catalog status`) becomes a REPL command (R7).<br>`show` stays as a deprecated alias for one release. | | `create` / `alter` / `drop` | as in SQL | `alter` children use `add`/`drop`. Replace `remove` (user role), `modify`/`add or modify` (settings), `define fragment`, `update security` and `update widgets`. Add the missing `drop`s: database connection, validation rule, external entity. | +**`describe` does three jobs; keep them apart.** Every document type that has a `create` also has a `describe`. Validation rules, indexes and access rules come out as part of `describe entity`, and `describe Module.Name` detects the type itself. The only gaps, app security and the security matrix, are covered by the `show` mapping above. But the verb currently mixes three kinds of answer: + +| Kind | Examples today | Output | Rule | +|---|---|---|---| +| **Model element** | `describe entity`, `describe page`, `describe microflow`, … | runnable MDL | round-trips (R12, §8.4) | +| **Part of a model element** | `describe fragment from page … widget`, `describe styling on page …`, `describe translations for nl_NL` | runnable MDL or a report | the same round-trip rules when the output is MDL | +| **Definition or lookup** | `describe widget combobox`, `describe glyph 'star'`, `describe catalog.entities`, `describe contract entity …`, `describe contract operation from openapi '…'` | a report, not MDL | starts with a `-- <kind> definition (not executable)` header; `--json` gives a documented shape | + +How to tell `describe` and `syntax` apart: **does the answer depend on the project?** +- `describe widget combobox` reads the widget package installed in *this* project. On PedApp that is Combo box 2.5.0 with 58 properties; another project with another version gets other properties. It therefore stays under `describe`, and does not move to `syntax`. +- `syntax` is built into the binary, the same for every project, and answers "how do I write X in MDL?". +- The two point at each other. For example, `syntax page widgets` tells the reader to run `describe widget type <name> -p app.mpr` for the properties a given project actually has. +- `describe catalog.<table>` stays too: it is SQL's `DESCRIBE TABLE`. + +**Rename:** `describe widget <name>` becomes **`describe widget type <name>`**. As it stands, it collides with describing a widget *instance* on a page (`describe fragment … widget`, and the proposed `describe page X widget Y`, §8.1). "Which kind of widget?" and "which widget on this page?" must not share a phrase. The old form stays as a deprecated alias. + Also: - `column` survives as a synonym for `attribute` in four `alter entity` actions. Remove it. - `rest call get …` breaks the `call <kind>` pattern. Use `call rest service …`, which is also Studio Pro's name for the activity, and fold `send rest request` into it where the semantics allow. @@ -546,7 +562,7 @@ create consumed rest service Shop.Api3 ( ### R12. `describe` emits the canonical form and nothing else -`describe` output is what reviewers read in PRs, so it defines the language in practice. +`describe` output is what reviewers read in PRs, so it defines the language in practice. This rule covers describing model elements. Definitions and lookups (R6) are reports, not MDL, and are marked as such. - Emit only canonical forms, which makes `describe` the reference implementation of these rules. - Omit defaults, because default noise inflates the output: @@ -1193,7 +1209,7 @@ Order, by how many existing scripts each item touches: | 3.2 | R8: page actions as words (`show page`, `save changes`); one spelling per keyword; `error message`; lowercase canonical (fixes `fmt` upper-casing keys) | M | | 3.3 | R5: XPath in `[ ]`, bare expressions (grants, workflow targeting and decisions, `Visible`/`Editable`); one constant reference | M | | 3.4 | R2: brackets (REST client, agents, image collection, database connection, message definitions, navigation/menus, `on error begin … end error`, `while`) | L | -| 3.5 | R6/R7: `list`/`describe`/`show` split; `drop` instead of `remove`; missing `drop`s; `call rest service` | M | +| 3.5 | R6/R7: drop `show` in favour of `list`/`describe`; `describe widget type` rename, and a not-executable header plus `--json` for definition reports; cross-links between `syntax` and `describe`; `drop` instead of `remove`; missing `drop`s; `call rest service` | M | | 3.6 | R9/R10: metadata placement (doc comment, `folder` clause); Studio Pro document names (`consumed rest service`, …); property lists for role, constant, demo user and settings headers | M | | 3.7 | §4 area items: domain-model aliases and inline validation rules; widget types resolved through the registry and the property-key convention; unified mapping grammar; HTTP concepts; workflow `caption`/`->`/`alter workflow` vocabulary | L | From b27a0dce5609675f74f54a103196db4cb362d3cb Mon Sep 17 00:00:00 2001 From: Ako <andrej@koelewijn.net> Date: Sat, 26 Sep 2026 13:44:32 +0000 Subject: [PATCH 07/15] docs: record argument, alter, beta-scope and header decisions in MDL proposal Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --- .../PROPOSAL_mdl_beta_syntax_freeze.md | 21 ++++++++++--------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/docs/11-proposals/PROPOSAL_mdl_beta_syntax_freeze.md b/docs/11-proposals/PROPOSAL_mdl_beta_syntax_freeze.md index 23d3e3789..86150343f 100644 --- a/docs/11-proposals/PROPOSAL_mdl_beta_syntax_freeze.md +++ b/docs/11-proposals/PROPOSAL_mdl_beta_syntax_freeze.md @@ -81,7 +81,7 @@ Each rule is stated so it can be added to the `design-mdl-syntax` skill as a che **About the examples.** - Every **Before** block was run through `mxcli check` at `5dc51ceb` and parses today. The `describe` output in R12 is real output from a scratch project. -- Every **After** block is *proposed* syntax and does not parse yet. They follow the decisions recorded in §7. Where §7 still leaves a choice open, they use the recommended option: `Param = expr` for arguments, and `set ( Key: value )` for alter. +- Every **After** block is *proposed* syntax and does not parse yet. They follow the decisions recorded in §7. ### R1. One idempotent create, with one defined meaning @@ -960,14 +960,15 @@ See the implementation plan in §9, which supersedes the short list that was her 3. **Microflow list operations must feel intuitive to developers who work with microflow diagrams in Studio Pro.** Each statement mirrors one activity: one **List operation** or **Aggregate list** activity per statement, named after the operation Studio Pro offers, and taking the inputs its dialog asks for. Because a diagram cannot nest one activity inside another, the statements cannot nest either (§4, Microflows). 4. **PedApp may be committed** as the Studio Pro-authored round-trip fixture (plan item 0.2). +5. **R3/R4, argument binding: `Param = expr`.** `=` binds a runtime value, `:` sets a model property. There is no `$` on the parameter name. One form for every call site: microflow calls, `show page`, button and menu actions, workflows and REST. +6. **Alter form: `set ( Key: value, … )`.** It is the same property list as `create`, so a fragment from `describe` can be pasted straight into an `alter`. +7. **Brownfield agent editing is a beta goal.** `alter microflow`/`alter nanoflow` (plan item 4.2, at least insert, replace and drop) and the no-op `create or modify` for microflows join the beta gate. +8. **The `mdl 1;` language header is added now** (plan item 1.5). It is an optional first statement that `describe` and `fmt` emit; after beta, removed aliases are gated by version. + ### Still open -1. **R3/R4, argument binding.** Should it be `Param = expr` (proposed: `=` binds runtime values, `:` sets model properties) or `Param: expr`? The reviews split on this. `=` aligns calls with `change`, `set` and the existing microflow describe output; `:` aligns with today's page describe output. Pick one; do not keep both. -2. **Alter form.** Should it be `alter X set ( Key: value )` (proposed, the same list as create) or SQL `alter X set Key = value`? The latter is more SQL-like, but it makes the same key take two separators. -3. **The `mdl 1;` header.** Adopt it now, or rely on aliases plus `fmt --upgrade` until the first post-beta break? -4. **Release cadence up to beta.** The `limit 1` flip and the required `;` each need one release of warnings, so beta has to be at least two releases away. -5. **Is brownfield agent editing a beta goal?** If it is, `alter microflow` (plan item 4.2) joins the beta gate. It is the largest single item in the plan. -6. **Where drift fingerprints are stored** (plan item 4.3): a local `.mxcli/state` file, or committed next to the scripts. This can wait until Phase 4. +1. **Release cadence up to beta.** The `limit 1` flip and the required `;` each need one release of warnings, so beta has to be at least two releases away. The date of the beta decides how Phases 2 and 4.2 are scheduled. +2. **Where drift fingerprints are stored** (plan item 4.3): a local `.mxcli/state` file, or committed next to the scripts. This can wait until Phase 4. ## 8. Two ways of working: MDL-first and data-first @@ -1136,7 +1137,7 @@ The plan has six phases: - Phases 0 and 1 are foundations: a safety net, and the machinery that makes syntax changes cheap. - Phase 2 is the **beta gate**: the changes that cannot be bridged by an alias. - Phases 3–5 can continue past beta, because every change in them keeps the old form parsing. -- Phase 4 (data-first editing) depends only on phases 0 and 1, so it can run in parallel with phases 2 and 3. +- Phase 4 (data-first editing) depends only on phases 0 and 1, so it can run in parallel with phases 2 and 3. Items 4.1 and 4.2 are on the beta gate, because brownfield editing is a beta goal (§7). Start them early: they are the longest items in the plan. Every item follows the repo's working rules (CLAUDE.md): - One concern per PR. @@ -1171,7 +1172,7 @@ Phase 0 safety net + bugs ──┬──> Phase 1 decisions + deprecation mac | 1.2 | **Deprecation registry.** Generalise the existing MDL065 pattern, where the AST records which spelling the source used (`ast_microflow.go:273`). Add a single table of `{code MDL-DEPRnnn, old form, canonical form, removed in}`. `check` and `exec` warn; a `--deprecations=error` flag fails instead. | M | `mdl/ast`, `mdl/linter`, new `deprecations.go` | a test fails if a grammar alternative marked as an alias has no registry entry | | 1.3 | **`mxcli fmt --upgrade`.** Each registry entry carries a rewrite, built on `mdl/formatter` / `cmd_fmt.go`. The output must parse with zero deprecation warnings and give an identical AST. | M | `mdl/formatter` | a property test over all of `mdl-examples/`: upgrade, then zero warnings, then AST equal | | 1.4 | **Canonical-form conformance gate.** Everything in `mxcli syntax`, the skills, `docs-site/` and `mdl-examples/` is parsed with `--deprecations=error`, starting with an allowlist. This stops the docs teaching old forms, a gap measured in §3. | S | `make lint` target | allowlist only shrinks | -| 1.5 | **Language header `mdl 1;`** (if decided). It is parsed and emitted by `describe`/`fmt`, and gates alias removal after beta. | S | grammar, visitor | round-trips | +| 1.5 | **Language header `mdl 1;`** (decided, §7). It is parsed and emitted by `describe`/`fmt`, and gates alias removal after beta. | S | grammar, visitor | round-trips | | 1.6 | **Grammar hygiene** that makes later phases cheaper: one shared trailing-comma list rule; ban bare `IDENTIFIER` in parser rules with a test modelled on `keyword_coverage_test.go`; move session commands out of `utilityStatement` into the REPL (R7). | M | `mdl/grammar` | tests; `make grammar` clean | ### Phase 2: the beta gate (§5, cannot be aliased) @@ -1190,7 +1191,7 @@ Each change breaks existing text, so each one lands with a registry entry, an `f **Beta gate:** - Phases 0, 1 and 2 are complete. - Phase 3's canonical forms are *decided and in the grammar*. Aliases may remain. -- Phase 4.2 has at least insert, replace and drop, if brownfield agent use is a beta goal. +- Phase 4.2 has at least insert, replace and drop, and `create or modify` on a microflow is a no-op when nothing differs (decided, §7: brownfield agent editing is a beta goal). Phase 4.1 (the generic `alter`) comes before it. ### Phase 3: consistency through aliases (R2–R10, §4) From df83781aaffbb5fd9938f58446794c85181872f1 Mon Sep 17 00:00:00 2001 From: Ako <andrej@koelewijn.net> Date: Sat, 26 Sep 2026 13:48:09 +0000 Subject: [PATCH 08/15] docs: add weekly release schedule to MDL beta proposal Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --- .../PROPOSAL_mdl_beta_syntax_freeze.md | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-) diff --git a/docs/11-proposals/PROPOSAL_mdl_beta_syntax_freeze.md b/docs/11-proposals/PROPOSAL_mdl_beta_syntax_freeze.md index 86150343f..a22736865 100644 --- a/docs/11-proposals/PROPOSAL_mdl_beta_syntax_freeze.md +++ b/docs/11-proposals/PROPOSAL_mdl_beta_syntax_freeze.md @@ -965,10 +965,11 @@ See the implementation plan in §9, which supersedes the short list that was her 7. **Brownfield agent editing is a beta goal.** `alter microflow`/`alter nanoflow` (plan item 4.2, at least insert, replace and drop) and the no-op `create or modify` for microflows join the beta gate. 8. **The `mdl 1;` language header is added now** (plan item 1.5). It is an optional first statement that `describe` and `fmt` emit; after beta, removed aliases are gated by version. +9. **Cadence: one release per week, beta in about four weeks** (around 2026-10-24). The schedule is in §9. + ### Still open -1. **Release cadence up to beta.** The `limit 1` flip and the required `;` each need one release of warnings, so beta has to be at least two releases away. The date of the beta decides how Phases 2 and 4.2 are scheduled. -2. **Where drift fingerprints are stored** (plan item 4.3): a local `.mxcli/state` file, or committed next to the scripts. This can wait until Phase 4. +1. **Where drift fingerprints are stored** (plan item 4.3): a local `.mxcli/state` file, or committed next to the scripts. This can wait until Phase 4. ## 8. Two ways of working: MDL-first and data-first @@ -1153,6 +1154,19 @@ Phase 0 safety net + bugs ──┬──> Phase 1 decisions + deprecation mac Phase 1 ──> Phase 4 data-first editing (alter microflow, drift, dry run) ``` +### Schedule (weekly releases, beta in about four weeks) + +Every break that has a warning period must ship its warning **at least one release before beta**. So all §5 warnings go out in week 2 at the latest, and the deprecation registry (1.2) has to land in week 1. + +| Release | Lands | Parallel track (Phase 4) | +|---|---|---| +| **Week 1** | ADR (1.1); deprecation registry (1.2); `mdl 1;` (1.5); round-trip harness and PedApp fixture (0.1, 0.2); first bug fixes (0.3–0.5); interim skill guidance (0.6) | generic `alter` skeleton (4.1); `mfmutator` target resolver (4.2a) | +| **Week 2** | **All §5 warnings ship:** `or replace` alias, bare `limit 1`, missing `;` and `/`, the old list-operation function form, optional `set`. New forms parse alongside the old ones (2.1, 2.4, 2.5). `fmt --upgrade` (1.3). | graph splice and placement (4.2b, 4.2c) | +| **Week 3** | Canonical `describe` (2.6). Phase 3 canonical forms added to the grammar, with the old forms as aliases. Conformance gate (1.4). Dead grammar removed (2.3). | `insert`, `replace`, `drop`; diff-then-patch `create or modify` for microflows (4.2e, 4.2g) | +| **Week 4: beta** | Breaks take effect: `limit 1` means a list; `;` required; unknown keys and `\` escapes are errors (2.2, 2.5) | 4.2 acceptance test on `VAL_Feedback` passes | + +**Risk.** Four weeks is tight for 4.2, which is the largest item and the riskiest code (the graph splice). If its acceptance test is not green in week 4, choose explicitly between slipping beta by a week or two, and shipping beta with `alter microflow` marked experimental while the microflow no-op guarantee stays on the gate. Phase 3's aliases and Phase 5 continue after beta; only the canonical *forms* have to be in the grammar by then. + ### Phase 0: safety net and bugs (no syntax change) | # | Item | Size | Where | Done when | From cd88e580d6fcd3b9fbdb2f8eed2105bf31db35c3 Mon Sep 17 00:00:00 2001 From: Ako <andrej@koelewijn.net> Date: Sat, 26 Sep 2026 14:03:11 +0000 Subject: [PATCH 09/15] docs: drift detection as optional optimistic locking via @base annotation Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --- .../PROPOSAL_mdl_beta_syntax_freeze.md | 63 ++++++++++++++++--- 1 file changed, 54 insertions(+), 9 deletions(-) diff --git a/docs/11-proposals/PROPOSAL_mdl_beta_syntax_freeze.md b/docs/11-proposals/PROPOSAL_mdl_beta_syntax_freeze.md index a22736865..659d0b982 100644 --- a/docs/11-proposals/PROPOSAL_mdl_beta_syntax_freeze.md +++ b/docs/11-proposals/PROPOSAL_mdl_beta_syntax_freeze.md @@ -967,9 +967,11 @@ See the implementation plan in §9, which supersedes the short list that was her 9. **Cadence: one release per week, beta in about four weeks** (around 2026-10-24). The schedule is in §9. +10. **Drift detection is optimistic locking, and is optional** (§8.2). A `@base '<fingerprint>'` annotation on the statement is emitted by `describe`, checked by `create or modify`, and updated by `exec`. Projects driven entirely by MDL skip it and instead require a dry run of all scripts to report no changes. There is no sidecar state file. + ### Still open -1. **Where drift fingerprints are stored** (plan item 4.3): a local `.mxcli/state` file, or committed next to the scripts. This can wait until Phase 4. +None. All decisions needed to start are recorded above. ## 8. Two ways of working: MDL-first and data-first @@ -1036,11 +1038,54 @@ Neither mode is better than the other in general. Each is the efficient one for - **Yes:** declarative replace is safe and cheapest. - **No** (someone edited it in Studio Pro, or it was never MDL): patch, or re-adopt it as MDL source first by running `describe`. Re-adopting is safe only for types whose round trip is proven. -That question can be answered mechanically, the way Terraform detects drift: -- Record a fingerprint of each document's canonical BSON when mxcli writes it. -- `create or modify` compares the stored document against that fingerprint. -- **Match:** the replace proceeds. -- **Drift:** the replace is refused with "changed outside MDL since the last apply; use `alter`, or `describe` to re-adopt, or `--force`". +That question can be answered mechanically, with **optimistic locking**: the same scheme as HTTP's `If-Match: <etag>`. There are two forms, one per kind of project. + +**1. `@base`: a per-statement version stamp, for mixed projects.** + +`describe` emits an annotation carrying the fingerprint (a hash of the canonical BSON) of the stored document it read: + +```mdl +@base 'sha256:3f9a…' +create or modify persistent entity Shop.Order ( + Number: integer, + Note: string(200), +); +``` + +- `create or modify` checks the annotation against the stored document and writes only if the two still match. Then `exec` rewrites the `@base` lines of the file it applied, so the next edit starts from the new version. It only updates annotations that are already there. +- The describe → edit → modify loop an agent uses on an existing app is therefore locked end to end, with no state file. The version travels with the statement through git, copy-paste and review. That keeps statements self-contained (ADR-0003). +- It is an **annotation, not a comment**. Formatters and LLMs strip or copy comments freely; an annotation has defined meaning and is validated. A statement copied to create another document carries a `@base` that does not match, and fails safely. +- The diff noise is small. The fingerprint changes only when the document changes, which happens because its statement was edited, so both land in the same hunk. When two developers apply different versions of one statement, the result is a git conflict on that line: a real conflict, shown where conflicts are already resolved. + +| Statement | Stored document | Result | +|---|---|---| +| no `@base` | missing | create | +| no `@base` | exists | modify to match (a no-op if nothing differs), exactly as without locking | +| `@base` matches | exists | modify, and `exec` updates `@base` | +| `@base` differs | exists | refused: "changed since you read it"; use `alter`, re-`describe`, or `--force` | + +**`@base` is optional.** A statement without it behaves exactly as it does today. + +**2. A dry run of all scripts must report no changes: for projects driven entirely by MDL.** + +Nothing but the scripts writes to such a project, so no per-statement lock is needed. Drift still happens in three ways: +- someone opens the app in Studio Pro (for example, a Mendix version upgrade rewrites documents); +- a marketplace module is updated; +- a one-off `alter` runs against the model and is never added to the scripts. The next apply silently undoes it, which makes this the main risk. + +Because `create or modify` writes nothing when nothing differs (R1), **any change reported by `mxcli exec --dry-run scripts/` is drift by definition**. Run in CI, that one check protects the whole project with no annotations. The rule for such projects: change the scripts, not the model. + +| Project style | Protection | +|---|---| +| Entirely MDL-driven | no `@base`; CI requires a dry run of all scripts to report no changes | +| Mixed (Studio Pro users, or agents patching existing documents) | `@base` on the statements `describe` produced | + +The two can be combined, for example scripts owning some modules and Studio Pro users owning others. + +**After beta.** +- **Finer-grained locking.** A whole-document check refuses non-conflicting edits: someone else added `Discount` while you changed `Note`. A per-element check refuses only when an element this statement would change or drop was changed by someone else. `alter` is already this fine-grained: a patch needs only its target to exist and be unambiguous. +- **Three-way merge.** A hash only detects a conflict. A merge also needs the base content, and git already has it: the previous version of the same statement. With base, theirs and yours, non-conflicting edits merge automatically, and real conflicts are reported per element. +- **Check and write together.** With `--mcp` the project is open in Studio Pro, so the backend must compare and write in one step, or there is a window for a lost update. This makes the choice between modes visible and safe, instead of something the agent has to remember. @@ -1120,7 +1165,7 @@ Silent loss is never acceptable. - New apps and modules: write declarative MDL. - Existing Studio Pro documents: change them with `alter`. - describe → replace only for documents whose stored state is still what MDL produced, and never on a Studio Pro-authored document of a type without a proven round trip. -2. **Drift detection** on `create or modify`: a per-document fingerprint recorded at write time, refusing the replace on drift (§8.2). Until it exists, the guidance in step 1 is the only guard. +2. **Drift detection** (§8.2): the optional `@base` annotation for mixed projects, and a no-changes dry run in CI for projects driven entirely by MDL. Until then, the guidance in step 1 is the only guard. 3. **A CI round-trip test** on a Studio Pro fixture: describe → exec → canonical BSON compared per unit, covering every document type. It would have caught every loss in §8.1. Fix those losses, or turn them into refusals. 4. **`alter microflow` / `alter nanoflow`** with content addressing and graph splicing (§8.3). This is the single largest gap for brownfield work. 5. **A real dry run.** Execute on an in-memory copy and diff canonical BSON per unit. The machinery exists in `canon.Reconcile`. Until then, `diff` saying "no changes" is not evidence of no change. @@ -1241,8 +1286,8 @@ Order, by how many existing scripts each item touches: | | e. **Operations,** in order: `insert after`/`before` → `replace` → `drop` → `set` (expression, caption, `on error`) → `add`/`drop parameter`. | | | | | | f. **Both backends:** the modelsdk engine and `--mcp` (the Studio Pro MCP backend), or an explicit "not supported by this backend" error. | | | | | | g. **Declarative `create or modify` as diff-then-patch:** match the declared flow against the stored one (by statement signature and output variable, as in 4.2a), derive the minimal set of insert/replace/drop operations, and apply them with the splice from 4.2b. An unchanged definition yields an empty patch and no write. This replaces `UpdateMicroflow`'s whole-document rebuild. | | | | -| 4.3 | **Drift detection.** Every write path records a per-unit canonical-BSON fingerprint through `canon.Reconcile`. `create or modify` refuses on drift, with `--force` to override. Open question: where the fingerprints live (a sidecar `.mxcli/state` file vs committed with the scripts). | M | `modelsdk/canon`, executor | a Studio Pro edit between two applies is detected; a control with no edit is not | -| 4.4 | **A real dry run:** `exec --dry-run`, replacing today's `diff`. Execute on an in-memory copy, diff canonical BSON per unit, and render changed units as a `describe` diff. Covers `alter` and every document type. | M | executor, `canon` | the §8.1 false negative (association storage) and false positives disappear | +| 4.3 | **Drift detection as optimistic locking.** `describe` emits `@base '<fingerprint>'` (canonical BSON hash per unit, computed with `canon`). `create or modify` refuses when a present `@base` does not match, with `--force` to override. `exec` rewrites existing `@base` lines after applying. The check and the write are atomic on both backends (modelsdk and `--mcp`). A statement without `@base` behaves exactly as before. | M | grammar (annotation), `modelsdk/canon`, executor | a Studio Pro edit between `describe` and `modify` is refused; a control with no edit applies; `exec` updates the stamp | +| 4.4 | **A real dry run** (also the drift check for projects driven entirely by MDL, §8.2): `exec --dry-run`, replacing today's `diff`. Execute on an in-memory copy, diff canonical BSON per unit, and render changed units as a `describe` diff. Covers `alter` and every document type. | M | executor, `canon` | the §8.1 false negative (association storage) and false positives disappear | | 4.5 | **Opaque passthrough or refusal** for content MDL cannot express, per document type as the 0.1 harness finds it: `preserved <kind> '<id>'` in `describe` output, carried by replace. | M | per doctype | no silent loss remains in the harness | | 4.6 | **Bulk patches:** `alter microflows|pages in M where contains (<pattern>) { … }`, building on 4.2a. `alter pages … where` and `update widgets` are folded into it as aliases. | M | grammar, executor | — | From caa6d4a6f5691bcc33a36152b74888e3b828d5f5 Mon Sep 17 00:00:00 2001 From: Ako <andrej@koelewijn.net> Date: Sat, 26 Sep 2026 14:12:32 +0000 Subject: [PATCH 10/15] docs: tie MDL meaning changes to the mdl 1 header; add prior-art section Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --- .../PROPOSAL_mdl_beta_syntax_freeze.md | 49 +++++++++++++++---- 1 file changed, 40 insertions(+), 9 deletions(-) diff --git a/docs/11-proposals/PROPOSAL_mdl_beta_syntax_freeze.md b/docs/11-proposals/PROPOSAL_mdl_beta_syntax_freeze.md index 659d0b982..2085d7538 100644 --- a/docs/11-proposals/PROPOSAL_mdl_beta_syntax_freeze.md +++ b/docs/11-proposals/PROPOSAL_mdl_beta_syntax_freeze.md @@ -918,12 +918,16 @@ alter configuration 'Default' set constant Shop.ApiUrl = 'https://test.example.c These change the meaning of text that parses today, or they reject text that is accepted today. They must land before beta or never. +**Changes of meaning and new rejections are tied to the language header, as Rust editions do** (§6, §10). A script that starts with `mdl 1;` gets the new meaning. A script without the header keeps the alpha meaning and gets a warning on every construct whose meaning differs. `fmt --upgrade` adds the header and rewrites those constructs. This applies to items 2–6 below. Existing scripts therefore never change meaning silently, and no user has to notice a one-week warning. + +Items 7–8 (removing forms that never worked or mis-store) and item 9 (describe output) do not depend on the header. Item 1 is an alias, so it does not either. + 1. **`create or modify` becomes the one idempotent create, with minimal-change semantics (R1).** `or replace` becomes an alias. Only the view-entity path gives the two names different behaviour, so aliasing is safe only once view entities also use the identity-carrying rewrite. The no-op guarantee changes behaviour for every document type that is rewritten today when nothing changed. 2. **`retrieve … limit 1`.** - Today it binds an **object** (the Mendix "first" range), so to any SQL reader it means something different from what it does. - Import mappings already solved this with `first | limit n`, and their own grammar comment argues that a list of one and an object "cannot share syntax". - Adopt `retrieve $x from … where … first;` for an object, so that `limit 1` means a list of one. - - Sequence: in the release before beta, warn on a bare `limit 1` and have `describe` emit `first`. At beta, flip the meaning. + - Tied to the header. Under `mdl 1;`, `limit 1` is a list of one. Without the header, it keeps its old meaning (an object) and warns. `describe` always emits `first` or `limit n` together with the header, so its output is never ambiguous. 3. **Make `set` mandatory, and turn list operations into statements that mirror Studio Pro's activities** (§4, Microflows), removing the `$x = find(...)` ambiguity. 4. **Unknown property keys become errors.** 5. **`;` becomes required**, and `/` is removed. @@ -932,7 +936,7 @@ These change the meaning of text that parses today, or they reject text that is 8. **Remove `float`, `currency` and `date`** as attribute types. They are accepted and mis-stored today. 9. **Omit synthetic widget names and derived layout** from describe. This changes describe output, which people have already committed to repos. -Everything else in this document can land after an alias period. +Everything else in this document can land after an alias period. Prior art for the whole proposal, and where it is new, is in §10. ## 6. Migration mechanism @@ -940,9 +944,10 @@ The language has no version marker today. `version-aware-mdl.md` proposes `set v 1. **Deprecated aliases keep parsing** and emit a warning with a stable code (`MDL-DEPR001`, …) that names the canonical form. `describe` never emits a deprecated form. 2. **`mxcli fmt --upgrade`** rewrites every deprecated form to its canonical form. This is mechanical for every alias in §3–§4, and it is what makes consolidation cheap for users. -3. **An optional language header**, `mdl 1;`, as the first statement. +3. **An optional language header**, `mdl 1;`, as the first statement (decided, §7). - `describe` and `fmt` emit it. - - With no header, a script gets the latest language version. + - **With no header, a script gets the alpha meaning (`mdl 0`) plus warnings**, never the latest. This matches the precedents: Rust treats a crate with no edition as the oldest edition, and Go treats a module with no `go` line as the oldest version. The meaning of a script never depends on which mxcli release happens to run it. + - Under `mdl 1`, the §5 changes of meaning and new rejections apply. Under `mdl 0` they are warnings. - After beta, a change that is not backwards compatible bumps the number, and the visitor gates removed aliases on it: under `mdl 1` they warn, under `mdl 2` they are refused. - It is independent of the Mendix target version. 4. **Skills, `mxcli syntax` and the quick reference** are regenerated from, or checked against, describe output. They should not be hand-maintained. The skill examples already disagree with the grammar in several places, such as `DELETE_BEHAVIOR PREVENT` and `rename … as`. @@ -1201,14 +1206,14 @@ Phase 0 safety net + bugs ──┬──> Phase 1 decisions + deprecation mac ### Schedule (weekly releases, beta in about four weeks) -Every break that has a warning period must ship its warning **at least one release before beta**. So all §5 warnings go out in week 2 at the latest, and the deprecation registry (1.2) has to land in week 1. +Changes of meaning are tied to the `mdl 1;` header (§5), so no existing script breaks on a given date. The schedule pressure is about having `mdl 1` complete, documented and emitted by beta. The deprecation registry (1.2) and the header (1.5) still land in week 1, because everything in week 2 builds on them. | Release | Lands | Parallel track (Phase 4) | |---|---|---| | **Week 1** | ADR (1.1); deprecation registry (1.2); `mdl 1;` (1.5); round-trip harness and PedApp fixture (0.1, 0.2); first bug fixes (0.3–0.5); interim skill guidance (0.6) | generic `alter` skeleton (4.1); `mfmutator` target resolver (4.2a) | -| **Week 2** | **All §5 warnings ship:** `or replace` alias, bare `limit 1`, missing `;` and `/`, the old list-operation function form, optional `set`. New forms parse alongside the old ones (2.1, 2.4, 2.5). `fmt --upgrade` (1.3). | graph splice and placement (4.2b, 4.2c) | +| **Week 2** | **The `mdl 1` semantics ship.** New forms parse alongside the old ones (2.1, 2.4, 2.5). Scripts without the header warn on every construct whose meaning differs. `fmt --upgrade` adds the header and rewrites (1.3). | graph splice and placement (4.2b, 4.2c) | | **Week 3** | Canonical `describe` (2.6). Phase 3 canonical forms added to the grammar, with the old forms as aliases. Conformance gate (1.4). Dead grammar removed (2.3). | `insert`, `replace`, `drop`; diff-then-patch `create or modify` for microflows (4.2e, 4.2g) | -| **Week 4: beta** | Breaks take effect: `limit 1` means a list; `;` required; unknown keys and `\` escapes are errors (2.2, 2.5) | 4.2 acceptance test on `VAL_Feedback` passes | +| **Week 4: beta** | `mdl 1` is the documented language, and `describe`/`fmt` emit it. Headerless scripts keep working, with warnings (2.2, 2.5). | 4.2 acceptance test on `VAL_Feedback` passes | **Risk.** Four weeks is tight for 4.2, which is the largest item and the riskiest code (the graph splice). If its acceptance test is not green in week 4, choose explicitly between slipping beta by a week or two, and shipping beta with `alter microflow` marked experimental while the microflow no-op guarantee stays on the gate. Phase 3's aliases and Phase 5 continue after beta; only the canonical *forms* have to be in the grammar by then. @@ -1241,10 +1246,10 @@ Each change breaks existing text, so each one lands with a registry entry, an `f | # | Item | Size | Warning period | Done when | |---|---|---|---|---| | 2.1 | **R1: `create or modify` is the keyword; `or replace` becomes an alias.** View entities get an identity-carrying rewrite instead of delete-and-recreate; `if not exists` works on every type. The no-op guarantee (an unchanged definition writes nothing) is enforced by the 0.1 harness for every type except microflows and nanoflows, which need 4.2. | M | `or replace` warns | GUID preserved on a view-entity replace (a test with a GUID != `$ID` control); re-running unchanged `describe` output writes no unit | -| 2.2 | **Strictness:** unknown or mis-shaped property keys are errors (REST, agents and business events first, then everywhere); `;` required; `/` removed; `\` is no longer an escape. | M | `;` and `/` warn for one release; unknown keys and `\` break immediately | parse tests | +| 2.2 | **Strictness:** unknown or mis-shaped property keys are errors (REST, agents and business events first, then everywhere); `;` required; `/` removed; `\` is no longer an escape. | M | tied to the header: errors under `mdl 1`, warnings without it | parse tests under both versions | | 2.3 | **Dead grammar removed:** `grant … on workflow`, `case … else`, `text`/`statictext`/`legacydatagrid`, `throw` (if not fixed in 0.3). | S | none (they never worked) | parse errors with a hint | | 2.4 | **Microflow assignment and list operations.** `set` becomes mandatory. List operations and aggregates become one statement per Studio Pro activity (`$A = filter $L by …`, `$B = filter $L where …`, `$n = count $A`; the full table is in §4), so nesting no longer parses. The `find`/`contains` ambiguity disappears; MDL-LISTOP02 becomes unreachable and is retired. The first PR checks the keywords against Studio Pro's dialog labels. | L | old function form warns (except `find`/`contains`, which can't) | `write-microflows` skill and examples migrated by `fmt --upgrade` | -| 2.5 | **`retrieve … first` vs `limit 1`.** In release N, `describe` emits `first` and a bare `limit 1` warns. In release N+1, the beta, `limit 1` means a list. | M | one release | MDL-RETRIEVE01 retired | +| 2.5 | **`retrieve … first` vs `limit 1`.** Under `mdl 1`, `limit 1` means a list and `first` an object. Without the header, the old meaning plus a warning. `describe` emits `first` together with the header. | M | tied to the header | MDL-RETRIEVE01 retired | | 2.6 | **Canonical `describe` (R12), one area per PR:** omit derived layout (extend the `@start` derived-vs-authored rule to `@position`/`@curve`/`@anchor`); omit defaults; no synthetic widget names (make the name optional in `widgetV3`, address grid columns explicitly); fold `join`/`merge` back into `case` and fall-through handlers; emit `elsif`. | L | none (output change) | the harness in 0.1 stays green; PutGet holds per area | **Beta gate:** @@ -1316,3 +1321,29 @@ Order, by how many existing scripts each item touches: - **Grammar changes ripple into generated artefacts:** LSP completions (`lsp_completions_gen.go`), `keyword_coverage_test.go`, the VS Code extension and the embedded skills. Each grammar PR runs `make build`, which regenerates them, and `make sync-skills`. - **The graph splice (4.2b) is the riskiest code.** It must never rewrite an `$ID` without rewriting every reference to it (CLAUDE.md rule 1), and its tests must use a Studio Pro-authored flow, because an mxcli-created flow cannot show identity loss (GUID == `$ID`). - **Scope creep in Phase 3.** Hold each rule to its recipe; anything beyond renaming belongs in its own proposal. + +## 10. Prior art: what is proven and what is new + +Nearly every construct in this proposal has a well-proven precedent. The ADR's *alternatives considered* can cite this table. + +| Proposed construct | Precedent | How close | +|---|---|---| +| `create`/`alter`/`drop`/`list`/`describe` | SQL DDL | exact (ADR-0003) | +| `create or modify` that changes only what differs | `kubectl apply`, Terraform plan/apply, declarative schema tools (Atlas, Skeema) | close. The principle is proven; the difficulty is in the implementation (see below). | +| Declarative modify as diff-then-patch (§8.3) | React reconciliation, Kubernetes controllers, Terraform | close | +| Content addressing (`after $Lines`) | React's `key` prop | close. React needs keys because matching by position produces wrong pairs; `canon/transplant.go` measured exactly that (§8.1). The output variable is the key. | +| `@base` optimistic locking | HTTP `ETag`/`If-Match`, Kubernetes `resourceVersion` | exact in principle | +| Three-way merge (after beta) | `kubectl apply` (it keeps the last applied configuration as an annotation for this), git | very close | +| Dry run in CI as the drift check | `terraform plan -detailed-exitcode` | exact | +| `()` properties, `{}` children (R2) | QML, SwiftUI, Flutter, HCL | close; QML is almost the page syntax | +| `:` declares, `=` binds (R3/R4) | Kotlin and Swift (`name: Type` vs `= value`), Python keyword arguments | close | +| Pattern-based bulk patches (§8.3) | Coccinelle (Linux kernel), OpenRewrite (Java), ast-grep, codemods | proven at large scale on code | +| Deprecation aliases, `fmt --upgrade`, header-tied changes of meaning | `go fix` with the `go` line in go.mod; Rust editions with `cargo fix --edition` | exact | +| One canonical form, emitted by the tool (R12) | `gofmt`, `terraform fmt`, Prettier | exact | +| The two round-trip laws (§8.4) | *lenses*, i.e. bidirectional transformations (Foster, Pierce et al.; Boomerang); Terraform's "no changes after import" | exact in theory, well known in practice | + +**What is new or less proven, and therefore where the risk sits:** + +1. **Patching a flowchart, not a tree.** Pattern-based patching is proven on code syntax trees. Splicing into a diagram graph, with sequence flows, merges and layout, has no textual precedent we know of; BPMN modelers do it internally, behind a UI. This is plan item 4.2b, the riskiest code in the plan, and its acceptance test runs on a Studio Pro-authored flow. +2. **Writing the version stamp back into the source.** Most tools keep this state separately: Terraform's state file, lockfiles, the Kubernetes server. A tool that rewrites your script is unusual, and some users will object. The mitigations: `@base` is optional; `exec` only updates stamps that are already there; projects driven entirely by MDL never use it. +3. **Round-tripping a model that MDL cannot fully express.** The principle is proven, but Terraform's long history of "perpetual diff" provider bugs shows how hard the practice is. Lens theory says a view that does not capture everything needs a *complement*: the hidden part carried alongside. The proposed `preserved` placeholder (§8.4) is that complement. The 7 losses in 12 writes measured in §8.1 are this problem, so the round-trip harness (0.1) is a precondition, not an optional extra. From 979be35aaf05cbd86e4daa9a10eb1f90de613406 Mon Sep 17 00:00:00 2001 From: Ako <andrej@koelewijn.net> Date: Sat, 26 Sep 2026 14:22:16 +0000 Subject: [PATCH 11/15] docs: ADR-0010..0012 for MDL canonical syntax, versioning and editing modes ADR-0010: one canonical form governed by rules R1-R12 (extends ADR-0003, amends its verb inventory: show is dropped). ADR-0011: spelling changes become registered deprecated aliases; meaning changes apply only under the mdl <n> language header. ADR-0012: MDL-first and data-first editing share one syntax and one patch engine; create or modify is diff-then-patch; round-trip laws; optional @base optimistic locking. Cross-references: ADR index, ADR-0003 status note, proposal frontmatter, CLAUDE.md pointer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --- CLAUDE.md | 2 + .../PROPOSAL_mdl_beta_syntax_freeze.md | 3 +- docs/13-decisions/0003-mdl-is-sql-shaped.md | 2 +- .../0010-mdl-canonical-syntax-rules.md | 73 +++++++++++++++++ .../0011-mdl-language-versioning.md | 64 +++++++++++++++ .../0012-mdl-first-and-data-first-editing.md | 81 +++++++++++++++++++ docs/13-decisions/README.md | 3 + 7 files changed, 226 insertions(+), 2 deletions(-) create mode 100644 docs/13-decisions/0010-mdl-canonical-syntax-rules.md create mode 100644 docs/13-decisions/0011-mdl-language-versioning.md create mode 100644 docs/13-decisions/0012-mdl-first-and-data-first-editing.md diff --git a/CLAUDE.md b/CLAUDE.md index 738439104..ec43f7c58 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -350,6 +350,8 @@ Three things no `--help` tells you: ## Before Writing MDL +New MDL syntax follows the canonical rules of ADR-0010 (checklist in `.claude/skills/design-mdl-syntax.md`); a change of meaning lands only behind a language version (ADR-0011). + **Read the matching skill first.** They are in `.claude/skills/` (contributor) and `.claude/skills/mendix/<name>/SKILL.md` (synced to user projects). Each one's frontmatter `description` says when to reach for it — that IS the index, so list the diff --git a/docs/11-proposals/PROPOSAL_mdl_beta_syntax_freeze.md b/docs/11-proposals/PROPOSAL_mdl_beta_syntax_freeze.md index 2085d7538..0f9ba412d 100644 --- a/docs/11-proposals/PROPOSAL_mdl_beta_syntax_freeze.md +++ b/docs/11-proposals/PROPOSAL_mdl_beta_syntax_freeze.md @@ -2,13 +2,14 @@ title: MDL Language Critique and Beta Syntax Freeze status: draft date: 2026-09-26 +decisions: [ADR-0010, ADR-0011, ADR-0012] --- # Proposal: MDL Language Critique and Beta Syntax Freeze **Status:** Draft **Date:** 2026-09-26 -**Related:** [ADR-0003](../13-decisions/0003-mdl-is-sql-shaped.md), [design-mdl-syntax skill](../../.claude/skills/design-mdl-syntax.md), [PROPOSAL_workflow_microflow_syntax_alignment.md](PROPOSAL_workflow_microflow_syntax_alignment.md) +**Related:** [ADR-0003](../13-decisions/0003-mdl-is-sql-shaped.md); resulting decisions [ADR-0010](../13-decisions/0010-mdl-canonical-syntax-rules.md), [ADR-0011](../13-decisions/0011-mdl-language-versioning.md), [ADR-0012](../13-decisions/0012-mdl-first-and-data-first-editing.md); [design-mdl-syntax skill](../../.claude/skills/design-mdl-syntax.md), [PROPOSAL_workflow_microflow_syntax_alignment.md](PROPOSAL_workflow_microflow_syntax_alignment.md) ## Summary diff --git a/docs/13-decisions/0003-mdl-is-sql-shaped.md b/docs/13-decisions/0003-mdl-is-sql-shaped.md index daaf03b71..abf4075ef 100644 --- a/docs/13-decisions/0003-mdl-is-sql-shaped.md +++ b/docs/13-decisions/0003-mdl-is-sql-shaped.md @@ -1,6 +1,6 @@ # ADR-0003: MDL is SQL-shaped -- **Status**: Accepted +- **Status**: Accepted. The verb inventory (`SHOW / LIST`) is amended by [ADR-0010](0010-mdl-canonical-syntax-rules.md) (R6: `show` is dropped); the rest stands. - **Date**: 2026-05-24 - **Related**: [PROPOSAL_mdl_syntax_design_guidelines.md](../11-proposals/PROPOSAL_mdl_syntax_design_guidelines.md); [`design-mdl-syntax` skill](../../.claude/skills/design-mdl-syntax.md) diff --git a/docs/13-decisions/0010-mdl-canonical-syntax-rules.md b/docs/13-decisions/0010-mdl-canonical-syntax-rules.md new file mode 100644 index 000000000..85a280843 --- /dev/null +++ b/docs/13-decisions/0010-mdl-canonical-syntax-rules.md @@ -0,0 +1,73 @@ +# ADR-0010: MDL has one canonical form, governed by twelve syntax rules + +- **Status**: Proposed +- **Date**: 2026-09-26 +- **Related**: extends [ADR-0003](0003-mdl-is-sql-shaped.md); [PROPOSAL_mdl_beta_syntax_freeze.md](../11-proposals/PROPOSAL_mdl_beta_syntax_freeze.md) §3, §4, §7, §10; [ADR-0011](0011-mdl-language-versioning.md), [ADR-0012](0012-mdl-first-and-data-first-editing.md); PR ako/mxcli#702 + +## Context + +ADR-0003 fixed MDL's *shape*: SQL-like verbs, qualified names, English keywords, property lists. It did not fix MDL's *conventions*, and in their absence each document type grew its own. A whole-language review before beta (the proposal, §3–§4) found: + +- **Argument binding has five spellings** across microflow calls, `show page`, button actions, workflows and REST. +- **Children nest by five different conventions** across REST clients, agents, published services, image collections and database connections. +- **Metadata has three homes.** Documentation, folder and layout each have up to three spellings. +- **Page actions are SCREAMING_SNAKE**: `SHOW_PAGE`, `SAVE_CHANGES`. Nothing else in MDL is. +- **Keywords have aliases everywhere.** `delete behavior` alone has 16 surface spellings for 3 values. +- **`describe` emits defaults, derived layout and invented names**, so its output reads worse than what the author wrote. + +After beta, every alias still in the grammar becomes a permanent compatibility obligation. Consolidating is cheap now and expensive later. MDL is authored increasingly by LLMs and reviewed in PR diffs, which strengthens both of ADR-0003's goals: regular patterns and one way to say each thing. + +## Decision + +Every MDL construct has **exactly one canonical form**, defined by the twelve rules below. `describe` emits only that form. Other forms survive only as deprecated aliases (ADR-0011). + +| # | Rule | +|---|---| +| R1 | **One idempotent create: `create or modify`.** It makes the stored document match the definition, and writes nothing when nothing differs (ADR-0012). `if not exists` is the separate "leave it alone" operation. Renames go through `alter`/`rename`, never through editing a definition. | +| R2 | **Three brackets, three meanings.** `( Key: value, … )` holds properties; `{ … }` holds declarative children, each shaped `<kind> [Name] ( props ) [ { children } ]`; `begin … end <keyword>` holds imperative flow. | +| R3 | **`:` sets a model property, `=` binds a runtime value.** `alter` uses the same property list as `create`: `set ( Key: value )`. | +| R4 | **One argument form everywhere: `Param = expression`**, with no `$` on the parameter name. Text-template parameters are always `with ({1} = …)`. | +| R5 | **Expressions are bare, and XPath is always in `[ … ]`.** Neither is ever written inside a string literal. There is one way to refer to a constant: `@Module.Const`. | +| R6 | **The verbs are `list` (plurals and relationship queries), `describe` (one thing), and `create`/`alter`/`drop`.** `show` is dropped. Element `describe` emits runnable MDL. *Definition* `describe` (`describe widget type`, `describe catalog.X`, …) emits a report marked "not executable". No `add`/`remove`/`define`/`update` verbs, and no synonyms such as `column` for `attribute`. | +| R7 | **MDL is what belongs in a checked-in `.mdl` file.** Session commands (`connect`, `set format`, `status`, `help`, …) are REPL commands, not grammar. | +| R8 | **Keywords are words, one spelling each.** No SCREAMING_SNAKE, no optional underscores. Lowercase is canonical. Property keys are case-preserved identifiers, never keywords. | +| R9 | **Each kind of metadata has one home.** Documentation is a `/** */` doc comment. Folder is a `folder '…'` clause. Canvas layout is `@position`/`@anchor`/`@curve`. Everything else is a property. | +| R10 | **Document types use Studio Pro's names**, with consistent `consumed`/`published` prefixes. | +| R11 | **Strict parsing.** `;` terminates every statement. Trailing commas are allowed in every list. `''` is the only string escape. Unknown or mis-shaped property keys are errors. Grammar that can never succeed is removed. | +| R12 | **`describe` emits the canonical form and nothing else.** It omits defaults, layout the tool derived itself, and names Mendix does not store. | + +Microflow list operations follow R2 and R12 in a way that is specific to Studio Pro: **one statement per List operation or Aggregate list activity**, named after the operation, taking the inputs its dialog asks for. As in the diagram, they cannot nest. + +## Consequences + +**Positive** + +- One example of a construct generalises to every document type. This is ADR-0003's LLM-fitness argument, now actually true. +- `describe` output becomes the reference implementation of the language. A fragment from it can be pasted into `create` or `alter` unchanged (R2, R3). +- R2's uniform node shape is what makes a single generic `alter` possible (ADR-0012). The rule has structural value, not only a cosmetic one. +- Several defect classes disappear at the grammar level: nested list operations, the `find` ambiguity, typos in property keys that are silently dropped, and XPath quoting that turns into runs of six quote characters. + +**Negative** + +- **Every existing script, skill, example and syntax page must migrate.** `fmt --upgrade` (ADR-0011) makes most of this mechanical, but not all. +- **`describe` output changes, and that output has been committed by users.** They will see one large diff when it switches to the canonical form. +- **R12's "omit derived layout" needs a reliable way to tell authored layout from derived layout.** That distinction exists today only for `@start`. +- The grammar temporarily grows, because every consolidated form keeps its alias until it is removed. +- R6 draws a line some users will find arbitrary: element describe must round-trip, definition describe need not. + +**Neutral** + +- ADR-0003's verb list (`SHOW / LIST`) and its note on the legacy `show` verb are replaced by R6. The rest of ADR-0003 stands. +- The rules become checklist items in the `design-mdl-syntax` skill, so new syntax is reviewed against them. + +## Alternatives considered + +- **Keep the aliases indefinitely** (be liberal in what you accept). Rejected: every alias is a second way to say something. It doubles what LLMs and reviewers must recognise and freezes accidental history into the post-beta language. +- **`Param: expr` for arguments.** It matches today's page `describe` output. Rejected in favour of R3's semantic split: `:` sets model properties, `=` binds runtime values, which also covers `change` and `set`. +- **SQL-style `alter X set Key = value`.** Rejected because the same key would take `:` in `create` and `=` in `alter`, which breaks copying a fragment between them. +- **`create or replace` as the idempotent keyword** (it matches SQL's `CREATE OR REPLACE VIEW`). Rejected because `modify` states the intent, *make the stored document match this*, and R1 adds minimal-change semantics that "replace" would contradict. +- **Keep `show` for non-element state.** Rejected in favour of a single `list`/`describe` split, which leaves no third verb to choose from. +- **Microflow list operations as nestable functions** (today's form). Rejected: the diagram has no nesting. The function form invited `count(filter(…))`, which Mendix cannot store, and it collides with the string functions `find` and `contains`. +- **Earlier proposals: `:=`, braces for blocks, lambdas, dropping `call`** (`PROPOSAL_mdl_syntax_improvements*.md`). Rejected as contrary to ADR-0003. + +Prior art for each rule is tabulated in the proposal's §10. diff --git a/docs/13-decisions/0011-mdl-language-versioning.md b/docs/13-decisions/0011-mdl-language-versioning.md new file mode 100644 index 000000000..3352c687b --- /dev/null +++ b/docs/13-decisions/0011-mdl-language-versioning.md @@ -0,0 +1,64 @@ +# ADR-0011: MDL evolves through deprecation aliases and a language header; meaning changes only across versions + +- **Status**: Proposed +- **Date**: 2026-09-26 +- **Related**: [ADR-0010](0010-mdl-canonical-syntax-rules.md); [PROPOSAL_mdl_beta_syntax_freeze.md](../11-proposals/PROPOSAL_mdl_beta_syntax_freeze.md) §5, §6, §9, §10; PR ako/mxcli#702 + +## Context + +ADR-0010 consolidates MDL onto one canonical form. That change comes in two kinds: + +- **Respellings.** Most changes give an existing meaning a new spelling: `SHOW_PAGE` becomes `show page`, `generalization` becomes `extends`. The old form can keep parsing. +- **Changes of meaning.** A few change what existing text *means*, or reject text that is accepted today: + - `retrieve … limit 1` binds an object today; under R11 it will be a list of one. + - `$x = find(…)` means different things depending on earlier statements. + - `'it\'s'` is an escape today. + - Unknown property keys are silently ignored today. + + Changing their meaning in place would silently alter scripts that users have committed. + +MDL has no version marker. The proposed `set version '10.18'` (`version-aware-mdl.md`) names the **Mendix target** version, a different axis. mxcli releases weekly, so "warn for one release, then flip" gives users one week to notice. After beta, the same mechanism will be needed for every later change. + +## Decision + +1. **Respellings become deprecated aliases.** + - Each alias is an entry in one registry: code `MDL-DEPRnnn`, old form, canonical form, and the version that removes it. + - `check` and `exec` warn on it, and `describe` never emits it. + - `mxcli fmt --upgrade` rewrites every registered alias mechanically. + - An alias without a registry entry fails a test. +2. **A script may start with the header `mdl <n>;`**, which declares the language version it is written in. Meaning changes and new rejections apply **only under the version that introduces them**: + - `mdl 1;` gets beta semantics. + - A script with no header is treated as `mdl 0` (the alpha meaning) and warns on every construct whose meaning differs. + - `fmt --upgrade` adds the header and rewrites those constructs. + - `describe` and `fmt` always emit the header. + - Aliases can be removed only at a version boundary: under the version that deprecates them they warn; under the next one they are refused. +3. **The header is independent of the Mendix target version.** + +## Consequences + +**Positive** + +- A script's meaning never depends on which mxcli release runs it. No committed script changes behaviour silently. +- Beta is not a flag day. Existing scripts keep running with warnings, and each user migrates when they choose, in one command. +- The same mechanism carries every future change, so after beta, "can we change this?" becomes "which version does it belong to?". +- Documentation, skills and examples can be held to the canonical form automatically, with deprecation warnings treated as errors in CI. + +**Negative** + +- **Two language versions must be maintained side by side.** The visitor must implement both meanings wherever they differ, and the tests must cover both. +- **A script with no header gets the *old* meaning.** New users, and LLMs that omit the header, get alpha semantics plus warnings until they add it. This is the price of never changing meaning silently, and the same price Rust and Go pay. +- **`fmt --upgrade` is a code generator that must be exactly right.** A wrong rewrite is itself a silent change of meaning. Every rewrite rule needs a test that the AST is equivalent after the rewrite. +- The registry is a new place every syntax change has to be recorded. + +**Neutral** + +- `mdl 0` is only ever implicit. Nobody writes it. +- Removing a deprecated form requires both a new language version and the passage of time, never only time. + +## Alternatives considered + +- **Warn for one release, then change meaning in place.** This was the proposal's first plan. Rejected: with weekly releases, the warning window is one week, and a user who skipped a release gets a silent change. +- **A script with no header gets the latest version.** Rejected: its meaning would then depend on the mxcli release that runs it. Rust (a missing `edition` means 2015) and Go (a missing `go` line means the oldest version) chose the oldest for this reason. +- **Tie the language version to the mxcli version.** Rejected: mxcli releases weekly, while the language should change rarely. +- **Tie the language version to the Mendix version.** Rejected: that is a different axis, and a project upgrading Mendix must not have its scripts change meaning. +- **No versioning; aliases forever.** Rejected: aliases cannot express a change of meaning, so R11's changes would be impossible. diff --git a/docs/13-decisions/0012-mdl-first-and-data-first-editing.md b/docs/13-decisions/0012-mdl-first-and-data-first-editing.md new file mode 100644 index 000000000..a4ffbf248 --- /dev/null +++ b/docs/13-decisions/0012-mdl-first-and-data-first-editing.md @@ -0,0 +1,81 @@ +# ADR-0012: MDL-first and data-first editing share one syntax and one patch engine + +- **Status**: Proposed +- **Date**: 2026-09-26 +- **Related**: builds on [ADR-0008](0008-identity-and-idempotence.md); [ADR-0010](0010-mdl-canonical-syntax-rules.md) (R1, R2, R12); [PROPOSAL_mdl_beta_syntax_freeze.md](../11-proposals/PROPOSAL_mdl_beta_syntax_freeze.md) §8, §9, §10; PR ako/mxcli#702 + +## Context + +mxcli is used in two ways that pull in opposite directions: + +- **New apps and modules.** MDL scripts are the source. Declarative definitions are the most efficient way to write them: each element is written once, in its final shape, with nothing to read first. +- **Existing Studio Pro apps.** This is where most agent work happens. The stored model is the truth, and MDL is a view of it. The efficient change is a small patch. + +Measured on two real apps (the proposal, §8.1), the second way does not work today: + +- **Re-running a document's own `describe` output as `create or modify`**, with no edit at all, silently lost data in **7 of 12** Studio Pro documents: + - association storage changed from table to column (a schema change); + - page translations were lost; + - annotation links were lost; + - export levels changed. +- **Microflows have no `alter`.** A one-line insert into a 16-activity flow meant re-sending all 107 lines. The whole-document rebuild (`UpdateMicroflow`, followed by re-pairing IDs by type and position in `canon/transplant.go`) then: + - deleted 5 merges; + - reset 11 of 15 curves; + - changed 51 of 161 element `$ID`s, some onto *different* nodes. +- **Surgical `alter page … set` preserved everything.** + +ADR-0008 made storage skip unchanged units. It cannot help when the rebuilt document is genuinely different from the stored one, and here it is. + +## Decision + +1. **Two modes, both first-class, one syntax.** + - **MDL-first** is declarative: `create or modify`, where the script is the source of truth. + - **Data-first** is a patch: `alter <type> X { set / insert / replace / drop }`, where the stored model is the truth. + - A fragment inside `alter` is written exactly as in `create`, and `describe` output is valid declarative source. +2. **One generic `alter`, addressed by content where elements have no names.** + - It covers every document type, including microflows and nanoflows. + - Microflow activities are addressed by output variable, then by caption, then by statement pattern with wildcards. + - When an address matches more than one element, that is an error that lists the matches. mxcli never guesses. +3. **Both modes write through one patch engine** that splices into the stored document. + - Untouched elements stay byte-identical; only new nodes are placed. + - `create or modify` is diff-then-patch: compare the definition with the stored document, derive the minimal patch, apply it. An empty patch writes nothing. +4. **Two round-trip laws hold for every document type** that has element `describe`, and CI enforces them against a Studio Pro-authored fixture: + - **GetPut:** executing `describe` output unchanged writes nothing. + - **PutGet:** describing what was written returns what was written. + + Content that MDL cannot express is carried through by a `preserved` placeholder, or the write is refused. It is never silently dropped. +5. **Concurrent change is detected by optional optimistic locking.** + - For mixed projects: `describe` emits `@base '<fingerprint>'`, `create or modify` refuses when that fingerprint no longer matches the stored document, and `exec` updates the stamps it finds. + - For projects driven entirely by MDL: `@base` is not used. A dry run of all scripts reporting no changes is the drift check. + - There is no separate state file. + +## Consequences + +**Positive** + +- The MDL-first and data-first cases each get the efficient form, and moving between them needs no second language. +- Agents can change existing apps at a cost proportional to the change, not to the document, without disturbing what they did not touch. This is the beta goal (§7 of the proposal). +- "No changes" becomes true in Studio Pro and in version control, not only in storage, which extends ADR-0008 from storage up to the language. +- The laws turn silent loss into either a test failure or a refusal. + +**Negative** + +- **The graph splice is new.** Patching a flowchart graph (sequence flows, merges, layout) has no textual precedent we know of. It is the riskiest code in the plan, and it must be tested on Studio Pro-authored flows, because mxcli-authored ones cannot show identity loss. +- **Diff-then-patch needs a stable way to match elements.** Where an activity has no output variable or caption, matching falls back to its statement signature. That is the same weakness React has for list items without `key` props. +- **`@base` means `exec` rewrites source files**, which is unusual (most tools keep this state separately), and some users will object. It is optional and only updates stamps already present. +- **A `preserved` placeholder in `describe` output is opaque to readers and to LLMs.** Each occurrence is a gap in what MDL can express, and must be tracked down. +- The whole-document rebuild paths (`UpdateMicroflow` and others) have to be replaced, not patched. + +**Neutral** + +- The guidance for agents is "choose the mode by who owns the document". It becomes skill text, and eventually an enforced check through `@base`. +- Three-way merge and per-element locking are natural extensions, deferred until after beta. + +## Alternatives considered + +- **Declarative only: always describe → edit → replace.** Rejected on the measurements above. Its cost is proportional to the document, and on Studio Pro content it is lossy. +- **Patch only.** Rejected for new apps: it forces building from empty shells, and scripts then describe a history rather than a design. +- **Keep the whole-document rebuild and improve ID re-pairing.** Rejected: re-pairing by type and position is structurally unable to be a fixed point once the builder emits objects in a different order from Studio Pro's. +- **A per-doctype `alter` grammar for microflows**, in the style of today's `alter workflow`. Rejected in favour of one generic `alter` over R2's uniform node shape, which leaves one grammar to learn instead of about 15. +- **Positional addressing** (the nth activity), as in JSON Patch's array indexes. Rejected as brittle under any unrelated edit. +- **A sidecar state file** for drift detection, as Terraform uses. Rejected in favour of the stamp in the statement, which travels with the script through git, copy-paste and review, and keeps statements self-contained (ADR-0003). This follows the HTTP `If-Match` model. diff --git a/docs/13-decisions/README.md b/docs/13-decisions/README.md index 7cf6b2c0b..dde0f810c 100644 --- a/docs/13-decisions/README.md +++ b/docs/13-decisions/README.md @@ -103,6 +103,9 @@ This preserves the audit trail. | [0007](0007-mcp-read-model-session-overlay.md) | MCP backend read model — disk base with session overlay | Proposed | | [0008](0008-identity-and-idempotence.md) | Skip unchanged writes; never renumber element IDs in place | Accepted | | [0009](0009-tunnel-is-linux-only.md) | The embedded tunnel ships in Linux builds only | Accepted | +| [0010](0010-mdl-canonical-syntax-rules.md) | MDL has one canonical form, governed by twelve syntax rules | Proposed | +| [0011](0011-mdl-language-versioning.md) | MDL evolves through deprecation aliases and a language header; meaning changes only across versions | Proposed | +| [0012](0012-mdl-first-and-data-first-editing.md) | MDL-first and data-first editing share one syntax and one patch engine | Proposed | ## Candidates to back-fill From 703d017d634649f315199ff012873b6585dc328b Mon Sep 17 00:00:00 2001 From: Ako <andrej@koelewijn.net> Date: Sat, 26 Sep 2026 14:22:16 +0000 Subject: [PATCH 12/15] skills: design-mdl-syntax checklist follows ADR-0010 rules Adds the R1-R12 canonical rules and alias/versioning checks, and fixes examples that contradicted the grammar (rename ... as, entity grant order) or the new rules (show, filter). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --- .claude/skills/design-mdl-syntax.md | 45 ++++++++++++++++++++++++----- 1 file changed, 37 insertions(+), 8 deletions(-) diff --git a/.claude/skills/design-mdl-syntax.md b/.claude/skills/design-mdl-syntax.md index 8d17086c9..7a7f3259a 100644 --- a/.claude/skills/design-mdl-syntax.md +++ b/.claude/skills/design-mdl-syntax.md @@ -35,9 +35,9 @@ Reuse existing patterns. Never create a second syntax for the same concept. | Remove | `drop <type> Module.Name` | `drop entity Shop.Product` | | List | `list <type>S [in module]` | `list entities in Shop` | | Inspect | `describe <type> Module.Name` | `describe entity Shop.Product` | -| Security | `grant/revoke <perm> on <target> to/from <role>` | `grant read on Shop.Product to Shop.User` | +| Security | `grant/revoke <perm> on <kind> <target> to/from <role>` | `grant read on entity Shop.Product to Shop.User` | -Do NOT use alternative verbs: `add` instead of `create`, `remove` instead of `drop`, `show` instead of `list`, `view` instead of `describe`. Note: `show` is the legacy verb — new commands use `list`. +Do NOT use alternative verbs: `add` instead of `create`, `remove` instead of `drop`, `show` instead of `list`, `view` instead of `describe`. `show` is dropped (ADR-0010 R6): plurals and relationship queries are `list`, single things are `describe`. Inside `alter`, children are added and removed with `add`/`drop`. ### 3. Optimize for LLMs @@ -56,7 +56,7 @@ Do NOT use alternative verbs: `add` instead of `create`, `remove` instead of `dr ### 5. Token Efficiency (Without Sacrificing Clarity) - Omit noise words: `create entity` not `create A NEW entity` -- Support `or modify` to avoid check-then-create +- Support `create or modify` (the one idempotent create, ADR-0010 R1) to avoid check-then-create - Allow type inference for obvious cases: `declare $count = 0` - Do NOT use symbols to save tokens at the cost of readability @@ -115,7 +115,9 @@ Rules: - Trailing comma allowed - One property per line (single line acceptable for 1-2 properties) -#### Colon `:` vs `as` — When to Use Each +#### Colon `:`, equals `=` and `as` — When to Use Each + +ADR-0010 R3: **`:` sets a model property; `=` binds a runtime value** (call arguments `M.F(Order = $Order)`, `change $o (Attr = v)`, `set $x = …`, text-template parameters `with ({1} = …)`). `alter` uses the same property list as `create`: `set ( Key: value )`. Use **colon** for property definitions (assigning a value to a named property): @@ -134,9 +136,10 @@ CUSTOM NAME map ( 'kvkNummer' as 'ChamberOfCommerceNumber', -- old name AS new name 'naam' as 'CompanyName', ) -alter entity Shop.Product rename Code as ProductCode -- old attr AS new attr ``` +Renames use `to`, like top-level `rename … to`: `alter entity Shop.Product rename attribute Code to ProductCode`. + **Rule of thumb**: if the left side is a *fixed property key* defined by the syntax, use `:`. If the left side is a *user-provided name* being mapped to another name, use `as`. ### Step 5: Validate @@ -181,7 +184,7 @@ create entity Shop.Customer (...); $items |> filter($.active) |> map($.name) -- RIGHT: keyword-based -filter $Items where Active = true +$Active = filter $Items by Active = true; -- one statement per Studio Pro list-operation activity ``` ### Positional Arguments @@ -204,10 +207,35 @@ create rule Shop.ProcessOrder ( -- Don't reuse it to mean property modification elsewhere unless established ``` +## Canonical Rules (ADR-0010) + +These are the canonical rules. Each PR that adds or changes syntax is checked against them. The rationale is in [ADR-0010](../../docs/13-decisions/0010-mdl-canonical-syntax-rules.md); examples are in `docs/11-proposals/PROPOSAL_mdl_beta_syntax_freeze.md` §3. + +| # | Rule | +|---|---| +| R1 | One idempotent create: `create or modify`. It changes only what differs, and writes nothing when nothing differs. `if not exists` is the "leave it alone" operation. Renames go through `alter`/`rename`. | +| R2 | `( Key: value, )` holds properties. `{ }` holds declarative children, each shaped `<kind> [Name] ( props ) [ { … } ]`. `begin … end <keyword>` holds imperative flow. | +| R3 | `:` sets a model property; `=` binds a runtime value. `alter … set ( Key: value )` takes exactly the keys of `create`. | +| R4 | One argument form everywhere: `Param = expression` (no `$` on the parameter name). Text templates always use `with ({1} = …)`. | +| R5 | Expressions are bare; XPath is in `[ … ]`. Neither ever goes in a string literal. Constants are referred to as `@Module.Const`. | +| R6 | Verbs: `list`, `describe`, `create`, `alter`, `drop`. Element `describe` emits runnable MDL; definition `describe` emits a report marked "not executable". | +| R7 | Session commands are REPL commands, not grammar. | +| R8 | Keywords are words, one spelling each: lowercase, no SCREAMING_SNAKE, no optional underscores. Property keys are identifiers, never keywords. | +| R9 | Documentation is a `/** */` doc comment; folder is a `folder '…'` clause; canvas layout uses `@` annotations; everything else is a property. | +| R10 | Document types use Studio Pro's names, with consistent `consumed`/`published` prefixes. | +| R11 | `;` is required; trailing commas are allowed in every list; `''` is the only string escape; unknown property keys are errors. | +| R12 | `describe` emits the canonical form only: no defaults, no derived layout, no names Mendix does not store. | + +A change of **meaning** to existing syntax is never made in place. It lands behind a language version (`mdl <n>;`, [ADR-0011](../../docs/13-decisions/0011-mdl-language-versioning.md)). A change of **spelling** keeps the old form as a registered deprecated alias. + ## Checklist Before merging any PR that adds new MDL syntax, verify: +- [ ] Conforms to R1–R12 above (and, where the construct exists in both modes, `alter` accepts the same fragment syntax as `create`, per ADR-0012) +- [ ] No new alias: any second spelling is a registered deprecation with an `fmt --upgrade` rewrite +- [ ] Any change of meaning is gated on the language header (ADR-0011) + - [ ] Follows `create`/`alter`/`drop`/`list`/`describe` pattern - [ ] Uses `Module.Element` qualified names (no bare names) - [ ] Property lists use `( key: value, ... )` format @@ -224,8 +252,9 @@ Before merging any PR that adds new MDL syntax, verify: ## Related Resources -- Full design rationale: `docs/11-proposals/PROPOSAL_mdl_syntax_design_guidelines.md` +- Decisions: [ADR-0003](../../docs/13-decisions/0003-mdl-is-sql-shaped.md), [ADR-0010](../../docs/13-decisions/0010-mdl-canonical-syntax-rules.md), [ADR-0011](../../docs/13-decisions/0011-mdl-language-versioning.md), [ADR-0012](../../docs/13-decisions/0012-mdl-first-and-data-first-editing.md) +- Beta syntax proposal (examples, plan): `docs/11-proposals/PROPOSAL_mdl_beta_syntax_freeze.md` +- Earlier design rationale: `docs/11-proposals/PROPOSAL_mdl_syntax_design_guidelines.md` - MDL Quick Reference: `docs/01-project/MDL_QUICK_REFERENCE.md` - Implementation workflow: `.claude/skills/implement-mdl-feature.md` -- Existing syntax proposals: `docs/11-proposals/PROPOSAL_mdl_syntax_improvements.md` - Grammar file: `mdl/grammar/MDLParser.g4` From 306a3ac78208f0788171af8db4503a1b9a1c8935 Mon Sep 17 00:00:00 2001 From: Ako <andrej@koelewijn.net> Date: Sat, 26 Sep 2026 17:02:41 +0000 Subject: [PATCH 13/15] docs: header-gated changes need no warning releases; mdl 1 is a preview until beta Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --- .../PROPOSAL_mdl_beta_syntax_freeze.md | 38 ++++++++++++------- .../0011-mdl-language-versioning.md | 5 ++- 2 files changed, 28 insertions(+), 15 deletions(-) diff --git a/docs/11-proposals/PROPOSAL_mdl_beta_syntax_freeze.md b/docs/11-proposals/PROPOSAL_mdl_beta_syntax_freeze.md index 0f9ba412d..9ac438f68 100644 --- a/docs/11-proposals/PROPOSAL_mdl_beta_syntax_freeze.md +++ b/docs/11-proposals/PROPOSAL_mdl_beta_syntax_freeze.md @@ -946,7 +946,7 @@ The language has no version marker today. `version-aware-mdl.md` proposes `set v 1. **Deprecated aliases keep parsing** and emit a warning with a stable code (`MDL-DEPR001`, …) that names the canonical form. `describe` never emits a deprecated form. 2. **`mxcli fmt --upgrade`** rewrites every deprecated form to its canonical form. This is mechanical for every alias in §3–§4, and it is what makes consolidation cheap for users. 3. **An optional language header**, `mdl 1;`, as the first statement (decided, §7). - - `describe` and `fmt` emit it. + - `describe` and `fmt` emit it **from beta on**. Before beta, `mdl 1` is a preview that warns "may still change"; at beta it is frozen, and later changes of meaning need `mdl 2`. - **With no header, a script gets the alpha meaning (`mdl 0`) plus warnings**, never the latest. This matches the precedents: Rust treats a crate with no edition as the oldest edition, and Go treats a module with no `go` line as the oldest version. The meaning of a script never depends on which mxcli release happens to run it. - Under `mdl 1`, the §5 changes of meaning and new rejections apply. Under `mdl 0` they are warnings. - After beta, a change that is not backwards compatible bumps the number, and the visitor gates removed aliases on it: under `mdl 1` they warn, under `mdl 2` they are refused. @@ -969,9 +969,9 @@ See the implementation plan in §9, which supersedes the short list that was her 5. **R3/R4, argument binding: `Param = expr`.** `=` binds a runtime value, `:` sets a model property. There is no `$` on the parameter name. One form for every call site: microflow calls, `show page`, button and menu actions, workflows and REST. 6. **Alter form: `set ( Key: value, … )`.** It is the same property list as `create`, so a fragment from `describe` can be pasted straight into an `alter`. 7. **Brownfield agent editing is a beta goal.** `alter microflow`/`alter nanoflow` (plan item 4.2, at least insert, replace and drop) and the no-op `create or modify` for microflows join the beta gate. -8. **The `mdl 1;` language header is added now** (plan item 1.5). It is an optional first statement that `describe` and `fmt` emit; after beta, removed aliases are gated by version. +8. **The `mdl 1;` language header is added now** (plan item 1.5). It is an optional first statement. It is a preview until beta, then frozen; `describe` and `fmt` emit it from beta on. After beta, removed aliases are gated by version. -9. **Cadence: one release per week, beta in about four weeks** (around 2026-10-24). The schedule is in §9. +9. **Cadence: one release per week; beta in about four weeks** (around 2026-10-24). Because changes of meaning are tied to the header, no change needs a warning release. Beta is a single release, cut when `mdl 1` is complete. The schedule is in §9. 10. **Drift detection is optimistic locking, and is optional** (§8.2). A `@base '<fingerprint>'` annotation on the statement is emitted by `describe`, checked by `create or modify`, and updated by `exec`. Projects driven entirely by MDL skip it and instead require a dry run of all scripts to report no changes. There is no sidecar state file. @@ -1205,18 +1205,30 @@ Phase 0 safety net + bugs ──┬──> Phase 1 decisions + deprecation mac Phase 1 ──> Phase 4 data-first editing (alter microflow, drift, dry run) ``` -### Schedule (weekly releases, beta in about four weeks) +### Schedule (weekly releases; beta when `mdl 1` is complete) -Changes of meaning are tied to the `mdl 1;` header (§5), so no existing script breaks on a given date. The schedule pressure is about having `mdl 1` complete, documented and emitted by beta. The deprecation registry (1.2) and the header (1.5) still land in week 1, because everything in week 2 builds on them. +**No change needs a warning release.** +- Changes of meaning apply only under `mdl 1;` (§5), so a script without the header keeps its meaning on every release. +- Changes of spelling keep the old form as an alias. +- The old output of `describe` has no header, so it keeps running with the old meaning. -| Release | Lands | Parallel track (Phase 4) | +Beta can therefore be a single release, cut whenever `mdl 1` is complete and verified. The order below is **build order**: the registry (1.2) and the header (1.5) come first because everything else is built on them. Weekly releases ship progress safely along the way. + +**`mdl 1` is a preview until beta, then frozen.** Parts of `mdl 1` ship in weekly releases before beta, so its meaning could still shift between those releases. To stop anyone depending on an unfinished `mdl 1`: +- Before beta, `mdl 1;` parses but warns "preview: may still change". +- `describe` and `fmt` do **not** emit the header before beta; `fmt --upgrade` adds it only on request. +- At beta, `mdl 1` is frozen, and `describe` and `fmt` start emitting it. From then on, any change of meaning needs `mdl 2`. + +| Order | Lands | Parallel track (Phase 4) | |---|---|---| -| **Week 1** | ADR (1.1); deprecation registry (1.2); `mdl 1;` (1.5); round-trip harness and PedApp fixture (0.1, 0.2); first bug fixes (0.3–0.5); interim skill guidance (0.6) | generic `alter` skeleton (4.1); `mfmutator` target resolver (4.2a) | -| **Week 2** | **The `mdl 1` semantics ship.** New forms parse alongside the old ones (2.1, 2.4, 2.5). Scripts without the header warn on every construct whose meaning differs. `fmt --upgrade` adds the header and rewrites (1.3). | graph splice and placement (4.2b, 4.2c) | -| **Week 3** | Canonical `describe` (2.6). Phase 3 canonical forms added to the grammar, with the old forms as aliases. Conformance gate (1.4). Dead grammar removed (2.3). | `insert`, `replace`, `drop`; diff-then-patch `create or modify` for microflows (4.2e, 4.2g) | -| **Week 4: beta** | `mdl 1` is the documented language, and `describe`/`fmt` emit it. Headerless scripts keep working, with warnings (2.2, 2.5). | 4.2 acceptance test on `VAL_Feedback` passes | +| **1** | ADR (1.1); deprecation registry (1.2); `mdl 1;` header as a preview (1.5); round-trip harness and PedApp fixture (0.1, 0.2); first bug fixes (0.3–0.5); interim skill guidance (0.6) | generic `alter` skeleton (4.1); `mfmutator` target resolver (4.2a) | +| **2** | `mdl 1` semantics behind the preview header (2.1, 2.2, 2.4, 2.5); `fmt --upgrade` (1.3) | graph splice and placement (4.2b, 4.2c) | +| **3** | Canonical `describe` (2.6); Phase 3 canonical forms added to the grammar, with the old forms as aliases; conformance gate (1.4); dead grammar removed (2.3) | `insert`, `replace`, `drop`; diff-then-patch `create or modify` for microflows (4.2e, 4.2g) | +| **Beta** | `mdl 1` is frozen, documented, and emitted by `describe`/`fmt`. Scripts without the header keep working, with warnings. | 4.2 acceptance test on `VAL_Feedback` passes | + +At a weekly cadence this is still roughly four weeks, but no date is imposed by compatibility. -**Risk.** Four weeks is tight for 4.2, which is the largest item and the riskiest code (the graph splice). If its acceptance test is not green in week 4, choose explicitly between slipping beta by a week or two, and shipping beta with `alter microflow` marked experimental while the microflow no-op guarantee stays on the gate. Phase 3's aliases and Phase 5 continue after beta; only the canonical *forms* have to be in the grammar by then. +**Risk.** Item 4.2 is the largest and the riskiest code (the graph splice). If its acceptance test is not green when everything else is, slip beta. Slipping now costs nobody compatibility, so it is the cheap option; the alternative is shipping `alter microflow` as experimental. Phase 3's aliases and Phase 5 continue after beta; only the canonical *forms* have to be in the grammar and in `mdl 1` by then. ### Phase 0: safety net and bugs (no syntax change) @@ -1242,9 +1254,9 @@ Changes of meaning are tied to the `mdl 1;` header (§5), so no existing script ### Phase 2: the beta gate (§5, cannot be aliased) -Each change breaks existing text, so each one lands with a registry entry, an `fmt --upgrade` rewrite where one is possible, and release notes. Where a warning period is possible, it lasts one release before the break. +Each change lands with a registry entry, an `fmt --upgrade` rewrite where one is possible, and release notes. Changes of meaning and new rejections are tied to the `mdl 1` header, so none of them needs a warning release. -| # | Item | Size | Warning period | Done when | +| # | Item | Size | Compatibility | Done when | |---|---|---|---|---| | 2.1 | **R1: `create or modify` is the keyword; `or replace` becomes an alias.** View entities get an identity-carrying rewrite instead of delete-and-recreate; `if not exists` works on every type. The no-op guarantee (an unchanged definition writes nothing) is enforced by the 0.1 harness for every type except microflows and nanoflows, which need 4.2. | M | `or replace` warns | GUID preserved on a view-entity replace (a test with a GUID != `$ID` control); re-running unchanged `describe` output writes no unit | | 2.2 | **Strictness:** unknown or mis-shaped property keys are errors (REST, agents and business events first, then everywhere); `;` required; `/` removed; `\` is no longer an escape. | M | tied to the header: errors under `mdl 1`, warnings without it | parse tests under both versions | diff --git a/docs/13-decisions/0011-mdl-language-versioning.md b/docs/13-decisions/0011-mdl-language-versioning.md index 3352c687b..a2d3c7c09 100644 --- a/docs/13-decisions/0011-mdl-language-versioning.md +++ b/docs/13-decisions/0011-mdl-language-versioning.md @@ -30,7 +30,7 @@ MDL has no version marker. The proposed `set version '10.18'` (`version-aware-md - `mdl 1;` gets beta semantics. - A script with no header is treated as `mdl 0` (the alpha meaning) and warns on every construct whose meaning differs. - `fmt --upgrade` adds the header and rewrites those constructs. - - `describe` and `fmt` always emit the header. + - **A version is a preview until it is frozen.** Before beta, `mdl 1;` parses but warns "preview: may still change", and `describe`/`fmt` do not emit it. At beta, `mdl 1` is frozen and `describe`/`fmt` always emit it. After that, any change of meaning needs a new version. - Aliases can be removed only at a version boundary: under the version that deprecates them they warn; under the next one they are refused. 3. **The header is independent of the Mendix target version.** @@ -39,7 +39,7 @@ MDL has no version marker. The proposed `set version '10.18'` (`version-aware-md **Positive** - A script's meaning never depends on which mxcli release runs it. No committed script changes behaviour silently. -- Beta is not a flag day. Existing scripts keep running with warnings, and each user migrates when they choose, in one command. +- Beta is not a flag day, and it needs no warning releases. Because no script changes meaning without opting in, beta is a single release, cut when `mdl 1` is complete. Existing scripts keep running with warnings, and each user migrates when they choose, in one command. - The same mechanism carries every future change, so after beta, "can we change this?" becomes "which version does it belong to?". - Documentation, skills and examples can be held to the canonical form automatically, with deprecation warnings treated as errors in CI. @@ -53,6 +53,7 @@ MDL has no version marker. The proposed `set version '10.18'` (`version-aware-md **Neutral** - `mdl 0` is only ever implicit. Nobody writes it. +- Freezing, not the release number, is what makes a version a contract. A preview version may change between weekly releases, and says so. - Removing a deprecated form requires both a new language version and the passage of time, never only time. ## Alternatives considered From 266e7ad676a06f420f15f2391b51f438f28925b2 Mon Sep 17 00:00:00 2001 From: Ako <andrej@koelewijn.net> Date: Sat, 26 Sep 2026 17:48:16 +0000 Subject: [PATCH 14/15] docs: drop CLAUDE.md pointer to ADR-0010/0011; it lives in design-mdl-syntax CLAUDE.md is at its context budget (TestRepoClaudeMDStaysWithinItsContextBudget). The rule applies only when designing syntax, so the design-mdl-syntax skill, which already carries R1-R12 and the versioning checks, is its home. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --- CLAUDE.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index ec43f7c58..738439104 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -350,8 +350,6 @@ Three things no `--help` tells you: ## Before Writing MDL -New MDL syntax follows the canonical rules of ADR-0010 (checklist in `.claude/skills/design-mdl-syntax.md`); a change of meaning lands only behind a language version (ADR-0011). - **Read the matching skill first.** They are in `.claude/skills/` (contributor) and `.claude/skills/mendix/<name>/SKILL.md` (synced to user projects). Each one's frontmatter `description` says when to reach for it — that IS the index, so list the From 97b07a0768e94abb266afe7c69f530ea87f49b01 Mon Sep 17 00:00:00 2001 From: Ako <andrej@koelewijn.net> Date: Sat, 26 Sep 2026 18:39:21 +0000 Subject: [PATCH 15/15] docs: accept ADR-0010..0012; mark @base as provisional pending user and CI validation Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --- docs/11-proposals/PROPOSAL_mdl_beta_syntax_freeze.md | 8 ++++---- docs/13-decisions/0010-mdl-canonical-syntax-rules.md | 2 +- docs/13-decisions/0011-mdl-language-versioning.md | 2 +- .../13-decisions/0012-mdl-first-and-data-first-editing.md | 4 +++- docs/13-decisions/README.md | 6 +++--- 5 files changed, 12 insertions(+), 10 deletions(-) diff --git a/docs/11-proposals/PROPOSAL_mdl_beta_syntax_freeze.md b/docs/11-proposals/PROPOSAL_mdl_beta_syntax_freeze.md index 9ac438f68..027768619 100644 --- a/docs/11-proposals/PROPOSAL_mdl_beta_syntax_freeze.md +++ b/docs/11-proposals/PROPOSAL_mdl_beta_syntax_freeze.md @@ -1,13 +1,13 @@ --- title: MDL Language Critique and Beta Syntax Freeze -status: draft +status: accepted date: 2026-09-26 decisions: [ADR-0010, ADR-0011, ADR-0012] --- # Proposal: MDL Language Critique and Beta Syntax Freeze -**Status:** Draft +**Status:** Accepted (ADR-0010, ADR-0011 and ADR-0012 accepted 2026-09-26) **Date:** 2026-09-26 **Related:** [ADR-0003](../13-decisions/0003-mdl-is-sql-shaped.md); resulting decisions [ADR-0010](../13-decisions/0010-mdl-canonical-syntax-rules.md), [ADR-0011](../13-decisions/0011-mdl-language-versioning.md), [ADR-0012](../13-decisions/0012-mdl-first-and-data-first-editing.md); [design-mdl-syntax skill](../../.claude/skills/design-mdl-syntax.md), [PROPOSAL_workflow_microflow_syntax_alignment.md](PROPOSAL_workflow_microflow_syntax_alignment.md) @@ -1070,7 +1070,7 @@ create or modify persistent entity Shop.Order ( | `@base` matches | exists | modify, and `exec` updates `@base` | | `@base` differs | exists | refused: "changed since you read it"; use `alter`, re-`describe`, or `--force` | -**`@base` is optional.** A statement without it behaves exactly as it does today. +**`@base` is optional, and provisional** (ADR-0012). A statement without it behaves exactly as it does today. Users may dislike stamps in their scripts, and CI/CD pipelines, which usually check out read-only and do not commit back, may not support a tool that rewrites them. It is therefore validated with users and in a pipeline before it is built. If it fails, it is replaced by a lock kept outside the script: a committed state file, or the base stored alongside the model. **2. A dry run of all scripts must report no changes: for projects driven entirely by MDL.** @@ -1304,7 +1304,7 @@ Order, by how many existing scripts each item touches: | | e. **Operations,** in order: `insert after`/`before` → `replace` → `drop` → `set` (expression, caption, `on error`) → `add`/`drop parameter`. | | | | | | f. **Both backends:** the modelsdk engine and `--mcp` (the Studio Pro MCP backend), or an explicit "not supported by this backend" error. | | | | | | g. **Declarative `create or modify` as diff-then-patch:** match the declared flow against the stored one (by statement signature and output variable, as in 4.2a), derive the minimal set of insert/replace/drop operations, and apply them with the splice from 4.2b. An unchanged definition yields an empty patch and no write. This replaces `UpdateMicroflow`'s whole-document rebuild. | | | | -| 4.3 | **Drift detection as optimistic locking.** `describe` emits `@base '<fingerprint>'` (canonical BSON hash per unit, computed with `canon`). `create or modify` refuses when a present `@base` does not match, with `--force` to override. `exec` rewrites existing `@base` lines after applying. The check and the write are atomic on both backends (modelsdk and `--mcp`). A statement without `@base` behaves exactly as before. | M | grammar (annotation), `modelsdk/canon`, executor | a Studio Pro edit between `describe` and `modify` is refused; a control with no edit applies; `exec` updates the stamp | +| 4.3 | **Drift detection as optimistic locking (provisional; not on the beta gate).** Before building, validate `@base` with users and in a CI/CD pipeline (ADR-0012). `exec --no-stamp` skips rewriting stamps, for pipelines. `describe` emits `@base '<fingerprint>'` (canonical BSON hash per unit, computed with `canon`). `create or modify` refuses when a present `@base` does not match, with `--force` to override. `exec` rewrites existing `@base` lines after applying. The check and the write are atomic on both backends (modelsdk and `--mcp`). A statement without `@base` behaves exactly as before. | M | grammar (annotation), `modelsdk/canon`, executor | a Studio Pro edit between `describe` and `modify` is refused; a control with no edit applies; `exec` updates the stamp | | 4.4 | **A real dry run** (also the drift check for projects driven entirely by MDL, §8.2): `exec --dry-run`, replacing today's `diff`. Execute on an in-memory copy, diff canonical BSON per unit, and render changed units as a `describe` diff. Covers `alter` and every document type. | M | executor, `canon` | the §8.1 false negative (association storage) and false positives disappear | | 4.5 | **Opaque passthrough or refusal** for content MDL cannot express, per document type as the 0.1 harness finds it: `preserved <kind> '<id>'` in `describe` output, carried by replace. | M | per doctype | no silent loss remains in the harness | | 4.6 | **Bulk patches:** `alter microflows|pages in M where contains (<pattern>) { … }`, building on 4.2a. `alter pages … where` and `update widgets` are folded into it as aliases. | M | grammar, executor | — | diff --git a/docs/13-decisions/0010-mdl-canonical-syntax-rules.md b/docs/13-decisions/0010-mdl-canonical-syntax-rules.md index 85a280843..08bd0f860 100644 --- a/docs/13-decisions/0010-mdl-canonical-syntax-rules.md +++ b/docs/13-decisions/0010-mdl-canonical-syntax-rules.md @@ -1,6 +1,6 @@ # ADR-0010: MDL has one canonical form, governed by twelve syntax rules -- **Status**: Proposed +- **Status**: Accepted - **Date**: 2026-09-26 - **Related**: extends [ADR-0003](0003-mdl-is-sql-shaped.md); [PROPOSAL_mdl_beta_syntax_freeze.md](../11-proposals/PROPOSAL_mdl_beta_syntax_freeze.md) §3, §4, §7, §10; [ADR-0011](0011-mdl-language-versioning.md), [ADR-0012](0012-mdl-first-and-data-first-editing.md); PR ako/mxcli#702 diff --git a/docs/13-decisions/0011-mdl-language-versioning.md b/docs/13-decisions/0011-mdl-language-versioning.md index a2d3c7c09..272e03af9 100644 --- a/docs/13-decisions/0011-mdl-language-versioning.md +++ b/docs/13-decisions/0011-mdl-language-versioning.md @@ -1,6 +1,6 @@ # ADR-0011: MDL evolves through deprecation aliases and a language header; meaning changes only across versions -- **Status**: Proposed +- **Status**: Accepted - **Date**: 2026-09-26 - **Related**: [ADR-0010](0010-mdl-canonical-syntax-rules.md); [PROPOSAL_mdl_beta_syntax_freeze.md](../11-proposals/PROPOSAL_mdl_beta_syntax_freeze.md) §5, §6, §9, §10; PR ako/mxcli#702 diff --git a/docs/13-decisions/0012-mdl-first-and-data-first-editing.md b/docs/13-decisions/0012-mdl-first-and-data-first-editing.md index a4ffbf248..cba4ca3a8 100644 --- a/docs/13-decisions/0012-mdl-first-and-data-first-editing.md +++ b/docs/13-decisions/0012-mdl-first-and-data-first-editing.md @@ -1,6 +1,6 @@ # ADR-0012: MDL-first and data-first editing share one syntax and one patch engine -- **Status**: Proposed +- **Status**: Accepted - **Date**: 2026-09-26 - **Related**: builds on [ADR-0008](0008-identity-and-idempotence.md); [ADR-0010](0010-mdl-canonical-syntax-rules.md) (R1, R2, R12); [PROPOSAL_mdl_beta_syntax_freeze.md](../11-proposals/PROPOSAL_mdl_beta_syntax_freeze.md) §8, §9, §10; PR ako/mxcli#702 @@ -63,6 +63,7 @@ ADR-0008 made storage skip unchanged units. It cannot help when the rebuilt docu - **The graph splice is new.** Patching a flowchart graph (sequence flows, merges, layout) has no textual precedent we know of. It is the riskiest code in the plan, and it must be tested on Studio Pro-authored flows, because mxcli-authored ones cannot show identity loss. - **Diff-then-patch needs a stable way to match elements.** Where an activity has no output variable or caption, matching falls back to its statement signature. That is the same weakness React has for list items without `key` props. - **`@base` means `exec` rewrites source files**, which is unusual (most tools keep this state separately), and some users will object. It is optional and only updates stamps already present. +- **CI/CD pipelines may not support it.** Pipelines usually check out read-only and do not commit back. So a stamp that `exec` updates in CI is lost, and a stale stamp could make the next apply refuse. In pipelines, the dry-run check (decision 5, second bullet) is the drift guard, and `exec` must be able to skip rewriting stamps. - **A `preserved` placeholder in `describe` output is opaque to readers and to LLMs.** Each occurrence is a gap in what MDL can express, and must be tracked down. - The whole-document rebuild paths (`UpdateMicroflow` and others) have to be replaced, not patched. @@ -70,6 +71,7 @@ ADR-0008 made storage skip unchanged units. It cannot help when the rebuilt docu - The guidance for agents is "choose the mode by who owns the document". It becomes skill text, and eventually an enforced check through `@base`. - Three-way merge and per-element locking are natural extensions, deferred until after beta. +- **The `@base` mechanism (decision 5, first bullet) is explicitly provisional.** It was accepted with the reservation that users may dislike stamps in their scripts and that CI/CD pipelines may not support them. It is not on the beta gate. Before it is implemented, it is validated with users and in a pipeline. If it fails that test, a new ADR supersedes this decision with an alternative that keeps the lock outside the script. The candidates are a committed state file (Terraform) and the base stored alongside the model (Kubernetes' last-applied annotation). The rest of this ADR does not depend on the choice. ## Alternatives considered diff --git a/docs/13-decisions/README.md b/docs/13-decisions/README.md index dde0f810c..0ed55822f 100644 --- a/docs/13-decisions/README.md +++ b/docs/13-decisions/README.md @@ -103,9 +103,9 @@ This preserves the audit trail. | [0007](0007-mcp-read-model-session-overlay.md) | MCP backend read model — disk base with session overlay | Proposed | | [0008](0008-identity-and-idempotence.md) | Skip unchanged writes; never renumber element IDs in place | Accepted | | [0009](0009-tunnel-is-linux-only.md) | The embedded tunnel ships in Linux builds only | Accepted | -| [0010](0010-mdl-canonical-syntax-rules.md) | MDL has one canonical form, governed by twelve syntax rules | Proposed | -| [0011](0011-mdl-language-versioning.md) | MDL evolves through deprecation aliases and a language header; meaning changes only across versions | Proposed | -| [0012](0012-mdl-first-and-data-first-editing.md) | MDL-first and data-first editing share one syntax and one patch engine | Proposed | +| [0010](0010-mdl-canonical-syntax-rules.md) | MDL has one canonical form, governed by twelve syntax rules | Accepted | +| [0011](0011-mdl-language-versioning.md) | MDL evolves through deprecation aliases and a language header; meaning changes only across versions | Accepted | +| [0012](0012-mdl-first-and-data-first-editing.md) | MDL-first and data-first editing share one syntax and one patch engine | Accepted | ## Candidates to back-fill