mxcli fmt --upgrade: registry rewrites, header-gated hook, execute-both property test

[ako]

Closes #735 (plan item 1.3, ADR-0011 decision 1).

Merges on top of the step-2 chain (#740, #745, #741, #738 via feature/731-create-or-modify). The branch contains that merge; the only conflict was docs-site/src/language/basics.md, where both sections are kept (the mdl 1 strictness table first, then fmt --upgrade).

What

• mdl/upgrade, the rewrite engine.
  • Aliases: every use the visitor records on Program.Deprecations becomes its canonical spelling. A keyword swap (DEPR001 create or replace → create or modify, DEPR002 show → list) happens in place, in the token's letter case. A structural rewrite (DEPR003/004, list call form → statement form) applies the visitor's ast.Fix. A use without a rewrite goes to Result.Unrewritten, with its reason.
  • Header-gated constructs: with AddHeader, every LanguageNote is rewritten before mdl 1; is added. gatedRewriters holds the codes that have a rewrite, unrewritable the ones that don't, with the reason. A construct with no rewrite blocks the header with HeaderBlockedError, which names every such construct and why. Nothing is written.
  • How rewrites are computed: the parse tree is the only place the tokens and their positions still exist, so the visitor that records a construct also computes its rewrite, as rune-offset edits (ast.Fix, mdl/visitor/visitor_upgrade_fixes.go). A Fix edits only the tokens that differ, so comments, layout and nested fixes survive. An occurrence that can't be rewritten carries NoFix, the reason.
  • Self-check: the output is re-parsed. It must parse, keep the statement count, and record no rewritable deprecation, so a second run is a no-op.
• mxcli fmt --upgrade [--header]. --header is opt-in while mdl 1 is a preview. Unrewritten uses are printed with their reason, and a refused header is an error that lists the constructs. This is documented in docs-site (language basics, with the table below) and in mxcli syntax language-header.

Rewrites (with --header)

Code	Rewrite
MDL-V1-SEMI	adds the missing ; right after the statement's last token, before any trailing comment
MDL-V1-SLASH	deletes the / line, or only the / token when it shares a line with the statement
MDL-V1-ESCAPE	writes the literal's mdl 0 value with '' as the only escape (see the cases below)
MDL-V1-LIMIT1	retrieve … limit 1 (one object) → retrieve … first
MDL-V1-SET	$x = e → set $x = e
MDL-V1-LIST	a call mdl 0 turns into an activity becomes the statement form (set $T = head($L) → $T = head $L, $F = find($L, Name = 'x') → $F = find $L by Name = 'x'). For find/contains on a variable declared String (a parameter or declare), mdl 0 builds the string function, so the call stays as it is and gains set
MDL-DEPR003 / MDL-DEPR004 (no header needed)	call form → statement form for every list operation and aggregate, including subtract($A,$B) → subtract $B from $A, sum($L.Attr) → sum $L by Attr, reduce(…, initial:, returns:) → reduce $L from … as … using …; by vs where exactly as the call form decided
MDL-V1-REPLACE02	create or replace user role / demo user (a plain create under mdl 0) → create …

MDL-V1-ESCAPE depends on where the literal sits:

• Plain literal, or one inside a re-rendered expression: rewritten to its mdl 0 value.
• Inside an expression stored as written (one that spans lines): no edit. mdl 0 already passes the backslash through to Mendix, so it means the same under mdl 1.
• Escaped line break inside a re-rendered expression: reported. Writing the break into the literal makes the builder store the expression as written. The execute-both test caught this: a log message stops being the template text and becomes a {1} parameter (bug-tests/264-log-node-expression-roundtrip.mdl).

Known-unrewritable

These are reported by fmt --upgrade --header, which then refuses the header. Tests list them explicitly, and each list may only shrink.

• Whole codes (unrewritable in mdl/upgrade/gated.go; TestGatedRegistryIsComplete holds every visitor.LanguageChanges() code to exactly one of gatedRewriters / unrewritable, and fails if a code is added to unrewritable):
  • MDL-V1-PROP: an unknown property key is ignored under mdl 0. Which key was meant can't be guessed.
  • MDL-V1-PROPVALUE: a mis-shaped value is ignored, or read by its shape. What was meant can't be guessed.
  • MDL-V1-REPLACE01: create or replace view entity drops and recreates under mdl 0, and no mdl 1 statement does that. create or modify keeps the identity the old meaning discards, and a drop in front would fail where the old statement created the entity.
• Occurrences of codes that otherwise have a rewrite (each carries its reason):
  • a nested list operation or non-variable operand (count(filter(…))), which needs a variable name nobody chose;
  • find/contains on a variable whose type the script does not state, e.g. a call's result;
  • an escaped line break inside a re-rendered expression;
  • a reduce whose seed (which moves) holds a backslash-escaped string;
  • a sum($L/Path/Attr) whose attribute is not a plain name.
• Corpus scripts that keep mdl 0 (keepsItsVersion in mdl/upgrade/examples_test.go): bug-tests/1101-nested-list-operand-dropped.fail.mdl (MDL-V1-LIST, nested calls) and bug-tests/264-log-node-expression-roundtrip.mdl (MDL-V1-ESCAPE, line break in a log message).
• Different AST, same model (buildsTheSameModelNotTheSameAST): ledger-53-string-contains.mdl and ledger-63-string-find.mdl. Under mdl 0, a List operation on a String input is turned into the string function by the flow builder. Under mdl 1 it is set $x = contains(…) directly. The execute-both test proves both write the same model.

Design choices the ADRs did not settle

1. --upgrade does not run the heuristic formatter. It upper-cases identifiers (Issue64.User → Issue64.USER) and changes what 374 of 701 example scripts build. TestFormatterChangesMeaning pins this.
2. The rewrite is textual, at parser positions, not an AST re-print. Keyword swaps keep the replaced token's case, and inserted keywords follow the operation keyword's case (FIND(…) → FIND $L WHERE …).
3. The header goes on line 1. A construct without a rewrite makes --header fail rather than add the header and carry on.
4. The property test runs the upgrade with the header. A script whose header is refused is upgraded without it, so its alias rewrites are still proven.
5. The default property run skips terminator-only upgrades. It does not execute scripts whose only edits are the header, ; and / lines: terminators never reach the AST, and the examples test proves all 701 scripts build the same statements. MXCLI_UPGRADE_ALL=1 executes everything (run locally, see below). This keeps the integration step within budget.

Test plan (what I ran, on the merged tree)

• make build and make lint pass.
• go test ./mdl/upgrade ./mdl/visitor ./mdl/deprecation ./mdl/langver ./cmd/mxcli passes.
  • Examples test: 715 scripts. 701 parse and 233 need an alias rewrite; 2 keep mdl 0. For each script, with and without the header: the output parses, records zero deprecations, builds the same statements, and fmt --upgrade is idempotent.
  • New unit tests: TestUpgrade_GatedRewrites (21 cases), TestUpgrade_UnrewritableBlocksTheHeader (5), TestUpgrade_ListCallFormToStatementForm (19, AST-equal), TestGatedRegistryIsComplete, TestLanguageChangesListsEveryDeclaredChange (reads the visitor's sources, so a new langver.Change can't escape the registry test), and TestFmtUpgrade_ReportsWhatItCannotRewrite.
• go test -tags integration ./mdl/upgrade/ ./mdl/roundtrip/ passes. TestUpgradeExecutesToTheSameModel takes about 6 min: 224 scripts execute to the same model and 25 are out of scope.
• MXCLI_UPGRADE_ALL=1 (all scripts, 15 min): 587 execute on PedApp and upgrade to the same model, 114 are out of scope (the original does not execute on PedApp), 2 keep mdl 0, 14 do not parse.
• Revert checks. Each mutation below made the named test fail; I restored the code after each.
  • SEMI fix removed → GatedRewrites/missing_semicolon, examples.
  • LIMIT1 fix removed → GatedRewrites/limit_1…, examples.
  • set not deleted in the set-call rewrite → GatedRewrites/set_with_a_list_call, examples.
  • subtract operands not swapped → ListCallForm/subtract (AST differs), examples.
  • by/where choice ignored → ListCallForm/filter…and…, GatedRewrites/find…by_expression, examples.
  • String operand treated as a list → GatedRewrites/find_on_a_String…, examples (stale buildsTheSameModelNotTheSameAST entry).
  • Escape rewritten inside a stored-as-written expression → GatedRewrites/escape_in_an_expression_stored_as_written, examples.
  • Line-break escape allowed in an expression → UnrewritableBlocksTheHeader/escaped_line_break…, examples. Before that guard existed, the execute-both test failed on 264-log-node-expression-roundtrip.mdl (Microflows$Microflow MF_LogMultilineMessage written differently).
  • Code added to unrewritable → TestGatedRegistryIsComplete.
  • Change dropped from LanguageChanges() → TestLanguageChangesListsEveryDeclaredChange.
• Existing controls (TestUpgradeExecuteBoth_Controls) pass unchanged.
• No Studio Pro verification: the change rewrites script text and writes no BSON. The execute-both test is the model-level proof.

Followups (not in this PR)

• find/contains on an operand whose type the script does not state could also be rewritten to the statement form. Today's flow builder still turns a String input into the string function under mdl 1, so the model would be the same, but the mdl 1 text would read as a List operation on a String. It is reported instead.
• The visitor stores a multi-line string literal as a source-preserved expression, so under mdl 1 a log message with a line break can't be a plain template literal. That's why 264-log-node-expression-roundtrip.mdl keeps mdl 0.
• Register MDL065 (legacy case/else in split type) in mdl/deprecation; plan item 3.5 (show <single thing> → describe); plan item 3.2 (fix formatter.Format's identifier upper-casing, then let fmt --upgrade format).

🤖 Generated with Claude Code

[ako]

Independent review: CI's Integration tests step timed out at 30 min (run 36310586491) because the execute-both test ran the whole corpus (~15 min locally; main's roundtrip package takes 85s on CI). Pushed f35637a: by default only the 223 scripts the upgrade rewrites beyond the header execute (202 same model, 21 out of scope, ~5 min locally); the 478 header-only scripts are listed and run with MXCLI_UPGRADE_ALL=1. Verified locally: unit tests, full roundtrip integration suite, make lint, and five revert checks on mdl/upgrade (case matching, re-parse check, gated-rewrite refusal, no-rewrite reporting, header already present) each fail the named tests.

🤖 Generated with Claude Code