TestApp in the round-trip harness; carry external-entity and OData-service content on create or modify (#743)

.claude/skills/fix-issue/findings/mdl-backend.jsonl

@@ -137,3 +137,4 @@
 {"area": "mdl/backend", "date": "2026-09-26", "symptom": "describe → exec of a snippet turns its Type 'Web' into '' (and would reset a non-default ExportLevel or canvas size)", "cause": "snippetToGen wrote Type, ExportLevel, CanvasWidth and CanvasHeight as constants on every write; the Type constant was \"\"", "file": "mdl/backend/modelsdk/snippet_write.go", "insight": "The snippet twin of #541 (carryStoredPageHeader). \"\" is not a member of PagesType (Native | Web), so every snippet mxcli created was also invalid. Carry the stored header on UpdateSnippet (width-agnostic canvas read — Studio Pro stores int64); a new snippet gets Web/Hidden/800×600, measured on all four Blank-template snippets", "refs": ["ako/mxcli#705", "#541"]}
 {"area": "mdl/backend", "date": "2026-09-26", "symptom": "describe → exec of a nanoflow detaches every annotation from its activity (AnnotationFlows 3 → 0, so the next describe prints the notes as free-floating) and deletes ExportLevel, UseListParameterByReference and ReturnVariableName; MarkAsUsed is cleared. Rules drop their annotation flows too", "cause": "nanoflowToGen and ruleToGen never wrote ObjectCollection.AnnotationFlows (microflowToGen did), and nanoflowToGen never set the three header keys; nanoflowFromGen did not read ReturnVariableName or MarkAsUsed and the builder hardcoded MarkAsUsed false", "file": "mdl/backend/modelsdk/nanoflow_write.go, rule_write.go, microflow.go; mdl/executor/cmd_microflows_build.go", "insight": "Three flow flavours share the object/flow converters but each has its own top-level writer, and each dropped a different subset — grep every *ToGen for the same list loop, not just the reported one. For keys with no MDL spelling and unknown version floors (Nanoflow.ExportLevel, UseListParameterByReference), carry ONLY the keys the stored unit has: nothing was ever written on create, Studio Pro fills an absent key, and an undeclared one makes the unit unopenable", "refs": ["ako/mxcli#705"]}
 {"area": "mdl/backend", "date": "2026-09-27", "symptom": "Running unchanged `describe user role` output (create or modify user role) rewrites Security$ProjectSecurity: /UserRoles[0] marker 2 -> 3; same for DemoUsers on a demo-user rewrite", "cause": "The encoder's default PartList marker is 3; Studio Pro writes 2 for ProjectSecurity.UserRoles and .DemoUsers (PedApp, TestApp, expr-checker agree). An untouched list passes through raw, so it only shows once an element in it is rewritten", "file": "mdl/backend/modelsdk/security_write.go", "insight": "A marker bug hides until the first rewrite of an element in that list; census the stored marker per (owner $Type, key) across Studio Pro fixtures and register it with RegisterPropertyListMarker", "refs": ["ako/mxcli#731"]}
+{"area": "mdl/backend", "date": "2026-09-27", "symptom": "Any write of a domain model (even rewriting a sibling entity) drops LocalizeDate from every DateTime attribute in the unit; external entity rewrites also rewrite Source/Key/Parts marker 2 -> 3", "cause": "attributeTypeToGen emitted DateTimeAttributeType without LocalizeDate and attributeTypeFromGen never read it; Rest$ODataKeyPart had no registered list marker so the encoder default 3 applied", "file": "mdl/backend/modelsdk/domainmodel.go, domainmodel_write.go", "insight": "A whole-unit rewrite re-serializes every sibling from the struct, so a property the struct cannot hold is lost on EVERY entity of the module, not just the edited one. Absent LocalizeDate means Mendix's default true (mxcli wrote it absent for years), so the reader must map absent to true or the fix would unlocalize every mxcli-authored DateTime. Only a Studio Pro fixture with DateTime attributes (TestApp) shows it", "refs": ["ako/mxcli#743"]}

.claude/skills/fix-issue/findings/mdl-executor.jsonl

@@ -733,3 +733,5 @@
 {"area": "mdl/executor", "date": "2026-09-27", "symptom": "describe microflow \u2026 with handles printed `-- handle: commit $Order on error {` for an activity with a custom error handler (TestApp Services.SaveOrder). The handle could not be used: an `alter microflow` target ends at the `{` that opens a fragment, so `insert after commit $Order on error { \u2026 }` parsed the handler brace as the fragment.", "cause": "printedStatement ended an action's statement at a line ending in `;` or `{` and kept the `{`, which belongs to the error-handler block describe opens, not to the statement.", "file": "mdl/executor/cmd_microflows_handles.go", "fix": "printedStatement strips the trailing `{` of an error-handler block opener, so the handle is `commit $Order on error`, which parses as a target and still matches the activity.", "insight": "A handle is only useful if it can be written back as a target in the grammar that consumes it; test a printed handle by parsing it, not only by resolving it.", "test": "TestPrintedStatement_ErrorHandlerBlockOpenerIsNotPartOfTheStatement"}
 {"area": "mdl/executor", "date": "2026-09-27", "symptom": "alter microflow (ako/mxcli#736) reported \"Altered microflow\" for statements mx check then rejected: two fragments in one statement declaring the same variable gave CE0111 Duplicate variable name; a fragment reading $X in the same statement as `drop $X` (either order) gave CE0109 Undefined variable. A loop fragment was refused as reading its own iterator, and drop/replace of a Studio Pro loop with more than one body activity was refused by the dangling-reference guard.", "cause": "The scope checks compared each operation with the flow as stored, never with what earlier operations of the same statement had declared, read or removed. The iterator of a fragment's loop is on its LoopSource, not an action output. Loop body flows are stored in the unit's Flows list, not in the loop, so removing the loop left them pointing at removed objects.", "file": "mdl/executor/cmd_alter_flow.go; mdl/backend/mfmutator/splice.go", "fix": "alterFlowContext tracks declaredByOps / readByOps / removedByOps across the statement's operations and checks each later operation against them; checkFragmentScope counts a fragment loop's iterator as its own; Drop and Replace remove graph.bodyFlows of a loop with it.", "insight": "A per-operation check against the stored document is only sound for a one-operation statement; every multi-operation test needs a case where operation 2 depends on operation 1. mx check on a copied TestApp is the cheap oracle: the CE numbers appear the moment a hygiene hole is hit.", "test": "TestAlterMicroflow_PedApp_ScopeSpansTheStatement; TestAlterMicroflow_PedApp_LoopFragmentDeclaresItsIterator; TestSplice_DropOrReplaceALoopTakesItsBodyFlows"}
 {"area": "mdl/executor", "date": "2026-09-27", "symptom": "describe of a published REST service with no resources printed `create or modify published rest service M.S ( \u2026 )` followed by a bare `;`, which does not parse (`mismatched input ';' expecting '{'`); found while making describe end every statement with `;` (ako/mxcli#744).", "cause": "The resource block `{ \u2026 }` is mandatory in the grammar (publishedRestService: \u2026 RPAREN LBRACE publishedRestResource* RBRACE), but the describer only printed it when the service had resources and otherwise closed the statement with `;` alone.", "file": "mdl/executor/cmd_published_rest.go", "fix": "Always print the block: `{\\n};` when there are no resources.", "insight": "A describe test that asserts substrings never parses the output; feeding every Describe*_Mock output through the parser (assertTerminated, #744) found this at once. The empty-collection branch of a describer is the one no fixture exercises.", "test": "TestDescribePublishedRestService_Mock (assertTerminated)"}
+{"area": "mdl/executor", "date": "2026-09-27", "symptom": "Running an external entity's describe output (create or modify external entity, either form) turns each attribute's Rest$ODataMappedValue into a DomainModels$StoredValue (RemoteName/RemoteType/Filterable... gone), re-mints attribute $IDs on the `from odata client` form, and rewrites String(36) as unlimited", "cause": "Both rewrite paths rebuilt attributes from the AST, which has no spelling for the OData mapping; the `from odata client` path minted fresh IDs; describe external entity printed GetTypeName() (`String`) instead of formatAttributeType", "file": "mdl/executor/cmd_entities.go (carryStoredAttributeState), cmd_odata.go", "insight": "Carry per attribute by name what the statement cannot spell, in the shared merge, and enumerate every path that rebuilds attributes: there were two statements for one document kind", "refs": ["ako/mxcli#743"]}
+{"area": "mdl/executor", "date": "2026-09-27", "symptom": "create or modify odata service over a Studio Pro service: EntitySet PageSize 10000 -> 0, entity sets reordered, member CanBeEmpty recomputed, ExportLevel deleted, AuthenticationTypes marker 1 -> 3", "cause": "The modify path rebuilt entity types/sets from the publish block (no spelling for PageSize without paging, set order, CanBeEmpty); the model had no ExportLevel; the writer used a marker-3 list copied from the legacy writer, not Studio Pro", "file": "mdl/executor/cmd_odata.go (carryPublishedEntityState), mdl/backend/modelsdk/odata_write.go", "insight": "Parity with the legacy writer is not evidence of what Studio Pro writes; measure markers on a Studio Pro fixture. A probe that runs describe output with the verb rewritten to create or modify (mdl/roundtrip TestTestAppCreateOrModifyProbe) shows what a verb switch would lose before switching it", "refs": ["ako/mxcli#743"]}

.claude/skills/mendix/write-microflows/SKILL.md

@@ -42,12 +42,12 @@ Choose the mode by who owns the microflow ([choose-edit-mode](../choose-edit-mod
 
 - **Created by your MDL scripts, and not edited in Studio Pro since:** edit the script
   (or fresh `describe` output) and re-run `create or modify`.
-- **Authored in Studio Pro:** there is **no `alter microflow` yet**, and re-emitting it
-  with `create or modify` renumbers element IDs, removes merges and resets connector
-  curves even for a one-line change. Keep the change minimal: put new logic in a new
-  sub-microflow and change the existing flow only to call it. Commit first, then
-  `describe` it again after `exec` and diff it against the original output. Anything
-  that differs and that you did not change is a loss.
+- **Authored in Studio Pro:** prefer `alter microflow X { insert/replace/drop … }`
+  (targets from `describe microflow X with handles`). `create or modify` of `describe`
+  output patches too: unchanged writes nothing; a top-level or `if`-branch statement
+  change is spliced in. Other changes (header, loop body, error handler, moved node)
+  rebuild the flow under mdl 0 (`MDL-V1-REBUILD`: IDs renumbered, merges and curves
+  lost) and are refused under `mdl 1;`.
 
 ## When to Use a Microflow vs a Nanoflow
 

.claude/skills/mendix/write-nanoflows/SKILL.md

@@ -23,12 +23,16 @@ Choose the mode by who owns the nanoflow ([choose-edit-mode](../choose-edit-mode
 
 - **Created by your MDL scripts, and not edited in Studio Pro since:** edit the script
   (or fresh `describe` output) and re-run `create or modify`.
-- **Authored in Studio Pro:** there is **no `alter nanoflow` yet**, and re-emitting it
-  with `create or modify` has dropped annotation links and changed the export level on
-  Studio Pro nanoflows, even with no edit at all. Keep the change minimal: put new logic
-  in a new nanoflow and change the existing one only to call it. Commit first, then
-  `describe` it again after `exec` and diff it against the original output. Anything
-  that differs and that you did not change is a loss.
+- **Authored in Studio Pro:** prefer `alter nanoflow X { insert/replace/drop … }`
+  (targets by output variable, caption or statement pattern). `create or modify` of `describe`
+  output also works as a patch: an unchanged definition writes nothing, and an inserted,
+  replaced or dropped statement (at the top level or in an `if` branch) is spliced in,
+  leaving every other node, merge and curve as stored. A change it cannot splice — the
+  header, anything inside a loop body or error handler, a moved node — rebuilds the
+  whole nanoflow under mdl 0
+  (warning `MDL-V1-REBUILD`: element IDs renumbered, merges removed, curves reset) and
+  is refused under `mdl 1;` (header and loop-body changes have no splice yet; move
+  nodes in Studio Pro).
 
 ## When to Use a Nanoflow vs a Microflow
 

.github/workflows/nightly.yml

@@ -54,9 +54,51 @@ jobs:
       - name: Setup mxbuild ${{ matrix.mendix-version }}
         run: ./bin/mxcli setup mxbuild --version ${{ matrix.mendix-version }}
 
+      # One step, every package: go test runs the packages in parallel, so the
+      # wall time is the slowest one's (mdl/roundtrip, ~21 min on this runner in
+      # ako/mxcli#757's measurement) and grows as the round trip gains fixtures.
+      # Per-PR CI splits this into parallel jobs instead (push-test.yml).
       - name: "Integration tests (Mendix ${{ matrix.mendix-version }})"
         run: make test-integration
-        timeout-minutes: 30
+        timeout-minutes: 60
+
+  # The upgrade property test over the WHOLE mdl-examples corpus, header-only
+  # scripts included (MXCLI_UPGRADE_ALL). Per-PR CI executes only the scripts
+  # the upgrade rewrites beyond the header and terminators; this is the run that
+  # proves langver's gating on the rest (ako/mxcli#757). Independent of the
+  # Mendix version — it runs on the committed PedApp fixture and never calls
+  # mx — so it runs once, not per matrix entry. It does not gate the release.
+  upgrade-full-corpus:
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        shard: ['1/3', '2/3', '3/3']
+    name: upgrade full corpus (${{ matrix.shard }})
+    steps:
+      - uses: actions/checkout@v7
+      - uses: actions/setup-go@v7
+        with:
+          go-version: '1.26.6'
+      - name: Cache ANTLR4 JAR
+        uses: actions/cache@v6
+        with:
+          path: ~/.m2/repository/org/antlr/antlr4
+          key: antlr4-4.13.2
+      - name: Install ANTLR4
+        run: pip install 'antlr4-tools==0.2.2'
+      - name: Generate parser
+        run: make grammar
+        env:
+          ANTLR4_TOOLS_ANTLR_VERSION: '4.13.2'
+      - name: Build
+        run: make build
+      - name: Upgrade property test, full corpus (${{ matrix.shard }})
+        run: make test-integration-upgrade
+        env:
+          MXCLI_UPGRADE_ALL: '1'
+          MXCLI_UPGRADE_SHARD: ${{ matrix.shard }}
+        timeout-minutes: 45
 
   nightly:
     needs: test

.github/workflows/push-test.yml

@@ -116,6 +116,10 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v7
+        with:
+          # testdata/testapp (ako/TestApp) is the round-trip harness's second
+          # Studio Pro-authored fixture (#743); without it that test skips.
+          submodules: true
       - uses: actions/setup-go@v7
         with:
           go-version: '1.26.6'
@@ -167,17 +171,101 @@ jobs:
         run: ./scripts/check-skill-mdl.sh ./bin/mxcli .claude/skills/mendix
       - name: Check docs-site MDL blocks
         run: ./scripts/check-skill-mdl.sh ./bin/mxcli docs-site/src
-      - name: Setup mxbuild
-        run: ./bin/mxcli setup mxbuild --version 11.12.2
-      # One engine since the legacy sdk/mpr backend was deleted
-      # (docs/plans/2026-09-14-retire-legacy-engine.md), so there is no matrix to
-      # narrow here any more and MXCLI_TEST_ENGINES is left unset.
-      - name: Integration tests
-        run: make test-integration
-        timeout-minutes: 30
+      # The integration tests are the `integration` job below.
       - name: Lint Go
         run: make lint-go
       - name: Vulnerability scan
         run: |
           go install golang.org/x/vuln/cmd/govulncheck@latest
           govulncheck ./...
+
+  # The integration tests, split into suites that run as parallel jobs
+  # (ako/mxcli#757). As one `make test-integration` step they took ~22 of its 30
+  # minutes once the upgrade property test landed, and the step's wall time was
+  # mdl/roundtrip's alone (~21 min) — go test already ran the packages in
+  # parallel, so no timeout below 30 min was going to hold as the round trip
+  # grows (#743 adds TestApp to it).
+  #
+  # Measured per package on the ubuntu runner (run 36319810164) and what each
+  # suite is expected to take, including ~3 min of checkout/build/mxbuild:
+  #   executor   mdl/executor                          ~15.5 min -> ~19 min
+  #   roundtrip  mdl/roundtrip, round-trip laws         ~1.7 min -> ~4 min
+  #   upgrade    upgrade property test, 3 shards        ~19 min  -> ~10 min each
+  #   other      cmd/mxcli{,/docker,/marketplace}       ~5.7 min -> ~8 min
+  # The full-corpus upgrade run (MXCLI_UPGRADE_ALL) is nightly, not here.
+  #
+  # Nothing here is dropped from per-PR CI: the round-trip laws and the
+  # execute-both upgrade test run at their default scope, the shards together
+  # executing every script exactly once (TestShardsPartition). Only the unit
+  # tests of packages WITHOUT integration tests are no longer re-run under the
+  # tag — `make test` in build-and-test runs those.
+  integration:
+    name: integration (${{ matrix.suite }}${{ matrix.shard && format(' {0}', matrix.shard) || '' }})
+    runs-on: ubuntu-latest
+    # Each suite's go test has its own -timeout (Makefile); this is the job's
+    # backstop, with room for the setup steps.
+    timeout-minutes: 45
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          - suite: executor
+            mxbuild: true
+          - suite: roundtrip
+          - suite: upgrade
+            shard: 1/3
+          - suite: upgrade
+            shard: 2/3
+          - suite: upgrade
+            shard: 3/3
+          - suite: other
+            mxbuild: true
+    steps:
+      - uses: actions/checkout@v7
+        with:
+          # testdata/testapp (ako/TestApp), the round trip's second Studio
+          # Pro-authored fixture (#743); only the roundtrip suite reads it, and
+          # without it that test skips.
+          submodules: ${{ matrix.suite == 'roundtrip' }}
+      - uses: actions/setup-go@v7
+        with:
+          go-version: '1.26.6'
+      - name: Cache ANTLR4 JAR
+        uses: actions/cache@v6
+        with:
+          path: ~/.m2/repository/org/antlr/antlr4
+          key: antlr4-4.13.2
+      - name: Install ANTLR4
+        run: pip install 'antlr4-tools==0.2.2'
+      - name: Generate parser
+        run: make grammar
+        env:
+          ANTLR4_TOOLS_ANTLR_VERSION: '4.13.2'
+      - name: Build
+        run: make build
+      # mdl/roundtrip runs on the committed PedApp fixture and never calls mx.
+      # One engine since the legacy sdk/mpr backend was deleted
+      # (docs/plans/2026-09-14-retire-legacy-engine.md), so MXCLI_TEST_ENGINES
+      # is left unset.
+      - name: Setup mxbuild
+        if: matrix.mxbuild
+        run: ./bin/mxcli setup mxbuild --version 11.12.2
+      - name: Integration tests (${{ matrix.suite }})
+        run: make test-integration-${{ matrix.suite }}
+        env:
+          MXCLI_UPGRADE_SHARD: ${{ matrix.shard }}
+        timeout-minutes: 30
+
+  # One stable check name for "every integration suite passed", whatever the
+  # matrix above becomes — a branch rule can require this instead of each leg.
+  integration-passed:
+    if: always()
+    needs: integration
+    runs-on: ubuntu-latest
+    steps:
+      - name: All integration suites passed
+        run: |
+          if [ "${{ needs.integration.result }}" != "success" ]; then
+            echo "integration: ${{ needs.integration.result }}"
+            exit 1
+          fi

.gitmodules

@@ -0,0 +1,3 @@
+[submodule "testdata/testapp"]
+	path = testdata/testapp
+	url = https://github.com/ako/TestApp.git