Skip to content

TestApp in the round-trip harness; carry external-entity and OData-service content on create or modify (#743) - #759

Merged
ako merged 15 commits into
mainfrom
feature/743-testapp-roundtrip-com
Sep 27, 2026
Merged

ako merged 15 commits into
mainfrom
feature/743-testapp-roundtrip-com

Conversation

@ako

@ako ako commented Sep 27, 2026

Copy link
Copy Markdown
Owner

Part of #743 (tracking #714). Scope items 1 and 3 are done, and item 2 is done for the OData service and the external entity. The workflow and OData client rewrites are not carried yet; their refusal now gives the right advice. The remainder is listed at the end.

What changed

1. ako/TestApp as a round-trip fixture (test(roundtrip) commit)

  • testdata/testapp is a git submodule pinned at 11a8fca. TestTestAppRoundTrip runs it with its own allowlist in mdl/roundtrip/testapp_allowlist_test.go; like PedApp's, that list may only shrink. When the submodule is not initialised, the test skips and says how to initialise it.
  • The harness now takes a fixture (dir, .mpr, allowlist). PedApp's test and allowlist are unchanged.
  • New unitKeywords mappings: Rest$ConsumedODataService → odata client, ODataPublish$PublishedODataService2 → odata service. External entities are also enumerated as external entity. Workflows were already mapped.
  • New TestTestAppCreateOrModifyProbe. For each kind whose describe still prints a plain create (workflow, odata client, external entity), it runs the describe output with the verb rewritten to create or modify, under the same laws and with its own allowlist. A kind switches its verb only when none of its documents is on that list. This test is the evidence behind item 3.
  • CI: submodules: true on the build-and-test checkout, and nothing else, because CI: integration step is near its 30-minute budget #757/CI: split integration tests into parallel suites; shard the upgrade property test (#757) #758 are changing the same workflow.

2. Carry

  • Domain model / external entity (fix(domain model)):
    • LocalizeDate is now read and written. Before, every DateTime in a rewritten domain model lost it, including DateTimes of sibling entities.
    • An absent LocalizeDate is read as true, Mendix's default and what mxcli has always written. Producers that don't know the value (contract import, MCP reader, API builder, System module) set true.
    • Rest$ODataKey Parts use list marker 2.
    • Both create or modify external entity forms now carry each attribute's OData mapping, its mapped default and LocalizeDate, matched by name. The from odata client form also keeps attribute $IDs.
    • describe external entity prints String(36) instead of String.
  • OData service (fix(odata service)):
    • The modify path carries entity-set PageSize when the statement does not page, the stored entity-set order, and each member's CanBeEmpty.
    • ExportLevel is read, and written only when stored.
    • AuthenticationTypes uses marker 1, which is what Studio Pro wrote on all 3 TestApp services. The old marker 3 matched the legacy writer, not Studio Pro.
  • Refusal advice (fix(refusal)): a plain create of an existing workflow or OData client now points to alter workflow / alter odata client and says why create or modify is not yet safe. The external-entity and OData-service messages still suggest create or modify, because those rewrites now carry.

3. Verb switch. describe odata service now emits create or modify. All 3 TestApp services passed the probe, and they now pass both laws in the main round trip. Workflow, OData client and external entity keep the plain create.

Design choices the ADRs did not settle

  • Absent LocalizeDate reads as true. mxcli has always written DateTimes without the key. Reading absent as false would have made the fix unlocalize every mxcli-authored DateTime on the next write of its domain model. Date is unchanged: it still writes no key. Making Date write false would change what it means, so under ADR-0011 it belongs behind mdl 1. Not done here.
  • The probe rewrites the verb instead of changing describe. Losses can be measured before the switch, and the switch is a separate, visible allowlist change.
  • ExportLevel and CanBeEmpty are carried only when stored. A service mxcli creates gets the same BSON as before.
  • External entity keeps the plain create, although its carry is done. Its probe now fails only on the NoGeneralization Has*Attr=false flags. That is Round-trip harness: untracked describe → exec losses on the Studio Pro fixture (microflows, entities, pages, snippets, menus, roles, JS actions) #721 B, shared by every entity, and it is fixed there, not here.

Test plan (all run locally)

  • make build, make lint (Go and TypeScript): pass.

  • go test ./mdl/executor/ ./mdl/backend/... ./model/... ./api/...: pass.

  • go test -tags integration ./mdl/roundtrip/ (PedApp, TestApp, probe, upgrade property test): pass, 808 s.

  • go test -tags integration ./mdl/backend/modelsdk/ ./modelsdk/... ./api/...: pass. Also passing: the integration tests in documentation_preserved_test.go and odata_role_grants_test.go.

  • Skip path: with testdata/testapp/TestApp moved away, both TestApp tests SKIP and the package passes.

  • Revert checks. For each of the following, I removed or stubbed the fix and watched its test fail:

    • LocalizeDate write, read, and absent → true.
    • ODataKey marker.
    • Mapped default read.
    • ExportLevel write.
    • CanBeEmpty write and read.
    • carryStoredAttributeState in the merge, and the mapped-default carry.
    • External-entity from odata client carry.
    • describe external entity type.
    • carryPublishedEntityState.
    • The refusal messages failed their tests before the change.

    At harness level:

    • Reverting the OData service verb makes all 3 TestApp services break exec.
    • Removing the service carry makes all 3 break getput.

    The AuthenticationTypes marker test failed with marker = 3 before the fix.

  • Studio Pro check: I ran mx check (mxbuild 11.14.0, TestApp's version) on a copy of TestApp. Then I executed the describe output of the 3 OData services, Clients.Orders, Odata.Devices and Clients.Customer, and ran mx check again. Both runs: 0 errors. I did not use the Studio Pro MCP tunnel.

  • CI time: the TestApp round trip adds about 6.5 min, and the probe about 12 s. CI: integration step is near its 30-minute budget #757/CI: split integration tests into parallel suites; shard the upgrade property test (#757) #758 own the CI time budget.

Remaining for #743 (not in this PR)

  • Workflow: carry EventSubProcesses, outcome Flows, activity names and sizes on create or modify workflow, or refuse it. Then switch the verb.
  • OData client: carry UseQuerySegment, ApplicationId, EndpointId, CatalogUrl, Icon, the proxy settings, Error/HeaderList/ConfigurationEntity microflows, CustomLocationTemplate, and the MetadataReferences/HttpHeaderEntries/ValidatedEntities markers, or refuse. Then switch the verb.
  • External entity: switch the verb once Round-trip harness: untracked describe → exec losses on the Studio Pro fixture (microflows, entities, pages, snippets, menus, roles, JS actions) #721 B (NoGeneralization Has*Attr flags) lands.
  • An explicit create or modify workflow or create or modify odata client written by a user still rewrites lossily. Only the plain-create advice changed.

🤖 Generated with Claude Code

ako and others added 10 commits September 27, 2026 13:37
Round-robin over the scripts that reach execution; every shard still
upgrades and checks every script. TestShardsPartition proves the shards
together execute each script exactly once. Locally 75+74+75 = 224, the
unsharded count.

Part of #757.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… nightly

The single make test-integration step took ~22 of its 30 minutes, its
wall time being mdl/roundtrip alone. Per-PR CI now runs executor,
roundtrip, upgrade (3 shards) and other as parallel jobs, with an
integration-passed aggregate. Nightly raises its step to 60 min and runs
MXCLI_UPGRADE_ALL in 3 shards once.

Closes #757.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rewrite (#743)

On the Studio Pro-authored ako/TestApp, running an entity's describe output
changed more than the entity:

- every DateTime attribute of the rewritten domain model lost LocalizeDate:
  the writer never set it and the reader never read it. The reader maps an
  absent LocalizeDate to true (Mendix's default, and what mxcli wrote until
  now), and producers that do not know it set true;
- an external entity's Rest$ODataKey Parts list went from marker 2 (what
  Studio Pro writes) to 3;
- `create or modify external entity`, in both forms, rebuilt each attribute
  without its OData mapping, so Rest$ODataMappedValue became a plain
  StoredValue; the `from odata client` form also re-minted attribute $IDs.
  The rewrite now carries, by name, the mapping, the mapped design-time
  default and LocalizeDate (carryStoredAttributeState);
- describe external entity printed `String` for a String(36), which executes
  as unlimited.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…create or modify (#743)

Running a published service's describe output as `create or modify` over the
three Studio Pro-authored services of ako/TestApp:

- set every entity set's PageSize 10000 -> 0 (describe prints PageSize only
  with UsePaging), reordered the entity sets into entity-type order and
  recomputed each member's CanBeEmpty. The modify path now carries all three
  from the stored service (carryPublishedEntityState); CanBeEmpty is read and
  written when stored, and derived from the key as before when not;
- deleted ExportLevel: the model had no field. It is read, and written only
  when stored, so a service mxcli creates is unchanged;
- wrote AuthenticationTypes with marker 3. Studio Pro writes 1 on all three;
  the 3 matched the legacy writer, not Studio Pro.

With that the rewrite writes nothing on all three, so describe switches to
`create or modify odata service`; the round-trip harness holds it to both laws
on TestApp. mx check on a TestApp copy after running the describe output of the
three services: 0 errors before and after.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… modify (#743)

A plain `create` of an existing workflow or OData client is refused, and the
refusal said "use create or modify". For these two types that is the rewrite
that still loses Studio Pro-authored content (event sub-processes and outcome
flows; UseQuerySegment, catalog, proxy and microflow settings, icon). The
message now points to `alter workflow` / `alter odata client` and says why the
whole-document rewrite is not yet safe.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e-or-modify probe (#743)

PedApp has no workflow, OData client or service, or external entity, so the
harness could not see what a rewrite of them loses. ako/TestApp, Studio
Pro-authored, is now pinned as the submodule testdata/testapp (11a8fca) and
round-tripped by TestTestAppRoundTrip with its own allowlist (may only shrink).
The tests skip, saying how to initialise it, when the submodule is absent; CI
checks it out (submodules: true, build-and-test only).

- the harness takes a fixture (dir, .mpr, allowlist); PedApp is unchanged;
- unitKeywords maps Rest$ConsumedODataService (odata client) and
  ODataPublish$PublishedODataService2 (odata service); external entities are
  also enumerated as `external entity` (workflows were already mapped);
- TestTestAppCreateOrModifyProbe runs the describe output of each kind that
  still prints a plain `create` (workflow, odata client, external entity) with
  the verb rewritten to `create or modify`, under the same laws and its own
  allowlist. A kind switches its describe verb once none of its documents is
  listed there; odata service switched on that evidence.

Measured at introduction: 142 of 775 TestApp documents keep both laws; the
rest are the #721 classes on a larger fixture, or the three deferred kinds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ext.apply

A pure refactor so create or modify can apply a derived patch through the
same checks and splice an alter statement uses (#747). No behaviour change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An existing flow is no longer rebuilt. The stored flow is described and
parsed back, compared with the declared definition statement by statement
(LCS over declaredMatches, recursing into if branches), and the minimal
insert/replace/drop is applied through the #739 splice via the same
alterFlowContext an alter statement uses. An unchanged definition writes
nothing and reports Unchanged.

A change the splice cannot express (header, loop or error-handler body,
a moved node) is refused under mdl 1 and still rebuilt under mdl 0 with the
MDL-V1-REBUILD warning (ADR-0011).

Strikes all #721 class A microflow and nanoflow entries from the PedApp
round-trip allowlist and the two nanoflow entries from studioProKnownLossy.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
create or modify re-parsed the stored flow's description under the
script's header. describe writes mdl 0, so under `mdl 1;` a stored line
break described as `\n` read as a backslash and an n; a script stating
exactly that then matched, reported Unchanged and wrote nothing, although
the value it states differs from the stored one.

The stored side is now described and parsed as mdl 0 always; the AST holds
values, so it compares with a declared side parsed under any version.
correctAmbiguousRanges applies in every case. Tests that exercised mdl 1
on VAL_Feedback relied on the misreading and now use a flow without a
backslash escape for their mdl 1 leg; TestPedAppListActivitiesUnderMdl1
asserts the mdl 1 description of a flow of list activities is Unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… loop (#747)

A declared loop that differed from the stored one only inside its body
became a replace of the whole loop: every node in it rebuilt with new
element IDs, and merges describe cannot show dropped - the rebuild's loss,
confined to the loop, and under mdl 1 without the refusal the design and
the skills promise for it. It is now a change the splice cannot make:
refused under mdl 1, the warned rebuild under mdl 0. A changed loop
iteration is still a replace.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ako and others added 5 commits September 27, 2026 16:09
… the 700-line skill budget

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…patch fixes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant