Skip to content

create or modify everywhere: describe emits it, or replace is its alias under mdl 1, module role in createStatement (#731) - #738

Merged
ako merged 36 commits into
mainfrom
feature/731-create-or-modify
Sep 27, 2026
Merged

ako merged 36 commits into
mainfrom
feature/731-create-or-modify

Conversation

@ako

@ako ako commented Sep 27, 2026

Copy link
Copy Markdown
Owner

Part of #731 (plan item 2.1, ADR-0010 R1). Tracking: #714.

What this does

  • describe emits create or modify for every document type. Newly moved: JavaScript action, user role, layout, REST client, OData client/service, external entity, database connection, data transformer, agent/model/knowledge base/consumed MCP service, workflow, validation rule, module; navigation create or replace → create or modify.
  • What that unmasked, fixed so the output really runs unchanged:
    • Security$ProjectSecurity.UserRoles/DemoUsers were re-encoded with marker 3; Studio Pro writes 2 (PedApp, TestApp and expr-checker agree). Registered.
    • JavaScript action rewrite now carries what describe cannot print, as the Java twin does: the "source not available" placeholder is not written as the .js source, and parameter Description/Category are carried by name.
    • carriedDocumentation: a doc comment that is exactly the stored text's normal form (no blank lines / trailing spaces / CRLF, which /** */ cannot spell) keeps the stored bytes.
  • Allowlist shrinks: the whole Round-trip harness: untracked describe → exec losses on the Studio Pro fixture (microflows, entities, pages, snippets, menus, roles, JS actions) #721 F class (45 JavaScript actions, 2 user roles) is struck, plus the putget law of microflow Administration.NewWebServiceAccount (fixed by the doc-comment carry). 101 of 223 PedApp documents round-trip.
  • create or replace under mdl 1 means create or modify on a view entity (identity-carrying rewrite instead of delete+recreate) and on user role / demo user (instead of a plain create); each is then a plain MDL-DEPR001 alias. Under mdl 0 the alpha meaning is kept and MDL-V1-REPLACE01 / MDL-V1-REPLACE02 warn (ADR-0011). Only translations stays exempt from the alias (it replaces the whole set).
  • create module role is a createStatement kind (so or replace/doc comment work like everywhere else); drop module role if exists added. Docs/skill/help updated.

Design choices the ADRs did not settle

  • The view-entity/role change is a change of meaning, so it is gated on the header rather than applied to headerless scripts; the language-change codes are MDL-V1-REPLACE01/02 (no convention existed yet beyond the MDL-V1-… example in visitor_language_header.go).
  • GUID fixture: PedApp has no view entity, and Studio Pro's MCP cannot author one (mxcli mcp capabilities: view entities not authorable; no save tool). The ako/TestApp clone already has a Studio Pro-authored view entity, MyFirstModule.VCar (GUID != $ID on entity and attributes). New fixture testdata/testapp-views (360 KB) is that app trimmed to the project root + MyFirstModule units; README documents what was removed.
  • create module has no modify behaviour (existing module is a no-op), so describe's create or modify module is just the canonical spelling.

Test plan (what I ran)

  • make build, make lint (Go + TS) — pass.
  • go test ./mdl/executor/ ./mdl/visitor/ ./mdl/grammar/ ./mdl/deprecation/ ./mdl/langver/ ./mdl/backend/... ./modelsdk/codec/ ./cmd/mxcli/ — pass.
  • go test -tags integration ./mdl/roundtrip/ — pass with the struck allowlist (includes new TestViewEntityCreateOrModifyKeepsGUID).
  • Manual round trip on a copy of the TestApp clone (describe → exec) for odata client/service, rest client, workflow, layout, JS action, external entity, user role, demo user, module role, published REST, navigation, module, entity, association.

Revert checks (each test fails with the fix reverted, passes with it):

  • describe spelling: reversing the 15 emitter edits fails 18 describe tests (incl. new parse-based TestDescribe{JavaScriptAction,UserRole,Navigation}_EmitsCreateOrModify).
  • TestProjectSecurity_UserRoleAndDemoUserListMarkers: markers 3 without the registration; control asserts the fixture has 2.
  • TestJavaScriptActionBody_PlaceholderIsNotWritten, TestJavaScriptActionRewrite_CarriesParameterDescription: fail with guard/carry disabled; each has a control.
  • TestCarriedDocumentation_KeepsStoredWhenOnlyTheSpellingDiffers failed before the change; reverting it in the harness makes NanoflowCommons.SignOut and NewWebServiceAccount fail again.
  • TestViewEntityCreateOrModifyKeepsGUID: with the mdl 1 gate disabled, the mdl 1 case re-mints entity + attribute GUIDs; the mdl 0 or replace case is the control that proves the assertion detects a lost GUID; precondition asserts GUID != $ID.
  • TestCreateOrReplaceMatchesModifyExceptExemptKinds now runs under mdl 0 and mdl 1; disabling the gate fails the three mdl 1 cases.
  • TestDropModuleRole_IfExistsSkipsMissingRole: fails with the IfExists branch disabled; the plain drop is the control.

Not in this PR (follow-ups)

  • if not exists on every document type (today entity/association only).
  • create or modify user role is additive and ignores manage all roles on an existing role; R1 "make it match" semantics would be an mdl 1 change.
  • Found while round-tripping TestApp: REST client describe prints Response: json without as $var (parse error); navigation describe output fails the reference check on System.Images.* icons; view-entity describe of an AutoNumber source column declares Long and fails the reference check; demo user describe prints password '***' which exec would store; JS action rewrite drops the .js EXTRA CODE section when the source is readable.

🤖 Generated with Claude Code

ako and others added 16 commits September 27, 2026 08:21
Rewriting a user role or demo user re-encoded Security$ProjectSecurity's
UserRoles/DemoUsers lists with marker 3; Studio Pro writes 2 (PedApp,
TestApp and expr-checker agree). Running unchanged describe user role
output therefore wrote the unit (#731).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With describe emitting create or modify, its output re-executes. Two
things it cannot express were lost on that rewrite, as the Java twin
already handled:
- the placeholder body printed when the .js source is unreadable was
  written as the action's source;
- parameter Description/Category (printed as a comment) were dropped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ation

A /** */ comment cannot spell blank lines, trailing whitespace or CRLF, so
describe output of Studio Pro prose restated it in normal form and the
rewrite overwrote the stored bytes with nothing changed. When the stated
text is exactly the stored text's normal form, keep the stored bytes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
describe printed a plain create for JavaScript actions, user roles,
layouts, REST/OData clients, OData services, external entities, database
connections, data transformers, agent-editor documents, workflows,
validation rules and modules, and create or replace (the deprecated
spelling) for navigation. A plain create fails with "already exists" on
the document it describes, so describe output did not run unchanged.

Strikes the #721 F class (45 JavaScript actions, 2 user roles) from the
round-trip allowlist, plus the putget law of
Administration.NewWebServiceAccount, fixed by the doc-comment carry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…731)

Under mdl 1, create or replace on a view entity is the identity-carrying
rewrite create or modify uses, instead of delete and recreate, and on a
user role or demo user it is create or modify instead of a plain create.
Each is then a plain MDL-DEPR001 alias. Under mdl 0 (no header) the alpha
meaning is kept and MDL-V1-REPLACE01/02 warns (ADR-0011).

The GUID test runs on testdata/testapp-views, a trimmed copy of ako/TestApp
whose view entity MyFirstModule.VCar was authored in Studio Pro (GUID !=
$ID); mdl 0 create or replace is the control that re-mints the GUID.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…dule role if exists (#731)

create module role was its own securityStatement rule with only an
or modify prefix, so it took none of the prefixes other document types
get: or replace was a parse error and a doc comment did not attach. It is
now a createStatement kind (or replace is the MDL-DEPR001 alias there),
and drop module role takes if exists like drop user role.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t mandatory under mdl 1

Each Studio Pro List operation / Aggregate list activity is one statement
whose operand is a variable, so nesting cannot be written:

  $Open = filter $Orders by Status = M.Status.Open;
  $Big  = filter $Orders where $currentObject/Total > 1000;
  $N    = count $Open;
  $Csv  = reduce $Orders from '' as String using $currentResult + $currentObject/Name;

The statement forms parse under every version. The call forms keep
parsing: a respelling everywhere except find/contains, registered as
MDL-DEPR003 (list operations) and MDL-DEPR004 (aggregates), both building
the same AST. find(...)/contains(...) clash with the string functions, so
they, a list call after `set`, and a nested call are version-gated
(MDL-V1-LIST): kept and warned under mdl 0, refused under mdl 1, where
`set $x = find(...)` is always the string function. A reassignment without
`set` is refused under mdl 1 and warned under mdl 0 (MDL-V1-SET).

`where` is always Find/Filter by expression; `by` and the call form keep
the by-member reading when the condition is `Member = value`
(ast.IsMemberEquality, shared by visitor and flow builder).

Part of #733.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
describe writes the language of the newest frozen version, or of the
script it runs in when that is newer. While mdl 1 is a preview a plain
describe keeps the call form; inside an `mdl 1;` script it prints the
statement form, which executes back to the same microflow (PedApp test).

Part of #733.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…forms

Part of #733.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rries

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…lementIDs, Writer.UpdateRawUnitPatch)

TransplantIDs pairs elements by type and position, which re-pairs the
surviving flows of a patched microflow onto their neighbours' $IDs after a
drop. A write that started from the stored bytes skips it; elision and the
storage-GUID guard still apply.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e, replace, drop)

Splices a fragment into the raw stored unit: appends only the fragment's
objects and flows, rewires the flow around the target by its pointers and
the rewired end, moves nodes past the insertion point to make room, and
never rewrites an $ID. Save refuses a unit in which anything still points at
a removed element or two elements share an $ID. Refuses what it cannot do
safely: after a decision, before a join, inside a loop body, drop/replace of
a decision or of an activity with an error handler, and a placement that
would overlap an object. The modelsdk backend writes through
UpdateRawUnitPatch (canon.Reconcile).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…#736)

alter microflow|nanoflow M.F { insert after|before <target> { … }
replace <target> with { … } drop <target>; }. Targets are the #713 content
addresses, resolved against the stored flow before any operation runs.
Fragments are built with the create-microflow builder, seeded with the
flow's variables, and cut out of their start and end events. A fragment
variable that clashes with one the flow has, or one it reads that is not
declared upstream of the insertion point, is an error; so is dropping an
activity whose output is still read. Acceptance on PedApp VAL_Feedback:
only the new log, the two flows around it and the shifted positions differ;
an empty alter writes nothing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… handle

A handle has to parse as an alter target, and a target ends at the { of a
fragment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Runs the shared splice on the stored flow and sends the difference as
ped_update_document path operations in one update, after checking the live
document still matches the .mpr (PED addresses entries by index). Drop and
replace are refused: PED does not roll back a removal when an update fails.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ce alone

describe now emits create or modify javascript action (#731), so its output
reaches the rewrite, which regenerated the .js file from the statement and
dropped the import list and the EXTRA CODE section. On TestApp,
NanoflowCommons.GetStraightLineDistance lost import { Big } and the deg2rad
helpers its user code calls. When the statement restates the stored user code
and parameter names, the file is left as it is.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ako

ako commented Sep 27, 2026

Copy link
Copy Markdown
Owner Author

Independent review. I pushed a9d617b (JS action: running unchanged describe output leaves the .js file alone). All unit, round-trip and lint checks pass on the branch, and I reverted each claimed fix to confirm its test fails without it.

Blocking: on main, describe printed a plain create for these types, and running that output was refused ("already exists"). With create or modify it now runs, and on the Studio Pro-authored ako/TestApp clone it silently rewrites the document with losses. I compared the BSON of each copy before and after one describe → exec:

  • workflow workflow.Workflow1_2: EventSubProcesses removed entirely. Outcome Flows dropped. Activity names re-spelled (start1 → Start). Workflow1 loses the same things inside its event sub-process.
  • external entity Clients.Orders: every attribute's Rest$ODataMappedValue becomes DomainModels$StoredValue, so the OData mapping (RemoteName/RemoteType/Filterable/...) is gone. Attribute $IDs are re-minted, and DateTime LocalizeDate is dropped.
  • odata client Clients.OrderODataClient: UseQuerySegment true → false. ApplicationId, EndpointId, CatalogUrl, Icon, the proxy settings, ErrorHandlingMicroflow and HeaderListMicroflow are dropped.
  • odata service MyFirstModule.CarOdataApi: EntitySet PageSize 10000 → 0 (describe only prints it with UsePaging). ExportLevel is dropped.
  • demo user: the rewritten user moves to the end of DemoUsers. This is churn only.

The round trip cannot see any of this: unitKeywords has no mapping for these types, and PedApp has no workflow, OData or external entity. The PedApp JS actions all printed the placeholder body, so the .js path was never run either. The followup about demo user '***' is out of date: #707 already treats '***' as "keep the stored password".

Either each of these rewrites needs to carry what describe cannot print, or these types keep a plain create until that is done. Which one is a call for the maintainer.

ako and others added 12 commits September 27, 2026 09:02
ADR-0010 R11, additive, so under every language version. One lexer rule
drops a comma whose next significant character closes a (), {} or [],
instead of a COMMA? in each of ~70 list rules: that kept every list's
error messages LL(1). Written into the parser, an unknown item after a
comma is reported as 'no viable alternative at input ,X' rather than
naming what the list expects, which lost e.g. the annotation-property
hint (mendixlabs#1014).

A comma still needs an item before it: (,) and (a,,) stay errors.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Loop body flows are stored in the unit's Flows list, not in the loop, so
removing a Studio Pro loop with two or more body activities left them
pointing at removed objects and Save refused the unit
(TestApp ACT_ConflictedWorkflowHelper_ApplyJumpTo). mx check 11.14 on the
dropped and replaced copies gives the baseline error list.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each operation was checked against the flow as stored only. Measured
with mx check 11.14 on TestApp: two fragments declaring the same
variable gave CE0111, and a fragment reading a variable a drop in the
same statement removed gave CE0109, after "Altered microflow". The
context now tracks what earlier operations declared, read and removed.
Also count a fragment loop's iterator as the fragment's own, so loop
fragments are no longer refused.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ADR-0010 R11. Both are new rejections, so they apply only under the
`mdl 1` header (ADR-0011): a headerless script parses as before and
check warns MDL-V1-SEMI / MDL-V1-SLASH for each occurrence.

A statement rule that consumes its own `;` (create java action … as
$$…$$;) counts as terminated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…cter (#732)

ADR-0010 R11. Under mdl 0 \n, \t, \r, \\ and \' in a string literal
are escapes, which contradicts Mendix expressions and makes 'C:\temp' a
tab. It changes what text means, so it is tied to the header
(ADR-0011): a headerless script keeps the old value, and check warns
MDL-V1-ESCAPE for each literal whose value would change.

The rule decides where a literal ends ('C:\' is complete under mdl 1,
unterminated under mdl 0), so it is fixed before lexing:
langver.ScanHeader reads the header from the source, and an mdl 1
script is lexed from a StrictEscapeStream, which a lexer predicate on
STRING_LITERAL checks. Every token keeps its stream, so the visitor
reads each literal under the rule it was lexed with: the 242
unquoteString(x.GetText()) calls become unquoteStringLit(x), and a test
keeps a new call from reading token text with mdl 0's escapes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The visitor needs the same suggestion for unknown property keys (#732)
and cannot import the executor. No change in behaviour.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ADR-0010 R11. The REST client (service, basic auth and operation
lists), published REST service, business event service, and the agent
editor's model, knowledge base, consumed MCP service, agent and agent
body blocks accepted any `Key: value` and read only the keys they knew,
so `pathh: '/users'` parsed, checked and executed with the path
missing. A value was also read by its shape rather than its key:
`Response: json from $X` set the request body.

Each list now has a schema (the keys its visitor reads, the value
shapes each takes, and what describe writes). Under mdl 1 a property
outside it is an error naming the key, the list and the nearest known
key; under mdl 0 the list is read as before and check warns
MDL-V1-PROP / MDL-V1-PROPVALUE.

suggest.Closest now also tries two edits, counting a neighbour swap as
one, for names of five letters or more (Verison, Passwd), which the
OData did-you-mean gets too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…dl 1

`first` is Mendix's "First object" range in every language version. Under
the mdl 1 header a bare `limit 1` is a Custom range, a list of one; without
it the alpha meaning (the object) is kept and warns MDL-V1-LIMIT1. The
visitor resolves the meaning into RetrieveStmt.First, so the writer, the
variable typing and MDL-RETRIEVE01 no longer read limit text. describe
prints the object range as `first`. `first` on an association retrieve
is refused (that source has no range).

MDL-RETRIEVE01 keys on the object range and names CE0100 for a loop, as
measured with mx check 11.13.

Tests: visitor per version, builder range per version, check per version,
describe spelling, and a PedApp round trip of both forms with a headerless
control.

Part of #734.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nder mdl 1

CLAUDE.md idiom 5, the quick reference, the syntax topic, the skills and
docs-site pages that taught `limit 1` for an object now write `first`
and say what `limit 1` means per language version. CHANGELOG entry.

Part of #734.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The microflow, nanoflow and workflow rules end in `SEMICOLON? SLASH?`
themselves, so `end;` followed by `/` left the statement's own SLASH
empty: the `/` was not reported, and the missing `;` was reported at
the `/`. Found by checking PedApp's describe output under mdl 1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Statement termination no longer recommends the / terminator, and the
string-literal section no longer claims backslash escapes are
unsupported: they are, until mdl 1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…732)

Under mdl 1 'it\'s' ends at the quote, and the errors that follow
point at the rest of the line. Name the cause.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ako and others added 8 commits September 27, 2026 11:15
…nt/service (ADR-0012)

Their create or modify rewrites do not yet carry what describe cannot
print (review of #738; details in #743), so re-running describe output on
Studio Pro content would silently lose it. A plain create refuses instead.
Tests pin the verb until #743 proves each carry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ADR-0010 R11/R12: describe output must mean the same under mdl 0 and
mdl 1, and mdl 1 makes `;` the only terminator. About twenty document
types printed a SQL*Plus `/` line after their statement, and pages,
snippets, layouts and some OData statements ended without `;`.

- Drop every `/` line from describe (agents, knowledge bases, MCP
  services, models, constants, contracts, associations, DB connections,
  entities, enumerations, image collections, modules, OData, published
  REST, security, microflows, nanoflows, rules).
- Pages, snippets and layouts end with `};`; a workflow with
  `end workflow;`.
- A published OData service ends with `;` after its property list,
  authentication clause or entity block, whichever is last; an external
  entity without attributes after its property list.
- A published REST service with no resources prints an empty `{ };`
  block: the block is mandatory in the grammar, so the bare `;` it
  printed before did not parse.

Tests: TestPedAppDescribeIsValidMdl1 checks the terminators of every
PedApp document type (plus each module, with and without `with all`,
navigation and settings) on the parse tree and re-parses the output under
an `mdl 1;` header; executor tests cover the types PedApp does not
contain, and every Describe*_Mock / #707 re-parse test now asserts the
same.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`diff` and `diff-local` render MDL for the compared side; with describe
no longer printing `/`, keeping it here would show a spurious line on
every statement and render text that is invalid under mdl 1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	mdl/executor/cmd_agenteditor_mock_test.go
#	mdl/executor/cmd_security_mock_test.go
… (valid under mdl 0 and mdl 1, required by #741)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ako
ako merged commit 279cd04 into main Sep 27, 2026
17 checks passed
ako added a commit that referenced this pull request Sep 27, 2026
…forms

On top of #740/#745/#741/#738, `fmt --upgrade --header` refused almost every
example script: those PRs gated new constructs on the header without a
rewrite. Each construct with a mechanical, meaning-preserving rewrite now has
one, computed from the parse tree by the visitor that records it (ast.Fix,
rune-offset edits, mdl/visitor/visitor_upgrade_fixes.go):

- MDL-V1-SEMI: add the missing `;`. MDL-V1-SLASH: delete the `/` line.
- MDL-V1-ESCAPE: write the literal's mdl 0 value with '' as the only escape;
  no edit inside an expression stored as written (mdl 0 already passed the
  backslash through); an escaped line break in a re-rendered expression is
  reported, since writing it into the literal changes what the builder
  stores (measured: a log message becomes a `{1}` template parameter).
- MDL-V1-LIMIT1: `limit 1` -> `first`. MDL-V1-SET: add `set`.
- MDL-DEPR003/004 and MDL-V1-LIST: call form -> statement form; find and
  contains on a declared String keep the call and gain `set`; a nested call,
  or an operand whose type the script does not state, is reported.
- MDL-V1-REPLACE02: `create or replace user role|demo user` -> `create`.

MDL-V1-PROP, MDL-V1-PROPVALUE and MDL-V1-REPLACE01 have no mechanical
rewrite; they are listed in `unrewritable` (may only shrink) and block the
header with HeaderBlockedError, which names each construct and why.
TestGatedRegistryIsComplete holds every visitor.LanguageChanges() code to
exactly one of the two lists.

The examples test lists the two corpus scripts that keep mdl 0
(keepsItsVersion) and the two whose AST differs but whose model is the same
(buildsTheSameModelNotTheSameAST), both may only shrink. The execute-both
property test upgrades a blocked script without the header, and by default
skips scripts whose only edits are the header and terminators, which never
reach the AST: 249 scripts execute (6 min); MXCLI_UPGRADE_ALL=1 runs all 701,
587 of them executing to the same model.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant