Mig 629 create data definitions from the variable structures

.agents/skills/generate-sample-data/SKILL.md

@@ -0,0 +1,113 @@
+---
+name: generate-sample-data
+description: Generate sample/test JSON data matching a migration variable structure (the data definition deployed to Inspire). Use when the user asks for sample data, test data, input data or example JSON for a variable structure, data definition or the current migration project.
+compatibility: Requires Node.js; DB access via psql, Docker or the Gradle VariablesExport task
+---
+
+# Generate sample data for a variable structure
+
+Optional inputs from the request: variable structure id, array sizes, output path, project config file
+(overrides `active-project-config`). Array sizes: `--top-array-size` = records of the outermost arrays (not
+nested in another array, typically `Clients`), `--array-size` = records of nested arrays. Both default to 3;
+a single unqualified size ("arrays of 5") sets both, "one client" / "one top-level record" sets only the top one.
+
+Produces a JSON file whose shape matches the variable structure of the **current migration project**, filled
+with plausible values. Two helpers live next to this file:
+
+- `variable-structure.sql` – reads one structure from the working DB (raw entries, no resolution).
+- `sample-data.mjs` – resolves paths, builds the tree, prints the field list or writes the final JSON
+  (accepts the SQL output **or** the `VariablesExport` CSV, so both sources share one resolution logic).
+
+`SKILL_DIR` below = the directory of this file (`.agents/skills/generate-sample-data`). Write intermediate
+files to the scratchpad directory, not into the repo.
+
+## 1. Resolve the current project
+
+All config lives in `migration-examples/src/main/resources/`.
+
+1. Project config file: `--project-config` argument if given, else the first non-empty line not starting with
+   `#` in `active-project-config` (no extension), else `project-config.toml`.
+2. Project name = `name` in that file. Every DB query **must** filter `project_name = <name>`.
+3. Structure id: argument if given → `defaultVariableStructure` in the project config (if not commented out)
+   → the only structure of the project → otherwise list the project's structures and ask the user.
+
+## 2. Load the structure – fallback order
+
+DB settings are `[dbConfig]` in `migration-config.toml` (host, port, dbName, user, password). Never echo the
+password; pass it via the `PGPASSWORD` env var. Commands below use POSIX shell syntax – adapt them to the
+shell in use (e.g. `$env:PGPASSWORD`, `gradlew.bat` in PowerShell/cmd). Use the first option that works:
+
+1. **`psql` on PATH** (native install, Windows service, remote host):
+   ```bash
+   PGPASSWORD='<pw>' psql -h <host> -p <port> -U <user> -d <dbName> -X -At -v ON_ERROR_STOP=1 \
+     -v proj='<project>' -v vs='<structureId>' -f "$SKILL_DIR/variable-structure.sql" > "$SCRATCH/structure.json"
+   ```
+2. **`psql` of a local PostgreSQL install that is not on PATH** – look in the platform's usual install
+   location (newest version) and run the same command with it, e.g. `C:/Program Files/PostgreSQL/*/bin`
+   (Windows), `/usr/lib/postgresql/*/bin` or `/usr/pgsql-*/bin` (Linux),
+   `/opt/homebrew/opt/postgresql*/bin` or `/Applications/Postgres.app/Contents/Versions/latest/bin` (macOS).
+3. **Docker container** publishing the configured port – find it with
+   `docker ps --filter publish=<port> --format '{{.Names}}'` (fallback: any running container whose image is
+   `postgres*`), then pipe the SQL to the psql inside it:
+   ```bash
+   docker exec -i -e PGPASSWORD='<pw>' <container> psql -U <user> -d <dbName> -X -At -v ON_ERROR_STOP=1 \
+     -v proj='<project>' -v vs='<structureId>' < "$SKILL_DIR/variable-structure.sql" > "$SCRATCH/structure.json"
+   ```
+4. **Gradle CSV export** (uses the project's own JDBC connection – works whenever the migration scripts work,
+   no psql needed). From the repo root:
+   ```bash
+   ./gradlew :migration-examples:VariablesExport -PscriptArgs="--variable-structure-id <structureId>"
+   ```
+   The CSV lands in `migration-examples/mapping/` and its name ends with `<structureId>.csv` (take the newest
+   match). Rows without `inspire_path` are not in the structure and are ignored by the script. Caveat: if the
+   project config selects specific document objects, the export may contain only their variables.
+5. **User-provided CSV** in the same `VariablesExport` format – ask for it if everything above fails, and
+   report what failed for each option.
+
+To list structures (step 1.3) with options 1–3, run
+`select id, name from variable_structure where project_name = '<project>'` the same way.
+
+## 3. Check the field list
+
+```bash
+node "$SKILL_DIR/sample-data.mjs" "$SCRATCH/structure.json"     # or the .csv
+```
+
+Prints the tree (`Name[]` = Array, `Name{}` = SubTree, then type and variable id) plus `WARN` lines on stderr
+(unresolvable references, missing variable rows). An empty structure or warnings are worth telling the user.
+
+How the structure works (the script implements it; know it to interpret the output):
+- Each structure entry's `path` is the **parent** path; the field name is appended. Field name =
+  structure entry `name` → variable `name` → variable id.
+- A path is a literal (`Data.Clients.Value`) or a reference to another variable (`$clientsArray` /
+  `{"variableId": …}`) meaning "inside that variable" – Array parents add `.Value`; references chain.
+- `Data` is the root and is dropped. `X.Value` = X is an array; another segment = subtree. An explicit
+  `Array`/`SubTree` variable at that path wins.
+
+## 4. Choose values and build the JSON
+
+Write `$SCRATCH/values.json` mapping **variable id** → value, or a list of values. Each time a field is emitted
+the next list item is used (cycling), so for a field inside arrays give as many distinct values as it will be
+emitted (product of the enclosing array sizes, e.g. with defaults 3 for `Clients[]`, 9 for `Clients[].Jobs[]`;
+with `--top-array-size 1` 1 and 3)
+– records should differ from each other, and related fields in the same record should fit together
+(first name / last name / email, city / state / zip). Unlisted ids get the defaults below.
+
+Values by type:
+- `String` – realistic for the name (`DRV_RecipientFirstName` → "John", `DRV_City` → "New York",
+  `*Email*` → "john.smith@example.com", `*Zip*` → "10001", `*_Indicator`, `*Flag*` → "Y"). If unsure: `"string"`.
+- `Integer` – whole number fitting the name (counts small, ids/numbers longer). Unsure: `0`.
+- `Currency` – JSON number with up to two decimals, e.g. `1250.75`. Unsure: `0`.
+- `DateTime` – ISO string; `"2026-03-15"` for dates, `"2026-03-15T14:30:00"` when the name implies time.
+- `Boolean` – `true` unless the name clearly implies otherwise.
+- Language variable (printed in the header) – a language code like `"en"`.
+
+Unless the user says otherwise every field gets a value and every array 3 records. Then:
+
+```bash
+node "$SKILL_DIR/sample-data.mjs" "$SCRATCH/structure.json" --values "$SCRATCH/values.json" \
+  --top-array-size 3 --array-size 3 --out "<output>"
+```
+
+Default output: `migration-examples/sample-data/<project>_<structureId>.json` (git-ignored) unless the user
+named a path. Show the user the path, a short excerpt, and any warnings.

.agents/skills/generate-sample-data/sample-data.mjs

@@ -0,0 +1,181 @@
+#!/usr/bin/env node
+// Builds sample JSON data for a migration variable structure.
+//
+// Usage:
+//   node sample-data.mjs <input.json|input.csv> [--values values.json] [--array-size 3] [--top-array-size N] [--out out.json]
+//
+// --array-size      records per nested array (default 3)
+// --top-array-size  records per outermost array, i.e. not nested in another array (default = --array-size)
+//
+// <input>  JSON produced by variable-structure.sql, or the CSV produced by the VariablesExport gradle task.
+// Without --values: prints the resolved field list (one line per leaf) so values can be chosen.
+// With --values:    writes the sample data JSON. values.json maps variable id -> value or list of values;
+//                   a list is consumed in order each time the field is emitted (cycling), so fields
+//                   inside arrays get distinct values. Missing ids fall back to a per-type default.
+
+import fs from "node:fs";
+
+const args = process.argv.slice(2);
+const opt = (name, dflt) => {
+    const i = args.indexOf(name);
+    if (i < 0) return dflt;
+    const v = args[i + 1];
+    args.splice(i, 2);
+    return v;
+};
+const valuesFile = opt("--values");
+const arraySize = Number(opt("--array-size", "3"));
+const topArraySize = Number(opt("--top-array-size", String(arraySize)));
+const outFile = opt("--out");
+const inputFile = args[0];
+if (!inputFile) {
+    console.error("Usage: node sample-data.mjs <input.json|input.csv> [--values values.json] [--array-size 3] [--top-array-size N] [--out out.json]");
+    process.exit(2);
+}
+
+const warnings = [];
+
+// ---- input -------------------------------------------------------------------------------------
+
+function splitCsvLine(line) {
+    const cols = [];
+    let cur = "", inQuotes = false, quoted = false;
+    for (let i = 0; i < line.length; i++) {
+        const c = line[i];
+        if (inQuotes) {
+            if (c === '"' && line[i + 1] === '"') { cur += '"'; i++; }
+            else if (c === '"') inQuotes = false;
+            else cur += c;
+        } else if (c === ",") { cols.push(cur); cur = ""; quoted = false; }
+        else if (cur.length === 0 && c === '"' && !quoted) { inQuotes = true; quoted = true; }
+        else cur += c;
+    }
+    cols.push(cur);
+    return cols;
+}
+
+// VariablesExport CSV: id,name,data_type,inspire_path,inspire_name,language_variable,originLocations
+// Variables without inspire_path are not part of the structure.
+function readCsv(text) {
+    const lines = text.split(/\r?\n/).filter(l => l.trim() !== "");
+    const header = splitCsvLine(lines[0]).map(h => h.trim());
+    const col = n => header.indexOf(n);
+    const variables = [];
+    let languageVariable = null;
+    for (const line of lines.slice(1)) {
+        const c = splitCsvLine(line);
+        const id = c[col("id")], path = c[col("inspire_path")];
+        if (!path) continue;
+        if ((c[col("language_variable")] || "").trim().toLowerCase() === "true") languageVariable = id;
+        variables.push({
+            id,
+            name: c[col("inspire_name")] || c[col("name")] || id,
+            dataType: c[col("data_type")] || null,
+            literalPath: path.startsWith("$") ? null : path,
+            refId: path.startsWith("$") ? path.slice(1) : null,
+        });
+    }
+    return { structureId: null, languageVariable, variables };
+}
+
+const raw = fs.readFileSync(inputFile, "utf8").replace(/^/, "");
+const input = inputFile.toLowerCase().endsWith(".csv") ? readCsv(raw) : JSON.parse(raw);
+const byId = new Map(input.variables.map(v => [v.id, v]));
+
+// ---- path resolution (mirrors VariablePath.resolve in InspireDocumentObjectBuilder.kt) -----------
+
+function parentPath(v, seen = new Set()) {
+    if (v.literalPath != null) return v.literalPath;
+    if (seen.has(v.id)) return null;
+    seen.add(v.id);
+    const parent = byId.get(v.refId);
+    if (!parent) return null;
+    const pp = parentPath(parent, seen);
+    if (!pp) return null;
+    if (parent.dataType === "Array") return `${pp}.${parent.name}.Value`;
+    if (parent.dataType === "SubTree") return `${pp}.${parent.name}`;
+    warnings.push(`'${v.id}' references '${parent.id}' of type ${parent.dataType}; only Array/SubTree can be referenced`);
+    return null;
+}
+
+const resolved = [];
+for (const v of input.variables) {
+    const pp = parentPath(v);
+    if (!pp) { warnings.push(`'${v.id}' has an unresolvable path - skipped`); continue; }
+    if (!v.dataType) warnings.push(`'${v.id}' has no variable record - treated as String`);
+    // segments without "Data" root and "Value" markers; isArray marks segments followed by ".Value"
+    const parts = `${pp}.${v.name}`.split(".");
+    const segs = [];
+    parts.forEach((p, i) => {
+        if (p.toLowerCase() === "value") return;
+        if (i === 0 && p === "Data") return;
+        segs.push({ name: p, isArray: parts[i + 1]?.toLowerCase() === "value" });
+    });
+    resolved.push({ ...v, dataType: v.dataType || "String", segs });
+}
+
+// ---- tree (mirrors buildVariableTree in InspireBuilderUtils.kt) ---------------------------------
+
+const root = { kind: "SubTree", children: new Map() };
+const explicitType = new Map(resolved.map(v => [v.segs.map(s => s.name).join("."), v]));
+for (const v of resolved) {
+    let node = root;
+    v.segs.forEach((s, i) => {
+        const key = v.segs.slice(0, i + 1).map(x => x.name).join(".");
+        let child = node.children.get(s.name);
+        if (!child) {
+            const ex = explicitType.get(key);
+            const kind = ex ? ex.dataType : (s.isArray ? "Array" : "SubTree");
+            child = { kind, id: ex?.id, path: key, children: new Map() };
+            node.children.set(s.name, child);
+        }
+        if (child.kind !== "Array" && child.kind !== "SubTree" && i < v.segs.length - 1) {
+            throw new Error(`Leaf variable '${key}' cannot contain child variables (via '${v.id}')`);
+        }
+        node = child;
+    });
+}
+
+// ---- output -------------------------------------------------------------------------------------
+
+if (!valuesFile) {
+    const lines = [];
+    const walk = (node, depth) => {
+        for (const [name, c] of node.children) {
+            const container = c.kind === "Array" ? "[]" : c.kind === "SubTree" ? "{}" : "";
+            lines.push(`${"  ".repeat(depth)}${name}${container}  ${c.kind}${c.id ? `  id=${c.id}` : ""}`);
+            walk(c, depth + 1);
+        }
+    };
+    walk(root, 0);
+    console.log(`structure: ${input.structureId ?? "(csv)"}  languageVariable: ${input.languageVariable ?? "-"}`);
+    console.log(lines.join("\n"));
+} else {
+    const values = JSON.parse(fs.readFileSync(valuesFile, "utf8"));
+    const counters = new Map();
+    const today = new Date().toISOString().slice(0, 10);
+    const defaults = { String: "string", Integer: 0, Currency: 0.0, Boolean: true, DateTime: today };
+    const leafValue = node => {
+        const n = counters.get(node.path) ?? 0;
+        counters.set(node.path, n + 1);
+        const v = node.id != null ? values[node.id] : undefined;
+        if (Array.isArray(v)) return v.length ? v[n % v.length] : defaults[node.kind];
+        if (v !== undefined) return v;
+        return defaults[node.kind] ?? "string";
+    };
+    // inArray: false until the first (outermost) array, which gets topArraySize records
+    const build = (node, inArray = false) => {
+        const obj = {};
+        for (const [name, c] of node.children) {
+            if (c.kind === "Array") obj[name] = Array.from({ length: inArray ? arraySize : topArraySize }, () => build(c, true));
+            else if (c.kind === "SubTree") obj[name] = build(c, inArray);
+            else obj[name] = leafValue(c);
+        }
+        return obj;
+    };
+    const json = JSON.stringify(build(root), null, 2) + "\n";
+    if (outFile) { fs.writeFileSync(outFile, json); console.log(`written ${outFile}`); }
+    else process.stdout.write(json);
+}
+
+for (const w of [...new Set(warnings)]) console.error(`WARN ${w}`);

.agents/skills/generate-sample-data/variable-structure.sql

@@ -0,0 +1,26 @@
+-- Raw variable structure entries for sample-data.mjs (path resolution happens in the script).
+-- psql variables: :'proj' = project name, :'vs' = variable structure id.
+-- Output: one JSON document { project, structureId, structureName, languageVariable, variables: [...] }
+-- variables[]: { id, name, dataType, literalPath, refId }
+--   name = structure entry name -> variable name -> variable id (same fallback as the deploy)
+--   exactly one of literalPath / refId is set; dataType is null when the variable row is missing
+SELECT jsonb_build_object(
+               'project', vs.project_name,
+               'structureId', vs.id,
+               'structureName', vs.name,
+               'languageVariable', vs.language_variable,
+               'variables', coalesce(jsonb_agg(jsonb_build_object(
+                       'id', e.key,
+                       'name', coalesce(e.value ->> 'name', v.name, e.key),
+                       'dataType', v.data_type,
+                       'literalPath', CASE jsonb_typeof(e.value -> 'path')
+                                          WHEN 'string' THEN e.value ->> 'path'
+                                          ELSE e.value -> 'path' ->> 'path' END,
+                       'refId', e.value -> 'path' ->> 'variableId'
+                                               ) ORDER BY e.key) FILTER (WHERE e.key IS NOT NULL), '[]'::jsonb))
+FROM variable_structure vs
+         LEFT JOIN LATERAL jsonb_each(vs.structure) e ON TRUE
+         LEFT JOIN variable v ON v.id = e.key AND v.project_name = vs.project_name
+WHERE vs.project_name = :'proj'
+  AND vs.id = :'vs'
+GROUP BY vs.project_name, vs.id, vs.name, vs.language_variable;

CHANGELOG.md

@@ -10,10 +10,13 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/)
 
 - Support base template deployment in evolve
 - Reworked migration-app, it now serves as a replacement for Gradle task runner
+- DeployDataDefinitions task that deploys data definitions based on the variable structures to Interactive
+- VariableStructuresExport/Import task that allows to modify variable structure data including name and target folder
 
 ### Changed
 
-- deployStyles with interactive output no longer deploys a wfd file, instead it uses the default StyleDefinition.wfd
+- DeployStyles with interactive output no longer deploys a wfd file, instead it uses the default StyleDefinition.wfd
+- DeployBaseTemplates task links base templates to their corresponding data definitions if available
 
 ### Fixed
 

CLAUDE.md

@@ -0,0 +1,4 @@
+# CLAUDE.md
+
+Agent skills for this repository live in `.agents/skills/<skill-name>/SKILL.md` (tool-neutral location).
+When a task matches a skill's `description`, read that `SKILL.md` and follow it.

migration-examples/.gitignore

@@ -3,6 +3,7 @@ logs/
 **/report/
 **/hierarchy/
 **/document-objects/
+**/sample-data/
 !src/main/groovy/com/quadient/migration/example/common/mapping/
 !src/main/groovy/com/quadient/migration/example/common/mapping/*.groovy
 !src/main/groovy/com/quadient/migration/example/common/report/

migration-examples/src/main/groovy/com/quadient/migration/example/common/DeployDataDefinitions.groovy

@@ -0,0 +1,26 @@
+//! ---
+//! displayName: Deploy Data Definitions
+//! category: Deployment
+//! description: Deploys one data definition per variable structure to Interactive
+//! ---
+package com.quadient.migration.example.common
+
+import groovy.transform.Field
+import org.slf4j.Logger
+import org.slf4j.LoggerFactory
+
+import static com.quadient.migration.example.common.util.InitMigration.initMigration
+
+@Field static Logger log = LoggerFactory.getLogger(this.class.name)
+def migration = initMigration(this.binding)
+deploymentResult = migration.deployClient.deployDataDefinitions()
+for (def item : deploymentResult.deployed) {
+    log.info "Deployed data definition '${item.id}' to '${item.targetPath}'"
+}
+for (def item : deploymentResult.warnings) {
+    log.warn "Data definition '${item.id}' deployed with warning: ${item.message}"
+}
+for (def item : deploymentResult.errors) {
+    log.error "Data definition '${item.id}' failed to deploy: ${item.message}"
+}
+log.info "Deployment finished. Deployed ${deploymentResult.deployed.size()} data definitions with ${deploymentResult.warnings.size()} warnings and ${deploymentResult.errors.size()} errors"