Planner prompt change: cross-step data contracts

[hanna-paasivirta]

Short Description

Adds golden tests of representative workflows, which surface a key quality issue in the global assistant: cross-step drift in job code generated in parallel. A prompt change allocates the responsibility of defining key and field names to the planner, before calling the job code agent instances. Another prompt tweak reinforces the planner/subagent boundary to prevent the planner from defining code structure except when specified by the user.

Fixes #569

Implementation Details

Problem

When the planner builds a workflow, each step's code is written by an isolated job-code agent call that cannot see any other step. Steps therefore disagreed on how data passes between them, producing workflows that look right but fail at runtime. In the baseline 8-step fan-out run, the validate step wrote the patient to state.event, a later step read state.validatedEvent, and the FHIR step read state.data — every downstream field undefined. In the baseline Kobo run the steps shared a key but not its fields: validation stored sex and dateOfBirth, the update step read gender and birthdate.

Change

One block of the planner system prompt (prompts.yaml, "Tool Call Ordering"). The baseline had the planner describe shared data by a label only and forbade mentioning state. Now the planner puts an identical contract line in every job-code message: the exact state key (e.g. state.patient), the exact field names inside it (from the user's message when given, otherwise chosen once), and ownership — one step produces it, downstream steps consume it verbatim and translate into their own target system's format internally. How the code produces or consumes it stays the job agent's call. (An intermediate version fixing only the key still left field drift; only key + fields closed both.) This is the lightweight form of the mapping spec human OpenFn implementers write before coding — at this scale it is 5–30 field names.

Results

We generated the same six workflows (three scenarios, each from a vague and a verbose prompt) under the baseline prompt and the final prompt, and read each generated YAML checking only the issue this change targets: do the steps agree on where shared data lives? Coherent means every state.X.y a step reads was actually written by an upstream step under exactly that key and field name, so the data arrives at runtime. The two failure modes are key drift (steps use different state keys for the same data — the object never arrives) and field drift (same key, different field names inside it — the object arrives with empty fields). This is judged by reading the code, not by test pass/fail.

Scenario (steps)	Baseline	Final
REST-to-REST vague (2)	coherent	coherent
REST-to-REST verbose (2)	coherent	coherent
Kobo→OpenMRS vague (2)	coherent	coherent
Kobo→OpenMRS verbose (5)	field drift — update reads registration.gender/birthdate, validate stored sex/dateOfBirth; error handler reads submissionId where the source has _id	coherent*
Fan-out vague (3–4)	coherent	coherent
Fan-out verbose (8)	key + field drift — state.event vs state.validatedEvent vs state.data across steps, plus gender vs sex; reproduced in two independent runs	coherent — all 8 steps read state.patient with the exact contract fields; gender/birthDate appear only inside target payloads (correct translation)

* One slip on an auxiliary value outside the contract (lookup stored a UUID string, update read .uuid off it); the contract object itself is clean.

Drift only ever appeared in the two verbose multi-system workflows — the ones with the most independently generated steps — and the final prompt eliminated it in both. The generated YAML for every cell is in services/global_chat/tests/acceptance/gold_workflows/tmp/ (__baseline / __fields-v2 labels).

Second change: requirements, not implementation

We found the planner occasionally prescribing code structure to subagents (e.g. telling one to put "placeholder constants at the top", which the job agent obeyed, producing invalid top-level declarations). The old guardrail only banned suggesting function names — an enumeration that misses everything not listed. It is replaced with the boundary itself: messages state what the step must do, must never do, and what it reads/writes per the contract — never what the code should look like; anything the user explicitly specified is a requirement and is relayed verbatim. Verified on two runs: the Kobo verbose scenario no longer gets structure prescriptions in planner messages, and the existing function-relay acceptance test (user names each(), fields(), tracker.import with CREATE_AND_UPDATE) still relays everything verbatim and now passes all criteria.

Future

For complex workflows a future version could generate a real, user-reviewable mapping artifact (including code-list/terminology mappings) or let job agents read upstream code directly. Not needed for workflows at this scale.

AI Usage

Please disclose whether you've used AI in this work (it's cool, we just want to
know!):

• Yes, I have not used AI
• No, I have not used AI

You can read more details in our
Responsible AI Policy