Merge planner and workflow agents

[hanna-paasivirta]

Plan: Planner as workflow agent (& workflow-edit tools) WIP

This plan builds on issue #530 (workflow_chat: replace structured outputs with tool use), adapted so the tools live on the PlannerAgent instead of inside workflow_chat. This can be implemented after the initial migration to the global assistant is done.

Summary

Remove the workflow agent as a subagent. The planner edits workflow structure directly through a set of small, deterministic tools (add_jobs, update_jobs, remove_jobs, add_edges, update_edges, remove_edges, set_trigger, create_workflow, update_workflow). Each tool is plain Python that mutates the planner's current_yaml with validation, no LLM call. The server owns IDs and job bodies; the model can never see or emit them.

The job code agent is unchanged: it stays a subagent because it earns the boundary (parallel fan-out, bulk code kept out of the planner's context).

This is not a huge amount of code in the first pass, but it changes the output a lot. The incremental-edit approach needed substantial fixing when we did it for job_chat, so testing is a first-class part of this plan, not a follow-up.

Why not a subagent anymore

A subagent boundary is justified by parallelism, context isolation, or specialised reasoning. The workflow agent has none of these:

• It can never run in parallel. It owns the single shared YAML, so every call is a serial bottleneck (unlike job code agents, which fan out).
• It isolates nothing. The planner already holds the redacted workflow structure in context every turn. The subagent re-reads what the planner just read and re-emits the whole document to express a two-line change. That round trip is the 30-40s cost on even trivial edits like adding a step.
• Structural edits need no reasoning. "Add step X with adaptor Y after Z" is a deterministic operation. One-shot whole-YAML regeneration is the slow and unreliable part: every re-emitted element (IDs, adaptors, edge conditions, webhook sub-keys) is an opportunity for silent corruption.

The subagent's real value was its knowledge (YAML semantics, adaptor rules) and its safety machinery (ID/body placeholders, parse-retry). Both move: the knowledge into the planner's system prompt, the safety into tool code, where it becomes stronger. A tool whose schema has no body or id parameter cannot corrupt job code or IDs by construction.

Why not a skill: skills are per-turn injected instructions, invoked explicitly. Workflow editing is the planner's core competence, always on, and a skill would change only what the planner knows, not what it can do: it would still emit whole YAML itself, which is the failure mode being removed. Skills remain the layer for user-invoked procedures (/diagnose, /qa) that sequence these tools.

Target architecture

User → global_chat → Router (Haiku)
                       ├─ job_chat (direct route, unchanged)
                       └─ PlannerAgent (Opus) ── owns current_yaml
                            ├─ workflow-edit tools (new, deterministic, in-process)
                            │    create_workflow, update_workflow, add_jobs, update_jobs,
                            │    remove_jobs, add_edges, update_edges, remove_edges, set_trigger
                            ├─ call_job_code_agent (subagent, parallelisable, unchanged)
                            ├─ inspect_job_code (unchanged)
                            └─ search_documentation (unchanged)

The planner IS the workflow agent. There is exactly one agent that reasons about workflow structure, and it manipulates it the way it already manipulates job code stitching: through deterministic in-process operations on self.current_yaml, streamed to the client per mutation via the existing _send_yaml / spinner / settled-status machinery.

Repo layout after the change

services/global_chat/
  global_chat.py                 # unchanged
  router.py                      # workflow route now lands on the planner
  planner.py                     # dispatches the new tools; loop/streaming unchanged
  subagent_caller.py             # call_workflow_agent removed; call_job_agent stays
  prompts.yaml                   # planner prompt gains the workflow-editing section
  tools/
    tool_definitions.py          # new tool schemas; CALL_WORKFLOW_AGENT_TOOL removed
    workflow_edit/
      handlers.py                # one function per tool: validate → mutate → dump
      validation.py              # adaptor names, key collisions, edge refs, trigger rules
      tests/                     # deterministic unit tests, no LLM

services/workflow_chat/          # deleted once Lightning's direct calls are sunset

Tool design (adapted from #530)

Carried over from #530 unchanged:

• All edit tools accept arrays for batch edits (one call adds five steps).
• Fail fast, atomically. All validation runs before any write; one failure means nothing is persisted and the error string is returned as the tool result. The planner's existing loop retries with the error in context, so correction happens in-turn (bounded by the existing max_tool_calls).
• Tools cannot touch what they don't name. No fuzzy matching; unknown key is an error with a did-you-mean hint, never a guess.
• set_trigger preserves webhook sub-keys (webhook_reply, webhook_response_config) unless explicitly passed. The "preserve EXACTLY" prompt rule becomes code.
• Unknown fields pass through untouched. A mutation writes only the fields it owns, so anything else Lightning puts in the YAML (canvas positions, webhook response config, future fields) survives by default. Under whole-YAML regeneration, only fields the prompt explicitly said to preserve survived.
• Read-only mode: mutating tools are simply not mounted for that request.

Simplified relative to #530:

• No save_workflow, no strict key accounting. The renames/deletes maps and unaccounted_keys reconciliation existed to make whole-structure re-emission safe. Granular tools make re-emission unnecessary: a restructure is remove_jobs + add_jobs + add_edges, each validated independently. create_workflow covers the from-scratch case (takes the full initial structure of steps/edges/trigger in one call so new workflows appear in one round; errors if any jobs already exist).
• "Start over" is deletion plus creation, not replacement. remove_jobs with every key (one batch call), then create_workflow / add_jobs. remove_jobs cascades: edges attached to removed jobs are removed too, and the tool result reports what was cascaded and how many removed jobs contained code. Deletion is always explicit, the model names every key it destroys, so nothing is lost silently; that was the real purpose of Workflow_chat: Replace structured outputs with tool use #530's key accounting, achieved without it. Prompt rule: confirm with the user before wholesale deletion of jobs that contain code.
• No rename_jobs. update_jobs takes an optional new_key; the server cascades the change through edge keys and references.
• No get_workflow. The planner receives the redacted structure with the user turn and again after every mutation (existing behaviour).
• No new agent loop, streaming protocol, or retry cap. The planner's existing loop, spinner/settled events and per-mutation _send_yaml already do all of this.
• search_knowledge stays deferred, as in Workflow_chat: Replace structured outputs with tool use #530. The adaptor list remains in the prompt for selection; validation of it lives in code.

Structure coverage (checked against the workflow_chat prompt and handlers)

Details of the project.yaml structure the tools must cover explicitly:

• Workflow name (top-level): set by create_workflow; renaming an existing workflow gets a small update_workflow tool (top-level fields only, currently just name).
• Triggers: keyed by name, type: cron (with cron_expression) or type: webhook; default enabled: false. set_trigger also covers cron_cursor_job_id, which references a job ID: the tool takes cron_cursor_job by job key and the server resolves it to the ID. Job rename/removal cascades update or clear it. Note: today restore_components only restores id fields on jobs/edges, so a model-emitted cron_cursor_job_id: __ID_JOB_x__ placeholder appears to leak unrestored into output — a latent bug that server-owned IDs remove by design (verify when migrating).
• Trigger edge: edges from the trigger use source_trigger (not source_job) and the trigger can connect to exactly ONE job (validated). Replacing or renaming the trigger via set_trigger cascades its edge (key and source_trigger field).
• Edge fields: condition_type is an enum (always, on_job_success, on_job_failure, js_expression); condition_expression is required iff js_expression (validated); enabled defaults to true. Edge keys are source->target and rename cascades rewrite them while preserving edge IDs — stronger than today, where ID survival across a rename depends on the model copying placeholders verbatim and silently regenerates UUIDs when it fails.
• Job constraints (validated): unique keys, unique display names, names under 100 characters, sanitised characters. Adaptor default is @latest; a version already set is never changed unless asked (prompt rule).

Every capability and instruction in workflow_chat moves to a named home. Nothing is preserved by accident.

workflow_chat today	New home
YAML structure prompt (yaml_structure_*)	Mostly obsolete: the model never emits YAML. Concepts (jobs, triggers, edges, condition types, one job from trigger) go in the planner prompt; the shape lives in the tool schemas.
ID preservation (extract_and_preserve_components, restore_components, __ID_*__ placeholders)	Deleted. Server owns IDs: tools have no id parameter; new components get UUIDs in the handler. Preservation is by construction.
Job body preservation (__CODE_BLOCK_*__)	Deleted. Tools have no body parameter; new jobs get the standard placeholder body in the handler.
Adaptor validation (validate_adaptors, currently log-only)	Hard validation in validation.py: unknown adaptor rejects the call with the valid-names hint, model corrects in-turn.
Job name sanitisation (sanitize_job_names)	Runs in add_jobs / update_jobs handlers.
Parse-retry loop (regenerate on bad YAML)	Obsolete: no YAML is generated. Replaced by tool-error retry in the existing planner loop.
Streaming YAML changes events	Existing planner _send_yaml, now per mutation, so each edit renders in the editor the moment it lands.
read_only mode	Mutating tools not mounted for the request. Payload gap: only the legacy direct endpoint has a read_only field today; it must be added to the global_chat payload as part of the Lightning migration or the mode is silently lost.
errors payload (error mode)	Appended to the planner's user content, same as page context today. Same payload gap as read_only: the field must be added to global_chat.
general_knowledge prompt section, adaptor list	Merged into the planner system prompt (cached prefix). Includes edge-semantics knowledge (multiple edges into one job = one run per edge, not a merge/wait) and webhook response-config rules (never add proactively; warn before switching away from after_completion/custom codes).
Mode intros and answering instructions (normal_mode_*, error_mode_*)	Behavioural rules port to the planner prompt: ask for clarification rather than build when intent is unclear, don't edit unnecessarily, be brief, describe the workflow as a chart (users never see YAML). JSON output-format sections are obsolete.
Handover / subagent mode	Obsolete: there is no subagent to hand over from. (The job_chat direct route's handover is untouched.)
History YAML redaction ("previously generated YAML has been redacted from history")	Non-issue by construction: the planner's returned history holds only user text and final assistant text; YAML travels in attachments and tool state, never in history.

Prompt migration is a real task, not a copy-paste: go through gen_project_prompts.yaml section by section and either port each rule into the planner prompt, encode it in a handler, or record it as deliberately dropped (e.g. all placeholder-handling rules).

Implementation steps

1. Handlers and validation (workflow_edit/): pure functions over the parsed YAML dict, full unit-test coverage. No planner changes yet. This is where correctness is won, deterministically and cheaply.
2. Tool schemas + planner dispatch: add the tools to tool_definitions.py, dispatch in _execute_tool, add spinner/settled messages per tool ("Adding steps...", "Added "Fetch Patients""). Remove CALL_WORKFLOW_AGENT_TOOL and call_workflow_agent. Tighten the planner's job-code path to the same strict-keys contract: find_job_in_yaml currently fuzzy-matches the planner's job_key (case, hyphens vs underscores, display name) before stitching code; make it exact key or error-with-hint, since the planner always has the exact keys in context. URL-derived lookups (page names, the router's job_key for the job_chat direct route) keep the normalisation — they parse user URLs, not model output. Review max_tool_calls (currently 10 in config.yaml): granular edits use more rounds than one call_workflow_agent did, though each round is far cheaper.
3. Prompt migration: planner system prompt gains the workflow-editing section per the table above. Beyond adding content, two existing rules invert or relax:
   • "Never name an adaptor yourself — the workflow agent is the expert" inverts: adaptor choice is now the planner's job (add_jobs takes an adaptor). The adaptor list (from latest_adaptors, as workflow_chat injects it today) moves into the planner system prompt. It's dynamic content in a cached prefix; it changes only when the adaptor cache refreshes, so the occasional cache bust is acceptable.
   • "Never call call_job_code_agent and call_workflow_agent in the same step" can relax: _execute_tool_blocks already runs non-job tools before job-code tools, and with deterministic structural tools that ordering guard is fully reliable. "Structure before code" stays as the ordering rule.
   • The "Trusting Subagent Responses" section drops its workflow-agent half; it now applies to the job code agent only.
4. Router: remove the workflow_agent destination and its handover path, and rewrite the routing rules — the whole class of workflow-vs-planner distinctions ("add an empty step" goes direct, the [Steps contain job code] check, workflow_has_job_code) disappears. Two destinations remain: job_chat direct route and planner. Note the cost shift: simple workflow questions ("what time does the trigger run?") previously answered by Sonnet via the direct route are now answered by the Opus planner — usually in one round with no tool call, since the redacted structure is already in its context, but pricier per question. Acceptable at current volume; check Langfuse after cutover.
5. Payload: add read_only and errors to the global_chat payload spec (see table above) and coordinate with Lightning so both modes survive the sunset of the direct workflow_chat endpoint.
6. Testing (below), then sunset: delete services/workflow_chat/ once Lightning's direct calls are gone.

Testing

This change affects output heavily and incremental edits bit us in job_chat, so the gate to merge is the test suite, not the implementation.

• Deterministic unit tests for every handler and validator: key collisions, unknown keys, rename cascades through edge keys and cron_cursor_job_id, trigger replacement cascading its edge and preserving webhook sub-keys, condition_expression required iff js_expression, atomicity (one bad item in a batch writes nothing), name sanitisation. These are free to run and catch the bug class that hurt in job_chat.
• Preservation invariants, now assertable exactly because the server owns the YAML: after any edit sequence, untouched job bodies and all IDs are byte-identical. Property-style tests over random edit sequences are cheap here and worth it.
• Acceptance suite (services/testing/ harness + gold workflows): a scenario matrix of real edit requests: add/remove/rename steps, retarget edges, change triggers, multi-step builds, "start over", ambiguous requests that should ask rather than edit. Judge both the final YAML and the number of tool rounds taken.
• Comparison run: run the same scenario set against the old call_workflow_agent path and the new tools (Langfuse traces on both) and review diffs in output quality and latency before cutover.

Cutover is a straight switch once the acceptance results and timings look good; no live A/B and no dual code paths kept around.

What this buys us

Structural edits go from a 30-40s serial subagent round-trip to in-process mutations, and small edits drop from full-YAML-out to a ~100-token tool call. The time and complexity freed up is what lets us add the subagents and skills we actually need next, e.g. a proofreading/QA phase that reviews a finished workflow for coherence and quality, which slots into the same planner tool loop without touching this design.

Out of scope (later)

• search_knowledge over adaptors/docs/examples (replaces the prompt-embedded adaptor list; Workflow_chat: Replace structured outputs with tool use #530 section 6).
• Proofreading/QA subagent or skill.
• User-defined skills (separate skills v0/v1 plan).