Fix df.loop: run loops as child sub-orchestrations and fix cross-generation child ID collision

[Copilot]

Problem

Two related defects in how df.loop drives continue_as_new:

1. Non-root loops restarted from the graph root. df.loop called continue_as_new inline in the main orchestration, so every new generation restarted from graph.root_node_id — re-executing prefix nodes on every iteration. For prefix ~> df.loop(body), the prefix ran once per iteration instead of once per instance.

2. Loops that spawned sub-orchestrations hung forever across generations. Once loops run as child orchestrations (see below), a loop body that itself spawns sub-orchestrations — a nested df.loop, or a JOIN/RACE branch — would stall. Duroxide's auto-generated child instance IDs ({parent}::sub::{event_id}) reset their event counter on each continue_as_new generation, so every loop generation re-derived the same child ID and collided with the previous (now terminal) child. The collided child was acked without processing and the loop never made progress.

Approach

Loops run as child sub-orchestrations

Each df.loop() node spawns a dedicated child sub-orchestration (execute_loop, registered as pg_durable::orchestration::execute-loop). The child handles all iterations via continue_as_new; when the loop exits it returns a SubtreeEnvelope to the parent. The parent orchestration awaits the child and continues with any suffix nodes — so prefix nodes run exactly once.

This is made possible by duroxide PR #31, which preserves the parent link across continue_as_new generations in a child orchestration. The duroxide dependency is switched to a git dependency on the pinodeca/continue-parent-link branch until that PR is merged and released.

// execute_loop_node — spawns child sub-orchestration with an explicit, generation-qualified id
let child_id = child_instance_id(ctx, "loop", node_id);
let raw = ctx.schedule_sub_orchestration_with_id(LOOP_NAME, child_id, loop_input).await?;
parse_subtree_envelope(&raw, "LOOP", results)

Security is preserved: execute_loop calls load_function_graph at the start of every generation (including after continue_as_new), re-validating submitted_by from the database and catching cross-iteration tampering exactly as the parent execute() does.

Generation-qualified child instance IDs

To fix the cross-generation collision, all sub-orchestration spawn sites (loop, join, join3 extras, race) now use schedule_sub_orchestration_with_id with a deterministic, generation-qualified child ID built by a new child_instance_id() helper:

{instance_id}::e{execution_id}::{tag}::{node_id}

execution_id increments on each continue_as_new, so a child spawned in generation N never collides with the terminal child from generation N-1, while the ID stays deterministic across replays of the same generation.

Changes

• Cargo.toml / Cargo.lock: switches duroxide to a git dependency on microsoft/duroxide branch pinodeca/continue-parent-link; adds [patch.crates-io] so duroxide-pg's transitive dependency resolves to the same version. Revert to crates.io pins once PR [DO NOT MERGE] Add native Duroxide provider, stop using duroxide-pg-opt #31 is merged and released.

• src/orchestrations/execute_function_graph.rs

  • execute_loop (new public sub-orchestration): runs iterations via continue_as_new; re-calls load_function_graph per generation for security; returns a SubtreeEnvelope on exit (a break inside the body is the loop's own terminator, so the loop always exits with Normal control)
  • execute_loop_node: spawns execute_loop and merges results back via parse_subtree_envelope
  • child_instance_id (new helper) + schedule_sub_orchestration_with_id at the loop, join, and race spawn sites — fixes the cross-generation child ID collision

• src/registry.rs: registers LOOP_NAME alongside NAME and SUBTREE_NAME

• USER_GUIDE.md: notes that each df.loop() now runs as its own child orchestration (prefix runs once, suffix runs after the loop exits, and df.instances shows the parent plus the loop's child instance)

• tests/e2e/sql/24_nonroot_loop.sql: regression tests covering:

  1. prefix ~> loop — prefix runs once, body runs N times
  2. prefix ~> loop ~> suffix — prefix and suffix each run once
  3. Named result from a prefix node accessible inside the loop body across generations
  4. Nested loop (df.loop(... ~> df.loop(...) ...)) — exercises the cross-generation child ID fix
  5. Loop inside a JOIN branch — exercises the same fix for parallel branches
  6. Non-root while-loop — prefix ~> loop(body, while-condition) ~> suffix exits via the while condition

Upgrade & Migration

No extension schema changes and no upgrade script changes. The new .so reads the same df schema as before; loop execution is entirely a runtime/orchestration change. The only dependency change is the temporary duroxide git pin noted above.

[pinodeca]

Regarding your statement:

A sub-orchestration approach was evaluated but is blocked by a duroxide 0.1.29
limitation: ContinueAsNew drops the parent link, so the parent orchestration never
receives the child's completion.

I created this PR in duroxide to unblock that approach:
microsoft/duroxide#31

Therefore, re-evaluate and:

• if the approach is still blocked, explain why and what would have to change to unblock it.
• if the approach is unblocked, re-implement using the sub-orchestration approach and when you're done update the PR description accordingly.

[pinodeca]

If microsoft/duroxide#33 merges and is part of the next release, this PR can revert back to using schedule_sub_orchestration rather than schedule_sub_orchestration_with_id thus avoiding managing an orchestration ID scheme.

Also, note that this PR should solve #230 and #233 (along with #227, which it initially targeted). Let's validate that by adding regression tests for the scenarios described in 230 and 233.