Skip to content

Querying a workflow that is running local activities makes its history fail replay (TMPRL1100): a local activity result is delivered live with the next workflow task's jobs #2143

Closed as duplicate

Description

@raufer

What are you really trying to do?

We run long-lived workflows whose main loop awaits local activities (writes).
Between those awaits, the loop handles whatever has arrived: a local activity
for each signal and an activity for each finished child workflow. Our tests
poll the running workflow with queries to follow its progress. Before every
change our test suite replays recorded histories with Replayer as a determinism check.
The same code, on the same SDK version, sometimes fails to replay its own
successfully completed histories, and only when the workflow was being queried
while it ran.

Describe the bug

Replay fails with [TMPRL1100], for example:

Nondeterminism error: Missing associated machine for LocalActivity(<seq>)
Nondeterminism error: No command scheduled for event HistoryEvent(id: <n>, StartChildWorkflowExecutionInitiated)

What the failing histories show:

  1. In workflow task N the workflow awaits a local activity, and queries keep
    arriving. The activity's marker is recorded among task N's commands.
  2. While task N runs, two signals and a child workflow's completion reach the
    server. They are the new events of task N+1.
  3. Live, the workflow got the local activity's result in the same
    activation as task N+1's jobs, for example [signal, signal, resolve child workflow, resolve local activity]. It handled all of them together and
    scheduled, say, [local activity, local activity, activity] (sequence
    numbers k, k+1, k+2).
  4. On replay, the result comes alone, as the last activation of task N.
    Task N+1's jobs come in the next activation. The workflow then schedules
    [local activity, activity, local activity]. The marker recorded for k+1
    now points at an activity rather than a local activity, and replay fails.

So in the live run, the workflow got the result only after the task that
recorded its marker had completed, in the same activation as the next task's
jobs. Nothing in the history records this, so replay delivers the result back
inside its own task.

Queries are what triggers it. The reproduction below is a toy of the same
shape. Without queries it never failed on 1.35.0 (0/80) or 1.36.1 (0/240), and
no live activation ever carried a local activity result together with another
kind of job. When the driver queries each running workflow every 5 ms, it fails
on 1.35.0 (11/80) and 1.36.1 (26/80). In 2 of those 80 runs on 1.36.1 the live
worker itself kept failing a workflow task with the same error, so this is not
only a Replayer problem. The 1.35.0 release notes say: "A workflow task that
arrives while a workflow waits on a local activity, such as a legacy query, now
waits until the current workflow task is reported. Previously it replaced that
workflow task." This looks like a remaining case of the same interaction.

The failing 1.35.0 and 1.36.1 histories record core flag 4, so by the 1.34.0
notes ("Replay now preserves which local activity results were delivered
together for newly recorded histories", temporalio/sdk-rust#1616) they are
newly recorded histories. As we read it, that change keeps results that were
delivered together within one workflow task together on replay. Here, the
activation that delivered the result belonged to the next task and also
carried that task's other jobs.

Lang flag 2 does not help. We forced
PROCESS_WORKFLOW_ACTIVATION_JOBS_AS_SINGLE_BATCH (_WorkflowLogicFlag 2) on
for new histories by adding it to _DEFAULT_ENABLED_WORKFLOW_LOGIC_FLAGS. The
toy still failed 17 of 80 runs on 1.36.1, and the failing histories record
lang flag 2.

Questions:

  1. Flag 2 does not prevent this in our reproduction. Is it meant to cover
    this case at all, and is it planned to become the default?
  2. Should Core deliver a local activity result within the workflow task that
    records its marker, even when a query arrives while the workflow waits on
    that activity? Or should it record enough for replay to reproduce the
    later delivery?

How often this happens in our own workflow's replay test (it polls with
queries; one replay per live run):

SDK replay failures
1.32.0 9 / 80 (more under load)
1.35.0 2 / 160
1.36.1 1 / 160
1.36.1, lang flag 2 forced on 0 / 160

Minimal Reproduction

One file; it needs only temporalio. It starts a local dev server, runs 80
workflows on one worker while querying each of them every 5 ms, replays every
finished history with Replayer, and counts the failures.

uv run --with temporalio==1.36.1 python repro.py                   # queries every 5 ms
QUERY_POLL_MS=0 uv run --with temporalio==1.36.1 python repro.py   # no queries: passes
FORCE_SINGLE_BATCH=1 uv run --with temporalio==1.36.1 python repro.py

The workflow works through steps, each a short local activity. Between steps
it starts a local activity per pending signal and an activity per finished
child. Beside the steps runs a slower local activity that makes the outside
world signal the workflow twice, and a child workflow that finishes at about
the same time.

SDK no queries queried every 5 ms
1.32.0 80 / 80 (looks like #1881) 80 / 80
1.35.0 0 / 80 11 / 80
1.36.1 0 / 240 26 / 80 (2 of them fail live)
1.36.1, flag 2 forced on 17 / 80 (2 of them fail live)
repro.py
"""Replay nondeterminism with local activities: a self-contained reproduction.

A workflow works through steps; each step is a short local activity (a
"write"). Between steps it handles whatever has arrived: one local activity
per signal, one activity for a finished child workflow. Beside the steps runs
a slower local activity that makes the outside world signal the workflow
twice, and a child workflow that finishes at about the same time. While a
run is going, the driver queries it every few milliseconds, as a test might.

Every finished run's history is replayed with ``Replayer``; the script counts
the replay failures.

    uv run --with temporalio==1.36.1 python repro.py

Knobs (environment variables, defaults in ``main``): RUNS, CONCURRENCY,
STEPS, STEP_MS, SLOW_MS, TASK_TIMEOUT_MS, QUERY_POLL_MS (query each running
workflow every N ms; 0 turns the polling off), MAX_CACHED (worker max_cached_workflows), TEMPORAL_CLI (path to an installed
``temporal`` CLI; the SDK downloads one otherwise), FORCE_SINGLE_BATCH=1
(turn lang logic flag 2 on for new histories), TRACE=1 (print the shapes of
live activations that resolve an activity together with other jobs).
"""

from __future__ import annotations

import asyncio
import dataclasses
import os
import random
import sys
import uuid
from collections import Counter
from datetime import timedelta

from temporalio import activity, workflow

with workflow.unsafe.imports_passed_through():
    import temporalio
    import temporalio.service
    from temporalio.client import Client, WorkflowHistory
    from temporalio.testing import WorkflowEnvironment
    from temporalio.worker import Replayer, Worker

TASK_QUEUE = "la-replay-repro"


@dataclasses.dataclass
class StepsInput:
    steps: int
    step_ms: int
    slow_ms: int


@dataclasses.dataclass
class WriteInput:
    item: str
    sleep_ms: int
    jitter_ms: int = 3
    notify: bool = False


@workflow.defn
class ChildWorkflow:
    @workflow.run
    async def run(self, delay_ms: int) -> str:
        await workflow.sleep(timedelta(milliseconds=delay_ms))
        return "child done"


class Activities:
    def __init__(self, client: Client) -> None:
        self._client = client

    @activity.defn
    async def write(self, inp: WriteInput) -> str:
        """Pretend to write somewhere. A notifying write makes the outside
        world send the workflow two signals while it runs."""
        total = (inp.sleep_ms + random.uniform(0, inp.jitter_ms)) / 1000
        if not inp.notify:
            await asyncio.sleep(total)
            return inp.item
        before = random.uniform(0, total)
        await asyncio.sleep(before)
        handle = self._client.get_workflow_handle(activity.info().workflow_id)
        await handle.signal(StepsWorkflow.submit, f"{inp.item}-a")
        await handle.signal(StepsWorkflow.submit, f"{inp.item}-b")
        await asyncio.sleep(max(0.0, total - before))
        return inp.item

    @activity.defn
    async def react(self, note: str) -> str:
        return note


@workflow.defn
class StepsWorkflow:
    def __init__(self) -> None:
        self._inbox: list[str] = []
        self._received = 0

    @workflow.signal
    def submit(self, item: str) -> None:
        self._inbox.append(item)
        self._received += 1

    @workflow.query
    def received(self) -> int:
        return self._received

    @workflow.run
    async def run(self, inp: StepsInput) -> int:
        slow: asyncio.Task[str] | None = None
        child: workflow.ChildWorkflowHandle | None = None
        rounds = 0
        handled = 0
        for step in range(inp.steps):
            if slow is None or slow.done():
                # Beside the steps: a slow write and a child workflow.
                rounds += 1
                child = await workflow.start_child_workflow(
                    ChildWorkflow.run,
                    inp.slow_ms,
                    id=f"{workflow.info().workflow_id}-child-{rounds}",
                )
                slow = asyncio.create_task(
                    workflow.execute_local_activity_method(
                        Activities.write,
                        WriteInput(f"slow-{rounds}", inp.slow_ms, inp.slow_ms // 2, True),
                        start_to_close_timeout=timedelta(seconds=30),
                    )
                )
            # The step's write.
            await workflow.execute_local_activity_method(
                Activities.write,
                WriteInput(f"step-{step}", inp.step_ms),
                start_to_close_timeout=timedelta(seconds=10),
            )
            # Handle whatever has arrived before the next step.
            pending = []
            items, self._inbox = self._inbox, []
            pending.extend(
                workflow.execute_local_activity_method(
                    Activities.write,
                    WriteInput(item, inp.step_ms),
                    start_to_close_timeout=timedelta(seconds=10),
                )
                for item in items
            )
            if child is not None and child.done():
                child = None
                handled += 1
                pending.append(
                    workflow.execute_activity_method(
                        Activities.react,
                        f"child-{handled}",
                        start_to_close_timeout=timedelta(seconds=10),
                    )
                )
            if pending:
                await asyncio.gather(*pending)
        # Let the last slow write, its signals and the last child finish.
        if slow is not None:
            await slow
        await workflow.wait_condition(lambda: self._received == 2 * rounds)
        if child is not None:
            await child
        return rounds


def _flags(history: WorkflowHistory) -> tuple[set[int], set[int]]:
    core: set[int] = set()
    lang: set[int] = set()
    for event in history.events:
        if event.HasField("workflow_task_completed_event_attributes"):
            meta = event.workflow_task_completed_event_attributes.sdk_metadata
            core.update(meta.core_used_flags)
            lang.update(meta.lang_used_flags)
    return core, lang


def _force_single_batch() -> None:
    """Turn on lang logic flag 2 (PROCESS_WORKFLOW_ACTIVATION_JOBS_AS_SINGLE_BATCH)
    for new histories, on SDK versions that have it."""
    import temporalio.worker._workflow as wf_mod
    import temporalio.worker._workflow_instance as inst_mod

    flag = inst_mod._WorkflowLogicFlag.PROCESS_WORKFLOW_ACTIVATION_JOBS_AS_SINGLE_BATCH
    flags = frozenset({*inst_mod._DEFAULT_ENABLED_WORKFLOW_LOGIC_FLAGS, flag})
    inst_mod._DEFAULT_ENABLED_WORKFLOW_LOGIC_FLAGS = flags
    wf_mod._DEFAULT_ENABLED_WORKFLOW_LOGIC_FLAGS = flags


LIVE_SHAPES: Counter[str] = Counter()


def _trace_live_activations() -> None:
    """Diagnostic: count live activations that resolve an activity together
    with other kinds of jobs."""
    import temporalio.worker._workflow as wf_mod

    original = wf_mod._WorkflowWorker._handle_activation

    async def traced(self, act):  # type: ignore[no-untyped-def]
        if not act.is_replaying:
            kinds = [job.WhichOneof("variant") for job in act.jobs]
            if "resolve_activity" in kinds and len(set(kinds)) > 1:
                LIVE_SHAPES[",".join(kinds)] += 1
        return await original(self, act)

    wf_mod._WorkflowWorker._handle_activation = traced


async def main() -> None:
    runs = int(os.environ.get("RUNS", "80"))
    concurrency = int(os.environ.get("CONCURRENCY", "40"))
    steps = int(os.environ.get("STEPS", "60"))
    step_ms = int(os.environ.get("STEP_MS", "10"))
    slow_ms = int(os.environ.get("SLOW_MS", "150"))
    task_timeout_ms = int(os.environ.get("TASK_TIMEOUT_MS", "10000"))
    run_timeout = float(os.environ.get("RUN_TIMEOUT_S", "60"))
    query_ms = float(os.environ.get("QUERY_POLL_MS", "5"))  # 0: no query polling
    max_cached = os.environ.get("MAX_CACHED")
    single_batch = os.environ.get("FORCE_SINGLE_BATCH") == "1"
    if single_batch:
        _force_single_batch()
    if os.environ.get("TRACE") == "1":
        _trace_live_activations()

    print(
        f"temporalio {temporalio.__version__}, python {sys.version.split()[0]}, "
        f"runs={runs} concurrency={concurrency} steps={steps} step_ms={step_ms} "
        f"slow_ms={slow_ms} task_timeout_ms={task_timeout_ms} "
        f"force_single_batch={single_batch} query_poll_ms={query_ms} "
        f"max_cached_workflows={max_cached or 'default'}",
        flush=True,
    )
    async with await WorkflowEnvironment.start_local(
        dev_server_existing_path=os.environ.get("TEMPORAL_CLI") or None,
    ) as env:
        client = env.client
        acts = Activities(client)
        replayer = Replayer(workflows=[StepsWorkflow, ChildWorkflow])
        failures: Counter[str] = Counter()
        failed_flags: list[tuple[set[int], set[int]]] = []
        stuck: list[str] = []
        queries = Counter[str]()

        async def poll(handle) -> None:  # type: ignore[no-untyped-def]
            """Query the running workflow in a tight loop, as a test might."""
            while True:
                try:
                    await handle.query(StepsWorkflow.received)
                    queries["ok"] += 1
                except Exception:
                    queries["error"] += 1
                await asyncio.sleep(query_ms / 1000)
        sem = asyncio.Semaphore(concurrency)

        async def one(i: int) -> None:
            async with sem:
                handle = await client.start_workflow(
                    StepsWorkflow.run,
                    StepsInput(steps, step_ms, slow_ms),
                    id=f"la-repro-{i}-{uuid.uuid4().hex[:8]}",
                    task_queue=TASK_QUEUE,
                    task_timeout=timedelta(milliseconds=task_timeout_ms),
                )
                poller = asyncio.create_task(poll(handle)) if query_ms else None
                try:
                    await asyncio.wait_for(handle.result(), run_timeout)
                except asyncio.TimeoutError:
                    # The live worker keeps failing a workflow task (see its log).
                    stuck.append(handle.id)
                    try:
                        await handle.terminate("repro: stuck")
                    except temporalio.service.RPCError:
                        pass  # it finished meanwhile
                    return
                finally:
                    if poller:
                        poller.cancel()
                history = await handle.fetch_history()
                result = await replayer.replay_workflow(
                    history, raise_on_replay_failure=False
                )
                if result.replay_failure:
                    message = str(result.replay_failure).splitlines()[0]
                    failures[message[:160]] += 1
                    failed_flags.append(_flags(history))
                    if len(failed_flags) <= 3:
                        path = f"failed-history-{len(failed_flags)}.json"
                        with open(path, "w") as f:
                            f.write(history.to_json())
                        print(f"run {i}: replay failed: {message[:140]} ({path})", flush=True)

        async with Worker(
            client,
            task_queue=TASK_QUEUE,
            workflows=[StepsWorkflow, ChildWorkflow],
            activities=[acts.write, acts.react],
            **({"max_cached_workflows": int(max_cached)} if max_cached else {}),
        ):
            await asyncio.gather(*(one(i) for i in range(runs)))

        print(f"\nreplay failures: {sum(failures.values())}/{runs - len(stuck)} completed runs")
        print(f"runs stuck live on a failing workflow task: {len(stuck)}/{runs}")
        if query_ms:
            print(f"queries: {dict(queries)}")
        for message, count in failures.most_common():
            print(f"  {count} x {message}")
        if failed_flags:
            core = set().union(*(c for c, _ in failed_flags))
            lang = set().union(*(lf for _, lf in failed_flags))
            print(f"flags in failing histories: core={sorted(core)} lang={sorted(lang)}")
        for shape, count in LIVE_SHAPES.most_common(8):
            print(f"  live activation x{count}: [{shape}]")


if __name__ == "__main__":
    asyncio.run(main())

Environment/Versions

  • OS and processor: reproduction on macOS, Apple silicon, Python 3.15.2. Our own workflow's runs: the same (macOS, Apple silicon, Python 3.15.2).
  • Temporal Version: SDK 1.32.0, 1.35.0 and 1.36.1 (temporalio from PyPI). The reproduction used the Temporal CLI 1.5.0 dev server (Server 1.29.0). Our own runs: the same SDK versions, against a local Temporal test server started by the SDK's testing module.
  • Are you using Docker or Kubernetes or building Temporal from source? Not for the reproduction (CLI dev server). Our own runs: no (local test server).

Additional context

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions