Repository navigation
Description
What are you really trying to do?
We run long-lived workflows whose main loop awaits local activities (writes).
Between those awaits, the loop handles whatever has arrived: a local activity
for each signal and an activity for each finished child workflow. Our tests
poll the running workflow with queries to follow its progress. Before every
change our test suite replays recorded histories with Replayer as a determinism check.
The same code, on the same SDK version, sometimes fails to replay its own
successfully completed histories, and only when the workflow was being queried
while it ran.
Describe the bug
Replay fails with [TMPRL1100], for example:
Nondeterminism error: Missing associated machine for LocalActivity(<seq>)
Nondeterminism error: No command scheduled for event HistoryEvent(id: <n>, StartChildWorkflowExecutionInitiated)
What the failing histories show:
- In workflow task N the workflow awaits a local activity, and queries keep
arriving. The activity's marker is recorded among task N's commands. - While task N runs, two signals and a child workflow's completion reach the
server. They are the new events of task N+1. - Live, the workflow got the local activity's result in the same
activation as task N+1's jobs, for example[signal, signal, resolve child workflow, resolve local activity]. It handled all of them together and
scheduled, say,[local activity, local activity, activity](sequence
numbers k, k+1, k+2). - On replay, the result comes alone, as the last activation of task N.
Task N+1's jobs come in the next activation. The workflow then schedules
[local activity, activity, local activity]. The marker recorded for k+1
now points at an activity rather than a local activity, and replay fails.
So in the live run, the workflow got the result only after the task that
recorded its marker had completed, in the same activation as the next task's
jobs. Nothing in the history records this, so replay delivers the result back
inside its own task.
Queries are what triggers it. The reproduction below is a toy of the same
shape. Without queries it never failed on 1.35.0 (0/80) or 1.36.1 (0/240), and
no live activation ever carried a local activity result together with another
kind of job. When the driver queries each running workflow every 5 ms, it fails
on 1.35.0 (11/80) and 1.36.1 (26/80). In 2 of those 80 runs on 1.36.1 the live
worker itself kept failing a workflow task with the same error, so this is not
only a Replayer problem. The 1.35.0 release notes say: "A workflow task that
arrives while a workflow waits on a local activity, such as a legacy query, now
waits until the current workflow task is reported. Previously it replaced that
workflow task." This looks like a remaining case of the same interaction.
The failing 1.35.0 and 1.36.1 histories record core flag 4, so by the 1.34.0
notes ("Replay now preserves which local activity results were delivered
together for newly recorded histories", temporalio/sdk-rust#1616) they are
newly recorded histories. As we read it, that change keeps results that were
delivered together within one workflow task together on replay. Here, the
activation that delivered the result belonged to the next task and also
carried that task's other jobs.
Lang flag 2 does not help. We forced
PROCESS_WORKFLOW_ACTIVATION_JOBS_AS_SINGLE_BATCH (_WorkflowLogicFlag 2) on
for new histories by adding it to _DEFAULT_ENABLED_WORKFLOW_LOGIC_FLAGS. The
toy still failed 17 of 80 runs on 1.36.1, and the failing histories record
lang flag 2.
Questions:
- Flag 2 does not prevent this in our reproduction. Is it meant to cover
this case at all, and is it planned to become the default? - Should Core deliver a local activity result within the workflow task that
records its marker, even when a query arrives while the workflow waits on
that activity? Or should it record enough for replay to reproduce the
later delivery?
How often this happens in our own workflow's replay test (it polls with
queries; one replay per live run):
| SDK | replay failures |
|---|---|
| 1.32.0 | 9 / 80 (more under load) |
| 1.35.0 | 2 / 160 |
| 1.36.1 | 1 / 160 |
| 1.36.1, lang flag 2 forced on | 0 / 160 |
Minimal Reproduction
One file; it needs only temporalio. It starts a local dev server, runs 80
workflows on one worker while querying each of them every 5 ms, replays every
finished history with Replayer, and counts the failures.
uv run --with temporalio==1.36.1 python repro.py # queries every 5 ms
QUERY_POLL_MS=0 uv run --with temporalio==1.36.1 python repro.py # no queries: passes
FORCE_SINGLE_BATCH=1 uv run --with temporalio==1.36.1 python repro.pyThe workflow works through steps, each a short local activity. Between steps
it starts a local activity per pending signal and an activity per finished
child. Beside the steps runs a slower local activity that makes the outside
world signal the workflow twice, and a child workflow that finishes at about
the same time.
| SDK | no queries | queried every 5 ms |
|---|---|---|
| 1.32.0 | 80 / 80 (looks like #1881) | 80 / 80 |
| 1.35.0 | 0 / 80 | 11 / 80 |
| 1.36.1 | 0 / 240 | 26 / 80 (2 of them fail live) |
| 1.36.1, flag 2 forced on | 17 / 80 (2 of them fail live) |
repro.py
"""Replay nondeterminism with local activities: a self-contained reproduction.
A workflow works through steps; each step is a short local activity (a
"write"). Between steps it handles whatever has arrived: one local activity
per signal, one activity for a finished child workflow. Beside the steps runs
a slower local activity that makes the outside world signal the workflow
twice, and a child workflow that finishes at about the same time. While a
run is going, the driver queries it every few milliseconds, as a test might.
Every finished run's history is replayed with ``Replayer``; the script counts
the replay failures.
uv run --with temporalio==1.36.1 python repro.py
Knobs (environment variables, defaults in ``main``): RUNS, CONCURRENCY,
STEPS, STEP_MS, SLOW_MS, TASK_TIMEOUT_MS, QUERY_POLL_MS (query each running
workflow every N ms; 0 turns the polling off), MAX_CACHED (worker max_cached_workflows), TEMPORAL_CLI (path to an installed
``temporal`` CLI; the SDK downloads one otherwise), FORCE_SINGLE_BATCH=1
(turn lang logic flag 2 on for new histories), TRACE=1 (print the shapes of
live activations that resolve an activity together with other jobs).
"""
from __future__ import annotations
import asyncio
import dataclasses
import os
import random
import sys
import uuid
from collections import Counter
from datetime import timedelta
from temporalio import activity, workflow
with workflow.unsafe.imports_passed_through():
import temporalio
import temporalio.service
from temporalio.client import Client, WorkflowHistory
from temporalio.testing import WorkflowEnvironment
from temporalio.worker import Replayer, Worker
TASK_QUEUE = "la-replay-repro"
@dataclasses.dataclass
class StepsInput:
steps: int
step_ms: int
slow_ms: int
@dataclasses.dataclass
class WriteInput:
item: str
sleep_ms: int
jitter_ms: int = 3
notify: bool = False
@workflow.defn
class ChildWorkflow:
@workflow.run
async def run(self, delay_ms: int) -> str:
await workflow.sleep(timedelta(milliseconds=delay_ms))
return "child done"
class Activities:
def __init__(self, client: Client) -> None:
self._client = client
@activity.defn
async def write(self, inp: WriteInput) -> str:
"""Pretend to write somewhere. A notifying write makes the outside
world send the workflow two signals while it runs."""
total = (inp.sleep_ms + random.uniform(0, inp.jitter_ms)) / 1000
if not inp.notify:
await asyncio.sleep(total)
return inp.item
before = random.uniform(0, total)
await asyncio.sleep(before)
handle = self._client.get_workflow_handle(activity.info().workflow_id)
await handle.signal(StepsWorkflow.submit, f"{inp.item}-a")
await handle.signal(StepsWorkflow.submit, f"{inp.item}-b")
await asyncio.sleep(max(0.0, total - before))
return inp.item
@activity.defn
async def react(self, note: str) -> str:
return note
@workflow.defn
class StepsWorkflow:
def __init__(self) -> None:
self._inbox: list[str] = []
self._received = 0
@workflow.signal
def submit(self, item: str) -> None:
self._inbox.append(item)
self._received += 1
@workflow.query
def received(self) -> int:
return self._received
@workflow.run
async def run(self, inp: StepsInput) -> int:
slow: asyncio.Task[str] | None = None
child: workflow.ChildWorkflowHandle | None = None
rounds = 0
handled = 0
for step in range(inp.steps):
if slow is None or slow.done():
# Beside the steps: a slow write and a child workflow.
rounds += 1
child = await workflow.start_child_workflow(
ChildWorkflow.run,
inp.slow_ms,
id=f"{workflow.info().workflow_id}-child-{rounds}",
)
slow = asyncio.create_task(
workflow.execute_local_activity_method(
Activities.write,
WriteInput(f"slow-{rounds}", inp.slow_ms, inp.slow_ms // 2, True),
start_to_close_timeout=timedelta(seconds=30),
)
)
# The step's write.
await workflow.execute_local_activity_method(
Activities.write,
WriteInput(f"step-{step}", inp.step_ms),
start_to_close_timeout=timedelta(seconds=10),
)
# Handle whatever has arrived before the next step.
pending = []
items, self._inbox = self._inbox, []
pending.extend(
workflow.execute_local_activity_method(
Activities.write,
WriteInput(item, inp.step_ms),
start_to_close_timeout=timedelta(seconds=10),
)
for item in items
)
if child is not None and child.done():
child = None
handled += 1
pending.append(
workflow.execute_activity_method(
Activities.react,
f"child-{handled}",
start_to_close_timeout=timedelta(seconds=10),
)
)
if pending:
await asyncio.gather(*pending)
# Let the last slow write, its signals and the last child finish.
if slow is not None:
await slow
await workflow.wait_condition(lambda: self._received == 2 * rounds)
if child is not None:
await child
return rounds
def _flags(history: WorkflowHistory) -> tuple[set[int], set[int]]:
core: set[int] = set()
lang: set[int] = set()
for event in history.events:
if event.HasField("workflow_task_completed_event_attributes"):
meta = event.workflow_task_completed_event_attributes.sdk_metadata
core.update(meta.core_used_flags)
lang.update(meta.lang_used_flags)
return core, lang
def _force_single_batch() -> None:
"""Turn on lang logic flag 2 (PROCESS_WORKFLOW_ACTIVATION_JOBS_AS_SINGLE_BATCH)
for new histories, on SDK versions that have it."""
import temporalio.worker._workflow as wf_mod
import temporalio.worker._workflow_instance as inst_mod
flag = inst_mod._WorkflowLogicFlag.PROCESS_WORKFLOW_ACTIVATION_JOBS_AS_SINGLE_BATCH
flags = frozenset({*inst_mod._DEFAULT_ENABLED_WORKFLOW_LOGIC_FLAGS, flag})
inst_mod._DEFAULT_ENABLED_WORKFLOW_LOGIC_FLAGS = flags
wf_mod._DEFAULT_ENABLED_WORKFLOW_LOGIC_FLAGS = flags
LIVE_SHAPES: Counter[str] = Counter()
def _trace_live_activations() -> None:
"""Diagnostic: count live activations that resolve an activity together
with other kinds of jobs."""
import temporalio.worker._workflow as wf_mod
original = wf_mod._WorkflowWorker._handle_activation
async def traced(self, act): # type: ignore[no-untyped-def]
if not act.is_replaying:
kinds = [job.WhichOneof("variant") for job in act.jobs]
if "resolve_activity" in kinds and len(set(kinds)) > 1:
LIVE_SHAPES[",".join(kinds)] += 1
return await original(self, act)
wf_mod._WorkflowWorker._handle_activation = traced
async def main() -> None:
runs = int(os.environ.get("RUNS", "80"))
concurrency = int(os.environ.get("CONCURRENCY", "40"))
steps = int(os.environ.get("STEPS", "60"))
step_ms = int(os.environ.get("STEP_MS", "10"))
slow_ms = int(os.environ.get("SLOW_MS", "150"))
task_timeout_ms = int(os.environ.get("TASK_TIMEOUT_MS", "10000"))
run_timeout = float(os.environ.get("RUN_TIMEOUT_S", "60"))
query_ms = float(os.environ.get("QUERY_POLL_MS", "5")) # 0: no query polling
max_cached = os.environ.get("MAX_CACHED")
single_batch = os.environ.get("FORCE_SINGLE_BATCH") == "1"
if single_batch:
_force_single_batch()
if os.environ.get("TRACE") == "1":
_trace_live_activations()
print(
f"temporalio {temporalio.__version__}, python {sys.version.split()[0]}, "
f"runs={runs} concurrency={concurrency} steps={steps} step_ms={step_ms} "
f"slow_ms={slow_ms} task_timeout_ms={task_timeout_ms} "
f"force_single_batch={single_batch} query_poll_ms={query_ms} "
f"max_cached_workflows={max_cached or 'default'}",
flush=True,
)
async with await WorkflowEnvironment.start_local(
dev_server_existing_path=os.environ.get("TEMPORAL_CLI") or None,
) as env:
client = env.client
acts = Activities(client)
replayer = Replayer(workflows=[StepsWorkflow, ChildWorkflow])
failures: Counter[str] = Counter()
failed_flags: list[tuple[set[int], set[int]]] = []
stuck: list[str] = []
queries = Counter[str]()
async def poll(handle) -> None: # type: ignore[no-untyped-def]
"""Query the running workflow in a tight loop, as a test might."""
while True:
try:
await handle.query(StepsWorkflow.received)
queries["ok"] += 1
except Exception:
queries["error"] += 1
await asyncio.sleep(query_ms / 1000)
sem = asyncio.Semaphore(concurrency)
async def one(i: int) -> None:
async with sem:
handle = await client.start_workflow(
StepsWorkflow.run,
StepsInput(steps, step_ms, slow_ms),
id=f"la-repro-{i}-{uuid.uuid4().hex[:8]}",
task_queue=TASK_QUEUE,
task_timeout=timedelta(milliseconds=task_timeout_ms),
)
poller = asyncio.create_task(poll(handle)) if query_ms else None
try:
await asyncio.wait_for(handle.result(), run_timeout)
except asyncio.TimeoutError:
# The live worker keeps failing a workflow task (see its log).
stuck.append(handle.id)
try:
await handle.terminate("repro: stuck")
except temporalio.service.RPCError:
pass # it finished meanwhile
return
finally:
if poller:
poller.cancel()
history = await handle.fetch_history()
result = await replayer.replay_workflow(
history, raise_on_replay_failure=False
)
if result.replay_failure:
message = str(result.replay_failure).splitlines()[0]
failures[message[:160]] += 1
failed_flags.append(_flags(history))
if len(failed_flags) <= 3:
path = f"failed-history-{len(failed_flags)}.json"
with open(path, "w") as f:
f.write(history.to_json())
print(f"run {i}: replay failed: {message[:140]} ({path})", flush=True)
async with Worker(
client,
task_queue=TASK_QUEUE,
workflows=[StepsWorkflow, ChildWorkflow],
activities=[acts.write, acts.react],
**({"max_cached_workflows": int(max_cached)} if max_cached else {}),
):
await asyncio.gather(*(one(i) for i in range(runs)))
print(f"\nreplay failures: {sum(failures.values())}/{runs - len(stuck)} completed runs")
print(f"runs stuck live on a failing workflow task: {len(stuck)}/{runs}")
if query_ms:
print(f"queries: {dict(queries)}")
for message, count in failures.most_common():
print(f" {count} x {message}")
if failed_flags:
core = set().union(*(c for c, _ in failed_flags))
lang = set().union(*(lf for _, lf in failed_flags))
print(f"flags in failing histories: core={sorted(core)} lang={sorted(lang)}")
for shape, count in LIVE_SHAPES.most_common(8):
print(f" live activation x{count}: [{shape}]")
if __name__ == "__main__":
asyncio.run(main())Environment/Versions
- OS and processor: reproduction on macOS, Apple silicon, Python 3.15.2. Our own workflow's runs: the same (macOS, Apple silicon, Python 3.15.2).
- Temporal Version: SDK 1.32.0, 1.35.0 and 1.36.1 (
temporaliofrom PyPI). The reproduction used the Temporal CLI 1.5.0 dev server (Server 1.29.0). Our own runs: the same SDK versions, against a local Temporal test server started by the SDK's testing module. - Are you using Docker or Kubernetes or building Temporal from source? Not for the reproduction (CLI dev server). Our own runs: no (local test server).
Additional context
- [Bug] Local activity resolutions regrouped on replay since 1.32.0, delivering the wrong payload #1881 (closed): local activity results regrouped on replay since 1.32.0.
- [Bug] non determinism when using
asyncio.gatherwith local activities #1578 (open): concurrent local activities withasyncio.gather. - Record which activations LAs are resolved in sdk-rust#1616: records which activation delivered each local
activity result and keeps those groups on replay (1.34.0). - Allow LAs to wake up workflow while another is still executing sdk-rust#1442: since 1.32.0, a local activity result can wake the
workflow while other local activities are still running.
Activity