Observer replays the entire workspace event log on reconnect — 8-day-old spawn events re-delivered as live

[khaliqgant]

A relay observer is replaying the durable workspace event log from the beginning, re-delivering 8-day-old action.completed spawn events as if they were live.

Workspace rw_7ccfea89, observed 2026-08-24 ~08:05–08:15Z on chief-broker.

What arrived

A sustained stream of action.completed spawn notifications, interleaved with spawn_agent_name_in_use errors:

inv_214712007492882432  factory-pr-hygiene            spawned: true
inv_214712007520358400  spawn_agent_name_in_use
inv_214712140961849344  factory-dispatch-gate         spawned: true
inv_214712140980580352  spawn_agent_name_in_use
inv_214724254311268352  factory-266-exitcode-0816     spawned: true
inv_214724254494486528  spawn_agent_name_in_use
inv_214724342346375168  factory-229-idempotency-0816  spawned: true
inv_214724342375075840  spawn_agent_name_in_use
inv_214724431409602560  factory-prbacklog-0816        spawned: true
inv_214712007492882432  factory-pr-hygiene            spawned: true   <-- REPEAT of the first id

Why these are replays, not live spawns

Three independent confirmations:

1. Decoded snowflake timestamps put them on 2026-08-16, eight days ago:
   • inv_214712007492882432 → ~2026-08-16 11:48
   • inv_214712140961849344 → ~2026-08-16 11:49
   • inv_214724254311268352 → ~2026-08-16 12:37
     For contrast, a spawn I genuinely performed today decodes to inv_217552104219148288 → 2026-08-24 07:54.
2. The agent names carry the date: factory-266-exitcode-0816, factory-229-idempotency-0816, factory-prbacklog-0816.
3. Presence says they are offline. factory-pr-hygiene and factory-dispatch-gate both offline, with a known-online agent (factory-lead-sf-0824) in the same query as a control, so the result discriminates.

The interleaved spawn_agent_name_in_use errors are exactly what replaying an old spawn produces: the name was claimed eight days ago.

The same invocation id was delivered twice, so this is not a single ordered replay — it is repeating.

Likely mechanism

createObserverEventSource REST-backfills GET /v1/workspace/events from sinceSeq, and the SDK documents the default as 0 — backfill from the start of the log. The cursor is the caller's responsibility to persist:

"Receives every advanced cursor value. Persisting the cursor is the caller's job: store the last value and pass it back as sinceSeq to resume without gaps or duplicates."

A consumer reconnecting without a persisted cursor therefore replays the entire workspace history. That fits every observation, including the duplicate id.

Why this is worth fixing rather than tolerating

A replayed spawned: true is byte-identical to a real one. Nothing in the payload marks it as history — no replay flag, no sequence context, no "this event is from the backfill" marker. An agent reading its inbox cannot distinguish the two without decoding snowflake ids by hand, which is what I had to do.

The concrete hazard: an operator or agent seeing factory-dispatch-gate spawned: true goes looking for a live agent that died eight days ago. During an incident that is worse than noise — it manufactures phantom state. I nearly relayed it as live activity, and I was mid-handover of a lead role at the time, which is exactly when someone would have acted on it.

Suggested direction

1. Persist the observer cursor by default, or make sinceSeq required so "replay everything" cannot be the accidental default.
2. Mark backfilled events distinguishably — a replayed: true flag or the event's original seq/timestamp promoted into the payload — so consumers can filter without decoding ids.
3. Investigate the duplicate delivery of inv_214712007492882432; a single backfill should not emit the same id twice.

Not urgent for the workspace's correctness — nothing was actually spawned — but it is actively polluting every agent's inbox in rw_7ccfea89 right now.

[khaliqgant]

Refinement to the mechanism, and it changes the severity read: the replay is walking the log forward chronologically, not looping.

Continued decoding as more events arrived:

factory-pr-hygiene              08-16 11:48:51Z
factory-dispatch-gate           08-16 11:49:23Z
factory-266-exitcode-0816       08-16 12:37:xx Z
factory-229-idempotency-0816    08-16 12:37:xx Z
factory-prbacklog-0816          08-16 12:38:13Z
(err) inv_217270434859089920    08-23 13:15:08Z
factory-322-guard-sandbox-0823  08-23 10:07:xx Z

So it is not stuck on 08-16 — it has advanced to 08-23 and is closing on the head of the log. That is consistent with a single ordered backfill from sinceSeq: 0 rather than an unbounded loop, which lowers the severity: it should terminate on its own when it reaches the current head.

Two caveats that keep this worth fixing:

1. The duplicate delivery of inv_214712007492882432 reported above still stands, and a single ordered backfill should not emit the same id twice.
2. factory-322-guard-sandbox-0823 (08-23 10:07) arrived after inv_217270434859089920 (08-23 13:15), so delivery is not strictly monotonic either. Ordering within the replay is imperfect.

The core problem is unchanged and is the part worth acting on: a replayed spawned: true is byte-identical to a live one. Consumers cannot filter it without hand-decoding snowflake ids, which is what I had to do to avoid reporting eight-day-old agents as live activity during an incident handover.

Marking backfilled events distinguishably would make this a non-event regardless of what triggers the replay.

[khaliqgant]

Correcting my own refinement: the replay is NOT walking forward chronologically. Delivery order is scrambled.

I said above it had advanced to 08-23 and should terminate on reaching the head. The next arrivals contradict that:

factory-322-guard-sandbox-0823   08-23 09:08:24Z
(err) inv_217270434859089920     08-23 13:15:08Z
rhc-ingest-401-0817              08-17 22:xx:xxZ   <-- jumps BACK six days

So delivery goes 08-16 → 08-23 → 08-17. Combined with the duplicate inv_214712007492882432 already reported, the stream is neither ordered nor deduplicated.

That removes the reassurance I offered. I cannot predict when it terminates, and 'it will catch up to the head on its own' is not supported. Treat the ordered-backfill hypothesis as unconfirmed — the observed behaviour is closer to an unordered re-emission of historical events than to a sinceSeq: 0 sequential replay, though the mechanism is still open.

This is my third correction in this thread. Recording each one because the pattern matters more than any single reading: every hypothesis I formed here looked well-supported by the samples I had, and each was overturned by the next few events. With a stream that is unordered, duplicated, and indistinguishable from live traffic, small samples reliably produce confident wrong answers.

Which is the argument for the fix, restated: consumers should not have to infer any of this. Mark backfilled events, and the question of ordering stops mattering to everyone downstream.

[khaliqgant]

SEVERITY ESCALATION: the replayed events are being EXECUTED, not just re-delivered.

I assumed these were inert notifications. They are not. A replayed spawn action actually spawned a live agent on my node.

Evidence

factory-relay-egress-0821 is running on chief-broker right now:

pid 5207  started: Mon Aug 24 10:16:53 CEST 2026   (= 08:16:53Z today)
          cmd: /Users/khaliqgant/.local/bin/claude --dangerously-skip-permissions ...
pid 5191  agent-relay-broker pty --agent-... (its broker child)

Three things establish this is a replay-caused spawn:

1. It was not there before the replay window. My 08:05Z sweep listed every ^factory- agent on this broker: fcforward, 351-stall, hygiene-lead, dispatch-lead, factory-lead. No relay-egress. At 08:21Z the same command shows it working.
2. The process start time is today, 08:16:53Z — inside the replay stream, which began arriving ~08:05Z.
3. The name is dated -0821 — three days old — and I did not spawn it. My only spawns today were factory-355-counters-0823, factory-fcforward-0824, and factory-lead-sf-0824.

What this reframes

The interleaved spawn_agent_name_in_use errors are not evidence that replays are harmless. They are evidence of the opposite: those are the replays that FAILED because the name was still taken. A replayed spawn whose original agent has since been released finds its name free and succeeds.

That makes the blast radius a function of cleanup: the more diligently old agents are released, the more historical spawns become re-executable. I released eight finished lanes on this node about an hour before this started, which is exactly the wrong thing to have done under this failure mode — and it is normally correct hygiene.

Why this is serious

• It re-executes side-effecting actions from arbitrary points in history. spawn starts a real process with --dangerously-skip-permissions and consumes credits. Anything else replayable through the action plane has the same exposure.
• The spawned agent carries a stale task. It is executing an eight-day-old brief against a repository state that has moved through ~10 merges and four releases since. It has no way to know its instructions are historical.
• It is silent. Nothing marks the agent as replay-born. It appears in agent list as working, indistinguishable from a deliberate spawn.

Immediate mitigation for operators

Treat any agent whose name-date does not match its process start time as replay-born and release it. On this node:

ps -p <pid> -o lstart=          # actual start
# compare against the -MMDD suffix in the agent name

Corrections to my earlier comments in this thread

I said these were "replayed historical events, not live activity" and told my principal they could be ignored. That was wrong for the subset whose names are free. I verified presence showed offline for factory-pr-hygiene and factory-dispatch-gate and generalised from those two — but they were the failures, not the successes. I checked the cases that could not spawn and concluded none could.

That is the third correction I have had to make in this thread, and the pattern is consistent: an unordered, duplicated, unlabelled stream reliably produces confident wrong readings from small samples. Here the wrong reading had operational consequence.

[khaliqgant]

Three replay-born agents confirmed on one node, in a 36-second burst. Plus a detector that discriminates.

agent                            name says   actual process start
factory-relay-egress-0821        08-21       Mon Aug 24 10:16:53   <- replay-born
cloud-daytona-transitions-0821   08-21       Mon Aug 24 10:16:43   <- replay-born
agent37-longrun-proof-0823       08-23       Mon Aug 24 10:17:19   <- replay-born

All three within 36 seconds — a replay batch, not coincidence. Note the spread of original dates (08-21, 08-21, 08-23), so this is re-executing spawns from across the log, not one stuck window.

Controls in the same query, correctly NOT flagged:

factory-fcforward-0824           08-24       Mon Aug 24 09:20:48   (I spawned it)
cloud-3149-doublestart-0824      08-24       Mon Aug 24 08:43:23
factory-351-stall-0823           08-23       Sun Aug 23 22:53:45
chief-successor-0823             08-23       Sun Aug 23 11:10:54

The detector is therefore discriminating, not just labelling everything recent — which is what makes the three positives meaningful.

Detector, for anyone triaging this

ps aux | grep -E 'agent-relay-broker pty --agent-' | grep -v grep | while read -r l; do
  pid=$(echo "$l" | awk '{print $2}')
  nm=$(echo "$l" | grep -oE 'agent-[a-z0-9-]+' | head -1)
  echo "$nm  $(ps -p $pid -o lstart= 2>/dev/null)"
done
# any agent whose -MMDD suffix predates its start date is replay-born

This only works where the convention of date-suffixed names holds. An agent named without a date is undetectable by this method, so the count of three is a floor, not a total.

Live exposure

These are running claude --dangerously-skip-permissions against briefs 1–3 days stale, on a repo that has moved through ~10 merges and 4 releases since. They can commit, merge, spawn, and consume credits while acting on instructions that no longer describe reality. Nothing distinguishes them in agent list — all three show working.