Skip to content

Events with equal timestamps reload out of append order (SqliteSessionService, DatabaseSessionService), breaking Workflow replay #7286

Description

@rishwanth-th

🔴 Required Information

Describe the Bug:

Session services reload a session's events ordered by (timestamp, id). Event ids
are random (Event.new_id()), so when two events carry the same timestamp they
come back in a random-but-stable order, not in the order they were appended. This
happens in SqliteSessionService (ORDER BY timestamp DESC, id DESC, with a
comment saying the id tie-break makes the order stable across reads) and in
DatabaseSessionService (order_by(timestamp.desc(), id.desc())).

For a resumable graph Workflow this breaks resume. ReplayManager builds the
replay sequence barrier from the reloaded events, so a swapped pair gives the
barrier a completion order the workflow can never follow, and the resume fails
after the 15 s barrier timeout:

RuntimeError: Replay divergence detected: Timed out waiting for sequence key 'wait_test_key@1' to be unblocked.

Equal timestamps are common: Python 3.12 on Windows ticks every 15.6 ms by default,
and fast function nodes emit several events per tick on any OS. It is a different
cause from #7027 (re-emitted outputs reordering the sequence in memory, fixed in
#7028): this one is in storage, and the repro below involves no workflow at all.

Steps to Reproduce:

  1. pip install "google-adk[db]==2.9.2" (plus asyncpg for Postgres)
  2. Save the script below as repro.py
  3. python repro.py sqlite, or python repro.py "sqlite+aiosqlite:///./r.db", or
    python repro.py "postgresql+asyncpg://user:pass@localhost:5432/db"
  4. The loaded order is the reverse of the append order

Expected Behavior:

get_session returns events in append order, including events with equal
timestamps.

Observed Behavior:

SqliteSessionService append order: ['b-appended-first', 'a-appended-second']
SqliteSessionService loaded order: ['a-appended-second', 'b-appended-first']
OUT OF ORDER

The same for DatabaseSessionService on SQLite and on Postgres 17.

In a resumable Workflow (a question loop: wait → classify → answer → wait,
SqliteSessionService), forcing ties with a 15.625 ms clock through
google.adk.platform.time.set_time_provider made resume fail with the divergence
above at loop 7; the same workflow with a strictly increasing clock passed every
loop.

Environment Details:

  • ADK Library Version: 2.9.2
  • Desktop OS: Windows 11; also a Linux container against Postgres 17.11
  • Python Version: 3.12.14
  • SQLAlchemy 2.0.54, aiosqlite 0.22.1, asyncpg 0.31.0

Model Information:

  • Are you using LiteLLM: Yes (not involved)
  • Which model is being used: none needed for the repro

🟡 Optional Information

Minimal Reproduction Code:

import asyncio
import sys
import tempfile
import time
from pathlib import Path

from google.adk.events import Event
from google.adk.sessions import DatabaseSessionService
from google.adk.sessions.sqlite_session_service import SqliteSessionService


async def main(target: str) -> None:
    if target == "sqlite":
        service = SqliteSessionService(db_path=str(Path(tempfile.mkdtemp()) / "r.db"))
    else:
        service = DatabaseSessionService(db_url=target)
    session = await service.create_session(app_name="repro", user_id="u")

    ts = time.time()
    await service.append_event(session, Event(id="b-appended-first", author="a", timestamp=ts))
    await service.append_event(session, Event(id="a-appended-second", author="a", timestamp=ts))

    loaded = await service.get_session(app_name="repro", user_id="u", session_id=session.id)
    order = [e.id for e in loaded.events]
    print("append order: ['b-appended-first', 'a-appended-second']")
    print("loaded order:", order)


asyncio.run(main(sys.argv[1]))

How often has this issue occurred?:

  • Always (100%) with the script above.
  • In a workflow, whenever two events share a timestamp; frequent with coarse clocks.

Additional Context:

Possible fixes: order by insertion rather than by a random id on ties (SQLite
rowid, or an autoincrement / sequence column in DatabaseSessionService), or make
event ids monotonic.

User-side workaround: a strictly increasing clock through
google.adk.platform.time.set_time_provider (no two events in a process share a
timestamp). It does not help across processes.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions