Python: add opt-in deferred session persistence to foundry_hosting

[Harsheet Shah (harsheet-shah)]

Summary

Move the tail AgentSession write off the response critical path when history_source="agent_server", where AgentServer already holds the durable transcript (so a still-pending session write is redundant with it). Gated behind a new defer_session_persistence flag (default False); default behavior is unchanged.

On the success path the session write is scheduled after response.completed instead of blocking it, cutting ~one storage round-trip (~140 ms p50, larger p99) from warm TTLB.

Correctness

Preserved by:

• a next-turn read-your-writes barrier that awaits a pending write for the same session before loading it, keeping per-session writes ordered;
• a shutdown drain in _cleanup_agent so graceful deactivation stays durable;
• synchronous persistence on the failure/interrupt paths (never deferred).

The only relaxed guarantee vs. the default: an abrupt, non-graceful termination in the brief window after response.completed may drop the last turn's session write, which is rebuilt from the AgentServer transcript on the next turn. Requesting the flag with history_source="agent" raises, since the session can be the sole record of the turn there.

Tests

Adds TestDeferredSessionPersistence covering the non-blocking behavior, shutdown drain, read-your-writes barrier, failure-path synchronous persist, and the config guard. Full test_responses.py suite passes.

[Copilot]

🟡 Changes recommended

Critical deferred-write loss and continuation-recovery issues, along with additional durability and scope gaps, remain unresolved.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Adds opt-in deferred AgentSession persistence for AgentServer-backed history to reduce response latency while preserving synchronous default behavior.

Changes:

• Adds background writes, ordering barriers, and shutdown draining.
• Preserves synchronous failure-path persistence and adds configuration validation.
• Adds focused persistence and lifecycle tests.

File summaries

File	Summary
python/packages/foundry_hosting/agent_framework_foundry_hosting/_responses.py	Implements deferred persistence. Unresolved findings cover failed-write recovery, workflow support, provider-state durability, tenant-scoped queues, and incomplete OAuth responses.
python/packages/foundry_hosting/tests/test_responses.py	Adds deferred-persistence coverage; the failure test does not prove synchronous behavior because the store completes immediately.

Review details

Suppressed comments (4)

python/packages/foundry_hosting/agent_framework_foundry_hosting/_responses.py:424

• AgentSession is not only the AgentServer transcript: context-provider state is stored in AgentSession.state, and agent-server history still permits non-history context providers. Because this option defers the entire snapshot, an abrupt exit can lose state changed by the last turn; the AgentServer transcript can rebuild messages but not that provider state. Please make this durability limitation explicit (or restrict deferral to transcript-derived sessions).

                pending writes. The only relaxed guarantee versus the default: after an abrupt,
                non-graceful termination in the brief window after `response.completed`, the last
                turn's session write may be lost, and it is rebuilt from the AgentServer transcript on
                the next turn. Defaults to False (persist synchronously before completing).

python/packages/foundry_hosting/agent_framework_foundry_hosting/_responses.py:636

• The pending queue is keyed only by the opaque session ID, but the default Foundry session store is user-isolated and is created from each request's platform context. Two users can supply the same conversation ID, causing one user's slow or stuck write to become the other user's barrier/parent and create cross-tenant blocking. Include the storage scope/tenant in this key or otherwise isolate pending queues per store scope.

        previous = self._pending_session_writes.get(session_id)

        async def _persist() -> None:
            if previous is not None:

python/packages/foundry_hosting/agent_framework_foundry_hosting/_responses.py:985

• This condition treats every non-exceptional run as a success, but _handle_response emits response.incomplete when tracker.oauth_consent_requested is true. A mid-run OAuth-consent response is a resumable, non-success turn whose session snapshot must be durable for the retry; deferring it can leave the client with an incomplete response but no persisted session if the background write fails or is interrupted. Restrict deferral to completed responses, for example by also requiring not tracker.oauth_consent_requested.

                defer_write = self._defer_session_writes and request_failure is None and not request_interrupted

python/packages/foundry_hosting/tests/test_responses.py:1683

• This test cannot distinguish synchronous failure persistence from an incorrectly deferred write: SessionStore.set completes immediately, so a background task can finish before these assertions run. Use a gated/recording store and verify the failed response does not complete until set is released, then assert the pending-write map is empty.

        # Failures are never deferred: the session is persisted synchronously on the failure
        # path, so nothing is left pending after the response resolves.
        assert resp.json()["status"] == "failed"
        assert not server._pending_session_writes  # pyright: ignore[reportPrivateUsage]
        assert await store.get(resp.json()["id"]) is not None

• Files reviewed: 2/2 changed files
• Comments generated: 3
• Review effort level: Lite (auto)

Note

Copilot is running an experiment and ran this review at Lite.

---

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

[Evan Mattson (moonbox3)]

Should the pending write be shielded from cancellation of the request waiting on it? A cancelled continuation propagates cancellation through await pending, cancels persistence for the already-completed turn, and suppress(BaseException) then lets the continuation proceed to load stale or missing state. Could we await asyncio.shield(pending), handle the write failure separately, and allow the continuation's CancelledError to propagate?

[Evan Mattson (moonbox3)]

What happens if the response stream is cancelled after _handle_inner_agent returns but before _handle_response emits response.completed? At that point this has already classified the turn as successful and scheduled a background write, so cancellation while yielding tracker.close() or the terminal event never sets request_interrupted; the interruption can finish without awaiting persistence and lose session state. Could the defer decision move to the outer completion boundary, or could that outer cancellation drain this write synchronously?