surestream

Broken LLM streams should raise, not return. surestream wraps the stream you already have (openai SDK, anthropic SDK, httpx, anything iterable) and makes truncation and stalls loud, then optionally resumes the answer where it stopped.

# before: the stream dies mid-answer, the loop just ends, you ship half an answer
for chunk in client.chat.completions.create(**request):
    handle(chunk)

# after: same chunks, but StreamTruncated / StreamStalled if the answer did not finish
for chunk in guard(client.chat.completions.create(**request), first_chunk_timeout=20, inter_chunk_timeout=10):
    handle(chunk)

• guard / aguard yield the same chunks, unchanged, and raise when the stream ends without a terminal signal or stops producing content. Keep-alive pings do not count as content.
• resumable / aresumable re-issue a broken request so that the server continues the partial answer, and splice the continuation into one stream.
• hedged / ahedged start a second request when the first token is slow, and keep whichever answers first.
• Small, typed, zero dependencies. Sync and asyncio. The SDKs are optional and never imported.

pip install git+https://github.com/cb-Sigma/surestream

Python 3.10+. Not on PyPI yet.

Contents

• What the SDKs do today
• API tour: guard · resumable · hedged · collect · testing
• What "resume" means, precisely
• Overhead
• What was tested, and what was not
• Related work · FAQ · Roadmap · Contributing

What the SDKs do today

Observed on 2026-11-13 with openai 3.50.0 and anthropic 2.8.0 (both on httpx2 2.16.0, Python 3.13.18, Linux), pointed at a local server that breaks its responses on purpose. Reproduce it with examples/what_the_sdks_do.py; the full output is in docs/observed-behaviour.md.

The stream...	openai SDK	anthropic SDK
ends early but cleanly (body terminated properly, no finish_reason / [DONE] / message_stop)	returns normally. Iteration just stops. chat.completions.stream() → get_final_completion() returns the partial text with finish_reason=None. Same for responses.create(stream=True): the last event is a text delta.	returns normally. Last event is a content_block_delta. messages.stream() → get_final_message() returns the partial text with stop_reason=None.
loses its TCP connection mid-body	raises openai.APIConnectionError	raises httpx2.RemoteProtocolError (not an SDK exception type)
goes completely silent	raises openai.APITimeoutError once the client's read timeout expires (2 s in this run; the SDK default is 10 minutes)	raises httpx2.ReadTimeout once the read timeout expires (same)
stalls while keep-alive pings keep arriving (client timeout=2.0, a ping every 0.5 s)	keeps blocking, no exception (still blocked when observation stopped after 6 s)	keeps blocking, no exception (same)

So two of the four failure shapes are silent: a response that is cut short cleanly (what you get when a gateway, proxy or the server itself swallows an upstream failure and ends the response normally) looks exactly like a short answer, and a stall behind keep-alives never times out because every ping resets the socket read timeout.

These are known and reported, not news:

• openai/openai-python#4063 (open, Oct 2026): "ChatCompletionStream.get_final_completion() can return an incomplete result after an early EOF".
• anthropics/anthropic-sdk-python#2051 (open, Nov 2026): "ping events keep a stalled stream alive; an event-idle timeout would let callers tell "stalled" from "slow"".
• openai/openai-python#2319 (closed, Apr 2025): a streaming call "stalling without throwing exception" in production, never reproduced reliably.

surestream does not patch the SDKs. It wraps what they return.

The stream...	with guard(...)
ends early but cleanly	StreamTruncated, carrying the text and chunks received
loses its connection	the SDK's exception, unchanged; partial text on stream.text
goes silent	StreamStalled after your deadline; response closed
stalls behind pings	StreamStalled after your deadline; response closed

API tour

The snippets below are abbreviated (messages, log and friends are yours). Complete scripts for each of them are in examples/; they run as they are, against the fake server that ships in surestream.testing, and are executed by the test suite. Start with examples/quickstart.py, which needs nothing but the standard library.

guard: fail loudly

from openai import OpenAI
from surestream import StreamStalled, StreamTruncated, guard

client = OpenAI()
stream = guard(
    client.chat.completions.create(model="your-model", messages=messages, stream=True),
    first_chunk_timeout=20,  # seconds until the first content-bearing chunk
    inter_chunk_timeout=10,  # seconds between content-bearing chunks
)
try:
    for chunk in stream:  # the SDK's own ChatCompletionChunk objects, untouched
        print(chunk.choices[0].delta.content or "", end="")
except StreamTruncated as exc:
    log.warning("answer cut off after %d chars", len(exc.text))  # exc.text, exc.chunks
except StreamStalled as exc:
    log.warning("no content for %ss (%s)", exc.timeout, exc.phase)  # "first_chunk" | "inter_chunk"

stream.summary  # StreamSummary(text=..., complete=..., finish_reason=..., usage=..., ttft=...)

Async is the same with aguard (asyncio):

from anthropic import AsyncAnthropic
from surestream import aguard

client = AsyncAnthropic()
stream = aguard(
    client.messages.create(model="your-model-id", max_tokens=1024, messages=messages, stream=True),
    first_chunk_timeout=20,
    inter_chunk_timeout=10,
)
async for event in stream:
    ...

Raw HTTP works too. Hand over the response (or httpx's client.stream(...) context manager) and surestream iterates its lines and closes it:

import httpx
from surestream import guard

with httpx.Client() as http:
    stream = guard(http.stream("POST", url, json=body), inter_chunk_timeout=10)
    for line in stream:  # raw SSE lines, exactly as httpx yields them
        ...

What is recognised automatically, by shape and without importing any SDK:

Stream	Terminal signal
openai SDK chat chunks and legacy completions; the same JSON as dicts from vLLM, llama.cpp, SGLang, ...	finish_reason on every choice
openai SDK Responses API events	response.completed / response.incomplete
anthropic SDK events (messages.create(stream=True) and the messages.stream() helper)	message_stop
raw SSE lines (str or bytes) from httpx, requests, aiohttp	the above, or data: [DONE]
anything else	is_terminal=lambda chunk: ..., or a ten-line custom adapter

Things to know:

• Timers only reset on content. Pings, SSE comments, role-only and empty deltas are passed through to you but do not buy the stream more time. Override with is_content=.
• Without a timeout, guard is a plain wrapper: no thread, it only checks termination.
• With a timeout in sync code, the stream is read on a helper thread. A thread blocked in a socket read cannot interrupt itself, and a socket timeout is reset by every ping, so the read moves to a helper thread and your thread waits with a deadline. When the deadline passes, surestream shuts the socket down, joins the thread, closes the response and raises. The cost is one short-lived thread per stream and a queue hand-off per chunk; the details, the alternatives that were rejected and the one case where the thread can outlive the deadline are in docs/design.md.
• Async needs no thread and no extra task: the pending read is cancelled from a timer. Time your code spends between chunks is never counted.
• Exceptions from the SDK or transport are never wrapped. stream.text and stream.chunks stay available after any exception.

resumable: continue the answer

You pass the request as a dict and a function that sends it. When the stream is truncated, stalls or drops, surestream sends a modified request that asks the server to continue the partial text and keeps yielding:

from openai import OpenAI
from surestream import resumable

client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")  # vLLM
request = dict(model="your-model", messages=messages, temperature=0, stream=True)

stream = resumable(
    lambda req: client.chat.completions.create(**req),
    request,
    continuation="vllm",  # assistant prefill + continue_final_message
    max_resumes=2,
    first_chunk_timeout=20,
    inter_chunk_timeout=10,
    on_resume=lambda info: log.info("resume #%d after %r", info.attempt, info.error),
)
for chunk in stream:  # one uninterrupted stream of ChatCompletionChunk objects
    print(chunk.choices[0].delta.content or "", end="")

stream.summary  # .text .complete .resumes .finish_reason .usage .usages

continuation=	For	Mechanism
"completions"	any OpenAI-compatible /v1/completions	prompt + generated text
"vllm"	vLLM chat	assistant prefill, continue_final_message=true, add_generation_prompt=false
"sglang"	SGLang chat	assistant prefill, continue_final_message=true
"llamacpp"	llama.cpp llama-server chat	a trailing assistant message is a prefill (server default)
"anthropic-prefill"	Anthropic models that accept prefill	trailing assistant message
"instruction"	any chat API, including Anthropic models that reject prefill	a user message quoting the partial answer and asking to continue (approximate)

Each one is traced to documentation or server source in docs/continuation.md. There is no strategy for the OpenAI Chat Completions or Responses APIs because they document no way to continue a partial assistant message.

Async, with the anthropic SDK:

from anthropic import AsyncAnthropic
from surestream import ResumeUnsupported, aresumable

client = AsyncAnthropic()
request = dict(model="your-model-id", max_tokens=1024, messages=messages, stream=True)
stream = aresumable(
    lambda req: client.messages.create(**req), request, continuation="anthropic-prefill", inter_chunk_timeout=15
)
try:
    async for event in stream:  # exactly one message_start ... one message_stop
        ...
except ResumeUnsupported as exc:  # e.g. the partial answer contained a tool call
    partial = exc.text

hedged: race a slow first token

from surestream import guard, hedged

race = hedged(lambda: client.chat.completions.create(**request), delay=1.5)
for chunk in guard(race, inter_chunk_timeout=10):
    ...
race.hedged, race.winner  # was a second request started, and which one won (0 or 1)

If no content-bearing chunk has arrived after delay seconds, the same request is issued a second time. Whichever produces the first token is yielded; the other is cancelled and its connection closed. Hedging only ever happens before the first token, and it costs a second request: use it for idempotent generations where time-to-first-token matters more than the duplicate. ahedged is the asyncio version.

collect: text plus a completeness flag

from surestream import collect

result = collect(client.chat.completions.create(**request), inter_chunk_timeout=10)
result.text, result.complete, result.finish_reason, result.usage, result.error

collect / acollect never raise for truncation, stalls or dropped connections; they hand back whatever arrived with complete=False and the exception in error. They also accept a guard or resumable stream.

Test your own handling

The fake server the library is tested with is public API and needs only the standard library:

# conftest.py
pytest_plugins = ["surestream.testing.plugin"]

# test_my_client.py
from surestream.testing import Fault


def test_truncated_answer_is_flagged(fake_llm):
    fake_llm.inject(Fault.truncate(after=3))  # or disconnect / stall / slow_first_token / error
    answer = my_client(base_url=fake_llm.openai_base_url).ask("hi")
    assert answer.incomplete

It speaks /v1/chat/completions, /v1/completions, /v1/messages and /v1/responses. There is also CutProxy, a small TCP proxy that cuts or stalls connections to a real server. See docs/testing.md.

What "resume" means, precisely

Resuming is not replaying. The server generates new tokens, conditioned on the text you already received.

• temperature=0, no sampler penalties: the result can be exact. Measured against a real llama.cpp server at temperature 0 with the repetition penalty off: resumed streams, including one cut three times, were character-identical to the uninterrupted run (details below).
• A repetition penalty breaks exactness, even at temperature 0. With llama-cpp-python's default repeat_penalty=1.1, none of the resumed runs matched: each diverged a few tokens after the seam into a different, fluent answer. Penalties look at tokens generated in the current request, and after a resume the partial answer is prompt, not generation. The same goes for frequency/presence penalties, DRY and mirostat.
• With sampling, you get a plausible continuation, not the answer the interrupted request would have produced. A fixed seed does not help.
• "instruction" is an approximation. The model is asked, not prefilled; it may repeat or rephrase. ResumeInfo.exact is False for it.
• Not resumed, by design: once content has been delivered, streams containing tool calls or reasoning/thinking, and requests using tools, structured-output constraints (response_format, guided decoding, grammars) or n > 1, raise ResumeUnsupported with the partial result instead. A text continuation of those would be wrong, not just different.
• If nothing was delivered yet, the original request is simply sent again, whatever it contains.
• In-band error events are not resumed by default, because an overload and a policy stop look alike. Opt in with resume_on=.
• Chunk ids and timestamps differ across the seam; each request gets the full max_tokens unless you pass a tokenizer; the resumed request re-sends the prompt plus the partial answer as input.

All of this, with sources, is in docs/continuation.md.

Measured on a real engine

llama-cpp-python 0.3.36 serving SmolLM2-135M-Instruct (Q8_0) on CPU, /v1/completions, openai SDK 3.50.0, a TCP proxy cutting the stream (examples/real_engine_check.py):

$ python examples/real_engine_check.py --url http://127.0.0.1:18080 --extra-body '{"repeat_penalty": 1.0}'
openai 3.50.0, model models/smollm2-135m-instruct-q8_0.gguf, api=completions, extra_body={'repeat_penalty': 1.0}

uninterrupted run: 80 chunks, 385 characters
cold first run identical to warm runs: True
two warm uninterrupted runs identical: True
reference text: [...]

(a) bare SDK, stream cut after 26 chunks
  body ended cleanly (as a gateway does):
    returned normally: 130 characters
  TCP connection dropped mid-body:
    raised openai.APIConnectionError: Connection error.

(b) guard()
  body ended cleanly: raised StreamTruncated with 130 characters received
  connection dropped: raised APIConnectionError with 130 characters received
  stream went silent: raised StreamStalled with 130 characters received

(c) resumable(continuation='completions'), temperature 0
  clean cut at 1/3: resumes=1 finish_reason=stop identical_to_uninterrupted=True
  dropped at 1/2: resumes=1 finish_reason=stop identical_to_uninterrupted=True
  silent at 2/3: resumes=1 finish_reason=stop identical_to_uninterrupted=True
  three cuts: resumes=3 finish_reason=stop identical_to_uninterrupted=True

4/4 resumed runs were character-identical to the uninterrupted run

The script discards a first warm-up request before taking its reference: in one session the very first request after a server start produced a different answer than every later identical request, a reminder that determinism at temperature 0 has conditions on the server side too.

The same run with the server's default repeat_penalty=1.1 gives 0/4, for the reason above. Both outputs, and the runs against a native llama-server, are in docs/observed-behaviour.md.

Overhead

Extra time per chunk compared with iterating the bare stream, measured with benchmarks/overhead.py on in-memory chunks (no network, so this is surestream's own cost), 20 001 chunks, median of 9 runs, on a shared 4-core x86-64 VM:

	Python 3.12.3	Python 3.13.18
guard(stream)	3.1 µs	2.8 µs
guard(stream, keep_chunks=False)	2.9 µs	2.7 µs
guard(stream, inter_chunk_timeout=30) (helper thread)	3.9 µs	3.4 µs
aguard(stream)	3.6 µs	3.3 µs
aguard(stream, inter_chunk_timeout=30)	5.6 µs	5.7 µs

The same benchmark on the same machine while it was busy with other work gave 4 to 9 µs. Either way, at 100 tokens per second that is under 0.1% of the time between two chunks. Memory: by default the received chunks are kept so that exceptions can carry them; pass keep_chunks=False for very long streams.

What was tested, and what was not

Tested (343 tests with the optional clients installed, 282 without; ResourceWarning is an error in the suite):

• Every guard path in sync and async form against an in-process fake server with fault injection: truncation, mid-body disconnect, silent stall, stall behind pings, slow first token (before and after response headers), in-band errors.
• Through the real openai (3.50.0) and anthropic (2.8.0) SDK clients, sync and async, including the .stream() helpers and the Responses API; through raw httpx (0.28.1, sync and async), requests (2.34.2) and aiohttp (3.14.5); and through http.client / asyncio streams.
• Cleanup: after a stall the server observes the client hanging up, and no helper thread or task is left.
• Healthy streams pass through with the same objects (or byte-identical lines) in the same order.
• resumable end to end for every continuation strategy against the fake, including seams that fall on whitespace, repeated cuts and exhausted budgets.
• A real llama.cpp engine on CPU (llama-cpp-python 0.3.36 and a native llama-server built from commit 8ca5a8008b), with a proxy that cuts, drops and stalls the stream: see above and docs/observed-behaviour.md.
• Python 3.10, 3.11, 3.12 and 3.13.

Not tested, stated plainly:

• vLLM, SGLang and every hosted API (OpenAI, Anthropic, ...) were only tested against fakes built from their documented formats and source. There was no GPU and no API key. In particular the "vllm", "sglang", "anthropic-prefill" and "instruction" strategies have never run against the real services. (The vLLM request fields were accepted by a real llama-server, which implements them as a compatibility alias.)
• TLS connections. The socket shutdown that interrupts a blocked read is the same call on a TLS socket, but only plaintext loopback connections were exercised.
• HTTP/2. Shutting a socket down ends every stream multiplexed on it.
• Windows and macOS. Development and tests ran on Linux.
• trio / anyio event loops: aguard is asyncio-only.
• Real-world network failures. A proxy closing a connection is a model of them, not a sample.

Related work

Nothing below is a criticism; these projects solve neighbouring problems, and several are a better fit depending on where you sit.

• LiteLLM handles broken streams inside its Router/proxy: before any content it retries or falls back, and with enable_mid_stream_fallback_continuation a chat stream that broke after plain text is continued on a fallback deployment using a prefill (reliability docs). If you already route through LiteLLM, use that. It is tied to the Router and has had gaps on individual routes: #24004 (/v1/messages; fixed Aug 2026), #28216 (MidStreamFallbackError not handled on Router.aresponses; closed as not planned), #25843 (key-level router settings; closed as not planned), #6957 (fallbacks while streaming; closed 2025), and an open request for a configurable time-to-first-token timeout, #50176. surestream is the same idea as a library function around whatever client you already use, without a router.
• Vercel AI SDK (TypeScript) has first-class stream timeouts (timeout: { totalMs, stepMs, chunkMs }). ai-retry adds retry and model fallback for it. resumable-stream (and its Python port on PyPI) solves a different problem: letting a browser reconnect to a stream that the server keeps producing, via Redis. It does not re-issue an upstream generation.
• Python resilience and client libraries. nopanic is a general resilience toolkit whose timeout policy supports per-item and first-item deadlines on streams; it does not look at LLM terminal signals, and it does not resume or hedge streams. lexilux is a full OpenAI-compatible client with its own "continue cut-off responses" helpers. llmfuse covers retries, breakers and failover for non-streaming calls. llm-stream-collector reassembles provider chunks into one message.
• Fault-injecting test doubles. LLMock (PyPI llmock) is a chaos mock server with truncate / disconnect / stall / slow-first-token / corrupt modes, a pytest fixture and a resilience report; it covers more providers than surestream.testing does. aimock (@copilotkit/aimock, Node) is a broader mock platform with mid-stream disconnects. surestream.testing exists because the library needed a dependency-free fake that also emulates continuation; if you only need a mock, look at those first.

A search of PyPI and the web on 2026-11-13 did not turn up a Python library that combines terminal-signal checking, content-aware deadlines and prefill-based resume as a wrapper around existing clients. If one exists, please open an issue so it can be listed here.

FAQ

Why not just set a timeout on the client? Do that too. But a socket read timeout resets on every byte, so keep-alive pings defeat it (see the table above), it cannot tell "slow to start" from "stopped in the middle", and it does nothing about a stream that ends early without an error.

Isn't checking finish_reason myself enough? For truncation, yes, if you remember to do it on every code path and for every provider's notion of "finished". guard is that check, written once, plus the deadlines.

Does it change the chunks? No. guard and hedged yield the objects they receive. resumable drops the repeated preamble of a continuation (role chunk, message_start) and, only when a prefill had to be sent without its trailing whitespace, shortens the text of at most the first chunk or two of the continuation.

Will the helper thread leak? Not for the openai SDK, the anthropic SDK, httpx, requests or http.client: surestream finds the socket, shuts it down, and the thread exits; the tests assert this. If it is handed a source it cannot interrupt, it still raises on time and emits a ResourceWarning. See docs/design.md.

Is a resumed answer the same answer? Only with greedy decoding, a deterministic server and no sampler penalties. Otherwise it is a continuation. See above.

Does it retry requests that failed before streaming started? Not as a feature. The SDKs already retry connection errors and 429/5xx at request time. resumable does re-send the original request if a stream breaks before any content arrived, because that case costs nothing to get right.

Tool calls? JSON mode? guard works with them: tool-call and reasoning deltas count as content, and termination is checked the same way. resumable refuses to continue them mid-answer and tells you so.

Does it work with LangChain / LlamaIndex / my framework? If you can get at an iterator of chunks whose shape surestream recognises, or write a ten-line adapter, yes. No framework integrations are shipped.

Roadmap

• Validate the vLLM, SGLang and Anthropic strategies against the real services (reports and fixes welcome; see CONTRIBUTING.md).
• Publish to PyPI.
• An overall deadline (total_timeout) next to the two per-chunk ones.
• Optional adapters for more native formats (Gemini, Ollama NDJSON, Bedrock event streams), each only once its terminal signals are verified.
• anyio/trio support for aguard.
• Resuming at a tool-call boundary, if a mechanism can be shown to be correct.

Contributing

Issues with a reproduction against surestream.testing.FakeLLMServer are the most useful thing you can send. Pull requests are welcome; please read CONTRIBUTING.md first. The short version: the core stays dependency-free, behaviour needs a source, and every path is tested in sync and async form.

pip install -e ".[dev]"
ruff check . && ruff format --check . && mypy && pytest

License and affiliation

Apache-2.0, see LICENSE.

surestream is an independent project. It is not affiliated with, endorsed by or supported by OpenAI, Anthropic, the vLLM, SGLang or llama.cpp projects, or any other vendor or project named here. Product names are used only to describe compatibility.