fix(streaming): keep chat moderation results in the accumulated snapshot - #3808
Open
hsusul wants to merge 1 commit into
Open
fix(streaming): keep chat moderation results in the accumulated snapshot#3808hsusul wants to merge 1 commit into
hsusul wants to merge 1 commit into
Conversation
`ChatCompletionStreamState._accumulate_chunk` carried `usage` and `system_fingerprint` forward from each chunk but ignored `moderation`. Moderation results arrive on their own chunk after the content chunks, so `get_final_completion()` and `current_completion_snapshot` reported `moderation=None` for streams that requested moderated completions, while the equivalent non-streaming response kept the field. Carry `moderation` into the snapshot when a chunk provides it. It is copied through `construct_type` because `ChatCompletionChunk.Moderation` and `ChatCompletion.Moderation` are distinct models, and it is only applied when present so a later chunk without moderation cannot clear it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes being requested
Component:
src/openai/lib/streaming/chat/_completions.py—ChatCompletionStreamState._accumulate_chunk.Problem.
ChatCompletionChunkandChatCompletionboth carry amoderationfield, and its docstring says it is "Present on the moderation chunk when moderated completions are requested" — i.e. it arrives on its own chunk, after the content chunks. The streaming accumulator carriesusageandsystem_fingerprintforward from every chunk but never readsmoderation, so the field is silently dropped from the accumulated completion.The result is that
client.chat.completions.stream(...)(andChatCompletionStreamStateused directly) reportsmoderation=Nonefor a stream that actually delivered moderation results, while the equivalent non-streamingclient.chat.completions.create(...)response keeps them. Callers that gate on moderation therefore see "no moderation ran" only when they stream.moderationwas added by87e46c25 feat(api): responses.moderation and chat_completions.moderation, which updated the generated types, resources and tests but did not touch this hand-written accumulator.Repro (no key, no network — the accumulator is fed synthetic chunks):
Before / after
NoneNoneNone(unchanged)The first-chunk case already worked by accident:
_convert_initial_chunk_into_snapshotsplats the whole first chunk into the snapshot, somoderationsurvived only when it happened to be on chunk 1.Root cause.
_accumulate_chunkends withmoderationis the one completion-level field that a later chunk can introduce and that is not in that list.The fix / why it is minimal. Three lines next to the existing carry-forwards:
is not Nonerather than assigned unconditionally likeusage/system_fingerprint, because the moderation chunk is not necessarily the last chunk; an unconditional assignment would clear a moderation payload that arrived earlier.construct_typebecausechat_completion_chunk.Moderationandchat_completion.Moderationare separate generated models with the same shape. Assigning the chunk instance straight onto the snapshot field stores the wrong class and makes the snapshot unserializable (TypeError: 'MockValSer' object cannot be converted to 'SchemaSerializer'fromto_dict()); one of the added tests assertscompletion.to_dict()["moderation"]round-trips.usage/system_fingerprintbehavior, to event emission, ordering, parsing, or any public signature. No other chunk field is touched.Why this is hand-maintained, not generated code.
src/openai/lib/streaming/chat/_completions.pyhas noFile generated from our OpenAPI specheader and lives undersrc/openai/lib/. The generated types (types/chat/chat_completion.py,types/chat/chat_completion_chunk.py) are already correct and are not modified here — the defect is only in the SDK-owned accumulator that has to bridge them.Tests (
tests/lib/chat/test_completions_streaming.py, following the existingtest_stream_obfuscation_stays_on_raw_chunkspattern — synthetic chunks, no HTTP, no key):test_stream_snapshot_keeps_moderation_from_a_later_chunk— the regression; fails onmain.test_stream_snapshot_keeps_moderation_from_the_first_chunk— the path that already worked stays working.test_stream_snapshot_moderation_survives_a_later_chunk_without_moderation— guards against an unconditional overwrite.test_stream_snapshot_has_no_moderation_when_the_stream_has_none— no field invented when the stream has none.Modes covered.
ChatCompletionStreamStateis the single accumulator behind bothChatCompletionStreamandAsyncChatCompletionStream, so sync and async share this code path and this fix; the tests drive the state object directly rather than duplicating an async wrapper that would exercise the same lines. Non-streamingparse/createare unaffected. No stream lifecycle, context-manager, or close behavior is involved.Additional context & links
Validation (macOS, Python 3.10.16 from
uv, branch at3884855):./scripts/lint— passed (RuffAll checks passed!, Pyright0 errors, 0 warnings, 0 informations, mypySuccess: no issues found in 1614 source files,import openaiok)../scripts/format— passed, no changes to the two files in this PR..venv/bin/python -m pytest tests/lib/chat/test_completions_streaming.py -k moderation -q -p no:xdist -o addopts=""— 1 failed, 3 passed with the source change reverted (AssertionError: assert None is not Noneoncompletion.moderation), 4 passed with it applied../scripts/test tests --ignore=tests/test_uv_workflows.py --ignore=tests/test_upload_examples.py— 9366 passed, 144 skipped (Pydantic v2) and 9352 passed, 158 skipped (Pydantic v1).Limits, stated plainly:
tests/test_uv_workflows.pyandtests/test_upload_examples.pywere excluded from the full run because they fail in my checkout for environment reasons, not because of this change:test_uv_workflows.pyshells out togit remote get-url originfrom a temp cwd and getsfatal: not a git repository, andtest_upload_examples.pytimes out generating large fixture files. I confirmed the identical failures with this commit's source change reverted, so they are pre-existing here.*.test.cjs) — it is unrelated to this change.