Conversation
PickleCoder and _MemoizingPickleCoder construct FastPrimitivesCoder with a requires_deterministic kwarg the constructor has never accepted, raising TypeError on every call instead of returning a deterministic coder. Route both through the existing _update_compatible_deterministic_fast_primitives_coder helper, matching FastPrimitivesCoder's own as_deterministic_coder. Fixes apache#39942
|
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment |
|
Run Python_Integration PreCommit 3.14 |
…coder-as-deterministic-coder-typeerror # Conflicts: # CHANGES.md
|
Assigning reviewers: R: @claudevdm for label python. Note: If you would like to opt out of this review, comment Available commands:
The PR bot will only process comments in the main thread (not review comments). |
|
remind me after tests pass |
|
Ok - I'll remind @AmirF194 after tests pass |
|
Reminder, please take a look at this pr: @claudevdm |
|
|
||
| def as_deterministic_coder(self, step_label, error_message=None): | ||
| return FastPrimitivesCoder(self, requires_deterministic=step_label) | ||
| return _update_compatible_deterministic_fast_primitives_coder( |
There was a problem hiding this comment.
Returning DeterministicFastPrimitivesCoder will fail at runtime for arbitrary classes.
Maybe add a warning like
PickleCoder was registered for a key type in '%s', but the runner requires a
deterministic key encoding and pickle is not deterministic. Keys in this step
will be encoded with the deterministic fallback coder instead of pickle. That
coder supports primitives, containers, protobuf messages, frozen dataclasses,
NamedTuples, enums, and classes defining both __getstate__ and __setstate__.
Any other key type will fail at encode time. If your key type is not one of
these, register a deterministic custom Coder for it or add a type hint so the
default coder is used.
Addresses review feedback on apache#39943: the deterministic fallback still fails at encode time for arbitrary classes, so log what changed and what it still cannot encode before the fallback is used.
|
Good point, pushed fa9623c. Added a warning at both call sites (PickleCoder and _MemoizingPickleCoder) using close to your wording, logged before the fallback coder is constructed. Ran the full coders_test.py suite plus a manual check that the warning fires with the right step label; 16 passed. |
…coder-as-deterministic-coder-typeerror # Conflicts: # CHANGES.md
PickleCoder.as_deterministic_coder()and_MemoizingPickleCoder.as_deterministic_coder()both constructFastPrimitivesCoderwith arequires_deterministickeyword argument:FastPrimitivesCoder.__init__(self, fallback_coder=PickleCoder())has never accepted that argument, so both raiseTypeErrorunconditionally. The bug goes back to commit 9be70c9, which changedPickleCoder.as_deterministic_coderfrom the workingDeterministicFastPrimitivesCoder(self, step_label)to this broken call, in the same diff that added arequires_deterministickwarg to a different class (coder_impl.FastPrimitivesCoderImpl, the Cython impl, not thecoders.FastPrimitivesCoderwrapper referenced here)._MemoizingPickleCoderlater copied the already-broken line.The invariant
as_deterministic_coder()is supposed to hold, that it always returns a working deterministic coder rather than raising, is exactly whatFastPrimitivesCoder's ownas_deterministic_coderalready does by delegating to_update_compatible_deterministic_fast_primitives_coder. This PR routes both call sites through that same helper instead.A reachable consumer:
GroupByEncryptedKey.expand()callscoder.as_deterministic_coder(...)insideexcept ValueError, expecting a non-deterministic key to log a warning. When the coder resolves toPickleCoder/_MemoizingPickleCoder, the uncaughtTypeErrorcrashes pipeline construction instead.Verified:
PickleCoder().as_deterministic_coder('x')and_MemoizingPickleCoder().as_deterministic_coder('x')raiseTypeErroronmaster, confirmed live against the installedapache-beampackage (byte-identicalcoders.py) and against this branch.PickleCoderTest.test_as_deterministic_coderfails onmaster, passes on this branch, both runs in the same container.coders_test.py,typecoders_test.py, andfast_coders_test.py(the standard-coder correctness suite, all classes includingFastPrimitivesCoder/DeterministicFastPrimitivesCoder) pass unchanged.ruff checkandyapf --diffare clean on both changed files.coder_impl) fast paths, since this fix is confined to the plain-Pythoncoders.pywrapper layer and does not touchcoder_impl.Fixes #39942