Skip to content

fix(reassembly): bypass copyreg's MRO walk for Deferred's protocol<2 pickling - #2435

Merged
JarryShaw merged 1 commit into
mainfrom
fix/deferred-pickle-graalpy-generic-2433
Oct 29, 2026
Merged

JarryShaw merged 1 commit into
mainfrom
fix/deferred-pickle-graalpy-generic-2433

Conversation

@JarryShaw

@JarryShaw JarryShaw commented Oct 29, 2026 •

Copy link
Copy Markdown
Owner

Please follow the guide below

  • Searched for similar pull requests
  • Followed the coding style (make pylint, make mypy, make isort)
  • make test passes, and a test case covers the change
  • Added a changelog entry under docs/source/changelog/ and regenerated CHANGELOG.md

What is the purpose of your pull request?

  • fix — corrects a defect

Description of your pull request and other information

Closes #2433. Deferred still failed to pickle under protocol 0/1 on GraalPy 25.4 even after #2424's __getstate__/__setstate__ fix, with TypeError: typing.Generic() takes no arguments from copyreg._reduce_ex.

Root cause: for protocol < 2, copyreg._reduce_ex walks cls.__mro__ for a base to compute a base-object state from, before __getstate__ is ever consulted. On CPython, Generic is a heap type whose __new__ is just the inherited, differently-bound object.__new__, so the walk passes it and stops at object (state=None). On GraalPy, Generic is itself a non-heap (built-in) type, so the walk's first check (not base.__flags__ & _HEAPTYPE) breaks on Generic directly and calls Generic(self), which raises. __getstate__/__setstate__ never run in that failure, since the crash happens before the walk's result is combined with them.

Fix: Deferred.__reduce_ex__ hard-codes the base=object, state=None outcome the walk always reached on CPython for protocol 0/1, bypassing copyreg._reduce_ex's MRO walk (and Generic with it) entirely. It also checks for, and defers to, a subclass's own __reduce__ override before building that hard-coded tuple — a regression caught in cross-review: without the check, a Deferred subclass overriding __reduce__ had it silently dropped at protocol 0/1 while protocol 2+ (via object.__reduce_ex__, unchanged) kept honouring it. Protocol >= 2 is otherwise untouched. Dropping Generic[_KT] instead was considered and rejected: test_deferred_proto_type_2417_unit.py reads Deferred.__parameters__ at runtime, which only exists because Generic is a real base, not TYPE_CHECKING-only.

Confirmed on real GraalPy 25.4.4 by cross-review: test_deferred_pickle_2423_unit.py goes from 8 failed on main to 16 passed/18 subtests on this fix, and the full tests/foundation/reassembly/ suite passes 173/854 subtests, 0 failed, on GraalPy. A new regression test (test_deferred_pickle_graalpy_2433_unit.py) reproduces the failure mode locally by monkeypatching copyreg._reduce_ex to raise the same TypeError, and a second new test pins the subclass-__reduce__ fix.

CPython: mypy 98/32 (baseline, unchanged), pylint 10.00/10, isort clean, tests/foundation/reassembly (176/890 subtests, including the two new regression tests) and tests/corekit (693/6767 subtests, unaffected by this change) pass, 100% coverage on the changed file.

@JarryShaw
JarryShaw force-pushed the fix/deferred-pickle-graalpy-generic-2433 branch from 780ff6b to 1d529f0 Compare October 29, 2026 19:15
@JarryShaw JarryShaw moved this to In review in PyPCAPKit Oct 29, 2026
@github-actions github-actions Bot added this to the PyPCAPKit v1.5 milestone Oct 29, 2026
@JarryShaw JarryShaw added fix Pull requests that fix a defect (fix: subject prefix) python Pull requests that update Python code review: running A cross-review is in flight against the current head - no verdict yet labels Oct 29, 2026
@JarryShaw

Copy link
Copy Markdown
Owner Author

Review: NEEDS CHANGES at 1d529f02e (Opus, different model from the author). The fix itself works — the reviewer actually got GraalPy 25.4.4 running locally and confirmed it reproduces #2433's exact failure on main and is fixed on this PR. Three things to fix:

  1. Mechanism docstring is wrong about the reason. It says Generic.__new__ satisfies the walk's stopping condition. Traced on real GraalPy: Generic is a non-heap (built-in) type there (__flags__ lacks _HEAPTYPE), so the walk's first check stops on it — __new__ binding is irrelevant. Reword to say that.
  2. Real regression: protocol 0/1 now ignore a subclass's own __reduce__ override (object.__reduce_ex__ checks for one; this PR's protocol<2 branch doesn't), while protocol 2–5 still honour it. Inconsistent. Fix: if type(self).__reduce__ is not object.__reduce__: return self.__reduce__() before the hard-coded tuple.
  3. No regression test that fails without the fix, per standing practice. The reviewer confirmed one is cheap: patch copyreg._reduce_ex to raise, pickle under protocol 0/1 — red on main, green on this PR. Add it, plus one for the __reduce__-override case in point 2.

Also: the PR body's corekit count (866/7649) is actually the combined reassembly+corekit run, not corekit alone (693/6767) — fix the label. And since GraalPy was actually run against this fix (not just reasoned about), replace "GraalPy isn't available here" with the real result before this closes #2433.

Confirmed correct and unaffected: protocol ≥2 is byte-identical to main for every object tested, old pickles made by main's __getstate__-only code still load under this PR and vice versa, keeping Generic[_KT] is justified (test_deferred_proto_type_2417_unit.py genuinely depends on it at runtime), no scope creep (3 files only), and the stated CPython test counts reproduce. mypy/pylint/isort and 3.8–3.13 not re-run (environment gap on the reviewer's side, not a finding).

@JarryShaw JarryShaw added review: needs-changes Cross-review at the current head says changes are required; see the verdict comment and removed review: running A cross-review is in flight against the current head - no verdict yet labels Oct 29, 2026
@github-actions

github-actions Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Coverage: 94.44% (unit tier, Python 3.15, b8ac76505, Unit Tests run success)

Package Statements Missed Branches Partial Cover
pcapkit (top level) 104 4 20 2 95.16%
pcapkit/const 19609 133 2620 120 98.84%
pcapkit/corekit 2605 74 818 37 96.52%
pcapkit/dumpkit 282 3 102 3 98.44%
pcapkit/foundation 3070 24 1072 26 98.79%
pcapkit/interface 138 4 52 3 96.32%
pcapkit/protocols 17593 193 4678 179 98.32%
pcapkit/toolkit 924 36 334 10 95.55%
pcapkit/utilities 502 7 144 5 98.14%
pcapkit/vendor 4449 1854 1020 167 55.11%

Per-file detail: the coverage-html artifact of this run.

@JarryShaw
JarryShaw force-pushed the fix/deferred-pickle-graalpy-generic-2433 branch from 1d529f0 to 641331f Compare October 29, 2026 19:57
@JarryShaw JarryShaw added review: running A cross-review is in flight against the current head - no verdict yet and removed review: needs-changes Cross-review at the current head says changes are required; see the verdict comment labels Oct 29, 2026
@JarryShaw

Copy link
Copy Markdown
Owner Author

Review: NEEDS CHANGES at 641331f47 (Opus, round 2, different model from the author). The code fix itself is confirmed correct and complete — this time including a real run on GraalPy 25.4.4 (oracle/graalpython release, checksum-verified): the new test file + test_deferred_pickle_2423_unit.py go from 10 failed on main to 19 passed/26 subtests on this head, the __reduce__-override guard is called at every protocol 0–5 on both CPython and GraalPy (verified with an independently-written test, not just reading theirs), and the full tests/foundation/reassembly/ suite passes 176/890 subtests with 100% coverage retained on data.py.

Only the new test file's docstrings need fixing — numbers borrowed from the wrong module, one self-contradiction, and a count that shouldn't be there per the owner's no-counted-numbers-in-docs ruling:

  • Module docstring claims "this module's tests go from 8 failed on main to 16 passed/18 subtests" and "173 tests/854 subtests" for the full suite — those are test_deferred_pickle_2423_unit.py's numbers (already correctly credited in the PR body), not this module's (which has 3 tests/8 subtests).
  • Same docstring says both "confirmed directly against real GraalPy 25.4.4" and "GraalPy itself cannot be run from this environment" in the same breath — self-contradictory.
  • test_protocol_0_1_survive_a_copyreg_reduce_ex_failure's docstring claims it was "confirmed on real GraalPy 25.4.4" — true now (this review just did it), but wasn't true when written.

Fix: drop the counts (owner ruling, no counted numbers in docs/prose) and the "cannot be run" sentence; state plainly that it's confirmed on GraalPy 25.4.4 now that it has been.

Non-blocking, noted for awareness only: test_protocol_2_and_above_never_call_copyreg_reduce_ex only exercises protocol 2 despite its name saying "2 and above"; a subclass whose __getstate__ returns {} now gets __setstate__({}) called at protocol 0/1 where copyreg used to skip it — consistent with protocol 2's existing behavior, not a regression.

@JarryShaw JarryShaw added review: needs-changes Cross-review at the current head says changes are required; see the verdict comment and removed review: running A cross-review is in flight against the current head - no verdict yet labels Oct 29, 2026
…pickling

copyreg._reduce_ex, which object.__reduce_ex__ hands protocol 0/1 to, walks
cls.__mro__ for a base to compute state from, before __getstate__ is ever
consulted. On CPython that walk passes through Generic (a heap type whose
__new__ is just the inherited, differently-bound object.__new__) and stops
at object (state=None); on GraalPy 25.4 Generic is itself a non-heap
(built-in) type, so the walk's first check breaks there directly and calls
Generic(self), raising "TypeError: typing.Generic() takes no arguments" --
independent of #2424/#2425's __getstate__/__setstate__ fix, since those only
supply the dict half of a reduction the walk fails to even build.

- Deferred.__reduce_ex__ now hard-codes base=object, state=None for protocol
  0/1 -- the only outcome the walk ever reached for this class -- and builds
  the reduction tuple directly via copyreg._reconstructor, with
  __getstate__()'s own (dict_or_None, slots) pair restored through
  __setstate__ exactly as before. Protocol >= 2 defers to
  object.__reduce_ex__ unchanged.
- It also checks for, and defers to, a subclass's own __reduce__ override
  before building that hard-coded tuple, matching what
  object.__reduce_ex__ itself already does for every protocol. Without the
  check, a Deferred subclass overriding __reduce__ had it silently dropped
  at protocol 0/1 while protocol 2+ (via object.__reduce_ex__ unchanged)
  kept honouring it -- a regression caught in cross-review, now covered by
  a new test alongside the GraalPy-failure regression test.
- Dropping Generic[_KT] instead was considered and rejected:
  test_deferred_proto_type_2417_unit.py reads Deferred.__parameters__ at
  runtime, which only exists because Generic is a real base, not a
  TYPE_CHECKING-only annotation.

No longer invokes copyreg._reduce_ex's MRO walk for protocol<2 at all, so
Generic is no longer part of the route pickling takes. Confirmed on real
GraalPy 25.4.4 by cross-review: test_deferred_pickle_2423_unit.py goes from
8 failed on main to 16 passed/18 subtests on this fix, and the full
tests/foundation/reassembly/ suite passes 173/854 subtests, 0 failed.

Build/test (CPython): mypy 98 errors/32 files (baseline, unchanged), pylint
10.00/10, isort clean; tests/foundation/reassembly (176 tests/890 subtests,
including the two new regression tests) and tests/corekit (693 tests/6767
subtests, unaffected by this change) pass; 100% coverage on the changed
file.

Closes #2433
@JarryShaw
JarryShaw force-pushed the fix/deferred-pickle-graalpy-generic-2433 branch from 641331f to b8ac765 Compare October 29, 2026 20:21
@JarryShaw JarryShaw added review: running A cross-review is in flight against the current head - no verdict yet and removed review: needs-changes Cross-review at the current head says changes are required; see the verdict comment labels Oct 29, 2026
@JarryShaw

Copy link
Copy Markdown
Owner Author

Review: GOOD TO GO at b8ac76505 (Opus, round 3, different model from the author). Confirmed the diff from round 2's head is exactly the claimed docstring-only change in the new test file (parsed code and # comments identical; only two docstrings changed) — the other three files are byte-identical to round 2. All three round-2 findings fixed: no counted numbers remain, the GraalPy-availability self-contradiction is resolved, and the "confirmed on GraalPy" claim is now correctly attributed to a cross-reviewer's actual run rather than stated pre-verification. Full tests/foundation/reassembly/ suite re-run clean: 176/890 subtests, 100% coverage on data.py, confirmed against the PR head (not the shared checkout).

Together with rounds 1–2's findings (both fixed and verified, including a real run on GraalPy 25.4.4), this PR is ready to merge.

@JarryShaw JarryShaw added review: good-to-go Cross-review at the current head says ready; CI state is separate and removed review: running A cross-review is in flight against the current head - no verdict yet labels Oct 29, 2026
@JarryShaw
JarryShaw merged commit ab178aa into main Oct 29, 2026
65 checks passed
@JarryShaw
JarryShaw deleted the fix/deferred-pickle-graalpy-generic-2433 branch October 29, 2026 20:41
@github-project-automation github-project-automation Bot moved this from In review to Done in PyPCAPKit Oct 29, 2026
@JarryShaw JarryShaw removed the review: good-to-go Cross-review at the current head says ready; CI state is separate label Oct 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fix Pull requests that fix a defect (fix: subject prefix) python Pull requests that update Python code

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

fix(reassembly): Deferred(Generic[_KT]) still fails to pickle under protocol 0/1 on GraalPy

1 participant