Skip to content

Reject readonly buffers before consuming buffered file data - #2260

Merged
martindurant merged 3 commits into
fsspec:masterfrom
ktz03:fix/readinto-readonly-buffer
Oct 14, 2026
Merged

martindurant merged 3 commits into
fsspec:masterfrom
ktz03:fix/readinto-readonly-buffer

Conversation

@ktz03

@ktz03 ktz03 commented Oct 10, 2026

Copy link
Copy Markdown
Contributor

AbstractBufferedFile.readinto() and readinto1() currently advance the file position before raising TypeError for a read-only target buffer. Retrying with a writable buffer then skips bytes. This affects bytes, read-only memoryviews, and read-only NumPy arrays; standard file objects reject them without moving the position.

Reject read-only buffers before consuming file data, retaining the existing closed-file error through a zero-length read on that error path. Writable-buffer behavior is unchanged. Add regressions for both methods, three cache types, initial/nonzero positions, read-only NumPy arrays, empty buffers, and closed files, plus a changelog entry.

Validation on Windows with Python 3.12.14, NumPy 2.5.3 and aiohttp 3.14.4:

  • The 50 added regressions produce 36 failures and 14 normal passes on the unchanged base; all 50 pass with the fix.
  • pytest fsspec/tests/test_spec.py fsspec/implementations/tests/test_http.py -q: 378 passed, 124 skipped, 1 xfailed. The unchanged base gives 328 passed with the same skips and xfail.
  • A real HTTP-served NumPy .npy checkpoint probe changes from 36 failures / 51 normal controls to 87 passes, including retry bytes, writable typed buffers, and np.load controls. This overlaps formal coverage and is not an additional suite count.
  • Repository-wide Ruff 0.14.3 lint and formatting checks pass.

The Windows test run emits an existing event-loop deprecation and an HTTP-fixture thread teardown warning; the latter was independently reproduced in the unchanged baseline fixture without calling readinto. Docker-dependent backends, downstream projects, other platforms and performance were not tested locally.

@martindurant martindurant left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is .readonly guaranteed to exist?

Comment thread fsspec/spec.py
"""
out = memoryview(b).cast("B")
if out.readonly:
self.read(0)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
self.read(0)

@adenzhou1350

Copy link
Copy Markdown
Contributor

Peer check on 50d7ccf: yes, out is a built-in memoryview after the cast, and readonly is documented in Python 3.10, the minimum supported version.

I independently reran the 50 added regressions on Windows/CPython 3.14: the base plus identical tests gives 36 failures/14 passes; this head gives 50 passes. Removing only self.read(0) leaves 48 passes and changes the two closed-file cases from the existing fsspec ValueError to TypeError.

That line preserves fsspec compatibility, rather than enforcing a universal stdlib rule: closed BytesIO.readinto(b"readonly") reports TypeError on both the Linux 3.12 and Windows 3.14 runtimes I checked. If changing exception precedence is intentional, those two expectations should change too; otherwise keep this validation or an explicit closed-file check. This is focused component verification, not a full-suite/CI claim.

@martindurant
martindurant merged commit 60fcf9d into fsspec:master Oct 14, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants