Skip to content

Support read_block with no delimiter and length=None - #2262

Merged
martindurant merged 5 commits into
fsspec:masterfrom
adenzhou1350:codex/read-block-none-no-delimiter
Oct 14, 2026
Merged

martindurant merged 5 commits into
fsspec:masterfrom
adenzhou1350:codex/read-block-none-no-delimiter

Conversation

@adenzhou1350

Copy link
Copy Markdown
Contributor

fsspec.utils.read_block documents and annotates length=None as reading through EOF. That works with a delimiter, but without one the function reaches assert length is not None:

from io import BytesIO
from fsspec.utils import read_block

read_block(BytesIO(b"abc"), 1, None)  # currently raises AssertionError; expected b"bc"

Use an unbounded read() for None, retaining read(length) for finite lengths. Add real-file and BytesIO regressions for start/interior/EOF/past-EOF offsets, with zero and finite-length controls, and clarify the docstring type. Existing delimiter handling is unchanged.

Validation against master 94afb3f6, Python 3.12:

  • Unmodified Windows and Linux: 8 failures / 16 passing controls.
  • Fixed Windows utility module: 157 passed.
  • Fixed Linux utility, filesystem-spec, local and memory modules: 773 passed, 28 skipped, 1 existing xfail, 1 existing multithreaded-fork warning.
  • Repository-pinned Ruff 0.14.3 check/format and diff check pass. Linux tested source bytes match the committed Git blobs.

Distinct from #2256 (delimiter + split_before) and #2261 (filesystem-level negative lengths). AI-assisted and independently reproduced. These are scoped native tests, not cloud-service or full-CI qualification.

Signed-off-by: Xucheng Zhou <aden1350@outlook.com>
Comment thread fsspec/utils.py
f.seek(offset)

# TODO: allow length to be None and read to the end of the file?
assert length is not None

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-test code should not contain asserts anyway (they can be stripped in bytecode) ...

Comment thread fsspec/utils.py Outdated
assert length is not None
b = f.read(length)
return b
return f.read() if length is None else f.read(length)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't this just f.read(length), since it accepts None and does the right thing?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, simplified to f.read(length) in 75a5878. Also resolved the changelog conflict with current master. Rechecked the final head: Windows utility/core tests 262 passed; Linux utility/spec/local/memory/core tests 932 passed (existing skips/xfail retained). Thanks!

@martindurant
martindurant merged commit faecd8d into fsspec:master Oct 14, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants