Skip to content

Preserve per-call options in HTTP listings - #2258

Merged
martindurant merged 4 commits into
fsspec:masterfrom
adenzhou1350:codex/fix-sync-http-list-options
Oct 14, 2026
Merged

martindurant merged 4 commits into
fsspec:masterfrom
adenzhou1350:codex/fix-sync-http-list-options

Conversation

@adenzhou1350

@adenzhou1350 adenzhou1350 commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Both HTTP implementations merge per-call request options in _ls_real() but send only filesystem defaults. Their empty-listing trailing-slash retry also drops the options.

Use the merged options for each request and preserve caller options on the retry. Constructor defaults remain unchanged. This extends the original synchronous-only fix to the identical async implementation rather than opening a sibling PR.

Four parametrized regressions exercise real aiohttp/Requests sessions and a loopback HTTP server, covering direct listing and trailing-slash retry. They verify received headers, forwarded request options, successful listings, and unchanged defaults on subsequent calls. The async tests use allow_redirects: public async-backed ls() consumes timeout as its synchronous wrapper's wait deadline, which this patch deliberately does not change. Synchronous tests also check request timeout forwarding.

Linux Python 3.12 validation against master 94afb3f6 and exact head 2094e3976dc8:

  • Unchanged production source plus the new tests: 4 failed, 179 passed, 6 skipped.
  • Repaired production source: 183 passed, 6 skipped, covering both HTTP implementations, generic filesystem and core tests.
  • An async initial-request-only fix: direct case passes, trailing-slash retry still fails.
  • Both full runs retain the same existing test_async_other_thread deprecation warning. Optional S3/FTP/ADL tests skip.
  • Repository-pinned Ruff 0.14.3 check/format and diff check pass.

This is native aiohttp/Requests loopback correctness evidence, not Pyodide/XHR, external-service or full CI qualification. AI-assisted implementation/tests, independently checked on unchanged source. Related merged #2229 fixes cached listing shape, not request-option propagation.

@adenzhou1350 adenzhou1350 changed the title Preserve per-call options in synchronous HTTP listings Preserve per-call options in HTTP listings Oct 11, 2026
@martindurant
martindurant merged commit 1997fe6 into fsspec:master Oct 14, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants