Skip to content

[urllib.request] Make Request generic over the type of data - #16608

Open
tqa236 wants to merge 1 commit into
python:mainfrom
tqa236:urllib-request-generic-data
Open

tqa236 wants to merge 1 commit into
python:mainfrom
tqa236:urllib-request-generic-data

Conversation

@tqa236

@tqa236 tqa236 commented Oct 24, 2026

Copy link
Copy Markdown

Closes #11470

Make urllib.request.Request generic over the type of data, so Request(url, data=b"...").data is bytes instead of the full _DataType union, as suggested in the linked issue.

There is a trade-off with this implementation though.

A plain Request annotation, as used by urlopen(), now means Request[Any]. That keeps Request[bytes] accepted everywhere, but .data is unchecked on code annotated with a plain Request. Before, it was the union.

from urllib.request import BaseHandler, Request, build_opener


class LogBodyHandler(BaseHandler):
    # Pre-processor called for every HTTP request before it is sent.
    def http_request(self, request: Request) -> Request:
        print("sending:", request.data.decode())  # no type error because assuming the body is bytes, but will crash at runtime: AttributeError: 'list' object has no attribute 'decode'
        return request


opener = build_opener(LogBodyHandler())

# Streaming the body as an iterable of chunks is allowed by urllib.
opener.open(Request("http://localhost:1/upload", data=[b"chunk1", b"chunk2"]))  

The alternative I considered but didn't choose is the covariant option, which will also slip through a runtime crash.

_DataT_co = TypeVar("_DataT_co", bound=_DataType, default=_DataType, covariant=True)
req = Request("http://x", data=b"body")   # Request[bytes]
req.data = [b"a", b"b"]                    # no errors from type checkers
req.data.decode()                          # no errors from type checkers; crashes at runtime

I did a small web scrapping to see which option is better and chose the current implementation because the scenario it slips through seem to be rarer (small samples though).

AI Disclosure: I used Claude Opus 5.5 to help resolve this issue.

Closes python#11470

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

According to mypy_primer, this change has no effect on the checked open source code. 🤖🎉

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

urllib.request.Request.data type erroring when using decode

1 participant