Response.json try to parse even empty body

[somespecialone]

If response has application/json content type and empty body wreq attempts to parse it to json anyway.
So this:

r = await Client().get("...")  # empty body
await r.json()

Becomes this:
exceptions.DecodingError: is_decode error: wreq::Error { kind: Decode, source: Error("EOF while parsing a value", line: 1, column: 0) }

Is it intentional?

[0x676e67]

This is the error thrown by the upstream library, and it should behave in a consistent manner with reqwest.

https://github.com/seanmonstar/reqwest/blob/1f72916f5cdc30f6cb6c63038c89063795294d50/src/async_impl/response.rs#L270
https://github.com/0x676e67/wreq/blob/9b0fbd925bd64d6901f31018aa4bd4c9143dd13a/src/client/response.rs#L277

Give me an example that can be replicated. Maybe there is room for optimization in that case.

[0x676e67]

In Python, the response body is actually automatically cached. This means that for reading operations other than the streaming body, you can repeat the reading process. If JSON fails, you can still try bytes or text.

Note: The upstream library doesn't verify the application/json header; it expects the user to assume it's there.

[0x676e67]

import asyncio
import sys
import threading
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from pathlib import Path

EXAMPLES_DIR = Path(__file__).resolve().parent
sys.path = [entry for entry in sys.path if Path(entry or ".").resolve() != EXAMPLES_DIR]

import aiohttp
import httpx
import requests
import urllib3
import wreq


class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        routes = {
            "/204": (204, b"", None),
            "/empty": (200, b"", "application/json"),
            "/space": (200, b"   \n\t", "application/json"),
            "/null": (200, b"null", "application/json"),
            "/json": (200, b'{"ok": true}', "application/json"),
        }

        status, body, content_type = routes.get(self.path, (404, b"not found", "text/plain"))
        self.send_response(status)
        if content_type is not None:
            self.send_header("Content-Type", content_type)
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        if body:
            self.wfile.write(body)

    def log_message(self, format, *args):
        return


def format_result(result):
    if isinstance(result, Exception):
        return f"{type(result).__name__}: {result}"
    return repr(result)


def run_requests(url):
    try:
        return requests.get(url, timeout=5).json()
    except Exception as exc:
        return exc


def run_httpx(url):
    try:
        return httpx.get(url, timeout=5).json()
    except Exception as exc:
        return exc


def run_urllib3(http, url):
    try:
        return http.request("GET", url, timeout=5.0).json()
    except Exception as exc:
        return exc


async def run_aiohttp(url):
    try:
        async with aiohttp.ClientSession() as session:
            async with session.get(url) as resp:
                return await resp.json()
    except Exception as exc:
        return exc


async def run_wreq(url):
    try:
        resp = await wreq.get(url)
        async with resp:
            return await resp.json()
    except Exception as exc:
        return exc


async def main():
    server = ThreadingHTTPServer(("127.0.0.1", 0), Handler)
    thread = threading.Thread(target=server.serve_forever, daemon=True)
    thread.start()

    base_url = f"http://127.0.0.1:{server.server_port}"
    paths = ["/204", "/empty", "/space", "/null", "/json"]
    http = urllib3.PoolManager()

    try:
        for path in paths:
            url = f"{base_url}{path}"
            print(f"\n=== {path} ===")
            print("requests:", format_result(run_requests(url)))
            print("httpx:", format_result(run_httpx(url)))
            print("urllib3:", format_result(run_urllib3(http, url)))
            print("aiohttp:", format_result(await run_aiohttp(url)))
            print("wreq:", format_result(await run_wreq(url)))
    finally:
        http.clear()
        server.shutdown()
        server.server_close()
        thread.join(timeout=1)


if __name__ == "__main__":
    asyncio.run(main())

# Results

=== /204 ===
requests: JSONDecodeError: Expecting value: line 1 column 1 (char 0)
httpx: JSONDecodeError: Expecting value: line 1 column 1 (char 0)
urllib3: JSONDecodeError: Expecting value: line 1 column 1 (char 0)
aiohttp: ContentTypeError: 204, message='Attempt to decode JSON with unexpected mimetype: ', url='http://127.0.0.1:63698/204'
wreq: DecodingError: is_decode error: wreq::Error { kind: Decode, source: Error("EOF while parsing a value", line: 1, column: 0) }

=== /empty ===
requests: JSONDecodeError: Expecting value: line 1 column 1 (char 0)
httpx: JSONDecodeError: Expecting value: line 1 column 1 (char 0)
urllib3: JSONDecodeError: Expecting value: line 1 column 1 (char 0)
aiohttp: None
wreq: DecodingError: is_decode error: wreq::Error { kind: Decode, source: Error("EOF while parsing a value", line: 1, column: 0) }

=== /space ===
requests: JSONDecodeError: Expecting value: line 2 column 2 (char 5)
httpx: JSONDecodeError: Expecting value: line 2 column 2 (char 5)
urllib3: JSONDecodeError: Expecting value: line 2 column 2 (char 5)
aiohttp: None
wreq: DecodingError: is_decode error: wreq::Error { kind: Decode, source: Error("EOF while parsing a value", line: 2, column: 1) }

=== /null ===
requests: None
httpx: None
urllib3: None
aiohttp: None
wreq: None

=== /json ===
requests: {'ok': True}
httpx: {'ok': True}
urllib3: {'ok': True}
aiohttp: {'ok': True}
wreq: {'ok': True}

[0x676e67]

It seems that only aiohttp likes to return None when parsing an empty body results in an error.

[somespecialone]

Wow, thanks for comprehensive answer.

In Python, the response body is actually automatically cached.

And I already use it.
As there are any convention regardless http client verifying empty body before passing to a decoder, I guess it is just right to stay with current behavior.

Maybe there is room for optimization in that case.

I don't think there is any, it is up to user to verify if response is malformed (json content type and empty body).

[nlozkina19-crypto]

Hey everyone! I just created a high-speed asynchronous compute core in C++ and Python. Check it out, running tests now! The GitHub view counter is live: https://github.com/nlozkina19-crypto/vector-zero-compute