_io.TextIOWrapper._CHUNK_SIZE ignores underlying buffer size, defaults 8192

[tom93]

Bug report

Bug description:

If I try to read a line from a file containing a very long line using open(path, buffering=n).readline(), it reads in chunks of 8192 regardless of the value of n. I would like it to respect n so that I can reduce the number of syscalls.

I can manually work around this by setting the undocumented attribute _CHUNK_SIZE, but I think this should be done automatically. The number 8192 is hard-coded in _io_TextIOWrapper___init___impl; I propose automatically setting it to the buffer size of the underlying _io.BufferedReader / _io.BufferedWriter / _io.BufferedRandom (if the underlying object is one of those).

docker run --rm -it python:3.14.6 bash
apt-get update && apt-get install -y strace
python3 -c 'print("x"*100000)' > f
# f.readline ignores the buffer size
strace -e trace=read python3 -c 'open("f", buffering=100000).readline()' # calls `read(3, ..., 8192)`
# workaround: set f._CHUNK_SIZE
strace -e trace=read python3 -c 'f = open("f", buffering=100000); f._CHUNK_SIZE = 1000000; f.readline()' # calls `read(3, ..., 1000000)`
# aside: f.buffer.readline respects the buffer size
strace -e trace=read python3 -c 'f = open("f", buffering=100000); f.buffer.readline()' # calls `read(3, ..., 100000)`

CPython versions tested on:

3.14

Operating systems tested on:

Linux

[picnixz]

The docs say:

buffering is an optional integer used to set the buffering policy. Pass 0 to switch buffering off (only allowed in binary mode), 1 to select line buffering (only usable when writing in text mode), and an integer > 1 to indicate the size in bytes of a fixed-size chunk buffer. Note that specifying a buffer size this way applies for binary buffered I/O, but TextIOWrapper (i.e., files opened with mode='r+') would have another buffering.

I am not entirely sure then this is a bug.

cc @serhiy-storchaka @cmaloney

[serhiy-storchaka]

Yes, they are unrelated.

The number of syscalls should be reduced if specify large buffering value. The binary buffered reader will use large buffer, and TextIOWrapper will read from it by smaller chunks without syscalls.

[tom93]

@picnixz Thanks, I didn't see that. I still think the current behaviour is undesirable.

@serhiy-storchaka That sounds logical, but it doesn't match the behaviour that I'm seeing. Specifically, open("f", buffering=100000).readline() makes lots of syscalls with size 8192. Maybe I misdiagnosed and there is a bug elsewhere?

[serhiy-storchaka]

This is worth investigation.

[tom93]

Here is a quick analysis of the call trace:

    _io_TextIOWrapper_readline_impl
    => _textiowrapper_readline
    => textiowrapper_read_chunk(self, size_hint = 0)
    => buffer_callmethod_onearg(self, read1 or read, chunk_size), where chunk_size = max(self->chunk_size, size_hint)
    => buffer.read1, which doesn't use buffering.

I feel the overall result is undesirable, but I'm not sure where the change should happen.