[vsaraikin]

HTTPFileSystem._exists sends a plain GET and closes the response without reading it. The server starts sending the whole file, and since the body is left unread aiohttp cannot return the connection to the pool, so every call opens a new connection, and a new TLS handshake over HTTPS. Against a local server with a 1 MB file:

               master                              this PR
exists x100    100 connections, 44.8 MB sent      1 connection, 0 MB sent

It now sends Range: bytes=0-0. A server that supports ranges answers 206 with one byte, which is read so the connection is reused. A server that ignores the header answers 200 as before. An empty file answers 416, which still means it exists. It stays a GET rather than a HEAD, since signed URLs are usually only valid for GET.

The test server now answers 416 for a range on an empty file, as real servers do. test_exists_requests_one_byte fails on master

[martindurant]

How sure can we be that servers will accept the Range header; i.e., we know that some servers don't process the header at all, but might they throw an error because it's there?

The server starts sending the whole file, and since the body is left unread

That would be true even for an empty body; but are you sure it isn't cleared when the connection goes out of scope?

[vsaraikin]

How sure can we be that servers will accept the Range header; i.e., we know that some servers don't process the header at all, but might they throw an error because it's there?

The server starts sending the whole file, and since the body is left unread

That would be true even for an empty body; but are you sure it isn't cleared when the connection goes out of scope?

1. master already sends Range when reading files, so this isn't new for servers
2. yes, I tested it. If the body isn't read, aiohttp closes the connection instead of returning it to the pool