HTTP2 Stream ID Conflict Occurs with ConnectionPool During Multi-threaded Concurrent Usage

[chenlcme]

When I submit HTTP2 requests concurrently using multiple threads through ConnectionPool, an error occurs. An analysis of the logs reveals that multiple threads are using the same stream ID, leading to an error returned by the server.

example:

import logging
import threading
import time

from httpcore import ConnectionPool

logging.basicConfig(
    level=logging.DEBUG,
    format='%(asctime)s %(levelname)s [%(threadName)s] %(message)s',
)

with ConnectionPool(http1=False, http2=True) as pool:
    thread_count = 3
    done_threads: list[str] = []

    def post_one():
        try:
            response = pool.request(
                "POST",
                "https://httpbin.org/post",
                headers=[(b"content-type", b"application/json")],
                content=b'{"hello": "world"}',
            )
            assert response.status == 200
        except Exception as e:
            logging.exception(e)
        finally:
            done_threads.append(threading.current_thread().name)

    for i in range(thread_count):
        thread = threading.Thread(target=post_one)
        thread.start()

    while len(done_threads) < thread_count:
        time.sleep(1)

    print(done_threads)

the key logs:

the hold log:
log.txt

I reviewed the source code of _sync.http2.HTTP2Connection and h2.connection.H2Connection. H2Connection states that it is not thread-safe, and I found the following code block has concurrency issues。I think the code block for acquiring a new stream ID needs to be locked.

[TayfurYldz]

The stream-ID race is plausible from current main, but the lock needs to protect a little more than just the call to get_next_available_stream_id().

After acquiring the stream-capacity semaphore, sync HTTP2Connection.handle_request() currently does:

stream_id = self._h2_state.get_next_available_stream_id()
self._events[stream_id] = []

with no lock around that h2 state transition. hyper-h2's connection object is not a thread-safety boundary, and multiple request threads share this one object for HTTP/2 multiplexing.

The important invariant is that allocating a stream ID and registering httpcore's bookkeeping for that stream must be one serialized operation. Locking only the getter but leaving the associated h2 mutation/bookkeeping able to interleave makes the fix unnecessarily fragile.

httpcore already has thread locks for shared connection state, but I would use a dedicated short critical section for stream allocation rather than serializing the entire request. HTTP/2 still needs concurrent streams; only mutation of the shared h2 connection state has to be coordinated.

Also, this should be tested with a deterministic barrier rather than a stress-only test: arrange for two threads to reach stream allocation together, then assert distinct stream IDs and successful bookkeeping. The async implementation has cooperative task concurrency rather than OS-thread concurrency, so the exact locking primitive differs, but the broader audit should include every place the shared H2Connection is mutated outside the existing read/write locks.