Skip to content

Major differences with Net::HTTP. #211

Closed

Description

@flavio-b

First of all, thank you for making these gems in the Async family. It's one of the most exciting developments in Ruby 3, in my opinion.

I've been working to replace threads, in several IO-heavy tasks, with Async. For example, downloading up to 500 images in parallel, and saving them to a tempfile:

Sync do |parent|
  urls.each do |url|
    parent.async { download_and_save(url) }
  end
end

The threaded version of this took 0.7s with 500 threads (as an experiment), and 2.5s with 12 threads as we cap to limit resource usage in prod.

download_and_save was written with Net::HTTP. I almost went back to threads because the download time increased to 13s in the Async version. Still better than doing it serially (55s), but much slower than threads.

Finally, I updated download_and_save to use Async::HTTP, and it managed in ~0.6s, with the advantage of consuming less memory and avoiding context switching! I was sold.

This was surprising since I always assumed the Fiber scheduler would work about the same with any library that does IO: that it could capture most blocking operations and pass work to other fibers. But, it sounds like either Net::HTTP has some blocking operations that can't be "seen" by the scheduler, or the HTTP2 aspect of Async::HTTP is making up the difference.

Can you share any comment on this? It look me a while to even think of trying Async::HTTP because I didn't know it would make such a difference.

Activity

  1. ioquatix commented on Sep 28, 2025

    @ioquatix
    Member

    I could not reproduce your results.

    Here is a benchmark comparing the 3 scenarios:

          describe Async::HTTP::Client with Thread
            describe Async::HTTP::Client with Thread measure Net::HTTP benchmark/async/http/client.rb:30
              ℹ 26 samples, mean: 216.94ms, standard deviation: 11.02ms, standard error: 2.16ms
          describe Async::HTTP::Client with Async
            describe Async::HTTP::Client with Async measure Net::HTTP benchmark/async/http/client.rb:52
              ℹ 29 samples, mean: 191.85ms, standard deviation: 10.43ms, standard error: 1.94ms
            describe Async::HTTP::Client with Async measure Async::HTTP benchmark/async/http/client.rb:72
              ℹ 18 samples, mean: 168.19ms, standard deviation: 7.03ms, standard error: 1.66ms
    

    Essentially, the performance looks about what I'd expect. Threads are okay, Net::HTTP with Async is better, and Async::HTTP is the best.

    Probably the main reason why Async:HTTP has an edge is persistent connections (and connection pooling/multiplexing).

    Can you figure out what you are doing that deviates from my benchmark?

    https://github.com/socketry/async-http/blob/main/benchmark/async/http/client.rb

    You can run the benchmark using bundle exec sus benchmark/async/http/client.rb.

  2. changed the title [-]Major differences with Net::HTTP[/-] [+]Major differences with `Net::HTTP`.[/+] on Sep 29, 2025
  3. locked and limited conversation to collaborators on Sep 29, 2025
  4. converted this issue into a discussion #212 on Sep 29, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions