This Ruby Gem is meant to scrape and parse results from Google, Bing, Baidu, Yandex, Yahoo, eBay, and more using SerpApi.
This library is deprecated in favor of the serpapi-ruby, which offers:
- 3x faster performance
- Better error handling
- Feature persistent connections
- Asynchronous search capabilities
The following services are provided:
SerpApi.com provides a script builder to get you started quickly.
Modern Ruby must already be installed:
$ gem install google_search_resultsTested Ruby versions:
- 2.5
- 3.0
- 3.1
- 3.2
See: GitHub Actions.
require 'google_search_results'
search = GoogleSearch.new(q: "coffee", serp_api_key: "secret_api_key")
hash_results = search.get_hashThis example runs a search for "coffee" using your secret api key.
The SerpApi.com service (backend)
- searches on Google using the search: q = "coffee"
- parses the messy HTML responses
- returns a standardized JSON response The class GoogleSearch
- Formats the request to the SerpApi.com server
- Executes a GET HTTP request
- Parses JSON into a Ruby Hash using the JSON standard library provided by Ruby Et voila..
Alternatively, you can search:
- Bing using BingSearch class
- Baidu using BaiduSearch class
- Yahoo using YahooSearch class
- Yandex using YandexSearch class
- eBay using EbaySearch class
- Home Depot using HomeDepotSearch class
- YouTube using YoutubeSearch class
See the playground to generate your code
- Google Search Results in Ruby
- Summary
- Error management
- Change log
- Roadmap
- Conclusion
- Contributing
The api_key can be set globally using a singleton pattern.
GoogleSearch.api_key = "secret_api_key"
search = GoogleSearch.new(q: "coffee")Or api_key can be provided for each search.
search = GoogleSearch.new(q: "coffee", api_key: "secret_api_key")To get the key, simply copy/paste it from serpapi.com/dashboard.
search_params = {
q: "search",
google_domain: "Google Domain",
location: "Location Requested",
device: "desktop|mobile|tablet",
hl: "Google UI Language",
gl: "Google Country",
safe: "Safe Search Flag",
num: "Number of Results",
start: "Pagination Offset",
api_key: "private key", # copy paste from https://serpapi.com/dashboard
tbm: "nws|isch|shop",
tbs: "custom search criteria",
async: true|false # allow async
}
# define the search
search = GoogleSearch.new(search_params)
# override an existing parameter
search.params[:location] = "Portland,Oregon,United States"
# search format returns raw HTML
html_results = search.get_html
# search format returns a Hash
hash_results = search.get_hash
# search in raw JSON format
json_results = search.get_json(the full documentation)[https://serpapi.com/search-api].
More search API are documented on SerpApi.com.
You will find more hands-on examples below.
We love true open source, continuous integration, and Test-Driven Development (TDD). We are using RSpec to test our infrastructure around the clock to achieve the best QoS (Quality of Service).
The directory test/ includes specification/examples.
Set your api key.
export API_KEY="your secret key"Install RSpec
gem install rspecTo run the test:
rspec testOr, if you prefer Rake
rake testlocation_list = GoogleSearch.new(q: "Austin", limit: 3).get_location
pp location_listIt prints the first 3 locations matching Austin (Texas, Texas, Rochester)
[
{
id: "585069bdee19ad271e9bc072",
google_id: 200635,
google_parent_id: 21176,
name: "Austin, TX",
canonical_name: "Austin, TX, Texas, United States",
country_code: "US",
target_type: "DMA Region",
reach: 5560000,
gps: [-97.7430608, 30.267153],
keys: ["austin", "tx", "texas", "united", "states"]
},
#...
]This API allows you to retrieve previous searches.
To do so, run a search to save a search_id.
search = GoogleSearch.new(q: "Coffee", location: "Portland")
original_search = search.get_hash
search_id = original_search[:search_metadata][:id]Now let's retrieve the previous search from the archive.
search = GoogleSearch.new
archive_search = search.get_search_archive(search_id)
pp archive_searchIt prints the search from the archive.
search = GoogleSearch.new
pp search.get_accountIt prints your account information.
search = GoogleSearch.new(q: 'cofffe', tbm: "isch")
image_results_list = search.get_hash[:images_results]
image_results_list.each do |image_result|
puts image_result[:original]
endTo download the image: wget #{image_result[:original]}
This code prints all the image links and downloads the image if you uncomment the line with wget (linux/osx tool to download images).
search = GoogleSearch.new({
q: 'coffee', # search search
tbm: "nws", # news
tbs: "qdr:d", # last 24h
num: 10
})
3. times do |offset|
search.params[:start] = offset * 10
news_results_list = search.get_hash[:news_results]
news_results_list.each do |news_result|
puts "#{news_result[:position] + offset * 10} - #{news_result[:title]}"
end
endThis script prints the first 3 pages of the news titles for the last 24 hours.
search = GoogleSearch.new({
q: 'coffee', # search search
tbm: "shop", # shopping
tbs: "tbs=p_ord:rv" # by best review
})
shopping_results_list = search.get_hash[:shopping_results]
shopping_results_list.each do |shopping_result|
puts "#{shopping_result[:position]} - #{shopping_result[:title]}"
endThis script prints all the shopping results ordered by review order, with position.
With SerpApi.com, we can build Google search from anywhere in the world. This code is looking for the best coffee shop per city.
["new york", "paris", "berlin"].each do |city|
# get location from the city name
location = GoogleSearch.new({q: city, limit: 1}).get_location.first[:canonical_name]
# get top result
search = GoogleSearch.new({
q: 'best coffee shop',
location: location,
num: 1, # number of results
start: 0 # offset
})
top_result = search.get_hash[:organic_results].first
puts "top coffee result for #{location} is: #{top_result[:title]}"
endWe do offer two ways to boost your searches thanks to the async parameter.
- Non-blocking - async=true (recommended)
- Blocking - async=false - it's more compute-intensive because the search would need to hold many connections.
company_list = %w(microsoft apple nvidia)
puts "submit batch of asynchronous searches"
search = GoogleSearch.new({async: true})
search_queue = Queue.new
company_list.each do |company|
# set search
search.params[:q] = company
# store request into a search_queue - no-blocker
result = search.get_hash()
if result[:search_metadata][:status] =~ /Cached|Success/
puts "#{company}: search done"
next
end
# Add result to the search queue
search_queue.push(result)
end
puts "wait until all searches are cached or success"
search = GoogleSearch.new
while !search_queue.empty?
result = search_queue.pop
# extract search id
search_id = result[:search_metadata][:id]
# retrieve search from the archive - blocker
search_archived = search.get_search_archive(search_id)
if search_archived[:search_metadata][:status] =~ /Cached|Success/
puts "#{search_archived[:search_parameters][:q]}: search done"
next
end
# add result to the search queue
search_queue.push(result)
end
search_queue.close
puts 'all searches completed'This code shows a simple implementation to run a batch of asynchronous searches.
GoogleSearch.api_key = ""
search = GoogleSearch.new(q: "Coffee", location: "Portland")
pp search.get_hashhttps://serpapi.com/search-api
BingSearch.api_key = ""
search = BingSearch.new(q: "Coffee", location: "Portland")
pp search.get_hashhttps://serpapi.com/bing-search-api
BaiduSearch.api_key = ""
search = BaiduSearch.new(q: "Coffee")
pp search.get_hashhttps://serpapi.com/baidu-search-api
YahooSearch.api_key = ""
search = YahooSearch.new(p: "Coffee")
pp search.get_hashhttps://serpapi.com/yahoo-search-api
YandexSearch.api_key = ""
search = YandexSearch.new(text: "Coffee")
pp search.get_hashhttps://serpapi.com/yandex-search-api
EbaySearch.api_key = ""
search = EbaySearch.new(_nkw: "Coffee")
pp search.get_hashhttps://serpapi.com/ebay-search-api
YoutubeySearch.api_key = ""
search = YoutubeSearch.new(search_query: "Coffee")
pp search.get_hashhttps://serpapi.com/youtube-search-api
HomedepotSearch.api_key = ""
search = HomedepotSearch.new(q: "Coffee")
pp search.get_hashhttps://serpapi.com/home-depot-search-api
WalmartSearch.api_key = ""
search = WalmartSearch.new(query: "Coffee")
pp search.get_hashhttps://serpapi.com/walmart-search-api
DuckduckgoSearch.api_key = ""
search = DuckduckgoSearch.new(query: "Coffee")
pp search.get_hashhttps://serpapi.com/duckduckgo-search-api
search = NaverSearch.new(query: "Coffee", api_key: "secretApiKey")
pp search.get_hashhttps://serpapi.com/duckduckgo-search-api
search = AppleStoreSearch.new(term: "Coffee", , api_key: "secretApiKey")
pp search.get_hashhttps://serpapi.com/duckduckgo-search-api
SerpApiSearch.api_key = ENV['API_KEY']
query = {
p: "Coffee",
engine: "youtube"
}
search = SerpApiSearch.new(query)
hash = search.get_hash
pp hash[:organic_results]see: google-search-results-ruby/test/search_api_spec.rb
This library follows the regular raise an exception when something goes wrong provided by Ruby. Any networking-related exception will be returned as is. Anything related to the client layer will be returned as a SerpApiException. A SerpApiException might be caused by a bug in the library. A networking problem will be caused by either SerpApi.com or your internet.
- 2.2
- Add Apple Store search engine
- Add Naver search engine
- 2.1 - Add more search engines: YouTube, DuckDuckGo, Home Depot, Walmart
- Improve error management and documentation.
- 2.0 - API simplified GoogleSearchResults -> GoogleSearch), fix gem issue with 2.6+ Ruby, out-of-the-box step to verify the package before delivery.
- 1.3.2 - rename variable client to search for naming consistency
- 1.3 - support for all major search engines
- 1.2 - stable version to support Google and a few more search engines
- 1.1 - client connection improvement to allow multithreading and fiber support
- 1.0 - first stable version with Google engine search with Google images
- 2.1 Improve exception / HTTP status handling
SerpApi supports all the major search engines. Google has more advanced support with all the major services available: Images, News, Shopping, and more. To enable a type of search, the field tbm (to be matched) must be set to:
- isch: Google Images API.
- nws: Google News API.
- shop: Google Shopping API.
- any other Google service should work out of the box.
- (no tbm parameter): regular Google search.
The tbs field lets you customize the search even more.
The full documentation is available here.
Contributions are welcome; feel free to submit a pull request!
To run the tests:
export API_KEY="your api key"
rake test