How to scrape Google search using Nokogiri - ruby

Question

I'd like to scrape a few Google search pages for the "Did you mean" spelling checking section.
For example, if I search for "cardiovascular diesese", it will be linked to
https://www.google.com/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q=cardiovascular%20diesese
I want to scrape the "Search instead for cardiovascular diesese" part.
How can I this by using Nokogiri and XPath?

Answer 1

If you can use the non-JavaScript URL, this should work:
require 'nokogiri'
require 'open-uri'
doc = Nokogiri::HTML(open("https://www.google.com/search?q=cardiovascular+diesese"))
doc.xpath("string(//span[#class='spell_orig']/a)") # => "cardiovascular diesese"
If you can render JavaScript and need to use your original example URL, this XPath selector should work once you've loaded the document into Nokogiri (tested with $x in Chrome):
doc.xpath("//a[#class='spell_orig'][boolean(#href)]/text()") # => "cardiovascular diesese"

Answer 2

Since you want to extract only a single result, you can use at_xpath shortcut which under the hood is still doing xpath/css.first. To locate element via Dev Tools you need to go to Elements Tab -> Right Click on the element -> Copy -> Copy Xpath.
To grab text:
doc.at_xpath("//*[#id='fprs']/a[2]/text()") #=> cardiovascular disease
# or you can use at_css which is faster for class names
doc.at_css("a.spell_orig/text()") #=> cardiovascular disease
To grab link:
doc.at_xpath("//*[#id='fprs']/a[2]/#href") #=> /search?hl=en&q=cardiovascular+diesese&nfpr=1&sa=X&ved=2ahUKEwjqhZfu0KbyAhVLRKwKHWbBDNsQvgUoAXoECAEQMg
# or you can use at_css which is faster for class names
doc.at_css("a.spell_orig/#href") #=> /search?hl=en&q=cardiovascular+diesese&nfpr=1&sa=X&ved=2ahUKEwjqhZfu0KbyAhVLRKwKHWbBDNsQvgUoAXoECAEQMg
Code and example in the online IDE:
require 'nokogiri'
require 'httparty'
headers = {
"User-Agent" => "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.19582"
}
params = {
q: "cardiovascular diesese",
hl: "en"
}
response = HTTParty.get("https://www.google.com/search",
query: params,
headers: headers)
doc = Nokogiri::HTML(response.body)
puts doc.at_xpath("//*[#id='fprs']/a[2]/text()"),
"https://www.google.com#{doc.at_xpath("//*[#id='fprs']/a[2]/#href")}"
# or at_css which is faster for class names and produces better XPath than written by hand
puts doc.at_css("a.spell_orig/text()"),
doc.at_css("a.spell_orig/#href")
-------
=begin
cardiovascular diesese
https://www.google.com/search?hl=en&q=cardiovascular+diesese&nfpr=1&sa=X&ved=2ahUKEwjS5Mevr6vyAhWMK80KHXg8AwoQvgUoAXoECAEQMQ
cardiovascular diesese
/search?hl=en&q=cardiovascular+diesese&nfpr=1&sa=X&ved=2ahUKEwjS5Mevr6vyAhWMK80KHXg8AwoQvgUoAXoECAEQMQ
=end
Alternatively, you can use Google Organic Results API from SerpApi. It's a paid API with a free plan that supports different languages.
The difference is that in this case, the figure out part of how to extract some elements from the page is missing. All that needs to be done is to iterate over a structured JSON.
Code to integrate:
require 'google_search_results'
params = {
api_key: ENV["API_KEY"],
engine: "google",
q: "cardiovascular diesese",
hl: "en"
}
search = GoogleSearch.new(params)
hash_results = search.get_hash
search_instead_for = hash_results[:search_information][:spelling_fix]
puts search_instead_for
-------
#=> cardiovascular disease
Disclaimer, I work for SerpApi.