I want to get the top trending queries in a particular category on Google Trends. I could download the CSV for that category but that is not a viable solution because I want to branch into each query and find the trending sub-queries for each.
I am unable to capture the contents of the following table, which contains the top 10 trending queries for a topic. Also for some weird reason taking a screenshot using capybara returns a darkened image.
<div id="TOP_QUERIES_0_0table" class="trends-table">
Please run the code on the Ruby console to see it working. Capturing elements/screenshot works fine for facebook.com or google.com but doesn't work for trends.
I am guessing this has to do with the table getting generated dynamically on page load but I'm not sure if that should block capybara from capturing the elements already loaded on the page. Any hints would be very valuable.
require 'capybara/poltergeist'
require 'capybara/dsl'
require 'csv'
class PoltergeistCrawler
include Capybara::DSL
def initialize
Capybara.register_driver :poltergeist_crawler do |app|
Capybara::Poltergeist::Driver.new(app, {
:js_errors => false,
:inspector => false,
phantomjs_logger: open('/dev/null')
})
end
Capybara.default_wait_time = 3
Capybara.run_server = false
Capybara.default_driver = :poltergeist_crawler
page.driver.headers = {
"DNT" => 1,
"User-Agent" => "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:22.0) Gecko/20100101 Firefox/22.0"
}
end
# handy to peek into what the browser is doing right now
def screenshot(name="screenshot")
page.driver.render("public/#{name}.jpg",full: true)
end
# find("path") and all("path") work ok for most cases. Sometimes I need more control, like finding hidden fields
def doc
Nokogiri.parse(page.body)
end
end
crawler = PoltergeistCrawler.new
url = "http://www.google.com/trends/explore#cat=0-45&geo=US&date=today%2012-m&cmpt=q"
crawler.visit url
crawler.screenshot
crawler.find(:xpath, "//div[#id='TOP_QUERIES_0_0table']")
Capybara::ElementNotFound: Unable to find xpath "//div[#id='TOP_QUERIES_0_0table']"
from /Users/karan/.rvm/gems/ruby-1.9.3-p545/gems/capybara-2.4.4/lib/capybara/node/finders.rb:41:in block in find'
from /Users/karan/.rvm/gems/ruby-1.9.3-p545/gems/capybara-2.4.4/lib/capybara/node/base.rb:84:insynchronize'
from /Users/karan/.rvm/gems/ruby-1.9.3-p545/gems/capybara-2.4.4/lib/capybara/node/finders.rb:30:in find'
from /Users/karan/.rvm/gems/ruby-1.9.3-p545/gems/capybara-2.4.4/lib/capybara/session.rb:676:inblock (2 levels) in '
from /Users/karan/.rvm/gems/ruby-1.9.3-p545/gems/capybara-2.4.4/lib/capybara/dsl.rb:51:in block (2 levels) in <module:DSL>'
from (irb):45
from /Users/karan/.rbenv/versions/1.9.3-p484/bin/irb:12:in'
The javascript error was due to the incorrect USER-Agent. Once I changed the User Agent to that of my chrome browser it worked !
"User-Agent" => "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.71 Safari/537.36"
Related
I am new to Mechanize and trying to overcome this probably very obvious answer.
I put together a short script to auth on an external site, then click a link that generates a CSV file dynamically.
I have finally got it to click on the export button, however, it returns an AWS URL.
I'm trying to get the script to download said CSV from this JSON Response (seen below).
Myscript.rb
require 'mechanize'
require 'logger'
require 'rubygems'
require 'nokogiri'
require 'open-uri'
require 'zlib'
USERNAME = "myemail"
PASSWORD = "mysecret"
USER_AGENT = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36"
mechanize = Mechanize.new do |a|
a.user_agent = USER_AGENT
end
form_page = mechanize.get('https://XXXX.XXXXX.com/signin')
form = form_page.form_with(:id =>'login')
form.field_with(:id => 'user_email').value=USERNAME
form.field_with(:id => 'user_password').value=PASSWORD
page = form.click_button
donations = mechanize.get('https://XXXXX.XXXXXX.com/pages/ACCOUNT/statistics')
puts donations.body
donations = mechanize.get('https://xxx.siteimscraping.com/pages/myaccount/statistics')
bs_csv_download = page.link_with(:text => 'Download CSV')
JSON response from website containing link to CSV I need to parse and download via Mechanize and/or nokogiri.
{"message":"Find your report at https://s3.amazonaws.com/reports.XXXXXXX.com/XXXXXXX.csv?X-Amz-Algorithm=AWS4-HMAC-SHA256\u0026X-Amz-Credential=AKIAIKW4BJKQUNOJ6D2A%2F20190228%2Fus-east-1%2Fs3%2Faws4_request\u0026X-Amz-Date=20190228T025844Z\u0026X-Amz-Expires=86400\u0026X-Amz-SignedHeaders=host\u0026X-Amz-Signature=b19b6f1d5120398c850fc03c474889570820d33f5ede5ff3446b7b8ecbaf706e"}
I very much appreciate any help.
You could parse it as JSON and then retrieve a substring from the response (assuming it always responds in the same format):
require 'json'
...
bs_csv_download = page.link_with(:text => 'Download CSV')
json_response = JSON.parse(bs_csv_download)
direct_link = json_response["message"][20..-1]
mechanize.get(direct_link).save('file.csv')
We're getting the 20th character in the "message" value with [20..-1] (-1 means till the end of the string).
In the Feedjira 2.0 announcement blog post, it says that if you want to set the user agent, that should be a configuration option, but it is not clear how to do this. Ideally, I would like to mimic the options previously provided in Feedjira 1.0, including user_agent, if_modified_since, timeout, and ssl_verify_peer.
http://feedjira.com/blog/2014/04/14/thoughts-on-version-two-point-oh.html
With Feedjira 1.0, you could set those options by making the following call (as described here):
feed_parsed = Feedjira::Feed.fetch_and_parse("http://sports.espn.go.com/espn/rss/news", {:if_modified_since => Time.now, :ssl_verify_peer => false, :timeout => 5, :user_agent => "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.94 Safari/537.36"})
The only example I have seen where configuration options are set was from a comment in a github pull request, which is as follows:
Feedjira::Feed.configure do |faraday|
faraday.request :user_agent, app: "MySite", version: APP_VERSION
end
But when I tried something similar, I received the following error:
undefined method `configure' for Feedjira::Feed:Class
It looks like a patch was added to allow a timeout option to be passed to the fetch_and_parse function:
https://github.com/feedjira/feedjira/pull/318/commits/fbdb85b622f72067683508b1d7cab66af6303297#diff-a29beef397e3d8624e10af065da09a14
However, until that is pushed live, a timeout and an open_timeout option can be passed by bypassing Feedjira for the fetching and instead using Faraday (or any library that can fetch HTTP requests, like Net::HTTP). You can also set ssl verify to false, and set the user agent, such as this:
require 'feedjira'
require 'pp'
url = "http://www.espn.com/espnw/rss/?sectionKey=athletes-life"
conn = Faraday.new :ssl => {:verify => false}
response = conn.get do |request|
request.url url
request.options.timeout = 5
request.options.open_timeout = 5
request.headers = {'User-Agent' => "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.94 Safari/537.36"}
end
feed_parsed = Feedjira::Feed.parse response.body
pp feed_parsed.entries.first
I haven't seen a way to check for "if_modified_since", but I will update answer if I do.
I have a lot of errors that I need to catch, so I put them all into two arrays and made a constant to hold them, however, when I run the program I receive the exception:
C:/Users/thomas_j_perkins/bin/ruby/tool/sql_tool/whitewidow/lib/imports/constants_and_requires.rb:62:in `<top (required)>': uninitialized constant RestClient::MaxRedirectsReached (NameError)
from whitewidow.rb:6:in `require_relative'
from whitewidow.rb:6:in `<main>'
Here's how the constants look:
LOADING_ERRORS = [RestClient::ResourceNotFound, RestClient::InternalServerError, RestClient::RequestTimeout,
RestClient::Gone, RestClient::SSLCertificateNotVerified, RestClient::Forbidden,
OpenSSL::SSL::SSLError, Errno::ECONNREFUSED, URI::InvalidURIError, Errno::ECONNRESET,
Timeout::Error, OpenSSL::SSL::SSLError, Zlib::GzipFile::Error, RestClient::MultipleChoices,
RestClient::Unauthorized, SocketError, RestClient::BadRequest, RestClient::ServerBrokeConnection,
RestClient::MaxRedirectsReached]
FATAL_ERRORS = [Mechanize::ResponseCodeError, RestClient::ServiceUnavailable, OpenSSL::SSL::SSLError,
RestClient::BadGateway]
Here's how I'm using them:
begin
# Do some cool stuff
rescue *FATAL_ERRORS => e
puts e
end
--
begin
# Do some more cool stuff
rescue *LOADING_ERRORS => e
puts e
end
Am I doing something wrong to where I will receive a top required error? Just in case you need it here's the entire requiring file that the error is specifying:
# Built in libraries
require 'rubygems'
require 'bundler/setup'
require 'mechanize'
require 'nokogiri'
require 'rest-client'
require 'timeout'
require 'uri'
require 'fileutils'
require 'yaml'
require 'date'
require 'optparse'
require 'tempfile'
require 'socket'
require 'net/http'
# Created libraries
require_relative '../../lib/modules/format'
require_relative '../../lib/misc/credits'
require_relative '../../lib/misc/legal'
require_relative '../../lib/misc/spider'
require_relative '../../lib/modules/copy'
require_relative '../../lib/modules/site_info'
require_relative '../../lib/modules/expansion/string_expan'
# Modules that need to be included
include Format
include Credits
include Legal
include Whitewidow
include Copy
include SiteInfo
# Constants used throughout the program
=begin
USER_AGENTS = { # Temporary fix for user agents until I can refactor the YAML file
1 => 'Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/webcrawler.html) Gecko/2008032620',
2 => 'Mozilla/5.0 (compatible; U; ABrowse 0.6; Syllable) AppleWebKit/420+ (KHTML, like Gecko)',
3 => 'Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.3pre) Gecko/20100403 Lorentz/3.6.3plugin2pre (.NET CLR 4.0.20506)',
4 => 'Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)',
5 => 'igdeSpyder (compatible; igde.ru; +http://igde.ru/doc/tech.html)',
6 => 'larbin_2.6.3 (ltaa_web_crawler#groupes.epfl.ch)',
7 => 'Mozilla/5.0 (Linux; Android 5.0.2; SAMSUNG SM-T550 Build/LRX22G) AppleWebKit/537.36 (KHTML, like Gecko) SamsungBrowser/3.3 Chrome/38.0.2125.102 Safari/537.36',
8 => 'Dalvik/2.1.0 (Linux; U; Android 6.0.1; Nexus Player Build/MMB29T)',
9 => 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:15.0) Gecko/20100101 Firefox/15.0.1',
10 => 'Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)',
}
=end
FORMAT = Format::StringFormat.new
PATH = Dir.pwd
VERSION = Whitewidow.version
SEARCH = File.readlines("#{PATH}/lib/lists/search_query.txt").sample
USER_AGENTS = YAML.load_file("#{PATH}/lib/lists/rand-age.yml")
OPTIONS = {}
USER_AGENT = USER_AGENTS[rand(1..10)]
SKIP = %w(/webcache.googleusercontent.com stackoverflow.com github.com)
LOADING_ERRORS = [RestClient::ResourceNotFound, RestClient::InternalServerError, RestClient::RequestTimeout,
RestClient::Gone, RestClient::SSLCertificateNotVerified, RestClient::Forbidden,
OpenSSL::SSL::SSLError, Errno::ECONNREFUSED, URI::InvalidURIError, Errno::ECONNRESET,
Timeout::Error, OpenSSL::SSL::SSLError, Zlib::GzipFile::Error, RestClient::MultipleChoices,
RestClient::Unauthorized, SocketError, RestClient::BadRequest, RestClient::ServerBrokeConnection,
RestClient::MaxRedirectsReached]
FATAL_ERRORS = [Mechanize::ResponseCodeError, RestClient::ServiceUnavailable, OpenSSL::SSL::SSLError,
RestClient::BadGateway]
I installed mechanize and rest-client
gem install mechanize
gem install rest-client
then I opened an IRB session
require mechanize
require rest-client
then tested you FATAL_ERROR array and was able to raise the error and handle it with your code.
So there is no problem with the way you are using the * splat operator.
The problem is in your LOADING_ERRORS array.
When I tried doing the same thing with your LOADING_ERRORS array, I got the same error message as you.
I cloned the rest-client git repository and searched in the lib/restclient/exceptions.rb file and it seems like there is no RestClient::MaxRedirectsReached defined.
If you remove that exception from your array, the code works.
After further research in the repository, there is a history.md file and it states:
Changes to redirection behavior: (#381, #484)
Remove RestClient::MaxRedirectsReached in favor of the normal
ExceptionWithResponse subclasses. This makes the response accessible on
the exception object as .response, making it possible for callers to tell
what has actually happened when the redirect limit is reached.
When following HTTP redirection, store a list of each previous response on
the response object as .history. This makes it possible to access the
original response headers and body before the redirection was followed.
Follow redirection consistently, regardless of whether the HTTP method was
passed as a symbol or string. Under the hood rest-client now normalizes the
HTTP request method to a lowercase string.
So it seems like that exception has been removed from the rest-client library.
You may want to replace it with RestClient::ExceptionWithResponse
Closed. This question needs debugging details. It is not currently accepting answers.
Edit the question to include desired behavior, a specific problem or error, and the shortest code necessary to reproduce the problem. This will help others answer the question.
Closed 7 years ago.
Improve this question
This code work on some pages, like klix.ba, but cant figure out why it doesn't work for others.
There is no error to explain what went wrong, nothing.
If puts page works, which means I can target the page, and parse it, why I cant get single elements?
require 'nokogiri'
require 'open-uri'
url = 'http://www.olx.ba/'
user_agent = "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.7) Gecko/2009021910 Firefox/3.0.7"
page = Nokogiri::XML(open(url,'User-Agent' => user_agent), nil, "UTF-8")
#puts page - This line work
puts page.xpath('a')
First of all, why are you parsing it as XML?
The following should be correct, considering your page is a HTML website:
page = Nokogiri::HTML(open(url,'User-Agent' => user_agent), nil, "UTF-8")
Furthermore, if you want to strip out all the links (a-tags), this is how:
page.css('a').each do |element|
puts element
end
If you are want to parse content from a web page you need to do this:
require 'nokogiri'
require 'open-uri'
url = 'http://www.olx.ba/'
user_agent = "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.7) Gecko/2009021910 Firefox/3.0.7"
page = Nokogiri::HTML(open(url,'User-Agent' => user_agent), nil, "UTF-8")
#puts page - This line work
puts page.xpath('a')
Here take a look at the Nokogiri documentation
One thing I would suggest is to use a debugger break point in your code (probably after assigning page). Look at the Pry-debugger gem.
So I would do something like this:
require 'nokogiri'
require 'open-uri'
require 'pry' # require the necessary library
url = 'http://www.olx.ba/'
user_agent = "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.7) Gecko/2009021910 Firefox/3.0.7"
page = Nokogiri::HTML(open(url,'User-Agent' => user_agent), nil, "UTF-8")
binding.pry # stop a moment in time in you code (break point)
#puts page - This line work
puts page.xpath('a')
I'd like to scrape a few Google search pages for the "Did you mean" spelling checking section.
For example, if I search for "cardiovascular diesese", it will be linked to
https://www.google.com/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q=cardiovascular%20diesese
I want to scrape the "Search instead for cardiovascular diesese" part.
How can I this by using Nokogiri and XPath?
If you can use the non-JavaScript URL, this should work:
require 'nokogiri'
require 'open-uri'
doc = Nokogiri::HTML(open("https://www.google.com/search?q=cardiovascular+diesese"))
doc.xpath("string(//span[#class='spell_orig']/a)") # => "cardiovascular diesese"
If you can render JavaScript and need to use your original example URL, this XPath selector should work once you've loaded the document into Nokogiri (tested with $x in Chrome):
doc.xpath("//a[#class='spell_orig'][boolean(#href)]/text()") # => "cardiovascular diesese"
Since you want to extract only a single result, you can use at_xpath shortcut which under the hood is still doing xpath/css.first. To locate element via Dev Tools you need to go to Elements Tab -> Right Click on the element -> Copy -> Copy Xpath.
To grab text:
doc.at_xpath("//*[#id='fprs']/a[2]/text()") #=> cardiovascular disease
# or you can use at_css which is faster for class names
doc.at_css("a.spell_orig/text()") #=> cardiovascular disease
To grab link:
doc.at_xpath("//*[#id='fprs']/a[2]/#href") #=> /search?hl=en&q=cardiovascular+diesese&nfpr=1&sa=X&ved=2ahUKEwjqhZfu0KbyAhVLRKwKHWbBDNsQvgUoAXoECAEQMg
# or you can use at_css which is faster for class names
doc.at_css("a.spell_orig/#href") #=> /search?hl=en&q=cardiovascular+diesese&nfpr=1&sa=X&ved=2ahUKEwjqhZfu0KbyAhVLRKwKHWbBDNsQvgUoAXoECAEQMg
Code and example in the online IDE:
require 'nokogiri'
require 'httparty'
headers = {
"User-Agent" => "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.19582"
}
params = {
q: "cardiovascular diesese",
hl: "en"
}
response = HTTParty.get("https://www.google.com/search",
query: params,
headers: headers)
doc = Nokogiri::HTML(response.body)
puts doc.at_xpath("//*[#id='fprs']/a[2]/text()"),
"https://www.google.com#{doc.at_xpath("//*[#id='fprs']/a[2]/#href")}"
# or at_css which is faster for class names and produces better XPath than written by hand
puts doc.at_css("a.spell_orig/text()"),
doc.at_css("a.spell_orig/#href")
-------
=begin
cardiovascular diesese
https://www.google.com/search?hl=en&q=cardiovascular+diesese&nfpr=1&sa=X&ved=2ahUKEwjS5Mevr6vyAhWMK80KHXg8AwoQvgUoAXoECAEQMQ
cardiovascular diesese
/search?hl=en&q=cardiovascular+diesese&nfpr=1&sa=X&ved=2ahUKEwjS5Mevr6vyAhWMK80KHXg8AwoQvgUoAXoECAEQMQ
=end
Alternatively, you can use Google Organic Results API from SerpApi. It's a paid API with a free plan that supports different languages.
The difference is that in this case, the figure out part of how to extract some elements from the page is missing. All that needs to be done is to iterate over a structured JSON.
Code to integrate:
require 'google_search_results'
params = {
api_key: ENV["API_KEY"],
engine: "google",
q: "cardiovascular diesese",
hl: "en"
}
search = GoogleSearch.new(params)
hash_results = search.get_hash
search_instead_for = hash_results[:search_information][:spelling_fix]
puts search_instead_for
-------
#=> cardiovascular disease
Disclaimer, I work for SerpApi.