I am trying to scrape a Crunchbase page but i got this error:
ryzal~/Desktop/Sites/scraper$ ruby scraper.rb
/Users/Ryzal/.rbenv/versions/2.3.1/lib/ruby/2.3.0/net/http.rb:933:in `connect_nonblock': SSL_connect returned=1 errno=0 state=SSLv3 read server hello A: sslv3 alert handshake failure (OpenSSL::SSL::SSLError)
from /Users/Ryzal/.rbenv/versions/2.3.1/lib/ruby/2.3.0/net/http.rb:933:in `connect'
from /Users/Ryzal/.rbenv/versions/2.3.1/lib/ruby/2.3.0/net/http.rb:863:in `do_start'
from /Users/Ryzal/.rbenv/versions/2.3.1/lib/ruby/2.3.0/net/http.rb:858:in `start'
from /Users/Ryzal/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/net-http-persistent-2.9.4/lib/net/http/persistent.rb:700:in `start'
from /Users/Ryzal/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/net-http-persistent-2.9.4/lib/net/http/persistent.rb:631:in `connection_for'
from /Users/Ryzal/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/net-http-persistent-2.9.4/lib/net/http/persistent.rb:994:in `request'
from /Users/Ryzal/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/mechanize-2.7.5/lib/mechanize/http/agent.rb:274:in `fetch'
from /Users/Ryzal/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/mechanize-2.7.5/lib/mechanize.rb:464:in `get'
from scraper.rb:10:in `<main>'
Below is the code:
require 'nokogiri'
require 'mechanize'
require 'json'
agent = Mechanize.new do |a|
a.ssl_version = :TLSv1
end
page = agent.get "https://www.crunchbase.com/app/search/people/0b543c0d8ea4c95cdf78a48583d501da2a76b26c"
member_links = page.links_with(href: %r{.*/person/\w+})
member_links.map do |link|
puts member_counter
member = link.click
# Get name
name = member.search('#profile_header_heading').text.strip
puts "#{name}"
end
I have tried both of these solutions:
Ruby Mechanize https error
Mechanize getting "Errno::ECONNRESET: Connection reset by peer - SSL_connect"
But still the same error persist.
Please help, thanks!
Try Following
agent = Mechanize.new
agent.agent.http.verify_mode = OpenSSL::SSL::VERIFY_NONE
Related
I am trying to connect to internal SOCKS proxy but its failing because the certificates are self signed... any id how I can change the verify_mode ?
require 'net/ftp'
require 'net/http'
require 'socksify'
Socksify::debug = true
# TCPSocket::verify_mode = OpenSSL::SSL::VERIFY_NONE
TCPSocket::socks_server = '127.0.0.1'
TCPSocket::socks_port = '9090'
Socksify::proxy(nil, nil) do |soc|
# Socksify::proxy('127.0.0.1', '9090') do |soc|
uri = URI('https://api.ipify.org?format=json')
puts Net::HTTP.get(uri)
ftp = Net::FTP.new
ftp.debug_mode = true
ftp.connect(#host, '2021')
ftp.passive = true
ftp.debug_mode = true
ftp.login $user_name, $hog_password
ftp.chdir('..')
puts "get file"
ftp.gettextfile('TSACN.ACH.JCL(COPY)', 'C:/#inbox/copy3.jcl')
ftp.close
end
C:/Ruby31/lib/ruby/gems/3.1.0/gems/net-protocol-0.1.3/lib/net/protocol.rb:46:in `connect_nonblock': SSL_connect returned=1 errno=0 peeraddr=54.91.59.199:443 state=error: certificate verify failed (self signed certificate in certificate chain) (OpenSSL::SSL::SSLError)
from C:/Ruby31/lib/ruby/gems/3.1.0/gems/net-protocol-0.1.3/lib/net/protocol.rb:46:in `ssl_socket_connect'
from C:/Ruby31/lib/ruby/gems/3.1.0/gems/net-http-0.2.2/lib/net/http.rb:1088:in `connect'
from C:/Ruby31/lib/ruby/gems/3.1.0/gems/net-http-0.2.2/lib/net/http.rb:995:in `do_start'
from C:/Ruby31/lib/ruby/gems/3.1.0/gems/net-http-0.2.2/lib/net/http.rb:984:in `start'
from C:/Ruby31/lib/ruby/gems/3.1.0/gems/net-http-0.2.2/lib/net/http.rb:628:in `start'
from C:/Ruby31/lib/ruby/gems/3.1.0/gems/net-http-0.2.2/lib/net/http.rb:504:in `get_response'
from C:/Ruby31/lib/ruby/gems/3.1.0/gems/net-http-0.2.2/lib/net/http.rb:475:in `get'
from ./gp_mf_ftp_sockify.rb:16:in `block in <main>'
from C:/Ruby31/lib/ruby/gems/3.1.0/gems/socksify-1.7.1/lib/socksify.rb:358:in `proxy'
from ./gp_mf_ftp_sockify.rb:13:in `<main>'
Summary: When I connect to Litmus via the gem, I get the following error:
OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=SSLv2/v3 read server hello A: tlsv1 unrecognized name
Accessing the api via curl with the same credentials works OK.
I found this solution for a similar error: SSL_connect SYSCALL returned=5 errno=0 state=SSLv2/v3 read server hello A - Faraday::Error::ConnectionFailed
Which recommends changing the SSL options in oauth:
ssl_options[:version] = :TLSv1
Is there a way to set the ssl options for Litmus? Is there another possible workaround?
Here's the full trace:
2.0.0-p247 :002 > Litmus::Base.new("xxx.litmus.com", "yyy#yyy.com", "zzz", true)
=> #<Litmus::Base:0x007fad4a66dc68>
2.0.0-p247 :003 > Litmus::EmailTest.list
OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=SSLv2/v3 read server hello A: tlsv1 unrecognized name
from /Users/andrei/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http.rb:918:in `connect'
from /Users/andrei/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http.rb:918:in `block in connect'
from /Users/andrei/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/timeout.rb:52:in `timeout'
from /Users/andrei/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http.rb:918:in `connect'
from /Users/andrei/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http.rb:862:in `do_start'
from /Users/andrei/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http.rb:851:in `start'
from /Users/andrei/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http.rb:1367:in `request'
from /Users/andrei/.rvm/gems/ruby-2.0.0-p247/gems/httparty-0.13.1/lib/httparty/request.rb:93:in `perform'
from /Users/andrei/.rvm/gems/ruby-2.0.0-p247/gems/httparty-0.13.1/lib/httparty.rb:521:in `perform_request'
from /Users/andrei/.rvm/gems/ruby-2.0.0-p247/gems/httparty-0.13.1/lib/httparty.rb:457:in `get'
from /Users/andrei/.rvm/gems/ruby-2.0.0-p247/gems/litmus-0.3.0/lib/litmus/test.rb:4:in `list'
from /Users/andrei/.rvm/gems/ruby-2.0.0-p247/gems/litmus-0.3.0/lib/litmus/email_test.rb:4:in `list'
from (irb):3
from /Users/andrei/.rvm/rubies/ruby-2.0.0-p247/bin/irb:16:in `<main>'
I'm thinking you'll need to patch base.rb's initialize method to accept an ssl version parameter, and then add something like this to the method:
def initialize(company, username, password, ssl = false, ssl_version)
protocol = ssl ? 'https' : 'http'
self.class.base_uri "#{protocol}://#{company}.litmus.com"
self.class.basic_auth(username, password)
self.class.ssl_version = ssl_version # <=
end
Then call it like:
Litmus::Base.new("xxx.litmus.com", "yyy#yyy.com", "zzz", true, :TLSv1)
Hope that helps, I did notice you posted an issue of the same topic in their Github repo. Perhaps you can fork & create a pull request.
Could anyone help me by giving small tips mentioning how the below can be written in Mechanize? I am totally new to the Gem Mechanize.
require "rubygems"
require "selenium-webdriver"
driver = Selenium::WebDriver.for :firefox
driver.get "https://www.example.com/"
element = driver.find_element :name => "username"
element.send_keys "#####"
element = driver.find_element :name => "password"
element.send_keys "******"
element.submit
element = driver.find_element(:name, "btnHome")
element.click
element=driver.find_element(:link, "Empdetals")
#print element.attribute(:href)
element.click
element = driver.find_element :name => "search.empdirectory"
element.send_keys "#######"
element = driver.find_element :name => "btnSearch"
element.click
driver.current_url
ERROR When I tried the 'mechanzie` version provided by #Prakash
D:\Ruby script>ruby gmail.rb
C:/Ruby193/lib/ruby/gems/1.9.1/gems/net-http-persistent-2.8/lib/net/http/persist
ent/ssl_reuse.rb:70:in `connect': SSL_connect returned=1 errno=0 state=SSLv3 rea
d server certificate B: certificate verify failed (OpenSSL::SSL::SSLError)
from C:/Ruby193/lib/ruby/gems/1.9.1/gems/net-http-persistent-2.8/lib/net
/http/persistent/ssl_reuse.rb:70:in `block in connect'
from C:/Ruby193/lib/ruby/1.9.1/timeout.rb:54:in `timeout'
from C:/Ruby193/lib/ruby/1.9.1/timeout.rb:99:in `timeout'
from C:/Ruby193/lib/ruby/gems/1.9.1/gems/net-http-persistent-2.8/lib/net
/http/persistent/ssl_reuse.rb:70:in `connect'
from C:/Ruby193/lib/ruby/1.9.1/net/http.rb:755:in `do_start'
from C:/Ruby193/lib/ruby/1.9.1/net/http.rb:750:in `start'
from C:/Ruby193/lib/ruby/gems/1.9.1/gems/net-http-persistent-2.8/lib/net
/http/persistent.rb:628:in `start'
from C:/Ruby193/lib/ruby/gems/1.9.1/gems/net-http-persistent-2.8/lib/net
/http/persistent.rb:570:in `connection_for'
from C:/Ruby193/lib/ruby/gems/1.9.1/gems/net-http-persistent-2.8/lib/net
/http/persistent.rb:926:in `request'
from C:/Ruby193/lib/ruby/gems/1.9.1/gems/mechanize-2.5.1/lib/mechanize/h
ttp/agent.rb:258:in `fetch'
from C:/Ruby193/lib/ruby/gems/1.9.1/gems/mechanize-2.5.1/lib/mechanize.r
b:407:in `get'
from C:/Ruby193/lib/ruby/gems/1.9.1/gems/mechanize-2.5.1/lib/mechanize.r
b:306:in `click'
from gmail.rb:6:in `block in <main>'
from C:/Ruby193/lib/ruby/gems/1.9.1/gems/mechanize-2.5.1/lib/mechanize.r
b:409:in `get'
from gmail.rb:4:in `<main>'
D:\Ruby script>
Yes, mechanize gem can be used to automate any interaction with a website, including logging into a site by submitting the userid/password and clicking on submit button/link, etc.
Unlike selenium-webdriver which makes direct calls to a browser, mechanize itself acts as a browser.
Do checkout the EXAMPLES page on mechanize documentation to learn to how mechanize can be used. The second example - with RubyForge - shows how to login to a site and work with the resultant page.
For a quick overview of how to work with mechanize check out RailsCasts episode on mechanize
Here is an example code for starting at http://www.google.com, clicking on 'Gmail' text, signing into Gmail, and listing the links within the page:
require 'mechanize'
a = Mechanize.new
a.get('http://www.google.com') do |page|
# Click the Gmail link
gmail_login_page = a.click(page.link_with(:text => "Gmail"))
# Submit the login form
gmail_page = gmail_login_page.form_with( :action => 'https://accounts.google.com/ServiceLoginAuth' ) do |f|
f.Email = "<username>#gmail.com"
f.Passwd = "**********"
end.click_button
# List all the links in the personal gmail page
gmail_page.links.each do |link|
text = link.text.strip
next unless text.length > 0
puts text
end
end
Hope it helps in getting started with Mechanize and explore it further!
For the past 2 days, I think I've been through every single available (google'able) post about this SSL-Error in connection with Net::HTTP:
OpenSSL::SSL::SSLError: SSL_connect SYSCALL returned=5 errno=0 state=SSLv2/v3 read server hello A
What I tried doing in the first place, was connecting to a EtherPad server (https://test.titanpad.com), signing in, and downloading a zip-archive via using rubygems & mechanize; however, unfortunately I don't even get up to that point due to said SSL-Error. After trying to debug the issue from within a mechanize object (f.e. by setting cert, ca_file, cert_store, verify_mode, etc. manually in the script), I went one level closer to the actual issue, trying to connect to https://test.titanpad.com by simply using Net::HTTP:
(in this example, I first connected to https://encrypted.google.com to make sure, SSL should work out; the attempt to connect to the EtherPad server starts from line 6)
irb(main):001:0> require 'net/https'
=> true
irb(main):002:0> google = Net::HTTP.new('encrypted.google.com', 443)
=> #<Net::HTTP encrypted.google.com:443 open=false>
irb(main):003:0> google.use_ssl = true
=> true
irb(main):004:0> google.ca_file = '/opt/local/share/curl/curl-ca-bundle.crt' if File.exists?('/opt/local/share/curl/curl-ca-bundle.crt')
=> "/opt/local/share/curl/curl-ca-bundle.crt"
irb(main):005:0> google.request_get('/')
=> #<Net::HTTPOK 200 OK readbody=true>
irb(main):006:0> etherpad = Net::HTTP.new('test.titanpad.com', 443)
=> #<Net::HTTP test.titanpad.com:443 open=false>
irb(main):007:0> etherpad.use_ssl = true
=> true
irb(main):008:0> etherpad.ca_file = '/opt/local/share/curl/curl-ca-bundle.crt' if File.exists?('/opt/local/share/curl/curl-ca-bundle.crt')
=> "/opt/local/share/curl/curl-ca-bundle.crt"
irb(main):009:0> etherpad.request_get('/')
OpenSSL::SSL::SSLError: SSL_connect SYSCALL returned=5 errno=0 state=SSLv2/v3 read server hello A
from /opt/local/lib/ruby1.9/1.9.1/net/http.rb:799:in `connect'
from /opt/local/lib/ruby1.9/1.9.1/net/http.rb:799:in `block in connect'
from /opt/local/lib/ruby1.9/1.9.1/timeout.rb:54:in `timeout'
from /opt/local/lib/ruby1.9/1.9.1/timeout.rb:99:in `timeout'
from /opt/local/lib/ruby1.9/1.9.1/net/http.rb:799:in `connect'
from /opt/local/lib/ruby1.9/1.9.1/net/http.rb:755:in `do_start'
from /opt/local/lib/ruby1.9/1.9.1/net/http.rb:744:in `start'
from /opt/local/lib/ruby1.9/1.9.1/net/http.rb:1284:in `request'
from /opt/local/lib/ruby1.9/1.9.1/net/http.rb:1195:in `request_get'
from (irb):9
from /opt/local/bin/irb:12:in `<main>'
Even when using verify_mode OpenSSL::SSL::VERIFY_NONE, OpenSSL bails out:
irb(main):010:0> etherpad.verify_mode = OpenSSL::SSL::VERIFY_NONE
=> 0
irb(main):011:0> etherpad.request_get('/')
OpenSSL::SSL::SSLError: SSL_connect SYSCALL returned=5 errno=0 state=SSLv2/v3 read server hello A
from /opt/local/lib/ruby1.9/1.9.1/net/http.rb:799:in `connect'
from /opt/local/lib/ruby1.9/1.9.1/net/http.rb:799:in `block in connect'
from /opt/local/lib/ruby1.9/1.9.1/timeout.rb:54:in `timeout'
from /opt/local/lib/ruby1.9/1.9.1/timeout.rb:99:in `timeout'
from /opt/local/lib/ruby1.9/1.9.1/net/http.rb:799:in `connect'
from /opt/local/lib/ruby1.9/1.9.1/net/http.rb:755:in `do_start'
from /opt/local/lib/ruby1.9/1.9.1/net/http.rb:744:in `start'
from /opt/local/lib/ruby1.9/1.9.1/net/http.rb:1284:in `request'
from /opt/local/lib/ruby1.9/1.9.1/net/http.rb:1195:in `request_get'
from (irb):11
from /opt/local/bin/irb:12:in `<main>'
After further playing around with openssl itself, it turns out the real trouble in this case is that the use of SSLv3 has to be forced for the handshake with the Jetty 6.1.20 server behind titanpad.com to work:
irb(main):001:0> require 'net/https'
=> true
irb(main):002:0> etherpad = Net::HTTP.new('test.titanpad.com', 443)
=> #<Net::HTTP test.titanpad.com:443 open=false>
irb(main):003:0> etherpad.use_ssl = true
=> true
irb(main):004:0> etherpad.ssl_version = "SSLv3"
=> "SSLv3"
irb(main):005:0> etherpad.ca_file = '/opt/local/share/curl/curl-ca-bundle.crt' if File.exists?('/opt/local/share/curl/curl-ca-bundle.crt')
=> "/opt/local/share/curl/curl-ca-bundle.crt"
irb(main):006:0> etherpad.request_get('/')
=> #<Net::HTTPFound 302 Found readbody=true>
Now while this obviously works when using Net::HTTP, there is no such option as to set the SSL version to use in Mechanize... and I'd therefore be really glad if someone could point out to me as to how I could enforce SSLv3 via said gem o.O
Thanks again!
System: Mac OSX 10.6.8
ruby 1.9.3p0 (2011-10-30 revision 33570) [x86_64-darwin10]
rubygems installed with mechanize: domain_name (0.5.2), mechanize (2.1.1), net-http-digest_auth (1.2), net-http-persistent (2.4.1), nokogiri (1.5.0), ntlm-http (0.1.1), unf (0.0.4), unf_ext (0.0.4), webrobots (0.0.13)
Has been fixed by porting the ssl_version functionality from Net::HTTP (via net-http-persistent) to Mechanize v. 2.1.2 (see https://github.com/tenderlove/mechanize/commit/4a228899855e0676ab69c2bf548170c8717465d8).
How can I prevent this exception when making a soap call to a server that is using a self signed certificate?
require "rubygems"
gem "httpclient", "2.1.2"
require 'http-access2'
require 'soap/rpc/driver'
client = SOAP::RPC::Driver.new( url, 'http://removed' )
client.options[ 'protocol.http.ssl_config.verify_mode' ] = OpenSSL::SSL::VERIFY_NONE
client.options[ 'protocol.http.basic_auth' ] << [ url, user, pass ]
at depth 0 - 18: self signed certificate
/opt/local/lib/ruby/1.8/soap/streamHandler.rb:200:in `send_post': 415: (SOAP::HTTPStreamError)
from /opt/local/lib/ruby/1.8/soap/streamHandler.rb:109:in `send'
from /opt/local/lib/ruby/1.8/soap/rpc/proxy.rb:170:in `route'
from /opt/local/lib/ruby/1.8/soap/rpc/proxy.rb:141:in `call'
from /opt/local/lib/ruby/1.8/soap/rpc/driver.rb:178:in `call'
Try:
client.options["protocol.http.ssl_config.verify_mode"] = nil