I'm trying to do the following:
page = Mechanize.new.get "https://sis-app.sph.harvard.edu:9030/prod/bwckschd.p_disp_dyn_sched"
But I only get this exception:
OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=SSLv2/v3 read server hello A: sslv3 alert illegal parameter
from /Users/amosng/.rvm/gems/ruby-1.9.3-p194/gems/net-http-persistent-2.7/lib/net/http/persistent/ssl_reuse.rb:70:in `connect'
from /Users/amosng/.rvm/gems/ruby-1.9.3-p194/gems/net-http-persistent-2.7/lib/net/http/persistent/ssl_reuse.rb:70:in `block in connect'
from /Users/amosng/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/timeout.rb:54:in `timeout'
from /Users/amosng/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/timeout.rb:99:in `timeout'
from /Users/amosng/.rvm/gems/ruby-1.9.3-p194/gems/net-http-persistent-2.7/lib/net/http/persistent/ssl_reuse.rb:70:in `connect'
from /Users/amosng/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/net/http.rb:755:in `do_start'
from /Users/amosng/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/net/http.rb:750:in `start'
from /Users/amosng/.rvm/gems/ruby-1.9.3-p194/gems/net-http-persistent-2.7/lib/net/http/persistent.rb:511:in `connection_for'
from /Users/amosng/.rvm/gems/ruby-1.9.3-p194/gems/net-http-persistent-2.7/lib/net/http/persistent.rb:806:in `request'
from /Users/amosng/.rvm/gems/ruby-1.9.3-p194/gems/mechanize-2.5.1/lib/mechanize/http/agent.rb:258:in `fetch'
from /Users/amosng/.rvm/gems/ruby-1.9.3-p194/gems/mechanize-2.5.1/lib/mechanize.rb:407:in `get'
from (irb):549
from /Users/amosng/.rvm/rubies/ruby-1.9.3-p194/bin/irb:16:in `<main>'
How can I get the webpage to load in Mechanize?
Sometimes you need to tell mechanize to use sslv3:
page = Mechanize.new{|a| a.ssl_version, a.verify_mode = 'SSLv3', OpenSSL::SSL::VERIFY_NONE}.get "https://sis-app.sph.harvard.edu:9030/prod/bwckschd.p_disp_dyn_sched"
Notice that I use OpenSSL::SSL::VERIFY_NONE. That means you are theoretically vulnerable to man-in-the-middle attack, but that's not something I generally worry about when I'm scraping a website.
Related
I use the ruby-saml gem and try to load my IDP's metadata but got this error:
> OneLogin::RubySaml::IdpMetadataParser.new.parse_remote('my IDP federation metadata XML URL')
Traceback (most recent call last):
12: from /Users/my_name_here/.rvm/rubies/ruby-2.6.4/bin/irb:23:in `<main>'
11: from /Users/my_name_here/.rvm/rubies/ruby-2.6.4/bin/irb:23:in `load'
10: from /Users/my_name_here/.rvm/rubies/ruby-2.6.4/lib/ruby/gems/2.6.0/gems/irb-1.0.0/exe/irb:11:in `<top (required)>'
9: from (irb):2
8: from /Users/my_name_here/.rvm/gems/ruby-2.6.4/gems/ruby-saml-1.11.0/lib/onelogin/ruby-saml/idp_metadata_parser.rb:53:in `parse_remote'
7: from /Users/my_name_here/.rvm/gems/ruby-2.6.4/gems/ruby-saml-1.11.0/lib/onelogin/ruby-saml/idp_metadata_parser.rb:178:in `get_idp_metadata'
6: from /Users/my_name_here/.rvm/rubies/ruby-2.6.4/lib/ruby/2.6.0/net/http.rb:1470:in `request'
5: from /Users/my_name_here/.rvm/rubies/ruby-2.6.4/lib/ruby/2.6.0/net/http.rb:919:in `start'
4: from /Users/my_name_here/.rvm/rubies/ruby-2.6.4/lib/ruby/2.6.0/net/http.rb:930:in `do_start'
3: from /Users/my_name_here/.rvm/rubies/ruby-2.6.4/lib/ruby/2.6.0/net/http.rb:996:in `connect'
2: from /Users/my_name_here/.rvm/rubies/ruby-2.6.4/lib/ruby/2.6.0/net/protocol.rb:44:in `ssl_socket_connect'
1: from /Users/my_name_here/.rvm/rubies/ruby-2.6.4/lib/ruby/2.6.0/net/protocol.rb:44:in `connect_nonblock'
OpenSSL::SSL::SSLError (SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate))
What does the above error mean?
I tried to see what certificate I'm using and it seems like to be this:
[13] pry(#<OpenSSL::SSL::SSLContext>)> OpenSSL::X509::DEFAULT_CERT_FILE
=> "/usr/local/etc/openssl#1.1/cert.pem"
[14] pry(#<OpenSSL::SSL::SSLContext>)> OpenSSL::X509::DEFAULT_CERT_DIR
=> "/usr/local/etc/openssl#1.1/certs"
Is there something wrong with my certificate?
Note: I have read SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate) but still don't know how to fix my issue. The person use a certificate manually but I assume I don't need to use one because the code should handle the certificate?
The IDP side certificate expires, they fix it and the error in question stopped.
I get the following error in apache log on puppet master:
Any ideas what could be wrong here:
[ E 2018-03-11 07:50:58.9057 10913/T8 age/Cor/Con/CheckoutSession.cpp:285 ]: [Client 1-1] Cannot checkout session because a spawning error occurred. The identifier of the error is cd3585fe. Please see earlier logs for details about the error.
App 10950 stdout:
App 10950 stdout:
[ E 2018-03-11 07:51:03.0922 10913/Te age/Cor/App/Implementation.cpp:304 ]: Could not spawn process for application /usr/share/puppet/rack/puppetmasterd: An error occurred while starting up the preloader.
Error ID: 01cd04d1
Error details saved to: /tmp/passenger-error-sAcbhg.html
Message from application: cannot load such file -- puppet/util/command_line (LoadError)
/usr/local/rvm/rubies/ruby-2.4.1/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
/usr/local/rvm/rubies/ruby-2.4.1/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
config.ru:32:in `block in <main>'
/usr/local/rvm/gems/ruby-2.4.1/gems/rack-2.0.4/lib/rack/builder.rb:55:in `instance_eval'
/usr/local/rvm/gems/ruby-2.4.1/gems/rack-2.0.4/lib/rack/builder.rb:55:in `initialize'
config.ru:1:in `new'
config.ru:1:in `<main>'
/usr/local/rvm/gems/ruby-2.4.1/gems/passenger-5.2.1/src/helper-scripts/rack-preloader.rb:110:in `eval'
/usr/local/rvm/gems/ruby-2.4.1/gems/passenger-5.2.1/src/helper-scripts/rack-preloader.rb:110:in `preload_app'
/usr/local/rvm/gems/ruby-2.4.1/gems/passenger-5.2.1/src/helper-scripts/rack-preloader.rb:156:in `<module:App>'
/usr/local/rvm/gems/ruby-2.4.1/gems/passenger-5.2.1/src/helper-scripts/rack-preloader.rb:30:in `<module:PhusionPassenger>'
/usr/local/rvm/gems/ruby-2.4.1/gems/passenger-5.2.1/src/helper-scripts/rack-preloader.rb:29:in `<main>'
My config:
Centos 6.5,
Puppet 3.8.7,
Apache/2.2.15,
Phusion Passenger 5.2.1
We have UI automation test framework based on cucumber. Recently we have moved from Ruby 1.9.x to 2.2.0 and after that we are facing problem in login to our application via test framework. It says SSL connect reset problem.
Error Trace:
07:01:26 An existing connection was forcibly closed by the remote host. - SSL_connect (Errno::ECONNRESET)
07:01:26 D:/Ruby223/lib/ruby/gems/2.2.0/gems/httpclient-2.7.1/lib/httpclient/ssl_socket.rb:46:in `connect'
07:01:26 D:/Ruby223/lib/ruby/gems/2.2.0/gems/httpclient-2.7.1/lib/httpclient/ssl_socket.rb:46:in `ssl_connect'
07:01:26 D:/Ruby223/lib/ruby/gems/2.2.0/gems/httpclient-2.7.1/lib/httpclient/ssl_socket.rb:24:in `create_socket'
07:01:26 D:/Ruby223/lib/ruby/gems/2.2.0/gems/httpclient-2.7.1/lib/httpclient/session.rb:739:in `block in connect'
07:01:26 D:/Ruby223/lib/ruby/2.2.0/timeout.rb:88:in `block in timeout'
07:01:26 D:/Ruby223/lib/ruby/2.2.0/timeout.rb:98:in `call'
07:01:26 D:/Ruby223/lib/ruby/2.2.0/timeout.rb:98:in `timeout'
07:01:26 D:/Ruby223/lib/ruby/gems/2.2.0/gems/httpclient-2.7.1/lib/httpclient/session.rb:735:in `connect'
07:01:26 D:/Ruby223/lib/ruby/gems/2.2.0/gems/httpclient-2.7.1/lib/httpclient/session.rb:497:in `query'
07:01:26 D:/Ruby223/lib/ruby/gems/2.2.0/gems/httpclient-2.7.1/lib/httpclient/session.rb:170:in `query'
07:01:26 D:/Ruby223/lib/ruby/gems/2.2.0/gems/httpclient-2.7.1/lib/httpclient.rb:1238:in `do_get_block'
07:01:26 D:/Ruby223/lib/ruby/gems/2.2.0/gems/httpclient-2.7.1/lib/httpclient.rb:1021:in `block in do_request'
07:01:26 D:/Ruby223/lib/ruby/gems/2.2.0/gems/httpclient-2.7.1/lib/httpclient.rb:1129:in `protect_keep_alive_disconnected'
07:01:26 D:/Ruby223/lib/ruby/gems/2.2.0/gems/httpclient-2.7.1/lib/httpclient.rb:1016:in `do_request'
07:01:26 D:/Ruby223/lib/ruby/gems/2.2.0/gems/httpclient-2.7.1/lib/httpclient.rb:858:in `request'
07:01:26 D:/Ruby223/lib/ruby/gems/2.2.0/gems/httpclient-2.7.1/lib/httpclient.rb:761:in `post'
<-- our Code for login kick in to call httpclient --->
07:01:26 D:/Ruby223/lib/ruby/gems/2.2.0/gems/cucumber-2.3.2/lib/cucumber/rb_support/rb_language.rb:96:in `load'
07:01:26 D:/Ruby223/lib/ruby/gems/2.2.0/gems/cucumber-2.3.2/lib/cucumber/rb_support/rb_language.rb:96:in `load_code_file'
07:01:26 D:/Ruby223/lib/ruby/gems/2.2.0/gems/cucumber-2.3.2/lib/cucumber/runtime/support_code.rb:142:in `load_file'
07:01:26 D:/Ruby223/lib/ruby/gems/2.2.0/gems/cucumber-2.3.2/lib/cucumber/runtime/support_code.rb:84:in `block in load_files!'
07:01:26 D:/Ruby223/lib/ruby/gems/2.2.0/gems/cucumber-2.3.2/lib/cucumber/runtime/support_code.rb:83:in `each'
07:01:26 D:/Ruby223/lib/ruby/gems/2.2.0/gems/cucumber-2.3.2/lib/cucumber/runtime/support_code.rb:83:in `load_files!'
07:01:26 D:/Ruby223/lib/ruby/gems/2.2.0/gems/cucumber-2.3.2/lib/cucumber/runtime.rb:254:in `load_step_definitions'
07:01:26 D:/Ruby223/lib/ruby/gems/2.2.0/gems/cucumber-2.3.2/lib/cucumber/runtime.rb:62:in `run!'
07:01:26 D:/Ruby223/lib/ruby/gems/2.2.0/gems/cucumber-2.3.2/lib/cucumber/cli/main.rb:32:in `execute!'
07:01:26 D:/Ruby223/lib/ruby/gems/2.2.0/gems/cucumber-2.3.2/bin/cucumber:8:in `<top (required)>'
I had the same issue a while back after moving to ruby 2.0.0
Follow the following steps to solve it
1)Visit the url http://curl.haxx.se/ca/cacert.pem save the contents as a .pem file(Do not save it as a text file.Make sure the extension is .pem)
2)Copy the file to any path in your local eg: C:\ruby200\ca_cert.pem (in my case)
3)Now add an Environment variable with Variable SSL_CERT_FILE and value "C:\ruby200\ca_cert.pem"(without quotes)(path is in my case.Replace with appropriate path)
(Environment variables can be added by navigating to Computer -> Advanced Settings -> Environment Variables)
4)Close all your command prompts and restart them.Things should work fine now
I think your gem is updated.Add these lines in your code to solve the problem
http.verify_mode = OpenSSL::SSL::VERIFY_NONE
before http.use_ssl = true
I hope this might solve the problem.
Dug through a lot of documentation and tried monkey patching. Not sure how to force proper certs, nor where to get them. everything works fine with regular http pages. Using Ruby 1.9.3
Here is stacktrace:
C:/Ruby193/lib/ruby/1.9.1/net/http.rb:800:in `connect': SSL_connect returned=1 e
rrno=0 state=SSLv3 read server certificate B: certificate verify failed (OpenSSL
::SSL::SSLError)
from C:/Ruby193/lib/ruby/1.9.1/net/http.rb:800:in `block in connect'
from C:/Ruby193/lib/ruby/1.9.1/timeout.rb:55:in `timeout'
from C:/Ruby193/lib/ruby/1.9.1/timeout.rb:100:in `timeout'
from C:/Ruby193/lib/ruby/1.9.1/net/http.rb:800:in `connect'
from C:/Ruby193/lib/ruby/1.9.1/net/http.rb:756:in `do_start'
from C:/Ruby193/lib/ruby/1.9.1/net/http.rb:745:in `start'
from C:/Ruby193/lib/ruby/1.9.1/open-uri.rb:306:in `open_http'
from C:/Ruby193/lib/ruby/1.9.1/open-uri.rb:776:in `buffer_open'
from C:/Ruby193/lib/ruby/1.9.1/open-uri.rb:203:in `block in open_loop'
from C:/Ruby193/lib/ruby/1.9.1/open-uri.rb:201:in `catch'
from C:/Ruby193/lib/ruby/1.9.1/open-uri.rb:201:in `open_loop'
from C:/Ruby193/lib/ruby/1.9.1/open-uri.rb:146:in `open_uri'
from C:/Ruby193/lib/ruby/1.9.1/open-uri.rb:678:in `open'
from C:/Ruby193/lib/ruby/1.9.1/open-uri.rb:33:in `open'
When trying to use the open-uri call open you can pass an option to not verify the SSL.
page = Nokogiri::HTML(open(url,{ssl_verify_mode: OpenSSL::SSL::VERIFY_NONE}))
I am using GMail gem for read and send mail functions, but it throws the following error:
Code:
require "gmail"
gmail_instance = Gmail.new(email,password)
Error:
SSL_connect SYSCALL returned=5 errno=0 state=SSLv2/v3 read server hello A
/Users/user/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/imap.rb:1454:in `connect'
/Users/user/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/imap.rb:1454:in `start_tls_session'
/Users/user/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/imap.rb:1047:in `initialize'
/Users/user/.rvm/gems/ruby-2.1.2/gems/gmail-0.4.0/lib/gmail/client/base.rb:26:in `new'
/Users/user/.rvm/gems/ruby-2.1.2/gems/gmail-0.4.0/lib/gmail/client/base.rb:26:in `connect'
/Users/user/.rvm/gems/ruby-2.1.2/gems/gmail-0.4.0/lib/gmail.rb:49:in `new'