So I have this script:
#!/usr/bin/ruby
require 'net/https'
require 'open-uri'
puts "HTTPS Client for Ruby!"
puts "Enter the URL"
site = gets.chomp
url = URI.parse(site)
http = Net::HTTP.new(url.host,url.port)
http.use_ssl = true
http.verify_mode = OpenSSL::SSL::VERIFY_PEER
http.cert_store = OpenSSL::X509::Store.new
http.cert_store.set_default_paths
http.cert_store.add_file('/home/user/sec/certs/cacert.pem')
page = Net::HTTP.get(url)
puts page
It works fine. It's able to grab the html of the homepage of pretty much any http or https website. However, I have an HTTPS enabled webserver set up in a virtual machine which it doesn't work with. Before I enabled SSL on the webserver this script grabbed the html just fine. So my question is, why do I receive this error:
/usr/lib/ruby/2.1.0/net/http.rb:920:in `connect': SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (OpenSSL::SSL::SSLError)
from /usr/lib/ruby/2.1.0/net/http.rb:920:in `block in connect'
from /usr/lib/ruby/2.1.0/timeout.rb:67:in `timeout'
from /usr/lib/ruby/2.1.0/net/http.rb:920:in `connect'
from /usr/lib/ruby/2.1.0/net/http.rb:863:in `do_start'
from /usr/lib/ruby/2.1.0/net/http.rb:852:in `start'
from /usr/lib/ruby/2.1.0/net/http.rb:583:in `start'
from /usr/lib/ruby/2.1.0/net/http.rb:478:in `get_response'
from /usr/lib/ruby/2.1.0/net/http.rb:455:in `get'
from https_client.rb:20:in `<main>'
When running the script trying to grab the html of my web server? The path that I've specified has an actual certificate there.
You get that error when the SSL certificate is self signed/not from a verified ssl provider.So assuming the website you are pulling from, is your own/you trust it and hasn't a verified certificate, change
http.verify_mode = OpenSSL::SSL::VERIFY_PEER
to
http.verify_mode = OpenSSL::SSL::VERIFY_NONE
same situation
Related
I have a script to scrape data with Mechanize, but I can't authenticate properly on some intranet sites because of NTLM authentication.
This is the code:
require 'mechanize'
url = 'http://intranet/somesite.asp'
agent = Mechanize.new
agent.auth(url, 'my_login', 'my_password')
agent.get(url) do |page|
puts page.title
puts page.body
end
This is the error returned:
/home/igallina/.rvm/gems/ruby-2.2.2/gems/mechanize-2.7.3/lib/mechanize/http/agent.rb:753:in `response_authenticate': 401 => Net::HTTPUnauthorized for http://sistemasnet/srd/Consultas/ConsultaGeral/TelaListagem.asp -- NTLM authentication failed -- available realms: (Mechanize::UnauthorizedError)
from /home/igallina/.rvm/gems/ruby-2.2.2/gems/mechanize-2.7.3/lib/mechanize/http/agent.rb:302:in `fetch'
from /home/igallina/.rvm/gems/ruby-2.2.2/gems/mechanize-2.7.3/lib/mechanize/http/agent.rb:788:in `response_authenticate'
from /home/igallina/.rvm/gems/ruby-2.2.2/gems/mechanize-2.7.3/lib/mechanize/http/agent.rb:302:in `fetch'
from /home/igallina/.rvm/gems/ruby-2.2.2/gems/mechanize-2.7.3/lib/mechanize/http/agent.rb:788:in `response_authenticate'
from /home/igallina/.rvm/gems/ruby-2.2.2/gems/mechanize-2.7.3/lib/mechanize/http/agent.rb:302:in `fetch'
from /home/igallina/.rvm/gems/ruby-2.2.2/gems/mechanize-2.7.3/lib/mechanize.rb:440:in `get'
from mechanize_scrape.rb:6:in `<main>'
I already tried all three methods with no success:
add_auth
auth
basic_auth
and also tried to give more parameters like realm and domain, although I don't really get what realm is.
Just went through mechanize issues, and realized they dropped NTLM support.
When I run the code below:
require "selenium-webdriver"
require 'rubygems'
require 'watir-webdriver'
b = Watir::Browser.new :phantomjs
b.goto 'http://www.google.com'
puts b.title
b.close
the following error is displayed:
/home/jotsarup/.gem/gems/selenium-webdriver-2.41.0/lib/selenium/webdriver/remote/http/common.rb:66:in `create_response': unexpected response, code=503, content-type="text/html" (Selenium::WebDriver::Error::WebDriverError)
<HTML><TITLE>503 Service Unavailable</TITLE>
<H1>503 Service Unavailable</H1>
Failed to connect to server <B>127.0.0.1</B></HTML>
from /home/jotsarup/.gem/gems/selenium-webdriver-2.41.0/lib/selenium/webdriver/remote/http/default.rb:66:in `request'
from /home/jotsarup/.gem/gems/selenium-webdriver-2.41.0/lib/selenium/webdriver/remote/http/common.rb:40:in `call'
from /home/jotsarup/.gem/gems/selenium-webdriver-2.41.0/lib/selenium/webdriver/remote/bridge.rb:634:in `raw_execute'
from /home/jotsarup/.gem/gems/selenium-webdriver-2.41.0/lib/selenium/webdriver/remote/bridge.rb:99:in `create_session'
from /home/jotsarup/.gem/gems/selenium-webdriver-2.41.0/lib/selenium/webdriver/remote/bridge.rb:68:in `initialize'
from /home/jotsarup/.gem/gems/selenium-webdriver-2.41.0/lib/selenium/webdriver/phantomjs/bridge.rb:32:in `initialize'
from /home/jotsarup/.gem/gems/selenium-webdriver-2.41.0/lib/selenium/webdriver/common/driver.rb:45:in `new'
from /home/jotsarup/.gem/gems/selenium-webdriver-2.41.0/lib/selenium/webdriver/common/driver.rb:45:in `for'
from /home/jotsarup/.gem/gems/selenium-webdriver-2.41.0/lib/selenium/webdriver.rb:67:in `for'
from /home/jotsarup/.gem/gems/watir-webdriver-0.6.8/lib/watir-webdriver/browser.rb:46:in `initialize'
from test_phantom.rb:7:in `new'
from test_phantom.rb:7:in `<main>'
phantomjs is not connected. I also tried Firefox and the results are the same.
It looks like you are failing to reach outside of your local machine based on "Failed to connect to server 127.0.0.1" 127.0.0.1 is your loopback address (for your machine) and I have seen this issue arise in the past when there is a firewall up. If you are in a company that requires traffic to be routed through the firewall I would recommend seeing if they see any traffic trying to make it out from your machine. If you're not in a company requiring a firewall then I would recommend dropping the firewall/proxy for testing.
Looks like you are behind the PROXY. Add the following snippet before starting the server:
ENV['HTTP_PROXY'] = ENV['http_proxy'] = nil
b = Watir::Browser.new :phantomjs
I'm trying to use RestClient to retrieve a page that's secured using an SSL client certificate. My code is as follows:
require 'restclient'
p12 = OpenSSL::PKCS12.new(File.read('client.p12'), 'password')
client = RestClient::Resource.new('https://example.com/',
:ssl_client_key => p12.key,
:verify_ssl => OpenSSL::SSL::VERIFY_NONE)
client.get
When I run it, I see the following failure:
1.9.3-p374 :007 > client.get
RestClient::BadRequest: 400 Bad Request
from /home/duncan/.rvm/gems/ruby-1.9.3-p374/gems/rest-client-1.6.7/lib/restclient/abstract_response.rb:48:in `return!'
from /home/duncan/.rvm/gems/ruby-1.9.3-p374/gems/rest-client-1.6.7/lib/restclient/request.rb:230:in `process_result'
from /home/duncan/.rvm/gems/ruby-1.9.3-p374/gems/rest-client-1.6.7/lib/restclient/request.rb:178:in `block in transmit'
from /home/duncan/.rvm/rubies/ruby-1.9.3-p374/lib/ruby/1.9.1/net/http.rb:745:in `start'
from /home/duncan/.rvm/gems/ruby-1.9.3-p374/gems/rest-client-1.6.7/lib/restclient/request.rb:172:in `transmit'
from /home/duncan/.rvm/gems/ruby-1.9.3-p374/gems/rest-client-1.6.7/lib/restclient/request.rb:64:in `execute'
from /home/duncan/.rvm/gems/ruby-1.9.3-p374/gems/rest-client-1.6.7/lib/restclient/request.rb:33:in `execute'
from /home/duncan/.rvm/gems/ruby-1.9.3-p374/gems/rest-client-1.6.7/lib/restclient/resource.rb:51:in `get'
from (irb):7
from /home/duncan/.rvm/rubies/ruby-1.9.3-p374/bin/irb:13:in `<main>'
I'm fairly sure this is a failure to authenticate, as I get the same error in a browser if I don't install the client certificate.
I'm using OpenSSL::SSL::VERIFY_NONE because the server has a self-signed certificate, and I believe this is the correct value to pass to ignore that.
Any suggestions on how to get this working would be greatly appreciated - even a pointer to some detailed documentation, or a suggestion of a different Gem could work. I've not had much luck with either the Gem docs or Google :(
Your HTTPS request is going to need the client certificate as well as the key. Try:
client = RestClient::Resource.new('https://example.com/',
:ssl_client_cert => p12.certificate,
:ssl_client_key => p12.key,
:verify_ssl => OpenSSL::SSL::VERIFY_NONE)
If that doesn't work you can try capturing the handshake packets (e.g. with WireShark) to verify that the API is offering the certificate.
I am adding functionality that scrapes an XML page from a source that requires the use of an HTTPS connection with authentication. I am trying to use Ryan Bates' Railscast #190 solution but I'm running into a 401 Authentication error.
Here is my test Ruby script:
require 'rubygems'
require 'nokogiri'
require 'open-uri'
url = "https://biblesearch.americanbible.org/passages.xml?q[]=john+3:1-5&version=KJV"
doc = Nokogiri::XML(open(url, :http_basic_authentication => ['username' ,'password']))
puts doc.xpath("//text_preview")
Here is the output of the console after I run my script:
/usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/net/http.rb:799:in `connect': SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (OpenSSL::SSL::SSLError)
from /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/net/http.rb:799:in `block in connect'
from /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/timeout.rb:54:in `timeout'
from /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/timeout.rb:99:in `timeout'
from /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/net/http.rb:799:in `connect'
from /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/net/http.rb:755:in `do_start'
from /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/net/http.rb:744:in `start'
from /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/open-uri.rb:306:in `open_http'
from /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/open-uri.rb:775:in `buffer_open'
from /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/open-uri.rb:203:in `block in open_loop'
from /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/open-uri.rb:201:in `catch'
from /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/open-uri.rb:201:in `open_loop'
from /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/open-uri.rb:146:in `open_uri'
from /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/open-uri.rb:677:in `open'
from /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/open-uri.rb:33:in `open'
from scrape.rb:6:in `<main>'
In my research, I saw one post in which it was suggested that in 1.9.3 the following option could be used:
doc = Nokogiri::XML(open(url, :http_basic_authentication => ['username' ,'password'], :ssl_verify_mode => OpenSSL::SSL::VERIFY_NONE))
However, this did not work either. I would appreciate some insight into addressing this challenge.
The given URL will be redirected to /v1/KJV/passages.xml?q[]=john+3%3A1-5 with HTTP status code 302 Found. OpenURI understands the redirection, but automatically deletes authentication header (maybe) for security reason. (*)
If you access "http://biblesearch.americanbible.org/v1/KJV/passages.xml?q[]=john+3%3A1-5" directly, you will get the expected result. :-)
(*) You can find in open-uri.rb:
if redirect
### snip ###
if options.include? :http_basic_authentication
# send authentication only for the URI directly specified.
options = options.dup
options.delete :http_basic_authentication
end
You can do this and it should work too:
open(url, :http_basic_authentication => [user, pass] )
doc = Nokogiri::HTML(open(url, :http_basic_authentication => [user, pass] ))
You can then parse the doc anyway you want.
By passing the http_basic_authentication in the header again in the second request, you will make up for the deleted header in the first request.
hope this works for you.
http://http-basic-authentication-nokogiri.blogspot.com/2014/08/http-basic-authentication-using-nokogiri.html
You say you need to use HTTPS, but you're using the HTTP protocol:
url = "http://biblesearch...."
OpenURI understands both HTTP and HTTPS. If you want to connect using HTTPS, change the protocol in the URL to HTTPS, then make the connection:
url = "https://biblesearch...."
I am attempting to use a remote system for user authentication. This chunk of code gets a response when I run it on MacOSX, but fails on my machine:
def create
uri = URI.parse('https://ourclient.example.com/')
http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = true
http.verify_mode = OpenSSL::SSL::VERIFY_NONE
request = Net::HTTP::Post.new('/login.jsp')
request.set_form_data({'login_name' => params[:login], 'password' => params[:password]})
response = http.request(request)
puts "Response BODY: #{response.body.inspect}"
end
Turning verify off gets rid of a warning on the Macs. On my machine, the http.request raises this exception:
OpenSSL::SSL::SSLError (SSL_connect returned=1 errno=0 state=SSLv2/v3 read server hello A: sslv3 alert illegal parameter):
app/controllers/sessions_controller.rb:16:in `create'
I get the same behavior using IRB without Rails. I did a clean install of Fedora 14 yesterday, installed the required development tools and libraries. I'm using Ruby 1.9.2-p180, and Rails 3.0.4. I thought I might have had my libraries misconfigured (I had Fedora 12 that had been upgraded a few times), but this is now a new install.
The remote system is probably Microsoft's IIS, but I'm not certain of that. Perhaps I can use an older SSL protocol, but my Google-fu can't find the incantation.
I would appreciate any tips on resolving this issue. Thanks,
Chris
how was created the ssl cert on the server? self-signed or ca-signed?
which is the error you receive if you remove the VERIFY_NONE?
use Mechanize gem for it
gem install mechanize
and try
require 'mechanize'
page = Mechanize.new{|a| a.ssl_version, a.verify_mode = 'SSLv3', OpenSSL::SSL::VERIFY_NONE}.get "**YOUR HTTPS LINK HERE**"