How can I prevent this exception when making a soap call to a server that is using a self signed certificate?
require "rubygems"
gem "httpclient", "2.1.2"
require 'http-access2'
require 'soap/rpc/driver'
client = SOAP::RPC::Driver.new( url, 'http://removed' )
client.options[ 'protocol.http.ssl_config.verify_mode' ] = OpenSSL::SSL::VERIFY_NONE
client.options[ 'protocol.http.basic_auth' ] << [ url, user, pass ]
at depth 0 - 18: self signed certificate
/opt/local/lib/ruby/1.8/soap/streamHandler.rb:200:in `send_post': 415: (SOAP::HTTPStreamError)
from /opt/local/lib/ruby/1.8/soap/streamHandler.rb:109:in `send'
from /opt/local/lib/ruby/1.8/soap/rpc/proxy.rb:170:in `route'
from /opt/local/lib/ruby/1.8/soap/rpc/proxy.rb:141:in `call'
from /opt/local/lib/ruby/1.8/soap/rpc/driver.rb:178:in `call'
Try:
client.options["protocol.http.ssl_config.verify_mode"] = nil
Related
I am trying to connect to internal SOCKS proxy but its failing because the certificates are self signed... any id how I can change the verify_mode ?
require 'net/ftp'
require 'net/http'
require 'socksify'
Socksify::debug = true
# TCPSocket::verify_mode = OpenSSL::SSL::VERIFY_NONE
TCPSocket::socks_server = '127.0.0.1'
TCPSocket::socks_port = '9090'
Socksify::proxy(nil, nil) do |soc|
# Socksify::proxy('127.0.0.1', '9090') do |soc|
uri = URI('https://api.ipify.org?format=json')
puts Net::HTTP.get(uri)
ftp = Net::FTP.new
ftp.debug_mode = true
ftp.connect(#host, '2021')
ftp.passive = true
ftp.debug_mode = true
ftp.login $user_name, $hog_password
ftp.chdir('..')
puts "get file"
ftp.gettextfile('TSACN.ACH.JCL(COPY)', 'C:/#inbox/copy3.jcl')
ftp.close
end
C:/Ruby31/lib/ruby/gems/3.1.0/gems/net-protocol-0.1.3/lib/net/protocol.rb:46:in `connect_nonblock': SSL_connect returned=1 errno=0 peeraddr=54.91.59.199:443 state=error: certificate verify failed (self signed certificate in certificate chain) (OpenSSL::SSL::SSLError)
from C:/Ruby31/lib/ruby/gems/3.1.0/gems/net-protocol-0.1.3/lib/net/protocol.rb:46:in `ssl_socket_connect'
from C:/Ruby31/lib/ruby/gems/3.1.0/gems/net-http-0.2.2/lib/net/http.rb:1088:in `connect'
from C:/Ruby31/lib/ruby/gems/3.1.0/gems/net-http-0.2.2/lib/net/http.rb:995:in `do_start'
from C:/Ruby31/lib/ruby/gems/3.1.0/gems/net-http-0.2.2/lib/net/http.rb:984:in `start'
from C:/Ruby31/lib/ruby/gems/3.1.0/gems/net-http-0.2.2/lib/net/http.rb:628:in `start'
from C:/Ruby31/lib/ruby/gems/3.1.0/gems/net-http-0.2.2/lib/net/http.rb:504:in `get_response'
from C:/Ruby31/lib/ruby/gems/3.1.0/gems/net-http-0.2.2/lib/net/http.rb:475:in `get'
from ./gp_mf_ftp_sockify.rb:16:in `block in <main>'
from C:/Ruby31/lib/ruby/gems/3.1.0/gems/socksify-1.7.1/lib/socksify.rb:358:in `proxy'
from ./gp_mf_ftp_sockify.rb:13:in `<main>'
The following call is made in our code base:
require 'open-uri'
open(url, :proxy => nil)
However, when the call is made, Open uri uses the http_proxy environment variable to make the call, which is effectively blocked by our firewall. According to the docs when passing nil or false to the proxy option, the environment variables should be ignored as well, but this doesn't seem to be the case.
If we either switch to Ruby version 1.8.7 or unset the environment variable, the call is made successfully without the proxy. The problem is that we need to use a proxy for calls to the internet, but we cannot use them for internal calls (which is the case here) due to our firewall configuration.
Any help would be greatly appreciated!
I can reproduce what you described
http_proxy=http://127.0.0.1 ruby -e "require 'open-uri'; open('http://google.com', proxy: nil)"
~/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http.rb:878:in `initialize': Connection refused - connect(2) (Errno::ECONNREFUSED)
from ~/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http.rb:878:in `open'
from ~/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http.rb:878:in `block in connect'
from ~/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/timeout.rb:52:in `timeout'
from ~/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http.rb:877:in `connect'
from ~/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http.rb:862:in `do_start'
from ~/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http.rb:851:in `start'
from ~/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/open-uri.rb:313:in `open_http'
from ~/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/open-uri.rb:708:in `buffer_open'
from ~/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/open-uri.rb:210:in `block in open_loop'
from ~/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/open-uri.rb:208:in `catch'
from ~/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/open-uri.rb:208:in `open_loop'
from ~/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/open-uri.rb:149:in `open_uri'
from ~/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/open-uri.rb:688:in `open'
from ~/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/open-uri.rb:34:in `open'
from -e:1:in `<main>'
I've digged the code, and I think I've found the problem.
So far open-uri manage correctly the proxy. For instance, in OpenURI.open_loop, it correctly considers that the proxy is set to nil (see open-uri.rb#195
case opt_proxy
when true
find_proxy = lambda {|u| pxy = u.find_proxy; pxy ? [pxy, nil, nil] : nil}
when nil, false
find_proxy = lambda {|u| nil} # <-- THIS LINE
But when it creates a Net::HTTP instance into OpenURI.open_http
http = klass.new(target_host, target_port)
it doesn't pass p_addr = nil. According the source code and doc for HTTP.new
def HTTP.new(address, port = nil, p_addr = :ENV, p_port = nil, p_user = nil, p_pass = nil)
http = super address, port
if proxy_class? then # from Net::HTTP::Proxy()
http.proxy_from_env = #proxy_from_env
http.proxy_address = #proxy_address
http.proxy_port = #proxy_port
http.proxy_user = #proxy_user
http.proxy_pass = #proxy_pass
elsif p_addr == :ENV then # <---- THIS LINE
http.proxy_from_env = true
else
http.proxy_address = p_addr
http.proxy_port = p_port || default_port
http.proxy_user = p_user
http.proxy_pass = p_pass
end
http
end
Above the doc says
To disable proxy detection set +p_addr+ to nil.
I've patched open-uri.rb by replacing the line 291 by
http = proxy ? klass.new(target_host, target_port) : klass.new(target_host, target_port, nil)
Now, it behaves as expected.
just filled an issue in Ruby bug tracker about that.
I'm trying to open browser url based on argument passed to script. Hence I wrote following ruby code:
require 'selenium-webdriver'
require 'test/unit'
class TestTitle < Test::Unit::TestCase
def setup
$driver = Selenium::WebDriver.for :firefox
if ARGV[0] == 'google'
$driver.get 'http://www.google.com'
elsif ARGV[0] == 'twitter'
$driver.get 'http://www.twitter.com'
end
end
def test_title
puts $driver.title
end
def teardown
$driver.quit
end
end
When I passed argument: ruby test.rb 'google', it results into following error:
c:/Ruby193/lib/ruby/1.9.1/test/unit.rb:167:in `block in non_options': file not found: google (ArgumentError)
from c:/Ruby193/lib/ruby/1.9.1/test/unit.rb:146:in `map!'
from c:/Ruby193/lib/ruby/1.9.1/test/unit.rb:146:in `non_options'
from c:/Ruby193/lib/ruby/1.9.1/test/unit.rb:207:in `non_options'
from c:/Ruby193/lib/ruby/1.9.1/test/unit.rb:52:in `process_args'
from c:/Ruby193/lib/ruby/1.9.1/minitest/unit.rb:891:in `_run'
from c:/Ruby193/lib/ruby/1.9.1/minitest/unit.rb:884:in `run'
from c:/Ruby193/lib/ruby/1.9.1/test/unit.rb:21:in `run'
from c:/Ruby193/lib/ruby/1.9.1/test/unit.rb:326:in `block (2 levels) in autorun'
from c:/Ruby193/lib/ruby/1.9.1/test/unit.rb:27:in `run_once'
from c:/Ruby193/lib/ruby/1.9.1/test/unit.rb:325:in `block in autorun'
Please help me understand what I'm doing wrong.
It appears that test-unit (as of 1.9.1) grabs command line options in its GlobOptions module. You are using ARGV[0] to pass browser name, but it thinks you're passing a file name. A workaround is to capture the value of ARGV[0] and then clear it before your test case runs:
browser = ARGV[0]
ARGV[0] = nil
Using Mechanize 2.6.0 on Ruby 1.9.3 I'm trying to fetch a web page over HTTPS from Windows 7x64. When I attempt to get() the URL the CPU usage goes to 100% and the method never returns:
require 'mechanize'
uri = "https://my.com/wiki/api.php?action=query&titles=US4&prop=info&format=xml"
agent = Mechanize.new
u,p = %w[myusername mypassword]
agent.add_auth( uri, u, p )
agent.agent.http.verify_mode = OpenSSL::SSL::VERIFY_NONE
info = agent.get( uri )
When I interrupt it, I get these stack traces (three different runs):
>> info = agent.get( page_api )
IRB::Abort: abort then interrupt!
from C:/Ruby193/lib/ruby/gems/1.9.1/gems/mechanize-2.6.0/lib/mechanize/http/www_authenticate_parser.rb:27:in `call'
from C:/Ruby193/lib/ruby/gems/1.9.1/gems/mechanize-2.6.0/lib/mechanize/http/www_authenticate_parser.rb:27:in `parse'
from C:/Ruby193/lib/ruby/gems/1.9.1/gems/mechanize-2.6.0/lib/mechanize/http/agent.rb:716:in `response_authenticate'
from C:/Ruby193/lib/ruby/gems/1.9.1/gems/mechanize-2.6.0/lib/mechanize/http/agent.rb:306:in `fetch'
from C:/Ruby193/lib/ruby/gems/1.9.1/gems/mechanize-2.6.0/lib/mechanize.rb:431:in `get'
from (irb):10
from C:/Ruby193/bin/irb:12:in `<main>'
>> info = agent.get( page_api )
IRB::Abort: abort then interrupt!
from C:/Ruby193/lib/ruby/gems/1.9.1/gems/mechanize-2.6.0/lib/mechanize/http/www_authenticate_parser.rb:29:in `call'
from C:/Ruby193/lib/ruby/gems/1.9.1/gems/mechanize-2.6.0/lib/mechanize/http/www_authenticate_parser.rb:29:in `new'
from C:/Ruby193/lib/ruby/gems/1.9.1/gems/mechanize-2.6.0/lib/mechanize/http/www_authenticate_parser.rb:29:in `parse'
from C:/Ruby193/lib/ruby/gems/1.9.1/gems/mechanize-2.6.0/lib/mechanize/http/agent.rb:716:in `response_authenticate'
from C:/Ruby193/lib/ruby/gems/1.9.1/gems/mechanize-2.6.0/lib/mechanize/http/agent.rb:306:in `fetch'
from C:/Ruby193/lib/ruby/gems/1.9.1/gems/mechanize-2.6.0/lib/mechanize.rb:431:in `get'
from (irb):11
from C:/Ruby193/bin/irb:12:in `<main>'
>> info = agent.get( page_api )
IRB::Abort: abort then interrupt!
from C:/Ruby193/lib/ruby/gems/1.9.1/gems/mechanize-2.6.0/lib/mechanize/http/www_authenticate_parser.rb:114:in `call'
from C:/Ruby193/lib/ruby/gems/1.9.1/gems/mechanize-2.6.0/lib/mechanize/http/www_authenticate_parser.rb:114:in `token'
from C:/Ruby193/lib/ruby/gems/1.9.1/gems/mechanize-2.6.0/lib/mechanize/http/www_authenticate_parser.rb:31:in `parse'
from C:/Ruby193/lib/ruby/gems/1.9.1/gems/mechanize-2.6.0/lib/mechanize/http/agent.rb:716:in `response_authenticate'
from C:/Ruby193/lib/ruby/gems/1.9.1/gems/mechanize-2.6.0/lib/mechanize/http/agent.rb:306:in `fetch'
from C:/Ruby193/lib/ruby/gems/1.9.1/gems/mechanize-2.6.0/lib/mechanize.rb:431:in `get'
from (irb):12
from C:/Ruby193/bin/irb:12:in `<main>'
How can I work around this problem and properly fetch an HTTPS URL via Ruby on Windows? (If there's a better solution than Mechanize for this—since I only need the source of the page to feed to Nokogiri anyhow—I'm open to not using Mechanize at all.)
Another datapoint: trying the same code on OS X produces the same result.
Here's the actual content of the page, using the alternative fetching method described in my workaround answer below:
p fetch_https_without_ssl_verification(uri, u, p)
#=> "\t\t <?xml version=\"1.0\"?><api><query><normalized><n from=\"Devtools/UI_Composer/DesignSpec/US7294\" to=\"Devtools/UI Composer/DesignSpec/US7294\" /></normalized><pages><page ns=\"0\" title=\"Devtools/UI Composer/DesignSpec/US7294\" missing=\"\" /></pages></query></api>"
If you simply need the contents of the URL (as I do) then using curl instead of Mechanize is far easier, and it works:
def fetch_https_without_ssl_verification( uri, user=nil, pass=nil )
`curl -s -k #{%Q{-u "#{user}#{":"<<pass if pass}"} if user} "#{uri}"`
end
I'm trying to write my first Ruby program, but have a problem. The code has to download 32 MP3 files over HTTP. It actually downloads a few, then times-out.
I tried setting a timeout period, but it makes no difference. Running the code under Windows, Cygwin and Mac OS X has the same result.
This is the code:
require 'rubygems'
require 'open-uri'
require 'nokogiri'
require 'set'
require 'net/http'
require 'uri'
puts "\n Up and running!\n\n"
links_set = {}
pages = ['http://www.vimeo.com/siai/videos/sort:oldest',
'http://www.vimeo.com/siai/videos/page:2/sort:oldest',
'http://www.vimeo.com/siai/videos/page:3/sort:oldest']
pages.each do |page|
doc = Nokogiri::HTML(open(page))
doc.search('//*[#href]').each do |m|
video_id = m[:href]
if video_id.match(/^\/(\d+)$/i)
links_set[video_id[/\d+/]] = m.children[0].to_s.split(" at ")[0].split(" -- ")[0]
end
end
end
links = links_set.to_a
p links
cookie = ''
file_name = ''
open("http://www.tubeminator.com") {|f|
cookie = f.meta['set-cookie'].split(';')[0]
}
links.each do |link|
open("http://www.tubeminator.com/ajax.php?function=downloadvideo&url=http%3A%2F%2Fwww.vimeo.com%2F" + link[0],
"Cookie" => cookie) {|f|
puts f.read
}
open("http://www.tubeminator.com/ajax.php?function=convertvideo&start=0&duration=1120&size=0&format=mp3&vq=high&aq=high",
"Cookie" => cookie) {|f|
file_name = f.read
}
puts file_name
Net::HTTP.start("www.tubeminator.com") { |http|
#http.read_timeout = 3600 # 1 hour
resp = http.get("/download-video-" + file_name)
open(link[1] + ".mp3", "wb") { |file|
file.write(resp.body)
}
}
end
puts "\n Yay!!"
And this is the exception:
/Users/test/.rvm/rubies/ruby-1.9.2-preview1/lib/ruby/1.9.1/net/protocol.rb:140:in `rescue in rbuf_fill': Timeout::Error (Timeout::Error)
from /Users/test/.rvm/rubies/ruby-1.9.2-preview1/lib/ruby/1.9.1/net/protocol.rb:134:in `rbuf_fill'
from /Users/test/.rvm/rubies/ruby-1.9.2-preview1/lib/ruby/1.9.1/net/protocol.rb:116:in `readuntil'
from /Users/test/.rvm/rubies/ruby-1.9.2-preview1/lib/ruby/1.9.1/net/protocol.rb:126:in `readline'
from /Users/test/.rvm/rubies/ruby-1.9.2-preview1/lib/ruby/1.9.1/net/http.rb:2138:in `read_status_line'
from /Users/test/.rvm/rubies/ruby-1.9.2-preview1/lib/ruby/1.9.1/net/http.rb:2127:in `read_new'
from /Users/test/.rvm/rubies/ruby-1.9.2-preview1/lib/ruby/1.9.1/net/http.rb:1120:in `transport_request'
from /Users/test/.rvm/rubies/ruby-1.9.2-preview1/lib/ruby/1.9.1/net/http.rb:1106:in `request'
from /Users/test/.rvm/rubies/ruby-1.9.2-preview1/lib/ruby/1.9.1/open-uri.rb:312:in `block in open_http'
from /Users/test/.rvm/rubies/ruby-1.9.2-preview1/lib/ruby/1.9.1/net/http.rb:564:in `start'
from /Users/test/.rvm/rubies/ruby-1.9.2-preview1/lib/ruby/1.9.1/open-uri.rb:306:in `open_http'
from /Users/test/.rvm/rubies/ruby-1.9.2-preview1/lib/ruby/1.9.1/open-uri.rb:767:in `buffer_open'
from /Users/test/.rvm/rubies/ruby-1.9.2-preview1/lib/ruby/1.9.1/open-uri.rb:203:in `block in open_loop'
from /Users/test/.rvm/rubies/ruby-1.9.2-preview1/lib/ruby/1.9.1/open-uri.rb:201:in `catch'
from /Users/test/.rvm/rubies/ruby-1.9.2-preview1/lib/ruby/1.9.1/open-uri.rb:201:in `open_loop'
from /Users/test/.rvm/rubies/ruby-1.9.2-preview1/lib/ruby/1.9.1/open-uri.rb:146:in `open_uri'
from /Users/test/.rvm/rubies/ruby-1.9.2-preview1/lib/ruby/1.9.1/open-uri.rb:669:in `open'
from /Users/test/.rvm/rubies/ruby-1.9.2-preview1/lib/ruby/1.9.1/open-uri.rb:33:in `open'
from test.rb:38:in `block in <main>'
from test.rb:37:in `each'
from test.rb:37:in `<main>'
I'd also appreciate your comments on the rest of the code.
For Ruby 1.8 I used this to solve my time-out issues. Extending the Net::HTTP class in my code and re-initialized with default parameters including an initialization of my own read_timeout should keep things sane I think.
require 'net/http'
# Lengthen timeout in Net::HTTP
module Net
class HTTP
alias old_initialize initialize
def initialize(*args)
old_initialize(*args)
#read_timeout = 5*60 # 5 minutes
end
end
end
Your timeout isn't in the code you set the timeout for. It's here, where you use open-uri:
open("http://www.tubeminator.com/ajax.php?function=downloadvideo&url=http%3A%2F%2Fwww.vimeo.com%2F" + link[0],
You can set a read timeout for open-uri like so:
#!/usr/bin/ruby1.9
require 'open-uri'
open('http://stackoverflow.com', 'r', :read_timeout=>0.01) do |http|
http.read
end
# => /usr/lib/ruby/1.9.0/net/protocol.rb:135:in `sysread': \
# => execution expired (Timeout::Error)
# => ...
# => from /tmp/foo.rb:5:in `<main>'
:read_timeout is new for Ruby 1.9 (it's not in Ruby 1.8). 0 or nil means "no timeout."