I have an error connecting in Ruby to the URL listed below, even though the URL exists. Why is that?
1.9.3p194 :003 > require 'uri'
=> true
1.9.3p194 :004 > require 'net/http'
=> true
1.9.3p194 :005 > url = "https://blogs.oracle.com/ksplice/entry/introducing_redpatch"
=> "https://blogs.oracle.com/ksplice/entry/introducing_redpatch"
1.9.3p194 :006 > url_parsed = URI.parse(url)
=> #<URI::HTTPS:0x00000001939288 URL:https://blogs.oracle.com/ksplice/entry/introducing_redpatch>
1.9.3p194 :007 > response = Net::HTTP.get_response(url_parsed)
Errno::ECONNRESET: Connection reset by peer
Rather than use Net::HTTP, simplify your life and use Ruby's OpenURI. Unless you need low-level control or visibility of low-level values, you'll find OpenURI is good enough:
require 'open-uri'
url = "https://blogs.oracle.com/ksplice/entry/introducing_redpatch"
open(url).read.size
=> 35493
Use this
url = "https://blogs.oracle.com/ksplice/entry/introducing_redpatch"
uri = URI.parse(url)
http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = true
http.verify_mode = OpenSSL::SSL::VERIFY_NONE
request = Net::HTTP::Get.new(uri.request_uri)
response = http.request(request)
response.body
It's taken from here: http://www.rubyinside.com/nethttp-cheat-sheet-2940.html
Related
I am using net/http to send a bunch of request to some internal IP addresses.
Here's a snippet of the code:
File.open("internalcorpIPs", "r") do |f|
f.each_line do |line|
puts line
res = Net::HTTP.get_response(URI.parse(line))
getCode = res.code
end
end
I'm strictly just making a request to http://IP and https://IP but it seems like this method only works if every single IP/line address is live. How do I skip IP addresses with no webserver (or 80/443 ports)?
Is it possible to make it read the line, and move on to the next if no response code was returned?
Thanks!
You could simply wrap your request in begin/rescue block like this:
File.open("internalcorpIPs", "r") do |f|
f.each_line do |line|
puts line
begin
# strip and encode uri from the file
uri = URI.parse(URI.encode(line.strip))
res = Net::HTTP.get_response(uri)
getCode = res.code
rescue Timeout::Error => e
puts e
false
end
end
end
But you will wait for 60 seconds at least before going in timeout, so I suggest to decrease the timeout. Furthermore, you could introduce an additional guard clause to check if the uri contains the scheme http:// or https://, otherwise raise an exception (or something else).
require 'net/http'
File.open("internalcorpIPs", "r") do |f|
f.each do |line|
puts line
begin
# strip and encode uri from the file
uri = URI.parse(URI.encode(line.strip))
# if uri misses the schema (http:// or https://) -> raise error
raise URI::Error, "uri #{uri} miss the scheme" unless uri.scheme
http = Net::HTTP.new(uri.host, uri.port)
http.open_timeout = 2 # seconds
http.read_timeout = 2 # seconds
http.start do |conn|
response = conn.request_get(path = '/')
puts response.code
end
rescue Timeout::Error, URI::Error, SocketError => e
puts e
false
end
end
end
Additional notes:
Open Timeout
Number of seconds to wait for the connection to open. Any number may be used, including Floats for fractional seconds. If the HTTP object cannot open a connection in this many seconds, it raises a Net::OpenTimeout exception. The default value is 60 seconds.
Read Timeout
Number of seconds to wait for one block to be read (via one read(2) call). Any number may be used, including Floats for fractional seconds. If the HTTP object cannot read data in this many seconds, it raises a Net::ReadTimeout exception. The default value is 60 seconds.
URI Scheme
Difference between generic uri (URI::Generic) and http uri (URI::HTTP).
uri = URI.parse('1.1.1.1')
=> #<URI::Generic 1.1.1.1>
uri.scheme
=> nil
uri.host
=> nil
uri.port
=> nil
uri.path
=> "1.1.1.1"
uri = URI.parse('http://1.1.1.1')
=> #<URI::HTTP http://1.1.1.1>
uri.scheme
=> "http"
uri.host
=> "1.1.1.1"
uri.port
=> 80
uri.path
=> ""
references:
Net::HTTP Api
URI Module
hope it helps!
UPDATE
URI.parse accepts a string as argument and automatically set the port if not specified:
❯ irb
2.2.0 :001 > require 'net/http'
=> true
2.2.0 :002 > uri = URI.parse('http://1.1.1.1')
=> #<URI::HTTP http://1.1.1.1>
2.2.0 :003 > uri.host
=> "1.1.1.1"
2.2.0 :004 > uri.port
=> 80
2.2.0 :005 > uri2 = URI.parse('http://mydomain')
=> #<URI::HTTP http://mydomain>
2.2.0 :006 > uri2.host
=> "mydomain"
2.2.0 :007 > uri2.port
=> 80
2.2.0 :008 > uri3 = URI.parse('https://mydomain')
=> #<URI::HTTPS https://mydomain>
2.2.0 :009 > uri3.host
=> "mydomain"
2.2.0 :010 > uri3.port
=> 443
I am trying to pull campaign stats from Clickbank API in ruby. When I run the sample code Clickbank provided. I get the following error:
uninitialized constant Net::HTTPS (NameError). What am I missing?
Example Code.
require 'net/http'
require 'net/https'
http = Net::HTTPS.new('api.clickbank.com')
http.use_ssl = false
path = '/rest/1.3/orders/list'
headers = {
'Authorization' => '<< DEVKEY >>:<< APIKEY>>',
'Accept' => 'application/json'
}
resp, data = http.get(path, nil, headers)
puts 'Code = ' + resp.code
puts 'Message = ' + resp.message
resp.each {|key, val| puts key + ' = ' + val}
puts data
Yes I put my dev and api key into
In Ruby 2.4.1 enable ssl as a parameter of Net::HTTP.start
Net::HTTP.start(uri.host, uri.port, use_ssl: true)
https://ruby-doc.org/stdlib-2.4.1/libdoc/net/http/rdoc/Net/HTTP.html#class-Net::HTTP-label-HTTPS
Use Net:HTTP and enable SSL instead of using Net::HTTPS and disabling SSL.
Example:
http = Net::HTTP.new('api.clickbank.com')
http.use_ssl = true
http.verify_mode = OpenSSL::SSL::VERIFY_NONE
You actually don't want to disable ssl as that API requires it. I was able to get it working like so based on the documentation for http found here: http://ruby-doc.org/stdlib-2.1.1/libdoc/net/http/rdoc/Net/HTTP.html
require 'net/http'
uri = URI('https://api.clickbank.com/rest/1.3/orders/list')
req = Net::HTTP::Get.new(uri)
# set headers on the request
req['Authorization'] = '<< DEVKEY >>:<< APIKEY>>'
req['Accept'] = 'application/json'
# perform the request
resp, data = Net::HTTP.start(uri.hostname, uri.port) {|http|
http.request(req)
}
puts 'Code = ' + resp.code
puts 'Message = ' + resp.message
resp.each {|key, val| puts key + ' = ' + val}
puts data
For example, to get response 200 OK from "example.com", necessary:
require 'net/http'
uri = URI('http://example.com/index.html')
res = Net::HTTP.get_response(uri)
puts res.code # => '200'
puts res.message # => 'OK'
How to make similar for pop.gmail.com?
Try this:
require "net/pop"
Net::POP3.enable_ssl(OpenSSL::SSL::VERIFY_NONE)
conn = Net::POP3.new("pop.gmail.com", 995)
conn.start(user_name, password)
conn.started?
require 'net/http'
require 'uri'
Net::HTTP.get_print URI.parse('https://forums.malwarebytes.org/index.php?showtopic=49893')
I am getting an error as follows::
c:/ruby/lib/ruby/1.8/net/protocol.rb:133:in sysread': end of file reached (EOFError)
from c:/ruby/lib/ruby/1.8/net/protocol.rb:133:inrbuf_fill'
from c:/ruby/lib/ruby/1.8/timeout.rb:56:in timeout'
from c:/ruby/lib/ruby/1.8/timeout.rb:76:intimeout'
from c:/ruby/lib/ruby/1.8/net/protocol.rb:132:in rbuf_fill'
from c:/ruby/lib/ruby/1.8/net/protocol.rb:116:inreaduntil'
from c:/ruby/lib/ruby/1.8/net/protocol.rb:126:in readline'
from c:/ruby/lib/ruby/1.8/net/http.rb:2029:inread_status_line'
from c:/ruby/lib/ruby/1.8/net/http.rb:2018:in read_new'
from c:/ruby/lib/ruby/1.8/net/http.rb:1059:inrequest'
from c:/ruby/lib/ruby/1.8/net/http.rb:957:in request_get'
from c:/ruby/lib/ruby/1.8/net/http.rb:380:inget_response'
from c:/ruby/lib/ruby/1.8/net/http.rb:547:in start'
from c:/ruby/lib/ruby/1.8/net/http.rb:379:inget_response'
from c:/ruby/lib/ruby/1.8/net/http.rb:337:in `get_print'
from urlparsing1.rb:3
I can’t reproduce the aforementioned behaviour on modern ruby version, but I think I know what causes it.
Let’s dive into execution process step by step:
▶ u = URI.parse('https://forums.malwarebytes.org/index.php?showtopic=49893')
#⇒ #<URI::HTTPS https://forums.malwarebytes.org/index.php?showtopic=49893>
OK, URI got.
▶ Net::HTTP.get_print u
#⇒ <html><head><meta http-equiv='refresh' content='0;
# url=/index.php?/topic/49893-removal-instructions-for-a-fast-antivirus/'>
# </head><body></body></html>=> nil
Ooups. Redirect. Let’s handle it:
▶ u_redir = Net::HTTP.get(u).scan(/url=(.*?)'/).first.first
#⇒ "/index.php?/topic/49893-removal-instructions-for-a-fast-antivirus/"
▶ u.path, u.query = u_redir.split '?'
#⇒ [
# [0] "/index.php",
# [1] "/topic/49893-removal-instructions-for-a-fast-antivirus/"
# ]
▶ Net::HTTP.get_print u
#⇒ [HTML CONTENT]
The summing up: I would guess you should handle redirects yourselves.
You can try by using this:
require 'net/http'
require 'uri'
uri = URI.parse('https://forums.malwarebytes.org/index.php?showtopic=49893')
request = Net::HTTP::Get.new uri.request_uri
res = Net::HTTP.start(uri.host, uri.port,
:use_ssl => uri.scheme == 'https') {|http| http.request request}
p res.body
This works great:
require 'net/http'
uri = URI('http://api.twitter.com/1/statuses/user_timeline.json')
args = {include_entities: 0, include_rts: 0, screen_name: 'johndoe', count: 2, trim_user: 1}
uri.query = URI.encode_www_form(args)
resp = Net::HTTP.get_response(uri)
puts resp.body
But changing from http to https leads to a meaningless error. I'm not asking why the error is meaningless, I would just like to know what's the closest means of doing get_response for https?
I've seen the 'HTTPS' example in the Net::HTTP doc but it looks not very impressive and will make me manually compose the URL from my parameters hash - no good.
Here is a example which works for me under Ruby 1.9.3
require "net/http"
uri = URI.parse("https://api.twitter.com/1/statuses/user_timeline.json")
args = {include_entities: 0, include_rts: 0, screen_name: 'johndoe', count: 2, trim_user: 1}
uri.query = URI.encode_www_form(args)
http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = true
request = Net::HTTP::Get.new(uri.request_uri)
response = http.request(request)
response.body