I have this snippet of code:
def httpsGet url
uri = URI.parse(url)
http = Net::HTTP.new(uri.host, uri.port)
request = Net::HTTP::Get.new(uri.request_uri)
http.use_ssl = true
request.initialize_http_header({"someHeader" => "82739840273985734"})
http.request(request)
end
i've been running a script that uses this just fine for the past week. the script basically calls out to some 3rd party service with different parameters many many times over and over again. suddenly, yesterday and today, this method seems to be hanging sometimes (i stuck puts in several places). it is annoying because this method sometimes hangs after 100 calls, sometimes 20 calls, sometimes many hours later...etc.
is that code not the best way to make an https call with headers in Ruby?
how do i debug this to ensure i'm not doing something wrong?
is the 3rd party service down? but even if so, shouldn't the connection in ruby time out? (like i get a timeout exception) ?
Take a look at open_timeout and ssl_timeout timeout defined for this library:
http = Net::HTTP.new(uri.host, uri.port)
http.open_timeout = 5 # create connection timeout after 5 seconds
http.ssl_timeout = 5 # read timeout after 5 seconds
Related
I have a bunch of web requests I am making in parallel right now using the parallel gem. This is causing all kinds of memory issues due to vfork. These web requests take around 30 seconds each. Is there a way I can queue them all up asynchronously and have them start at the same time without using the parallel gem?
Right now I use Faraday to do the web requests. The code for each request looks like this:
conn = Faraday.new(url: TRIGGER_URL)
conn.post do |req|
req.headers['Content-Type'] = 'application/json'
req.options.timeout = 540
req.body = {auth_key: AUTH_KEY, image_url: image_url, space_id: space_id, scene_num: scene_num, cylinder_mode: cylinder_mode}.to_json
end.body
The async-http gem did exactly what I want.
https://github.com/socketry/async-http
There are several methods in the Net::HTTP API shipped with Ruby core. Some like Net::HTTP.get, Net::HTTP.get_response and Net::HTTP.post_form do not persist connections. However, ::start and ::new do persist connections. In fact, ::start takes a block remains with open connection until block exits:
uri = URI("example.com")
Net::HTTP.start(uri.host, uri.port) do |http|
response1 = http.get('path1')
response2 = http.get('path2')
end
Now you can use ::new instead of ::start:
http = Net::HTTP.new('example.com')
http.start
response1 = http.get('path1')
response2 = http.get('path2')
But must I manually close the connection? Is this a necessity or can I get away with not using this:
begin
http = Net::HTTP.new('example.com')
http.start
response1 = http.get('path1')
response2 = http.get('path2')
ensure
http.finish
end
If I skip the finish call, will this create some kind of bloat or memory leak in the application?
Using new does not open the connection, according to the documentation:
Creates a new Net::HTTP object without opening a TCP connection or
HTTP session.
But to answer your question, yes, you should close your connections after using them. The connection will close eventually, but it is simply bad practice not to clean up your own unmanaged resources when you are done with them.
The preferred method is the way you demonstrated, using start with a block, which ensures the connection will close when the block exits.
EDIT
I was not specific enough. I was referring to the class method start, not the instance method start. The class method signature looks like this:
start(address, port, p_addr, p_port, p_user, p_pass, &block)
start(address, port=nil, p_addr=nil, p_port=nil, p_user=nil, p_pass=nil, opt, &block)
Documentation for it can be found here, though I am sure you can figure out the parameters already.
I've written a consumer for a payment API. My code simply issues a POST request and gets a response from the API. I've implemented that with Net::HTTP, here are the relevant lines of code:
http = Net::HTTP.new(uri.host, 443)
http.use_ssl = true
http.verify_mode = OpenSSL::SSL::VERIFY_NONE
request = Net::HTTP::Post.new(uri.request_uri)
request.set_form_data(params)
response = http.request(request)
This worked for years, however, recently some requests have reached timeouts when the API is under stress. The API maintainer came up with this explanation:
We pass on the data to RabbitMQ synchronously after flushing the HTTP response. Apparently, some HTTP libs wait for the connection to be closed before the program continues on the consumer side and we think this is happening here. Please reconfigure your consumer not to wait for close but to continue right after the response has been flushed.
I'm not sure how Net::HTTP is implemented and whether it really waits for the close when the response has been flushed. The docs don't say anything about it nor is there a setting to control any of this. And to make matters worse, I don't really know how to simulate this.
Any ideas are very welcome!
I guess the following experiment (with Ruby 2.3) should give the answer, I post it here in case someone else stumbles across this question in the future.
server.rb:
require 'socket'
server = TCPServer.new('localhost', 2345)
loop do
socket = server.accept
request = socket.gets
STDERR.puts request
response = "Hello World at #{Time.now}!\n"
socket.print "HTTP/1.1 200 OK\r\n" +
"Content-Type: text/plain\r\n" +
"Content-Length: #{response.bytesize}\r\n" +
"Connection: close\r\n"
socket.print "\r\n"
socket.print response
socket.flush
sleep 10
socket.close
end
client.rb:
require 'net/http'
http = Net::HTTP.new('localhost', 2345)
request = Net::HTTP::Post.new('/')
response = http.request(request)
puts response.body
Running the server, the client will send one request and exit. It does so immediately, so the flush is sufficient to have the client code continue. Restarting the client within the 10 seconds wait of the server, causes the client the hang until the 10 seconds have fully elapsed, then printing the Hello World and once again immediately exiting.
In other words: Such a simple Net::HTTP client does not wait for the connection to close but continues to execute it's code once the server has flushed.
I have working code that gets data over https (below). In fact it runs some test through php. I used standard timeout that works fine. Now while "waiting" for server response I need to implement timer. Because in some cases the test won't finish - the php code will not finish - the ruby time out works ok. So I need to kill some process to capture the error in the existing https session.
How can I implement my own time out for https request on top of existing time out?
The existing timeout will be always greater than custom timeout. eg existing timeout is 10mins and the custom will be 5 mins.
uri = URI.parse(url)
start = Time.new
http_read_timeout=60*10
connection = Net::HTTP.new(uri.host, 443)
connection.use_ssl = true
begin
response = connection.start() do |http|
http.open_timeout = 50
http.read_timeout = http_read_timeout
http.request_get(uri.request_uri)
# here I need to place a code that is triggered
# in case of custom timeout is reached
end
rescue Timeout::Error
# "Connection failed
time_out_message ="security time out - after #{http_read_timeout} sec"
return time_out_message
end
puts "finished"
I don't get it. What does your custom timeout do? You are making an HTTP request...it either returns or times out.
You're already setting the timeout value. Your code can't reach into the future & tell you what the external code would eventually return, if it did...so what do you want it to do, exactly?
But if you really just need an external Timeout wrapper, you can use Timeout::Timeout. Like this:
require 'timeout'
Timeout::timeout(your_timeout_period) do
run_some_code
rescue => err
do_something_with err
# and maybe the below?
raise
end
I have a very basic TCP server implemented in Ruby. In general it does what it's supposed to, but every once in a while I get "The connection to the server was reset while the page was loading" error. I have a feeling that it has something to do with close terminating the connection too soon. If so, how do I wait for all the data to be sent? Or is it something else?
require 'socket'
server = TCPServer.new('', 80)
loop do
session = server.accept
begin
session.print Time.now
ensure
session.close
end
end
I'm not an expert in this area, but here is what I believe is happening....
The browser sends a GET request with the header field "Connection: keep-alive". So the browser is expecting to keep the connection alive at least until it receives a complete chunk of the response. Under this protocol, the server response must include a header specifying the length of the response, so that the browser knows when it has received the complete response. After this point, the connection can be closed without the browser caring.
The original example closes the connection too quickly, before the browser can validate that a complete response was received. Curiously, if I run that example and refresh my browser several times, it will load about every 1 in 10 tries. Maybe this erratic behavior is due to the browser occasionally executing fast enough to beat my server closing the connection.
Below is a code example that executes consistently in my browser:
require 'socket'
response = %{HTTP/1.1 200 OK
Content-Type: text;charset=utf-8
Content-Length: 12
Hello World!
}
server = TCPServer.open(80)
loop do
client = server.accept
client.puts response
sleep 1
client.close
end
I suspect it's because the browser is expecting an HTTP response with headers &c. Curiously, you can make the "reset" error happen every time if you put before the "ensure" a sleep of, say, one second.
How to fix it depends upon what you are after. If this is not to be an HTTP server, then don't use the browser to test it. Instead, use telnet or write a little program. If it is to be an HTTP server, then take a look at webrick, which is built into Ruby MRI >= 1.8. Here's how:
#!/usr/bin/ruby1.8
require 'webrick'
# This class handles time requests
class TimeServer < WEBrick::HTTPServlet::AbstractServlet
def do_GET(request, response)
response.status = 200
response['Content-Type'] = 'text/plain'
response.body = Time.now.to_s
end
end
# Create the server. There are many other options, if you need them.
server = WEBrick::HTTPServer.new(:Port=>8080)
# Whenever a request comes in for the root page, use TimeServer to handle it
server.mount('/', TimeServer)
# Finally, start the server. Does not normally return.
server.start
Also, should note that including Connection: close in the response header doesn't seem to help me at all with this connection reset error in my browser (FFv3.6). I have to include both the content-length header field, and include the sleep method to put some delay in the connection closing in order to get a consistent response in my browser.