Cannot get timeout to work properly with net http in Ruby - ruby

I am having issues with the timeout of my HTTP Get requests and have tried several things to get it working with no success. Here's my get_request function:
require 'net/http'
require 'openssl'
def get_request(url, headers = "")
uri = URI.parse(url)
request = Net::HTTP::Get.new(uri)
unless headers.empty?
headers.each { |key, value| request[key] = value }
end
if uri.port == 443
if #proxy
response = Net::HTTP.start(uri.host, uri.port, #proxy_ip, #proxy_port, :use_ssl => true, :verify_mode => OpenSSL::SSL::VERIFY_NONE) { |http|
http.request(request)
}
else
response = Net::HTTP.start(uri.host, uri.port, :use_ssl => true, :verify_mode => OpenSSL::SSL::VERIFY_NONE) { |http|
http.request(request)
}
end
else
response = Net::HTTP.start(uri.host, uri.port) { |http|
http.request(request)
}
end
if response.code.to_i == 302 or response.code.to_i == 301
get_request(response["Location"])
else
return response
end
return response
end
I have tried inserting:
http.read_timeout = 3
right before each of these lines:
http.request(request)
but this does not seem to work. The script still hangs for a good 20+ seconds.
I have also tried to insert:
request.read_timeout = 3
right below:
request = Net::HTTP::Get.new(uri)
but this still does not work. I just simply want the script to stop the GET request after 3 seconds, but after placing a puts "Loading get request" line after the method starts, I notice that the request still takes a solid 10+ seconds.
This suggestion has been posted in several areas, but it just doesn't seem to be working for me.

I use open-uri fot that reason, never had a problem with that.
The method below starts a procedure on a Tomcat server.
The response happens to take some time depending on the load on the server.
Before it failed sometimes.
def start_sending(url)
require 'open-uri'
open(url, :read_timeout => 5 * 60) do |response|
# if there is a response it always contains "Ok" for this url
if response.read[/Return: Ok/i]
log "sending ok"
return true
else
log "error sending, no confirmation received\n#{response.read}"
return false
end
end
rescue => e
log_error("#{e.message}\n#{e.backtrace}")
end

Related

Net https requests gives me what appears to be hex

In ruby, I'm doing an HTTP GET request to a website that uses SSL, and for some reason it only gives me the data I need when proxying it through an HTTP proxy (burp suite).
For example:
if uri.port == 443
response = Net::HTTP.start(uri.host, uri.port, "172.16.38.182", "8080", :use_ssl => true, :verify_mode => OpenSSL::SSL::VERIFY_NONE) {|http|
http.request(request)
}
else
response = Net::HTTP.start(uri.host, uri.port) {|http|
http.request(request)
}
end
by proxying this through BurpSuite, I can see the data that I need. But if I try this below:
if uri.port == 443
response = Net::HTTP.start(uri.host, uri.port, :use_ssl => true, :verify_mode => OpenSSL::SSL::VERIFY_NONE) {|http|
http.request(request)
}
else
response = Net::HTTP.start(uri.host, uri.port) {|http|
http.request(request)
}
end
then I get hex characters that looks like this:
Any idea why this would be the case? Is it because of the specific SSL/TLS version used or something and the web application proxy knows how to interact with it but net/https doesn't? can't figure this out.
Turns out the reason this looks the way it does is because it's returned with gzip encoding.
I was able to successfully decode it by using the following:
if response['content-encoding'] == 'gzip'
gz = Zlib::GzipReader.new(StringIO.new(response.body.to_s))
uncompressed_string = gz.read
end

'JSON.parse' error handling invalid/non JSON format

I request information from an HTTP streaming service. It provides data in JSON format. Here is the documentation. Here is a part of the code I am using:
require 'uri'
require 'net/https'
require 'json'
uri = URI("https://api.tradier.com/v1/markets/events/session")
http = Net::HTTP.new(uri.host, uri.port)
request = Net::HTTP::Post.new(uri.request_uri)
http.read_timeout = 30
http.use_ssl = true
http.verify_mode = OpenSSL::SSL::VERIFY_PEER
# Headers
request["Accept"] = "application/json"
request["Authorization"] = "Bearer xxx"
# Send synchronously
response = http.request(request)
# parses response
parse = JSON.parse(response.body)
#out puts values only from response
sessionid = parse.values[0]["sessionid"]
url = parse.values[0]["url"]
uri = URI("#{url}?sessionid=#{sessionid}&symbols=aapl")
http = Net::HTTP.new(uri.host, uri.port)
request = Net::HTTP::Post.new(uri.request_uri)
http.read_timeout = 30
http.use_ssl = true
http.verify_mode = OpenSSL::SSL::VERIFY_PEER
# Headers
request["Accept"] = "application/json"
request["Authorization"] = "Bearer xxx"
http.request request do |response|
response.read_body do |data|
puts data.class
# info = JSON.parse(data, :quirks_mode => true)
# puts info.values
end
end
I want to have the system continue the program. It seems that I need to use begin and rescue, but I cannot get them to work. When I request data, I get the following error:
`parse': 757: unexpected token at '{"type":"trade","symbol":"AAPL","exch":"Q","price":"191.23","size":"1081622","cvol":"18308460","date":"1528747200000","last":"191.23"}{"type":"summary","symbol":"AAPL","open":"191.35","high":"191.97","low":"190.21","prevClose":"191.7","close":"191.23"}' (JSON::ParserError)
The endpoint you are using is documented in Tradier API docs and it's a streaming endpoint.
It appears that the response is not chunked to contain just one JSON document per chunk. It however does appear that the documents are separated by linefeeds, making the response look like:
{ "json": "data" }
{ "more": "data" }
And that is not valid JSON. You probably need to parse them one by one by doing something like:
http.request request do |response|
response.read_body do |data|
data.each_line do |chunk|
info = JSON.parse(chunk)
puts info.inspect
end
end
end
If the response chunking happens in the middle of JSON documents, you must use some kind of buffered reader.

An error has occurred that says {"code":"BadArgument","requestId":"XXXX","message":"JSON format error."} on computer vision api of Azure

require 'net/http'
uri = URI('https://westus.api.cognitive.microsoft.com/vision/v1.0/analyze')
uri.query = URI.encode_www_form({
# Request parameters
'visualFeatures' => 'Adult',
'details' => 'Celebrities',
'language' => 'en'
})
request = Net::HTTP::Post.new(uri.request_uri)
# Request headers
request['Content-Type'] = 'application/json'
# Request headers
request['Ocp-Apim-Subscription-Key'] = 'caa91cccdgywtduwdxxf530'
# Request body
request.body = "http://instudy.jp/wp-content/uploads/2015/03/doraemon-coming.jpg"
response = Net::HTTP.start(uri.host, uri.port, :use_ssl => uri.scheme == 'https') do |http|
http.request(request)
end
puts response.body
I wanted to check the photo whether adult things or not so I wrote the code.
I think there is nothing wrong so I don't know why error's occurred.
environment: ruby 2.3.1
As per the documentaion, the vision api is expecting an image url in 'url' key of request body. Also in your case, you are not passing a valid json as request body. So, replace
request.body = "http://instudy.jp/wp-content/uploads/2015/03/doraemon-coming.jpg"
with
request_body = Hash.new
request_body[:url] = "http://instudy.jp/wp-content/uploads/2015/03/doraemon-coming.jpg"
request.body = request_body.to_json

HTTP Bad Request Debugging Methods

So I'm getting an HTTP Bad Request error when trying a POST to an endpoint. I've been at it for two hours and can't figure out the problem. I've looked at documentation and combed stackoverflow. Help on fixing the code is appreciated, but more importantly, can someone point me to methods for troubleshooting/debugging this myself? A good blog post maybe?
require 'logger'
require 'json'
require 'uri'
require 'net/http'
def put_into_list(payload) # payload is an array of strings
logger = Logger.new("#{File.dirname(__FILE__)}/etc/daily.log", 0, 100 * 1024 * 1024)
logger.level = Logger::DEBUG
config = grab_config
uri = URI("https://website.com/endpoint")
header = {'Content-Type' => 'text/json'}
Net::HTTP.start(uri.host, uri.port, :use_ssl => uri.scheme == 'https',
:verify_mode => OpenSSL::SSL::VERIFY_NONE) do |http|
req = Net::HTTP::Post.new(uri.request_uri, header)
req.basic_auth config['key'], config['pass']
payload.each do |entry|
req.body = entry.to_json
logger.info("Setting req.body to #{req.body}")
begin
res = http.request req
logger.info("Got response of: #{res}")
logger.info("Got response body of: #{res.body}")
rescue => j
logger.info("Rescuing: #{j}")
logger.info("Backtrace: #{j.backtrace}")
end
end
end
end
And this is what my logging file looks like:
I, [2017-06-09T08:44:05.616776 #16932] INFO -- : Setting req.body to "{'prospect':{'email':'test#gmail.com'}}"
I, [2017-06-09T08:44:05.703313 #16932] INFO -- : Got response of: #<Net::HTTPBadRequest:0x2d77aa8>
I, [2017-06-09T08:44:05.704207 #16932] INFO -- : Got response body of: {"status":{"status":"ERROR","code":"E_UNNOWN","msg":"Unknown error."}}

Ruby API (RabbitMQ create user)

Started working on Ruby a week back. Writing an API to connect to RabbitMQ messaging queue. The command line for adding a new user works.
$ curl -i -u guest:guest -H "content-type:application/json" -XPUT -d'{"password":"pwd","tags":"administrator"}' http://localhost:15672/api/users/username
I need to make this Http Put request from Ruby. The following is my code:
def test_add_user
uri = URI.parse('http://localhost:15672/api/users/karthik/')
uri.to_s
http = Net::HTTP.new(uri.host, uri.port)
request = Net::HTTP::Put.new(uri.path)
request.basic_auth 'guest', 'guest'
request['Content-Type'] = 'application/json'
request['Accept'] = 'application/json'
request.set_form_data({'password' => 'secret', 'tags' => 'management'})
http.start do |http|
res = http.request(request)
puts res
end
end
This is the result I get
o.test_add_user
#<Net::HTTPUnsupportedMediaType:0x007fd7fb6fe1d8>
=> nil
Does Media type exception relate with Content-Type?
Only application/json is allowed
Should I use anything like to_json? If yes, where should it be used? Thanks in advance.
Regards
Karthik
Thank you Hector and ptd. Fixed it. Attached the working code for future reference.
def test_add_user
uri = URI.parse('http://localhost:15672/api/users/Test1/')
http = Net::HTTP.new(uri.host, uri.port)
request = Net::HTTP::Put.new(uri.path)
request.basic_auth 'guest', 'guest'
request['Content-Type'] = 'application/json'
request['Accept'] = 'application/json'
request.body = {'password' => 'secret', 'tags' => 'management'}.to_json
http.start do |http|
res = http.request(request)
puts res
end
end
Adds a new user to the RabbitMQ queue

Resources