Net https requests gives me what appears to be hex - ruby

In ruby, I'm doing an HTTP GET request to a website that uses SSL, and for some reason it only gives me the data I need when proxying it through an HTTP proxy (burp suite).
For example:
if uri.port == 443
response = Net::HTTP.start(uri.host, uri.port, "172.16.38.182", "8080", :use_ssl => true, :verify_mode => OpenSSL::SSL::VERIFY_NONE) {|http|
http.request(request)
}
else
response = Net::HTTP.start(uri.host, uri.port) {|http|
http.request(request)
}
end
by proxying this through BurpSuite, I can see the data that I need. But if I try this below:
if uri.port == 443
response = Net::HTTP.start(uri.host, uri.port, :use_ssl => true, :verify_mode => OpenSSL::SSL::VERIFY_NONE) {|http|
http.request(request)
}
else
response = Net::HTTP.start(uri.host, uri.port) {|http|
http.request(request)
}
end
then I get hex characters that looks like this:
Any idea why this would be the case? Is it because of the specific SSL/TLS version used or something and the web application proxy knows how to interact with it but net/https doesn't? can't figure this out.

Turns out the reason this looks the way it does is because it's returned with gzip encoding.
I was able to successfully decode it by using the following:
if response['content-encoding'] == 'gzip'
gz = Zlib::GzipReader.new(StringIO.new(response.body.to_s))
uncompressed_string = gz.read
end

Related

Net http connecting over plaintext even though SSL options are set

Using the net/http gem in Ruby, I am simply trying to connect over port 443/tcp using the following:
response = Net::HTTP.new(uri.host, uri.port, #proxy_ip, #proxy_port, use_ssl: true, verify_mode: OpenSSL::SSL::VERIFY_NONE).start do |http|
request = Net::HTTP::Get.new(uri)
headers.each { |key, value| request[key] = value } unless headers.empty?
http.request(request)
end
However, the response is giving me an error showing that I'm connecting over port 80:
[12] pry(main)> response.body
=> "<html>\r\n<head><title>400 The plain HTTP request was sent to HTTPS port</title></head>\r\n<body>\r\n<center><h1>400 Bad Request</h1></center>\r\n<center>The plain HTTP request was sent to HTTPS port</center>\r\n</body>\r\n</html>\r\n"
My initial way of connecting is this way:
response = Net::HTTP.start(uri.host, uri.port, #proxy_ip, #proxy_port, use_ssl: true, verify_mode: OpenSSL::SSL::VERIFY_NONE) do |http|
http.request(request)
end
However, I needed to implement authentication for the proxy, and apparently I need to do this with the HTTP.new, so this code block was revised to try accommodating proxy authentication.
I have tried taking out #proxy_ip and #proxy_port, but this results in another error:
response = Net::HTTP.new(uri.host, uri.port, use_ssl: true, verify_mode: OpenSSL::SSL::VERIFY_NONE).start do |http|
request = Net::HTTP::Get.new(uri)
headers.each { |key, value| request[key] = value } unless headers.empty?
http.request(request)
end
error:
TypeError: Failed to open TCP connection to {:use_ssl=>true, :verify_mode=>0}:80 (no implicit conversion of Hash into String)
from /usr/local/rvm/rubies/ruby-2.7.6/lib/ruby/2.7.0/net/http.rb:960:in `initialize'
Caused by TypeError: no implicit conversion of Hash into String
from /usr/local/rvm/rubies/ruby-2.7.6/lib/ruby/2.7.0/net/http.rb:960:in `initialize'
My uri is just basically https://www.google.com/page?query=1&query=2
I'm just simply trying to make an HTTPS call while authenticating to a proxy (if one exists and is assigned in the environment variables).
The Net::HTTP.new constructor does not support SSL options directly. You have to add them later in the sequence. The signature for Net:HTTP.new is:
Net::HTTP.new(host, port, proxy_host, proxy_port, proxy_user, proxy_pass)
There is also a requirement to set use_ssl before start is called, otherwise the proxy setup will fail.
So with those things in mind, this should work:
http = Net::HTTP.new(host, port, proxy_host, proxy_port, proxy_user, proxy_pass)
http.use_ssl = true
http.verify_mode = OpenSSL::SSL::VERIFY_NONE
res = http.start { |connection|
request = Net::HTTP::Get.new(uri.request_uri)
headers.each { |key, value| request[key] = value } unless headers.empty?
connection.request(request)
}

Cannot get timeout to work properly with net http in Ruby

I am having issues with the timeout of my HTTP Get requests and have tried several things to get it working with no success. Here's my get_request function:
require 'net/http'
require 'openssl'
def get_request(url, headers = "")
uri = URI.parse(url)
request = Net::HTTP::Get.new(uri)
unless headers.empty?
headers.each { |key, value| request[key] = value }
end
if uri.port == 443
if #proxy
response = Net::HTTP.start(uri.host, uri.port, #proxy_ip, #proxy_port, :use_ssl => true, :verify_mode => OpenSSL::SSL::VERIFY_NONE) { |http|
http.request(request)
}
else
response = Net::HTTP.start(uri.host, uri.port, :use_ssl => true, :verify_mode => OpenSSL::SSL::VERIFY_NONE) { |http|
http.request(request)
}
end
else
response = Net::HTTP.start(uri.host, uri.port) { |http|
http.request(request)
}
end
if response.code.to_i == 302 or response.code.to_i == 301
get_request(response["Location"])
else
return response
end
return response
end
I have tried inserting:
http.read_timeout = 3
right before each of these lines:
http.request(request)
but this does not seem to work. The script still hangs for a good 20+ seconds.
I have also tried to insert:
request.read_timeout = 3
right below:
request = Net::HTTP::Get.new(uri)
but this still does not work. I just simply want the script to stop the GET request after 3 seconds, but after placing a puts "Loading get request" line after the method starts, I notice that the request still takes a solid 10+ seconds.
This suggestion has been posted in several areas, but it just doesn't seem to be working for me.
I use open-uri fot that reason, never had a problem with that.
The method below starts a procedure on a Tomcat server.
The response happens to take some time depending on the load on the server.
Before it failed sometimes.
def start_sending(url)
require 'open-uri'
open(url, :read_timeout => 5 * 60) do |response|
# if there is a response it always contains "Ok" for this url
if response.read[/Return: Ok/i]
log "sending ok"
return true
else
log "error sending, no confirmation received\n#{response.read}"
return false
end
end
rescue => e
log_error("#{e.message}\n#{e.backtrace}")
end

Unauthorized readbody=true when posting message to hangout chat room with webhook

Trying to post a message to a hangout chat room. I've generated a webhook for the room and used it as uri in the following code. The rest are basic net/http stuff.
require 'net/http'
require 'uri'
message = 'hello'
# prep and send the http request
uri = URI.parse("https://chat.googleapis.com/v1/spaces/AAAAcroWtl4/messages?key=abc&token=xyz")
request = Net::HTTP::Post.new(uri)
request.content_type = "application/json"
request.body = '"content":[{"type":"text","text":"'+message+'"}]'
req_options = { use_ssl: uri.scheme == "https" }
response = Net::HTTP.start(uri.hostname, uri.port, req_options) do |http|
http.request(request)
end
puts response.inspect
The response contains the following text.
#<Net::HTTPUnauthorized 401 Unauthorized readbody=true>
Is there something wrong with the request body?
Edit: key and token changed in the question.
This same code worked with stride rooms, the only difference is that it had
request["Authorization"] = "Bearer #{access_token}"
Since the token is already in the uri I figured this would not be needed.
This worked for me
require 'net/http'
require 'uri'
require 'json'
message = 'hello'
# prep and send the http request
uri = URI.parse("//webhook")
request = Net::HTTP::Post.new(uri)
request['Content-Type'] = "application/json"
request.body = { text: message }.to_json
response = Net::HTTP.start(uri.hostname, uri.port, use_ssl: true) do |http|
http.request(request)
end
puts response.inspect

An error has occurred that says {"code":"BadArgument","requestId":"XXXX","message":"JSON format error."} on computer vision api of Azure

require 'net/http'
uri = URI('https://westus.api.cognitive.microsoft.com/vision/v1.0/analyze')
uri.query = URI.encode_www_form({
# Request parameters
'visualFeatures' => 'Adult',
'details' => 'Celebrities',
'language' => 'en'
})
request = Net::HTTP::Post.new(uri.request_uri)
# Request headers
request['Content-Type'] = 'application/json'
# Request headers
request['Ocp-Apim-Subscription-Key'] = 'caa91cccdgywtduwdxxf530'
# Request body
request.body = "http://instudy.jp/wp-content/uploads/2015/03/doraemon-coming.jpg"
response = Net::HTTP.start(uri.host, uri.port, :use_ssl => uri.scheme == 'https') do |http|
http.request(request)
end
puts response.body
I wanted to check the photo whether adult things or not so I wrote the code.
I think there is nothing wrong so I don't know why error's occurred.
environment: ruby 2.3.1
As per the documentaion, the vision api is expecting an image url in 'url' key of request body. Also in your case, you are not passing a valid json as request body. So, replace
request.body = "http://instudy.jp/wp-content/uploads/2015/03/doraemon-coming.jpg"
with
request_body = Hash.new
request_body[:url] = "http://instudy.jp/wp-content/uploads/2015/03/doraemon-coming.jpg"
request.body = request_body.to_json

Ruby Sinatra create post request without PEM and OpenSSL::SSL::VERIFY_NONE

I try create POST request with SSL but without OpenSSL::SSL::VERIFY_NONE because it is opend up security attacks and without PEM certificate.
But I catch problems, my ruby code for send POST request:
post '/test/test1' do
cross_origin
post_data = request.body.read
res_Data = JSON.parse(post_data)
userName = res_Data['username']
#responseFromServer=''
uri = URI('https://test.com/test1')
Net::HTTP.start(uri.host, uri.port,
:use_ssl => uri.scheme == 'https',
:verify_mode => OpenSSL::SSL::VERIFY_NONE) do |http|
request = Net::HTTP::Post.new uri.request_uri
request.basic_auth 'aa', 'bb'
request.body = {'username' =>userName}.to_json
response = http.request request
#responseFromServer = response.body.to_s
end
newJson = JSON.parse(#responseFromServer)
status_msg = newJson['status']['status_msg']
if (status_msg == "Success")
return 'true'
end
return 'false'
end
It is method worked but he use OpenSSL::SSL::VERIFY_NONE. How to create method for send POST request without OpenSSL::SSL::VERIFY_NONE and PEM sertificate?
EDIT
SSL/HTTPS request
Update: There are some good reasons why this code example is bad. It introduces a potential security vulnerability if it's essential you use the server certificate to verify the identity of the server you're connecting to. There's a fix for the issue though!
require "net/https"
require "uri"
uri = URI.parse("https://secure.com/")
http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = true
http.verify_mode = OpenSSL::SSL::VERIFY_NONE
request = Net::HTTP::Get.new(uri.request_uri)
response = http.request(request)
response.body
response.status
response["header-here"] # All headers are lowercase
SSL/HTTPS request with PEM certificate
require "net/https"
require "uri"
uri = URI.parse("https://secure.com/")
pem = File.read("/path/to/my.pem")
http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = true
http.cert = OpenSSL::X509::Certificate.new(pem)
http.key = OpenSSL::PKey::RSA.new(pem)
http.verify_mode = OpenSSL::SSL::VERIFY_PEER
request = Net::HTTP::Get.new(uri.request_uri)
My question: How to create POST method without PEM and OpenSSL::SSL::VERIFY_NONE?
This question is quite misleading, but I try my best to figure it out. Here is my advise:
Do you want to connect to a service that is only available through https and you do not care if the certificate is valid?
Then you can use :verify_mode => OpenSSL::SSL::VERIFY_NONE when initializing the Net::HTTP client.
You will have some kind of transport security, but you cannot be sure the server you are talking to is the one you think it is. You are vulnerable.
Do you want to connect to a service that is available both through https and http, and you do not care about transport security?
Then you should use the http://... endpoint.
Do you want to connect to a service and you care about transport security?
Then you should definitely use the https://... endpoint.
Do not override :verify_mode!
If you are getting certificate verification errors, make sure you have the correct Certificate Authority installed on your system.

Resources