Ruby - remote file download ...timeout? - ruby

Excuse the tabs. I'm trying to download a file from remote to local and I keep getting one back that is exactly 310 bytes (regardless of what file I choose to download). I tried setting the timeout to 0, but this isn't working. What am I doing wrong? Thanks!
#downloadUrl = 'https://username:password#api.net'
Net::HTTP.start(#downloadUrl) do |http|
response = http.get('/file.ext')
open('/Users/me/file.ext', "wb", :read_timeout=>0) do |file|
file.write(response.body)
end
end
EDIT: I don't want to use httpclient, I want to use standard net/http. I am almost there, but I keep getting initialize': getaddrinfo: nodename nor servname provided, or not known (SocketError) thrown at Net::HTTP.start(url.path). But when I remote "https", I get ECONNREFUSED. ...Getting closer?
url = URI.parse('https://api.net/file.ext')
#request = Net::HTTP.start(url.path)
#request.basic_auth 'username', 'password'
sock = Net::HTTP.new(url.host, 443)
sock.use_ssl = true
sock.ssl_version='SSLv3'
sock.start do |http|
response = http.get(#request)
open('/Users/me/file.ext', "wb", :read_timeout=>0) do |file|
file.write(response.body)
end
end

Using httpclient is much simpler when accessing via SSL.
gem install httpclient
I haven't tried this out, but this should work for you. Here is the rdoc.
require 'httpclient'
domain = "https://api.net/file.ext"
username = "username"
password = "password"
clnt = HTTPClient.new
clnt.set_auth(domain, username, password)
res = clnt.get_content(https_url)
You can refer to the "writing a binary file in ruby" question for saving your response to a file.
Updated Using net/http
You should be doing Net::HTTP.start(url.host, url.port) instead of Net:HTTP.start(url.path). See RDoc for Net::HTTP.start.
url = URI.parse("https://api.net/file.ext")
download_file = opne("/Users/me/file.ext", "wb")
request = Net::HTTP.start(url.host, url.port)
# .. set basic auth, verify peer etc
begin
request.request_get(url.path) do |resp|
resp.read_body { |segment| download_file.write(segment) }
end
ensure
download_file.close
end

Related

Checking Facebook Status Using Ruby

I wanted to check facebook status of another page from ruby script. First of all is it possible. I have been doing the following:
I got developer account
I got app key and secret
I installed json_pure gem
Here is my code:
require 'rubygems'
require 'json/pure'
require 'net/http'
url ="https://graph.facebook.com/user_id/feed?access_token=app_id|app_secret"
uri = URI.parse(URI.encode(url.strip))
#to remove specia codes encode
#to remoce whitespace strip
req = Net::HTTP::Get.new(uri.to_s)
res = Net::HTTP.start(uri.host, uri.port) {|http|
http.request(req)
}
html = res.body
res = JSON.parse(html)
Here is the error:
C:/Ruby187/lib/ruby/1.8/net/protocol.rb:135:in `sysread': An existing connection
was forcibly closed by the remote host. (Errno::ECONNRESET)
I would recommend using the koala gem instead of Net::Http
#graph = Koala::Facebook::API.new(oauth_access_token)
status = #graph.get_connections("me", user_id, "status")

How to do basic authentication over HTTPs in Ruby?

After looking a lot, I've found some solutions that seem working, but not for me...
For example, I have this script:
require 'net/http'
require "net/https"
#http=Net::HTTP.new('www.xxxxxxx.net', 443)
#http.use_ssl = true
#http.verify_mode = OpenSSL::SSL::VERIFY_NONE
#http.start() {|http|
req = Net::HTTP::Get.new('/gb/PastSetupsXLS.asp?SR=31,6')
req.basic_auth 'my_user', 'my_password'
response = http.request(req)
print response.body
}
When I run it, it gives me a page that requests for authentication, but if I write the following URL in the browser, I get into the website without problems:
https://my_user:my_password#www.xxxxxxx.net/gb/PastSetupsXLS.asp?SR=31,6
I have also tried with open-uri:
module OpenSSL
module SSL
remove_const :VERIFY_PEER
end
end
OpenSSL::SSL::VERIFY_PEER = OpenSSL::SSL::VERIFY_NONE
def download(full_url, to_here)
writeOut = open(to_here, "wb")
writeOut.write(open(full_url, :http_basic_authentication=>["my_user", "my_password"]).read)
writeOut.close
end
download('https://www.xxxxxxx.net/gb/PastSetupsXLS.asp?SR=31,6', "target_file.html")
But the result is the same, the site is asking for user authentication.
Any tips of what am I doing wrong?. Must I encode the password in Base 64?
I wrote a piece of code based on examples given in the Net::HTTP docs and tested it on my local WAMP server - it works fine. Here's what I have:
require 'net/http'
require 'openssl'
uri = URI('https://localhost/')
Net::HTTP.start(uri.host, uri.port,
:use_ssl => uri.scheme == 'https',
:verify_mode => OpenSSL::SSL::VERIFY_NONE) do |http|
request = Net::HTTP::Get.new uri.request_uri
request.basic_auth 'matt', 'secret'
response = http.request request # Net::HTTPResponse object
puts response
puts response.body
end
And my .htaccess file looks like this:
AuthName "Authorization required"
AuthUserFile c:/wamp/www/ssl/.htpasswd
AuthType basic
Require valid-user
My .htpasswd is just a one liner generated with htpasswd -c .htpasswd matt for password "secret". When I run my code I get "200 OK" and contents of index.html. If I remove the request.basic_auth line, I get 401 error.
UPDATE:
As indicated by #stereoscott in the comments, the :verify_mode value I used in the example (OpenSSL::SSL::VERIFY_NONE) is not safe for production.
All available options listed in the OpenSSL::SSL::SSLContext docs are: VERIFY_NONE, VERIFY_PEER, VERIFY_CLIENT_ONCE, VERIFY_FAIL_IF_NO_PEER_CERT, out of which (according to the OpenSSL docs) only the first two ones are used in the client mode.
So VERIFY_PEER should be used on production, which is the default btw, so you can skip it entirely.
The following is what ended up working for me:
require "uri"
require "net/http"
url = URI("https://localhost/")
https = Net::HTTP.new(url.host, url.port)
https.use_ssl = true
request = Net::HTTP::Get.new(url)
request["Authorization"] = "Basic " + Base64::encode64("my_user:my_password")
response = https.request(request)
puts response.read_body
I came up with this by building a new HTTP Request in Postman, specifying the URL, choosing an Authorization Type of "Basic Auth," and inputting the credentials.
Clicking the Code icon (</>) and selecting "Ruby - Net::HTTP" will then generate a code snippet, giving you the output above.
Postman took care of encoding the credentials, but this answer helped me to dynamically set these values. You also can likely omit the "cookie" key as part of the request.

Socksify and SSL for HTTPS request

For a project I need to make an HTTPS call through a Socks5 proxy so I can access the remote API of a customer.
The original code I'm using is this:
url = "/proxyValidate?service=#{migration_station_verification_url}&ticket=#{service_ticket}"
if Rails.env == 'development'
uri = URI.parse(soe_cas_url + url)
Net::HTTP.SOCKSProxy('127.0.0.1', 1080).start(uri.host, uri.port) do |http|
res = http.get(uri.path)
doc = Nokogiri::XML(res)
doc.remove_namespaces!
doc.xpath('//sessionId').first.content
end
else
http = Net::HTTP.new(soe_cas_url)
http.use_ssl = true
req = Net::HTTP::Get.new(url)
res = http.request(req)
doc = Nokogiri::XML(res)
doc.remove_namespaces!
doc.xpath('//sessionId').first.content
end
The goal is to use the proxy only in development environment. When I make the call using the proxy I receive the error:
wrong status line: "<!DOCTYPE
Googling this shows I need to set the use_ssl flag to true, which works outside the development cycle. When using the socksify there is no such mechanism. If I call this inside the do block, I receive an error that the session is already started. Calling it outside the block returns an exception that no such function exists.
Anyone can tell me how to run SSL/HTTPS through the socksify proxy?
EDIT:
I'm currently playing around with pure sockets to get what I want, using the following code:
socket = Socket::TCPSocket.open('127.0.0.1', 1080)
ssl_context = OpenSSL::SSL::SSLContext.new
ssl_context.verify_mode = OpenSSL::SSL::VERIFY_NONE
ssl_context.ca_file = File.join(Rails.root, 'ssl', 'cacert.pem')
ssl_socket = OpenSSL::SSL::SSLSocket.new(socket, ssl_context)
ssl_socket.sync_close = true
ssl_socket.connect
request = "GET #{url} HTTP/1.1\r\nAccept: */*\r\nHost: #{soe_cas_url}"
ssl_socket.puts(request)
ssl_socket.puts("")
response = ""
while (line = ssl_socket.gets)
response << line
end
This however fails at the ssl_socket.connect with the following error:
ssl_connect syscall returned=5 errno=0 state=sslv2/v3 read server hello a
I've tried using self-signed certificates, no certificates etc but nothing seems to work. I do know the endpoint relies on self-signed certificates on staging environmnet.

Check if a given username is taken on Facebook

How to check if a username is already in use on Facebook?
My solution was trying to access http://www.facebook.com/USER and check the http headers (200 = OK; 404 = NOT FOUND). I could use this code:
require 'open-uri'
require 'net/http'
def remote_file_exists?(url,httpcode)
url = URI.parse(url)
Net::HTTP.start(url.host, url.port) do |http|
return http.head(url.request_uri).code == httpcode
end
end
The problem is that Facebook always returns 302 (Found), then redirects to https://www.facebook.com/USER.
I can require net/https and create a new function:
def https_url_exists? (url,httpcode)
url = URI.parse(url)
net = Net::HTTP.new(url.host, url.port)
net.use_ssl = true
net.verify_mode = OpenSSL::SSL::VERIFY_NONE
net.start do |http|
return (http.head(url.request_uri).code == httpcode)
end
end
Now the problem is that some users use dots on their usernames. For example, username might be user.name. Facebook use redirections for this.
What's the best way to check if USERNAME exists on facebook? How to get USER.NAME if USERNAME redirects to it?
You can use https://graph.facebook.com/username. This will return a json response with info to see if it exists as well as enough information to identify it as a user or page.
Once you have a valid user you can get user name First,Last info using:
https://graph.facebook.com/{userId}?fields=first_name,last_name

How to implement cookie support in ruby net/http?

I'd like to add cookie support to a ruby class utilizing net/http to browse the web. Cookies have to be stored in a file to survive after the script has ended. Of course I can read the specs and write some kind of a handler, use some cookie.txt format and so on, but it seems to mean reinventing the wheel. Is there a better way to accomplish this task? Maybe some kind of a cooie jar class to take care of cookies?
The accepted answer will not work if your server returns and expects multiple cookies. This could happen, for example, if the server returns a set of FedAuth[n] cookies. If this affects you, you might want to look into using something along the lines of the following instead:
http = Net::HTTP.new('https://example.com', 443)
http.use_ssl = true
path1 = '/index.html'
path2 = '/index2.html'
# make a request to get the server's cookies
response = http.get(path)
if (response.code == '200')
all_cookies = response.get_fields('set-cookie')
cookies_array = Array.new
all_cookies.each { | cookie |
cookies_array.push(cookie.split('; ')[0])
}
cookies = cookies_array.join('; ')
# now make a request using the cookies
response = http.get(path2, { 'Cookie' => cookies })
end
Taken from DZone Snippets
http = Net::HTTP.new('profil.wp.pl', 443)
http.use_ssl = true
path = '/login.html'
# GET request -> so the host can set his cookies
resp, data = http.get(path, nil)
cookie = resp.response['set-cookie'].split('; ')[0]
# POST request -> logging in
data = 'serwis=wp.pl&url=profil.html&tryLogin=1&countTest=1&logowaniessl=1&login_username=blah&login_password=blah'
headers = {
'Cookie' => cookie,
'Referer' => 'http://profil.wp.pl/login.html',
'Content-Type' => 'application/x-www-form-urlencoded'
}
resp, data = http.post(path, data, headers)
# Output on the screen -> we should get either a 302 redirect (after a successful login) or an error page
puts 'Code = ' + resp.code
puts 'Message = ' + resp.message
resp.each {|key, val| puts key + ' = ' + val}
puts data
update
#To save the cookies, you can use PStore
cookies = PStore.new("cookies.pstore")
# Save the cookie
cookies.transaction do
cookies[:some_identifier] = cookie
end
# Retrieve the cookie back
cookies.transaction do
cookie = cookies[:some_identifier]
end
The accepted answer does not work. You need to access the internal representation of the response header where the multiple set-cookie values are stores separately and then remove everything after the first semicolon from these string and join them together. Here is code that works
r = http.get(path)
cookie = {'Cookie'=>r.to_hash['set-cookie'].collect{|ea|ea[/^.*?;/]}.join}
r = http.get(next_path,cookie)
Use http-cookie, which implements RFC-compliant parsing and rendering, plus a jar.
A crude example that happens to follow a redirect post-login:
require 'uri'
require 'net/http'
require 'http-cookie'
uri = URI('...')
jar = HTTP::CookieJar.new
Net::HTTP.start(uri.host, uri.port, use_ssl: uri.scheme == 'https') do |http|
req = Net::HTTP::Post.new uri
req.form_data = { ... }
res = http.request req
res.get_fields('Set-Cookie').each do |value|
jar.parse(value, req.uri)
end
fail unless res.code == '302'
req = Net::HTTP::Get.new(uri + res['Location'])
req['Cookie'] = HTTP::Cookie.cookie_value(jar.cookies(uri))
res = http.request req
end
Why do this? Because the answers above are incredibly insufficient and flat out don't work in many RFC-compliant scenarios (happened to me), so relying on the very lib implementing just what's needed is infinitely more robust if you want to handle more than one particular case.
I've used Curb and Mechanize for a similar project.
Just enable cookies support and save the cookies to a temp cookiejar...
If your using net/http or packages without cookie support built in, you will need to write your own cookie handling.
You can send receive cookies using headers.
You can store the header in any persistence framework. Whether it is some sort of database, or files.

Resources