currently I am writing a program that needs to check tons of possible urls searching for any that actually exist. To be precise, I mean exist as in you can visit the url and there's actual content of some sort.. not string parsing to see if it's in url format.
The program generates a list of possible variants for a filename and then checks each one until it gets a url that actually exists, so most of the url remains the same. Examples would be,
https://www.test.com/folder1/FILE.png
https://www.test.com/folder1/File.png
https://www.test.com/folder1/file.png
https://www.test.com/folder1/file1.png
That said, my code currently works fine.. however it ends up taking about 2-4 secods per url check and I don't know of a way to speed it up. Is there any faster or better way to validate urls or am I just out of luck?
This is my function to validate urls:
require "net/http"
def url_exist? url_path
url = URI.parse(url_path)
req = Net::HTTP.new(url.host, url.port)
req.use_ssl = true
res = req.request_head(url.path)
if res.code == "200" || res.code == "403"
return true
end
end
Thank you for taking the time to read this and any help will be much appreciated.
Your code creates a new connection for each URL. It should be faster to send multiple requests over the same connection via HTTP keep-alive.
In Ruby, you can open such connection via Net::HTTP.start, e.g.:
require 'net/http'
class URLChecker
def initialize(base_url)
uri = URI(base_url)
Net::HTTP.start(uri.host, uri.port, use_ssl: uri.is_a?(URI::HTTPS)) do |http|
#http = http
yield self
end
end
def exist?(path)
res = #http.head(path)
res.code == '200' || res.code == '403'
end
end
URLChecker.new('https://stackoverflow.com') do |uc|
p uc.exist?('/questions/tagged/ruby') #=> true
p uc.exist?('/questions/tagged/python') #=> true
p uc.exist?('/questions/tagged/foobar') #=> false
end
Maybe I'm just blind but many post about passing headers in Net::HTTP follows the lines of
require 'net/http'
uri = URI("http://www.ruby-lang.org")
req = Net::HTTP::Get.new(uri)
req['some_header'] = "some_val"
res = Net::HTTP.start(uri.hostname, uri.port) {|http|
http.request(req)
}
puts res.body
(From Ruby - Send GET request with headers metaphori's answer)
And from the Net::HTTP docs (https://docs.ruby-lang.org/en/2.0.0/Net/HTTP.html)
uri = URI('http://example.com/cached_response')
file = File.stat 'cached_response'
req = Net::HTTP::Get.new(uri)
req['If-Modified-Since'] = file.mtime.rfc2822
res = Net::HTTP.start(uri.hostname, uri.port) {|http|
http.request(req)
}
open 'cached_response', 'w' do |io|
io.write res.body
end if res.is_a?(Net::HTTPSuccess)
But what is the advantage of doing the above when you can pass the headers via the following way?
options = {
'headers' => {
'Content-Type' => 'application/json'
}
}
request = Net::HTTP::Get.new('http://www.stackoverflow.com/', options['headers'])
This allows you to parameterize the headers and can allow for multiple headers very easily.
My main question is, what is the advantage of passing the headers in the creation of Net::HTTP::Get vs passing them after the creation of Net::HTTP::Get
Net::HTTPHeader already goes ahead and assigns the headers in the function
def initialize_http_header(initheader)
#header = {}
return unless initheader
initheader.each do |key, value|
warn "net/http: duplicated HTTP header: #{key}", uplevel: 1 if key?(key) and $VERBOSE
if value.nil?
warn "net/http: nil HTTP header: #{key}", uplevel: 1 if $VERBOSE
else
value = value.strip # raise error for invalid byte sequences
if value.count("\r\n") > 0
raise ArgumentError, 'header field value cannot include CR/LF'
end
#header[key.downcase] = [value]
end
end
end
So doing
request['some_header'] = "some_val" almost seems like code duplication.
There is no advantage for setting headers one way or another, at least not that I can think of. It comes down to your own preference. In fact, if you take a look at what happens when you supply headers while initializing a new Net::Http::Get, you will find that internally, Ruby simply sets the headers onto a #headers variable:
https://github.com/ruby/ruby/blob/c5eb24349a4535948514fe765c3ddb0628d81004/lib/net/http/header.rb#L25
And if you set the headers using request[name] = value, you can see that Net::Http does the exact same thing, but in a different method:
https://github.com/ruby/ruby/blob/c5eb24349a4535948514fe765c3ddb0628d81004/lib/net/http/header.rb#L46
So the resulting object has the same configuration no matter which way you decide to pass the request headers.
How can I send HTTP GET request with parameters via ruby?
I have tried a lot of examples but all of those failed.
I know this post is old but for the sake of those brought here by google, there is an easier way to encode your parameters in a URL safe manner. I'm not sure why I haven't seen this elsewhere as the method is documented on the Net::HTTP page. I have seen the method described by Arsen7 as the accepted answer on several other questions also.
Mentioned in the Net::HTTP documentation is URI.encode_www_form(params):
# Lets say we have a path and params that look like this:
path = "/search"
params = {q: => "answer"}
# Example 1: Replacing the #path_with_params method from Arsen7
def path_with_params(path, params)
encoded_params = URI.encode_www_form(params)
[path, encoded_params].join("?")
end
# Example 2: A shortcut for the entire example by Arsen7
uri = URI.parse("http://localhost.com" + path)
uri.query = URI.encode_www_form(params)
response = Net::HTTP.get_response(uri)
Which example you choose is very much dependent on your use case. In my current project I am using a method similar to the one recommended by Arsen7 along with the simpler #path_with_params method and without the block format.
# Simplified example implementation without response
# decoding or error handling.
require "net/http"
require "uri"
class Connection
VERB_MAP = {
:get => Net::HTTP::Get,
:post => Net::HTTP::Post,
:put => Net::HTTP::Put,
:delete => Net::HTTP::Delete
}
API_ENDPOINT = "http://dev.random.com"
attr_reader :http
def initialize(endpoint = API_ENDPOINT)
uri = URI.parse(endpoint)
#http = Net::HTTP.new(uri.host, uri.port)
end
def request(method, path, params)
case method
when :get
full_path = path_with_params(path, params)
request = VERB_MAP[method].new(full_path)
else
request = VERB_MAP[method].new(path)
request.set_form_data(params)
end
http.request(request)
end
private
def path_with_params(path, params)
encoded_params = URI.encode_www_form(params)
[path, encoded_params].join("?")
end
end
con = Connection.new
con.request(:post, "/account", {:email => "test#test.com"})
=> #<Net::HTTPCreated 201 Created readbody=true>
I assume that you understand the examples on the Net::HTTP documentation page but you do not know how to pass parameters to the GET request.
You just append the parameters to the requested address, in exactly the same way you type such address in the browser:
require 'net/http'
res = Net::HTTP.start('localhost', 3000) do |http|
http.get('/users?id=1')
end
puts res.body
If you need some generic way to build the parameters string from a hash, you may create a helper like this:
require 'cgi'
def path_with_params(page, params)
return page if params.empty?
page + "?" + params.map {|k,v| CGI.escape(k.to_s)+'='+CGI.escape(v.to_s) }.join("&")
end
path_with_params("/users", :id => 1, :name => "John&Sons")
# => "/users?name=John%26Sons&id=1"
Although the HTTP spec says that headers are case insensitive; Paypal, with their new adaptive payments API require their headers to be case-sensitive.
Using the paypal adaptive payments extension for ActiveMerchant (http://github.com/lamp/paypal_adaptive_gateway) it seems that although the headers are set in all caps, they are sent in mixed case.
Here is the code that sends the HTTP request:
headers = {
"X-PAYPAL-REQUEST-DATA-FORMAT" => "XML",
"X-PAYPAL-RESPONSE-DATA-FORMAT" => "JSON",
"X-PAYPAL-SECURITY-USERID" => #config[:login],
"X-PAYPAL-SECURITY-PASSWORD" => #config[:password],
"X-PAYPAL-SECURITY-SIGNATURE" => #config[:signature],
"X-PAYPAL-APPLICATION-ID" => #config[:appid]
}
build_url action
request = Net::HTTP::Post.new(#url.path)
request.body = #xml
headers.each_pair { |k,v| request[k] = v }
request.content_type = 'text/xml'
proxy = Net::HTTP::Proxy("127.0.0.1", "60723")
server = proxy.new(#url.host, 443)
server.use_ssl = true
server.start { |http| http.request(request) }.body
(i added the proxy line so i could see what was going on with Charles - http://www.charlesproxy.com/)
When I look at the request headers in charles, this is what i see:
X-Paypal-Application-Id ...
X-Paypal-Security-Password...
X-Paypal-Security-Signature ...
X-Paypal-Security-Userid ...
X-Paypal-Request-Data-Format XML
X-Paypal-Response-Data-Format JSON
Accept */*
Content-Type text/xml
Content-Length 522
Host svcs.sandbox.paypal.com
I verified that it is not Charles doing the case conversion by running a similar request using curl. In that test the case was preserved.
The RFC does specify that header keys are case-insensitive, so unfortunately you seem to have hit an annoying requirement with the PayPal API.
Net::HTTP is what is changing the case, although I'm surprised they're not all getting downcased:
# File net/http.rb, line 1160
def []=(key, val)
unless val
#header.delete key.downcase
return val
end
#header[key.downcase] = [val]
end
"Sets the header field corresponding to the case-insensitive key."
As the above is a simple class it could be monkey-patched. I will think further for a nicer solution.
Use following code to force case sensitive headers.
class CaseSensitivePost < Net::HTTP::Post
def initialize_http_header(headers)
#header = {}
headers.each{|k,v| #header[k.to_s] = [v] }
end
def [](name)
#header[name.to_s]
end
def []=(name, val)
if val
#header[name.to_s] = [val]
else
#header.delete(name.to_s)
end
end
def capitalize(name)
name
end
end
Usage example:
post = CaseSensitivePost.new(url, {myCasedHeader: '1'})
post.body = body
http = Net::HTTP.new(host, port)
http.request(post)
If you are still looking for an answer that works. Newer versions have introduced some changes to underlying capitalize method by using to_s. Fix is to make the to_s and to_str return the self so that the returned object is an instance of ImmutableKey instead of the base string class.
class ImmutableKey < String
def capitalize
self
end
def to_s
self
end
alias_method :to_str, :to_s
end
Ref: https://jatindhankhar.in/blog/custom-http-header-and-ruby-standard-library/
I got several issues with the code proposed by #kaplan-ilya because the Net::HTTP library tries to detect the post content-type, and the I ended up with 2 content-type and other fields repeated with different cases.
So the code below should ensure than once a particular case has been choosen, it will stick to the same.
class Post < Net::HTTP::Post
def initialize_http_header(headers)
#header = {}
headers.each { |k, v| #header[k.to_s] = [v] }
end
def [](name)
_k, val = header_insensitive_match name
val
end
def []=(name, val)
key, _val = header_insensitive_match name
key = name if key.nil?
if val
#header[key] = [val]
else
#header.delete(key)
end
end
def capitalize(name)
name
end
def header_insensitive_match(name)
#header.find { |key, _value| key.match Regexp.new(name.to_s, Regexp::IGNORECASE) }
end
end
I want to do an HTTP POST that looks like an HMTL form posted from a browser. Specifically, post some text fields and a file field.
Posting text fields is straightforward, there's an example right there in the net/http rdocs, but I can't figure out how to post a file along with it.
Net::HTTP doesn't look like the best idea. curb is looking good.
I like RestClient. It encapsulates net/http with cool features like multipart form data:
require 'rest_client'
RestClient.post('http://localhost:3000/foo',
:name_of_file_param => File.new('/path/to/file'))
It also supports streaming.
gem install rest-client will get you started.
Another one using only standard libraries:
uri = URI('https://some.end.point/some/path')
request = Net::HTTP::Post.new(uri)
request['Authorization'] = 'If you need some headers'
form_data = [['photos', photo.tempfile]] # or File.open() in case of local file
request.set_form form_data, 'multipart/form-data'
response = Net::HTTP.start(uri.hostname, uri.port, use_ssl: true) do |http| # pay attention to use_ssl if you need it
http.request(request)
end
Tried a lot of approaches but only this was worked for me.
I can't say enough good things about Nick Sieger's multipart-post library.
It adds support for multipart posting directly to Net::HTTP, removing your need to manually worry about boundaries or big libraries that may have different goals than your own.
Here is a little example on how to use it from the README:
require 'net/http/post/multipart'
url = URI.parse('http://www.example.com/upload')
File.open("./image.jpg") do |jpg|
req = Net::HTTP::Post::Multipart.new url.path,
"file" => UploadIO.new(jpg, "image/jpeg", "image.jpg")
res = Net::HTTP.start(url.host, url.port) do |http|
http.request(req)
end
end
You can check out the library here:
http://github.com/nicksieger/multipart-post
or install it with:
$ sudo gem install multipart-post
If you're connecting via SSL you need to start the connection like this:
n = Net::HTTP.new(url.host, url.port)
n.use_ssl = true
# for debugging dev server
#n.verify_mode = OpenSSL::SSL::VERIFY_NONE
res = n.start do |http|
curb looks like a great solution, but in case it doesn't meet your needs, you can do it with Net::HTTP. A multipart form post is just a carefully-formatted string with some extra headers. It seems like every Ruby programmer who needs to do multipart posts ends up writing their own little library for it, which makes me wonder why this functionality isn't built-in. Maybe it is... Anyway, for your reading pleasure, I'll go ahead and give my solution here. This code is based off of examples I found on a couple of blogs, but I regret that I can't find the links anymore. So I guess I just have to take all the credit for myself...
The module I wrote for this contains one public class, for generating the form data and headers out of a hash of String and File objects. So for example, if you wanted to post a form with a string parameter named "title" and a file parameter named "document", you would do the following:
#prepare the query
data, headers = Multipart::Post.prepare_query("title" => my_string, "document" => my_file)
Then you just do a normal POST with Net::HTTP:
http = Net::HTTP.new(upload_uri.host, upload_uri.port)
res = http.start {|con| con.post(upload_uri.path, data, headers) }
Or however else you want to do the POST. The point is that Multipart returns the data and headers that you need to send. And that's it! Simple, right? Here's the code for the Multipart module (you need the mime-types gem):
# Takes a hash of string and file parameters and returns a string of text
# formatted to be sent as a multipart form post.
#
# Author:: Cody Brimhall <mailto:brimhall#somuchwit.com>
# Created:: 22 Feb 2008
# License:: Distributed under the terms of the WTFPL (http://www.wtfpl.net/txt/copying/)
require 'rubygems'
require 'mime/types'
require 'cgi'
module Multipart
VERSION = "1.0.0"
# Formats a given hash as a multipart form post
# If a hash value responds to :string or :read messages, then it is
# interpreted as a file and processed accordingly; otherwise, it is assumed
# to be a string
class Post
# We have to pretend we're a web browser...
USERAGENT = "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en-us) AppleWebKit/523.10.6 (KHTML, like Gecko) Version/3.0.4 Safari/523.10.6"
BOUNDARY = "0123456789ABLEWASIEREISAWELBA9876543210"
CONTENT_TYPE = "multipart/form-data; boundary=#{ BOUNDARY }"
HEADER = { "Content-Type" => CONTENT_TYPE, "User-Agent" => USERAGENT }
def self.prepare_query(params)
fp = []
params.each do |k, v|
# Are we trying to make a file parameter?
if v.respond_to?(:path) and v.respond_to?(:read) then
fp.push(FileParam.new(k, v.path, v.read))
# We must be trying to make a regular parameter
else
fp.push(StringParam.new(k, v))
end
end
# Assemble the request body using the special multipart format
query = fp.collect {|p| "--" + BOUNDARY + "\r\n" + p.to_multipart }.join("") + "--" + BOUNDARY + "--"
return query, HEADER
end
end
private
# Formats a basic string key/value pair for inclusion with a multipart post
class StringParam
attr_accessor :k, :v
def initialize(k, v)
#k = k
#v = v
end
def to_multipart
return "Content-Disposition: form-data; name=\"#{CGI::escape(k)}\"\r\n\r\n#{v}\r\n"
end
end
# Formats the contents of a file or string for inclusion with a multipart
# form post
class FileParam
attr_accessor :k, :filename, :content
def initialize(k, filename, content)
#k = k
#filename = filename
#content = content
end
def to_multipart
# If we can tell the possible mime-type from the filename, use the
# first in the list; otherwise, use "application/octet-stream"
mime_type = MIME::Types.type_for(filename)[0] || MIME::Types["application/octet-stream"][0]
return "Content-Disposition: form-data; name=\"#{CGI::escape(k)}\"; filename=\"#{ filename }\"\r\n" +
"Content-Type: #{ mime_type.simplified }\r\n\r\n#{ content }\r\n"
end
end
end
Here is my solution after trying other ones available on this post, I'm using it to upload photo on TwitPic:
def upload(photo)
`curl -F media=##{photo.path} -F username=#{#username} -F password=#{#password} -F message='#{photo.title}' http://twitpic.com/api/uploadAndPost`
end
Fast forward to 2017, ruby stdlib net/http has this built-in since 1.9.3
Net::HTTPRequest#set_form): Added to support both application/x-www-form-urlencoded and multipart/form-data.
https://ruby-doc.org/stdlib-2.3.1/libdoc/net/http/rdoc/Net/HTTPHeader.html#method-i-set_form
We can even use IO which does not support :size to stream the form data.
Hoping that this answer can really help someone :)
P.S. I only tested this in ruby 2.3.1
Ok, here's a simple example using curb.
require 'yaml'
require 'curb'
# prepare post data
post_data = fields_hash.map { |k, v| Curl::PostField.content(k, v.to_s) }
post_data << Curl::PostField.file('file', '/path/to/file'),
# post
c = Curl::Easy.new('http://localhost:3000/foo')
c.multipart_form_post = true
c.http_post(post_data)
# print response
y [c.response_code, c.body_str]
restclient did not work for me until I overrode create_file_field in RestClient::Payload::Multipart.
It was creating a 'Content-Disposition: multipart/form-data' in each part where it should be ‘Content-Disposition: form-data’.
http://www.ietf.org/rfc/rfc2388.txt
My fork is here if you need it: git#github.com:kcrawford/rest-client.git
Well the solution with NetHttp has a drawback that is when posting big files it loads the whole file into memory first.
After playing a bit with it I came up with the following solution:
class Multipart
def initialize( file_names )
#file_names = file_names
end
def post( to_url )
boundary = '----RubyMultipartClient' + rand(1000000).to_s + 'ZZZZZ'
parts = []
streams = []
#file_names.each do |param_name, filepath|
pos = filepath.rindex('/')
filename = filepath[pos + 1, filepath.length - pos]
parts << StringPart.new ( "--" + boundary + "\r\n" +
"Content-Disposition: form-data; name=\"" + param_name.to_s + "\"; filename=\"" + filename + "\"\r\n" +
"Content-Type: video/x-msvideo\r\n\r\n")
stream = File.open(filepath, "rb")
streams << stream
parts << StreamPart.new (stream, File.size(filepath))
end
parts << StringPart.new ( "\r\n--" + boundary + "--\r\n" )
post_stream = MultipartStream.new( parts )
url = URI.parse( to_url )
req = Net::HTTP::Post.new(url.path)
req.content_length = post_stream.size
req.content_type = 'multipart/form-data; boundary=' + boundary
req.body_stream = post_stream
res = Net::HTTP.new(url.host, url.port).start {|http| http.request(req) }
streams.each do |stream|
stream.close();
end
res
end
end
class StreamPart
def initialize( stream, size )
#stream, #size = stream, size
end
def size
#size
end
def read ( offset, how_much )
#stream.read ( how_much )
end
end
class StringPart
def initialize ( str )
#str = str
end
def size
#str.length
end
def read ( offset, how_much )
#str[offset, how_much]
end
end
class MultipartStream
def initialize( parts )
#parts = parts
#part_no = 0;
#part_offset = 0;
end
def size
total = 0
#parts.each do |part|
total += part.size
end
total
end
def read ( how_much )
if #part_no >= #parts.size
return nil;
end
how_much_current_part = #parts[#part_no].size - #part_offset
how_much_current_part = if how_much_current_part > how_much
how_much
else
how_much_current_part
end
how_much_next_part = how_much - how_much_current_part
current_part = #parts[#part_no].read(#part_offset, how_much_current_part )
if how_much_next_part > 0
#part_no += 1
#part_offset = 0
next_part = read ( how_much_next_part )
current_part + if next_part
next_part
else
''
end
else
#part_offset += how_much_current_part
current_part
end
end
end
there's also nick sieger's multipart-post to add to the long list of possible solutions.
I had the same problem (need to post to jboss web server). Curb works fine for me, except that it caused ruby to crash (ruby 1.8.7 on ubuntu 8.10) when I use session variables in the code.
I dig into the rest-client docs, could not find indication of multipart support. I tried the rest-client examples above but jboss said the http post is not multipart.
The multipart-post gem works pretty well with Rails 4 Net::HTTP, no other special gem
def model_params
require_params = params.require(:model).permit(:param_one, :param_two, :param_three, :avatar)
require_params[:avatar] = model_params[:avatar].present? ? UploadIO.new(model_params[:avatar].tempfile, model_params[:avatar].content_type, model_params[:avatar].original_filename) : nil
require_params
end
require 'net/http/post/multipart'
url = URI.parse('http://www.example.com/upload')
Net::HTTP.start(url.host, url.port) do |http|
req = Net::HTTP::Post::Multipart.new(url, model_params)
key = "authorization_key"
req.add_field("Authorization", key) #add to Headers
http.use_ssl = (url.scheme == "https")
http.request(req)
end
https://github.com/Feuda/multipart-post/tree/patch-1
Using http.rb gem:
HTTP.post("https://here-you-go.com/upload",
form: {
file: HTTP::FormData::File.new(file_path)
})
Details
Haha, seems like doing this without a gem is a well guarded secret.
I used HTTParty gem:
HTTParty.post(
'http://localhost:3000/user',
body: {
name: 'Foo Bar',
email: 'example#email.com',
avatar: File.open('/full/path/to/avatar.jpg')
}
)
https://github.com/jnunemaker/httparty/blob/master/examples/multipart.rb
https://github.com/jnunemaker/httparty
gem install httparty