I need to set Content-type = "application/pdf" to a parameter of a request with the method "set_form" from net/http class but it always shows me Content-Type = "application/octet-stream".
I already checked out the documentation for set_form but I don't know how to properly set the Content-Type.
http = Net::HTTP.new(uri.host, uri.port)
request = Net::HTTP::Post.new("#{path}")
request.set_form([
["parameter1","#{parameter1}"],
["parameter2","#{parameter2}"],
["attachment", File.new("/home/name.pdf", "rb")]
], 'multipart/form-data')
response = http.request(request)
I have tried this code to set opt hash like documentation but I have received the same response:
uri = URI.parse("#{host}")
http = Net::HTTP.new(uri.host, uri.port)
request = Net::HTTP::Post.new("#{path}")
request.set_form([
["parameter1","#{parameter1}"],
["parameter2","#{parameter2}"],
["attachment", File.new("/home/name.pdf", "rb"), { "Content-Type" => "application/pdf" }]
], 'multipart/form-data')
response = http.request(request)
The actual output is still Content-Type: application/octet-stream but I need:
... Content-Disposition: form-data; name=\"attachment\"; filename=\"name.pdf\"\r\nContent-Type: application/pdf\r\n\r\n%PDF-1.5\r%\xE2\xE ...
I have found this (https://dradisframework.com/blog/2017/04/dradis-attachments-api-using-ruby/), not sure if it helps.
This code builds individual body parts for the request.
require 'net/http'
uri = URI('http://dradis.ip/pro/api/nodes/18/attachments')
Net::HTTP.start(uri.host, uri.port) do |http|
# Read attachments associated to a specific node:
get_request = Net::HTTP::Get.new uri
get_request['Authorization'] = 'Token token="iOEFCQDR-miTHNTjiBxObjWC"'
get_request['Dradis-Project-Id'] = '8'
get_response = http.request(get_request)
puts get_response.body
# Attach some other files to that node:
BOUNDARY = "AaB03x"
file1 = '/your/local/path/image1.png'
file2 = '/your/local/path/image2.png'
post_body = []
post_body << "--#{BOUNDARY}\r\n"
post_body << "Content-Disposition: form-data; name=\"files[]\"; filename=\"#{File.basename(file1)}\"\r\n"
post_body << "Content-Type: image/png\r\n"
post_body << "\r\n"
post_body << File.read(file1)
post_body << "\r\n--#{BOUNDARY}\r\n"
post_body << "Content-Disposition: form-data; name=\"files[]\"; filename=\"#{File.basename(file2)}\"\r\n"
post_body << "Content-Type: image/png\r\n"
post_body << "\r\n"
post_body << File.read(file2)
post_body << "\r\n--#{BOUNDARY}--\r\n"
post_request = Net::HTTP::Post.new uri
post_request['Authorization'] = 'Token token="iOEFCQDR-miTHNTjiBxObjWC"'
post_request['Dradis-Project-Id'] = '8'
post_request.body = post_body.join
post_request["Content-Type"] = "multipart/form-data, boundary=#{BOUNDARY}"
post_response = http.request(post_request)
puts post_response.body
end
For those who are still looking for how to do it with Net::HTTP, you should be doing:
request.set_form(
[
['attachment', File.open('myfile.pdf'), content_type: 'application/pdf', filename: 'my-better-filename.pdf']
],
'multipart/form-data'
)
Options content_type and filename are optional. By default, content type is application/octet-stream and filename is the name of the file you are opening.
In my opinion this is not possible for multipart requests as you can see in the code for the set_form method, when a different content type is set it should raise an error.
# File net/http/header.rb, line 452
def set_form(params, enctype='application/x-www-form-urlencoded', formopt={})
#body_data = params
#body = nil
#body_stream = nil
#form_option = formopt
case enctype
when /\Aapplication\/x-www-form-urlencoded\z/i,
/\Amultipart\/form-data\z/i
self.content_type = enctype
else
raise ArgumentError, "invalid enctype: #{enctype}"
end
end
There are surely workarounds as in my other answer or you can simply use different http clients like HTTParty which support a different content type for multipart requests, start reading the example here and continue in the source code on Github. I haven't found any better documentation.
I want decode body what do i get of apigateway request to a lambda in ruby 2.5 and get hash with params.
This one comes with 'Content-type: "form-data"'
----------------------------106174163609970174188191\r\nContent-Disposition: form-data;
name=\"packageName\"\r\n\r\ncom.orbemnetworks.americatv\r\n----------------------------106174163609970174188191\r\nContent-Disposition: form-data;
name=\"subscriptionId\"\r\n\r\nsuscripcion_mensual\r\n----------------------------106174163609970174188191\r\nContent-Disposition: form-data;
name=\"token\"\r\n\r\ndaomloifokamoigolhhapeap.AO-J1OxQBm7mW7z6pPPfwE9bJ23n5oeQwOBf46gKFSjjdHmkRHJDIR-RCbXtuDD6L_C_KfYEVETbvpu0M72t9-FXTNdHbB67dfJqzGMpx197Pb_IN8kLyU6ng-b8Yvo-1r-xJzcC04Rg\r\n----------------------------106174163609970174188191\r\nContent-Disposition: form-data;
name=\"idusuario\"\r\n\r\n1577633\r\n----------------------------106174163609970174188191\r\nContent-Disposition: form-data;
name=\"os\"\r\n\r\nA\r\n----------------------------106174163609970174188191--\r\n
My code was inspired (read: mostly copied from) Faraday Gem at: Faraday Gem Github
You will need to use gem 'multipart-parser', '~> 0.1.1' and
create a util.rb file:
require 'multipart_parser/reader'
module Util
extend self
# parse a multipart MIME message, returning a hash of any multipart errors
def parse_multipart(event)
body = event["body"]
boundary = MultipartParser::Reader::extract_boundary_value(event.dig("headers","Content-Type"))
reader = MultipartParser::Reader.new(boundary)
result = { errors: [], parts: [] }
def result.part(name)
hash = self[:parts].detect { |h| h[:part].name == name }
[hash[:part], hash[:body].join]
end
reader.on_part do |part|
result[:parts] << thispart = {
part: part,
body: []
}
part.on_data do |chunk|
thispart[:body] << chunk
end
end
reader.on_error do |msg|
result[:errors] << msg
end
reader.write(body)
result
end
def parse_multipart_hash(event)
event_parts = parse_multipart(event)[:parts]
event_parts.inject({}) do |hash, part|
if part[:part].filename.nil?
hash.merge({part[:part].name => part[:body]})
else
hash.merge({part[:part].name => {mime: part[:part].mime, filename: part[:part].filename, data: part[:body].join}})
end
end
end
end
Then you can use it like so:
require_relative './util'
def lambda_handler(event:, context:)
parsed_event_body = Util.parse_multipart_hash(event)
end
Not sure how this will handle images, but it works fine for CSV. You can modify parse_multipart to just return the hash, but I left it because it seem to provide a more generic structure to the data.
You can use the first method like this:
parts = Util.parse_multipart_hash(event)
meta, fieldname = parts.part("fieldname")
Rack's multipart parser can parse multipart form data
require "rack"
content_type = "multipart/form-data; boundary=xYzZY"
sanitized_multipart_form_data = multipart_form_data.gsub(/\r?\n/, "\r\n")
io = StringIO.new(sanitized_multipart_form_data)
tempfile = Rack::Multipart::Parser::TEMPFILE_FACTORY
bufsize = Rack::Multipart::Parser::BUFSIZE
query_parser = Rack::Utils.default_query_parser
result = Rack::Multipart::Parser.parse(io, sanitized_multipart_form_data.length, content_type, tempfile, bufsize, query_parser)
params = result.params
In case it helps, I've published the snippet above as a ruby gem
https://rubygems.org/gems/multipart_form_data_parser
https://github.com/nisanth074/multipart_form_data_parser
I am trying to fetch data from a nested json. Not able to understand the issue over here. Please ignore the fields that I am passing to ChildArticle class. I can sort that out.
URL for JSON - http://api.nytimes.com/svc/mostpopular/v2/mostshared/all-sections/email/30.json?api-key=31fa4521f6572a0c05ad6822ae109b72:2:72729901
Below is my code:
url = 'http://api.nytimes.com'
#Define the HTTP object
uri = URI.parse(url)
http = Net::HTTP.new(uri.host, uri.port)
#If the api being scraped uses https, then set use_ssl to true.
http.use_ssl = false
#Define the request_url
#Make a GET request to the given url
request = '/svc/mostpopular/v2/mostshared/all-sections/email/30.json?api-key=31fa4521f6572a0c05ad6822ae109b72:2:72729901'
response = http.send_request('GET', request)
#Parse the response body
forecast = JSON.parse(response.body)
forecast["results"]["result"].each do |item|
date = Date.parse(item["published_date"].to_s)
if (#start <= date) && (#end >= date)
article = News::ChildArticle.new(author: item["author"], title: item["title"], summary: item["abstract"],
images: item["images"],source: item["url"], date: item["published_date"],
guid: item["guid"], link: item["link"], section: item["section"],
item_type: item["item_type"], updated_date: item["updated_date"],
created_date: item["created_date"],
material_type_facet: item["material_type_facet"])
#articles.concat([article])
end
end
I get below error -
[]': no implicit conversion of String into Integer (TypeError) atforecast["results"]["result"].each do |item|`
Looks like forecast['results'] is simply an array, not a hash.
Take a look at this slightly modified script. Give it a run in your terminal, and check out its output.
require 'net/http'
require 'JSON'
url = 'http://api.nytimes.com'
uri = URI.parse(url)
http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = false
request = '/svc/mostpopular/v2/mostshared/all-sections/email/30.json?api-key=31fa4521f6572a0c05ad6822ae109b72:2:72729901'
response = http.send_request('GET', request)
forecast = JSON.parse(response.body)
forecast["results"].each.with_index do |item, i|
puts "Item #{i}:"
puts '--'
item.each do |k, v|
puts "#{k}: #{v}"
end
puts '----'
end
Also, you may want to inspect the JSON structure of the API return from that URL. If you go to that URL, open your JavaScript console, and paste in
JSON.parse(document.body.textContent)
you can inspect the JSON structure very easily.
Another option would be downloading the response to a JSON file, and inspecting it in your editor. You'll need a JSON prettifier though.
File.open('response.json', 'w') do |f|
f.write(response.body)
end
I have to send two XML documents in my request to the UPS API (here's my original question What is the root of this XML document? )
How would I do this?
def make_initial_request
uri = URI.parse(UPS_API['confirm_url'])
https = Net::HTTP.new(uri.host, uri.port)
https.use_ssl = true
headers = {'Content-Type' => 'text/xml'}
request = Net::HTTP::Post.new(uri.path, headers)
request.body = xml_for_initial_request #<-- how do i split this into two documents?
#request.body = second_xml_document #<-- i want something like that. could i just use << ?
begin
response = https.request(request)
rescue
return nil
end
puts "response: #{response.code} #{response.message}: #{response.body}"
return nil if response.body.include?("Error")
end
You should use MIME Multipart messages if the API support them (ruby gem).
Otherwise just try to concatenate files' contents request.body = "#{xml_for_initial_request}\n#{second_xml_document}"
I want to do an HTTP POST that looks like an HMTL form posted from a browser. Specifically, post some text fields and a file field.
Posting text fields is straightforward, there's an example right there in the net/http rdocs, but I can't figure out how to post a file along with it.
Net::HTTP doesn't look like the best idea. curb is looking good.
I like RestClient. It encapsulates net/http with cool features like multipart form data:
require 'rest_client'
RestClient.post('http://localhost:3000/foo',
:name_of_file_param => File.new('/path/to/file'))
It also supports streaming.
gem install rest-client will get you started.
Another one using only standard libraries:
uri = URI('https://some.end.point/some/path')
request = Net::HTTP::Post.new(uri)
request['Authorization'] = 'If you need some headers'
form_data = [['photos', photo.tempfile]] # or File.open() in case of local file
request.set_form form_data, 'multipart/form-data'
response = Net::HTTP.start(uri.hostname, uri.port, use_ssl: true) do |http| # pay attention to use_ssl if you need it
http.request(request)
end
Tried a lot of approaches but only this was worked for me.
I can't say enough good things about Nick Sieger's multipart-post library.
It adds support for multipart posting directly to Net::HTTP, removing your need to manually worry about boundaries or big libraries that may have different goals than your own.
Here is a little example on how to use it from the README:
require 'net/http/post/multipart'
url = URI.parse('http://www.example.com/upload')
File.open("./image.jpg") do |jpg|
req = Net::HTTP::Post::Multipart.new url.path,
"file" => UploadIO.new(jpg, "image/jpeg", "image.jpg")
res = Net::HTTP.start(url.host, url.port) do |http|
http.request(req)
end
end
You can check out the library here:
http://github.com/nicksieger/multipart-post
or install it with:
$ sudo gem install multipart-post
If you're connecting via SSL you need to start the connection like this:
n = Net::HTTP.new(url.host, url.port)
n.use_ssl = true
# for debugging dev server
#n.verify_mode = OpenSSL::SSL::VERIFY_NONE
res = n.start do |http|
curb looks like a great solution, but in case it doesn't meet your needs, you can do it with Net::HTTP. A multipart form post is just a carefully-formatted string with some extra headers. It seems like every Ruby programmer who needs to do multipart posts ends up writing their own little library for it, which makes me wonder why this functionality isn't built-in. Maybe it is... Anyway, for your reading pleasure, I'll go ahead and give my solution here. This code is based off of examples I found on a couple of blogs, but I regret that I can't find the links anymore. So I guess I just have to take all the credit for myself...
The module I wrote for this contains one public class, for generating the form data and headers out of a hash of String and File objects. So for example, if you wanted to post a form with a string parameter named "title" and a file parameter named "document", you would do the following:
#prepare the query
data, headers = Multipart::Post.prepare_query("title" => my_string, "document" => my_file)
Then you just do a normal POST with Net::HTTP:
http = Net::HTTP.new(upload_uri.host, upload_uri.port)
res = http.start {|con| con.post(upload_uri.path, data, headers) }
Or however else you want to do the POST. The point is that Multipart returns the data and headers that you need to send. And that's it! Simple, right? Here's the code for the Multipart module (you need the mime-types gem):
# Takes a hash of string and file parameters and returns a string of text
# formatted to be sent as a multipart form post.
#
# Author:: Cody Brimhall <mailto:brimhall#somuchwit.com>
# Created:: 22 Feb 2008
# License:: Distributed under the terms of the WTFPL (http://www.wtfpl.net/txt/copying/)
require 'rubygems'
require 'mime/types'
require 'cgi'
module Multipart
VERSION = "1.0.0"
# Formats a given hash as a multipart form post
# If a hash value responds to :string or :read messages, then it is
# interpreted as a file and processed accordingly; otherwise, it is assumed
# to be a string
class Post
# We have to pretend we're a web browser...
USERAGENT = "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en-us) AppleWebKit/523.10.6 (KHTML, like Gecko) Version/3.0.4 Safari/523.10.6"
BOUNDARY = "0123456789ABLEWASIEREISAWELBA9876543210"
CONTENT_TYPE = "multipart/form-data; boundary=#{ BOUNDARY }"
HEADER = { "Content-Type" => CONTENT_TYPE, "User-Agent" => USERAGENT }
def self.prepare_query(params)
fp = []
params.each do |k, v|
# Are we trying to make a file parameter?
if v.respond_to?(:path) and v.respond_to?(:read) then
fp.push(FileParam.new(k, v.path, v.read))
# We must be trying to make a regular parameter
else
fp.push(StringParam.new(k, v))
end
end
# Assemble the request body using the special multipart format
query = fp.collect {|p| "--" + BOUNDARY + "\r\n" + p.to_multipart }.join("") + "--" + BOUNDARY + "--"
return query, HEADER
end
end
private
# Formats a basic string key/value pair for inclusion with a multipart post
class StringParam
attr_accessor :k, :v
def initialize(k, v)
#k = k
#v = v
end
def to_multipart
return "Content-Disposition: form-data; name=\"#{CGI::escape(k)}\"\r\n\r\n#{v}\r\n"
end
end
# Formats the contents of a file or string for inclusion with a multipart
# form post
class FileParam
attr_accessor :k, :filename, :content
def initialize(k, filename, content)
#k = k
#filename = filename
#content = content
end
def to_multipart
# If we can tell the possible mime-type from the filename, use the
# first in the list; otherwise, use "application/octet-stream"
mime_type = MIME::Types.type_for(filename)[0] || MIME::Types["application/octet-stream"][0]
return "Content-Disposition: form-data; name=\"#{CGI::escape(k)}\"; filename=\"#{ filename }\"\r\n" +
"Content-Type: #{ mime_type.simplified }\r\n\r\n#{ content }\r\n"
end
end
end
Here is my solution after trying other ones available on this post, I'm using it to upload photo on TwitPic:
def upload(photo)
`curl -F media=##{photo.path} -F username=#{#username} -F password=#{#password} -F message='#{photo.title}' http://twitpic.com/api/uploadAndPost`
end
Fast forward to 2017, ruby stdlib net/http has this built-in since 1.9.3
Net::HTTPRequest#set_form): Added to support both application/x-www-form-urlencoded and multipart/form-data.
https://ruby-doc.org/stdlib-2.3.1/libdoc/net/http/rdoc/Net/HTTPHeader.html#method-i-set_form
We can even use IO which does not support :size to stream the form data.
Hoping that this answer can really help someone :)
P.S. I only tested this in ruby 2.3.1
Ok, here's a simple example using curb.
require 'yaml'
require 'curb'
# prepare post data
post_data = fields_hash.map { |k, v| Curl::PostField.content(k, v.to_s) }
post_data << Curl::PostField.file('file', '/path/to/file'),
# post
c = Curl::Easy.new('http://localhost:3000/foo')
c.multipart_form_post = true
c.http_post(post_data)
# print response
y [c.response_code, c.body_str]
restclient did not work for me until I overrode create_file_field in RestClient::Payload::Multipart.
It was creating a 'Content-Disposition: multipart/form-data' in each part where it should be ‘Content-Disposition: form-data’.
http://www.ietf.org/rfc/rfc2388.txt
My fork is here if you need it: git#github.com:kcrawford/rest-client.git
Well the solution with NetHttp has a drawback that is when posting big files it loads the whole file into memory first.
After playing a bit with it I came up with the following solution:
class Multipart
def initialize( file_names )
#file_names = file_names
end
def post( to_url )
boundary = '----RubyMultipartClient' + rand(1000000).to_s + 'ZZZZZ'
parts = []
streams = []
#file_names.each do |param_name, filepath|
pos = filepath.rindex('/')
filename = filepath[pos + 1, filepath.length - pos]
parts << StringPart.new ( "--" + boundary + "\r\n" +
"Content-Disposition: form-data; name=\"" + param_name.to_s + "\"; filename=\"" + filename + "\"\r\n" +
"Content-Type: video/x-msvideo\r\n\r\n")
stream = File.open(filepath, "rb")
streams << stream
parts << StreamPart.new (stream, File.size(filepath))
end
parts << StringPart.new ( "\r\n--" + boundary + "--\r\n" )
post_stream = MultipartStream.new( parts )
url = URI.parse( to_url )
req = Net::HTTP::Post.new(url.path)
req.content_length = post_stream.size
req.content_type = 'multipart/form-data; boundary=' + boundary
req.body_stream = post_stream
res = Net::HTTP.new(url.host, url.port).start {|http| http.request(req) }
streams.each do |stream|
stream.close();
end
res
end
end
class StreamPart
def initialize( stream, size )
#stream, #size = stream, size
end
def size
#size
end
def read ( offset, how_much )
#stream.read ( how_much )
end
end
class StringPart
def initialize ( str )
#str = str
end
def size
#str.length
end
def read ( offset, how_much )
#str[offset, how_much]
end
end
class MultipartStream
def initialize( parts )
#parts = parts
#part_no = 0;
#part_offset = 0;
end
def size
total = 0
#parts.each do |part|
total += part.size
end
total
end
def read ( how_much )
if #part_no >= #parts.size
return nil;
end
how_much_current_part = #parts[#part_no].size - #part_offset
how_much_current_part = if how_much_current_part > how_much
how_much
else
how_much_current_part
end
how_much_next_part = how_much - how_much_current_part
current_part = #parts[#part_no].read(#part_offset, how_much_current_part )
if how_much_next_part > 0
#part_no += 1
#part_offset = 0
next_part = read ( how_much_next_part )
current_part + if next_part
next_part
else
''
end
else
#part_offset += how_much_current_part
current_part
end
end
end
there's also nick sieger's multipart-post to add to the long list of possible solutions.
I had the same problem (need to post to jboss web server). Curb works fine for me, except that it caused ruby to crash (ruby 1.8.7 on ubuntu 8.10) when I use session variables in the code.
I dig into the rest-client docs, could not find indication of multipart support. I tried the rest-client examples above but jboss said the http post is not multipart.
The multipart-post gem works pretty well with Rails 4 Net::HTTP, no other special gem
def model_params
require_params = params.require(:model).permit(:param_one, :param_two, :param_three, :avatar)
require_params[:avatar] = model_params[:avatar].present? ? UploadIO.new(model_params[:avatar].tempfile, model_params[:avatar].content_type, model_params[:avatar].original_filename) : nil
require_params
end
require 'net/http/post/multipart'
url = URI.parse('http://www.example.com/upload')
Net::HTTP.start(url.host, url.port) do |http|
req = Net::HTTP::Post::Multipart.new(url, model_params)
key = "authorization_key"
req.add_field("Authorization", key) #add to Headers
http.use_ssl = (url.scheme == "https")
http.request(req)
end
https://github.com/Feuda/multipart-post/tree/patch-1
Using http.rb gem:
HTTP.post("https://here-you-go.com/upload",
form: {
file: HTTP::FormData::File.new(file_path)
})
Details
Haha, seems like doing this without a gem is a well guarded secret.
I used HTTParty gem:
HTTParty.post(
'http://localhost:3000/user',
body: {
name: 'Foo Bar',
email: 'example#email.com',
avatar: File.open('/full/path/to/avatar.jpg')
}
)
https://github.com/jnunemaker/httparty/blob/master/examples/multipart.rb
https://github.com/jnunemaker/httparty
gem install httparty