I'm trying to upload a file with XHR request using PUT method with Sinatra.
My first idea was to upload the file and writing the stream directly into a MongoDB GridFS
#fs.open("test.iso", "w") do |f|
f.write request.body.read
end
It works, but, it loads the entire file into the RAM and it write it into the MongoDB GridFS.
I'd like to avoid this behavior by writing it continuously to the GridFS (stream the file, not loading it and put it in the GridFS) without loading the entire file into the RAM : because for huge files (like 1GB or more) it's clearly a bad practice (due to RAM consumption).
How can I do that ?
EDIT :
Can I have Sinatra / Rack not read the entire request body into memory? method is creating a TempFile, the thing is I want to only work with streams to optimize memory consumption on server-side.
Like a php://input would do in PHP.
EDIT 2 :
Here is how I'm currently handling it :
HTML/JS part
Ruby/Sinatra part
It looks like the streaming support in Sinatra is only in the opposite direction for long running GET requests out to the client.
If you do a POST multipart upload, Sinatra will write the data to a tempfile and provide you with the details in the params Hash.
require 'sinatra'
require 'fileutils'
post '/upload' do
tempfile = params['file'][:tempfile]
filename = params['file'][:filename]
FileUtils.mv(tempfile.path, "test.iso")
"posted"
end
While in the same sinatra directory:
$ echo "testtest" > /tmp/file_to_upload
$ curl --form "file=#/tmp/file_to_upload" http://localhost:4567/upload
posted
$ cat test.iso
testtest
Related
I am building a small Sinatra API that will run some calculations and store results on the server. The POST route requires a JSON payload. The JSON payload is a combination of floats, arrays of floats and base64 encoded data.
This is the route
post '/calc' do
rad_data = JSON.parse(request.body.read)
# Perform calculations
end
Everything works fine when the json is around 2.8MB, but when the size gets to 3.9MB I receive the error
2021-11-20 17:33:43 +0000 Rack app ("POST /calc" - (78.110.164.58)): #<RangeError: RangeError>
(It is also weird that 78.110.164.58 is not the ip address of the machine and i have no idea where it comes from.)
The structure of the payload is exactly the same. The larger one has simply longer arrays. Also, if I load the file directly on the server everything works, so it is not a problem with the json content.
Based on this answer I have added the key_space_limit to the beginning of the app, but it makes no difference.
require 'sinatra'
require 'sinatra/reloader'
require 'rack'
if Rack::Utils.respond_to?("key_space_limit=")
puts "Yes"
Rack::Utils.key_space_limit = 2**64
end
Just in case it helps, I send my request with Ruby rest-client
RestClient.post(uri_post, JSON.dump(rad_data))
In the Ruby-doc Net/HTTP there is a detailed example for streaming response bodies - it applies when you try to download a large file.
I am looking for an equivalent code snippet to upload a file via PUT. Spent quite a bit of time trying to make code work with no luck. I think I need to implement a particular interface and pass it the to request.body_stream
I need streaming because I want to alter the content of the file while it is being uploaded so I want to have access to the buffered chunks while the upload takes place. I would gladly use a library like http.rb or rest-client as long as I can use streaming.
Thanks in advance!
For reference following is the working non streamed version
uri = URI("http://localhost:1234/upload")
Net::HTTP.start(uri.host, uri.port) do |http|
request = Net::HTTP::Put.new uri
File.open("/home/files/somefile") do |f|
request.body = f.read()
end
# Ideally I would need to use **request.body_stream** instead of **body** to get streaming
http.request request do |response|
response.read_body do |result|
# display the operation result
puts result
end
end
end
I'm trying to make a POST request from the command line to my Flask app, and I want it to include an image. But I don't know how to include it with the command. I've only used strings as data successfully.
So, if my POST request looks like this:
curl -i -H "Content-Type: application/json" -X POST -d '{"username":"user1", "password":"password", "image":##What do I put here?##}' http://localhost:5000/my_app/api/users
I don't know what to put in that image part of the JSON. I'm tagging flask in this question because it might be a specific answer with regards to flask.
I would like to include an actual image here, and then on the Flask side of things, put the image in a folder of the app where all the uploads go, then save the path to the image in the database for later access. But, to do that, I need to know how to send an image in the first place. Any thoughts?
Seems you're mixing things up here.
From your example seems you want to upload an image in a JSON object. This is generally bad for 2 reasons:
Overhead: the image data should be encoded in printable characters, e.g. using base64. This creates a huge overhead on the data itself causing the JSON decoder to slow down.
Testing: You can't try this using curl on the commandline. You should make some command line utility to test the request.
HTTP knows about data uploads. So to mantain the JSON structure without the slowdown of above, you should upload your image as traditional data upload using a field, and another field for the JSON data structure.
Using curl this is achieved with the -F option.
curl -i -Ffiledata=#file.jpg -Fdata='{"username":"user1", "password":"password"}' http://localhost:5000/my_app/api/users
With the above command you are sending an HTTP POST with a file upload named filedata and a 'data` field containing your 'JSON' payload to parse in the receiving view handler.
You must use 2 HTTP fields because in HTTP upload you're using multipart encoding.
Behind the scenes
You're sending the image contents encoded in base64 but since the data is decoded by the application framework knowing exactly what to do (JSON parse must figure it out while parsing) it's a lot faster.
When I run the curl command
curl -v -H "Content-type: application/json" -X POST -d '{"name":"abc", "id":"12", "subject":"my subject"}' http://localhost:9292
to send a POST request with data to my Rack application, my code prints out {}. That is coming from puts req.POST() in the code below.
Why does it print out {} instead of the POST data? And how do I correctly access the POST data in my Rack application?
require 'json'
class Greeter
def call(env)
req = Rack::Request.new(env)
if req.post?
puts req.POST()
end
[200, {"Content-Type" => "application/json"}, [{x:"Hello World!"}.to_json]]
end
end
run Greeter.new
From reading the docs for POST, looks like it is giving you parsed data based on other content types. If you want to process "application/json", you probably need to
JSON.parse( req.body.read )
instead. To check this, try
puts req.body.read
where you currently have puts req.POST.
req.body is an I/O object, not a string. See the body documentation and view the source. You can see that this is in fact the same as mudasobwa's answer.
Note that other code in a Rack application may expect to read the same I/O, such as the param parsers in Sinatra or Rails. To ensure that they see the same data and not get an error, you may want to call req.body.rewind, possibly both before and after reading the request body in your code. However, if you are in such a situation, you might instead consider whether your framework has options to process JSON directly via some option on the controller or request content-type handler declaration etc - most likely there will be an option to handle this kind of request within the framework.
Try:
env['rack.input'].read
I found it in "How to receive a JSON object with Rack" and, though it still sounds weird to me, it likely works.
You can try:
req.params
Hope this can help you.
Edit: I'm using Ruby with Sinatra.
UPDATE: here is the code I'm using which doesn't work...
get '/' do
session[:time] = Time.now
z = Zlib::Deflate.new(6, 31)
z.deflate(File.read('public/Assets/Styles/build.css'))
z.flush
z.finish
z.close
erb :home
end
...I don't get any errors. But when I check the file via Firebug's Yslow plugin it tells me that file isn't GZIP'ed
I'm trying to understand how I GZIP web page content and static files like JavaScript and CSS using zlib?
I know I can pass a string of data to Zlib::Deflate.deflate but I'm using Sinatra with ERB files. So do I pass in a path to the ERB file and the Js/CSS files? Or can I pass in the directory where scripts/styles are stored? Would I pass in a path to the ERB file or the symbol that references the ERB file?
Unless you are writing your own HTTP server, your server needs to handle this. The client first has to let the server know that it accepts gzip content encoding, and then the server can deliver gzip content encoding.
Zlib::Deflate.deflate will not produce gzip-encoded data. It will only produce zlib-encoded data. You would need to use Zlib::Deflate.new with the windowBits argument equal to 31 to start a gzip stream.