How can I stop file writing ( upload form remote url ) when file size exceeds 500KB ?
I am using following code to upload a remote file
require 'open-uri'
open('temp/demo.doc', 'wb') do |file|
file << open('http://example.com/demo.doc').read
end
this code is working properly and I am able to get files in temp folder. Now I want if filesize exceeds 500KB then it should stop writing file. In other words I want only 500KB of file if it is more than 500KB
IO#read, takes a bytes argument, so you can specify the size of what you want to read from IO as below:
require 'open-uri'
open('temp/demo.doc', 'wb') do |file|
file << open('http://example.com/demo.doc').read(500000)
end
you can also play with things like file.stat.size but given you are piping directly to file you would have to do more things to get this to work.
Related
I need to download a large zipped file, unzip it and modify each string before I save them to array.
I prefer to read downloaded zipped file line(entry) at a time, and manipulate each line(entry) as they load, rather then load the whole file in the memory.
I experimented with many IO methods of opening a file this way, but I struggle to pass a line(entry) to Zip::InputStream object. This is what I have:
require 'tempfile'
require 'zip'
require 'open-uri'
f = open(FILE_URL) #FILE_URL contains download path to .zip file
Zip::InputStream.open(f) do |io| #io is a String
while (io.get_next_entry)
io.each do |line|
# manipulate the line and push it to an array
end
end
end
if I use open(FILE_URL).each do |zip_entry|, I cannot figure out how to pass zip_entry to Zip::InputStream. Simply Zip::InputStream.open(zip_entry) does not work...
is this scenario possible, or do I have to have content of zipped file downloaded in to Tempfile completely? Any pointers so solve will be helpful
I'm writing a small application that takes data from an uploaded csv file, parses each line and then rearranges the data in a new csv file, which is then downloaded.
Here is the code that creates the new csv file and downloads it:
CSV.open("template.csv", "w") do |csv|
#formatted_lines.each do |line|
csv << line
end
# Download CSV
send_file("template.csv", :disposition => 'attachment', :filename => File.basename("template.csv"))
end
I end up with a csv file with 270 lines, even though the #formatted_lines array has 280 lines/arrays in it. There is nothing wrong with the data in the original csv file that would cause an error when it is parsed. Why would it cut off the last 10 lines?
You're not actually closing the file before trying to send it, it's possible that the last ten lines are just buffered and haven't been written to disk. Try doing the send_file after the CSV.open block.
I'm working on a file upload feature in a Sinatra app. It's small and simple and was done just using Ruby's File class and saving a temporary file to a directory by hand. I'm trying to implement the same functionality using Tempfile.
I've got the upload working, but now when I click a link to download the file, the filename is just a number. It downloads and reads the file correctly but it doesn't retain the filename or type of file. Before I made my changes the file would open up in the browser by redirecting to the newly uploaded file's endpoint on the server. I'd like to get that functionality back.
My code is as follows:
post "/positions/:id/attachment" do
html_settings
new_data = post_data
if params[:file_attachment][:file].present?
file = params[:file_attachment][:file]
# file looks like this when uploaded:
#{:filename=>"Screen Shot 2013-11-26 at 4.36.13 PM.png", :type=>"image/png", :name=>"file_attachment[file]", :tempfile=>#<File:/var/folders/85/0kp_g81s1ws16zths3s8d9p80000gn/T/RackMultipart20131127-2757-1kdficq>, :head=>"Content-Disposition: form-data; name=\"file_attachment[file]\"; filename=\"Screen Shot 2013-11-26 at 4.36.13 PM.png\"\r\nContent-Type: image/png\r\n"}
# Tempfile object
temp_file = Tempfile.new(file[:filename], 'uploads') # Create tempfile, save to uploads folder
begin
write_tempfile(file, temp_file)
new_data['file_attachment']['file'] = temp_file
new_data['multipart'] = true
# At this point, the new_data hash looks the same except for a small difference in the path name
# Before tempfile - {"file_attachment"=> {"display_name"=>"test","file"=>#<File:uploads/Screen Shot 2013-11-26 at 11.35.36 AM.png>}, "id"=>"1"}
# With tempfile - {"file_attachment"=> {"display_name"=>"test", "file"=>#<File:/path/to/uploads/Screen Shot 2013-11-26 at 4.36.13 PM.png20131127-2757-eb6w6r>}, "id"=>"1", "multipart"=>true}
response = api_post(attachment_upload_endpoint(params[:id]), new_data)
ensure
delete_tempfile(temp_file)
response
end
end
end
Helper methods:
def write_tempfile(file, temp)
file[:tempfile].rewind # Rewind before reading
temp.write(file[:tempfile].read) # Write to the temp file
temp.rewind # Rewind in order to be read
end
def delete_tempfile(temp_file)
#close! calls #close AND #unlink. #unlink deletes the file
temp_file.close!
end
After the file is uploaded there is a link to https://myserver.com/positions/1/file_attachments/46
Does anyone understand why now, when I click on that link, it downloads the file with the filename 46 and not in the browser anymore?
I also get this notification in the console:
Resource interpreted as Document but transferred with MIME type binary/octet-stream
Thanks.
I was able to get it working with some extra parsing for the extension:
ext = file[:filename].split('.').last
temp_file = Tempfile.new([file[:filename], ".#{ext}"])
I'm using the example code from this page:
http://www.wooptoot.com/file-upload-with-sinatra
When I try to upload an image file (png or jpg), it uploads successfully and I can see the file in the proper directory, but it gets corrupted in the process. I cannot open the image. Doing a diff with the original files, I see several newlines that are missing in the uploaded version.
I'm running Ruby 1.9.3p392 on Windows.
Edit:
I tried a test outside the context of Sinatra
File.open('57-new.jpg', "wb") do |f|
f.write(File.open('57.jpg', 'rb').read)
end
That works. The only difference is the addition of the binary flags. When using Sinatra I can set the binary flag on the write operation, but I'm not sure how I can set it on the read since I seem to be passed a file object by the request.
File.open('uploads/' + params['myfile'][:filename], "wb") do |f|
f.write(params['myfile'][:tempfile].read)
end
Okay, so it looks like all I needed was the binary flag when opening the new file.
File.open('uploads/' + params['myfile'][:filename], "wb") do |f|
f.write(params['myfile'][:tempfile].read)
end
I am creating a Zip file containing both text files and image files. The code works as expected when running on MacOS, but it fails when running on Windows because the image file contents are not read correctly.
The snippet below always reads PNG image files as '‰PNG', adding a 5 bytes file in the Zip for each PNG image.
Is it an issue regarding Windows environment?
zip_fs.file.open(destination, 'w') do |f|
f.write File.read(file_name)
end
from Why are binary files corrupted when zipping them?
io.get_output_stream(zip_file_path) do |out|
out.write File.binread(disk_file_path)
end
You need to tell Ruby to read/write the files in binary mode. Here are some variations on a theme:
zip_fs.file.open(destination, 'wb') do |f|
File.open(file_name, 'rb') do |fi|
f.write fi.read
end
end
zip_fs.file.open(destination, 'wb') do |f|
f.write File.read(file_name, 'mode' => 'rb')
end
zip_fs.file.open(destination, 'wb') do |f|
f.write File.readbin(file_name)
end
A potential problem with the code is the input file is being slurped, which, if it's larger than the available space, would be a bad thing. It'd be better to read the input file in blocks. This is untested but should work:
BLOCK_SIZE = 1024 * 1024
zip_fs.file.open(destination, 'wb') do |f|
File.open(file_name, 'rb') do |fi|
while (block_in = fi.read(BLOCK_SIZE)) do
f.write block_in
end
end
end
The file that was opened will never be closed. Use File.binread(file_name)
My initial code was written to show that binary mode needed to be used, and used open because it's "more traditional", but forgot to use the block mode. I modified my sample code to fix that problem.
However, the file would be closed implicitly by Ruby as the interpreter shuts down when the script ends, as part of housekeeping that occurs. However, it's better to explicitly close the file. If the OP is using RubyZip like I think, that will automatically happen if a block is passed to open. Otherwise, read and readbin will both read to EOF and close the file. Code using those methods needs to be sensitive to the need to read blocks if the input file is an unknown size or larger than available buffer space.
I had a similar problem when I was reading Lib files. Here's my solution:
File.open(path + '\Wall.Lib')
Where the path corresponded to a javascript file that inputs filenames.