I'm using the example code from this page:
http://www.wooptoot.com/file-upload-with-sinatra
When I try to upload an image file (png or jpg), it uploads successfully and I can see the file in the proper directory, but it gets corrupted in the process. I cannot open the image. Doing a diff with the original files, I see several newlines that are missing in the uploaded version.
I'm running Ruby 1.9.3p392 on Windows.
Edit:
I tried a test outside the context of Sinatra
File.open('57-new.jpg', "wb") do |f|
f.write(File.open('57.jpg', 'rb').read)
end
That works. The only difference is the addition of the binary flags. When using Sinatra I can set the binary flag on the write operation, but I'm not sure how I can set it on the read since I seem to be passed a file object by the request.
File.open('uploads/' + params['myfile'][:filename], "wb") do |f|
f.write(params['myfile'][:tempfile].read)
end
Okay, so it looks like all I needed was the binary flag when opening the new file.
File.open('uploads/' + params['myfile'][:filename], "wb") do |f|
f.write(params['myfile'][:tempfile].read)
end
Related
I need to download a large zipped file, unzip it and modify each string before I save them to array.
I prefer to read downloaded zipped file line(entry) at a time, and manipulate each line(entry) as they load, rather then load the whole file in the memory.
I experimented with many IO methods of opening a file this way, but I struggle to pass a line(entry) to Zip::InputStream object. This is what I have:
require 'tempfile'
require 'zip'
require 'open-uri'
f = open(FILE_URL) #FILE_URL contains download path to .zip file
Zip::InputStream.open(f) do |io| #io is a String
while (io.get_next_entry)
io.each do |line|
# manipulate the line and push it to an array
end
end
end
if I use open(FILE_URL).each do |zip_entry|, I cannot figure out how to pass zip_entry to Zip::InputStream. Simply Zip::InputStream.open(zip_entry) does not work...
is this scenario possible, or do I have to have content of zipped file downloaded in to Tempfile completely? Any pointers so solve will be helpful
My test suite has a cucumber front end with a ruby backend, running the latest version of watir-webdriver and its dependencies atop the latest version of OSX. My cucumber environment is setup to execute in Firefox.
The export feature of our app creates a zip file but to test the import feature, I need the contents of the zip file.
My actual test needs to unpack that zip file and select the individual files in it for use in testing the import feature of our web application.
Can anyone point me to a reference that can help me figure out how to write that?
Based off my experience, you download this file the same way that a normal user might. So first off, you just click the download button or whatever and then can access the file wherever it is and check out its contents.
Assuming the downloads just go to your Downloads folder by default, there is some simple code you can use to select the most recently downloaded item:
fn = Dir.glob("~/Downloads/*.zip").max { |a,b| File.ctime(a) <=> File.ctime(b)}
Then just use the unzip shell command to unzip the file. No reason to add another gem into the mix when you can just use generic shell commands.
`unzip #{fn}`
Then, you'd use Dir.glob again to get the filenames of everything inside the unzipped files folder. Assuming the file was named "thing.zip", you do this:
files = Dir.glob("~/Downloads/thing/*")
If you want to files to be downloaded directly to your project folder, you can try this. This also prevents the popup from asking you if you really want to save the file which is handy. I think this still works but haven't used it in some time. The above stuff works for sure though.
profile = Selenium::WebDriver::Firefox::Profile.new
download_dir = Dir.pwd + "/test_downloads"
profile['browser.download.dir'] = download_dir
profile['browser.helperApps.neverAsk.saveToDisk'] = "application/zip"
b = Watir::Browser.new. :firefox, :profile => profile
I ended up adding the rubyzip gem at https://github.com/rubyzip/rubyzip
the solution is on that link but i modified mine a little bit. I added the following to my common.rb file. see below:
require 'Zip'
def unpack_zip
test_home='/Users/yournamegoeshere/SRC/watir_testing/project files'
sleep(5) #<--manually making time to hit the save download dialog
zip_file_paths = []
Find.find(test_home) do |path|
zip_file_paths << path if path =~ /.*\.zip$/
end
file_name=zip_file_paths[0]
Zip::File.open(file_name) do |zip_file|
# Handle entries one by one
zip_file.each do |entry|
# Extract to file/directory/symlink
puts "Extracting #{entry.name}"
entry.extract(test_home + "/" + entry.name)
# Read into memory
content = entry.get_input_stream.read
end
# Find specific entry
entry = zip_file.glob('*.csv').first
puts entry.get_input_stream.read
end
end
This solution works great!
I have a ruby script that downloads a remote ZIP file from a server using rubys opencommand. When I look into the downloaded content, it shows something like this:
PK\x03\x04\x14\x00\b\x00\b\x00\x9B\x84PG\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\n\x00\x10\x00foobar.txtUX\f\x00\x86\v!V\x85\v!V\xF6\x01\x14\x00K\xCB\xCFOJ,RH\x03S\\\x00PK\a\b\xC1\xC0\x1F\xE8\f\x00\x00\x00\x0E\x00\x00\x00PK\x01\x02\x15\x03\x14\x00\b\x00\b\x00\x9B\x84PG\xC1\xC0\x1F\xE8\f\x00\x00\x00\x0E\x00\x00\x00\n\x00\f\x00\x00\x00\x00\x00\x00\x00\x00#\xA4\x81\x00\x00\x00\x00foobar.txtUX\b\x00\x86\v!V\x85\v!VPK\x05\x06\x00\x00\x00\x00\x01\x00\x01\x00D\x00\x00\x00T\x00\x00\x00\x00\x00
I tried using the Rubyzip gem (https://github.com/rubyzip/rubyzip) along with its class Zip::ZipInputStream like this:
stream = open("http://localhost:3000/foobar.zip").read # this outputs the zip content from above
zip = Zip::ZipInputStream.new stream
Unfortunately, this throws an error:
Failure/Error: zip = Zip::ZipInputStream.new stream
ArgumentError:
string contains null byte
My questions are:
Is it possible, in general, to download a ZIP file and extract its content in-memory?
Is Rubyzip the right library for it?
If so, how can I extract the content?
I found the solution myself and then at stackoverflow :D (How to iterate through an in-memory zip file in Ruby)
input = HTTParty.get("http://example.com/somedata.zip").body
Zip::InputStream.open(StringIO.new(input)) do |io|
while entry = io.get_next_entry
puts entry.name
parse_zip_content io.read
end
end
Download your ZIP file, I'm using HTTParty for this (but you could also use ruby's open command (require 'open-uri').
Convert it into a StringIO stream using StringIO.new(input)
Iterate over every entry inside the ZIP archive using io.get_next_entry (it returns an instance of Entry)
With io.read you get the content, and with entry.name you get the filename.
Like I commented in https://stackoverflow.com/a/43303222/4196440, we can just use Zip::File.open_buffer:
require 'open-uri'
content = open('http://localhost:3000/foobar.zip')
Zip::File.open_buffer(content) do |zip|
zip.each do |entry|
puts entry.name
# Do whatever you want with the content files.
end
end
I am creating a Zip file containing both text files and image files. The code works as expected when running on MacOS, but it fails when running on Windows because the image file contents are not read correctly.
The snippet below always reads PNG image files as '‰PNG', adding a 5 bytes file in the Zip for each PNG image.
Is it an issue regarding Windows environment?
zip_fs.file.open(destination, 'w') do |f|
f.write File.read(file_name)
end
from Why are binary files corrupted when zipping them?
io.get_output_stream(zip_file_path) do |out|
out.write File.binread(disk_file_path)
end
You need to tell Ruby to read/write the files in binary mode. Here are some variations on a theme:
zip_fs.file.open(destination, 'wb') do |f|
File.open(file_name, 'rb') do |fi|
f.write fi.read
end
end
zip_fs.file.open(destination, 'wb') do |f|
f.write File.read(file_name, 'mode' => 'rb')
end
zip_fs.file.open(destination, 'wb') do |f|
f.write File.readbin(file_name)
end
A potential problem with the code is the input file is being slurped, which, if it's larger than the available space, would be a bad thing. It'd be better to read the input file in blocks. This is untested but should work:
BLOCK_SIZE = 1024 * 1024
zip_fs.file.open(destination, 'wb') do |f|
File.open(file_name, 'rb') do |fi|
while (block_in = fi.read(BLOCK_SIZE)) do
f.write block_in
end
end
end
The file that was opened will never be closed. Use File.binread(file_name)
My initial code was written to show that binary mode needed to be used, and used open because it's "more traditional", but forgot to use the block mode. I modified my sample code to fix that problem.
However, the file would be closed implicitly by Ruby as the interpreter shuts down when the script ends, as part of housekeeping that occurs. However, it's better to explicitly close the file. If the OP is using RubyZip like I think, that will automatically happen if a block is passed to open. Otherwise, read and readbin will both read to EOF and close the file. Code using those methods needs to be sensitive to the need to read blocks if the input file is an unknown size or larger than available buffer space.
I had a similar problem when I was reading Lib files. Here's my solution:
File.open(path + '\Wall.Lib')
Where the path corresponded to a javascript file that inputs filenames.
I have a Ruby script that produces a Latex document using an erb template. After the .tex file has been generated, I'd like to make a system call to compile the document with pdflatex. Here are the bones of the script:
class Book
# initialize the class, query a database to get attributes, create the book, etc.
end
my_book = Book.new
tex_file = File.open("/path/to/raw/tex/template")
template = ERB.new(tex_file.read)
f = File.new("/path/to/tex/output.tex")
f.puts template.result
system "pdflatex /path/to/tex/output.tex"
The system line puts me in interactive tex input mode, as if the document were empty. If I remove the call, the document is generated as normal. How can I ensure that the system call isn't made until after the document is generated? In the meantime I'm just using a bash script that calls the ruby script and then pdflatex to get around the issue.
The File.new will open a new stream that won't be closed (saved to disk) until the script ends of until you manually close it.
This should work:
...
f = File.new("/path/to/tex/output.tex")
f.puts template.result
f.close
system "pdflatex /path/to/tex/output.tex"
Or a more friendly way:
...
File.open("/path/to/tex/output.tex", 'w') do |f|
f.puts template.result
end
system "pdflatex /path/to/tex/output.tex"
The File.open with a block will open the stream, make the stream accessible via the block variable (f in this example) and auto-close the stream after the block execution. The 'w' will open or create the file (if the file already exists the content will be erased => The file will be truncated)