I'm trying to use Mechanize::Download so I can work with FLV video streams on the fly without buffering the whole video into memory, however Mechanize isn't following the pluggable parser for 'video/flv' and keeps returning a Mechanize::File instead. Is there a way to use Mechanize to return an IO-like stream of a file? The file only needs to be read once so seeking isn't necessary. I need mechanize to submit the right Cookies and referrers to get the file.
irb(main):072:0> nn.agent.pluggable_parser
=> #<Mechanize::PluggableParser:0x0000000192eea0 #parsers={"text/html"=>Mechanize::Page, "application/xhtml+xml"=>Mechanize::Page, "application/vnd.wap.xhtml+xml"=>Mechanize::Page, "image"=>Mechanize::Image, "text/xml"=>Mechanize::XmlFile, "application/xml"=>Mechanize::XmlFile, "video/flv"=>Mechanize::Download}, #default=Mechanize::Download>
irb(main):073:0> nn.agent.get(vid.video_url).response['content-type']
=> "video/flv"
irb(main):074:0> nn.agent.get(vid.video_url).class
=> Mechanize::File
irb(main):075:0>
Related
For testing, I send a Rack::Request straight to the app, not using the server.
def request_via_API( app, method, path, params={} ) # app should be API
env = Rack::MockRequest.env_for( path, {:method => method, :params=>params} )
app.handle Rack::Request.new(env)
end
works great for testing direct input, but I'm stymied by file upload. My real system works great from the browser with a file upload. But now I want to test it via the API, and don't know how to get the file contents into the request via any of the Rack classes/methods. (I tried making sense of Rack::Test::UploadedFile but didn't succeed).
Thanks, Alistair
You were definitely on the right path. You can even use your function request_via_API without any modifications, e.g.:
request_via_API(app, 'POST', '/', {
:field => "value",
:text_source => Rack::Multipart::UploadedFile.new(PATH_TO_YOUR_FILE, MIME_TYPE_OF_YOUR_FILE)
})
This means you need to have some file somewhere. If you use fixtures, your test upload file should be around them. You can omit MIME time, but it defaults to text/plain.
If you use barebones Rack, you get the following hash after calling Rack::Multipart.parse_multipart:
{
"field" => "value",
"text_source" => {
:filename => File.basename(PATH_TO_YOUR_FILE),
:type => MIME_TYPE_OF_YOUR_FILE,
:name => "text_source",
:tempfile => Tempfile.new("RackMultipart"), # copied from PATH_TO_YOUR_FILE
:head => "Content-Disposition: form-data; name=\"text_source\"; filename=\"#{File.basename(PATH_TO_YOUR_FILE)}\"\r\n" +
"Content-Type: #{MIME_TYPE_OF_YOUR_FILE}\r\n" +
"Content-Length: #{BYTESIZE_OF_YOUR_FILE}\r\n"
}
}
The text_source key can have any other name, of course.
Rack::MockRequest#env_for automatically tries to create multipart form data request if:
HTTP method is not GET
there is no :input option provided
:params option is a Hash
:params option values contain at least one instance of Rack::Multipart::UploadedFile
You can see the details in the source code here and here.
I think relying on multipart request generation by Rack::MockRequest and Rack::Multipart is useful only for mocking HTML forms with file upload and file upload mechanisms that act the same. So, there's no need to use Rack::Multipart#build_multipart or Rack::Multipart::Generator directly.
If you have more complicated multipart scenarios or different file upload mechanism, you must pass opts argument with :input key instead of :params to Rack::MockRequest#env_for. How you generate that value for :input is your problem as far as Rack mocking capabilities are concerned. It only wraps it in StringIO if it is a String as you can see here. Otherwise, it is the same thing that will be passed as rack.input in the Rack environment hash and therefore it must conform to the Rack input stream spec (i.e., be an IO-like object).
Because this was also quite a challenge for me and I used it as an exercise to deepen my knowledge of Rack I created a simple project on GitHub to explore this file upload mocking.
Note: I tried to fix everything to Rack 1.5.2 except for the link to the Rack SPEC (so beware). Links to Ruby StdLib lead to the current version.
I'm using ruby 1.9.2 along with Rails 3.1.4 and Paperclip 2.4.5.
My issue is trying to save a paperclip attachment from a URI loses the file extension and saves the file without one resulting in issues with things like fancybox that require an extension.
Some example code:
uri = "http://featherfiles.aviary.com/2012-06-13/bbe5f0de1/0c5a672b88ea47ecb4631ac173e27430.png"
open(uri)
#=> #<File:/var/folders/zc/d69gxhzx10x_bvjrkqgyjgxr0000gn/T/open-uri20120613-27204-i6cldv>
Because there is no extension on the temp file paperclip is saving the file without one resulting in issues.
Has anyone run into this issue? I've seen multiple answers about using paperclip to store images from a URI but none seem to address the same problem we're running
Don't use the temporary file! It's there as a placeholder as the file is read from the port, and should be considered a private resource for OpenURI. Instead, use open(url).read and work with the resulting content by saving it.
Do something like:
require 'uri'
require 'open-uri'
url = 'http://www.iana.org/domains/example/index.html'
filename = File.basename(URI.parse(url).path)
File.open(filename, 'wb') do |fo|
fo.write(open(url).read)
end
Temporarily spooling to disk during an operation, especially a network operation, is common. Once the file's content has been accumulated, then it is available to be passed off to the app. read is blocking, so your code will stop there until the file is returned to you. Then you can play with it.
Extension isn't important for temporary file, but if you want use this file in code or save to another place. You can do it:
temp_file = open(params[:url])
def temp_file.original_filename; File.basename(base_uri.path); end
Now, you can save this temporary file to permanent space or use it in code; Original filename will be used automatically.
Im not sure if this will help in your case, but I was noticing similar issues in my project.
The issue turned out to be not caused by Paperclip nor open-uri, but the receiver of the paperclip file (in my case Spree Commerce). Check that you are assigning the paperclip object to the right object, and that it is being interpreted correctly.
The fix that worked for me was to change:
#product.images << Spree::Image.create({
:attachment => open(image_url)
}, :without_protection => true)
to
#product.master.images << Spree::Image.create({
:attachment => open(image_url)
}, :without_protection => true)
Good luck with your issue
Have you inclued the :extension in your path/url option?
For example:
has_attached_file :image,
...
:url => '/images/highlights/:id_partition/:style_:id.:extension',
:path => ':rails_root/files/images/highlights/:id_partition/:style_:id.:extension'
This will probably solve your problem.
You can force an extension there, but I don't think that's recommended.
Update – Paperclip can do this on its own!
Posted by Aditya Sanghi (thanks a lot!):
current_comments.pictures.create!(file: URI.parse(image_url))
Although keep in mind, that you still need to handle 500, 404, etc
errors (Paperclip can raise them).
Thanks to: https://mensfeld.pl/2013/12/rails-paperclip-open-uri-downloading-files-from-the-internet-and-saving-them-with-paperclip/
Yes, it is a problem but we can get around this with fancybox.
In the link tag(for image) add :type => 'image'
- #images.each do |image|
= link_to image_tag(image.attachment.url), image.attachment.url, class: "fancybox", type: 'image'
By specifying 'type', Fancybox overrides the type as image
https://groups.google.com/forum/?fromgroups=#!topic/fancybox/QgjquBCLynU
I have a rails 3 app where I am using the 'face' gem to reference the Face.com API. The api method takes a parameter of the form:
:file => File.new(path_to_file, 'rb')
which works.
I am trying to change the flow of the app so that the file can be uploaded via a form, do some work with RMagick and then make the API call, all without saving the file to disk.
I can generate the RMagick 'Image' with
image = Magick::Image.from_blob(upload_image_field.read)
I can then manipulate the file with RMagick and even save the results into the database with:
self.data = image.to_blob #normally 'upload_image_field.read' if not using RMagick
My problem is that I can't change the image file (or the blob) into something that the API will recognize (without saving it to disk and then referencing the file on disk).
For example using this in the API method fails:
:file => image.to_blob
How do I convert he blob into the same format as
File.new(path_to_file, 'rb')
Thanks
OK, I could be wrong on this one... but I wanted to dig this up. Unfortunately, you just have to live with saving it as a file. The reason is because the API makes an HTTP POST. Unfortunately, this needs to be a file.
References from: [https://github.com/rociiu/face/tree/master/lib/face]:
recognition.rb:
def faces_detect(opts={})
opts.assert_valid_keys(:urls, :file, :detector, :attributes, :callback, :callback_url)
make_request(:faces_detect, opts)
end
utils.rb:
def make_request(api_method, opts={})
....
response = JSON.parse( RestClient.post(API_METHODS[ api_method ], opts.merge(api_crendential)).body )
....
end
So, why is it a problem to save to a file then?
Im scraping a website using Celerity gem and I want to save an image but I dont know how to do it XD
With the next Celerity command I get an IO object
irb(main):260:0* image = #browser.image(:xpath, ".//*[#class='notdTop']/img").download
=> #<IO:0x277e07ae>
How can I save this object to a jpg file?? I tried this, but didnt work:
irb(main):261:0> image.flush
IOError: not opened for writing
but the IO object is not closed because I got this:
irb(main):264:0> image.closed?
=> false
Anyone can help me please?
Try:
image.save(filename)
http://rubydoc.info/gems/celerity/0.8.9/Celerity/Image#save-instance_method
When I do a find query from the mongodb JavaScript console, I get a beautiful JSON response. However, when I do a find with the ruby driver, it stores the JSON as some Ruby object. How do I convert this back to JSON? Or, better yet, how do I just get the JSON without doing the extra conversions?
I got it working. I was using ruby-1.9.1-p378 with rvm. I removed that. Now, I'm using the system ruby-1.8.7-p174 that came with the SnowLeopard install DVD. But, I was still getting an error with the to_json method except this time it was saying, stack level too deep. I did:
require 'json/pure'
instead of
require 'json'
Then, I changed the code to look something like this:
http://github.com/banker/mongulator/blob/master/mongulator.rb#L53
Here's the relevant part of the code:
cursor = persons.find(
{"loc" => {"$near" => [params[:lat].to_f, params[:lng].to_f]}},
{:limit => 20})
content_type "application/json"
JSON.pretty_generate(cursor.to_a)
The complete file is here:
http://github.com/acani/acani-sinatra/blob/master/acani.rb
And, it worked, even with pretty json like the facebook graph api gives you. To return the JSON all on one line, just do something like:
cursor.to_a.to_json
The JSON Code you get on the javascript console is also converted. It's not the native output of MongoDB, the Native format is BSON. To Get JSON on the javascript console it must be converted. In Ruby you should be able to do the same thing with the .to_json instance method of your Object.
now you can do cursor.find.to_a.inspect