I'm developing a Sinatra app that consists of an upload form, with a progress bar indicating how much of the upload has completed.
The process, as described by ryan dahl, is the following:
HTTP upload progress bars are rather obfuscated- they typically involve a process running on the server keeping track of the size of the tempfile that the HTTP server is writing to, then on the client side an AJAX call is made every couple seconds to the server during the upload to ask for the progress of the upload.
Every upload has a random session-id, and to keep track of the association i employ a class variable in my app (i know, that's horrible -- if you've got better ideas, please tell me)
configure do
##assoc = {}
end
I have a POST route for the upload, and a GET one for the AJAX polling.
Inside the POST route i save the association of session-id, Tempfile, and total size.
post '/files' do
tmp = params[:file][:tempfile]
# from here on, ##assoc[#sid] should have a value, even in other routes
##assoc[#sid] = { :file => tmp, :size => env['CONTENT_LENGTH'] }
File.open("#{options.filesdir}/#{filename}", 'w+') do |file|
file << tmp.read
end
end
In the GET route, i calculate the percentage based on the Tempfile's current size:
get '/status/:sid' do
h = ##assoc[params[:sid]]
unless h.nil?
percentage = (h[:file].size / h[:size].to_f) * 100
"#{percentage}%"
else
"0%"
end
end
The problem is that until the POST request hasn't completed (i.e., after it has read all of the Tempfile) the h.nil? returns true, which doesn't really make sense as I've just assigned ##assoc[#sid] a value in the other route.
So, what am I missing here?
EDIT: I've tried
set :reload, false
set :environment, :production
config { ##assoc ||= {} }
I also tried throwing a relational db at it (SQLite with DataMapper)
Neither worked.
I think i got what the problem is:
tmp = params[:file][:tempfile] doesn't return until the file has been fully received.
##assoc[#sid] = { :file => tmp, :size => env['CONTENT_LENGTH'] }
should be
##assoc[params[:sid]] = { :file => tmp, :size => env['CONTENT_LENGTH'] }
Related
Hi I'm having trouble downloading multiple files with axlsx. The problem is I'm sending an array of Id's to the controller and asking it to download the report using the render command. It raises an AbstractController::DoubleRenderError. I was thinking of overriding the error but realized it's a bad idea, I don't know what else to do... Any suggestions? Thanks.
My controller code looks like this:
def download_report
params[:user_id].each do |user_id|
#report = Report.find_by(:user_id => user_id)
render :xlsx => "download_report", :filename => "#{#report.user.last_name}.xlsx"
end
end
My axlsx template:
wb = xlsx_package.workbook
wb.add_worksheet(name: "Reports") do |sheet|
wb.styles do |s|
# template code
end
end
It is the built in expectation of Rails that you would call render once per request. And, the browser is going to expect one response per request. So, you are going to have to do something else!
You can use render_to_string, and combine the results into a zip file, serving that. See the bottom of this response.
Or, you could create a single spreadsheet and have each user's report show up on their own worksheet.
Or, on the client side, you could use javascript to request each spreadsheet and download each one separately.
The zip one would be something like this code, which uses render_to_string, rubyzip, and send_data:
def download_report
compressed_filestream = Zip::ZipOutputStream.write_buffer do |zos|
params[:user_id].each do |user_id|
#report = Report.find_by(:user_id => user_id)
content = render_to_string :xlsx => "download_report", :filename => "#{#report.user.last_name}.xlsx"
zos.put_next_entry("user_#{user_id}.xlsx")
zos.print content
end
end
compressed_filestream.rewind
send_data compressed_filestream.read, :filename => 'download_report.zip', :type => "application/zip"
end
Axlsx requires rubyzip, so you should have it already. And you probably want to lookup each user and use their name for the spreadsheet, unless you have it otherwise.
Say I have some HTML documents stored on S3 likes this:
http://alan.aws-s3-bla-bla.com/posts/1.html
http://alan.aws-s3-bla-bla.com/posts/2.html
http://alan.aws-s3-bla-bla.com/posts/3.html
http://alan.aws-s3-bla-bla.com/posts/1/comments/1.html
http://alan.aws-s3-bla-bla.com/posts/1/comments/2.html
http://alan.aws-s3-bla-bla.com/posts/1/comments/3.html
etc, etc
I'd like to serve these with a Rack (preferably Sinatra) application, mapping the following routes:
get "/posts/:id" do
render "http://alan.aws-s3-bla-bla.com/posts/#{params[:id]}.html"
end
get "/posts/:posts_id/comments/:comments_id" do
render "http://alan.aws-s3-bla-bla.com/posts/#{params[:posts_id]}/comments/#{params[:comments_id}.html"
end
Is this a good idea? How would I do it?
There would obviously be a wait while you grabbed the file, so you could cache it or set etags etc to help with that. I suppose it depends on how long you want to wait and how often it is accessed, its size etc as to whether it's worth storing the HTML locally or remotely. Only you can work that bit out.
If the last expression in the block is a string that will automatically be rendered, so there's no need to call render as long as you've opened the file as a string.
Here's how to grab an external file and put it into a tempfile:
require 'faraday'
require 'faraday_middleware'
#require 'faraday/adapter/typhoeus' # see https://github.com/typhoeus/typhoeus/issues/226#issuecomment-9919517 if you get a problem with the requiring
require 'typhoeus/adapters/faraday'
configure do
Faraday.default_connection = Faraday::Connection.new(
:headers => { :accept => 'text/plain', # maybe this is wrong
:user_agent => "Sinatra via Faraday"}
) do |conn|
conn.use Faraday::Adapter::Typhoeus
end
end
helpers do
def grab_external_html( url )
response = Faraday.get url # you'll need to supply this variable somehow, your choice
filename = url # perhaps change this a bit
tempfile = Tempfile.open(filename, 'wb') { |fp| fp.write(response.body) }
end
end
get "/posts/:whatever/" do
tempfile = grab_external_html whatever # surely you'd do a bit more here…
tempfile.read
end
This might work. You may also want to think about closing that tempfile, but the garbage collector and the OS should take care of it.
I am using Prawn to generate a PDF from my controller, and when accessed directly at the url, it works flawlessly, I.E. localhost:3000/responses/1.pdf
However, when I try to generate this file on the fly for inclusion in a Mailer, everything freezes up and it times out.
I have tried various methods for generating / attaching the file and none have changed the outcome.
I also tried modifying the timeout for Net::HTTP to no avail, it just takes LONGER to time out.
If I run this command on the Rails Console, I receive a PDF data stream.
Net::HTTP.get('127.0.0.1',"/responses/1.pdf", 3000)
But if I include this code in my controller, it times out.
I have tried two different methods, and both fail repeatedly.
Method 1
Controller:
http = Net::HTTP.new('localhost', 3000)
http.read_timeout = 6000
file = http.get(response_path(#response, :format => 'pdf')) #timeout here
ResponseMailer.confirmComplete(#response,file).deliver #deliver the mail!
Method 1 Mailer:
def confirmComplete(response,file)
email_address = response.supervisor_id
attachments["test.pdf"] = {:mime_type => "application/pdf", :content=> file}
mail to: email_address, subject: 'Thank you for your feedback!'
end
The above code times out.
Method 2 Controller:
ResponseMailer.confirmComplete(#response).deliver #deliver the mail!
Method 2 Mailer:
def confirmComplete(response)
email_address = response.supervisor_id
attachment "application/pdf" do |a|
a.body = Net::HTTP.get('127.0.0.1',"/responses/1.pdf", 3000) #timeout here
a.filename = "test.pdf"
end
mail to: email_address, subject: 'Thank you for your feedback!'
end
If I switch the a.body and a.filename, it errors out first with
undefined method `filename=' for #<Mail::Part:0x007ff620e05678>
Every example I find has a different syntax or suggestion but none fix the problem that Net::HTTP times out. Rails 3.1, Ruby 1.9.2
The problem is that, in development, you're only running one server process, which is busy generating the email. That process is sending another request (to itself) to generate a PDF and waiting for a response. The request for the PDF is basically standing in line at the server so that it can get it's PDF, but the server is busy generating the email and waiting to get the PDF before it can finish. And thus, you're waiting forever.
What you need to do is start up a second server process...
script/rails server -p 3001
and then get your PDF with something like ...
args = ['127.0.0.1','/responses/1.pdf']
args << 3001 unless Rails.env == 'production'
file = Net::HTTP.get(*args)
As an aside, depending on what server you're running on your production machine, you might run into issues with pointing at 127.0.0.1. You might need to make that dynamic and point to the full domain when in production, but that should be easy.
I agree with https://stackoverflow.com/users/811172/jon-garvin's analysis that you're only running one server process, but I would mention another solution. Refactor your PDF generation so you don't depend on your controller.
If you're using Prawnto, I'm guessing you have a view like
# app/views/response.pdf.prawn
pdf.text "Hello world"
Move this to your Response model: (or somewhere else more appropriate, like a presenter)
# app/models/response.rb
require 'tmpdir'
class Response < ActiveRecord::Base
def pdf_path
return #pdf_path if #pdf_generated == true
#pdf_path = File.join(Dir.tmpdir, rand(1e11).to_s)
Prawn::Document.generate(#pdf_path) do |pdf|
pdf.text "Hello world"
end
#pdf_generated = true
#pdf_path
end
def pdf_cleanup
if #pdf_generated and File.exist?(#pdf_path.to_s)
File.unlink #pdf_path
end
end
end
Then in your ResponsesController you can do:
# app/controllers/responses_controller.rb
def show
#response = Response.find params[:id]
respond_to do |format|
# this sends the PDF to the browser (doesn't email it)
format.pdf { send_file #response.pdf_path, :type => 'application/pdf', :disposition => 'attachment', :filename => 'test.pdf' }
end
end
And in your mailer you can do:
# this sends an email with the PDF attached
def confirm_complete(response)
email_address = response.supervisor_id
attachments['test.pdf'] = {:mime_type => "application/pdf", :content => File.read(response.pdf_path, :binmode => true) }
mail to: email_address, subject: 'Thank you for your feedback!'
end
Since you created it in the tmpdir, it will be automatically deleted when your server restarts. You can also call the cleanup function.
One final note: you might want to use a different model name like SupervisorReport or something - Response might get you in namespacing trouble later)
I need feedback on the design for uploading and delayed processing of a file using heroku, paperclip, delayed job and, if necessary, s3. Parts of it have been discussed in other places but I couldn't find a complete discussion anywhere.
Task description:
Upload file (using paperclip to s3/db on heroku). File needs to be private as it contains sensitive data.
Queue file for processing (delayed job)
Job gets run in queue
File is retrieved (from s3/db), and processing is completed
File is deleted (from s3/db)
Since I am using delayed job, I have to decide between storing the file in the database or on s3. I am assuming that storing the file on the web server is out of the question as I am using heroku and delayed job. Uploading files to s3 takes a long time. But, storing files in db is more expensive. Ideally, we would want the processing to finish as quickly as possible.
What is the more common design pattern? Store files on s3? Store files in db? Any particular recommended gems used to retrieve and process files stored in s3 (aws-s3? s3?)?
Heroku has a timeout of 30 seconds on any server request (learnt the hard way), so definitely storing files on s3 is a must.
Try carrierwave (carrierwave railscasts) instead of paperclip, as I prefer the added helpers that come onboard, plus there a number of great plugins, like carrierwave_direct for uploading large files to s3, which integrate nicely with carrierwave.
Delayed_job (railscasts - delayed_job) will work nicely for deleting files from s3 and any other background processing that may be required.
My gem file includes the following:
gem 'delayed_job'
gem "aws-s3", :require => 'aws/s3'
gem 'fog'
gem 'carrierwave'
gem 'carrierwave_direct'
fog gem is a nice way to have all your account info in a single place and sets up everything quite nicely. For the AWS gem how-to, good resource.
Here is a sample controller when submitting a form to upload (there are definitely better ways of doing this, but for illustrative purposes)
def create
#asset = Asset.new(:description => params[:description], :user_id => session[:id], :question_id => #question.id)
if #asset.save && #asset.update_attributes(:file_name => sanitize_filename(params[:uploadfile].original_filename, #asset.id))
AWS::S3::S3Object.store(sanitize_filename(params[:uploadfile].original_filename, #asset.id), params[:uploadfile].read, 'bucket_name', :access => :private, :content_type => params[:uploadfile].content_type)
if object.content_length.to_i < #question.emailatt.to_i.megabytes && object.content_length.to_i < 5.megabytes
url = AWS::S3::S3Object.url_for(sanitize_filename(params[:uploadfile].original_filename, #asset.id), 'bucket_name')
if #asset.update_attributes(:download_link => 1)
if Usermailer.delay({:run_at => 5.minutes.from_now}).attachment_user_mailer_download_notification(#asset, #question)
process_attachment_user_mailer_download(params[:uploadfile], #asset.id, 24.hours.from_now, #question.id)
flash[:notice] = "Thank you for the upload, we will notify this posts author"
end
end
end
else
#asset.destroy
flash[:notice] = "There was an error in processing your upload, please try again"
redirect_to(:controller => "questions", :action => "show", :id => #question.id)
end
end
private
def sanitize_filename(file_name, id)
just_filename = File.basename(file_name)
just_filename.sub(/[^\w\.\-]/,'_')
new_id = id.to_s
new_filename = "#{new_id}" + just_filename
end
def delete_process(uploadfile, asset_id, time, question_id)
asset = Asset.find(:first, :conditions => ["id = ?", asset_id])
if delete_file(uploadfile, asset_id, time) && asset.destroy
redirect_to(:controller => "questions", :action => "show", :id => question_id)
end
end
def process_attachment_user_mailer_download(uploadfile, asset_id, time, question_id)
asset = Asset.find(:first, :conditions => ["id = ?", asset_id])
if delete_file(uploadfile, asset_id, time) && #asset.delay({:run_at => time}).update_attributes(:download_link => 0)
redirect_to(:controller => "questions", :action => "show", :id => question_id)
end
end
#S3 METHODS FOR CREATE ACTION
#deletes the uploaded file from s3
def delete_file(uploadfile, asset_id, time)
AWS::S3::S3Object.delay({:run_at => time}).delete(sanitize_filename(uploadfile.original_filename, asset_id), 'bucket_name')
end
Lots of unnecessary code, I know (wrote this when I was starting with Rails). Hopefully it will give some idea of the processes involved in writing this type of app. Hope it helps.
For my part I'm using :
Delayed Job
Paperclip
Delayed Paperclip which uploads the original file
on S3 and create a delayed job with the custom post processing. It
can add a column to you model stating that the file is being
processed.
Only a few lines to set up. And you can do a lot with paperclip interpolations and generators.
I'm new to Rails. I'm developing a store builder.
What I want
I want a root level url for each shop.
http://greatsite.com/my-shop-name
My Solution
shop_controller.rb
def show
if params[:url]
#shop_ref = params[:url]
#shop = Shop.where(:url => #shop_ref).first
else
#shop_ref = params[:id]
#shop = Shop.find(#shop_ref)
redirect_to "/" + #shop.url
return
end
if #shop.nil?
render 'show_invalid_shop', :object => #shop_ref and return
end
render 'show' => #shop
end
def create
#shop_url = (0...8).map{65.+(rand(25)).chr}.join.downcase
#shop = Shop.new(:url => #shop_url)
if #shop.save
redirect_to "/" + #shop.url
else
render :action => "new"
end
end
routes.rb
...
resources :shops
match ':url' => 'shops#show', :constraints => { :url => /[a-z|0-9]{4,30}/ }
...
The Problem
Crap Performance. (It's ugly as sin too, of course.)
Every time someone creates a new shop (which is one click from our home page), it creates a new shop and does a redirect. In New Relic, I see this is killing performance - a lot of time is spent in "Request Queuing".
Is there any neater and faster way of achieving what I want?
I'm not sure why the redirects would be causing such a headache, but:
Could you do something like:
Create the shop via an AJAX call.
On a successful create via AJAX render the show view, and return the html "string".
Replace the contents of the page with JS, and use pushstate to update the URL.
Might be useful to look at: http://pjax.heroku.com/
It's not exactly pretty, but if redirects are really that bad it might help?
I wouldn't recommend this, as it violates the REST principle...
But you could have create call/render the show action after it's done it's object creation (just like you do with "new" when it fails). That would eliminate the redirect but still show the same content as if it had.
There's a lot of reasons why you wouldn't want to do this. I'd look for performance improvements in other places first.