I'm using a ruby script to generate an html page that I'm uploading to s3. My amazon bucket is configured to work as a static website, and the html page my script is generating is the index page.
I've created a couple of models to build a connection to s3 using the fog gem, generate the html page, and then push it up to s3.
When I use the script to push the file up to s3, I don't get any errors, but only half the file gets uploaded. It happens regardless of file size. I inevitably wind up with about 15 rows missing from the html table I generated.
The relevant code from my FogManager model is this:
def connect
#connection = Fog::Storage.new({
provider: 'AWS',
aws_access_key_id: AWS_ACCESS_KEY,
aws_secret_access_key: AWS_SECRET_KEY,
persistent: true
})
end
def locate_directory
#directory = #connection.directories.get(DIRECTORY)
end
def upload_report_page(index_file)
file = #directory.files.get(index_file)
file.body = File.open(index_file)
file.acl = 'public-read'
file.save
end
My script looks something like:
filename = "index.html"
s3 = FogManager.new
s3.connect
s3.locate_directory
# code for generating the html page
s3.upload_report_page(filename)
All of the code works, but the whole html file isn't getting uploaded to s3.
I'm using the latest version of fog (1.15.0) with ruby 2.0.0p0
I've been able to manually go into irb and establish an s3 connection, locate the file, and upload the new one. I use these commands:
require 'fog'
connection = Fog:Storage({provider: 'AWS',
aws_access_key_id: 'xxx',
aws_secret_access_key: 'xxx',
persistent: true})
directory = connection.directories.get('my_directory')
file = directory.files.get('index.html')
file.body = File.open('index.html')
file.acl = 'public-read'
file.save
When I upload the file this way, it works properly, and the whole file gets uploaded, but it defeats the purpose of having the script.
Any insight would be greatly appreciated!
Related
I am using the ROO gem to parse an Excel file uploaded by the user (Active Storage with AWS S3) on Heroku.
Since I am on Heroku I cannot download it on filesystem and then parse it. How can I open it from HTTP?
Model
class Report < ApplicationRecord
has_one_attached :search_report
def parsing_method(path_http)
xlsx = Roo::Spreadsheet.open(path_http)
end
Yes, you can download it in Heroku into the filesystem.
I doubt theres much space, but you should be able to use the temp file. Most system stores an upload into the temp file system before pushing it somewhere else.
I'm already doing it, since Roo was patch to be able to open Excel files from the stream, but no other type (csv, ods, etc).
def create_temp_file
filename = some_model.some_attachment.blob.filename
#tmp = Tempfile.new([filename.base, filename.extension_with_delimiter], binmode: true)
#tmp.write(ome_model.some_attachment.download)
#tmp.rewind
#tmp
end
# this is how you use it, note the rescue part
def access_the_file
spreadsheet = Roo::Spreadsheet.open(create_temp_file)
# access the spreadsheet as usual
rescue
#tmp&.unlink
end
This preserves the file prefixes and extensions (important to let Roo infer what the file is) and ensures the file get deleted when done.
ModelName.filename.blob.open do |file|
write your code here ...
end
You can temporarily download the file with open method
report.search_report.open do |file|
xlsx = Roo::Spreadsheet.open(file)
end
In a similar application, I used this:
def parsing_method(path_http)
xlsx = Roo::Excelx.new(path_http, nil, :ignore)
end
It should work for you.
I am currently able to send a docx file for the user to download, question is how do I save the file created for download from the send_file method to store into a local directory in a ruby app?
Below is the code that uses send_file:
send_file rand_file, :type => 'docx', :filename => "#{#report.report_name}.docx"
Save the file prior to calling send_file and then reference it as such
file = File.open(temp_file_path, 'w+')
file.binmode
file.write(rand_file.read)
file.close
file = Tempfile.new('temp_file_path', 'tmp')
send_file file, ...
I've finally solve my problem by using tips from #katafrakt and #rantingsonrails by using the FileUtils method of copying just before the send_file command. Below are my code on how i did it.
temp_file_path = "./document/#{#report.report_name}.docx"
FileUtils::copy_file(rand_file,temp_file_path)
Currently, I am sending GET requests to S3 using aws-sdk ruby as follow:
#!/usr/bin/ruby
#
require 'aws-sdk'
s3 = Aws::S3::Resource.new(region: 'test', endpoint:'http://10.0.23.45:8081')
my_bucket = s3.bucket('test.bucket-name')
my_bucket.objects.limit(50).each do |obj|
puts " #{obj.key} => #{obj.etag}"
end
But the request is trying to hit this url endpoint(virtual hosting):
http://test.bucket-name.10.0.23.45:8081
I would like to use path style addressing instead. This is what I want the request url endpoint to look like:
http://10.0.23.45:8081/test.bucket-name/
Any idea how to set path style addressing instead of virtual hosting address? Thanks.
I found the answer for my own question after looking at the source code of ruby aws-sdk Source Code
Aws.config[:s3] = { force_path_style: true }
Adding the above line forced to use path style addressing.
You need to set option :virtual_host to true according to documentation.
So in your case something like this should work:
s3.bucket('10.0.23.45:8081').object('test.bucket-name').public_url(virtual_host: true)
#=> "http://10.0.23.45:8081/test.bucket-name/"
Hello I am currently building a site using Rails and Heroku as well as GoDaddy for a custom domain name.
I am using Carrierwave with Fog, Amazon S3, and have set up a cloudfront with my Heroku domain name as the Origin Domain. Currently, the site loads up images very slow which is why I set up a cloudfront, but my carrierwave's config file doesn't seem to change the urls to the cloudfront counterparts. Here is my carrierwave config file. I would like users to be able to upload images through Carrierwave and then have those served back with my cloudfront.
CarrierWave.configure do |config|
config.fog_provider = 'fog/aws' # required
config.fog_credentials = {
provider: 'AWS', # required
aws_access_key_id: 'acesskey', # required
aws_secret_access_key: 'secretkey', # required
region: 'us-east-1', # optional, defaults to 'us-east-1'
}
config.fog_directory = 'directory' # required
# config.fog_public = false # optional, defaults to true
config.asset_host = 'randomjunk.cloudfront.net'
config.fog_public = false
config.fog_attributes = { 'Cache-Control' => "max-age=#{365.day.to_i}" } # optional, defaults to {}
end
It works as long as config.fog_public is false but when I change it to config.fog_public = true the urls are now correct but the images are not found. Is this something to do with my configuration settings or have I set up my cloudfront with Heroku incorrectly?
It depends on the settings that were used when you created the files initially. If they were created with config.fog_public = false they will be private files and access will be done with a special signed url to s3. I think the asset host is just used for files that are public (hence what you are seeing). If you upload files with fog_public = true then I don't think you would see the not found. You may also be able to just set it to true and re-save images to change that attribute (I'm less sure about specifics here, you certainly should be able to change them to public though).
I have some cucumber tests that I'm writing, and to test different scenarios I setup a config.yml file. I load it in in my env.rb using:
require 'calabash-cucumber/cucumber'
require 'yaml'
# Config file to hold user settings
#CONFIG = YAML::load_file(File.join(File.dirname(File.expand_path(__FILE__)), 'config.yml'))
CONFIG = YAML.load_file("features/support/config.yml")
The commented out line is another variation I've tried to load the file.
When I call the data, I do so like this:
#userName = CONFIG[env.to_sym][side.to_sym][type.to_sym][:username]
Yet, when it gets to this line, it gives me the following error:
"undefined method '[]' for nil:NilClass (NoMethodError)"
(The [] is an empty square box in the console.)
I'm not sure how to rectify this issue, I'm 99% sure I'm loading the YAML file correctly and using it correctly.
Any ideas or suggestions would be great!
Edit: Sample YAML data:
:env:
:side:
type:
:username: "username"
Say you have config.yml file:
env:
side:
type:
username: 'User'
YAML.load_file() won't symbolize hash by default, so you need to access values using string, not symbols.
puts #userName = CONFIG['env']['side']['type']['username']
#=> User
Update:
As mentioned by the Tin Man, it depends what is your YML file look like.
Loading this yml file would produce symbolized hash:
:env:
:side:
:type:
:username: 'User'
So if you want to access username key:
#userName = CONFIG[:env][:side][:type][:username]