Trying to delete a s3 object by using ruby with something like this:
After('#removeS3Files') do
prefix = 'pifiles/files/'
s3 = AWS::S3.new
bucket = s3.buckets['scratching']
if (defined?(s3files)).nil?
s3files = []
end
s3files << bucket.objects.with_prefix(prefix).collect(&:key)
s3files.each do |key|
bucket.objects.delete(key)
end
end
I'm getting the error "Failed to delete 1 objects (AWS::S3::Errors::BatchDeleteError)"
Related
I'm attempting to pull all .xslx files in my local directory into my S3 bucket, but I'm struggling on how best to direct my script to locate and pull all files with that file extension.
Here is what I currently have, any directions/suggestions are appreciated.
def initialize(bucket:, object_key:, input_file_path:)
#bucket = bucket
#object_key = object_key
#input_file_path = input_file_path
end
def call
object = s3_resource.bucket(bucket).object(object_key)
object.upload_file(input_file_path)
end
private
attr_reader :bucket, :object_key, :input_file_path
Just skip the call when the input_file_path does not match:
def call
return unless input_file_path.end_with?('.xslx')
object = s3_resource.bucket(bucket).object(object_key)
object.upload_file(input_file_path)
end
In our rails 6 project I want to upload excel file into S3 bucket and send file link to email and provide download xls feature from email. Please help me.
I have generated the excel file using the code below, and also set S3 configuration in our project
xlsx = GenerateSpreadsheetService.new(all_data, file_name).call
Now my question is that how to upload the above excel file into S3 bucket and send file link to email and provide download xls feature from email
class ImportCsvDataJob < ApplicationJob
queue_as :default
def perform(file_name, column)
begin
keywords = []
all_data = []
file_path = get_file_path(file_name)
spreadsheet = Roo::Excelx.new(file_path)
(2..spreadsheet.last_row).each do |i|
keywords << {"language_code": "en", "location_code": 2840, "keyword": "allintitle: #{spreadsheet.row(i)}"}
end
response = GetTaskService.new(keywords).call
response["tasks"].map{|task|
keyword = task["data"]["keyword"].gsub('allintitle: ','')
result = GetTaskService.new.get_values(task["id"])
all_data << {keyword: keyword.to_s, value: result["tasks"][0]["result"][0]["se_results_count"]}
}
## Create records
CsvImport.create(all_data)
## Generate excel file
xlsx = GenerateSpreadsheetService.new(all_data, file_name).call
## send email
SendEmailJob.perform_now(xlsx, file_name)
## remove temp storage file after processing
File.delete(Rails.root + "public/spreadsheets/#{file_name}")
rescue Exception => e
Rails.logger.info "--error-----#{e.message}--"
end
end
end
class GenerateSpreadsheetService
def initialize(data = [], file_name)
#data = data
#file_name = file_name
end
def call
ActionController::Base.new().render_to_string(template: "csv_imports/template.xlsx.axlsx", layout: false, formats: [:axlsx], locals: {:data => #data})
end
end
class SendEmailJob < ApplicationJob
queue_as :default
def perform(xlsx, file_name)
SendEmailService.new(xlsx, file_name).call
end
end
class SendEmailService
def initialize(xlsx, file_name)
#xlsx = xlsx
#email = "test#example.com"
#subject = "excel file"
#file_name = file_name
end
def call
KeywordMailer.send_csv(#email, #subject, #xlsx, #file_name).deliver_now!
end
end
https://docs.aws.amazon.com/AmazonS3/latest/dev/UploadObjSingleOpRuby.html
Amazon gives you the answer:
require 'aws-sdk-s3'
s3 = Aws::S3::Resource.new(region:'us-west-2')
obj = s3.bucket('bucket-name').object('key')
obj.upload_file('/path/to/source/file')
You simply need to set the region, bucket-name and then the file path on the last line is wherever you store this xls file. It looks like you're storing it at /public/spreadsheets with some random file name.
How did I solved this problem?
Generate excel sheet
time = Time.zone.now.strftime("%Y%m%d%H%M%S")
xlsx = GenerateSpreadsheetService.new(all_data, file_name).call
Create an empty directory
path = Tempfile.new("data_#{time}.xlsx").path
Write the excel file into this directory
file = File.open(path,"wb") {|f| f.write(xlsx) }
s3 = Aws::S3::Resource.new(region: APP_CONFIG["s3_region"], credentials: Aws::Credentials.new(APP_CONFIG["s3_access_key"], APP_CONFIG["s3_secret_key"]))
obj = s3.bucket(APP_CONFIG["s3_bucket"]).object("data_#{time}.xlsx")
obj.upload_file(path)
presigned_url = obj.presigned_url(:put, bucket: APP_CONFIG["s3_bucket"], key: "data_#{time}.xlsx", expires_in: 604800)
save public_url and presigned_url
FileUrl.create(key: "data_#{time}.xlsx", file_public_url: obj.public_url, file_presigned_url: presigned_url) if presigned_url.present?
I'm new to AWS and I'm trying to loop through my bucket objects to generate urls to read the objects. I found the following reference in the AWS documentation:
bucket.objects.myobject.url_for(:read)
I have the following method which contains a loop that can at least print the key for each object BUT I'm struggling to get the url_for to work. Any ideas?
def aws_s3_url
s3_client = Aws::S3::Resource.new(region: ENV['AWS_REGION'])
bucket = s3_client.bucket(ENV['S3_BUCKET'])
bucket.objects.each do |name|
puts name.key
end
end
All help is appreciated.
I don't know the specific use-case you have, but you don't need the URL to read the objects in the bucket since the AWS SDK maps the files in the bucket to instances of Object.
To read the contents of the file, try this:
#s3_client = Aws::S3::Resource.new(region: ENV['AWS_REGION'])
def file_content(key)
bucket = #s3_client.bucket(ENV['S3_BUCKET'])
obj = #s3_client.get_object(bucket: bucket, key: key)
obj.body.read
end
def get_all_files
bucket = #s3_client.bucket(ENV['S3_BUCKET'])
bucket.objects.each do |o|
puts file_content(o.key)
end
end
To return the public URL for the object, you can try:
Aws::S3::Object.public_url:
def get_url(key)
bucket = #s3_client.bucket(ENV['S3_BUCKET'])
obj = #s3_client.get_object(bucket: bucket, key: key)
obj.public_url
end
I am trying to list only the objects from the s3 folder (not a real folder I know) called distribution but I want to remove the reference to the name and any slashes around the object. The output should just look like 021498cd-ca73-4675-a57a-c12b3c652aac whereas currently it looks like distribution/021498cd-ca73-4675-a57a-c12b3c652aac/
So far I have tried;
def files
s3 = Aws::S3::Resource.new
s3.client
bucket = s3.bucket('test')
files = []
bucket.objects.each do |obj|
if obj.key.include?('distribution/')
temp_files = puts "#{obj.key}"
files = temp_files.select do |file|
file.gsub("distribution/", "")
end
else
end
end
end
But this doesn't seem to be working at all.
Your explanation is pretty simple but your code is implying something else.
However, this should help with what you are trying to achieve.
def files
s3 = Aws::S3::Resource.new
s3.client
bucket = s3.bucket('test')
files = []
bucket.objects.each do |obj|
if obj.key.include?('distribution/')
files << "#{file.gsub(/(distribution)|\//, '')}"
end
end
end
The files array will contain all the file names with garbage stripped.
I am trying to iterate over the children of a AWS::S3::Tree object, like so:
conn = AWS::S3.new(:access_key_id => 'ACCESSKEYID', :secret_access_key => 'ACCESSKEY')
bucket = conn.buckets['bucketname']
tree = bucket.objects.with_prefix('assets/images').as_tree
directories = tree.children.select(&:branch?).collect(&:prefix)
This works fine on most of the paths I use as a prefix, but one folder (that has tens of thousands of sub folders) returns this error:
/lib/aws/s3/object_collection.rb:311:in `next_markers': Unable to find marker in S3 list objects response (RuntimeError)
I have overridden the gem method to debug like so:
module AWS
class S3
class ObjectCollection
protected
def next_markers page
raise page.inspect
marker = (last = page.contents.last and last.key)
if marker.nil?
raise 'Unable to find marker in S3 list objects response xxxxx'
else
{ :marker => marker }
end
end
end
end
end
And this outputs the returned data:
{:delimiter=>"/", :contents=>[], :common_prefixes=>[{:prefix=>"assets/images/100/"}, {:prefix=>"assets/images/1000/"}, {:prefix=>"assets/images/1001/"}, etc etc
Why is the call returning an empty array for :contents ?
This happens when the XML from S3 is not valid and the contents can not be parsed. If you enable wiretracing (add :http_wire_trace => true to AWS::S3.new) and you should be able to examine the offending http response body.