In my project I am assigning individual UUIDs to every upload that gets moved between two folders in my S3 Bucket, for the purposes of individualising their links later on.
However, while attempting to format the Object Key as displayed in the method below, all the code below
parts = old_key.split('/') greys out/ no longer functions, indicating a Syntax error that I can't see.
def move_and_rename(target_folder, prefix_generator: nil)
prefix_generator ||= -> { SecureRandom.uuid }
s3_objects.each do |obj|
prefix = prefix_generator.call
old_key = obj.key
parts = old_key.split('/')
new_last_part = "#{prefix}-#{parts.last}”
new_key = (parts[0..-2] + [new_last_part]).join('/')
new_key = s3_folder ? obj.key.sub(s3_folder, target_folder) : (parts[0..-2] + [new_last_part]).join('/')
obj.move_to(bucket: bucket_name, key: new_key)
yield(old_key: old_key, new_key: new_key, obj: obj, prefix: prefix) if block_given?
end
self
end
Or perhaps I have this structured completely wrong within the method? Any pointers would be appreciated.
I'm attempting to pull all .xslx files in my local directory into my S3 bucket, but I'm struggling on how best to direct my script to locate and pull all files with that file extension.
Here is what I currently have, any directions/suggestions are appreciated.
def initialize(bucket:, object_key:, input_file_path:)
#bucket = bucket
#object_key = object_key
#input_file_path = input_file_path
end
def call
object = s3_resource.bucket(bucket).object(object_key)
object.upload_file(input_file_path)
end
private
attr_reader :bucket, :object_key, :input_file_path
Just skip the call when the input_file_path does not match:
def call
return unless input_file_path.end_with?('.xslx')
object = s3_resource.bucket(bucket).object(object_key)
object.upload_file(input_file_path)
end
Trying to delete a s3 object by using ruby with something like this:
After('#removeS3Files') do
prefix = 'pifiles/files/'
s3 = AWS::S3.new
bucket = s3.buckets['scratching']
if (defined?(s3files)).nil?
s3files = []
end
s3files << bucket.objects.with_prefix(prefix).collect(&:key)
s3files.each do |key|
bucket.objects.delete(key)
end
end
I'm getting the error "Failed to delete 1 objects (AWS::S3::Errors::BatchDeleteError)"
I am trying to list only the objects from the s3 folder (not a real folder I know) called distribution but I want to remove the reference to the name and any slashes around the object. The output should just look like 021498cd-ca73-4675-a57a-c12b3c652aac whereas currently it looks like distribution/021498cd-ca73-4675-a57a-c12b3c652aac/
So far I have tried;
def files
s3 = Aws::S3::Resource.new
s3.client
bucket = s3.bucket('test')
files = []
bucket.objects.each do |obj|
if obj.key.include?('distribution/')
temp_files = puts "#{obj.key}"
files = temp_files.select do |file|
file.gsub("distribution/", "")
end
else
end
end
end
But this doesn't seem to be working at all.
Your explanation is pretty simple but your code is implying something else.
However, this should help with what you are trying to achieve.
def files
s3 = Aws::S3::Resource.new
s3.client
bucket = s3.bucket('test')
files = []
bucket.objects.each do |obj|
if obj.key.include?('distribution/')
files << "#{file.gsub(/(distribution)|\//, '')}"
end
end
end
The files array will contain all the file names with garbage stripped.
How do I do the equivalent of 's3cmd ls s3://some_bucket/foo/bar' in Ruby?
I found the Amazon S3 gem for Ruby and also the Right AWS S3 library, but somehow it's not immediately obvious how to do a simple 'ls' like command on an S3 'folder' like location.
Using the aws gem this should do the trick:
s3 = Aws::S3.new(YOUR_ID, YOUR_SECTRET_KEY)
bucket = s3.bucket('some_bucket')
bucket.keys('prefix' => 'foo/bar')
I found a similar question here: Listing directories at a given level in Amazon S3
Based on that I created a method that behaves as much as possible as 's3cmd ls <path>':
require 'right_aws'
module RightAws
class S3
class Bucket
def list(prefix, delimiter = '/')
list = []
#s3.interface.incrementally_list_bucket(#name, {'prefix' => prefix, 'delimiter' => delimiter}) do |item|
if item[:contents].empty?
list << item[:common_prefixes]
else
list << item[:contents].map{|n| n[:key]}
end
end
list.flatten
end
end
end
end
s3 = RightAws::S3.new(ID, SECRET_KEY)
bucket = s3.bucket('some_bucket')
puts bucket.list('foo/bar/').inspect
In case some looks for the answer to this question for the aws-sdk version 2, you can very easily do this this way:
creds = Aws::SharedCredentials.new(profile_name: 'my_credentials')
s3_client = Aws::S3::Client.new(region: 'us-east-1',
credentials: creds)
response = s3_client.list_objects(bucket: "mybucket",
delimiter: "/")
Now, if you do
response.common_prefixes
It will give you the "Folders" of that particular subdirectory, and if you do
response.contents
It will have the files of that particular directory
The official Ruby AWS SDK now supports this: http://docs.aws.amazon.com/AWSRubySDK/latest/AWS/S3/Tree.html
You can also add the following convenience method:
class AWS::S3::Bucket
def ls(path)
as_tree(:prefix => path).children.select(&:branch?).map(&:prefix)
end
end
Then use it like this:
mybucket.ls 'foo/bar' # => ["/foo/bar/dir1/", "/foo/bar/dir2/"]
a quick and simple method to list files in a bucket folder using the ruby aws-sdk:
require 'aws-sdk'
s3 = AWS::S3.new
your_bucket = s3.buckets['bucket_o_files']
your_bucket.objects.with_prefix('lots/of/files/in/2014/09/03/').each do |file|
puts file.key
end
Notice the '/' at the end of the key, it is important.
I like the Idea of opening the Bucket class and adding a 'ls' method.
I would have done it like this...
class AWS::S3::Bucket
def ls(path)
objects.with_prefix("#{path}").as_tree.children.select(&:leaf?).collect(&:member).collect(&:key)
end
end
s3 = AWS::S3.new
your_bucket = s3.buckets['bucket_o_files']
your_bucket.ls('lots/of/files/in/2014/09/03/')