In my project I am assigning individual UUIDs to every upload that gets moved between two folders in my S3 Bucket, for the purposes of individualising their links later on.
However, while attempting to format the Object Key as displayed in the method below, all the code below
parts = old_key.split('/') greys out/ no longer functions, indicating a Syntax error that I can't see.
def move_and_rename(target_folder, prefix_generator: nil)
prefix_generator ||= -> { SecureRandom.uuid }
s3_objects.each do |obj|
prefix = prefix_generator.call
old_key = obj.key
parts = old_key.split('/')
new_last_part = "#{prefix}-#{parts.last}”
new_key = (parts[0..-2] + [new_last_part]).join('/')
new_key = s3_folder ? obj.key.sub(s3_folder, target_folder) : (parts[0..-2] + [new_last_part]).join('/')
obj.move_to(bucket: bucket_name, key: new_key)
yield(old_key: old_key, new_key: new_key, obj: obj, prefix: prefix) if block_given?
end
self
end
Or perhaps I have this structured completely wrong within the method? Any pointers would be appreciated.
Related
I'm new to AWS and I'm trying to loop through my bucket objects to generate urls to read the objects. I found the following reference in the AWS documentation:
bucket.objects.myobject.url_for(:read)
I have the following method which contains a loop that can at least print the key for each object BUT I'm struggling to get the url_for to work. Any ideas?
def aws_s3_url
s3_client = Aws::S3::Resource.new(region: ENV['AWS_REGION'])
bucket = s3_client.bucket(ENV['S3_BUCKET'])
bucket.objects.each do |name|
puts name.key
end
end
All help is appreciated.
I don't know the specific use-case you have, but you don't need the URL to read the objects in the bucket since the AWS SDK maps the files in the bucket to instances of Object.
To read the contents of the file, try this:
#s3_client = Aws::S3::Resource.new(region: ENV['AWS_REGION'])
def file_content(key)
bucket = #s3_client.bucket(ENV['S3_BUCKET'])
obj = #s3_client.get_object(bucket: bucket, key: key)
obj.body.read
end
def get_all_files
bucket = #s3_client.bucket(ENV['S3_BUCKET'])
bucket.objects.each do |o|
puts file_content(o.key)
end
end
To return the public URL for the object, you can try:
Aws::S3::Object.public_url:
def get_url(key)
bucket = #s3_client.bucket(ENV['S3_BUCKET'])
obj = #s3_client.get_object(bucket: bucket, key: key)
obj.public_url
end
I am working on a CLI Project and trying to open up a web page by using url variable declared in another method.
def self.open_deal_page(input)
index = input.to_i - 1
#deals = PopularDeals::NewDeals.new_deals
#deals.each do |info|
d = info[index]
#product_url = "#{d.url}"
end
#product_url.to_s
puts "They got me!"
end
def self.deal_page(product_url)
#self.open_deal_page(input)
deal = {}
html = Nokogiri::HTML(open(#product_url))
doc = Nokogiri::HTML(html)
deal[:name] = doc.css(".dealTitle h1").text.strip
deal[:discription] = doc.css(".textDescription").text.strip
deal[:purchase] = doc.css("div a.button").attribute("href")
deal
#binding.pry
end
but I am receiving this error.
`open': no implicit conversion of nil into String (TypeError)
any possible solution? Thank you so much in advance.
Try returning your #product_url within your open_deal_page method, because now you're returning puts "They got me!", and also note that your product_url is being created inside your each block, so, it won't be accessible then, try creating it before as an empty string and then you can return it.
def open_deal_page(input)
...
# Create the variable
product_url = ''
# Assign it the value
deals.each do |info|
product_url = "#{info[index].url}"
end
# And return it
product_url
end
In your deal_page method tell to Nokogiri to open the product_url that you're passing as argument.
def deal_page(product_url)
...
html = Nokogiri::HTML(open(product_url))
...
end
I am trying to list only the objects from the s3 folder (not a real folder I know) called distribution but I want to remove the reference to the name and any slashes around the object. The output should just look like 021498cd-ca73-4675-a57a-c12b3c652aac whereas currently it looks like distribution/021498cd-ca73-4675-a57a-c12b3c652aac/
So far I have tried;
def files
s3 = Aws::S3::Resource.new
s3.client
bucket = s3.bucket('test')
files = []
bucket.objects.each do |obj|
if obj.key.include?('distribution/')
temp_files = puts "#{obj.key}"
files = temp_files.select do |file|
file.gsub("distribution/", "")
end
else
end
end
end
But this doesn't seem to be working at all.
Your explanation is pretty simple but your code is implying something else.
However, this should help with what you are trying to achieve.
def files
s3 = Aws::S3::Resource.new
s3.client
bucket = s3.bucket('test')
files = []
bucket.objects.each do |obj|
if obj.key.include?('distribution/')
files << "#{file.gsub(/(distribution)|\//, '')}"
end
end
end
The files array will contain all the file names with garbage stripped.
When I do for a single file it works:
aws_s3 = AWS::S3.new(S3_CONFIG)
bucket = aws_s3.buckets[S3_CONFIG["bucket"]]
object = bucket.objects["user/1/photos/image_1.jpg"]
new_object = bucket.objects["users/1/photos/image_1.jpg"]
object.copy_to new_object, {:acl => :public_read}
But I want to move the entire "/photos" folder throws No Such Key. Probably the s3 keys are only the full path for each file. How to do that?
aws_s3 = AWS::S3.new(S3_CONFIG)
bucket = aws_s3.buckets[S3_CONFIG["bucket"]]
object = bucket.objects["user/1/photos"]
new_object = bucket.objects["users/1/photos"]
object.copy_to new_object, {:acl => :public_read}
Thanks!
Did it:
bucket.objects.with_prefix("user/1/photos").each do |object|
...
end
I needed additional code to get this working. Basically chop off the base from the source prefix, then add that to the destination prefix:
def copy_files_s3(bucket_name, source, destination)
source_bucket = #s3.buckets[bucket_name]
source_bucket.objects.with_prefix(source).each do |source_object|
new_file_name = source_object.key.dup
new_file_name.slice! source
new_object = source_bucket.objects["#{destination}#{new_file_name}"]
source_object.copy_to new_object, {acl: :public_read}
end
end
A "folder" is not an object in S3, that is why you can not get it by key, but the folder path is actually a prefix for all the keys of the objects contained by the folder.
Another important thing, you have to url encode the keys otherwise you may end up with an unknown key error.
require 'aws-sdk'
require 'aws-sdk-s3'
require 'securerandom'
require 'uri'
require "erb"
include ERB::Util
def copy_folder(folder, destination)
bucket_name = 'your_bucket'
credentials = Aws::Credentials.new('key', 'secret')
s3_client = Aws::S3::Client.new(region:'the_region', credentials: credentials)
enumerate_keys_with_prefix(source).each do |source_object|
source_key = url_encode(source_object.key)
destination_key = source_object.key.dup.sub(source, "")
s3_client.copy_object({bucket: bucket_name, copy_source: bucket_name+'/'+source_key, key: destination+'/'+destination_key, acl: "public-read"})
end
end
def enumerate_keys_with_prefix(prefix)
bucket_name = 'your_bucket'
credentials = Aws::Credentials.new('key', 'secret')
s3 = Aws::S3::Resource.new(region:'the_region', credentials:credentials)
return s3.bucket(bucket_name).objects(prefix: prefix)
end
I have this code that tries to return a value from a method:
temp = "123"
return temp
and I have this line that calls the method and assigns the return value:
person_connections = #client.get_person_connections(:id => current_user_id )
but when I try to inspect person_connections, it shows some different object string. Any idea how to return the actual value of the temp variable?
def get_person_connections(options = {})
person_id = options[:id]
path = "/people/id=" + person_id + ":(num-connections)"
query_connections(path, options)
self
end
and
private
def query_connections(path, options={})
fields = options.delete(:fields) || LinkedIn.default_profile_fields
if options.delete(:public)
path +=":public"
elsif fields
path +=":(#{fields.map{ |f| f.to_s.gsub("_","-") }.join(',')})"
end
headers = options.delete(:headers) || {}
params = options.map { |k,v| v.is_a?(Array) ? v.map{|i| "#{k}=#{i}"}.join("&") : "#{k}=#{v}" }.join("&")
path += "?#{params}" if not params.empty?
temp_var = get(path, headers)
hash = JSON.parse(temp_var)
conn = hash["numConnections"]
end
As Samy said in a comment:
In Ruby, the last statement will be returned.
So if we take a look at get_person_connections, we see that the last line is self. What it means is that it returns the instance on which the method was called, #client in this case.
Additional notes: the solution would be to remove self, although if the method is used elsewhere be careful as returning self is often used to allow chaining of methods (though it hardly makes sense to do that on a get method).