I am trying to read around 1.5GB size from s3 using ruby sdk for aws.
S3 gem is - gem 'aws-sdk' ( https://github.com/aws/aws-sdk-ruby )
most_recent_s3_object = s3_resource.bucket(bucket_name).objects.max_by(&:last_modified)
Here most_recent_s3_object class is Aws::S3::ObjectSummary
When I do most_recent_s3_object.get I get broken pipe error.
Is there a way around to get this file in a whole for this size?
Related
I'm trying to get the secrets from SSM Parameter store. The issue is we're on aws-sdk-v1 (ruby). For V2, V3 I can get plenty of examples, but not for V1. e.g. code snippet for aws-sdk--v2.
ssm_client = Aws::SSM::Client.new(
region: region
)
param_response = ssm_client.get_parameter(
name: parameter_id,
with_decryption: true
).to_h
Do anyone know how to do it if I'm on aws-sdk-v1.
PS: Upgrading from aws-sdk V1 to V2/V3 is not the viable options, please suggest considering the solution should run on aws-sdk-v1.
Ruby version: '1.9.3'
SSM is already present in the aws-sdk-v1.
There was a dependency issue in my Gemfile. I was using aws-sdk-resources ~> 3, which is fundamentally incompatible with aws-sdk ~> 1.
Is it possible to get the machine specs from the instance type?
get_spec("t1.small") => {CPU:64, RAM:8 ....HVM:true}
is there such kind of method?
The Amazon EC2 API does not expose these stats programmatically. For quick reference I tend to use: http://www.ec2instances.info/
They have a static JSON file you can fetch programmatically:
http://www.ec2instances.info/instances.json
No, there is no this type of machine specs you can get directly from aws ruby SDK.
But you can develop this function by yourself to refer the url Amazon EC2 Instances types. Build an input yaml file with full informations, and search key word t1.small .
instance:
t2.small:
cpu: 1
mem: 2
t2.micro:
cpu: 1
mem: 1
with the ruby code, you can use the gem of yaml
$ irb
irb(main):001:0> require 'yaml'
=> true
irb(main):002:0> config=YAML::load_file('instances.yaml')
=> {"instance"=>{"t2.small"=>{"cpu"=>1, "mem"=>2}, "t2.micro"=>{"cpu"=>1, "mem"=>1}}}
irb(main):003:0> config['instance']['t2.small']
=> {"cpu"=>1, "mem"=>2}
I am trying to download a track from Soundcloud using the ruby sdk (soundcloud 0.2.0 gem) with an app. I have registered the app on soundcloud and the client_secret is correct. I know this because I can see my profile info and tracks using the app.
Now when I try to download a track using the following code
#track = current_user.soundcloud_client.get(params[:track_uri])
data = current_user.soundcloud_client.get(#track.download_url)
File.open("something.mp3","wb"){|f|f.write(data)}
and when I open the file it has nothing in it. I've tried many approaches including the following one,
data = current_user.soundcloud_client.get(#track.download_url)
file = File.read(data)
And this one gives me an error
can't convert nil into String
on line 13 which is in
app/controllers/store_controller.rb:13:in `read'
that is the File.read function.
I have double checked that the track I am trying to download is public and downloadable.
I tried to test the download_url that is being used explicitly by copying it from console and sending a request using Postman and it worked. I am not sure why it is not working with the app when other things are working so well.
What I want to do is to successfully be able to either download or at least get the data which I could store somewhere.
Version details : -
ruby 1.9.3p194 (2012-04-20 revision 35410) [x86_64-linux]
Rails 3.2.18
soundcloud 0.2.0
There are few assumptions that you have to understand before doing this thing.
Not every track on SoundClound can be downloaded! Only tracks that are flagged as downloadable can be downloaded - your code has to consider that option!
Your track URL has to be "resolved" before you get to download_url and after you get download_url you have to use your client_id to get the final download URL.
Tracks can be big, and downlowding them requires time! You should never do tasks like this straight from your Rails app in your controller or model. If the tasks runs longer you always use some background worker or some other kind of background processing "thing" - Sidekiq for example.
Command-line client example
This is example of working client, that you can use to download tracks from SoundClound. Its using official Official SoundCloud API Wrapper for Ruby, assumes that you are using Ruby 1.9.x and its not dependent on Rails in any way.
# We use Bundler to manage our dependencies
require 'bundler/setup'
# We store SC_CLIENT_ID and SC_CLIENT_SECRET in .env
# and dotenv gem loads that for us
require 'dotenv'; Dotenv.load
require 'soundcloud'
require 'open-uri'
# Ruby 1.9.x has a problem with following redirects so we use this
# "monkey-patch" gem to fix that. Not needed in Ruby >= 2.x
require 'open_uri_redirections'
# First there is the authentication part.
client = SoundCloud.new(
client_id: ENV.fetch("SC_CLIENT_ID"),
client_secret: ENV.fetch("SC_CLIENT_SECRET")
)
# Track URL, publicly visible...
track_url = "http://soundcloud.com/forss/flickermood"
# We call SoundCloud API to resolve track url
track = client.get('/resolve', url: track_url)
# If track is not downloadable, abort the process
unless track["downloadable"]
puts "You can't download this track!"
exit 1
end
# We take track id, and we use that to name our local file
track_id = track.id
track_filename = "%s.aif" % track_id.to_s
download_url = "%s?client_id=%s" % [track.download_url, ENV.fetch("SC_CLIENT_ID")]
File.open(track_filename, "wb") do |saved_file|
open(download_url, allow_redirections: :all) do |read_file|
saved_file.write(read_file.read)
end
end
puts "Your track was saved to: #{track_filename}"
Also note that files are in AIFF (Audio Interchange File Format). To convert them to mp3 you do something like this with ffmpeg.
ffmpeg -i 293.aif final-293.mp3
I am attempting to download a file from a private S3-bucket, through Boxen puppet scripts. However, I haven't found any examples how to do so. All I found was readme's discussing the environment variables (which I set).
But how can I download an archive from S3 and install it locally? Any good examples? Is this done through homebrew or a puppet script?
Thanks
puppet-minecraft does this with a publicly reachable AWS S3 bucket. Perhaps it can help you.
Check out the manifest init.pp >here<, where you'll find this snippet showing the URL that's grabbing the item from AWS S3.
package { 'Minecraft':
source => 'https://s3.amazonaws.com/Minecraft.Download/launcher/Minecraft.dmg',
provider => 'appdmg'
}
Mincraft isn't the only example. I found others in my repo by running on my Mac.:
mdfind -onlyin /opt/boxen/repo/shared s3 | grep manifest
/opt/boxen/repo/shared/vmware_fusion/manifests/init.pp
/opt/boxen/repo/shared/ruby/manifests/version.pp
/opt/boxen/repo/shared/minecraft/manifests/init.pp
/opt/boxen/repo/shared/java/manifests/init.pp
/opt/boxen/repo/shared/istatmenus4/manifests/init.pp
/opt/boxen/repo/shared/heroku/manifests/init.pp
/opt/boxen/repo/shared/github_for_mac/manifests/init.pp
i am able to zip all required files(using ruby gems of AWS SDK) & upload them to the S3 bucket. when i tried to download the zip files from S3 to local server & unzip them , i am geeting following error...
[2013-05-06T07:19:37+00:00] FATAL: TypeError: aws_unzip[db_unzip] (aws::unzip line 14) had an error: TypeError: can't dup NilClass
Even when i try to unzip manually, i can see "zip files is corrupted"...But the zip file which is present in S3 location is not corrupted(i tested to extract them manually by downloading them, it extracted well)....
Can any one help where i am doing mistake in reading zip files from bucket to local server?????
When downloading it is getting corrupted...
My code is
File.open(dd, 'w') {|f| f.write(obj.read.force_encoding('utf-8'))}
Try 'wb' mode - zip is a binary file. Also that force_encoding is very suspicious.