I have a bunch of files on s3. I have fog set up with a .fog config file so I can fire up fog and get a prompt. Now how do I access and edit a file on s3, if I know its path?
The easiest thing to do is probably to use IRB or PRY to get a local copy of the file, or write a simple script to download, edit and then re-upload it. Assume you have a file named data.txt.
You can use the following script to initialize a connection to S3.
require 'fog'
connection = Fog::Storage.new({
:provider => 'AWS',
:aws_secret_access_key => YOUR_SECRET_ACCESS_KEY,
:aws_access_key_id => YOUR_SECRET_ACCESS_KEY_ID
})
directory = connection.directories.get("all-my-data")
Then use the directory object to get a copy of your file on your local file-system.
local_file = File.open("/path/to/my/data.txt", "w")
file = directory.files.get('data.txt')
local_file.write(file.body)
local_file.close
Edit the file using your favorite editor and then upload it to S3 again.
file = directory.files.get('data.txt')
file.body = File.open("/path/to/my/data.txt")
file.save
Related
How do I copy all the files present in an s3 directory(same prefix) to another directory in the same bucket using fog?
For eg: Copy all files with prefix <bucket>/foo/ to <bucket>/bar/
I don't think there is a direct way to do that per se, and that instead you would need to iterate over the appropriate files to do the move. I think it would look something like this:
require 'rubygems'
require 'fog'
# create a connection
connection = Fog::Storage.new({
provider: 'AWS',
aws_access_key_id: YOUR_AWS_ACCESS_KEY_ID,
aws_secret_access_key: YOUR_AWS_SECRET_ACCESS_KEY
})
directory = connection.directories.get(BUCKET, prefix: '/foo/')
directory.files.each do |file|
file.copy(BUCKET, "/bar/#{file.key.split('/').last}")
end
I am trying to read a file's contents and use it in my ruby code. In this step, I am not trying to do anything on the bootstrapped node. All I want to do is read a JSON file that will reside in cookbook's files folder and read the contents of the file and do something. I just want to use the value coming from JSON in my code itself. The code example is shown below. Any help is appreciated.
Attributes: default.rb
default["xyz"]["ohs_servers"]=[
{"hostname"=> "intf301.linux.xyz.com","name" => "INTFIN_OHS_001", "short_name" => "OGS", "port" => "9931"},
{"hostname"=> "intf302.linux.xyz.com","name" => "INTFIN_OHS_001", "short_name" => "OHS", "port" => "9931"}
]
Machines: machines.rb
require 'rubygems'
require 'json'
require 'pp'
json = File.read('environment.json')
obj = JSON.parse(json)
number = obj["name"]
x = node["xyz"]["ohs_servers"][number]["hostname"]
JSON file in cookbook's files folder: environment.json
{
"template_name": "environment_template",
"number": 0
}
Even if I don't really get why you don't want to use attributes for this:
What you want is to ensure the cookbook files are in the cache even if there's no resource calling them, the way to go is to configure the client.rb on the node with the no_lazy_load attribute to true
Quoting the documentation about this option:
no_lazy_load Use to download all cookbook files and templates at the
beginning of the chef-client run. Default value: true.
I'm unsure if the default value has changed with 12 or on wich version, but I'm quite sure it was false in chef 11 (loading file or template when the provider referencing them is called)
Then you can read your file using
File::read("#{Chef::Config['file_cache_path']}/cookbooks/my_cookbook/files/my_file.json")
Edit: Just saw the comment of Stephen King, I more or less paraphrased Seth Vargo's answer here :/
use cookbook_file and then add run_action(:create)
cookbook_file "myfile.txt" do
path "somepathyouwantthefilebe/myfile.txt"
source "myfile.txt" #the name of the file in files folder of your cookbook"
end.run_action(:create) # read notes** bellow
then you can have some ruby code to read from it
for example
File::read("somepathyouwantthefilebe/myfile.txt")
** the run action is nessecary since you are combining ruby code and resources in chef-zero
I'm trying to read a CSV file directly from s3.
I'm getting the s3 URL but I am not able to open it as it's not in the local system. I don't want to download the file and read it.
Is there any other way to achieve this?
There are few ways, depending on the gems that you are using. For example, one of the approaches from official documentation:
s3 = Aws::S3::Client.new
resp = s3.get_object(bucket:'bucket-name', key:'object-key')
resp.body
#=> #<StringIO ...>
resp.body.read
#=> '...'
Or if you are using CarrierWave/Fog:
obj = YourModel.first
content = obj.attachment.read
You can open the file from URL directly:
require 'open-uri'
csv = open('http://server.com/path-to-your-file.csv').read
I think s3 doesn't provide you any way of reading the file without downloading it.
What you can do is save it in a tempfile:
#temp_file = Tempfile.open("your_csv.csv")
#temp_file.close
`s3cmd get s3://#{#your_path} #{#temp_file.path}`
For further information: http://www.ruby-doc.org/stdlib-1.9.3/libdoc/tempfile/rdoc/Tempfile.html
remote_file block copies only one specific file.
Is there any possibility in Chef to copy all files from specific folder on ftp?
my current code is quite weird as for me:
require 'net/ftp'
ftp = Net::FTP::new("server")
ftp.login("user", "password")
ftp.chdir("/folder")
fileList = ftp.nlst('*.jar')
fileList.each do |file|
remote_file "C:\\Temp\\" + file do
source "ftp://user:password#server/folder/" + file
action :create_if_missing
end
end
ftp.close
If your solution works, why not wrap it in an LWRP? They are quite easy to create, and would tuck away the implementation in its own file. This is what I would do.
See: http://docs.opscode.com/chef/lwrps_custom.html
And for a real-life - easy to understand - example, see:
https://github.com/opscode-cookbooks/ssh_known_hosts/blob/master/providers/entry.rb
https://github.com/opscode-cookbooks/ssh_known_hosts/blob/master/resources/entry.rb
I am writing an application in Ruby on Elastic Beanstalk in which I download a file from a remote server and write it to an object in a bucket.
require 'open-uri'
...
s3 = AWS::S3.new
bucket = s3.buckets['mybucket']
f = open(params[:url]) #using open-uri
obj = bucket.objects[params[:key]]
obj.write[f] #<< fails here
The last line, however, fails with the following exception in the log:
:data must be provided as a String, Pathname, File, or an object that responds to #read and #eof?
I know, however, from executing the same #open on my machine, that f is a StringIO object, which does have #read and #eof?.
I was getting same error during zip file upload on S3 and finally this worked for me:
zip_data = File.read(zip_file_path)
means, zip_data will be the object at the zip file path that is located in your tmp directory.
Hope, this will work for you also.