How do I copy all the files present in an s3 directory(same prefix) to another directory in the same bucket using fog?
For eg: Copy all files with prefix <bucket>/foo/ to <bucket>/bar/
I don't think there is a direct way to do that per se, and that instead you would need to iterate over the appropriate files to do the move. I think it would look something like this:
require 'rubygems'
require 'fog'
# create a connection
connection = Fog::Storage.new({
provider: 'AWS',
aws_access_key_id: YOUR_AWS_ACCESS_KEY_ID,
aws_secret_access_key: YOUR_AWS_SECRET_ACCESS_KEY
})
directory = connection.directories.get(BUCKET, prefix: '/foo/')
directory.files.each do |file|
file.copy(BUCKET, "/bar/#{file.key.split('/').last}")
end
Related
I would like to copy http://seapower/spring.txt and http://seapower/has_sprung.txt and append second one to the first one in a new file named src_filepath.txt:
remote_file 'src_filepath.txt' do
source 'http://seapower/spring.txt', 'http://seapower/has_sprung.txt'
checksum node['nginx']['foo123']['checksum']
owner 'root'
group 'root'
mode '0755'
end
It doesn't work and just copy the first file to src_filepath.txt
Something like this is probably a good place to start and then tweak however you like:
cache1 = "#{Chef::Config[:file_cache_path]}/content1"
cache2 = "#{Chef::Config[:file_cache_path]}/content2"
# this will not redownload if cache1 exists and has not been updated
remote_file cache1 do
source "http://source.url/content1"
end
# this will not redownload if cache1 exists and has not been updated
remote_file cache2 do
source "http://source.url/content2"
end
# this will not update the file if the contents has not changed
file "/my/combined/file" do
content lazy { IO.read(cache1) + IO.read(cache2) }
end
This is not something Chef supports directly. You could use multiple remote_file resources and either a ruby_block or execute plus cat to implement the concat.
remote_file does not support concatenation, so you would not be able to implement this using that resource directly, however you could piece together the desired result using the file resource and Net::HTTP like so:
file_path = '/path/to/your_whole_file'
unless File.exist?(file_path) &&
Digest::SHA256.hexdigest(File.read(file_path)) == 'your_file_checksum'
file file_path do
content(
Net::HTTP.get(URI('http://source.url/content1')) +
Net::HTTP.get(URI('http://source.url/content2'))
)
owner 'root'
group 'root'
mode '0755'
end
end
The reason for the Digest::SHA256 call at the beginning is to prevent Chef from trying to download both files during every Chef run. Note that you may have to require the net/http and digest gems at the top of your recipe for this to work.
Also, because it's against best practices to put Ruby code directly into your recipes, you may want to wrap the above code in a simple custom resource.
I am trying to read a file's contents and use it in my ruby code. In this step, I am not trying to do anything on the bootstrapped node. All I want to do is read a JSON file that will reside in cookbook's files folder and read the contents of the file and do something. I just want to use the value coming from JSON in my code itself. The code example is shown below. Any help is appreciated.
Attributes: default.rb
default["xyz"]["ohs_servers"]=[
{"hostname"=> "intf301.linux.xyz.com","name" => "INTFIN_OHS_001", "short_name" => "OGS", "port" => "9931"},
{"hostname"=> "intf302.linux.xyz.com","name" => "INTFIN_OHS_001", "short_name" => "OHS", "port" => "9931"}
]
Machines: machines.rb
require 'rubygems'
require 'json'
require 'pp'
json = File.read('environment.json')
obj = JSON.parse(json)
number = obj["name"]
x = node["xyz"]["ohs_servers"][number]["hostname"]
JSON file in cookbook's files folder: environment.json
{
"template_name": "environment_template",
"number": 0
}
Even if I don't really get why you don't want to use attributes for this:
What you want is to ensure the cookbook files are in the cache even if there's no resource calling them, the way to go is to configure the client.rb on the node with the no_lazy_load attribute to true
Quoting the documentation about this option:
no_lazy_load Use to download all cookbook files and templates at the
beginning of the chef-client run. Default value: true.
I'm unsure if the default value has changed with 12 or on wich version, but I'm quite sure it was false in chef 11 (loading file or template when the provider referencing them is called)
Then you can read your file using
File::read("#{Chef::Config['file_cache_path']}/cookbooks/my_cookbook/files/my_file.json")
Edit: Just saw the comment of Stephen King, I more or less paraphrased Seth Vargo's answer here :/
use cookbook_file and then add run_action(:create)
cookbook_file "myfile.txt" do
path "somepathyouwantthefilebe/myfile.txt"
source "myfile.txt" #the name of the file in files folder of your cookbook"
end.run_action(:create) # read notes** bellow
then you can have some ruby code to read from it
for example
File::read("somepathyouwantthefilebe/myfile.txt")
** the run action is nessecary since you are combining ruby code and resources in chef-zero
remote_file block copies only one specific file.
Is there any possibility in Chef to copy all files from specific folder on ftp?
my current code is quite weird as for me:
require 'net/ftp'
ftp = Net::FTP::new("server")
ftp.login("user", "password")
ftp.chdir("/folder")
fileList = ftp.nlst('*.jar')
fileList.each do |file|
remote_file "C:\\Temp\\" + file do
source "ftp://user:password#server/folder/" + file
action :create_if_missing
end
end
ftp.close
If your solution works, why not wrap it in an LWRP? They are quite easy to create, and would tuck away the implementation in its own file. This is what I would do.
See: http://docs.opscode.com/chef/lwrps_custom.html
And for a real-life - easy to understand - example, see:
https://github.com/opscode-cookbooks/ssh_known_hosts/blob/master/providers/entry.rb
https://github.com/opscode-cookbooks/ssh_known_hosts/blob/master/resources/entry.rb
I built a ruby gem with a binary. I use like this:
myruby "param"
It is a helper for building integration, and needs a setting for each project. I have settings in settings.rb for several projects. Is it possible to require a .rb file based on the present working dir? When I run:
/home/usr/admin/sources/myproject1/ $ myruby start
I want it to require the settings from:
/home/usr/admin/sources/myproject1/settings.rb
How could I do this if it's possible? I tried:
require_relative '#{Dir.pwd}/settings.rb'
which did not work.
File.expand_path('../', __FILE__)
gives you the path to the current directory. Thus if you have a file in bin/foo and you want to require something in lib/foo/settings.rb simply use
require File.join(File.expand_path('../../'), __FILE__), 'lib/foo/settings.rb')
Note the double ../ because the first is required to strip out from __FILE__ the current filename.
If the file is in /home/usr/admin/sources/myproject1/bin/foo
File.expand_path('../', __FILE__)
# => /home/usr/admin/sources/myproject1/bin
File.expand_path('../../', __FILE__)
# => /home/usr/admin/sources/myproject1
File.join(File.expand_path('../../', __FILE__), 'lib/foo/settings.rb')
# => /home/usr/admin/sources/myproject1/lib/foo/settings.rb
If you want to include the file with a relative path from the working directory, use
require File.join(Dir.pwd, 'settings.rb')
However, I don't think it's a good idea to hard-code a path in this way. You may probably want to pass the settings as argument to the command line.
It doesn't really make sense to create a gem that depends on a path of a file hard-coded on your machine.
File.expand_path(File.dirname(__FILE__)) will return the directory relative to the file this command is called from.
The difference between require and require_relative is that require_relative loads files from directory relative to the file where it is written. While require searches for files in directories from $LOAD_PATH if given a relative path or can load files from absolute path, which can be created with File.expand_path
I have a bunch of files on s3. I have fog set up with a .fog config file so I can fire up fog and get a prompt. Now how do I access and edit a file on s3, if I know its path?
The easiest thing to do is probably to use IRB or PRY to get a local copy of the file, or write a simple script to download, edit and then re-upload it. Assume you have a file named data.txt.
You can use the following script to initialize a connection to S3.
require 'fog'
connection = Fog::Storage.new({
:provider => 'AWS',
:aws_secret_access_key => YOUR_SECRET_ACCESS_KEY,
:aws_access_key_id => YOUR_SECRET_ACCESS_KEY_ID
})
directory = connection.directories.get("all-my-data")
Then use the directory object to get a copy of your file on your local file-system.
local_file = File.open("/path/to/my/data.txt", "w")
file = directory.files.get('data.txt')
local_file.write(file.body)
local_file.close
Edit the file using your favorite editor and then upload it to S3 again.
file = directory.files.get('data.txt')
file.body = File.open("/path/to/my/data.txt")
file.save