How to select thousands of file in chef recipe - ruby

I have a lot of csv files that should be placed in a server.
So I put it in a directory my-cookbook/files/default/ and I want to load all csv files by using Dir.glob method, but I don't know how to set proper relative path for it.
Dir.glob("#{relative_path}/*.csv").each do |file|
cookbook_file "/var/foo/#{File.basename(file)}" do
source File.basename(file)
end
end
I tried with Dir.glob("*.csv"), but it didn't work.
How can I load all files in files/default directory?

This isn't really how Chef is designed to be used. You would be better off putting all your files in something like a git repo or remote tarball and using things like the git or remote_file resources to download them. cookbook_file is really only a hack for small, one-off things. Honestly I don't really use it much, I would recommend you do the same.

Related

How DASK_jobqueue SLURMCluster can access local python module in parent directory

I'm using dask_jobqueue to establish a SLURMCluster. I'm trying to pass python files in the parent directory to the workers. I tried different ways including sys.path.append, setting PYTHONPATH in my .bashrc file, and setting PYTHONPATH in env_extra (it seems DASK version can't recognize job_script_prologue), but they all fail. The script works only when the imported Python files are in the same directory as the script. I just wonder if this is really the case (which means I should re-write my code to keep all files in the same directory), or I'm missing something here?
Also, I tried to use with performance_report to record some statistics in an HTML file, but it seems it does not work with dask_jobqueue, right?

What is the standard way to describe the paths to config files of my ruby application (linux)?

What is the standard way to describe the paths to config files of my ruby application (linux)? (not rails, but I think it doesn't matter right now)
How does this "fit" into standard paths like /etc/ and /usr/local/etc/?
Let's say I want to put config in /usr/local/etc/myapp/config.json during global installation,
but in $HOME/.config/myapp/config.json during local installation.
I want to provide the default configuration file with my application, not pass it from somewhere else (although this is also possible, but only as an option).
Can gemspec do this? I found in the description of Gem::Specification only paths to bin and exe files, but not to configuration files.

Chef, related path from cookbook to file storage

Is it possibile to get absolute path to cookbook directory(I don't need path to cache).
What I want to do?
My structure:
Chef
-Cookbooks(tomcat, java, etc.)
-Profiles(json files with configs)
-Resources(zip files with programs)
runScript0.bat
runScript1.bat
....
How can I get path to Resources folder from recipes, I need to get files from this folder w/o hardcodded path. Of course I can download programs from svn/git, but I need this option too.
The structure you describe is not inside a cookbook so those files do not exist as far as Chef is concerned. Even inside a cookbook, you need to use a cookbook_file or remote_directory resource to write to a known place first.

What is a usual workflow for dealing with shared files with capistrano?

My .csv recreates on every release, and, as I understand, to keep its data unchanged between deploys I need to put it in /shared directory and simlink to it from my deploy.rb.
Is this the right route? (I have this question because I don't seem to find much info on how to do this with respect to, eg, databases, for some reason. /shared directory is mostly used for .conf files and paperclip-like directories).
When using capistrano, your application code will be "uploaded" to some directory on the server. Capistrano uses this structure:
/path_to_folder:
current - symlink to the directory with the current release
releases - contains all kept releases
shared - files that should persist between releases
So to your question - copy the .csv file somewhere into "shared" directory and then in the config/deploy.rb add this:
namespace :deploy do
task :create_symlinks do
run "ln -s #{shared_path}/something.csv #{latest_release}/db/something.csv"
end
end
after 'deploy:update_code', 'deploy:create_symlinks'
Replace "something" with the file name that you copied. You can also put the csv file into some directory under "shared" if you want to, I'd use "db" in this case. If you do so, don't forget to update path in the symlink.

Extra Copy of New Rsync Files

I am attempting to mirror a directory on a remote server using rsync. However, I would like a copy of all newly created files to be stored in a separate directory on the local machine.
For example, if a new file is added on the remote server, I would like it to mirror regularly (for example, to ~/mirror), but save an additional copy of only the new file in another folder, (for example, ~/staging). To be clear, only the new files should appear in staging.
My first approach was to allow rsync to update the timestamps, and then use that to make a copy. However, I would now like to preserve timestamps.
Can anyone provide ideas on a simple approach? I am open to use of additional utilities other than rsync.
You might consider making hardlinks in the extra directory.
ln --force --target-directory=~/staging ~/mirror/*
Edit:
If this is a Linux system, incron will trigger on inotify events and would allow you to make copies of files as they are added to a directory you specify.

Resources