How do I share a single file, instead of sharing a whole folder like
config.vm.synced_folder "host/folder", "box/folder"
?
In other words is there a way to have something like:
config.vm.synced_folder "host/folder/file.conf", "box/folder/file.conf"
The accepted answer (by cdmo) didn't work for me, but was close and led me to the right solution, so cheers. To copy just one file I needed to change it to:
config.vm.synced_folder "c:/hostpath/", "/guestpath/", type: "rsync",
rsync__args: ["-r", "--include=file.text", "--exclude=*"]
Mind the arguments and their order, -r MUST be present and --include MUST precede --exclude.
If needed, instead of -r option you may use -a option which combines -r with several other options for preservation of permissions, ownership etc. (see rsync help).
My testing configuration: Vagrant 2.0.2/Win10/VirtualBox 5.2.6
Use the include flag in your rsync args array:
config.vm.synced_folder "host/folder/", "box/folder/", type: "rsync",
rsync__args: ["--include=file.conf"]
You can use the file provisioner to do this:
Vagrant.configure("2") do |config|
# ... other configuration
config.vm.provision "file", source: "~/.gitconfig", destination: ".gitconfig"
end
You can't synchronize a single file. But maybe you can achieve the same effect by synchronizing the folder using RSync with the rsync__exclude option:
rsync__exclude (string or array of strings) - A list of files or directories to exclude from the sync. The values can be any acceptable rsync exclude pattern.
Vagrant by default utilizes the file/folder synchronization mechanisms offered by the provider technologies (e.g., VirtualBox, VMWare, etc.). If the providers can't do synchronization on a per-file basis, then Vagrant can't either. Vagrant can also use NFS, RSync, or SMB for file synchronization. However, looking at the code, it would appear that each expects the per-folder paradigm.
Another possibility is to use one of the provisioning options to selectively synchronize the appropriate file(s). If you want to take the easy way out you could use the shell provisioner to just copy the file you need. For something more interesting, this is a reasonable guide to getting started with Puppet in Vagrant, and you might be interested in the puppet file Type.
As Benjamin Mosior explained, you can only synchronise a whole folder.
As a workaround, I ended up synchronising the whole folder to a temporary folder and in a later step creating a symlink where I need it to the single file inside this temporary folder.
Related
I have a playbook with the following task that must copy the 2 Gb file from local to remote servers and extract files:
- name: Copy archived file to target server and extract
unarchive:
src: /path_to_source_dir/file.tar.gz
dest: /path_to_dest_dir
This task fails because ansible copies file to /home mount point on the target server and there's not enough space there:
sftp> put /path_to_source_dir/file.tar.gz /home/my_user_name/.ansible/tmp/ansible-tmp-1551129648.53-14181330218552/source
scp: /home/my_user_name/.ansible/tmp/ansible-tmp-1551129648.53-14181330218552/source: No space left on device
The reason for that is because ansible.cfg has a default parameter:
remote_tmp = ~/.ansible/tmp
How to overwrite this parameter from the playbook (if possible) and make ansible to copy file to the same destination directory specified in the task? So it would be like this:
remote_tmp = /path_to_dest_dir/.ansible/tmp
And the destination path is going to be different each time for a different target server!
Cleaning /home is not an option for me.
The answer here unfortunately is not very clear to me.
There are a few different ways to achieve what you are looking to do. Which one is a matter of preference and your use case.
You found the first way, setting an environment variable before running the playbook. Great for a quick on-the-fly job. Remembering to do that every time you run a certain playbook is indeed annoying. A slight variation of that is to use the environment keyword to set that variable for the play. You can also set environment variable in a role, block or a single task. https://docs.ansible.com/ansible/devel/reference_appendices/playbooks_keywords.html?highlight=environment%20directive. Here is an example of it in use: https://docs.ansible.com/ansible/devel/reference_appendices/faq.html?highlight=environment.
Using the environment keyword in a play et al works well for a specific application of automation, but what if you want Ansible to always use a different remote tmp path for specific servers? Like all variables, the remote_tmp can be sourced from inventory host and group variables not just the config file or environment variables. You need to mind you variable precedence if it is being set in different places. With this you could set remote_tmp in your inventory for that host or a group of hosts. Ansible will always use that path for that host or hosts in that group without having to define it in every play or roles. If you need to change that path, you change it in your inventory and it changes the behavior for all playbook runs without any additional edits. Here is an example of it being used as a host variable in static inventory: https://docs.ansible.com/ansible/devel/reference_appendices/faq.html?highlight=remote_tmp#running-on-solaris
So while the specific issue of "dynamically" setting the remote tmp directory on a host is not a best practice topic per se, it does become an example of the best practice of making the most of variables in Ansible.
For reference, remote temporary directories are handled by the shell plugins. While many parameters are shared, there are others that are specific to the shell Ansible using. Ansible uses sh by default. https://docs.ansible.com/ansible/latest/plugins/shell/sh.html
Hope that helps. Happy automating.
I use ~/.ssh/config to manage hosts that I need to interact with frequently. Sometimes I would like to access the aliases from that file in scripts that do not use ssh directly, ie if I have a .ssh/config with
Host host1
User user1
Hostname server1.somewhere.net
I would like to be able to say something like sshcfg['host1'].Hostname to get server1.somewhere.net in scripting languages, particularly Python and something sensible in Bash.
I would prefer to do this with standard tools/libraries if possible. I would also prefer the tools to autodetect the the current configuration from the environment rather than have to be explicitly pointed at a configuration file. I do not know if there is a way to have alternate configs in ssh but if there is I would like a way to autodetect the currently active one. Otherwise just defaulting to "~/.ssh/config" would do.
It is possible to specify an alternative configuration file on the command line, but ~/.ssh/config is always the default. For an alternative configuration file use the -F configfile option.
You can either try parsing the original config file, or have a file that is better suited for manipulation and generate an alternative configuration file or part of the default configuration file from that.
Using Paramiko
Paramiko is a pure-Python (3.6+) implementation of the SSHv2 protocol, providing both client and server functionality.
You may not need every feature Paramiko is providing, but based on their documentation, this module could do just what you need: paramiko.config.SSHConfig
Representation of config information as stored in the format used by OpenSSH. Queries can be made via lookup. The format is described in OpenSSH’s ssh_config man page. This class is provided primarily as a convenience to posix users (since the OpenSSH format is a de-facto standard on posix) but should work fine on Windows too.
You can define a non-standard config-file,the same way you would specify an alternate ssh-config file to ssh command line, as mentioned in a previous answer:
config = SSHConfig.from_file(open("some-path.config"))
# Or more directly:
config = SSHConfig.from_path("some-path.config")
# Or if you have arbitrary ssh_config text from some other source:
config = SSHConfig.from_text("Host foo\n\tUser bar")
You can then retrieve whichever configuration you need like so
For example with a configuration file like this:
Host foo.example.com
PasswordAuthentication no
Compression yes
ServerAliveInterval 60
You could access different settings using:
my_config = SSHConfig()
my_config.parse(open('~/.ssh/config'))
conf = my_config.lookup('foo.example.com')
assert conf['passwordauthentication'] == 'no'
assert conf.as_bool('passwordauthentication') is False
assert conf['compression'] == 'yes'
assert conf.as_bool('compression') is True
assert conf['serveraliveinterval'] == '60'
assert conf.as_int('serveraliveinterval') == 60
In case you only need available hostnames, you can list all available ones, and then get information about their configuration using that context (More info here)
get_hostnames()
Return the set of literal hostnames defined in the SSH config (both explicit hostnames and wildcard entries).
This might not be compliant with your request for a standard library, though the project seems to be actively maintained and widely used as well
I want to delete the line configuration (This word "configuration" is inside the angular brackets in the file "hdfs-site.xml"). I tried using this code but no luck.
ruby_block "delete_lines_in_hdfs_site" do
block do
file = Chef::Util::FileEdit.new("/opt/hadoop-2.4.1/etc/hadoop/hdfs-site.xml")
file.search_file_delete_line(/<configuration>/)
end
end
You need to end the ruby block with
file.write_file
Otherwise the changes are not written out.
While I agree that using a template is preferred and considered best practices, there are times you need to edit an existing file which may have variations and not fit the template model.
You do not add to or remove lines from files with chef. Instead you replace the configuration file with yours, which you put into your cookbook under files/ or templates/ forlder.
template "/opt/hadoop-2.4.1/etc/hadoop/hdfs-site.xml"
or
cookbook_file "/opt/hadoop-2.4.1/etc/hadoop/hdfs-site.xml"
When you just add/replace lines in config files, you cannot be sure, that after upgrade of installed software your config files are right.
I have what I think is a fairly common task of taking a local archive file, transferring it to a server, and extracting it there. I'm using the unarchive module for this but struggling with an elegant way to deal with the local filename always being different because the filename includes the software version. For example, here's a excerpt from a playbook:
- name: Upload code
unarchive: src={{payload_file}} dest={{install_dir}}
I can run this and use -e to set the variable:
ansible-playbook -e payload_file=/tmp/payload-4.2.1.zip -i production deploy.yml
My question is is this the best way to handle this situation? All the docs have hardcoded example file names and paths and I haven't seen any evidence in the docs that makes a variable that will be different each deployment straightforward to set.
While passing extra args on the command line will work, I generally don't like having to type to run the job -- this is to say that I don't think your solution is inelegant or bad in any way.
However, if you're willing to change your process a little, you can avoid having to add options when calling your playbook.
Option 1: Put the payload-version.zip file in a directory and target that directory with a glob. Any file in that directory will get unarchived (if there's only one file, this achieves the same behavior):
- name: Upload code
unarchive: src={{item}} dest={{install_dir}}
with_fileglob:
- /tmp/payloads/*
Option 2: Symlink the versioned file to a simple, general name of payload-latest.zip and target it with your play (using the modified date or setting a fact for metadata). I assume you have a build process or script to generate the zip, so add a step to symlink the build version to payload-latest.zip and target that in your play.
I get a strange error when a chef-client tries to execute remote_resource for a big local file.
From stack trace I guess ruby copy files itself. My disk has a lot of free space. Also var and tmp folders has at leas 2 Gbytes. If I do this job myself with cp command or I replace remote_file resource with execute one it's okay.
Chef complains about lack of disk space.
This resource fails for a file of 4G size with message No space on device.
remote_file "/tmp/copy.img" do
source "file://tmp/origin.img"
action :create
end
I made workaround with bash resource and it works.
execute "disk-image" do
command "cp /tmp/origin.img /tmp/copy.img"
not_if "cmp /tmp/origin.img /tmp/copy.img"
end
It's not going to work. remote_file downloads the remote file to somewhere within /var/chef IIRC, then copies to its destination.
Since /var has only 2Gb of space and the file is 4Gb big, it correctly throws the No space left on device error.
Thank you #lamont for the explanation. To cut to the chase a bit, the only solution that worked for me was to add the following to my Chef recipe, prior to any calls to remote_file:
ENV['TMP'] = '/mytmp'
ENV['TMPDIR'] = '/mytmp'
where /mytmp is a directory on a volume with enough space to hold my file.
The promising feature of adding:
file_staging_uses_destdir true
to /etc/chef/client.rb currently does not work, due to this bug: https://tickets.opscode.com/browse/CHEF-5311.
9/20/2016: Chef 12.0 shipped with file_stating_uses_destdir being defaulted to true so this should no longer be an issue (the remote_file but where it streams to /tmp may still exist).
First the real simple statement: If you've got a 4GB file in /tmp and you only have 2GB left in /tmp, then obviously copying the 4GB will fail, and nothing can help you. I'm assuming you have at least 4GB in /tmp and only 2GB left in /var which is the only interesting case to address.
As of 11.6.0 (to 11.10.2 at least) chef-client will create a tempfile using ruby's Tempfile.new and will copy the contents to that temp file and then will mv it into place. The tempfile location will be determined by ENV['TMPDIR'] and that differs based on your O/S distro (e.g. on a Mac that will be something like /var/folders/rs/wqwmj_1n59135dhhbvfdrlqh0001yj/T/ and not just /tmp or even /var/tmp), so it may not be obvious where the intermediate tempfile is created. You may be running into that problem. You should be able to see from the chef-client -l debug output what tempfile location chef is using and if you df -k that directory you might see that it is 100%.
Also, look at df -i to see if you've run out of inodes somehow which will also throw a no space left on device error.
You can set chef-client globally to use the destination directory as the tmpdir for creating files via adding this to client.rb:
file_staging_uses_destdir true
Then if your destination dir is '/tmp' the tempfile will get created there and then will simply get renamed in the directory in order to deploy it. That ensures that if there's enough space on the target device to hold the result, then the resource should always succeed to write the tempfile. It also avoids the problem if /tmp and the destdir are on different filesystems that the mv to rename and deploy the file will get translated into a copy-and-unlink-src operation which can fail in several different ways.
The answer by #cassianoleal is not correct in stating that chef-client always uses /var/cache as a temp location. Changing file_cache_path will also not have an effect. That is confusing a common pattern of downloading remote_files into the Chef file_cache_path directory for how remote_file works internally -- those are not the same thing. There is no file_cache_path in the question, so there should not be any file_cache_path in the answer.
The behavior of remote_file with file:// URLs is a bit round-a-bout, but that is because they're necessary for all other URLs (as #cassianoleal correctly mentioned). The behavior with file_staging_uses_destdir is probably correct, however, since you do want to take into account edge conditions where you run out of room and truncate the file or the server crashes in the middle of a copy operation and you don't want a half-populated file left over. By writing to a tempfile and closing it and then renaming a lot of those edge conditions are avoided.