Since Ansible backup feature is questionable a little with lack of configuration. I'm looking into some solution.
Normally in script I would have backup function that you can call with file name and it would copy the file to separate location with changed name.. for example bkp_location = /tmp/backup//
Lets say I want to backup /etc/systemconf/network I pass it to function and it would copy it to backup directory under etc_systemconf_network ( it replace / with _ so we can tell where it come from )
What would be the best solution in Ansible for something like that ? That I could call it in every role etc...
Maybe one backup.yml in root directory and have it include and pass variable ( file name ) to it, would that work ?
Edit:
Backup feature I speak of:
there is an option backup: yes for some modules ( this is shared function between them as far as I know ) but does not offer any modification to what it does.
Like what would be the backup file name, where it would be located... ? so I have to handle that externally... kind of mid-step between.. but seems like include backup.yml and pass variable to it will do the trick.
cat backup.yml
- name: creating backup
copy: src="{{ path_of_file }}" dest="{{ bkp_location }}/backup{{ path_of_file }}{{ contenttoaddwhilebackingup }}" remote_src=true
in running playbook
include: backup.yml
So if you run a playbook like this
ansible-playbook random.yml -e 'bkp_location=/tmp/backup/ path_of_file=/etc/systemconf/network contenttoaddwhilebackingup=26march2021'
It will create backup like this
ls -lrt /etc/systemconf/
-rw-r--r-- 1 root root 2 Mar 25 15:22 network
ls -lrt /tmp/backup/
-rw-r--r-- 1 root root 2 Mar 25 15:22 backupnetwork26march2021
Related
I want to set two groups to a directory
app_group
td-agent
If use file module, there is a group attribute can do
- name: Set group to directory
file:
path: /app
group: app_group
state: directory
recurse: true
mode: '0774'
The current permissions like
$ ls -la /app/app1/
drwxrwx---. 5 root app_group 31 Oct 11 14:12 .
drwxr-xr-x. 3 root root 21 May 12 2021 ..
drwxrwx---. 3 app_user app_group 12 Apr 28 13:30 product
But I want to set Fluentd's user group td-group can also read files in /app directory. Like /app/app1/product/001.log. How to do?
If change mode to mode: o+rx, other user can read/execute the directory. Will it change current permission for app_user and app_group?
Since the underlying operating system seems to be Linux, the answer to your question
How to set permission to a directory with multiple groups?
will be you can't do that. At least not in that way. Ansible can configure only what is available.
So you may have a look into the basic concepts of how to manage Linux permissions for users, groups, and others.
In example
Linux group permissions
Granting write permissions to a group to a folder
How does group permissions work?
and maybe put the fluentd user into app_group. An other approach could be introducing Access Control Lists (ACL).
In example
How can I give permissions of files to multiple groups?
Two Groups Separate Permissions, One Directory?
There are couple of folders in the root dir of HDFS:
dir1
subdir1
table1
table2
subdir2
dir2
subdir1
table1
table2
dir3
They all have subfolders that contain different Parquet files that are queried with Hive.
I can't load one of the subfolders (for example table1 inside dir2) even though the permissions look ok to me, I get the EXECUTE error when trying to load it.
The code is running in a Jupyter notebook.
Users are organized in groups.
I've added rwx permissions for the directory in question to the group by using the following command:
hdfs dfs -setfacl -R -m group:user_group:rwx /dir2/subdir2
The error I'm getting looks like this:
Cannot create staging directory 'hdfs://server:8020/dir2/subdir1/table1/.hive-staging_hive_2019-08-01_13-04-22': Permission denied: user=username, access=EXECUTE, inode="/dir2":hdfs:supergroup:drwxrwx---
I've added read and execute permissions on dir2 to the user group but the error persists. It looks to me from this error that somehow the default permissions are applied and they are ---
So, to summarize;
group has read and execute privileges on the root dir, and read, write and execute privileges on the table directories, but it keeps failing with permissions for root directory.
This is how the permissions look:
# file: /dir2
# owner: hdfs
# group: supergroup
user::rwx
user:some_group1:r-x
group::---
group:some_group2:rwx
group:user_group:r-x
group:hive:rwx
group:some_group3:r-x
group:some_group4:r-x
mask::rwx
other::---
default:user::rwx
default:user:some_group1:r-x
default:group::---
default:group:some_group2:rwx
default:group:hive:rwx
default:group:some_group3:r-x
default:group:some_group4:r-x
default:mask::rwx
default:other::---
# file: /dir2/subdir1/table1
# owner: some_user
# group: supergroup
user::rwx
user:some_group1:r-x
group::---
group:some_group2:rwx
group:user_group:rwx
group:hive:rwx
group:some_group3:r-x
group:some_group4:rwx
mask::rwx
other::---
default:user::rwx
default:user:some_group1:r-x
default:group::---
default:group:some_group2:rwx
default:group:user_group:rwx
default:group:hive:rwx
default:group:some_group3:r-x
default:group:some_group4:rwx
default:mask::rwx
default:other::---
The problem was eventually solved by creating new directories that replaced the old ones. The new directories were created with the correct user and credentials.
For example, I created subdir1_new, moved the data there, renamed subdir1 to subdir1_old and renamed subdir1_new to subdir1. Not a lot of folders were affected by this issue so it didn't take a long time.
I know it's not the actual solution, but I couldn't figure out what exactly was happening and this workaround did the trick.
I just noticed that some unknown folder has been created in my /etc/ansible directory.
Here it is ,
monitoring-server#monitoring-server:/etc/ansible$ ls
[' ansible.cfg hosts java.retry java.yml nginx.retry nginx.yml roles test.retry test.yml
This [' is the folder.
monitoring-server#monitoring-server:/etc/ansible$ ls -l
total 56
drw-r--r-- 3 root root 4096 Jul 27 07:12 ['
I need to be root to open it.
root#monitoring-server:/etc/ansible/['# tree
.
└── opt
└── rsyslog']
2 directories, 0 files
Well, the /opt/rsyslog is one of the directories I made on some remote servers using ansible.
How is this folder created and Why is it created?
Directory tree is ['/opt/rsyslog'] – seems like you mistyped string path as list sometime ago and fed it into module with local host as target.
As long as /etc/ansible also seems to be your working directory (which is really weird practice), path [' / opt / rsyslog'] has been created.
You may safely remove it.
If you look at a host which was set up be SaltStack, then it is sometimes like looking at a binary file with vi.
You have no clue how the config/file was created.
This makes trouble shooting errors hard. Reverse engineering where a file comes from takes too much time.
My goal: Make it easy to find the way from looking at the unix config file on the minion (created by salt) to the source where this configuration came from. Like $Id$ in svn and cvs.
One idea a friend and I had:
The state file.managed should (optionally) add the source of the file.
Example:
My sls file contains this:
file_foo_bar:
file.managed:
- source:
- salt://foo/bar
Then the created file should contain this comment.
# Source: salt://foo/bar
Of course this is not simple, since there are different ways to put comments into configuration files.
Is this feasible? Or is there a better solution to my goal.
Update
Usually I know what I did wrong and can find the root easily. The problem arises if several people work on a state tree.
This is a starting point where you can get the date and time of the modified file when its managed by Salt by using Salt Pillar.
Lets call our variable salt_managed. Create a pillar file like the following:
{% set managed_text = 'Salt managed: File modified on ' + salt.cmd.run('date "+%Y-%m-%d %H:%M:%S"') %}
salt_managed: {{ managed_text | yaml_dquote }}
Then on the minion when you call the pillar you will get the following result:
$ salt-call pillar.get salt_managed
local:
Salt managed: File modified on 2016-10-18 11:12:40
And you can use this by adding it on the top of your config files for example like this:
{{ pillar.get('salt_managed') }}
Update:
I found a work around that might be useful for someone. Lets say we have a multiple states that could modify the same file. How can we know that State X is the responsible for modifying that file ? by doing the following steps:
1- I have created a state like this one:
Create a File:
file.managed:
- name: /path/to/foofile
- source: salt://statedir/barfile
Add file header:
file.prepend:
- name: /path/to/foofile
- text: "This file was managed by using this salt state {{ sls }}"
The contents of barfile is:
This is a new file
2- Call the state from the minion and this will be the result:
$ salt-call state.sls statedir.test
local:
----------
ID: Create a File
Function: file.managed
Name: /path/to/foofile
Result: True
Comment: File /path/to/foofile updated
Started: 07:50:45.254994
Duration: 1034.585 ms
Changes:
----------
diff:
New file
mode:
0644
----------
ID: Add file header
Function: file.prepend
Name: /path/to/foofile
Result: True
Comment: Prepended 1 lines
Started: 07:50:46.289766
Duration: 3.69 ms
Changes:
----------
diff:
---
+++
## -1,1 +1,2 ##
+This file was managed by using this salt state statedir.test
This is a new file
Summary for local
------------
Succeeded: 2 (changed=2)
Failed: 0
------------
Total states run: 2
Currently the content of foofile is:
This file was managed by using this salt state statedir.test
This is a new file
I'm attempting to embed a shell command in to my Chef Recipe, however when Chef executes the command it seems to get things wrong. Here's the resource in question:
script "create libs symlink" do
interpreter "bash"
user "root"
cwd "/home/robin/test"
code <<-EOH
ln -s $(ls -1 | grep '^[0-9.-]\+$') curr-version-libs
EOH
end
The /home/robin/test directory contains a folder called 19.26-3, so I'm expecting a symlink called curr-version-libs pointing at 19.26-3.
Instead, I'm ending up with a circular symlink:
drwxr-xr-x 4 root root 4096 Jan 17 22:35 19.26-3
drwxr-xr-x 2 root root 4096 Jan 17 22:35 config
lrwxrwxrwx 1 root root 17 Jan 28 17:31 curr-version-libs -> curr-version-libs
It seems that the $(ls -1 | grep '^[0-9.-]+$') is being removed and I'm ending up with the command ln -s curr-version-libs.
Does anyone know what's going on here? I've tried using an execute resource, but I get the same results.
If your 19.26-3 directory exists before chef run starts, then it is easy. If you are creating a symbolic link, I would recommend using link resource for that.
version = `ls /home/robin/test/ -1 | grep '^[0-9.-]+$'`.strip
link "/home/robin/test/curr-version-libs" do
to ::File.join( "/home/robin/test", version )
end
But if it is not there, I would recommend using ruby_block and defining your link resource dynamically.
ruby_block "create libs symlink" do
block do
version = `ls /home/robin/test/ -1 | grep '^[0-9.-]+$'`.strip
res = Chef::Resource::Link.new( "/home/robin/test/curr-version-libs", run_context )
res.to ::File.join( "/home/robin/test", version )
res.run_action :create
end
end
Edit: I corrected the answer by fixing regex and and calling strip before assigning to version as Robin proposed.
It appears you are calling out to the shell to create a sym link. In that case, a much better way to do this is to use the Chef link resource. I would never use the script or execute resource to do what you are doing.
Using the link resource, you would do the following:
link "/home/robin/test/curr-version-libs" do
to '/home/robin/test/19.26-3'
user 'root'
group 'root'
link_type :symbolic
action :create
end
A quick side commentary: I have mentored and tutored a number of folks to come up on Chef. Those who come to understand what is offered by the resources, providers, and lightweight resources (aka. LWRPs) are much happier and effective than those who just try to drop their old shell scripts into their cookbooks.
I highly recommend reading the Resource and Providers and Lightweight Resources Documentation
have you tried escaping dollar sign?
ln -s \$(ls -1 | grep '^[0-9.-]\+$') curr-version-libs