I'm designing a kind of playbook lib with individual tasks
so in the usual roles repo, I have something like:
roles
├── common
│ └── tasks
│ ├── A.yml
│ ├── B.yml
│ ├── C.yml
│ ├── D.yml
│ ├── login.yml
│ ├── logout.yml
│ └── save.yml
├── custom_stuff_workflow
│ └── tasks
│ └── main.yml
└── other_stuff_workflow
└── tasks
└── main.yml
my main.yml in custom_stuff_workflow then contain something like:
---
- include: login.yml
- include: A.yml
- include: C.yml
- include: save.yml
- include: logout.yml
and this one in the other workflow:
---
- include: login.yml
- include: B.yml
- include: A.yml
- include: D.yml
- include: save.yml
- include: logout.yml
I can't find a way to do it in a natural way:
one way that worked was having all tasks in a single role and tagging the relevant tasks while including a custom_stuff_workflow
The problem I have with that is that tags cannot be set in the calling playbook: it's only to be set at command line
as I'm distributing this ansible repo with many people in the company, I can't rely on command line invocations (it would be nice to have a #! header in yml to be processed by ansible-playbook command)
I could also copy the relevant tasks (inside common in the above tree) in each workflow, but I don't want to repeat them around
Can someone see a solution to achieve what I'd like without repeating the tasks over different roles?
I guess the corner stone of my problem is that I define tasks as individual and it looks not natural in ansible...
Thanks a lot
PS: note that the tasks in the workflow have to be done in specific order and the only natural steps to abstract would be the login and save/logout
PPS: I've seen this question How do I call a role from within another role in Ansible? but it does not solve my problem as it's invoking a full role and not a subset of the tasks in a role
Just in case someone else bumps into this, version 2.2 of Ansible now has include_role. You can now do something like this:
---
- name: do something
include_role:
name: common
tasks_from: login
Check out the documentation here.
Yes, Ansible doesn't really like tasks as individual components. I think it wants you to use roles, but I can see why you wouldn't want to use roles for simple, reusable tasks.
I currently see two possible solutions:
1. Make those task-files into roles and use dependencies
Then you could do something like this in e.g. custom_stuff_workflow
dependencies:
- { role: login }
See: https://docs.ansible.com/playbooks_roles.html#role-dependencies
2. Use include with "hardcoded" paths to the task files
- include: ../../common/tasks/login.yml
That worked pretty well in a short test playbook I just did. Keep in mind, you can also use parameters etc. in those includes.
See: http://docs.ansible.com/ansible/latest/playbooks_reuse.html
I hope I understood that question correctly and this helps.
Using include_role: with option tasks_from is a good idea. However this still includes parts of the role. For example it loads role vars and meta dependencies. If apply is used to apply tags to an included file, then same tags are applied to meta dependencies. Also, the ansible output lists as the included role's name in its output, which is confusing.
It is possible to dynamically locate a role and include a file using first_found. One can find the role path searching DEFAULT_ROLES_PATH and load a file from tasks folder. Ansible uses the same variable when sarching a role, so long as the role is in a path that Ansible can find, then the file will be loaded.
This method is as dynamic as using include_role with option tasks_from
Example:
- name: Include my_tasks.yml from my_ansible_role
include_tasks: "{{lookup('first_found', params)}}"
vars:
params:
files: my_ansible_role/tasks/my_tasks.yml
paths: "{{ lookup('config', 'DEFAULT_ROLES_PATH') }}"
You can use the built-in variable playbook_dir to include tasks from other roles.
- name: "Create SSL certificates."
include_tasks: "{{ playbook_dir }}/roles/common/tasks/ssl.yml"
with_items: "{{ domains }}"
The ssl.yml is in the common role. It can have any number of tasks; also, passing variables like the example is possible.
Related
Tree structure:
.
├── README.md
├── galaxy
│ ├── requirements.yaml
│ └── roles
│ └── github-runner.yaml
└── inventories
└── hosts.yaml
3 directories, 4 files
galaxy/requirements.yaml content:
---
roles:
- name: monolithprojects.github_actions_runner
galaxy/roles/github-runner.yaml content:
---
- name: GitHub self-hosted runner
hosts: my_host
roles:
- role: monolithprojects.github_actions_runner
become: true
runner_user: ubuntu
github_account: my-github-username
github_repo: my-github-repository
access_token: redacted
I'm using this role to install GitHub Actions self-hosted runner on one of my servers. But, I can't find a way to load and pass that access_token parameter from another file (e.g. .env or secrets.yaml) or from my local environment variable.
I don't prefer to store any secrets in my Git repo even they can be revoked and I may apply some Git tricks to clean my Git history. I just want to start clean and right because I don't want to deal with the mess later.
I tried using vars_files and include_vars attributes in the YAML file above but failed. I also tried passing from my local environment variable (like in code block below) - but that failed too.
... content of galaxy/roles/github-runner.yaml ...
access_token: "{{ lookup('env', 'PERSONAL_ACCESS_TOKEN') }}"
Is it possible to set that access_token from a secret file or environment variable? Or it's not possible in this case?
Note: I'm planning to reuse more roles from Ansible Galaxy in future and I will add them as needed under that galaxy/roles folder.
The ansible docu says
If Ansible were to load ansible.cfg from a world-writable current working directory, it would create a serious security risk.
That makes sense but causes a problem in my ci-pipeline fir my project:
.
├── group_vars
├── host_vars
├── playbooks
├── resources
├── roles
| ├── bootstrap
| └── networking
├── ansible.cfg
├── inventory.yml
├── requirements.yml
├── site.yml
└── vault.yml
I have two "local" roles which are checked in under source control of the ansible project under ./roles, but the roles are not found when i run ansible-playbook --syntax-check site.yml
$ ansible-playbook --syntax-check site.yml
[WARNING] Ansible is being run in a world writable directory (/builds/papanito/infrastructure), ignoring it as an ansible.cfg source. For more information see https://docs.ansible.com/ansible/devel/reference_appendices/config.html#cfg-in-world-writable-dir
[WARNING]: provided hosts list is empty, only localhost is available. Note
that the implicit localhost does not match 'all'
ERROR! the role 'networking' was not found in /builds/papanito/infrastructure/playbooks/roles:/root/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles:/builds/papanito/infrastructure/playbooks
The error appears to have been in '/builds/papanito/infrastructure/playbooks/networking.yml': line 14, column 7, but may
be elsewhere in the file depending on the exact syntax problem.
The offending line appears to be:
roles:
- { role: networking, become: true }
^ here
ERROR: Job failed: exit code 1
--------------------------------------------------------
Obviously cause roles are searched
A roles/ directory, relative to the playbook file.
Thus my ansible.cfg defined to look in ./roles
# additional paths to search for roles in, colon separated
roles_path = ./roles
So based on the ansible docu I can use the environment variable ANSIBLE_CONFIG as I do as follows in the gitlab-ci.yml
variables:
SITE: "site.yml"
PLAYBOOKS: "playbooks/**/*.yml"
ANSIBLE_CONIG: "./ansible.cfg"
stages:
- verify
before_script:
.....
ansible-verify:
stage: verify
script:
- ansible-lint -v $SITE
- ansible-lint -v $PLAYBOOKS
- ansible-playbook --syntax-check $SITE
- ansible-playbook --syntax-check $PLAYBOOKS
But I still get the error above. What do I miss?
site.yml
- import_playbook: playbooks/networking.yml
- import_playbook: playbooks/monitoring.yml
playbooks/networking.yml
- name: Setup default networking
hosts: all
roles:
- { role: networking, become: true }
- { role: oefenweb.fail2ban, become: true }
I know the topic is old, but you have a typo in your config file. You are missing an F in ANSIBLE_CONFIG, so write this instead
variables:
SITE: "site.yml"
PLAYBOOKS: "playbooks/**/*.yml"
ANSIBLE_CONFIG: "./ansible.cfg"
BTW, it helped to solve my problem
Looks like a Hierarchy setup issue, there is no task associated within the roles bootstrap, networking; Instead looks like the playbooks are in a different folder called playbooks.
Refer directory layout: https://docs.ansible.com/ansible/latest/user_guide/playbooks_best_practices.html
I know that you can change between different inventory files using the -i flag which can be used to switch between different hosts.
In my case, the hosts to be acted upon change between deployments, so I take the hosts in as --extra-vars and use delegate_to to deploy to that host (see below for more details).
I was hoping for a way to switch between files containing environment variables in a similar fashion. For example, lets say I have the following simplified directory structure:
/etc/ansible/
├── ansible.cfg
├── hosts
└── project/
└── environments/
├── dev/
│ └── vars.yml
└── prd/
└── vars.yml
The structure of vars.yml in both environments would be exactly the same, just with the variables having different values due to the differences between environments.
I've found a few places that talk about doing something similar such as these:
https://rock-it.pl/managing-multiple-environments-with-ansible-best-practices/
http://rosstuck.com/multistage-environments-with-ansible
http://www.geedew.com/setting-up-ansible-for-multiple-environment-deployments/
In those guides, they act against statically declared hosts. One thing that help me seems to be the directories called group_vars. It looks like the inventory points to the config with the same name, and assumingly uses those variables when the hosts: directive of a play contains the host(s) specified in the inventory header.
However, Since I dynamically read in the servers that we're acting against via the CLI flag --extra-vars, I can't take that approach because I will always have something like this for my plays:
...
hosts: localhost
tasks:
...
- name: do something
...
delegate_to: {{ item }}
with_items: {{ given_hosts }}
Or I run a task first that takes the servers and adds them to a new host like this:
- name: Extract Hosts
hosts: localhost
tasks:
- name: Adding given hosts to new group...
add_host:
name: "{{ item }}"
groups: some_group
with_items:
- "{{ list_of_hosts | default([]) }}"
and then uses the dynamically created group:
- name: Restart Tomcat for Changes to Take Effect
hosts: some_group
tasks:
- name: Restarting Tomcat...
service:
name: tomcat
state: restarted
So I need to find a way to specify which vars.yml to use. Because I use Jenkins to kick off the Ansible playbook via CLI over SSH, I was hoping for something like the following:
ansible-playbook /path/to/some/playbook.yml --include-vars /etc/ansible/project/dev/vars.yml
At the least, how would I explicitly include a vars.yml file in a playbook to use the variables defined within?
You can use:
extra vars with #: --extra-vars #/etc/ansible/project/dev/vars.yml
or
include_vars:
- include_vars: "/etc/ansible/project/{{ some_env }}/vars.yml"
to load different variables depending in your environment.
I have a playbook organized as follows (simplified for the sake of this question):
├── deploy.yml
├── hosts
├── requirements.yml
├── roles
│ └── web
│ ├── meta
│ │ └── main.yml
│ └── tasks
│ └── main.yml
└── site.retry
My simplified deploy.yml is:
---
- name: Everything I need
hosts: somewhere
roles:
- web
And my simplified roles/web/tasks/main.yml is
---
- name: Various things that work
become: yes
[whatever]
- name: the thing that I have a problem with
become: yes
davidedelvento.nbextension: name=foo state=present
This fails with:
ERROR! no action detected in task. This often indicates a misspelled module name, or incorrect module path.
So I tried to change roles/web/tasks/main.yml to
---
- name: Various things that work
become: yes
[whatever]
- name: the thing that I have a problem with
become: yes
roles:
- { role: davidedelvento.nbextension, name: foo, state: present}
which fails in the same way. I understand the failure (since I cannot call a role from a task, which instead I'm doing -- but the error could be clearer....)
However I am not clear how can I accomplish what I'd like, namely doing whatever nbextension is doing at that point in time. I could move that role from roles/web/tasks/main.yml to roles/web/meta/main.yml and that works, but it is executed before the Various things that work and I need it to be executed after. How to accomplish that?
Note that I wrote nbextension, however the same problem happen with similar other roles from the galaxy.
EDIT: Note also that the extension is correctly installed and can be used from a standalone, single-file playbook such as
---
- name: Example
hosts: all
become: yes
roles:
- { role: davidedelvento.nbextension, name: foo, state: present}
however I need it to "integrate" in the larger project described above for the "web" role (I have more roles that I'm not showing)
EDIT2: note that the galaxy ansible role used for this question has been renamed to jupyterextension but as I said the issue (and solution) is the same for any role
Ok, so I've found two ways to deal with this issue.
split the role in two (or more) parts and use the galaxy's role as a dependency to the things that it needs to prepend. In general, I like this idea, but in my particular use case I don't, since I would need create 3 roles for something that is really one.
Use include_role module, with the caveat that at the moment it is "flagged as preview", namely it is not guaranteed to have a backwards compatible interface. However it works quite well for my current setup:
- name: the thing that I have not a problem with anymore
become: yes
include_role:
name: davidedelvento.nbextension
with_items:
- foo
- bar
loop_control:
loop_var: name
Background: I want to write a ansible playbook that works with different OS. One of the basic thing it does is to install packages using package command (newly introduced in ansible 2.0).
The problem is the list of packages are different on each OS. I would like to specify common ones somewhere, and then place Ubuntu specific ones and CentOS specific ones separately. Then for CentOS ones, CentOS6 and CentOS7 also have some common ones and different ones, so I would like to put common ones for CentOS somewhere, and CentOS6 and CentOS7 specific ones somewhere else.
The immediate way jumps into my mind is to have group_vars, however, the way I know wouldn't work. Suppose under group_vars I have files all, ubuntu, centos, centos6 and centos7 to define variables, the order of which takes precedence is indeterministic. For a CentOS6 host, it is in groups all, centos and centos6. There is no way to specify centos6 should take precedence over centos and centos is over all.
Another way I found was to set hash_behaviour = merge in ansible.cfg, however, this modifies the behavior globally, which overkills. There might be cases where default behavior is wanted, so modifying it globally is not a nice approach.
How should I organize everything in a clean way, and share and re-use common things as much as possible?
OS condition inside the role
One common way that I use is to define OS dedicated default vars files inside the vars directory of the role and then include them at the beginning of the main.yml task. Ansible helps me to automatically identify the currently used distribution with {{ ansible_distribution }} and {{ ansible_distribution_version }}, so I just have to name the OS dedicated yml files accordingly.
Role dir tree:
my_role/
├── handlers
│ ├── main.yml
│ └── Ubuntu14.04.yml
├── tasks
│ ├── main.yml
│ └── Ubuntu14.04.yml
├── templates
│ └── Ubuntu14_04.config.j2
└── vars
└── Ubuntu14.04.yml
In the main.yml you first include default OS specific variables and then run OS specific tasks:
tasks/main.yml:
---
- name: include distribution specific vars
include_vars: "{{ ansible_distribution }}{{ ansible_distribution_version }}.yml"
- name: include distribution specific install
include: "{{ ansible_distribution }}{{ ansible_distribution_version }}.yml"
- name: Do general stuff
...
Now you can easily define your packages inside the vars file for each distribution, e.g., Ubuntu14.04:
vars/Ubuntu14.04.yml:
---
my_role_packages:
- a
- b
And finally install them in your distribution specific setup task.
tasks/Ubuntu14.04.yml:
---
- name: Install packages for Ubuntu14.04
apt:
name={{ item }}
state=present
update_cache=yes
with_items: "{{ my_role_packages }}"
You can now also easily define OS specific handlers and templates, e.g.,:
handlers/main.yml:
---
- name: distribution specific handler
include: "{{ ansible_distribution }}{{ ansible_distribution_version }}.yml"
Hash Merging
About globally setting hash_behaviour=merge, here a quote from the official Ansible documentation:
Some users prefer that variables that are hashes (aka ‘dictionaries’
in Python terms) are merged. This setting is called ‘merge’.
We generally recommend not using this setting unless you think you
have an absolute need for it, and playbooks in the official examples
repos do not use this setting.
Originally I come from the SaltStack world and was used to merging hashes from my defaults map.jinja with the dedicated pillars, but in Ansible I started relying more on variable prefixes, so instead of
nginx:
pkg: "nginx"
repo: "deb http://nginx.org/packages/ubuntu/ trusty nginx"
I'd write
nginx_pkg: "nginx"
nginx_repo: "deb http://nginx.org/packages/ubuntu/ trusty nginx"
in order to avoid accidentally overwriting hashes when going up the variable hierarchy. If in some cases you still prefer merging, you could use the jinja2 combine filter in your variable files: dict_a: {{ dict_b|combine(dict_c) }}.
Variable grouping
The Ansible documentation puts a lot of emphasize on heavily using group_vars and I found that to be good advice. A general approach here is to define my groups in my inventories files, like:
hosts/test:
[app]
app-test-a
app-test-b
[app_test:children]
app
hosts/live:
[app]
app-live-a
app-live-b
[app_live:children]
app
Now I can easily use group_vars to include variables based on the defined groups:
group_vars/
├── all.yml
├── app_test.yml
├── app_live.yml
└── app.yml
Ansible Galaxy and DebObs
I also recommend checking out Ansible Galaxy roles. They are always a good starting point to get some ideas (including this one). Another good source of inspiration is the DebObs repository.