Errors when use "json_query" in Ansible - ansible

When use json_query in Ansible, got the error along with my code as shown below.
The Python used in Ansible is 3.6.8, and used the same version installed jmespath: pip install jmespath, so, this should not be the issue. The Ansible code should be fine as well.
fatal: [localhost]: FAILED! => {"msg": "template error while templating string: no filter named 'json_query'. String: {{ jsondata | json_query(jmesquery) }}"
The following is the Ansible codes:
---
- name: ReadJsonfile
hosts: localhost
tasks:
- name: Display the JSON file content
shell: cat config.json
register: result
- name: save the JSON data to a Variable as a Fact
set_fact:
jsondata: "{{ result.stdout | from_json }}"
- name: setDomainName
set_fact:
domain_name: "{{ jsondata | json_query(jmesquery) }}"
vars:
jmesquery: 'domain.name'
- name: setDomainUsername
set_fact:
domain_username: "{{ jsondata | json_query(jmesquery) }}"
vars:
jmesquery: 'domain.user'
- name: setDomainPassword
set_fact:
domain_password: "{{ jsondata | json_query(jmesquery) }}"
vars:
jmesquery: 'domain.password'
- name: setadmin_Listenport
set_fact:
admin_ListenPort: "{{ jsondata | json_query(jmesquery) }}"
vars:
jmesquery: 'domain.admin.listenport'
- name: Debug the values
debug: msg=" Admin Listen Port => {{ admin_ListenPort }}, DomainName => {{ domain_name }}, DomainUserName => {{ domain_username }} , Domain Password => {{ domain_password }}"

From the error message, I suspect you are using ansible 2.10
The json_query filter is part of the community.general collection which needs to be installed separately starting from this version (this is clearly stated in the documentation)
ansible-galaxy collection install community.general
That being said:
your actual jmespath queries will not return what you expect
using shell to get a file content from remote is a bad practice
that's lots of set_facts and json_query to simply get data that is readily available
Since yaml is a strict superset of json, any valid json is also a valid yaml. In your case, you only have to load the content of the file and you have the data.
One solution is to fetch the file locally and then use include_vars.
In this particular case, using slurp looks like the best option.
This is what (I believe, since you did not provide an example) your config file looks like. I pushed that file in /tmp/config.json for my example.
{
"domain": {
"name": "toto",
"user": "doejohn",
"password": "sosecret",
"admin": {
"listenport": 5501
}
}
}
This is the demo playbook
---
- name: Load remote json content demo
hosts: localhost
gather_facts: false
vars:
config_file: /tmp/config.json
# Of course the below vars will fire errors
# if you call them before slurping data from remote.
# Note: loading `data` in a jinja expression on its own
# forces to transform it back rom string to json data.
# You can accomplish the same result in one go
# using the `from_json` filter
# i.e. => domain: "{{ (slurped_config.content | b64decode | from_json).domain }}"
data: "{{ slurped_config.content | b64decode }}"
domain: "{{ data.domain }}"
# Note2: the above will work and adapt to N hosts in your play: you will get
# the correct data for each host in every task.
tasks:
- name: Slurp config file content
slurp:
src: "{{ config_file }}"
register: slurped_config
- name: Show result
vars:
message: |-
name is: {{ domain.name }}
user is: {{ domain.user }}
password is: {{ domain.password }}
port is: {{ domain.admin.listenport }}
debug:
msg: "{{ message.split('\n') }}"
which gives:
PLAY [Load remote json content demo] ******************************************************
TASK [Slurp config file content] **********************************************************
Wednesday 21 April 2021 17:57:47 +0200 (0:00:00.010) 0:00:00.010 *******
ok: [localhost]
TASK [Show result] ************************************************************************
Wednesday 21 April 2021 17:57:48 +0200 (0:00:00.199) 0:00:00.209 *******
ok: [localhost] => {
"msg": [
"name is: toto",
"user is: doejohn",
"password is: sosecret",
"port is: 5501"
]
}
PLAY RECAP ********************************************************************************
localhost : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
Wednesday 21 April 2021 17:57:48 +0200 (0:00:00.027) 0:00:00.237 *******
===============================================================================
Slurp config file content ---------------------------------------------------------- 0.20s
Show result ------------------------------------------------------------------------ 0.03s

Related

Ansible Task Using 'ansible.builtin.unvault' lookup

The ansible code below takes an ansible vault (vault.yml) and then uses the ansible.builtin.unvault lookup to retrieve and save the entire vault as the variable full_vault. The output of the debug shows the code in json. This code is working as expected.
- name: Pull vault into Variable from encrypted YAML file
hosts: localhost
gather_facts: no
tasks:
- name: Get specific value from vault file
set_fact:
full_vault: "{{ lookup('ansible.builtin.unvault', 'vault.yml') | from_yaml }}"
- name: Display Vault
ansible.builtin.debug:
msg: "Vault: {{ full_vault }}"
The challenge I am having is trying to use the ansible.builtin.vault lookup to put the full_vault variable back into an ansible vault. How can I accomplish this in a single task?
I am using the environment variable ANSIBLE_VAULT_PASSWORD_FILE=pass.txt for encryption/decryption.
Your question and example is focusing on the ansible.builtin.unvault lookup which is abolutely not needed in your situation. The only case I can think of where this would be needed is if you fetch your vault password from an other system/app/source while running your playbook. But since it is available with classic env vars to ansible, you just have to use the encrypted file which will be decrypted on the fly.
For the rest of the example, let's imagine your vault.yml file contains (decrypted):
my_login: vip
my_pass: v3rys3cr3t
some_other_key: toto
Using the above encrypted file is as easy as
---
- hosts: localhost
gather_facts: false
vars_files:
- vault.yml
tasks:
- name: Dummy use of login and pass
ansible.builtin.debug:
msg: "Login in {{ my_login }} and password is {{ my_pass }}"
Now if you want to easily load that file with all its content, change a value for a key in the contained dict and push back the content encrypted with the same configured password, here is a first draft that you will probably have to enhanced. But it worked for my local test with your current configuration.
The update_vault.yml playbook
---
- hosts: localhost
gather_facts: false
vars:
vault_file: vault.yml
new_pass: n3ws3cr3t
tasks:
- name: Import vaulted variables in a namespace (for further easier manipulation)
ansible.builtin.include_vars:
file: "{{ vault_file }}"
name: my_vault
- name: Dummy task just to show above worked
debug:
msg:
- Login is {{ my_vault.my_login }}.
- Password is {{ my_vault.my_pass }}
- Some other key is {{ my_vault.some_other_key }}
- name: Update an element and push back to encrypted file
vars:
new_vault_content: "{{ my_vault | combine({'my_pass': new_pass}) }}"
vault_pass_file: "{{ lookup('ansible.builtin.env', 'ANSIBLE_VAULT_PASSWORD_FILE') }}"
vault_pass: "{{ lookup('ansible.builtin.file', vault_pass_file) }}"
copy:
content: "{{ new_vault_content | to_nice_yaml | ansible.builtin.vault(vault_pass) }}"
dest: "{{ vault_file }}"
decrypt: false
gives:
$ ansible-playbook update_vault.yml
PLAY [localhost] ***********************************************************************************************************************************************************************************************************************
TASK [Import vaulted variables in a namespace (for furthre easier manipulation)] *******************************************************************************************************************************************************
ok: [localhost]
TASK [Dummy task just to show above worked] ********************************************************************************************************************************************************************************************
ok: [localhost] => {
"msg": [
"Login is vip.",
"Password is v3rys3cr3t",
"Some other key is toto"
]
}
TASK [Update an element and push back to encrypted file] *******************************************************************************************************************************************************************************
changed: [localhost]
PLAY RECAP *****************************************************************************************************************************************************************************************************************************
localhost : ok=3 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
And you can easilly check the file was correctly updated:
$ ansible-vault view vault.yml
my_login: vip
my_pass: n3ws3cr3t
some_other_key: toto
Note that the above playbook is not idempotent. If you run it a second time, the decrypted content of your file will stay identical (with the same new password), but the file will still be changed as the vault salt will change and the encrypted content will be different.

How do you iterate/parse through a CSV file to use in Ansible playbook?

I am new to Ansible and don't have much coding experience in general. I am trying to figure out how to provision RHEL servers using the DellEMC OpenManage modules.
The first step to this is figuring out how to parse a CSV, that we are putting necessary information for the later templating. (IP, hostname, MAC, etc. ...) I can get it to print the data in general, but cant figure out how to parse/iterate through it.
CSV Sample
server_name,idrac_ip,idrac_user,idrac_pwd,idrac_nw,idrac_gw,idrac_mac,mgmt_ip,mgmt_nw,mgmt_gw,mgmt_mac,bond0_100_ip,bond0_200_ip,dns_1,bond0_100_nw,bond0_100_gw,bond0_100_vip,bond0_200_nw,bond0_200_gw,bond0_200_vip,os,fs,sio_type,mdm_name
test1,1.1.1.10,root,Password1234,1.1.1.0/24,1.1.1.1,00:00:00:00:00:01,1.1.1.142,1.1.62.0/24,1.1.2.1,98:03:9B:46:25:B3,1.1.1.22,1.1.1.21,1.1.1.26,1.1.61.15/24,1.1.61.1,1.1.61.29,1.1.66.0/24,1.1.66.1,1.1.1.29,RHEL 7.6,1,Master,MDM-1
Here is my playbook to print the info in general.
---
- name: Parse
hosts: localhost
connection: local
tasks:
- name: Load CSV Data into object
read_csv:
path: 'Test_Lab_Servers.csv'
fieldnames: server_name,idrac_ip,idrac_user,idrac_pwd,idrac_nw,idrac_gw,idrac_mac,mgmt_ip,mgmt_nw,mgmt_gw,mgmt_mac,bond0_100_ip,bond0_200_ip,dns_1,bond0_100_nw,bond0_100_gw,bond0_100_vip,bond0_200_nw,bond0_200_gw,bond0_200_vip,os,fs,sio_type,mdm_name
delimiter: ','
register: csv_output
delegate_to: localhost
- name: Print data
debug:
msg: "{{ csv_output }}"
Any advice?
According the read_csv module parameter documentation fieldnames are
needed if the CSV does not have a header
only. Therefore you could remove it to get a list object which might be easier to parse.
- name: Print data
debug:
msg: "{{ csv_output.list }}"
As already mentioned within the comments, to iterate over the list elements you may use loop.
- name: Print data
debug:
msg: "{{ item }}"
loop: "{{ csv_output.list }}"
With extended loop variables you will have a better control about the loop output.
- name: Print data
debug:
msg: "{{ item }}"
loop: "{{ csv_output.list }}"
loop_control:
extended: yes
label: "{{ ansible_loop.index0 }}"
Resulting into an output of
TASK [Print data] ***************
ok: [localhost] => (item=0) =>
msg:
bond0_100_gw: 1.1.61.1
bond0_100_ip: 1.1.1.22
bond0_100_nw: 1.1.61.15/24
bond0_100_vip: 1.1.61.29
bond0_200_gw: 1.1.66.1
bond0_200_ip: 1.1.1.21
bond0_200_nw: 1.1.66.0/24
bond0_200_vip: 1.1.1.29
dns_1: 1.1.1.26
fs: '1'
idrac_gw: 1.1.1.1
idrac_ip: 1.1.1.10
idrac_mac: 00:00:00:00:00:01
idrac_nw: 1.1.1.0/24
idrac_pwd: Password1234
idrac_user: root
mdm_name: MDM-1
mgmt_gw: 1.1.2.1
mgmt_ip: 1.1.1.142
mgmt_mac: 98:03:9B:46:25:B3
mgmt_nw: 1.1.62.0/24
os: RHEL 7.6
server_name: test1
sio_type: Master
...
Specific items (fields) like server_name you could get simply by using
msg: "{{ item.server_name }}"
resulting into an output of
TASK [Print data] ************
ok: [localhost] => (item=0) =>
msg: test1
ok: [localhost] => (item=1) =>
msg: test2
ok: [localhost] => (item=2) =>
msg: test3
...

How to replace environment variable with value in ansible variable

I have a JSON object of files and the destinations they need to be symlinked moved to. I would like to be able to have environment variables in the destinations and have them evaluated to know where they should be moved to. I'm attempting to do a regex_search with a lookup, but it isn't giving me the desired result.
Here is the json:
dotfiles.json
{
"mac": [
{
"name": ".gitconfig",
"destination": "$HOME/"
}
]
}
playbook: ansible/bootstrap.yaml
---
- name: "Bootstrapping Machine"
hosts: localhost
connection: local
ignore_errors: true
vars:
dotfiles: "{{ lookup('file', '../dotfiles.json') | from_json }}"
roles:
- role: "MacOSX"
when: "ansible_distribution == 'MacOSX'"
tasks: ansible/roles/MacOSX/tasks/main.yaml
- name: Symlink dotfiles
ansible.builtin.file:
src: "../../../dotfiles/{{ item.name }}"
dest: "{{ destination }}"
state: link
vars:
destination: "{{ item.destination | regex_replace('\\$\\w+', lookup('env', '\\1')) }}"
with_items: "{{ dotfiles.mac | list }}"
The destination variable evaluates to /.
Below is a quick and dirty fix of your original idea. This will only work if you target localhost (as lookups only run on localhost) hence will fail for a remote target (as the env var will be gathered for the current user on your local environment).
Moreover, the regexes/replace/lookups will probably fail as soon as you introduce more than one env var in your string(s). But at least you get a first working example for your particular situation and can build over it.
The following example playbook (using the same json file as in your question):
- name: "Replace dollar env var with local env value"
hosts: localhost
gather_facts: false
vars:
dotfiles: "{{ lookup('file', 'dotfiles.json') }}"
tasks:
- name: Replace dollar env var with local env value
vars:
env_var: >-
{{ item.destination | regex_replace('\$(\w+)/', '\1') }}
destination: >-
{{ item.destination | regex_replace('\$\w+[^/]', lookup('env', env_var)) }}{{ item.name }}
debug:
msg: "{{ destination }}"
loop: "{{ dotfiles.mac }}"
Gives (username redacted)
$ ansible-playbook bootstrap.yml
PLAY [Replace dollar env var with local env value] *****************************************************************************
TASK [Replace dollar env var with local env value] *****************************************************************************
ok: [localhost] => (item={'name': '.gitconfig', 'destination': '$HOME/'}) => {
"msg": "/home/somelocaluser/.gitconfig"
}
PLAY RECAP *****************************************************************************
localhost : ok=1 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0

Receiving Error when I read file line by line in ansible

The goal is to read the input file for a string that matches the regex expression then to match the expression to my config file.. when there is a match ansible is to replace the existing line in the config file with the matching line in the delta file.
I was able to perform this task but noticed that ansible would read one line and essentially be done. I added the .splitlines() option to my code so that it would read line by line and perform the same action but i received the following error:
- name: Search for multiple reg expressions and replace in config file
vars:
# pull data from config file
input: "{{ lookup('file', '{{ record }}').splitlines() }}"
delta: "{{ input | regex_search('^.*[a-zA-Z]+.*_.*[a-zA-Z]+.*?', multiline=True )}}"
delta1: "{{ input | regex_search('^.*[a-zA-Z]+.*_.*[a-zA-Z]+.*', multiline=True)}}"
record: "/etc/ansible/roles/file_config/files/records/records.config.{{ inventory_hostname }}"
lineinfile:
path: /dir/dir/WCRUcachedir/records.config
# Line to search/Match against
regexp: "{{item.From}}"
# Line to replace with
line: "{{item.To}}"
state: present
backup: yes
with_items:
- { From: '{{delta}}', To: '{{delta1}}' }
This happened to be my end result
"msg": "An unhandled exception occurred while templating '{{ input | regex_search('^.*[a-zA-Z]+.*_.*[a-zA-Z]+.*', multiline=True )}}'. Error was a <class 'ansible.errors.AnsibleError'>, original message: Unexpected templating type error occurred on ({{ input | regex_search('^.*[a-zA-Z]+.*_.*[a-zA-Z]+.*', multiline=True )}}): expected string or buffer"
}
these are what i believe my conflicting lines are
input: "{{ lookup('file', '{{ record }}').splitlines() }}"
AND
delta1: "{{ input | regex_search('^.[a-zA-Z]+._.[a-zA-Z]+.', multiline=True)}}"
OK, you do have another problem.
In the vars section, when you're setting delta and delta1, regex_search is expecting a string, but your passing a list (which splitlines() created). But you need it to work on one line at a time.
So, rather that input, use item, which will be set in the loop:
vars:
input: "{{ lookup('file', '{{ record }}').splitlines() }}"
delta: "{{ item | regex_search('^.*[a-zA-Z]+.*_.*[a-zA-Z]+.*?')}}"
delta1: "{{ item | regex_search('^.*[a-zA-Z]+.*_.*[a-zA-Z]+.*')}}"
Obviously, you don't need the multiline=True any more.
Now, the loop will look like this:
lineinfile:
path: /etc/opt/CCURcache/records.config
regexp: "{{ delta }}"
line: "{{ delta1 }}"
state: present
backup: yes
loop: "{{ input }}"
when: delta != ""
Yes, you only have one item to loop over. That item has two elements, From and To.
From is {{ input | regex_search('^.*[a-zA-Z]+.*_.*[a-zA-Z]+.*?', multiline=True )}}
To is {{ input | regex_search('^.*[a-zA-Z]+.*_.*[a-zA-Z]+.*', multiline=True)}}
All you did in the vars section was define strings, which will get executed later, when the regex and line are used in the module.
Now, assuming you need to put these together, you need to zip them:
lineinfile:
path: /etc/opt/CCURcache/records.config
regexp: "{{item.0}}"
line: "{{item.1}}"
state: present
backup: yes
loop: "{{ delta|zip(delta1)|list }}"
Here's a simple example:
---
- hosts: localhost
connection: local
gather_facts: no
vars:
list1:
- key_1
- key_2
- key_n
list2:
- value_1
- value_2
- value_n
tasks:
- name: Loop over lists
debug:
msg: "key is {{ item.0 }}; value is {{ item.1 }}"
loop: "{{ list1|zip(list2)|list }}"
And the results:
PLAY [localhost] *******************************************************************************
TASK [Loop over lists] *************************************************************************
ok: [localhost] => (item=['key_1', 'value_1']) => {
"msg": "key is key_1; value is value_1"
}
ok: [localhost] => (item=['key_2', 'value_2']) => {
"msg": "key is key_2; value is value_2"
}
ok: [localhost] => (item=['key_n', 'value_n']) => {
"msg": "key is key_n; value is value_n"
}
PLAY RECAP *************************************************************************************
localhost : ok=1 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
Good luck!

Loops within loops

I've set up some application information in my Ansible group_vars like this:
applications:
- name: app1
- name: app2
- name: app3
- name: app4
settings:
log_dir: /var/logs/app4
associated_files:
- auth/key.json
- name: app5
settings:
log_dir: /var/logs/app5
repo_path: new_apps/app5
I'm struggling to get my head around how I can use these "sub loops".
My tasks for each application are:
Create some folders based purely on the name value
Create a log folder if a settings/log_dir value exists
Copy associated files over, if specified
The syntax for these tasks isn't the problem here, I'm comfortable with those - I just need to know how to access the information from this applications variable. Number 3 in particular seems troublesome to me - I need to loop within a loop.
To debug this, I've been trying to run the following task:
- debug:
msg: "{{ item }}"
with_subelements:
- "{{ applications }}"
- settings
Here's the output:
with_items: I get the error with_items expects a list or a set
with_nested: I can see the top level information (e.g. msg: {{ item }} outputs an array of app1, app2 etc)
with_subelements: I get the error subelements lookup expects a dictionary, got 'None'
It's possible/probable that the way I've set the variable up in the first instance is wrong. If there's a better way to do this, it's not a problem to change it.
You can't use with_subelements because settings is a dictionary, not a list. If you were to restructure your data so that settings is a list, like this:
applications:
- name: app1
- name: app2
- name: app3
- name: app4
settings:
- name: log_dir
value: /var/logs/app4
- name: associated_files
value:
- auth/key.json
- name: app5
settings:
- name: log_dir
value: /var/logs/app5
- name: repo_path
value: new_apps/app5
You could then write something like the following to iterate over each setting for each application:
---
- hosts: localhost
gather_facts: false
vars_files:
- applications.yml
tasks:
- debug:
msg: "set {{ item.1.name }} to {{ item.1.value }} for {{ item.0.name }}"
loop: "{{ applications|subelements('settings', skip_missing=true) }}"
loop_control:
label: "{{ item.0.name }}.{{ item.1.name }} = {{ item.1.value }}"
(I'm using loop_control here just to make the output nicer.)
Using the sample data you posted in applications.yml, this will produce as output:
PLAY [localhost] *********************************************************************
TASK [debug] *************************************************************************
ok: [localhost] => (item=app4.log_dir = /var/logs/app4) => {
"msg": "set log_dir to /var/logs/app4 for app4"
}
ok: [localhost] => (item=app4.associated_files = ['auth/key.json']) => {
"msg": "set associated_files to ['auth/key.json'] for app4"
}
ok: [localhost] => (item=app5.log_dir = /var/logs/app5) => {
"msg": "set log_dir to /var/logs/app5 for app5"
}
ok: [localhost] => (item=app5.repo_path = new_apps/app5) => {
"msg": "set repo_path to new_apps/app5 for app5"
}
PLAY RECAP ***************************************************************************
localhost : ok=1 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0

Resources