I want to write an Ansible role to be able to alter a given Kafka topic. I am using a dictionary of key/value pairs.
The command module is then used to execute a Kafka script that take a string of comma separated values. For instance, use app_kafka_topic list:
---
app_kafka_topic:
cleanup.policy :
- "delete"
retention.ms :
- "146800000"
partitions :
- "6"
replication-factor :
- "2"
and create the string:
"cleanup.policy=delete,retention.ms=146800000,partitions=6,replication-factor=2"
This is what I have so far.
- name: Reading the Default Topic Properties
set_fact:
app_kafka_topic_properties_dicts: |
{% set res = [] -%}
{% for key in app_kafka_topic.keys() -%}
{% for value in app_kafka_topic[key] -%}
{% set ignored = res.extend([{'topic_property': key, 'value':value}]) -%}
{%- endfor %}
{%- endfor %}
{{ res }}
- name: Create Topic with Default Properties
command: "{{ kafka_bin_dir }}/{{ kafka_config_script }}
--zookeeper {{ prefix }}-kafka-{{ Kafka_node }}.{{ DNSDomain}}:{{ zookeeper_port }}
--entity-type topics
--alter
--entity-name {{ kafka_topic }}
--add-config
{{ properties }}"
with_items: "{{ app_kafka_topic_properties_dicts }}"
register: createdTopic
vars:
properties: |-
{% for key in app_kafka_topic.keys() %}
{% for value in app_kafka_topic[key] %}
"{{ key }}={{ value }}"
{%- endfor %}
{%- endfor %}
However, the properties variable is not concatenating the values to the end of a string. Is there a way to append the values to a string and separate them by a comma?
Is this the code that you're looking for?
play.yml
- hosts: localhost
gather_facts: no
vars:
string: ""
app_kafka_topic:
cleanup.policy :
- "delete"
retention_ms :
- "146800000"
partitions :
- "6"
replication_factor :
- "2"
tasks:
- set_fact:
string: "{{ string }}{{ (index > 0)|ternary(',','') }}{{ item.key }}={{ item.value[0] }}"
loop: "{{ app_kafka_topic|dict2items }}"
loop_control:
index_var: index
- debug:
var: string
$ ansible-playbook play.yml | grep string
"string": "retention_ms=146800000,cleanup.policy=delete,replication_factor=2,partitions=6"
Related
I am a beginner to ansible.
How can I get the content of the csv file printed in my shell?
I tryed to register the calculated value of content key and to display it via
- ansible.builtin.debug:
msg: "{{ csv_content }}"
in another task, but I cannot see it into my playbook output.
vars:
current_date: "{{ '%Y-%m-%d' | strftime }}"
tasks:
- name: Dump results to /tmp/myfile.csv
copy:
dest: /tmp/mycsv_{{ '%Y-%m-%d' | strftime }}.csv
content: |
{% for host in hosts_list %}
{% ---things--- %}
{% set idm=host.inventory_hostname.split('_')[0].split('-')[1] %}
{% set idm_padded = '%03d' % idm|int %}
{% ---things--- %}
{{ [idm_padded, --things-- ] | map('trim') | join(';') }}
{% --things--- %}
{% endfor %}
vars:
hosts_list: "{{ ansible_play_hosts | map('extract', hostvars) | list }}"
register: csv_content
run_once: yes
- ansible.builtin.debug:
msg: "{{ csv_content }}"
Rather than trying to get the content from the copy task, reverse your logic: render the content into a variable first, then use that variable as the argument to the content key in the copy task:
- hosts: localhost
gather_facts: false
tasks:
- set_fact:
csv_content: |
{% for host in hosts_list %}
{% set idm=2 %}
{% set idm_padded = '%03d' % idm|int %}
{{ [idm_padded, "things" ] | map('trim') | join(';') }}
{% endfor %}
vars:
hosts_list:
- foo
- bar
- baz
run_once: true
- debug:
msg: "{{ csv_content }}"
- name: Dump results to /tmp/myfile.csv
copy:
dest: mycsv_{{ '%Y-%m-%d' | strftime }}.csv
content: "{{ csv_content }}"
run_once: true
I want to generate a list of dictionaries through the loop. I came across that we can use with_sequence to generate the integer sequences in ansible. I am getting the following error:
The offending line appears to be:
- { "dir": "{{ mat_cleaner_input_file_path_flow }}/{{ item }}" }
with_sequence: start=0 end={{ http_range }}
^ here
We could be wrong, but this one looks like it might be an issue with
missing quotes. Always quote template expression brackets when they
start a value. For instance:
with_items:
- {{ foo }}
Should be written as:
with_items:
- "{{ foo }}"
My config.j2:
{{ cleaner_config | to_nice_yaml(indent=2) }}
My task.yml:
---
- name: Test dictionaries playbook
hosts: localhost
connection: local
vars:
mat_cleaner_input_file_path_flow: "/var/opt/miq/sftp/edr-flow"
mat_cleaner_input_file_path_http: "/var/opt/miq/sftp/edr-http"
mat_cleaner_input_retention_period: 21600
http_range: 5
cleaner_config:
- { "dir": "{{ mat_cleaner_input_file_path_flow }}/{{ item }}" }
with_sequence: start=0 end={{ http_range }}
tasks:
- name: Set Cleaner Config
template:
src: /home/osboxes/CleanerConfig/cleaner.config.j2
dest: /home/osboxes/CleanerConfig/output
delegate_to: localhost
with_items: cleaner_config
I want the final output file to be:
dir: /var/opt/miq/sftp/edr-flow/0
dir: /var/opt/miq/sftp/edr-flow/1
dir: /var/opt/miq/sftp/edr-flow/2
The format of the output file you want is text. It's not a list. You can create it by Jinja, e.g.
mat_cleaner_input_file_path_flow: "/var/opt/miq/sftp/edr-flow"
http_range: 3
cleaner_config: |
{% for i in range(http_range) %}
{{ mat_cleaner_input_file_path_flow }}/{{ i }}
{% endfor %}
gives
cleaner_config: |-
/var/opt/miq/sftp/edr-flow/0
/var/opt/miq/sftp/edr-flow/1
/var/opt/miq/sftp/edr-flow/2
Then, because it's not a list, you don't have to use to_nice_yaml in the template. Use indent instead if you want to indent the lines, e.g.
shell> cat cleaner.config.j2
{{ cleaner_config|indent(2) }}
- template:
src: cleaner.config.j2
dest: output
gives
shell> cat output
/var/opt/miq/sftp/edr-flow/0
/var/opt/miq/sftp/edr-flow/1
/var/opt/miq/sftp/edr-flow/2
Under normal circumstances, you would simplify the code, omit the cleaner_config variable, and put the Jinja iteration into the template, e.g.
mat_cleaner_input_file_path_flow: "/var/opt/miq/sftp/edr-flow"
http_range: 3
shell> cat cleaner.config.j2
{% for i in range(http_range) %}
{{ mat_cleaner_input_file_path_flow }}/{{ i }}
{% endfor %}
If you want to use a list
mat_cleaner_input_file_path_flow: "/var/opt/miq/sftp/edr-flow"
http_range: 3
cleaner_config_text: |
{% for i in range(http_range) %}
- {{ mat_cleaner_input_file_path_flow }}/{{ i }}
{% endfor %}
cleaner_config: "{{ cleaner_config_text|from_yaml }}
gives
cleaner_config:
- /var/opt/miq/sftp/edr-flow/0
- /var/opt/miq/sftp/edr-flow/1
- /var/opt/miq/sftp/edr-flow/2
Now you need to_nice_yaml to format the list, e.g.
shell> cat cleaner.config.j2
{{ cleaner_config|to_nice_yaml|indent(2) }}
gives
shell> cat output
- /var/opt/miq/sftp/edr-flow/0
- /var/opt/miq/sftp/edr-flow/1
- /var/opt/miq/sftp/edr-flow/2
How to loop over dict and filetree? I want to recursively template files with .j2 suffix (key) to destination location (value), also basename should be renamed (remove .j2 suffix). Its a perfect use case. Unfortunatelly ansible is not good with complex data structures.
Input:
vars:
applications:
application1:
svcpaths:
localfolder/bardir1: remotefolder/bardir1
localfolder/bardir2: remotefolder/bardir2
localfolder/bardir3: remotefolder/bardir3
application2:
svcpaths:
localfolder/bardir5: remotefolder/bardir5
localfolder/bardir6: remotefolder/bardir6
My try:
- name: Files to template
template:
src: "{{ item.src }}"
dest: "{{ item.destination }}/{{ item.name | regex_replace('.j2','') }}"
loop: |
[
{% for c in applications %}
{% if applications[c]['svcpaths'] is defined and applications[c]['svcpaths'] |list|length >0 %}
{% for o,m in applications[c]['svcpaths'].items() %}
{% for i in lookup('filetree', o ) %}
{% if i.state == 'file' and i.path | regex_search('.\.j2') %}
{
"name": "{{ i.path }}",
"src": "{{ i.src }}",
"destination": "{{ m }}"
},
{% endif %}
{% endfor %}
{% endfor %}
{% endif %}
{% endfor %}
]
I know that using jinja in plays is not good and I want to avoid it, if its possible. Also input datastructure should not be changed.
Thannks
If I understand what you're trying to do, I think there is a reasonably simple solution. If you write a task file like this called template_files.yml:
---
- name: render templates to dest_dir
loop: "{{ query('filetree', src_dir) }}"
# we need this to avoid conflicts with the "item" variable in
# the calling playbook.
loop_control:
loop_var: template
when: template.src.endswith('.j2')
template:
src: "{{ template.src }}"
dest: "{{ dest_dir }}/{{ (template.src|basename)[:-3] }}"
Then you can write a playbook like this:
---
- hosts: localhost
gather_facts: false
vars:
applications:
application1:
svcpaths:
localfolder/bardir1: /tmp/remotefolder/bardir1
localfolder/bardir2: /tmp/remotefolder/bardir2
localfolder/bardir3: /tmp/remotefolder/bardir3
application2:
svcpaths:
localfolder/bardir5: /tmp/remotefolder/bardir5
localfolder/bardir6: /tmp/remotefolder/bardir6
tasks:
# generate a list of {key: src, value: destination}
# dictionaries from your data structure.
- set_fact:
templates: "{{ templates|default([]) + item|dict2items }}"
loop: "{{ applications|json_query('*.svcpaths')}}"
# show what the generated variable looks like
- debug:
var: templates
# template all the things
- include_tasks: template_files.yml
loop: "{{ templates }}"
vars:
src_dir: "{{ item.key }}"
dest_dir: "{{ item.value }}"
Given that I have a set of local files that look like this:
localfolder/bardir1/example.txt.j2
localfolder/bardir2/example.txt.j2
localfolder/bardir3/example.txt.j2
localfolder/bardir5/example.txt.j2
localfolder/bardir6/example.txt.j2
Running the playbook results in:
/tmp/remotefolder/bardir6/example.txt
/tmp/remotefolder/bardir5/example.txt
/tmp/remotefolder/bardir3/example.txt
/tmp/remotefolder/bardir2/example.txt
/tmp/remotefolder/bardir1/example.txt
I think that's probably easier to read and understand than the Jinja-template based solution you're using.
I've got this yaml object
users:
user1: some description
user2: another desc
user3: ...
How can I get this object's keys? Result should be [user1, user2, user3]
EDIT: I'm using yaml variables in ansible
Array:
{{ users.keys() }}
Task:
- name: with_dict
debug:
msg: "{{ item.key }}: {{ item.value }}"
with_dict: "{{ users }}"
Template:
{% for key, value in users.iteritems() %}
{{ key|e }}: {{ value|e }}
{% endfor %}
Ansible uses Jinja, so you can do:
[ {{ obj.keys() | join(", ") }} ]
My playbook contains vars that are passed to a role. When I run it, I get [DEPRECATION WARNING]: Skipping task due to undefined Error, in the future this will be a fatal error..
Here's what I have:
---
- hosts: hadoopL0X
become: yes
become_method: sudo
vars:
logrotate_scripts:
- name: "{{ item }}"
with_items:
- zookeeper
- sa
path: "/var/log{{ item }}/{{ item }}.log "
options:
- daily
- rotate 3
- missingok
- compress
- notifempty
roles:
- log-rotation
...
The role is as such:
log-rotation/tasks/main.yml
---
- name: Setup logrotate.d scripts
template:
src: logrotate.d.j2
dest: "{{ logrotate_conf_dir }}{{ item }}"
with_items: "{{ logrotate_scripts }}"
...
log-rotation/defaults/main.yml
---
logrotate_conf_dir: "/etc/logrotate.d/"
logrotate_scripts: []
...
log-rotation/templates/logrotate.d.j2
# {{ ansible_managed }}
"{{ item.path }}" {
{% if item.options is defined -%}
{% for option in item.options -%}
{{ option }}
{% endfor -%}
{% endif %}
{%- if item.scripts is defined -%}
{%- for name, script in item.scripts.iteritems() -%}
{{ name }}
{{ script }}
endscript
{% endfor -%}
{% endif -%}
}
Any help would be much appreciated!
with_items can only be used with tasks, it cannot be used when defining variables, and because of that item isn't defined. It also looks like that the service variable isn't defined as well.