Is there a way to make templates optional in Nomad job files?
If test.data does not exist in Consul KV I want the job to simply ignore the template.
Below example using keyOrDefault nearly does what I want but still creates an empty file in my container "testfile".
I don't want any file to be created at all if key does not exist.
template {
destination = "local/testfile"
perms = "100"
data = "{{ keyOrDefault \"service/foo/bar/test.data\" \"\" }}"
}
If possible I would like to include the entire template in an if statement.
Related
I am using Ansible and have a directory structure like the example below:
configs
something
files
1.conf
2.conf
// and so on
Those files are templates and I am using Ansible to parse these templates and create them automatically in the destination server.
My problem is that with_fileglob is working only of first level directory and cannot seem to enable some recursive mode.
I have
- name: "Apply templates"
template:
src: "{{ item }}"
dest: "{{ item | replace('.j2', '') }}"
with_fileglob:
- "{{ user_configs_path }}/*"
by the way user_configs_path=configs exists and all good here.
The above does nothing.
If I add something under configs, example configs/blabla.j2 and re-run the playbook it is parsed and copied fine.
So seems somehow that the directories are not searched recursively.
I am not limited to only use the fileglob solution so feel free to suggest anything I can learn to reach my goal.
Basically I want to recursively iterate all directories for files only, and in a loop apply the template module to them and create them in remote server
Thank you
Regarding
My problem is that with_fileglob is working only of first level directory and cannot seem to enable some recursive mode.
and according the documentation fileglob
Matches all files in a single directory, non-recursively, that match a pattern. It calls Python’s “glob” library.
for
How to make Ansible with_fileglob work recursively for all subdirectories
one would need to enhance the module code.
Regarding
Basically I want to recursively iterate all directories for files only, and in a loop apply the template module to them and create them in remote server
as solution and depending on your requirements and what you try to achieve, you could use just
find module – Return a list of files based on specific criteria, in example .conf files. It works with parameter recursive: true and will you provide with a list of full path(s) over which you can loop after.
shell module – Execute shell commands on targets like find and register: result
Similar Q&A
Ansible playbook to find out specific files in sub directories
Ansible: How to find latest files from a directory recursively?
How to search for files containing a particular text with Ansible?
According to the ansible documentation, I can use slurp to read a remote file.
I have a java properties file on a remote host that I want to slurp so I did:
- name slurp xyz properties
slurp:
src: /some/path/on/the/remote/my.properties
register: myprops
- debug:
msg: "{{ myprops['content'] | b64decode }}"
If I do that I get the content.
Now I want to use that conent in ansible. E.g. through a lookup. Something like this:
{{lookup('somePropertyInPropertiesFile', myprops['content'])}}
But this doesn't work since the lookup module only allows lookup in files.
How can I pass the slurped file to the lookup?
I'm using ansible 2.9.9
Given ansible lookups work on the control host, you can also get a file from remote to local using fetch module. Then use ini lookup to read a specific property from the properties file.
I couldn't test the code but something like below should work.
- name: Fetch my properties
fetch:
src: /some/path/on/the/remote/my.properties
dest: /tmp/
flat: yes
- debug:
msg: "content is {{ lookup('ini', 'content type=properties file=/tmp/my.properties') }}"
flat: yes will copy the file under /tmp without creating a dir with hostname in the given destination dir which is the default behavior. This might be useful if you have a single host or do not care if the file gets overwritten.
The answer of Moon is correct, but I want to add more details about java property files.
It seems that ansible can only hanlde simple java property files like:
user.name=robert
user.pass=somerandompassword
But the java properties file format also allowes values to span serveral lines, e.g.
targetCities=\
Detroit,\
Chicago,\
Los Angeles
When you have entries like this, ansible doesn't parse them correctly. Ansible's result is:
ok: [..............] => {
"msg": "content is \\\nDetroit,\\\nChicago,\\\nLos Angeles"
}
But the java Properties documentation say:
Properties are processed in terms of lines. There are two kinds of line, natural lines and logical lines. A natural line is defined as a line of characters that is terminated either by a set of line terminator characters (\n or \r or \r\n) or by the end of the stream. A natural line may be either a blank line, a comment line, or hold all or some of a key-element pair. A logical line holds all the data of a key-element pair, which may be spread out across several adjacent natural lines by escaping the line terminator sequence with a backslash character .
Thus
targetCities=\
Detroit,\
Chicago,\
Los Angeles
should be equivalent to
targetCities=Detroit,Chicago,Los Angeles
which is not the case as explained above.
EDIT
Sadly some property files, like tomcat's catalina.properties, can not be parsed at all.
fatal: [..............]: FAILED! => {"msg": "An unhandled exception occurred while running
the lookup plugin 'ini'. Error was a <class 'ConfigParser.ParsingError'>, original
message: File contains parsing errors: <???>\n\t[line 35]: u'org.apache.jasper.,org.apache.naming.,org.apache.tomcat.\\r\\n'\n\t[line 110]:
...
I have a file /tmp/components_list with a content like this:
ComponentA: '1263'
ComponentB: '989'
ComponentC: '1354'
I want to register variables in ansible (without quotes), according to the content of file and use them in the yml code.
So, as a result I need something like that:
- name: Get variables from file
Some actions with a file /tmp/components_list
- name: Using these variables
shell: docker run --name component artifactory:5100/radware/Component:{{ComponentA}}
So it should be a number in the variable ComponentA.
How can I do it by using ansible? Thanks!
You can use include_vars statement, see also How to include vars file in a vars file with ansible?
I have a scenario where i want to process csv file and load to someother database:
Cases
pic csv file and load to mysql with the same name as csv
then do some modification on loaded rows using python task file
after that extract data from mysql and load to some other database
CSV files are coming from remote server to one airflow server in a folder.
We have to pick these csv file and process through python script.
Suppose i pick one csv file then i need to pass this csv file to rest of the operator in a dependency manner like
filename : abc.csv
task1 >> task2 >> task3 >>task4
So abc.csv should be available for all the task.
Please tell how to proceed.
Your scenarios don't have anything to do with realtime. This is ingesting on a schedule/interval. Or perhaps you could use a SensorTask Operator t detect data availability.
Implement each of your requirements as functions and call them from operator instances.
Add the operators to a DAG with a schedule appropriate for your incoming feed.
How you pass and access params is
-kwargs python_callable when initing an operator
-context['param_key'] in execute method when extending an operator
-jinja templates
relevant...
airflow pass parameter from cli
execution_date in airflow: need to access as a variable
The way tasks communicate in Airflow is using XCOM, but it is meant for small values, not for file content.
If you want your tasks to work with the same csv file you should save it on some location and then pass in the XCOM the path to this location.
We are using the LocalExecutor, so the local file system is fine for us.
We decided to create a folder for each dag with the name of the dag. Inside that folder we generate a folder for each execution date (we do this in the first task, that we always call start_task). Then we pass the path of this folder to the subsequent tasks via Xcom.
Example code for the start_task:
def start(share_path, **context):
execution_date_as_string = context['execution_date'].strftime(DATE_FORMAT)
execution_folder_path = os.path.join(share_path, 'my_dag_name', execution_date_as_string)
_create_folder_delete_if_exists(execution_folder_path)
task_instance = context['task_instance']
task_instance.xcom_push(key="execution_folder_path", value=execution_folder_path)
start_task = PythonOperator(
task_id='start_task',
provide_context=True,
python_callable=start,
op_args=[share_path],
dag=dag
)
The share_path is the base directory for all dags, we keep it in the Airflow variables.
Subsequent tasks can get the execution folder with:
execution_folder_path = task_instance.xcom_pull(task_ids='start_task', key='execution_folder_path')
using ansible looking for some idea for changing values directly on the destination system instead of copying files from source.
example:
I have many files in tmpl format on the destination system and would like to change the values related to this <%= applicationname_zk_quorum %> by using ansible variable's dynamically.
/applactionname/conf/zk.properties.tmpl
[root#applicationname-7 ~]# cat /applicationname/conf/zk.properties.tmpl
#nodes in quorum
applicationname.zk.quorum=<%= applicationname_zk_quorum %>
#applicationname.zk.quorum=zook1:6666,zook2:6666,zook3:6666
applicationname.zk.node=/applicationnames/node/
#Session timeout in milliseconds
applicationname.zk.sessionTimeout=72000
I understand that ansible variables only support for handling yaml and don't have any idea if anything it is possible to apply something for tmpl file.
thanks a lot for all your support and guidance!!!.
Please go through jinja2 documents and Ansible template module
Suppose you have this file conf/zk.properties.tmpl.j2 in your current ansible module's templates folder
- template:
src: conf/zk.properties.tmpl.j2
dest: /applicationname/conf/zk.properties.tmpl
owner: root
group: root
mode: 0644
You can define the variable in conf/zk.properties.tmpl.j2. For example:
#nodes in quorum
applicationname.zk.quorum=<%= {{ applicationname_zk_quorum }} %>
applicationname.zk.node=/applicationnames/node/
#Session timeout in milliseconds
applicationname.zk.sessionTimeout=72000
In vars folder, you can define this variable applicationname_zk_quorum as a yml file, for example, tmpl.yml.
applicationname.zk.quorum: "zook1:6666,zook2:6666,zook3:6666"