I created a Variable (from Airflow UI):
Key: env_variables
Value: {'xx': 'yy`}
and trying to access using var.json.env_variables.xx inside bash operator. But the value of "xx" is dynamic, which
I am passing from REST API and accessing it using dag_run.conf['param'] (dag_run.conf['param'] should return xx). Eventually I want to run var.json.env_variables.xx. How can I achieve that inside bash operator in Airflow?
I am trying to run the following code but its showing Jinja Syntax Error.
task = BashOperator(
bash_command="export KUBECONFIG=$KUBECONFIG:{{var.json.env_variables.{{dag_run.conf['param']}} }}
What I want:
It should fetch the value of var.json.env_variables.xx. I have also tried using params but no luck, params.path is returning None.
task = BashOperator(
bash_command="export KUBECONFIG=$KUBECONFIG:{{params.path}},
params: {
"path":env_variables.get('{{dag_run.conf["param"]}}')
}
Any kind of help is highly appreciated.
You don't need the extra {{...}} in the Jinja expression around dag_run.conf and you'll also need the typical item access of a dictionary. Try using a template expression like this:
nested_vars = BashOperator(
task_id="nest_vars",
bash_command="export KUBECONFIG=$KUBECONFIG:{{ var.json.env_variables[dag_run.conf['param']] }}"
)
This is the log entry when passing in a triggering config of {"param": "xx"} to the operator instantiation above:
INFO - Running command: ['bash', '-c', 'export KUBECONFIG=$KUBECONFIG:yy']
Related
I am trying to pass to 3 AWS launch template userdata a bash script. This script calls other scripts from Github depending on a specific variable. Since each launch template must call different scripts what is the best way to accomplish it. I am currently trying to configure a data source template_file but I can´t find a way to do what I need.
This is a piece of the bash script where I put a variable that need to change its value depending on which launch template is being built every time:
#------------------------------------------------------------------------------------------
# Define here scripts (separated with 1 space) that will be executed on first run:
AMI_SCRIPTS="ami_base_lynis.sh ${ami_script}"
#------------------------------------------------------------------------------------------
download_and_run_scripts
This is the template file data source:
data "template_file" "AMIs"{
template = "${file("../AMIs/s1_aws_userdata.sh")}"
vars = {
ami = var.dci_appserver_ami
}
}
And this is the user data attribute:
user_data_base64 = base64encode(data.template_file.AMIs.rendered)
This is not working for me as it will replace the variable has the same value for all 3 launch templates. How can I assign each time a different value?
The syntax you used for user_data_base64 tells me that you're using Terraform v0.12 or later, so you should no longer use template_file as shown in the template_file documentation:
In Terraform 0.12 and later, the templatefile function offers a built-in mechanism for rendering a template from a file. Use that function instead, unless you are using Terraform 0.11 or earlier.
Because of that, I'm going to answer using the templatefile function instead.
Inside each of your launch template resource blocks, you can call templatefile with different values for the template variables in order to get a different result each time:
resource "aws_launch_template" "example1" {
# ...
user_data = base64encode(templatefile("${path.module}/../AMIs/s1_aws_userdata.sh", {
ami = var.dci_appserver_ami
ami_script = "script1.sh"
}))
}
resource "aws_launch_template" "example2" {
# ...
user_data = base64encode(templatefile("${path.module}/../AMIs/s1_aws_userdata.sh", {
ami = var.dci_appserver_ami
ami_script = "script2.sh"
}))
}
resource "aws_launch_template" "example3" {
# ...
user_data = base64encode(templatefile("${path.module}/../AMIs/s1_aws_userdata.sh", {
ami = var.dci_appserver_ami
ami_script = "script3.sh"
}))
}
You could in principle factor out constructing the templates into a local value if you want to do it more systematically, but since your question didn't include any indication that you are doing anything special with the launch templates here I've just written the simplest possible approach, where each launch template has its own template-rendering expression.
I have an ansible playbook that gets its vars passed in from an extra-vars.json file. It gets passed in at the command line with --extra-vars "#extra-vars.json".
This is an abbreviated version of the var file
{
"source" : {
"access_token" : "abc",
"git_instance_url" : "foo.com",
"repo" : "some-group/some-project/some-repo"
},
"target" : {
"access_token" : "xyz",
"git_instance_url" : "foo.bar.com",
"repo_path" : "lorem/ipsum"
}
}
Because of the var structure, when I call the vars in my playbook I have to use dot notation i.e. {{ source.repo_path }} or {{ target.access_token }}. My problem is that I would like to remove a couple of these vars from the extra-vars.json and pass them individually at the command line. If I remove source.git_instance_url from extra-vars.json I can pass it in without any precedence conflicts.
My issue is that I can't figure out how to pass dot notation vars in at the command line. I don't want to change my playbook to do this. If I pass in --extra-vars "source.git_instance_url=bar.baz.com" I get an error source is undefined.
I tried using bracket notation source[git_instance_url]=bar.baz.com with no success.
Is there a way to pass dot notation vars at the command line or am I going to have to change my playbook from {{ source.git_instance_url }} ==> {{ source_git_instance_url }} to be able to accomplish this?
When passing extra vars, these are always "string variables". I learned it the hard way when trying to pass in boolean variables.
You could pass them as json:
ansible-playbook -e '{"source": { "git_instance_url": "foo" }}' playbook.yml
But I don't know right now, if they get merged with the source var from your vars file. I'd guess, one overwrites the other. So you probably end up with either the source var from your string or with the one from the file.
I extract a JsonArray containing a list of string, and I want to validate by a regex each string inside this object.
Problem, I don't seem to find any answers on the Taurus' website.
Do you know how I can do it ?
Example below:
# Verification of value inside the JsonArray
extract-jsonpath:
names: $.names
- foreach: name in names
do:
- jsonpath: ${name} # if this JSONPATH is not found, assert will fail
validate: true # validate against an expected value
expected-value: "\\w" # value we're expecting to validate. [default: false]
regexp: true # if the value is regular expression, default: true
expect-null: false # expected value is null
invert: false # invert condition
I don't think it is possible with Taurus YAML syntax as:
foreach keyword generates a normal JMeter ForEach Controller
These jsonpath, validate, etc. are applied to a Sampler by default, they will not work if you add them just as children of the ForEach Controller
Assuming above points I would suggest adding a JSR223 PostProcesssor to perform all the checks. In Taurus it is being done via JSR223 Blocks like:
- url: https://api.example.com/v1/media/search
extract-jsonpath:
names: $.names
jsr223:'1.upto(vars.get("names_matchNr") as int,{if (vars.get("names_$it").matches("\\w+")) {prev.setSuccessful(false)}})'
See The Groovy Templates Cheat Sheet for JMeter article to get more ideas with regards to what can be done using Groovy scripts.
When creating a Go template, you can give it a name, like in this example, "my_home_template":
var tmplHome = template.Must(template.New("my_home_template").Funcs(funcMap).ParseFiles("templates/base.tmpl", "templates/content_home.tmpl"))
How can I get that template name and use it inside the actual template file?
Ultimately I just want to define a convenient css class, like so:
<body class="my_home_template">
Here's a working solution, taking mkopriva's advice:
When executing a template, pass some custom parameter with dummy data. Here, I just create a "PageHome" parameter to pass to the template, and value is a simple "1", but it could be any value:
tmplHome.ExecuteTemplate(w, "base", map[string]interface{}{"PageHome": "1", "Data": events, "UserFirstName": &u.FirstName, "UserProfilePic": &u.ProfilePic})
Then, inside the template itself, a simple if statement to check if the parameter exists, and do something accordingly:
{{ if .PageHome }}
<body class="PageHome">
{{ else }}
<body>
{{ end }}
All my other template executions don't pass a "PageHome" parameter at all, so the if statement never passes as true for them.
There's probably a more advanced solution using a functions via a template function map, and having a consistent "PageType":"something" parameter in all template executions, but in the end you still have to define a parameter per template execution and still have to build up if statements in your templates anyways.
I want to validate that the input values passed to the variables as extra_args.
I want to run a pre-task that passes if the variable contains a string value, else fails if it contains anything else.
The values are passed to them as extra_args when executing the playbook.
I want to run a pre-task that passes if the variable contains a string value, else fails if it contains anything else.
This task fails if the variable is not a string object:
- fail:
when: variable is not string
But be aware that all values passed as extra-vars will be strings, because that's what they are -- anything you type on your keyboard is a valid string. As there is no type declaration, even if a variable contains a numerical value, it will be stored in a string object.
It is different to variable values defined in YAML which undergo type autodetection performed by YAML parser. For example if you type myvar: true in YAML, it will be considered Boolean object true, but if you pass the same value with --extra-vars "myvar:true", it will be a string object true.
You need to specify another condition.
Here are few filters and tests in ansible you may find it useful
http://docs.ansible.com/ansible/latest/playbooks_filters.html
http://docs.ansible.com/ansible/latest/playbooks_tests.html
for validation you might use it as follows:
tasks:
- fail: msg="Variable '{{ item }}' is not a string"
when: string | search("^[a-zA-Z]*$")
I prefer to use the 'assert' module for such cases like this.
- name: Test if the type of 'variable' is string
assert:
that:
- variable is defined
- variable is string
fail_msg: |
variable: {{ variable | d() | to_nice_json }}
seealso: type check examples: https://github.com/ssato/ansible-role-assertive-programming-examples/blob/master/tasks/pre_type_checks.yml
BTW, if you want to define variables with types you want using --extra-vars (-e) option, you need to prepare yaml files define these variables and let them loaded using '#' such like '-e #/path/to/the/yaml_file'.