google deployment manager access nested properties with --properties argument - google-deployment-manager

If Im using a jinja template I can override properties via command line like this:
--properties zone:us-central1-a,machineType:n1-standard-1,image:debian-9
But I see no documentation or examples for doing this with nested properties like labels or environmentVariables for example:
resources:
- name: resource-name
type: 'gcp-types/cloudfunctions-v1:projects.locations.functions'
properties:
labels:
testlabel1: testlabel1value
testlabel2: testlabel2value
environmentVariables:
TEST: 'zzzzzzzzz'
How do set properties like these? this does not work: --properties labels:testlabel1:newvalue

The short answer here is that the --properties flag is not meant to pass property values to the template. A template cannot run without a configuration file, the --properties flag is meant to replace the config file. Each parameter you pass is the same as listing them in a config file.
Essentially using --template my-template.py --properties zone:us-central1-f is the equivalent of running --config myConfig.yaml where the YAML is defined as such:
imports:
- path: my-template.py
resources:
- name: some-resource
type: my-temaplte.py
properties:
zone: us-central1-f
The --properties flag is not meant to pass raw data to replace non-variables.
Although this does not directly answer your question, you shouldn't normally need to define nested values in the flag. Your template will generally call on direct variables taken from the object properties.
despite this, I did try some tests, and as far as I can tell, you can't do this.

After some trial and error I managed to pass an object via command line like this:
--properties ^~^labels:{'testlabel1: testlabel1value','testlabel2: testlabel2value'}~environmentVariables:{'TEST: zzzzzzzzz'}
This sequence of symbols ^~^ is how you can change delimiter. You have to put it at the beginning of your properties. More info about escaping you can find here.
I put single apostrophe over single key value pair because we need that space between key and value. Otherwise it's interpereted as key with null value.
If you're using Bash shell you should also escape {,} symbols.

Related

Jenkinsfile environment variables not available with parameter variables

I am using a Jenkins plugin to upload test run results to Jira. Using this plugin I can send two JSON blobs of data for the import, but the variables in those JSON blobs can only be environment variables (not variables generally available in the Jenkinsfile).
When I run it is recognizing environment variables that come from the parameters block (this is a parameterized build), but it does not recognize any environment variables I set, either in an environment {} block in the pipeline or by nesting the build step in a withEnv() {} block.
As a sanity check, right before the step in question, I echo two environment variables, one from the parameters block and one from the environment block, and both spit out to the console as expected, but then, as consumed by the plugin, only the variables coming from the parameters block are read as variables, with the rest being left as string.
So is there some difference in how these environment variables are stored/managed behind the scenes that might play into this?
So, for example, here are the parameters and environment blocks:
parameters {
choice(name: 'ENVIRONMENT', choices: ['dev', 'test', 'staging', 'prod'], description: 'Select the environment to run against.')
choice(name: 'TESTS', choices: ['All', 'API', 'Web'], description: 'Select the tests to run.')
}
environment {
PROJECT_KEY = "$jiraProjectKey"
TEST_PLAN_KEY = "$testPlanKeys[$env.ENVIRONMENT]"
PRODUCT_NAME = "$productName"
TEAM_NAME = "$teamName"
}
When I used these environment variables in the JSON blobs to set the Summary field of a Test Execution in Jira with a line that looks like this:
...
"summary": "${ENVIRONMENT} - ${PRODUCT_NAME} - ${TESTS} Tests",
...
The resulting issue summary is:
dev - ${PRODUCT_NAME} - API Tests
So it will properly interpret the environment variables set by the parameters block, but not ones I set explicitly in the environment block.
In the JSON blobs that you are sending inline make sure that for multiline strings you are using """ to delimit those strings and not '''.
Replace:
... importInfo: '''{...'''
by:
...importInfo: """{..."""

Jenkins Pipeline throws "syntax error: bad substitution" when Passing in Parameter

I have a Terraform project that I was trying to use Jenkin's Custom Checkbox plugin (Custom Checkbox Parameter) with so that I can build separate applications dynamically using the same IaC, however, I'm getting the following error when passing in the name parameter for that plugin into the Terraform plan and apply commands.
syntax error: bad substitution
The idea for all this is just to click on "select all" or each individual app and run the build, and this will create the IaC for the given application(s).
I have a terraform plan that I am running as a smoke test to verify the parameters above are being passed in correctly before running the apply. This looks like the following:
sh 'terraform plan -var-file="terraform-dev.tfvars" -var "app_name=[${params[${please-work}]}]" -input=false'
The documentation for the plugin states that you can reference the items checked by using this format: "${params['please-work']}" which is what I've done above. That said, one caveat to this is that Im having to set the values in quotes for this to work since the variables are being set in the Terraform using list(string).
NOTE: I have tested that all this works if I just hardcode the app names with the escapes as following:
sh 'terraform plan -var-file="terraform-dev.tfvars" -var "app_name=[\\"app-1\\",\\"app-2\\"]" -input=false'
Again, what I need is for this to work with the -var "app_name=[${params[${please-work}]}]" without throwing that error.
If needed, here is the setup for the JSON that the plugin is using:
Additionally, I can see the values are being set the way I need them to be set when running the echo of echo "${params['please-work']}" on the initial build step. So these are coming back as "app-1", "app-2"
Again, all but that one bit is working and I've tried various ways to escape the needed strings to get this work and I need insight on a path forward. This would be greatly appreciated.
You are casting the script argument in your sh step method as a literal string, and therefore it will not interpolate the pipeline variable of type object params within the Groovy pipeline interpreter. You also are passing the variable value for the app_name with [] syntax (attempted list constructor?), which is not syntactically valid for shell, Terraform, or JSON, but is for Jenkins Pipeline and Groovy with undesired behavior (unclear what is desired here). Finally, please-work is a literal string and not a Jenkins Pipeline or Groovy variable, and since params is technically an object and not a Map, you must use the . syntax and not the [] syntax for accessors. You must update with:
sh(label: 'Execute Terraform Plan', script: "terraform plan -var-file='terraform-dev.tfvars' -var 'app_name=${params.please-work}' -input=false")
If another issue arises after fixing all of this, then it would be recommended to convert the plugin usage to the pipeline with a parameters directive, and also to probably remove the unusual characters e.g. - from the parameter name.
Thanks for helping me think through this, Matt. I was able to resolve the issue with the following shell script in the declarative pipeline:
sh "terraform plan -var-file='terraform-dev.tfvars' -var 'app_name=[${params['please-work']}]' -input=false"
This is working now.

Fail to use yaml reference in ansible inventory plugin

I would like to use this config with an inventory plugin
# test_inventory_xxx.yml
plugin: cloudscale # or openstack or ...
inventory_hostname: &inventory_hostname_value uuid
compose:
setting_of_inventory_hostname: *inventory_hostname_value
I get no error, but the value is not set. And it is valid yaml. (At least my checker nor myself see an error.
So I decided to simplify it by using the constructed plugin, which is standard:
# inventory_constructed.yaml
plugin: constructed
# add variables to existing inventory
keyed_groups:
- key: from_inventory
prefix: inventory
parent_group: &common_parent_group test_group_1
compose:
var_from_constructed: 1233456789
also_from_constr: "'also'" # must be in quotes 2x!
some_from_constr: &ref1 1234567777
ref_from_constr: *ref1 # this works fine
ref_to_test: *common_parent_group # <--- this line returns an error
strict: yes
Now I get the error: Could not set ref_to_test for host my_host: 'test_group_1' is undefined
But it passes when I uncomment the marked line. (the ref &common_parent_group is still defined, but not used with *common_parent_group.) Why is test_group_1 undefined in one case, but not in the other?
How to reproduce: ansible -i some_of/your_inventory -i inventory_constructed.yaml -m debug -a var=vars
What do I do wrong? Or what else is the problem?
(I tought it is an missing feature, so original info in https://github.com/ansible/ansible/issues/69043)
It seems like parent_group takes a literal string while ref_to_test takes a Jinja2 expression (because it's under compose). It should fail the same way if you write
ref_to_test: test_group_1
because test_group_1 simply isn't a Jinja2 variable. You'll have to write
ref_to_test: "'test_group_1'"
just like above so Jinja2 sees 'test_group_1' which is a literal string. This also means you can't use an alias because parent_group does not evaluate its content with Jinja2 and therefore shouldn't include quotes in its content.

Fetch variable from yaml in puppet manifest

I'm doing one project for puppet, however currently stuck in one logic.
Thus, want to know can we fetch variable from .yaml, .json or plain text file in puppet manifest file.
For example,
My puppet manifest want to create user but the variable exist in the .yaml or any configuration file, hence need to fetch the varibale from the outside file. The puppet manifest also can do looping if it exist multiple users in .yaml file.
I read about hiera but let say we are not using hiera is there any possible way.
There are a number of ways you can do this using a combination of built-in and stdlib functions, at least for YAML and JSON.
Using the built-in file function and the parseyaml or parsejson stdlib functions:
Create a file at mymodule/files/myfile.yaml:
▶ cat files/myfile.yaml
---
foo: bar
Then in your manifests read it into a string and parse it:
$myhash = parseyaml(file('mymodule/myfile.yaml'))
notice($myhash)
That will output:
Notice: Scope(Class[mymodule]): {foo => bar}
Or, using the loadyaml or loadjson stdlib functions:
$myhash = loadyaml('/etc/puppet/data/myfile.yaml')
notice($myhash)
The problem with that approach is that you need to know the path to file on the Puppet master. Or, you could use a Puppet 6 deferred function and read the data from a file on the agent node.
(Whether or not you should do this is another matter entirely - hint: answer is you almost certainly should be using Hiera - but that isn't the question you asked.)

How to read Heroku's nested process.env vars / object in nconf?

I'm trying to deploy Ghost 1.2.0 to Heroku. With previous versions of Ghost (<= 0.11.x), they used a config.js file where you could just do:
database: {
client: 'postgres',
connection: {
host: process.env.POSTGRES_HOST,
user: process.env.POSTGRES_USER,
password: process.env.POSTGRES_PASSWORD,
database: process.env.POSTGRES_DATABASE,
port: process.env.POSTGRES_PORT
},
debug: false
}, …
But in this version they're using nconf that replaces this config.js for environment dependent JSON files like config.production.json.
JSON files unlike JS objects can't have vars as values. I could hardcode my credentials to the JSON file, but I don't want to do that because:
Looks like a bad practise to me, and
Heroku rotates credentials periodically and updates applications where this database is attached. (Quoted from Heroku)
After some digging into nconf README and issues I understood that it would be possible to imitate this expected database object just with:
nconf.env({
separator: '__' // Two dashes
});
and defining vars as:
heroku config:set DATABASE__CLIENT=postgres
heroku config:set DATABASE__CONNECTION__HOST=<value>
...
but, no matter what, I get undefined when I later call:
nconf.get('database');
nconf.get('DATABASE'); // In case it was case-sensitive...
Instead, if I call:
nconf.get('DATABASE__CLIENT'); // postgres
it works. I could try (and I will) to modify Ghost scripts to read all variables this way, but as long as it expects a database object it'll be so cool to get it working the right way.
So, has anybody figured out how to correctly recreate an object with Heroku's env vars?
I've finally found the solution.
Unless you want to modify nconf.env(settings) like:
nconf.env({
separator: '__', // Two dashes
lowerCase: true
});
This will make it possible to pass lowerCase: true to env() so that if an environment variable is called SOMETHING or SOMEthing, it will also be gettable using something [Source]
I recommend to use already lowercase env vars.
So,
heroku config:set database__client=postgres
will be readable using:
nconf.get('database:client');
Looks like nconf has a different character separator to define nested variables called separator and another one to read them called logicalSeparator (its default value is :)

Resources