Similar to e.g. this: Parse out key=value pairs into variables
.env
key1=value1
key2=value2
config.yml
some:
key1: ${key1}
key2: ${key2}
How can I replace the values in config.yml with the values from .env?
So far I got this:
awk 'FNR==NR {n=index($1,"=");if(n){vars[substr($i, 1, n - 1)]=substr($i, n + 1)}; next } { for (i in vars) gsub("${"i"}", vars[i]) }1' .env file.yml
^- Can't figure out how to replace $
I can make it work with {key1} but not with ${key1}. How can I achieve this?
The simplest would be:
. .env
export key1 key2
envsubst '$key1 $key2' < config.yml
Can't figure out how to replace $
Seems you have to escape $ and { and }, like:
gsub("\\$\\{"i"\\}", vars[i])
Without using awk:
(set -a; source .env; envsubst < config.yml)
some:
key1: value1
key2: value2
(...): Runs command line in a sub-shell to avoid polluting current shell's environment
set -a: Exports all declaring variables
source .env: Source in .env file
envsubst < config.yml: Substitutes env variables in config.yml
Related
Given yaml config file that looks like this:
key1:
key11:value1
key12:value2
key2:
key21:value3
How can I convert it in a bash script (preferable with yq) to env vars prefixed with a string? Desired output for env:
TF_VAR_key11=value1
TF_VAR_key12=value2
TF_VAR_key21=value3
Assuming the missing spaces in the input's subelements between key and value are intentional, so we are just dealing with an array of string values containing :, and separated by whitespace.
yq '.[] | split(" ") | .[] | split(":") | "TF_VAR_" + .[0] + "=" + .[1]' file.yaml
Which implementation of yq are you using? This works for mikefarah/yq. To be used with kislyuk/yq, add the -r option.
You'd first load the YAML configuration into memory.
from yaml import loads
with open("config.yaml") as f:
config = loads(f.read())
Then, iterate over the dictionary values, which appear to also be dictionaries. For each of these dictionaries, write the key=val pair to the new file.
env_str = ""
for inner_dict in config.values():
for key, val in inner_dict.items():
env_str = f"TF_VAR_{key}={val}\n"
Using python as suggested is easy and readable, if you need to do everything in bash, you can check this thread which has various solutions:
How can I parse a YAML file from a Linux shell script?
I am trying to pass bash variable in yq
test.yml
configuration:
Properties:
corporate-url: https://stackoverflow.com/
temp = '.configuration.Properties.corporate-url'
export $temp
Value1=$(yq '.[env($temp)]' test.yml)
expected output:
https://stackoverflow.com/
but I am getting this error(Actual output)
Error: Value for env variable '$variable1' not provided in env()
Please note:
I am trying to fetch corporate-url value, using a bash variable, constraint is that I cannot pass string directly in yq as the value of temp changes as this snippet is running inside a for loop which changes value of temp every time so cannot hard code for a particular value.
Reference YQ Documentation:
https://mikefarah.gitbook.io/yq/operators/env-variable-operators
ApisDraft folder contains multiple yml files
ApisDraft=$(find drafts/* -maxdepth 1 -type f)
for ApiFixOrgsTags in $ApisDraft
do
my_var=$(yq '.securityDefinitions.[].tokenUrl' $ApiFixOrgsTags)
ConfigProper='.configuration.Properties.'
CatelogProper='.configuration.catalogs.[].Properties.'
variable1=$ConfigProper${my_var}
variable2=$CatelogProper${my_var}
# to remove white all spaces
variable1= echo $variable1 | sed -E 's/(\.) */\1/g'
variable2= echo $variable2 | sed -E 's/(\.) */\1/g'
export $variable1
export $variable2
Value1=$(yq "$variable1" $ApiFixOrgsTags)
Value2=$(yq '.[env($variable2)]' $ApiFixOrgsTags)
done
In this case, you don't need to put it in the environment. Let the shell expand it so yq just sees the value of the variable:
yq "$temp" test.yml # => https://stackoverflow.com/
I am using yq v4.27.2 and want to achieve this using the environment variable as the key:
yq -n '.key1.key2="value"'
key1:
key2: value
However, I get the error as:
export KEY=key1.key2; yq -n '.env(KEY)="value"'
Error: 1:6: invalid input text "KEY)=env(VALUE)"
With this, the whole key is treated as one node,
export KEY="key1.key2"; yq -n '.[env(KEY)]="value"'
key1.key2: value
What is the right syntax? Thanks
Found the syntax after reading the source code here https://github.com/mikefarah/yq/blob/master/pkg/yqlib/operator_env_test.go#L61
export KEY=.key1.key2; ./yq -n 'eval(strenv(KEY))="value"'
key1:
key2: value
I have various environment variables declared inside an env_file referenced in a docker-compose.yml
export FOO=bar
export FAZ=baz
In a config file within a container I reference these environment variables as such:
Today is a great day to ${FOO} and ${FAZ}
I want to be able to capture all instances of text start with ${ and ending with } (ex. ${<SOMETHING>}) and then replace it with the environment variable of key <SOMETHING>.
The following works in the shell, but I cannot get this to work with sed -i against a file, or within a bash script.
echo "TODAY IS ${DAY}" | sed -r 's/(<foobar>\$[\w{}]+)/<foobar>/g
where the environment variable DAY is "FUNDAY" would produce: "TODAY IS FUNDAY"
I think i found something which may interest you. Please note that it complies to using only sed, in place editing and bash. Before we get into it, lets see the input and output:
Environment file/variables: As seen below, our file "myenv" has 3 variables and their values are as seen below.
%_Host#User> file myenv
myenv: ASCII text
%_Host#User> cat myenv
export FOO='This is FOO'
export BAR='This is BAR'
export BAZ='This is BAZ'
%_Host#User> env|egrep "FOO|BAR|BAZ"
FOO=This is FOO
BAR=This is BAR
BAZ=This is BAZ
%_Host#User>
Target file to perform the operation upon: Our target is to read the file sample.txt and replace values of variables with their actual values, if they are found in the loaded environment file. Our 4th line is not defined in our variable file so it should be untouched or unchanged.
%_Host#User> cat sample.txt
${FOO} is TRUE
${BAR} is TRUE
${BAZ} is TRUE
${BRAZ} is TRUE
%_Host#User>
SCRIPT Result and final output: To demonstrate, I am printing the lines as they are read from the target file sample.txt. If the variable inside the container (eg: ${FOO} is found in any of the line, we replace the it with its value (eg: This is FOO in this case) which we can obtain from the env
%_Host#User> ./env.sh sample.txt ; cat sample.txt
[1.] Line is: [${FOO} is TRUE]
[1.] VAR:[FOO] has VALUE:[This is FOO]
[2.] Line is: [${BAR} is TRUE]
[2.] VAR:[BAR] has VALUE:[This is BAR]
[3.] Line is: [${BAZ} is TRUE]
[3.] VAR:[BAZ] has VALUE:[This is BAZ]
[4.] Line is: [${BRAZ} is TRUE]
[4.] VAR:[BRAZ] not found and NO CHANGE IN FILE !!
This is FOO is TRUE
This is BAR is TRUE
This is BAZ is TRUE
${BRAZ} is TRUE
%_Host#User>
As seen above, we have managed to change the ${FOO} with This is FOO as expected and left any other/s (which were not in our env file) untouched.
SCRIPT and its working:
#!/bin/bash
i=1
cfgfile="$1"
cat $cfgfile | while read line
do
echo "[${i}.] Line is: [$line]"
var=$(echo $line | sed 's#^.*\${\(\w*\)}.*$#\1#g')
val=$(env |grep "$var"| cut -f2 -d"=")
env|grep "${var}=" >/dev/null
if [ $? -eq 0 ]
then
echo "[$i.] VAR:[$var] has VALUE:[$val]"
# Perform inplace editing.
sed -i "s#\${$var}#$val#g" "$cfgfile"
else
echo "[$i.] VAR:[$var] not found and NO CHANGE IN FILE !!"
fi
((i++)) ; echo
done
Above script is following these steps:
i=1 is just our counter. Not so important. Only for demo purpose.
cfgfile="$1" to accept file from prompt.
We cat the file and read it line by line in a while loop.
We echo the line and then extract FOO from ${FOO} and its actual value from env i.e. This is FOO and store it in var and val. simple enough.
We test whether this variable FOO is defined in env or not and check for command status.
If it is defined i.e. $? is 0 or True, we go ahead and perform in place editing or else we leave the line alone and no change.
I believe you should try your original command with double quotes (") instead of single ('). It should work.
Please let us know if this was any useful.
Cheers.
I have a mustache like template file that needs to be parsed. e.g.
abc def ghijk{{ var1 }} lmno {{ var2 }} pq
rst={{ var3 }} uvwzyx
Variables like {{ var1 }} need to be replaced by a variable with the same name, which is already defined previous in the bash script, e.g var1="foobar".
I am thinking of using while read and awk to accomplish this, but don't know what is the correct way to do string manipulation in this case.
Thanks in advance!
export var1="duper"
export var2="tester"
export var3=1231
sed -e 's/{{ *\([^} ]*\) *}}/$\1/g' -e 's/^/echo "/' -e 's/$/"/' input | sh
Gives:
abc def ghijkduper lmno tester pq
rst=1231 uvwzyx
Here's an all awk version that requires a file of key=value pairs for the replacements. If I make a vars file like:
var1 =foobar
var2 =elf
var3 =monkey data
where I "cheated" and included the whitespaces associated with vars in your data file. Then I made an executable awk file like:
#!/usr/bin/awk -f
BEGIN {FS="="}
NR==FNR {vars[$1]=$2; next}
set_delims==0 {FS="[{][{]|[}][}]"; $0=$0; set_delims=1 }
{
for(i=1;i<=NF;i++) {
printf( "%s%s", ($i in vars) ? vars[$i] : $i, i==NF?"\n":"")
}
}
If the executable awk file is called awko it can be run like awko vars data:
abc def ghijkfoobar lmno elf pq
rst=monkey data uvwzyx
I had a similar issue recently and found this nice mustache implementation in bash: https://github.com/tests-always-included/mo
If you have all your variables already defined in the current bash context it's as simple as calling:
./mo template.file
With a slightly different usage it also supports other mustache features like partials or arrays.
To do this in a scalable manner (lots of macros to expand), you'll probably want to look into using a macro processor. GNU m4 springs to mind, though it only allows alphanumeric and _ characters in macro names, so may be tricky for this particular task.
To get you started, you can use sed to do the replacements with this sed script (I called it mustache.sed):
s/{{ var1 }}/foobar/g
s/{{ var2 }}/bazbar/g
s/{{ var3 }}/quxbar/g
In use with your example:
$ sed -f mustache.sed mustache.txt
abc def ghijkfoobar lmno bazbar pq
rst=quxbar uvwzyx
$
You could put this sed script all on one sed command line, but I think using the script makes it more readable.