I have a config file:
name: test
configmap:
foo1: bar1
foo2: bar2
secrets:
secrets1: value1
whatever comes under configmap should be converted to ENV Variables. Like in above example, only 2 values will be converted to ENV variables:
export foo1=bar1
export foo2=bar2
I followed this https://unix.stackexchange.com/questions/539009/export-environment-variables-parsed-from-yaml-text-file
but could not get it working.
You can use a tool like yq to create an output like:
key1=value1 key2=value2
Passing that to export will create the env vars.
export $(yq e '.configmap | to_entries | map(.key + "=" + .value) | join(" ")' input)
Local shell example, were your input is stored in a file called input:
$ echo $foo1 $foo2
$
$ export $(yq e '.configmap | to_entries | map(.key + "=" + .value) | join(" ")' input)
$
$ echo $foo1 $foo2
bar1 bar2
$
Using awk:
While I would recommend using a proper YAML parser as in the answer by #OstoneO, you could use awk to accomplish this task:
$ unset foo1; unset foo2
$ export $(awk -F": " '/configmap:/{f=1} {if(f && NF==2) {printf "export %s=%s ", $1, $2}} /secrets:/{f=0}' test.yaml);
$ echo $foo1 $foo2
bar1 bar2
Related
I am trying to pass bash variable in yq
test.yml
configuration:
Properties:
corporate-url: https://stackoverflow.com/
temp = '.configuration.Properties.corporate-url'
export $temp
Value1=$(yq '.[env($temp)]' test.yml)
expected output:
https://stackoverflow.com/
but I am getting this error(Actual output)
Error: Value for env variable '$variable1' not provided in env()
Please note:
I am trying to fetch corporate-url value, using a bash variable, constraint is that I cannot pass string directly in yq as the value of temp changes as this snippet is running inside a for loop which changes value of temp every time so cannot hard code for a particular value.
Reference YQ Documentation:
https://mikefarah.gitbook.io/yq/operators/env-variable-operators
ApisDraft folder contains multiple yml files
ApisDraft=$(find drafts/* -maxdepth 1 -type f)
for ApiFixOrgsTags in $ApisDraft
do
my_var=$(yq '.securityDefinitions.[].tokenUrl' $ApiFixOrgsTags)
ConfigProper='.configuration.Properties.'
CatelogProper='.configuration.catalogs.[].Properties.'
variable1=$ConfigProper${my_var}
variable2=$CatelogProper${my_var}
# to remove white all spaces
variable1= echo $variable1 | sed -E 's/(\.) */\1/g'
variable2= echo $variable2 | sed -E 's/(\.) */\1/g'
export $variable1
export $variable2
Value1=$(yq "$variable1" $ApiFixOrgsTags)
Value2=$(yq '.[env($variable2)]' $ApiFixOrgsTags)
done
In this case, you don't need to put it in the environment. Let the shell expand it so yq just sees the value of the variable:
yq "$temp" test.yml # => https://stackoverflow.com/
I have a yaml file say "test.yaml". The below is the content of yaml file.
...
test:
config:
abc: name1
xyz: name2
...
Now I want to read the value of abc and xyz alone from the yaml inside a shell script and store it in two variables inside shell script. test.yaml file contains additional data apart from the above one which I don't need to bother about that inside this shell script.
Eg: test.sh
var1=name1 //test[config[abc]]
var2=name2 //test[config[xyz]]
How do I read specific data (as key-value) from yaml inside a shell script. It would be really helpful if someone helps me out on this. Thanks in advance!!!
Here's an example with yq. All of the following assumes that the values do not contain newlines.
Given
$ cat test.yaml
---
test:
config:
abc: name1
xyz: name2
then
yq e '.test.config | to_entries | map(.value) | .[]' test.yaml
outputs
name1
name2
You can read them into variables like
{ read -r var1; read -r var2; } < <(yq e '.test.config | to_entries | map(.value) | .[]' test.yaml)
declare -p var1 var2
declare -- var1="name1"
declare -- var2="name2"
I would read them into an associative array with the yaml key though:
declare -A conf
while IFS="=" read -r key value; do conf["$key"]=$value; done < <(
yq e '.test.config | to_entries | map([.key, .value] | join("=")) | .[]' test.yaml
)
declare -p conf
declare -A conf=([abc]="name1" [xyz]="name2" )
Then you can write
echo "test config for abc is ${conf[abc]}"
# or
for var in "${!conf[#]}"; do printf "key %s, value %s\n" "$var" "${conf[$var]}"; done
I'm using "the Go implementation"
$ yq --version
yq (https://github.com/mikefarah/yq/) version 4.16.1
Let say I have YAML file that looks like this:
FOO: somefoo
BAR: somebar
I would like to convert this (using yq) into the following so that I can source the contents into environment variables:
export BAR='somebar'
export FOO='somefoo'
I can do it it with jq by converting the input to JSON first, but I can't seem to figure out how to do it with yq only. (I am using yq 4.x, <4.18).
So, concretely, how could I do the following using just yq?
INPUT="FOO: somefoo
BAR: somebar"
echo "$INPUT" | yq e 'to_json' - | jq -r 'keys[] as $k | "export \($k)='\''\(.[$k])'\''"'
You could switch to kislyuk's yq which uses native jq under the hood. Then, you would just need to_entries to access key and value, string interpolation in combination with the -r flag to produce the output, and #sh to escape for shell compliance:
yq -r 'to_entries[] | "export \(.key)=\(.value | #sh)"'
export FOO='somefoo'
export BAR='somebar'
Use the key operator and string concatenation:
$ echo "$INPUT" | yq $'.[] | "export " + key + "=\'" + . + "\'"'
export FOO='somefoo'
export BAR='somebar'
Tested with yq 4.27.2
The #sh operator has been added in yq v4.31.1 (with my humble contribution). Now you can do it pretty much the same way as in jq:
yq '.[] | "export " + key + "=" + #sh'
The quoting algorithm is a bit different from jq as it starts to quote only at characters that need quoting, so the literal output will likely differ, but will be later parsed equally.
# input
FOO: somefoo
BAR: somebar and some
# result
export FOO=somefoo
export BAR=somebar' and some'
With older yq versions you can still implement a primitive but safe quoting algorithm using other yq functions (the quoting is a little lovely nightmare, though):
# POSIX shell quoting starting "
yq ".[] | \"export \" + key + \"='\" + sub(\"'\", \"'\''\") + \"'\""
# POSIX shell quoting starting '
yq '.[] | "export " + key + "='\''" + sub("'\''", "'\'\\\'\''") + "'\''"'
# BASH "dollar" quoting
yq $'.[] | "export " + key + "=\'" + sub("\'", "\'\\\'\'") + "\'"'
Actually this is exactly what jq does with its #sh. In all cases this ends up as:
export FOO='somefoo'
export BAR='somebar and some'
Desired end result
I'm trying to convert the following (MS-SQL) string
INSERT INTO Foo (Bar) VALUES (CAST('1958-08-22 21:00:00.000' AS DateTime))
to SQLite syntax
INSERT INTO Foo (Bar) VALUES (-358491600)
Approach
I'm successfully doing this with the following sed arguments:
sed -r "s#cast\('(.*)' as datetime\)#date -d '\1' '+%s'#ige"
(calling date -d '...' '+%s' to convert the date to epoch)
Problem
Running the same command over the complete line:
echo "INSERT INTO Foo (Bar) values (cast('1958-08-22 21:00:00.000' as datetime))" | \
sed -r "s#cast\('(.*)' as datetime\)#date -d '\1' '+%s'#ige"
...produces an error: sh: 1: Syntax error: "(" unexpected
From what I've tracked, parenthesis cause the line to fail:
echo "() cast('1958-08-22 21:00:00.000' as datetime)" | \
sed -r "s#cast\('(.*)' as datetime\)#date -d '\1' '+%s'#ige"
Removing the e switch properly converts the command. What am I doing wrong?
this sed with ge flag does your job:
sed -r 's/(.*CAST[^\x27]*\x27)([^\x27]*)(\x27 AS DateTime.*)/
echo "\1"$(date -d"\2" "+%s")"\3"/ge' file
with your example:
kent$ cat f
INSERT INTO Foo (Bar) VALUES (CAST('1958-08-22 21:00:00.000' AS DateTime));
INSERT INTO Foo (Bar) VALUES (CAST('1958-08-23 22:00:00.000' AS DateTime));
kent$ sed -r 's/(.*CAST[^\x27]*\x27)([^\x27]*)(\x27 AS DateTime.*)/echo "\1"$(date -d"\2" "+%s")"\3"/ge' file
INSERT INTO Foo (Bar) VALUES (CAST('-358488000' AS DateTime));
INSERT INTO Foo (Bar) VALUES (CAST('-358398000' AS DateTime));
if you don't want to have the As DateTime in output, just make proper groups, I think you can manage it.
If you run your command under strace to see what exacly will be executed you will see the following:
$ echo "INSERT INTO Foo (Bar) values (cast('1958-08-22 21:00:00.000' as datetime))" | strace -ff sed -r "s#cast\('(.*)' as datetime\)#date -d '\1' '+%s'#ige" 2>&1 | grep 'execve('
execve("/bin/sed", ["sed", "-r", "s#cast\\('(.*)' as datetime\\)#dat"...], [/* 27 vars */]) = 0
[pid 8179] execve("/bin/sh", ["sh", "-c", "INSERT INTO Foo (Bar) values (da"...], [/* 27 vars */] <unfinished ...>
It means that sed tries to execute not only text that was matched the pattern but the whole line.. So, probably, you can't do that with sed (I will be glad if I'm mistaken.)
So, I suggest to gather all dates from file, convert them and then replace one by one. For example:
$ cat q.sql
INSERT INTO Foo (Bar) VALUES (CAST('1958-08-22 21:00:00.000' AS DateTime));
INSERT INTO Foo (Bar) VALUES (CAST('1958-08-23 22:00:00.000' AS DateTime));
$ sed "s|.*CAST('\([^']\+\)'.*|\1|" q.sql | while read DATE; do sed -i "s|$DATE|$(date -d "$DATE" '+%s')|" q.sql; done
$ cat q.sql
INSERT INTO Foo (Bar) VALUES (CAST('-358495200' AS DateTime));
INSERT INTO Foo (Bar) VALUES (CAST('-358405200' AS DateTime));
How to get the array name from the below ?
Getting the name of the array from config :
jobcfgUniqName=`echo ${config_data} | awk -F "#" '{print $3}'`
Creating an array of it :
for ((xx = 0; xx <= ${#joblognameSearch[#]}; xx++))
do
print $joblognameSearch[$xx]
eval ($jobcfgUniqName)[$xx]=`grep -B 3 -i error $joblogPath/$joblognameSearch[$xx]`
print jobcfgUniqName : ${jobcfgUniqName}
done
This line I tried changing many ways but did not work :
eval ($jobcfgUniqName)[$xx]
You can use declare bulletin of BASH to replace your eval by this:
declare arr_"$jobcfgUniqName"[$xx]=`grep -B 3 -i error $joblogPath/$joblognameSearch[$xx]`
Now you will have dynamic array create with prefix arr_ and some variable name $jobcfgUniqName.
TESTING:
# set the array
s='abc'
declare arr_"$s"[0]='foo'
declare arr_"$s"[1]='bar'
# retrieve the values
v1=arr_"$s"[0]
v2=arr_"$s"[1]
echo "${!v1}"
foo
echo "${!v2}"
bar
Add echo.
Example:
#!/bin/bash
A="abcd dcba"
B=A
C='eval "echo \$$B"'
eval "$C"
$ bash 1.sh
abcd dcba