I have a file test.yaml with content:
'12345'
key1: 'foo'
key2: 'bar'
I have a system/env variable called ENV1 with value "baz"
How can I have this file as an outcome with bash?
'12345':
key1: 'foo'
key2: 'bar'
key3: 'baz'
$ sed "\$a\ key3: '$ENV1'" file
'12345'
key1: 'foo'
key2: 'bar'
key3: 'baz'
or set -i for in place.
Another alternative
$ cat file <(echo " key3: '$ENV1'")
I think this the easiest approach (this will write to the file):
echo " key3: '$ENV1'" >> file
If just want to print appending the content of the environment variable (not modifying the file)
cat file <(echo " key3: '$ENV1'")
Related
I am using yq v4.27.2 and want to achieve this using the environment variable as the key:
yq -n '.key1.key2="value"'
key1:
key2: value
However, I get the error as:
export KEY=key1.key2; yq -n '.env(KEY)="value"'
Error: 1:6: invalid input text "KEY)=env(VALUE)"
With this, the whole key is treated as one node,
export KEY="key1.key2"; yq -n '.[env(KEY)]="value"'
key1.key2: value
What is the right syntax? Thanks
Found the syntax after reading the source code here https://github.com/mikefarah/yq/blob/master/pkg/yqlib/operator_env_test.go#L61
export KEY=.key1.key2; ./yq -n 'eval(strenv(KEY))="value"'
key1:
key2: value
Similar to e.g. this: Parse out key=value pairs into variables
.env
key1=value1
key2=value2
config.yml
some:
key1: ${key1}
key2: ${key2}
How can I replace the values in config.yml with the values from .env?
So far I got this:
awk 'FNR==NR {n=index($1,"=");if(n){vars[substr($i, 1, n - 1)]=substr($i, n + 1)}; next } { for (i in vars) gsub("${"i"}", vars[i]) }1' .env file.yml
^- Can't figure out how to replace $
I can make it work with {key1} but not with ${key1}. How can I achieve this?
The simplest would be:
. .env
export key1 key2
envsubst '$key1 $key2' < config.yml
Can't figure out how to replace $
Seems you have to escape $ and { and }, like:
gsub("\\$\\{"i"\\}", vars[i])
Without using awk:
(set -a; source .env; envsubst < config.yml)
some:
key1: value1
key2: value2
(...): Runs command line in a sub-shell to avoid polluting current shell's environment
set -a: Exports all declaring variables
source .env: Source in .env file
envsubst < config.yml: Substitutes env variables in config.yml
Suppose I have two similar files:
a.yaml
data:
- name: a1
args: ["cmd", "something"]
config:
- name: some
val: thing
- name: a2
args: ["cmd2", "else"]
[...other array values...]
tags: ["something-in-a"]
values: ["else-in-a"]
substitutions:
key1: a-value
key2: a-value
key3: a-value
b.yaml
data:
- name: b1
args: ["cmd", "something"]
config:
- name: some
val: thing
- name: b2
args: ["cmd2", "else"]
[...other array values...]
tags: ["something-in-b"]
values: ["else-in-b"]
substitutions:
key1: b-value
key2: b-value
key3: b-value
My goal is to combine parts of a and b file such that I have a new file which consists of file content before substitutions: from b.yaml and content including and after substitutions: from a.yaml
So in this case, my desired output would be like this:
c.yaml
data:
- name: b1
args: ["cmd", "something"]
config:
- name: some
val: thing
- name: b2
args: ["cmd2", "else"]
[...other array values...]
tags: ["something-in-b"]
values: ["else-in-b"]
substitutions:
key1: a-value
key2: a-value
key3: a-value
The parts before and after substitutions: in both file contents might have different lengths.
Currently, my method is like this:
head -q -n `awk '/substitution/ {print FNR-1}' b.yaml` b.yaml >! c.yaml ; \
tail -q -n `awk '/substitution/ {ROWNUM=FNR} END {print NR-ROWNUM+1}' a.yaml` a.yaml >> c.yaml; \
rm a.yaml b.yaml; mv c.yaml a.yaml; # optional newfile renaming to original
But I wonder if there's an alternative or better method for combining parts of different files based on a common text string in bash?
Use awk, you just need to flag the flow based on the string:
awk '$1 == "substitutions:"{skip = FNR==NR ? 1:0}!skip' b.yaml a.yaml
Explaination:
FNR==NR: if true, process lines in the first file b.yaml, otherwise the 2nd file a.yaml
!skip: if TRUE, print the line, otherwise skip the line.
{
head -B9999 'substitutions:' a.yaml | head -n -1
head -A9999 'substitutions:' b.yaml
} > c.yaml
A oneliner:
{ head -B9999 'substitutions:' a.yaml | head -n -1; head -A9999 'substitutions:' b.yaml; } > c.yaml
The -A9999 and -B9999 are a bit dirty, here's a solution with sed's:
{
sed '/substitutions:/,$d' a.yaml
echo substitutions:
sed '1,/substitutions:/d' b.yaml
} > c.yaml
I would like to do something like:
opt-flags : &opt_flags -DCMAKE_BUILD_TYPE=Release
dbg-flags : &dbg_flags -DCMAKE_BUILD_TYPE=Debug
common-flags: &common -DENABLE_EXAMPLES=ON -DENABLE_TESTS=ON
# concatenate previous definitions to create composed definitions
dbg: *common *dbg_flags
opt: *common *opt_flags
This doesn't work directly. Is it possible to do something equivalent to this in YAML?
No you cannot do that, an alias replaces a complete node.
However if you are dealing with mappings, you can use the merge key language-independent type if your parser supports it to combine multiple sets of keys into a new mapping:
opt-flags : &opt_flags -DCMAKE_BUILD_TYPE=Release
dbg-flags : &dbg_flags -DCMAKE_BUILD_TYPE=Debug
common-flags: &common -DENABLE_EXAMPLES=ON -DENABLE_TESTS=ON
dbg:
<< : [*common_flags, *dbg_flags]
opt:
<< : [*common_flags, *opt_flags]
This however will make two entries each, and not concatenate the strings scalars that are anchored, and will need a program that can combine the multiple values, for which the ordering is not guaranteed.
Unfortunately in 2022 you still cannot concatenate or join aliases with other aliases or strings. For mappings, there is another syntax which works the same as "merge keys" (described in the accepted answer) but is less ambiguous and easier to read IMO. You can reference multiple anchors like so (this works in docker-compose btw):
x-foo: &foo
VAR1: value1
x-bar: &bar
VAR2: value2
foobar:
<<: *foo
<<: *bar
# foobar:
# VAR1: value1
# VAR2: value2
Also worth noting that you can nest anchors too:
x-foo: &foo
VAR1: value1
bar: &bar
VAR2: value2
foobar:
<<: *foo
bar:
<<: *bar
VAR3: value3
# foobar:
# VAR1: value1
# bar:
# VAR2: value2
# VAR3: value3
I have a scenario like this:
manifest.yaml
key1: "value1"
key2: "value2"
And I want to update this manifest.yaml file at run time to this
key1: "value1"
key2: "value2"
new_key: "new_value1"
so how can I add a new key at run time?
Achieved the same by the following command:
sed -i '/key2: "value2"/a new_key: "new_value1"' manifest.yml
and in case if space(5) has to be added before inserting new_key: "new_value1" then it can be done in the following way :
sed -i '/key2: "value2"/a \ \ \ \ \ new_key: "new_value1"' manifest.yml