I have a simple yaml file called foo.yaml
foo:
- a
- c
bar:
- foo: bar
foo2: bar2
I'm trying to add a new value (b) to foo, in alphabetical order. I can add the value with +=, but it doesn't get alphabatized
$ yq '.foo += "b"' foo.yaml
foo:
- a
- c
- b
bar:
- foo: bar
foo2: bar2
If I use + I can use sort, but I only get the raw values. e.g.:
$ yq '.foo + "b" | sort()' foo.yaml
- a
- b
- c
I tried to set this into a bash variable and then use it with =, but it appears as a multi-line text
$ variable=$(yq '.foo + "b" | sort()' foo.yaml)
$ yq ".foo = \"$variable\"" foo.yaml
foo: |-
- a
- b
- c
bar:
- foo: bar
foo2: bar2
Is there an easier way to insert a new value into foo alphabetically, while keeping the rest of the yaml in tact?
The reason you are getting the raw values is that you've told yq to traverse into 'foo'. Instead try:
yq '.foo = (.foo + "b" | sort)' file.yaml
yields:
foo:
- a
- b
- c
bar:
- foo: bar
foo2: bar2
Explanation:
you need to update the entry in 'foo'
then, in brackets, set the new value. Normally you can use +=, but because you want to sort I've used '='
Disclaimer: I wrote yq
Related
Consider the following hello.yaml:
foos:
- foo: foo1
bar: hello
- foo: foo2
bar: world
If I want to update the bar value where foo = "foo1", I can invoke the following command:
yq '( .foos[] | select(.foo == "foo1") | .bar) |= "goodbye cruel"' hello.yaml
And that correctly outputs:
foos:
- foo: foo1
bar: goodbye cruel
- foo: foo2
bar: world
However, if I do not know that I have an item that matches, I would like to insert the appropriate entries e.g. something like yq '( .foos[] | select(.foo == "foo3") | .bar) ... would output
foos:
- foo: foo1
bar: hello
- foo: foo2
bar: world
- foo: foo3
bar: goodbye cruel
Is there a way in yq to "upsert" the array, or do I have to evaluate if the key exists upfront and perform one of two commands to insert or update?
Many thanks
Like Inian said; there is no upsert operation (at the moment). This is how I would do it - not sure if there is a better way?
yq '
with(.foos ;
select( all_c(.foo != "foo3")) | . += {"foo": "foo3"}
) |
(.foos[] | select(.foo == "foo3") | .bar) = "cool"
' hello.yaml
Explanation:
In the with block, match arrays that don't have foo: foo3, and add it.
Next, find all the elements with foo: foo3 and update them.
Disclaimer: I wrote yq
I have a file with a yaml data as below:
cat x.yml
foo:
- bar: 1
- zoo: 2
I am able to insert the text but this is messing the indentation(see 2nd line):
sed -r '/^[ ]+- bar:/a- hoo: 3' x.yml
foo:
- bar: 1
- hoo: 3
- zoo: 2
Then, I tried to backreference the leading spaces but seems like it is not working with /a flag.
sed -r '/^([ ]+)- bar:/a\1- hoo: 3' x.yml
foo:
- bar: 1
1- hoo: 3
- zoo: 2
Any help to get the following using a one-liner ?
foo:
- bar: 1
- hoo: 3
- zoo: 2
I suggest to switch to GNU sed's s command:
sed -E 's/( *)- bar:.*/&\n\1- hoo: 3/' file
Output:
foo:
- bar: 1
- hoo: 3
- zoo: 2
See: man sed and The Stack Overflow Regular Expressions FAQ
Best option is probably to use a parser. If you know exactly where the values should be, you can just pop them in there. Otherwise you'd have to loop and look for the "bar" key. This is using the YAML module.
use strict;
use warnings;
use YAML;
my $yaml = do { local $/; <> };
my $href = Load($yaml);
for (0 .. $#{ $href->{foo} }) {
if (grep { $_ eq "bar" } keys %{ $href->{foo}[$_] }) {
splice #{ $href->{foo} }, $_+1, 0, { hoo => 1 };
}
}
print Dump $href;
It outputs:
foo:
- bar: 1
- hoo: 1
- zoo: 2
Otherwise you can use Perl like so:
$ perl -pe's/^( *- *)bar.*\K/$1hoo: 1\n/s' x.yml
foo:
- bar: 1
- hoo: 1
- zoo: 2
Capture from beginning of line ^ a dash surrounded by dashes. Expect "bar", then absorb everything after it into the regex match, including the newline at the end (hence the /s modifier). Keep (\K) everything that was matched, and after it, add on the captured dash-string, plus your new content and a newline. Done.
First off, I agree with Inian saying a YML parser would be more appropriate here.
Nevertheless, you could use the s command and capture groups instead like
$ sed -r 's/^([ ]+)- bar:(.+)$/\1- bar:\2\n\1- hoo: 3/' x.yml
which gives
foo:
- bar: 1
- hoo: 3
- zoo: 2
I would like to do something like:
opt-flags : &opt_flags -DCMAKE_BUILD_TYPE=Release
dbg-flags : &dbg_flags -DCMAKE_BUILD_TYPE=Debug
common-flags: &common -DENABLE_EXAMPLES=ON -DENABLE_TESTS=ON
# concatenate previous definitions to create composed definitions
dbg: *common *dbg_flags
opt: *common *opt_flags
This doesn't work directly. Is it possible to do something equivalent to this in YAML?
No you cannot do that, an alias replaces a complete node.
However if you are dealing with mappings, you can use the merge key language-independent type if your parser supports it to combine multiple sets of keys into a new mapping:
opt-flags : &opt_flags -DCMAKE_BUILD_TYPE=Release
dbg-flags : &dbg_flags -DCMAKE_BUILD_TYPE=Debug
common-flags: &common -DENABLE_EXAMPLES=ON -DENABLE_TESTS=ON
dbg:
<< : [*common_flags, *dbg_flags]
opt:
<< : [*common_flags, *opt_flags]
This however will make two entries each, and not concatenate the strings scalars that are anchored, and will need a program that can combine the multiple values, for which the ordering is not guaranteed.
Unfortunately in 2022 you still cannot concatenate or join aliases with other aliases or strings. For mappings, there is another syntax which works the same as "merge keys" (described in the accepted answer) but is less ambiguous and easier to read IMO. You can reference multiple anchors like so (this works in docker-compose btw):
x-foo: &foo
VAR1: value1
x-bar: &bar
VAR2: value2
foobar:
<<: *foo
<<: *bar
# foobar:
# VAR1: value1
# VAR2: value2
Also worth noting that you can nest anchors too:
x-foo: &foo
VAR1: value1
bar: &bar
VAR2: value2
foobar:
<<: *foo
bar:
<<: *bar
VAR3: value3
# foobar:
# VAR1: value1
# bar:
# VAR2: value2
# VAR3: value3
Assume a text file that contains lines starting with foo and bar, respectively. Assume further that I would like to print only every fourth line of the ones starting with bar; the lines starting with foo should always be printed.
foo bar qux
# Deliberate empty line
bar baz 1
bar baz 2
bar baz 3
bar baz 4
bar baz 5
bar baz 6
bar baz 7
bar baz 8
# A miscellaneous code comment
The following code prints every fourth line irrespective of the first word and is thus not what I am looking for.
awk '/^bar/ NR == 1 || NR % 4 == 0' infile
What would the correct code be (preferentially with awk)?
EDIT:
Thanks to fedorqui for his excellent suggestion. Considering the potential appearance of empty lines and comments in the input file, I am using the following code:
user$ awk '!/^bar/ || (/^bar/ && !(++c%4))' file
foo bar qux
# Deliberate empty line
bar baz 4
bar baz 8
# A miscellaneous code comment
Just use a counter:
awk '/^foo/ || (/^bar/ && !(++c%4))' file
This prints lines that accomplish either of these:
start with "foo"
start with "bar" and this happens for the 4th time, 8th... That is, every four times a line starts with "bar".
See it in action:
$ cat a
foo1
bar1
bar2
bar3
foo2
foo3
bar4
bar5
bar6
bar7
bar8
bar9
$ awk '/^foo/ || (/^bar/ && !(++c%4))' a
foo1
foo2
foo3
bar4
bar8
This should do the trick:
awk '/^bar/ && NR % 4 == 0 || /^foo/' infile
Given we have two formatted strings that are unrelated to each other.
#test.rb
string_1 = "Title\nfoo bar\nbaz\nfoo bar baz boo"
string_2 = "Unrelated Title\ndog cat farm\nspace moon"
How can I use ruby or call shell commands to have each of these string display as columns in terminal? The key is that the data of each string are not building a correlated row, ie this is not a table, rather 2 lists side by side.
Title Unrelated Title
foo bar dog cat farm
baz space moon
foo bar baz boo
You can try using paste and column command together. Note that this is a shell command so spaces between the assignment operator should be corrected.
$ string_1="Title\nfoo bar\nbaz\nfoo bar baz boo"
$ string_2="Unrelated Title\ndog cat farm\nspace moon"
$ paste -d '|' <(echo -e "$string_1") <(echo -e "$string_2") | column -s'|' -t
Title Unrelated Title
foo bar dog cat farm
baz space moon
foo bar baz boo
We paste the lines with | as delimiter and tell column command to use | as a reference to form columns.
In Ruby, you could do it this way:
#!/usr/bin/env ruby
string_1 = "Title\nfoo bar\nbaz\nfoo bar baz boo"
string_2 = "Unrelated Title\ndog cat farm\nspace moon"
a1 = string_1.split("\n")
a2 = string_2.split("\n")
a1.zip(a2).each { |pair| puts "%-20s%s" % [pair.first, pair.last] }
# or
# a1.zip(a2).each { |left, right| puts "%-20s%s" % [left, right] }
This produces:
Title Unrelated Title
foo bar dog cat farm
baz space moon
foo bar baz boo
Hi , If you Use temp files
string_1 = "Title\nfoo bar\nbaz\nfoo bar baz boo"
string_2 = "Unrelated Title\ndog cat farm\nspace moon"
echo -e $string_1 >a.txt
echo -e $string_2 >b.txt
paste a.txt b.txt
I hope it will help.