bash search and replace a line after a certain line - bash

I have a big yaml file containing multiple declaration blocks, related to different services.
The structure is similar to the following (but repeated for multiple applications):
- name: commerce-api
type: helm
version: 0.0.5
I would like to find the block of code that is containing commerce-api and replace the version property value with something else.
The thing is, I wrote this script:
bumpConfig() {
LINE=$(awk "/- name: $1$/{print NR + $2}" "$CONFIG_YML")
sed -i "" -E "${LINE}s/version: $3.*$/\version: $4/" "$CONFIG_YML"
}
bumpConfig "commerce-api" 2 "$OLD_APP_VERSION" "$NEW_APP_VERSION"
Which is kind of allowing me to do what I want, but the only problem is that, the property version is not always on the third line.
How can I make my script to look for the first occurrence of version given the service name to be commerce-api?
Is this even possible using awk?

Adding some variation to the input file:
$ cat config.yml
- name: commerce-api-skip
type: helm
version: 0.0.5
- name: commerce-api
type: helm
bogus line1: bogus value1
version: 0.0.5
bogus line2: bogus value2
- name: commerce-api-skip-too
type: helm
version: 0.0.5
One awk idea:
bumpConfig() {
awk -v name="$1" -v old="$2" -v new="$3" '
/- name: / { replace=0
if ($NF == name)
replace=1
}
replace && $1=="version:" { if ($NF == old)
$0=substr($0,1,index($0,old)-1) new
}
1
' "${CONFIG_YML}"
}
Taking for a test drive:
CONFIG_YML='config.yml'
name='commerce-api'
OLD_APP_VERSION='0.0.5'
NEW_APP_VERSION='0.0.7'
bumpConfig "${name}" "${OLD_APP_VERSION}" "${NEW_APP_VERSION}"
This generates:
- name: commerce-api-skip
type: helm
version: 0.0.5
- name: commerce-api
type: helm
bogus line1: bogus value1
version: 0.0.7
bogus line2: bogus value2
- name: commerce-api-skip-too
type: helm
version: 0.0.5
Once OP is satisfied with the result:
if running GNU awk the file can be updated 'in place' via: awk -i inplace -v name="$1" ...
otherwise the output can be saved to a temp file and then copy the temp file over the original: awk -v name="$1" ... > tmpfile; mv tmpfile "${CONFIG_YML}"

Entirely in sed
sed -i '' "s/^version: $3/version: $4/' "$CONFIG_YML"
/^- name: $1\$/,/^- name:/ restricts the s command to just the lines between the requested name and the next - name: line.

#!/bin/bash
OLD_APP_VERSION=0.0.5
NEW_APP_VERSION=0.0.7
CONFIG_YML=config.yml
bumpConfig() {
gawk -i inplace -v name="$1" -v old="$2" -v new="$3" '
1
/^- name: / && $3 == name {
while (getline > 0) {
if (/^ version: / && $2 == old)
$0 = " version: " new
print
if (!NF || /^-/ || /^ version: /)
break
}
}
' "${CONFIG_YML}"
}
bumpConfig commerce-api "${OLD_APP_VERSION}" "${NEW_APP_VERSION}"

Related

Replace yaml config with sed

I have the following yaml file called file.yaml
# This is a file store
- file-store:
version: 2
enabled: no
- file-store2:
version: 4
enabled: yes
Is there a way to enable the file-store option for the first group?. I tried to use this one but it doesn't work:
sed -i '/^ *file-store:/,/^ *[^:]*:/s/enabled: no/enabled: yes/' file.yaml
Expected output:
# This is a file store
- file-store:
version: 2
enabled: yes
- file-store2:
version: 4
enabled: yes
In addition to the recommendations given by #HatLess you can try to use this pure sed solution. I know, it's too wordy but it's more reliable even though not 100%.
# Capture the block between "- file-store:" and "- file store.*:"
/^[ ]*- file-store:/,/^[ ]*- file-store[^:]*:/ {
# Look for te string having "enabled:"
# and replace the last non-whitespace sequence with "yes"
/enabled:/ s/[^ ][^ ]*$/yes/
}
and the short inline version of the script:
sed '/^[ ]*- file-store:/,/^[ ]*- file-store[^:]*:/ { /enabled:/ s/[^ ][^ ]*$/yes/ }'
You can use yq
$ cat file
file1:
version: 2
enabled: no
file2:
version: 4
enabled: yes
To change the value of enabled, use:
$ yq -e '.file1.enabled = "yes"' file
{
"file1": {
"version": 2,
"enabled": "yes"
},
"file2": {
"version": 4,
"enabled": true
}
}
To save in place, add -yi flag
As your question includes sed, here is a fragile solution
$ sed '/file1/ {n;N;s/no/yes/}' file
file1:
version: 2
enabled: yes
file2:
version: 4
enabled: yes

My sed command to insert lines into a file is not working - is special characters the issue?

I am trying to add a couple of lines in a text file with sed.
I think I have special characters that are giving me the issue.
I want to insert lines between
username: system:node:{{EC2PrivateDNSName}}
and
kind: ConfigMap
This is what I want to insert -
- groups:
- eks-role
- system:master
rolearn: arn:aws:iam::xxxxx:role/eks
username: eks
mapUsers: |
- userarn: arn:aws:iam::xxxxx:user/test-ecr
username: test-ecr
groups:
- eks-role
I have also tried using forward slashes around the special characters to no avail.
Here is the sed command I have now that does not work - it seems not to insert anything. I assume it can't find the line "username: system:node:{{EC2PrivateDNSName}}".
sed '/^username\: system:node\:{{EC2PrivateDNSName}}$/r'<(
echo " - groups:"
echo " - eks-role"
echo " - system:master"
echo " rolearn: arn:aws:iam::xxxxx:role/eks"
echo " username: eks"
echo " mapUsers: |"
echo " - userarn: arn:aws:iam::xxxxx:user/test-ecr"
echo " username: ecr"
echo " groups:"
echo " - eks-role"
) -i -- temp-aws-auth.yaml
Here is the contents of the file that I want to insert into -
apiVersion: v1
data:
mapRoles: |
- groups:
- system:bootstrappers
- system:nodes
rolearn: arn:aws:iam::xxxxx:role/eksctl-ops-nodegroup-linux-ng-sys-NodeInstanceRole-763ALQD2ZGXK
username: system:node:{{EC2PrivateDNSName}}
kind: ConfigMap
metadata:
creationTimestamp: "2020-12-09T15:54:56Z"
name: aws-auth
namespace: kube-system
resourceVersion: "1298"
UPDATE: Taking into consideration OPs answer/comment re: missing spaces, and a bit more fiddling, I was able to get the following sed command to work, too:
sed '/^.*username.*EC2PrivateDNSName.*$/r'<(cat replace.txt) temp-aws-auth.yaml
Assumptions:
OP is unable to use a yaml-aware tool to perform the edit
username ... EC2PrivateDNSName only shows up in one place in the file (or, alternatively, it shows up in multiple places and OP wishes to add a new line after each occurrence)
Replacement data:
$ cat replace.txt
- groups:
- eks-role
- system:master
rolearn: arn:aws:iam::xxxxx:role/eks
username: eks
mapUsers: |
- userarn: arn:aws:iam::xxxxx:user/test-ecr
username: test-ecr
groups:
- eks-role
NOTE: If the replacement data is in a variable it can fed into awk as a herestring.
One awk idea:
awk '
FNR==NR { a[FNR]=$0 # store first file (replacement data) into an array
next } # skip to next line in first file
{ print } # print current line of second file
/username.*EC2PrivateDNSName/ { for (i in a) # if we found our match then dump the contents of array a[] to stdout
print a[i]
next
}
' replace.txt temp-aws-auth.yaml
Or as a single-line:
awk 'FNR==NR {a[FNR]=$0; next} {print} /username.*EC2PrivateDNSName/ { for (i in a) print a[i]; next}' replace.txt temp-aws-auth.yaml
This generates:
apiVersion: v1
data:
mapRoles: |
- groups:
- system:bootstrappers
- system:nodes
rolearn: arn:aws:iam::xxxxx:role/eksctl-ops-nodegroup-linux-ng-sys-NodeInstanceRole-763ALQD2ZGXK
username: system:node:{{EC2PrivateDNSName}}
- groups:
- eks-role
- system:master
rolearn: arn:aws:iam::xxxxx:role/eks
username: eks
mapUsers: |
- userarn: arn:aws:iam::xxxxx:user/test-ecr
username: test-ecr
groups:
- eks-role
kind: ConfigMap
metadata:
creationTimestamp: "2020-12-09T15:54:56Z"
name: aws-auth
namespace: kube-system
resourceVersion: "1298"
I found out the issue with my original command - Sed needs the spaces included in the line it is looking for!
Since the line I was looking for has spaces in it :
' username: system:node:{{EC2PrivateDNSName}}'
I had to add the spaces to my sed statement :
sed '/^ username\: system:node\:{{EC2PrivateDNSName}}$/r'<(
Thanks for the feedback!
Happy holidays!!
This might work for you (GNU sed & cat):
cat <<\! |sed ':a;/username: system:node:{{EC2PrivateDNSName}}/{n;/kind: ConfigMap/!ba;h;s/.*/cat -/ep;g}' file
- groups:
- eks-role
- system:master
rolearn: arn:aws:iam::xxxxx:role/eks
username: eks
mapUsers: |
- userarn: arn:aws:iam::xxxxx:user/test-ecr
username: test-ecr
groups:
- eks-role
!
Make a here-document with the lines to be inserted.
Pipe the here-document through to a sed command.
If a line contains username: system:node:{{EC2PrivateDNSName}}, print it and fetch the next line.
If the following line does not contain kind: ConfigMap go to the start of the sed cycle and start again.
Otherwise, copy the current line, replace/print the current line by the lines to inserted from the here-document and then over write the replacement by the copy in the hold space.
N.B. The replacement lines are inserted into the document by way of the substitute command and the e flag, that evaluates what is substituted into the pattern space i.e. the cat - that is the here-document that is passed through via the pipe.

Sed awk text formatting

I would like to filter and order text files with something like awk or sed. It does not need to be a single command, a small bash script should be fine too.
#
home: address01
name: name01
info: info01
number: number01
company: company01
#
name: name02
company: company02
info: info02
home: home02
#
company: company03
home: address03
name: name03
info: info03
info: info032
number: number03
company: company032
#
name: name04
info: info04
company: company04
number: number04
number: number042
info: info042
I only need name, number, and info. There is always exactly one name, but there can be 0,1 or 2 number and info. The # is the only thing which is consistent and always on the same spot.
output should be:
name01,number01,,info01,
name02,,,info02,
name03,number03,,info03,info032
name04,number04,number042,info04,info042
What I tried so far:
awk -v OFS=',' '{split($0,a,": ")} /^name:/{name=a[2]} /^number:/{number=a[2]} /^info:/{info=a[2]; print name,number,info}' > dump.csv
Consider changing the logic to print on '#' or after the last line (assuming last block not terminated with #):
awk -v OFS=',' '
{split($0,a,": ")}
/^name:/{name=a[2]}
/^number:/{number=a[2]}
/^info:/{info=a[2]}
/^#/ { print name,number,info}
END { print name,number,info}
' < w.txt > dump.csv

Bash: Replace each occurrence of string in file with next value from array

I have an .yml file with three address entries and some other data, and an array containing three new addresses to replace these with
file.yml:
[...]
- address: s1.example.com
user: ubuntu
role: storage
- address: w1.example.com
user: ubuntu
role: worker
- address: w2.example.com
user: ubuntu
role: worker
[...]
array:
addr[0]: storage.domain.com
addr[1]: worker1.domain.com
addr[2]: worker2.domain.com
expected result:
[...]
- address: storage.domain.com
user: ubuntu
role: storage
- address: worker1.domain.com
user: ubuntu
role: worker
- address: worker2.domain.com
user: ubuntu
role: worker
[...]
I'm using sed, as I would like to write the new lines directly to the original file
I have tried a number of times, but the array incrementing always fails
Attempt 1
sed -i "s/- address: .*/- address: ${addr[$i]}/g" file.yml
This seems to exclusively write the first item in the array:
- address: storage.domain.com
[...]
- address: storage.domain.com
[...]
- address: storage.domain.com
[...]
Attempt 2
if (grep -e "- address:" file.yml); then
sed -i "s/- address: .*/- address: ${addr[$i]}/g"
((i++))
fi
This seems to grep all results at the same time, and forwards nothing to sed as I haven't figured that one out yet.
Output:
- address: s1.example.com
- address: w1.example.com
- address: w2.example.com
sed: no input files
Recently I started doing this with similar jobs:
Print all array members with some separator, ex. :. Put this as the first input to sed.
The first line (ie. the array members printed with some separator) are put into hold space
Then for each the - address: found:
I copy the hold space into pattern space, extract the first element of the array and append it with - address :. And print.
And remove the first element from hold space
The script below:
# replicate input
cat <<EOF >file.yml
[...]
- address: s1.example.com
user: ubuntu
role: storage
- address: w1.example.com
user: ubuntu
role: worker
- address: w2.example.com
user: ubuntu
role: worker
[...]
EOF
addr[0]=storage.domain.com
addr[1]=worker1.domain.com
addr[2]=worker2.domain.com
# the sed script
sed -nE '
1{
# hold the first line with array members separated with :
H
# don't print anything
d
}
# if the line is address
/- address: .*/{
g
s/\n//
# remove all except first array member from hold space
s/:.*//
# prepend it with _- address_
s/^/- address: /
# remove the first member from hold space
x
s/[^:]*://
x
}
# print the output
p
' <( IFS=':'; printf "%s\n" "${addr[*]}"; ) file.yml
and the same oneliner:
sed -n '1{;H;d};/- address: .*/{g;s/\n//;s/:.*//;s/^/- address: /;x;s/[^:]*://;x};p' <( IFS=':'; printf "%s\n" "${addr[*]}"; ) file.yml
will output:
[...]
- address: storage.domain.com
user: ubuntu
role: storage
- address: worker1.domain.com
user: ubuntu
role: worker
- address: worker2.domain.com
user: ubuntu
role: worker
[...]
#! /bin/bash
# initialise the array the way you want
addr[0]="storage.domain.com"
addr[1]="worker1.domain.com"
addr[2]="worker2.domain.com"
awk -F: -v addr="${addr[*]}" ' BEGIN{count=0 ; split(addr, addr_array, " ") }
{
if( $1 ~ /address/ ) {
for(i=1;i<=NF-1;++i){
printf "%s:", $i
}
printf "%s\n", addr_array[++count]
}
else
{
print
}
}' file.yml
If you want to overwrite the original file,
addr[0]="storage.domain.com"
addr[1]="worker1.domain.com"
addr[2]="worker2.domain.com"
gawk -i inplace -F: -v addr="${addr[*]}" ' BEGIN{count=0 ; split(addr, addr_array, " ") }
{
if( $1 ~ /address/ ) {
for(i=1;i<=NF-1;++i){
printf "%s:", $i
}
printf "%s\n", addr_array[++count]
}
else
{
print
}
}' file.yml
One possible solution is as follows:
addr=(storage.domain.com worker1.domain.com worker2.domain.com)
i=0
line_count=1
while IFS= read -r line; do
if [[ $line == *"- address:"* ]]; then
sed -i "${line_count}s/- address: .*/- address: ${addr[i]}/" file.yml
i=$((i+1))
fi
line_count=$((line_count+1))
done < file.yml
Above script iterates through the file, line by line and then replace the matching line with the content of the array.
This might work for you (GNU sed):
printf ": %s\n" "${addr[#]}" |
sed '/address:/R /dev/stdin' ymlFile |
sed '/address:/{N;s/:.*:/:/}'
Print the addr array out to stdout so that each address is on a separate line.
In the first invocation of sed, insert each address on a separate line following the regexp address:.
In the second invocation of sed remove the old addresses.

To remove line based on string

I have file like test.yaml file, the text content in the file like below.
servers:
- uri: "http://demo.nginx1.com/status"
name: "NGinX Monitor1"
- uri: "http://demo.nginx2.com/status"
name: "NGinX Monitor2"
I want to remove - uri line and immediate next line (start with name:) where host name = demo.nginx1.com.
I want out put like below.
servers:
- uri: "http://demo.nginx2.com/status"
name: "NGinX Monitor2"
I tied like below..
cat test.yaml | grep -v demo.nginx1.com | grep -v Monitor1 >> test_back.yaml
mv test_back.yaml test.yaml
I am getting expected out put. But it's re creating the file and I don't want to re create the file
Please help me with suitable command that i can use..
Just a simple logic using GNU sed
sed '/demo.nginx1.com/,+1 d' test.yaml
servers:
- uri: "http://demo.nginx2.com/status"
name: "NGinX Monitor2"
For in-place replacement, add a -i flag as -i.bak
sed -i.bak '/demo.nginx1.com/,+1 d' test.yaml
To see the in-place replacement:-
cat test.yaml
servers:
- uri: "http://demo.nginx2.com/status"
name: "NGinX Monitor2"
As I dislike using regular expressions to hack something you can parse - here's how I'd tackle it, using perl and the YAML module:
#!/usr/bin/env perl
use strict;
use warnings;
use YAML;
use Data::Dumper;
#load yaml by reading files specified as args to stdin,
#or piped in. (Just like how you'd use 'sed')
my $config = Load ( do { local $/ ; <>} );
#output data structure for debug
print Dumper $config;
#use grep to filter the uri you don't want.
#{$config -> {servers}} = grep { not $_ -> {uri} =~ m/demo.nginx2/ } #{$config -> {servers}};
#resultant data structure
print Dumper $config;
#output YAML to STDOUT
print Dump $config;

Resources