To remove line based on string - shell

I have file like test.yaml file, the text content in the file like below.
servers:
- uri: "http://demo.nginx1.com/status"
name: "NGinX Monitor1"
- uri: "http://demo.nginx2.com/status"
name: "NGinX Monitor2"
I want to remove - uri line and immediate next line (start with name:) where host name = demo.nginx1.com.
I want out put like below.
servers:
- uri: "http://demo.nginx2.com/status"
name: "NGinX Monitor2"
I tied like below..
cat test.yaml | grep -v demo.nginx1.com | grep -v Monitor1 >> test_back.yaml
mv test_back.yaml test.yaml
I am getting expected out put. But it's re creating the file and I don't want to re create the file
Please help me with suitable command that i can use..

Just a simple logic using GNU sed
sed '/demo.nginx1.com/,+1 d' test.yaml
servers:
- uri: "http://demo.nginx2.com/status"
name: "NGinX Monitor2"
For in-place replacement, add a -i flag as -i.bak
sed -i.bak '/demo.nginx1.com/,+1 d' test.yaml
To see the in-place replacement:-
cat test.yaml
servers:
- uri: "http://demo.nginx2.com/status"
name: "NGinX Monitor2"

As I dislike using regular expressions to hack something you can parse - here's how I'd tackle it, using perl and the YAML module:
#!/usr/bin/env perl
use strict;
use warnings;
use YAML;
use Data::Dumper;
#load yaml by reading files specified as args to stdin,
#or piped in. (Just like how you'd use 'sed')
my $config = Load ( do { local $/ ; <>} );
#output data structure for debug
print Dumper $config;
#use grep to filter the uri you don't want.
#{$config -> {servers}} = grep { not $_ -> {uri} =~ m/demo.nginx2/ } #{$config -> {servers}};
#resultant data structure
print Dumper $config;
#output YAML to STDOUT
print Dump $config;

Related

How can I replace the value inside a yaml file from another yaml file in bash

I would like to replace a single value inside a yaml file (file1.yml) with the value of another yaml file (file2.yml). The key inside both file:
app:
key: 12345
So far here is what I have done using sed with no success:
#!/bin/bash
old_value=`cat file1.yml | grep key | head -n1 | cut -f 2 -d ':'`
new_value=`cat file2.yml | grep key | head -n1 | cut -f 2 -d ':'`
sed "s/$old_value/$new_value/g;" file1.yml
I guess I should not be sending the standard output with sed and should be using awk.
To manipulate yaml files, you should employ a yaml processor, like mikefarah/yq or kislyuk/yq.
Using mikefarah/yq:
new="$(yq '.key' file2.yml)" yq -i '.key = env(new)' file1.yml
Using kislyuk/yq:
yml="$(yq -y -s '.[0].key = .[1].key | .[0]' file1.yml file2.yml)"
cat <<< "$yml" > file1.yml
Because in a yaml file the same value may exist in multiple places you want to use sed to perform a full search and then replace the value you are looking for; or use an specialized tool like yq (a jq wrapper)
For example, this yaml file is valid
app1:
key: "1234"
app2:
key: "1234"
with sed you will run the following to change key: "1234" to key: "5678" in just app2
sed '/^app2:/{n;s/key:.*/key: "5678"/;}' file.yaml
But doing the same using yq using in-place edit would look like:
yq -i -y '.app2.key = "5678"' file.yml

How do I embed YAML inside a YAML as text in bash script?

I am assembling a YAML build pipeline using bash as follows.
cat <<EOT >> ${BUILD_SOURCESDIRECTORY}/azure-pipelines.yml
- template: templates/deploy-to-env-ssh.yml#infrastructure
parameters:
dockerHostEndpoint: ${DOCKER_HOST}
jobName: ${BASENAME}
stackName: ${STACK_NAME}
composeFile: ${STACK_NAME}.yml
schedule: ???
$(cat schedule.yml)
tz: ${TZ}
EOT
What I want is to store the following YAML into schedule as a string which I can reuse in a another part of the pipeline.
version: 1.4
jobs:
DockerJob:
cmd: docker ps
time: "*"
notifyOnSuccess:
- type: stdout
data:
- stdout
But it seems it needs to be indented.
You can use the pr utility:
cat <<EOF >> ${BUILD_SOURCESDIRECTORY}/azure-pipelines.yml
- template: templates/deploy-to-env-ssh.yml#infrastructure
parameters:
dockerHostEndpoint: ${DOCKER_HOST}
jobName: ${BASENAME}
stackName: ${STACK_NAME}
composeFile: ${STACK_NAME}.yml
schedule: $(printf "\n" && pr -to 8 schedule.yml)
tz: ${TZ}
EOT
I use printf "\n" because you'd need to place the $(…) at the first column if you want to write $(…) on a new line, since every line including the first one will be offset by the given number of spaces.

Replace a value in yaml file with multi line string using sed

I have been trying to replace/add value of a field in yaml with an env variable that has multi line string, using below syntax
replacewith=" |-
multi
line
string"
sed -i -e "s/^\(\s*key-in-yaml\s*:\s*\).*/\1 $replacewith/" somefile.yaml
We can assume that key-in-yaml doesn't have any value by default. Above comand results in
sed: -e expression #1, char 37: unterminated `s' command
I also want the indentation to be maintained.
If this is the content of yaml file
apiVersion: operators.coreos.com/v1alpha1
kind: ClusterServiceVersion
metadata:
annotations:
alm-examples:
capabilities: Basic Install
after that sed command i was expecting
apiVersion: operators.coreos.com/v1alpha1
kind: ClusterServiceVersion
metadata:
annotations:
alm-examples: |-
multi
line
string
capabilities: Basic Install
With your shown samples, please try following, in case you are ok with awk.
awk -v str="$replacewith" '
1;
/annotations:/{ found=1 }
found && /alm-examples: \|-/{
print str
found=""
}
' Input_file
Once you are happy with above results(which will be shown on terminal) and in case you want to save them into Input_file itself then append > temp && mv temp Input_file to above command.
sed -E '/alm-examples/s/(.*)$/printf "\1 $replacewith"/e' somefile.yaml
This uses s///e to shell out to printf which will handle the multiline string better than attempts to inline it with sed commands. It is printf expanding the string, not sed, because the sed command is in single quotes.
This also works, with & replacing the \1, because the line break doesn't get passed to printf either way:
sed -E '/alm-examples/s/.*/printf "& $replacewith"/e' somefile.yaml
Or, to depend on annotations: in the prior line:
sed -E '/annotations:/{N;/alm-examples/s/.*/printf "& $replacewith"/e}' somefile.yaml
I have decided to use yq and it has been working great even for complex fields in yaml for example a.b.c[0].d=env-var-multiline.

How to escape single quotes in Jenkins pipeline sed command?

I cant seem to figure out how to get to write this in Jenkins pipeline?
sed -i "txt" "s/id 'com.github' ver '1.0'/id 'com.github'/g" file.txt
This is what i have been trying so far
sh """sed -i "txt" "s/id \'com.github\' ver \'1.0\'/id \'com.github\'/g" file.txt"""
I have a file with content -
id 'com.github' ver '1.0'
I want to remove
ver '1.0'
from
id 'com.github' ver '1.0'
using a bash command. How can I do this?
I would use ed instead.
printf '%s\n' "s/\(id 'com.github'\) ver '1.0'/\\1/" wq | ed file.txt
Note that the quoting issue can be fixed in sed (you are unnecessarily escaping the single quotes), but ed is more standard and designed for what you are trying to do: edit a file in place.
This worked for me in Jenkinsfile -
script {
dir('mydir') {
def command = $/sed -i "s/id 'com.github' ver '1.0'/id 'com.github'/" ./file.txt/$
SED_OUT = sh(returnStdout: true, script: command)
}
}

How can I get content of values under key images in YAML file from the shell

YAML file like this:
http:
Domain: {{ environment.domains.httpport }}
images:
emas_fe_weex: 20170810-ed0b13f
eweex_basic_manager: 20150109-e0fafa3
replicaCount:
xxxx: 1
resources:
{}
How can I get the following with shell?
emas_fe_weex: 20170810-ed0b13f
eweex_basic_manager: 20150109-e0fafa3
It is best to process YAML with a YAML parser e.g. with Python and ruamel.yaml (disclaimer: I am the author of that package). With the input in input.yaml:
< input.yaml python -c "import sys, ruamel.yaml; yaml=ruamel.yaml.YAML(); yaml.dump(yaml.load(sys.stdin)['http']['images'], sys.stdout)"
will output:
emas_fe_weex: 20170810-ed0b13f
eweex_basic_manager: 20150109-e0fafa3
I agree with Anthon: YAML is sufficiently complicated to require use of a YAML parser (like XML, JSON, CSV, etc)
Here are a few examples with other scripting languages, depending on your taste:
Ruby
ruby -ryaml -e '
data = YAML.load($stdin)
puts YAML.dump(data["http"]["images"])
' < file.yaml
Perl (requires YAML::XS from CPAN)
perl -MYAML::XS -0777 -nE '
$data = Load($_);
say Dump($data->{http}{images})
' < file.yaml
Tcl (requires tcllib)
echo '
package require yaml
set data [yaml::yaml2dict -file "file.yaml"]
puts [yaml::dict2yaml [dict get $data http images]]
' | tclsh
If you are using a system where grep is available, you can get them both with it. Assuming the data is in a file called http.yaml:
grep -e emas_fe_weex -e eweex_basic_manager http.yaml

Resources