I have below content in my file abcd.yml with below indentation:
buildConfig:
env:
credentials: # want to remove this from file
- id: TEST_ID # want to remove this from file
user: username # want to remove this from file
password: password # want to remove this from file
scan:
credentials:
- id: scan_id
user: username
password: password
Tried the below :
sed -i '/credentials:/d' abcd.yml
sed -i '/- id: TEST_ID/d' abcd.yml
sed -i '/user: username/d' abcd.yml
sed -i '/password: password/d' abcd.yml
But it is removing all the occurrence of the above strings from file which I don't want.
Expected Output:
buildConfig:
env:
scan:
credentials:
- id: scan_id
user: username
password: password
I need to do this for 1000 files .Hence a script is required to do it .The file is in .yml format.
Use a yaml parser like yq for this:
$ yq eval 'del(.buildConfig.env.credentials)' abcd.yml > newfile.yml
$ cat newfile.yml
buildConfig:
env: {}
scan:
credentials:
- id: scan_id
user: username
password: password
tested with version v4.30.8
Related
I have below content in yml file:
category:
toolSettings: settings.xml
Below snippet needs to be added to the existing under category:
env:
variables:
- user: ABC
- passowrd: BCD
Expected Output:
category:
env:
variables:
- user: ABC
- passowrd: BCD
toolSettings: settings.xml
Tried below:
yq e '."category" +=({env: {variables:[ {"user":"ABC"},{"passowrd":"BCD"}]}})' jules.yml > tmp.yml
yq -i '.category.env.variables[0].user="ABC"' jules.yml > tmp.yml
yq -i '.category.env.variables[1].passowrd="BCD"' jules.yml > tmp.yml
But none of the above are not working .
github.com/mikefarah/yq,
yq Version: 4.26.1
One should use either yq -i or a redirection on stdout, not both together.
The following code:
# Create original file
cat >jules.yml <<'EOF'
category:
toolSettings: settings.xml
EOF
# Edit that file in-place
yq -i '
.category.env.variables.user = "ABC"
| .category.env.variables.password = "DEF"
' jules.yml
# Write file to output
echo "New jules.yml file follows:"
echo "---"
cat jules.yml
...leaves jules.yml with the content:
category:
toolSettings: settings.xml
env:
variables:
user: ABC
password: DEF
...as you can see at https://replit.com/#CharlesDuffy2/EnragedPreviousScans#runme.bash
im trying to edit big textfiles with sed in one command
what i try to do is to execute sed and replace a certain string with the same string and added newlines+strings, so i can add text.
example command:
sed -i "s/openssl_verify_mode: 'none'/openssl_verify_mode: 'none'\n& domain: 'test.net'\n& user_name: 'admin'\n& password: 'mypassword'/g" /var/www/test-pro/config/configuration.yml
i use \n& to get the newlines. what am i doing wrong/what do i have to write?
sed has an append command that you can you use to add text after a match:
sed -e "/openssl_verify_mode: 'none'/a\\
openssl_verify_mode: domain: 'test.net'\\
openssl_verify_mode: user_name: 'admin'\\
openssl_verify_mode: password: 'mypassword'\\
" input-file
But it would probably be much cleaner to add the new text to a file and use the read command:
$ cat > new-text << EOF
openssl_verify_mode: domain: 'test.net'
openssl_verify_mode: user_name: 'admin'
openssl_verify_mode: password: 'mypassword'
EOF
$ sed -e "/openssl_verify_mode: 'none'/rnew-text" input
If you want to parameterize the string "openssl_verify_mode" (ie, if you want to use a different string), do it at the shell level. eg:
$ s="openssl_verify_mode"
$ sed -e "/${s}: 'none'/a\\
$s: domain: 'test.net'\\
$s: user_name: 'admin'\\
$s: password: 'mypassword'\\
" input
I strongly recommend against -i, and have omitted it from the answer. Feel free to abuse it if you like.
I can see a way to create kubectl command like this where I can pass some parameterize values ad well.
My question is, with what type of file we will save this, Is this a bash script? and how to run and supply the parameter?
export SECRET_NAME="my-app.secret"
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
name: $SECRET_NAME
type: Opaque
data:
password: $(echo -n "s33msi4" | base64 -w0)
username: $(echo -n "jane" | base64 -w0)
EOF
Yes, it can be treated as a bash script. As Jetchisel already mentioned in his comment, it contains a structure called Here Document used with cat command but it also contains export command which sets and exports a variable. So as a whole it can be treated as a simple bash script with 2 instructions.
In order to run it and create a new Secret object (which is your ultimate goal), follow these steps:
Fix the indentation which is crucial in yaml files:
export SECRET_NAME="my-app.secret"
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
name: $SECRET_NAME
type: Opaque
data:
password: $(echo -n "s33msi4" | base64 -w0)
username: $(echo -n "jane" | base64 -w0)
EOF
Save the above content as a file. You can call it secret.sh.
Source it (source and . are the same command):
. secret.sh
You should see the following message:
secret/my-app.secret created
Alternatvely you can paste it directly into the console. As you can see, it also works:
### don't copy it, this is the example console output
### you will see once you paste the above script in your bash shell
$ export SECRET_NAME="my-app.secret"
$
$ cat <<EOF | kubectl apply -f -
> apiVersion: v1
> kind: Secret
> metadata:
> name: $SECRET_NAME
> type: Opaque
> data:
> password: $(echo -n "s33msi4" | base64 -w0)
> username: $(echo -n "jane" | base64 -w0)
> EOF
secret/my-app.secret created
I am trying to add a couple of lines in a text file with sed.
I think I have special characters that are giving me the issue.
I want to insert lines between
username: system:node:{{EC2PrivateDNSName}}
and
kind: ConfigMap
This is what I want to insert -
- groups:
- eks-role
- system:master
rolearn: arn:aws:iam::xxxxx:role/eks
username: eks
mapUsers: |
- userarn: arn:aws:iam::xxxxx:user/test-ecr
username: test-ecr
groups:
- eks-role
I have also tried using forward slashes around the special characters to no avail.
Here is the sed command I have now that does not work - it seems not to insert anything. I assume it can't find the line "username: system:node:{{EC2PrivateDNSName}}".
sed '/^username\: system:node\:{{EC2PrivateDNSName}}$/r'<(
echo " - groups:"
echo " - eks-role"
echo " - system:master"
echo " rolearn: arn:aws:iam::xxxxx:role/eks"
echo " username: eks"
echo " mapUsers: |"
echo " - userarn: arn:aws:iam::xxxxx:user/test-ecr"
echo " username: ecr"
echo " groups:"
echo " - eks-role"
) -i -- temp-aws-auth.yaml
Here is the contents of the file that I want to insert into -
apiVersion: v1
data:
mapRoles: |
- groups:
- system:bootstrappers
- system:nodes
rolearn: arn:aws:iam::xxxxx:role/eksctl-ops-nodegroup-linux-ng-sys-NodeInstanceRole-763ALQD2ZGXK
username: system:node:{{EC2PrivateDNSName}}
kind: ConfigMap
metadata:
creationTimestamp: "2020-12-09T15:54:56Z"
name: aws-auth
namespace: kube-system
resourceVersion: "1298"
UPDATE: Taking into consideration OPs answer/comment re: missing spaces, and a bit more fiddling, I was able to get the following sed command to work, too:
sed '/^.*username.*EC2PrivateDNSName.*$/r'<(cat replace.txt) temp-aws-auth.yaml
Assumptions:
OP is unable to use a yaml-aware tool to perform the edit
username ... EC2PrivateDNSName only shows up in one place in the file (or, alternatively, it shows up in multiple places and OP wishes to add a new line after each occurrence)
Replacement data:
$ cat replace.txt
- groups:
- eks-role
- system:master
rolearn: arn:aws:iam::xxxxx:role/eks
username: eks
mapUsers: |
- userarn: arn:aws:iam::xxxxx:user/test-ecr
username: test-ecr
groups:
- eks-role
NOTE: If the replacement data is in a variable it can fed into awk as a herestring.
One awk idea:
awk '
FNR==NR { a[FNR]=$0 # store first file (replacement data) into an array
next } # skip to next line in first file
{ print } # print current line of second file
/username.*EC2PrivateDNSName/ { for (i in a) # if we found our match then dump the contents of array a[] to stdout
print a[i]
next
}
' replace.txt temp-aws-auth.yaml
Or as a single-line:
awk 'FNR==NR {a[FNR]=$0; next} {print} /username.*EC2PrivateDNSName/ { for (i in a) print a[i]; next}' replace.txt temp-aws-auth.yaml
This generates:
apiVersion: v1
data:
mapRoles: |
- groups:
- system:bootstrappers
- system:nodes
rolearn: arn:aws:iam::xxxxx:role/eksctl-ops-nodegroup-linux-ng-sys-NodeInstanceRole-763ALQD2ZGXK
username: system:node:{{EC2PrivateDNSName}}
- groups:
- eks-role
- system:master
rolearn: arn:aws:iam::xxxxx:role/eks
username: eks
mapUsers: |
- userarn: arn:aws:iam::xxxxx:user/test-ecr
username: test-ecr
groups:
- eks-role
kind: ConfigMap
metadata:
creationTimestamp: "2020-12-09T15:54:56Z"
name: aws-auth
namespace: kube-system
resourceVersion: "1298"
I found out the issue with my original command - Sed needs the spaces included in the line it is looking for!
Since the line I was looking for has spaces in it :
' username: system:node:{{EC2PrivateDNSName}}'
I had to add the spaces to my sed statement :
sed '/^ username\: system:node\:{{EC2PrivateDNSName}}$/r'<(
Thanks for the feedback!
Happy holidays!!
This might work for you (GNU sed & cat):
cat <<\! |sed ':a;/username: system:node:{{EC2PrivateDNSName}}/{n;/kind: ConfigMap/!ba;h;s/.*/cat -/ep;g}' file
- groups:
- eks-role
- system:master
rolearn: arn:aws:iam::xxxxx:role/eks
username: eks
mapUsers: |
- userarn: arn:aws:iam::xxxxx:user/test-ecr
username: test-ecr
groups:
- eks-role
!
Make a here-document with the lines to be inserted.
Pipe the here-document through to a sed command.
If a line contains username: system:node:{{EC2PrivateDNSName}}, print it and fetch the next line.
If the following line does not contain kind: ConfigMap go to the start of the sed cycle and start again.
Otherwise, copy the current line, replace/print the current line by the lines to inserted from the here-document and then over write the replacement by the copy in the hold space.
N.B. The replacement lines are inserted into the document by way of the substitute command and the e flag, that evaluates what is substituted into the pattern space i.e. the cat - that is the here-document that is passed through via the pipe.
I have file like test.yaml file, the text content in the file like below.
servers:
- uri: "http://demo.nginx1.com/status"
name: "NGinX Monitor1"
- uri: "http://demo.nginx2.com/status"
name: "NGinX Monitor2"
I want to remove - uri line and immediate next line (start with name:) where host name = demo.nginx1.com.
I want out put like below.
servers:
- uri: "http://demo.nginx2.com/status"
name: "NGinX Monitor2"
I tied like below..
cat test.yaml | grep -v demo.nginx1.com | grep -v Monitor1 >> test_back.yaml
mv test_back.yaml test.yaml
I am getting expected out put. But it's re creating the file and I don't want to re create the file
Please help me with suitable command that i can use..
Just a simple logic using GNU sed
sed '/demo.nginx1.com/,+1 d' test.yaml
servers:
- uri: "http://demo.nginx2.com/status"
name: "NGinX Monitor2"
For in-place replacement, add a -i flag as -i.bak
sed -i.bak '/demo.nginx1.com/,+1 d' test.yaml
To see the in-place replacement:-
cat test.yaml
servers:
- uri: "http://demo.nginx2.com/status"
name: "NGinX Monitor2"
As I dislike using regular expressions to hack something you can parse - here's how I'd tackle it, using perl and the YAML module:
#!/usr/bin/env perl
use strict;
use warnings;
use YAML;
use Data::Dumper;
#load yaml by reading files specified as args to stdin,
#or piped in. (Just like how you'd use 'sed')
my $config = Load ( do { local $/ ; <>} );
#output data structure for debug
print Dumper $config;
#use grep to filter the uri you don't want.
#{$config -> {servers}} = grep { not $_ -> {uri} =~ m/demo.nginx2/ } #{$config -> {servers}};
#resultant data structure
print Dumper $config;
#output YAML to STDOUT
print Dump $config;