im trying to edit big textfiles with sed in one command
what i try to do is to execute sed and replace a certain string with the same string and added newlines+strings, so i can add text.
example command:
sed -i "s/openssl_verify_mode: 'none'/openssl_verify_mode: 'none'\n& domain: 'test.net'\n& user_name: 'admin'\n& password: 'mypassword'/g" /var/www/test-pro/config/configuration.yml
i use \n& to get the newlines. what am i doing wrong/what do i have to write?
sed has an append command that you can you use to add text after a match:
sed -e "/openssl_verify_mode: 'none'/a\\
openssl_verify_mode: domain: 'test.net'\\
openssl_verify_mode: user_name: 'admin'\\
openssl_verify_mode: password: 'mypassword'\\
" input-file
But it would probably be much cleaner to add the new text to a file and use the read command:
$ cat > new-text << EOF
openssl_verify_mode: domain: 'test.net'
openssl_verify_mode: user_name: 'admin'
openssl_verify_mode: password: 'mypassword'
EOF
$ sed -e "/openssl_verify_mode: 'none'/rnew-text" input
If you want to parameterize the string "openssl_verify_mode" (ie, if you want to use a different string), do it at the shell level. eg:
$ s="openssl_verify_mode"
$ sed -e "/${s}: 'none'/a\\
$s: domain: 'test.net'\\
$s: user_name: 'admin'\\
$s: password: 'mypassword'\\
" input
I strongly recommend against -i, and have omitted it from the answer. Feel free to abuse it if you like.
Related
I have below content in my file abcd.yml with below indentation:
buildConfig:
env:
credentials: # want to remove this from file
- id: TEST_ID # want to remove this from file
user: username # want to remove this from file
password: password # want to remove this from file
scan:
credentials:
- id: scan_id
user: username
password: password
Tried the below :
sed -i '/credentials:/d' abcd.yml
sed -i '/- id: TEST_ID/d' abcd.yml
sed -i '/user: username/d' abcd.yml
sed -i '/password: password/d' abcd.yml
But it is removing all the occurrence of the above strings from file which I don't want.
Expected Output:
buildConfig:
env:
scan:
credentials:
- id: scan_id
user: username
password: password
I need to do this for 1000 files .Hence a script is required to do it .The file is in .yml format.
Use a yaml parser like yq for this:
$ yq eval 'del(.buildConfig.env.credentials)' abcd.yml > newfile.yml
$ cat newfile.yml
buildConfig:
env: {}
scan:
credentials:
- id: scan_id
user: username
password: password
tested with version v4.30.8
I am finding a bash command for a conditional replacement with offset. The existing posts that I've found are conditional replacement without offset or with a fixed offset.
Task: If uid contains 8964, then insert the line FORBIDDEN before DOB.
Each TXT file below represents one user, and it contains (in the following order)
some property(ies)
unique uid
some quality(ies)
unique DOB
a random lorem ipsum
I hope I can transform the following files
# file1.txt (uid doens't match 8964)
admin: false
uid: 123456
happy
movie
DOB: 6543-02-10
lorem ipsum
seo varis lireccuni paccem noba sako
# file2.txt (uid matches 8964)
citizen: true
hasSEAcct: true
uid: 289641
joyful hearty
final debug Juno XYus
magazine
DOB: 1234-05-06
saadi torem lopez dupont
into
# file1.txt (uid doens't match 8964)
admin: false
uid: 123456
happy
movie
DOB: 6543-02-10
lorem ipsum
seo varis lireccuni paccem noba sako
# file2.txt (uid matches 8964)
citizen: true
hasSEAcct: true
uid: 289641
joyful hearty
final debug Juno XYus
magazine
FORBIDDEN
DOB: 1234-05-06
saadi torem lopez dupont
My try:
If uid contains 8964, then do a 2nd match with DOB, and insert FORBIDDEN above DOB.
sed '/^uid: [0-9]*8964[0-9]*$/{n;/^DOB: .*$/{iFORBIDDEN}}' file*.txt
This gives me an unmatched { error.
sed: -e expression #1, char 0: unmatched `{'
I know that sed '/PAT/{n;p}' will execute {n;p} if PAT is matched, but it seems impossible to put /PAT2/{iTEXT} inside /PAT/{ }.
How can I perform such FORBIDDEN insertion?
$ awk '
/^uid/ && /8964/ {f=1} #1
/^DOB/ && f {print "FORBIDDEN"; f=0} #2
1 #3
' file
If a line starting with "uid" matches "8964", set flag
If a line starts with "DOB" and flag is set, print string and unset flag
print every line
$ awk -v RS='' '/uid: [0-9]*8964/{sub(/DOB/, "FORBIDDEN\nDOB")} 1' file
Alternatively, treat every block separated by a blank line as a single record, then sub in "FORBIDDEN\nDOB" if there's a match. I think the first one's better practice. As a very general rule, once you start thinking in terms of fields/records, it's time for awk/perl.
In my opinion, this is a good use-case for sed.
Here is a GNU sed solution with some explanation:
# script.sed
/^uid:.*8964/,/DOB/ { # Search only inside this range, if it exists.
/DOB/i FORBIDDEN # Insert FORBIDDEN before the line matching /DOB/.
}
Testing:
▶ gsed -f script.sed FILE2
citizen: true
hasSEAcct: true
uid: 289641
joyful hearty
final debug Juno XYus
magazine
FORBIDDEN
DOB: 1234-05-06
saadi torem lopez dupont
▶ gsed -f script.sed FILE1
admin: false
uid: 123456
happy
movie
DOB: 6543-02-10
lorem ipsum
seo varis lireccuni paccem noba sako
Or on one line:
▶ gsed -e '/^uid:.*8964/,/DOB/{/DOB/i FORBIDDEN' -e '}' FILE*
tried on gnu sed
sed -Ee '/^uid:\s*\w*8964\w*$/{n;/^DOB:/iFORBIDDEN' -e '}' file*.txt
I have file like test.yaml file, the text content in the file like below.
servers:
- uri: "http://demo.nginx1.com/status"
name: "NGinX Monitor1"
- uri: "http://demo.nginx2.com/status"
name: "NGinX Monitor2"
I want to remove - uri line and immediate next line (start with name:) where host name = demo.nginx1.com.
I want out put like below.
servers:
- uri: "http://demo.nginx2.com/status"
name: "NGinX Monitor2"
I tied like below..
cat test.yaml | grep -v demo.nginx1.com | grep -v Monitor1 >> test_back.yaml
mv test_back.yaml test.yaml
I am getting expected out put. But it's re creating the file and I don't want to re create the file
Please help me with suitable command that i can use..
Just a simple logic using GNU sed
sed '/demo.nginx1.com/,+1 d' test.yaml
servers:
- uri: "http://demo.nginx2.com/status"
name: "NGinX Monitor2"
For in-place replacement, add a -i flag as -i.bak
sed -i.bak '/demo.nginx1.com/,+1 d' test.yaml
To see the in-place replacement:-
cat test.yaml
servers:
- uri: "http://demo.nginx2.com/status"
name: "NGinX Monitor2"
As I dislike using regular expressions to hack something you can parse - here's how I'd tackle it, using perl and the YAML module:
#!/usr/bin/env perl
use strict;
use warnings;
use YAML;
use Data::Dumper;
#load yaml by reading files specified as args to stdin,
#or piped in. (Just like how you'd use 'sed')
my $config = Load ( do { local $/ ; <>} );
#output data structure for debug
print Dumper $config;
#use grep to filter the uri you don't want.
#{$config -> {servers}} = grep { not $_ -> {uri} =~ m/demo.nginx2/ } #{$config -> {servers}};
#resultant data structure
print Dumper $config;
#output YAML to STDOUT
print Dump $config;
I want to break down a file that has multiple lines that follow this style:
e-mail;year/month/date;groups;sharedFolder
An example line from file:
alan.turing#cam.ac.uk;1912/06/23;visitor;/visitorData
Essentially I want to break each line up into four arrays that can be accessed later on in a loop to create a new user for each line.
I have declared the arrays already have a file saved as variable 'filename'
Usernames need to be the first three letters of the surname and the first three letters of the first name.
Passwords need to be the users birthdate as day/month/year.
So far this is what I have. Am I on the right track? Are there places I have gone wrong or could improve on?
#reads file and saves into appropriate arrays
while read -r line
do
IFS = $';' read -r -a array <<< "$line"
mailArray += "$(array[0])"
dateArray += "$(array[1])"
groupArray += "$(array[2])"
folderArray += "$(array[3])"
done < $filename
#create usernames from emails
for i in "$(mailArray[#])"
do
IFS=$'.' read -r -a array <<< "$i"
part1 = ${array[0]:0:3}
part2 = ${array[1]:0:3}
user = $part2
user .= $part1
userArray += ("$user")
done
#create passwords from birthdates
for i in "$(dateArray[#])"
do
IFS=$'/' read -r -a array <<< "$i"
password = $part3
password .= $part2
password .= $part1
passArray += ("$password")
done
Not sure if arrays are required here, and if you just want to create username, password from the lines in the desired format, please see below:
# Sample Input Data:
bash$> cat d
alan.turing#cam.ac.uk;1912/06/23;visitor;/visitorData
rob.zombie#cam.ac.uk;1966/06/23;metalhead;/metaldata
donald.trump#stupid.com;1900/00/00;idiot;/idiotique
bash$>
# Sample Output from script:
bash$> ./dank.sh "d"
After processing the line [alan.turing#cam.ac.uk;1912/06/23;visitor;/visitorData], we have the following parameters extracted:
Name: alan
Surname: turing
Birthdate: 1912/06/23
username: alatur
Password: 1912/06/23
After processing the line [rob.zombie#cam.ac.uk;1966/06/23;metalhead;/metaldata], we have the following parameters extracted:
Name: rob
Surname: zombie
Birthdate: 1966/06/23
username: robzom
Password: 1966/06/23
After processing the line [donald.trump#stupid.com;1900/00/00;idiot;/idiotique], we have the following parameters extracted:
Name: donald
Surname: trump
Birthdate: 1900/00/00
username: dontru
Password: 1900/00/00
Now the script, which does this operation.
# Script.
bash$> cat dank.sh
#!/bin/bash
cat "$1" | while read line
do
name=`echo $line | sed 's/^\(.*\)\..*\#.*$/\1/g'`
surname=`echo $line | sed 's/^.*\.\(.*\)\#.*$/\1/g'`
bdate=`echo $line | sed 's/^.*;\(.*\);.*;.*$/\1/g'`
temp1=`echo $name | sed 's/^\(...\).*$/\1/g'`
temp2=`echo $surname | sed 's/^\(...\).*$/\1/g'`
uname="${temp1}${temp2}"
echo "
After processing the line [$line], we have the following parameters extracted:
Name: $name
Surname: $surname
Birthdate: $bdate
username: $uname
Password: $bdate
"
done
bash$>
Basically, i am just running couple of sed commands to extract what is useful and required, and storing them in variables and then one could use them anyways they want to. You could redirect them to a file if you want or print out a pipe separated output.. upto you.
Let me know..
I have the following lines:
Name Server:NS92.WORLDNIC.COM(or some other value)
Name Server:
Name Server:
Name Server:
Please see the screenshot for better understanding: http://imgur.com/q6Ir4lo
How do I get rid of the 'Name Server:' line but keep the line with the value?
I tried /Name Server:{0,0}/d but it deletes all lines.
Thanks
I was able to get the following two lines to work:
I believe the [:space:] is POSIX compliant:
cat test |sed '/^Name Server:[[:space:] \t]\?$/d'
An alternative is simply:
cat test |sed '/^Name Server:[ \t]\?$/d'
I've also found in sed, that most of the meta-characters (eg + ? ) need to be escaped for sed to recognize them correctly.
This works for me:
echo "Name Server:NS92.WORLDNIC.COM" | sed 's/^Name Server://'
cut -d ":" -f 2 < ff | sed '/^$/d'
Uses ':' as delimiter and splits the line (-d option), then selects the second field (-f option)