How can I use a cat and sed to read data from a file and insert it into another file under known line?
For example I have a file named script1.txt that contains a few hundred lines, one of the line has the value "COMMANDS="commands"
If I wanted use sed to insert a line under it, simply I can use sed as the command bellow.
sed -i '/^COMMANDS=.*/a NEW LINE HERE' script1.txt
But if I want to insert a multi lines and these lines inside a file, and these line changes every a few hours.. how can i do that ?
I tried:
DATA=$(cat data.txt)
sed -i '/^COMMANDS=.*/a '$DATA'' script1.txt
I got the error bellow.
sed: -e expression #1, char 1: unknown command: `"'
Is there a way other than sed to insert the data from file under known line with no issues?
This might work for you (GNU sed):
sed -i '/^COMMANDS=/r dataFile' file
This will append the contents of the file dataFile after the line beginning COMMANDS= and update file
If the data you want to append is multi-line, you might want to replace newlines with \n.
#!/bin/sh
DATA="$(awk '{gsub(/[]\/$*.^&[]/, "\\\\&");printf (FNR>1)?"\\n%s":"%s",$0}END{print ""}' data.txt)"
sed -i -e '/^COMMANDS=.*/a\' -e "$DATA" script1.txt
Here the awk command escapes sed special characters (for basic regular expressions), then prints "%s" for the first line, and "\\n%s" for the others. A newline is printed at the end, but it's somewhat pointless as $() strips it anyway.
The sed command is almost the same but multiple expressions are used which is equivalent to a multi-line sed script (The a text sed alternative syntax can act weirdly with leading spaces/backslashes).
Related
I am trying to comment a line in a file using search pattern, and then insert new line next to it.
search_variable=Dlog4j2.formatMsgNoLookups
new_variable="wrapper.java.additional.47=-Dlog4j2.formatMsg=false"
cat testfile.txt
wrapper.java.additional.47=-Dlog4j2.formatMsgNoLookups=true
This one works, but trying to use variable to comment the line and
sed -i '/Dlog4j2.formatMsgNoLookups/s/^/#/g' testfile.txt
output:
#wrapper.java.additional.47=-Dlog4j2.formatMsgNoLookups=true
Desired output
cat testfile.txt
#wrapper.java.additional.47=-Dlog4j2.formatMsgNoLookups=true
wrapper.java.additional.47=-Dlog4j2.formatMsg=false
With GNU sed:
search_variable="Dlog4j2.formatMsgNoLookups"
new_variable="wrapper.java.additional.47=-Dlog4j2.formatMsg=false"
sed -i "s/.*${search_variable}.*/#&\n${new_variable}/" testfile.txt
Output to testfile.txt:
#wrapper.java.additional.47=-Dlog4j2.formatMsgNoLookups=true
wrapper.java.additional.47=-Dlog4j2.formatMsg=false
For the meaning of & see using sed with ampersand (&).
The curly brackets can also be omitted in this case.
This can also be helpful: Difference between single and double quotes in bash
how to remove comment lines (as # bal bla ) and empty lines (lines without charecters) from file with one sed command?
THX
lidia
If you're worried about starting two sed processes in a pipeline for performance reasons, you probably shouldn't be, it's still very efficient. But based on your comment that you want to do in-place editing, you can still do that with distinct commands (sed commands rather than invocations of sed itself).
You can either use multiple -e arguments or separate commands with a semicolon, something like (just one of these, not both):
sed -i 's/#.*$//' -e '/^$/d' fileName
sed -i 's/#.*$//;/^$/d' fileName
The following transcript shows this in action:
pax> printf 'Line # with a comment\n\n# Line with only a comment\n' >file
pax> cat file
Line # with a comment
# Line with only a comment
pax> cp file filex ; sed -i 's/#.*$//;/^$/d' filex ; cat filex
Line
pax> cp file filex ; sed -i -e 's/#.*$//' -e '/^$/d' filex ; cat filex
Line
Note how the file is modified in-place even with two -e options. You can see that both commands are executed on each line. The line with a comment first has the comment removed then all is removed because it's empty.
In addition, the original empty line is also removed.
#paxdiablo has a good answer but it can be improved.
(1) The '/^$/d' clause only matches 100% blank lines.
If you want to also match lines that are entirely whitespace (spaces, tabs etc.) use this instead:
'/^\s*$/d'
(2) The 's/#.*$//' clause only matches lines that start with the # character in column 0.
If you want to also match lines that have only whitespace before the first # use this instead:
'/^\s*#.*$/d'
The above criteria may not be universal (e.g. within a HEREDOC block, or in a Python multi-line string the different approaches could be significant), but in many cases the conventional definition of "blank" lines include whitespace-only, and "comment" lines include whitespace-then-#.
(3) Lastly, on OSX at least, the #paxdiablo solution in which the first clause turns comment lines into blank lines, and the second clause strips blank lines (including what were originally comments) doesn't work. It seems to be more portable to make both clauses /d delete actions as I've done.
The revised command incorporating the above is:
sed -e '/^\s*#.*$/d' -e '/^\s*$/d' inputFile
This tiny jewel removes all # comments, no matter where they begin in a line (see caution below):
sed -e 's/\s*#.*$//'
Example:
text="
this is a # test
#this is a test
#this is a #test
this is # another #test
"
$echo "$text" | sed -e 's/\s*#.*$//'
this is a
this is
Next this removes any resulting blank lines:
$echo "$text" | sed -e 's/\s*#.*$//' | sed -e '/^\s*$/d'
Caution: Depending on the syntax and/or interpretation of the lines your processing, this might not be an appropriate solution, as it just stupidly removes end of lines, even if the '#' is part of your data or code. However, for use cases where you'll never use a hash except for as an end of line comment then it works fine. So just as with all coding, context must be taken into consideration.
Alternative variant, using grep:
cat file.txt | grep -Ev '(#.*$)|(^$)'
you can use awk
awk 'NF{gsub(/^[ \t]*#/,"");print}' file
First example(paxdiablo) is very good except its not change file, just output result. If you want to change it inline:
sudo sed -i 's/#.*$//;/^$/d' inputFile
On (one of) my linux boxes, sed understands extended regular expressions with the -r option, so:
sed -r '/(^\s*#)|(^\s*$)/d' squid.conf.installed
is very useful for showing all non-blank, non comment lines.
The regex matches either start of line followed by zero or more spaces or tabs followed by either a hash or end of line, and deletes those matching lines from the input.
In zabbix-agent.conf I have lines:
# Example: Server=127.0.0.1,192.168.1.0/24,::1,2001:db8::/32,zabbix.example.com
Server=127.0.0.1
I want to replace line
Server=127.0.0.1
with my
Server=zabbix.mydomain.com
But if I do
sed -i -e 's/Server=127.0.0.1/Server=zabbix.mydomain.com/g' /etc/zabbix/zabbix_agentd.conf
it found also line with commented example and replace string in it. I get:
#<----->Example: Server=zabbix.mydomain.com,192.168.1.0/24,::1,2001:db8::/32,zabbix.example.com
Server=zabbix.mydomain.com
How to replace only one line?
You need to
Match the text at the start of string
Escape the dots
Remove g flag since the match will only be found at the string start.
Also, you do not need the -e option, you can use
sed -i 's/^Server=127\.0\.0\.1/Server=zabbix.mydomain.com/' /etc/zabbix/zabbix_agentd.conf
See the online demo:
#!/bin/bash
s='# Example: Server=127.0.0.1,192.168.1.0/24,::1,2001:db8::/32,zabbix.example.com
Server=127.0.0.1'
sed 's/^Server=127\.0\.0\.1/Server=zabbix.mydomain.com/g' <<< "$s"
Output:
Example: Server=127.0.0.1,192.168.1.0/24,::1,2001:db8::/32,zabbix.example.com
Server=zabbix.mydomain.com
This might work for you (GNU sed):
sed -i '/^Server=127\.0\.0\.1/cServer=zabbix.mydomain.com' file
Change line beginning Server=127.0.0.1 to Server=zabbix.mydomain.com.
The other answers are almost fine but they would also replace a line like:
Server=127.0.0.10
A complete solution with any sed could be:
sed -i 's/^Server=127\.0\.0\.1$/Server=zabbix.mydomain.com/' /etc/zabbix/zabbix_agentd.conf
^ and $ anchor the string to the beginning and end of line, respectively. Dots need backslash escape, else they stand for any character.
I have a file, which has multiple lines.
For example:
a
ab#
ad.
a12fs
b
c
...
I want to use sed or awk delete the line, if the line include symbols or numbers. (For example, I want to delete: ab#, ad., a12fs.... lines)
or in another words, I just want to keep the line which include [a-z][A-Z] .
I know how to delete number line,
sed '/[0-9]/d' file.txt
but I do not know how to delete symbols lines.
Or there has any easy way to do that?
To keep blank lines:
grep '^[[:alpha:]]*$' file
sed '/[^[:alpha:]]/d' file
awk '/^[[:alpha:]]*$/' file
To remove blank lines:
grep '^[[:alpha:]]+$' file
sed -E -n '/^[[:alpha:]]+$/p' file
awk '/^[[:alpha:]]+$/' file
grep works well too and is even simpler: just do the reverse: keep the lines that interest you, which are way easier to define
grep -i '^[a-z]*$' file.txt
(match lines containing only letters and empty lines, and -i option makes grep case-insensitive)
to remove empty lines as well:
grep -i '^[a-z]+$' file.txt
caution when using Windows text files, as there's a carriage return at the end of the line, so nothing would match depending on grep versions (tested on windows here and it works)
but just in case:
grep -iP '^[a-z]*\r?$'
(note the P option to enable perl expressions or \r is not recognized)
You can use this sed:
sed '/^[A-Za-z0-9]\+$/!d' file
(OR)
sed '/[^A-Za-z0-9]/d' file
$ awk '!/[^[:alpha:]]/' file.txt
a
b
c
Suppose my file a.conf is as following
Include /1
Include /2
Include /3
I want to replace "Include /2" with a new line, I write the code in .sh file :
line="Include /2"
rep=""
sed -e "s/${line}/${rep}/g" /root/new_scripts/a.conf
But after running the sh file, It give me the following error
sed: -e expression #1, char 14: unknown option to `s'
If you are using a newer version of sed you can use -i to read from and write to the same file. Using -i you can specify a file extension so a backup will be made, incase something went wrong. Also you don't need to use the -e flag unless you are using multiple commands
sed -i.bak "s/${line}/${rep}/g" /root/new_scripts/a.conf
I have just noticed that as the variables you are using are quoted strings you may want to use single quotes around your sed expression. Also your string contains a forward slash, to avoid any errors you can use a different delimiter in your sed command (the delimiter doesn't need to be a slash):
sed -i.bak 's|${line}|${rep}|g' /root/new_scripts/a.conf
You have to write the changes to a new file and then, move the new file over the old one. Like this:
line="Include 2"
rep=""
sed -e "s/${line}/${rep}/g" /root/new_scripts/a.conf > /root/new_scripts/a.conf-new
mv /root/new_scripts/a.conf-new /root/new_scripts/a.conf
The redirection (> /root/new_scripts/a.conf) wipes the contents of the file before sed can see it.
You need to pass the -i option to sed to edit the file in-place:
sed -i "s/${line}/${rep}/g" /root/new_scripts/a.conf
You can also ask sed to create a backup of the original file:
sed -i.bak "s/${line}/${rep}/g" /root/new_scripts/a.conf
So, if you have to replace a substring in a file, you can use sed command like this, say we have a file as file.txt, so replacing a substring in it can be done like this
searchString="abc";
replaceString="def";
sed -i '' "s|$searchString|$replaceString|g" file.txt
This will all the occurrences of "abc" with "def" in file.txt. Also, this keeps a check for any / character present in the variables used, and with no backup file made.