Trying to replace a line with a multi-line file. I can easily do this with a single-line file or multi-line string (see below).
#!/bin/bash
NEW_STRING="apple\nbanana\ncarrot"
sed -i "3s/.*/$(echo "${NEW_STRING}")/" tmp.txt
# Outputs...
# line 1
# line 2
# apple
# banana
# carrot
# line 4
# line 5
# ...
# etc
However, when I change the code to use a multi-file, such as replace.txt, I receive the following error:
sed: -e expression #1, char 10: unterminated `s' command
broken_script.bash
#!/bin/bash
FILE=`cat replace.txt`
sed -i "3s/.*/$(echo "${FILE}")/" tmp.txt
replace.txt
++++
++++++++++++++++++++++++++++++++++++++++++++++
apple size=8, align=2, ..., etc.
banana size=64, align=8, ..., etc.
...
carrot size=92, align=4, ..., etc.
Note broken_script.bash works if I delete replace.txt to be a single-line (i.e. just ++++).
Does anyone see what I'm doing wrong? Why doesn't this work with a multi-line file as the replacement text (i.e. like the single-line file or multi-line string)?
To replace lines with the contents of a file, you can use the r command:
sed -e 3rreplace.txt -e 3d tmp.txt
As asked in a comment, sed -e 3d -e 3rreplace.txt does not work because the d command immediately returns to the top of the program after reading the next line and never executes the r command.
First of all: Drop the echo part. You can use the variable directly.
Back to the actual problem:
The difference is how you encode the newline. In the first command you wrote \n do denote a newline. That \n is not interpreted by 'bash', but directly sent to sed. In the second command, the file content is sent to sed, including literal newline characters. For a file with the two lines line1 and line2 the command sed sees is
3s/.*/line1
line2/
sed cannot handle such multi-lined commands.
Non-sed solution:
As it seems, you just want to replace the third line of one file with the content of another file. This can be done with
cat <(head -n 2 file1) file2 <(tail -n +4 file1)
head -n2 file1 prints the first two lines of file1.
tail -n +4 file 1 prints all lines of file1 starting at line 4.
<(command) is called process substitution and emulates a file containing the output of command.
cat concatenates the three "files".
sed can do this with some quote juggling
$ seq 5 | sed -e '/3/{r replace.txt' -e 'd}'
1
2
++++
++++++++++++++++++++++++++++++++++++++++++++++
apple size=8, align=2, ..., etc.
banana size=64, align=8, ..., etc.
...
carrot size=92, align=4, ..., etc.
4
5
for in place replacement you need to provide file
$ sed -i -e '/3/{r replace.txt' -e 'd}' file
of course test first or take backup.
Related
how to remove comment lines (as # bal bla ) and empty lines (lines without charecters) from file with one sed command?
THX
lidia
If you're worried about starting two sed processes in a pipeline for performance reasons, you probably shouldn't be, it's still very efficient. But based on your comment that you want to do in-place editing, you can still do that with distinct commands (sed commands rather than invocations of sed itself).
You can either use multiple -e arguments or separate commands with a semicolon, something like (just one of these, not both):
sed -i 's/#.*$//' -e '/^$/d' fileName
sed -i 's/#.*$//;/^$/d' fileName
The following transcript shows this in action:
pax> printf 'Line # with a comment\n\n# Line with only a comment\n' >file
pax> cat file
Line # with a comment
# Line with only a comment
pax> cp file filex ; sed -i 's/#.*$//;/^$/d' filex ; cat filex
Line
pax> cp file filex ; sed -i -e 's/#.*$//' -e '/^$/d' filex ; cat filex
Line
Note how the file is modified in-place even with two -e options. You can see that both commands are executed on each line. The line with a comment first has the comment removed then all is removed because it's empty.
In addition, the original empty line is also removed.
#paxdiablo has a good answer but it can be improved.
(1) The '/^$/d' clause only matches 100% blank lines.
If you want to also match lines that are entirely whitespace (spaces, tabs etc.) use this instead:
'/^\s*$/d'
(2) The 's/#.*$//' clause only matches lines that start with the # character in column 0.
If you want to also match lines that have only whitespace before the first # use this instead:
'/^\s*#.*$/d'
The above criteria may not be universal (e.g. within a HEREDOC block, or in a Python multi-line string the different approaches could be significant), but in many cases the conventional definition of "blank" lines include whitespace-only, and "comment" lines include whitespace-then-#.
(3) Lastly, on OSX at least, the #paxdiablo solution in which the first clause turns comment lines into blank lines, and the second clause strips blank lines (including what were originally comments) doesn't work. It seems to be more portable to make both clauses /d delete actions as I've done.
The revised command incorporating the above is:
sed -e '/^\s*#.*$/d' -e '/^\s*$/d' inputFile
This tiny jewel removes all # comments, no matter where they begin in a line (see caution below):
sed -e 's/\s*#.*$//'
Example:
text="
this is a # test
#this is a test
#this is a #test
this is # another #test
"
$echo "$text" | sed -e 's/\s*#.*$//'
this is a
this is
Next this removes any resulting blank lines:
$echo "$text" | sed -e 's/\s*#.*$//' | sed -e '/^\s*$/d'
Caution: Depending on the syntax and/or interpretation of the lines your processing, this might not be an appropriate solution, as it just stupidly removes end of lines, even if the '#' is part of your data or code. However, for use cases where you'll never use a hash except for as an end of line comment then it works fine. So just as with all coding, context must be taken into consideration.
Alternative variant, using grep:
cat file.txt | grep -Ev '(#.*$)|(^$)'
you can use awk
awk 'NF{gsub(/^[ \t]*#/,"");print}' file
First example(paxdiablo) is very good except its not change file, just output result. If you want to change it inline:
sudo sed -i 's/#.*$//;/^$/d' inputFile
On (one of) my linux boxes, sed understands extended regular expressions with the -r option, so:
sed -r '/(^\s*#)|(^\s*$)/d' squid.conf.installed
is very useful for showing all non-blank, non comment lines.
The regex matches either start of line followed by zero or more spaces or tabs followed by either a hash or end of line, and deletes those matching lines from the input.
I want to replace a line in a file with multiple lines. I know I can use \n in the sed replace, but that is rather ugly. I was hoping to HEARDOCs.
So I can do this to replace the line with multiple lines:
$ cat sedtest
DINGO=bingo
$ sed -i -e "s/^DINGO.*$/# added by $(whoami) on $(date)\nDINGO=howdy/" sedtest
$ cat sedtest
# added by user on Sun Feb 3 08:55:44 EST 2019
DINGO=howdy
In the command I want to put the replacement in new lines so it's easier to read/understand. So far I have been using HEREDOCs when I want to add new lines to a file:
CAT << EOF | sudo tee -a file1 file2 file3
line one
line two
line three
EOF
And this has worked well for appending/adding. Is it possible to do something similar but instead use the output as the replacement in sed or is there some other way to do what I'm looking for?
Is this what you're trying to do?
$ awk 'NR==FNR{new=(NR>1?new ORS:"") $0;next} $0=="DINGO=bingo"{$0=new} 1' - file <<!
# added by $(whoami) on $(date)
DINGO=howdy
!
# added by user on Sun, Feb 3, 2019 8:50:41 AM
DINGO=howdy
Note that the above is using literal string operations so it'll work for any characters in the old or new strings unlike your sed script which would fail given /s or any ERE regexp character or capture groups or backreferences or... in the input (see Is it possible to escape regex metacharacters reliably with sed for details).
This might work for you (GNU sed):
cat <<! | sed -i -e '/^DINGO/r /dev/stdin' -e '//d' file
# added by $(whoami) on $(date)
DINGO=howdy
!
This replaces lines starting DINGO with the here-document which is piped to stdin as file within the sed command.
An alternative:
cat <<! | sed -i -e '/^DINGO/e cat /dev/stdin' -e 's/bingo/howdy' file
# added by $(whoami) on $(date)
!
N.B. In the alternative solution the here-doc will only be read once!
This question already has answers here:
Sed Insert Multiple Lines
(3 answers)
Closed 1 year ago.
I have a multiline variable that I captured from STDOUT.
I want to insert an echo command using this multiline variable to line 15 in another script (target).
#!/bin/bash
TEST=`cat foo`
echo "$TEST"
sed -i "15i echo \"$TEST\" > someotherfile" target
Contents of foo :
apples
oranges
bananas
carrots
I thought the sed command read in line feeds, which I confirmed my foo has:
user#test$ cat foo | tr -cd '\n' | wc -c
4
When I run my test.sh script, I see what's in $TEST, but am getting an error for the sed command:
user#test$ ./test.sh
apples
oranges
bananas
carrots
sed: -e expression #1, char 18: unknown command: `o'
What am I doing wrong?
Thanks in advance.
GNU sed is assumed, as implied by the syntax used in the question.
#!/bin/bash
# Read contents of file 'foo' into shell variable $test.
test=$(<foo)
# \-escape the newlines in $test for use in Sed.
testEscapedForSed=${test//$'\n'/\\$'\n'}
sed -i "15i echo \"$testEscapedForSed\" > someotherfile" target
Your problem was that passing multi-line strings to sed functions such as i (insert) requires the newlines embedded in those strings to be \-escaped, so that sed knows where the string ends and additional commands, if any, start.
A (nonstandard) parameter expansion is used to replace all newlines in $test with themselves prefixed by \, using ANSI C-quoted string $'\n' to generate actual newline chars.
Also note:
I've renamed TEST to test, because all-uppercase shell-variable names should be avoided.
I've used modern command-substitution syntax $(..) in lieu of legacy syntax `...`.
$(<foo) is a slightly more efficient - although nonstandard - way of reading the content of a file at once.
Try:
Solution1:
awk 'NR==15{print;system("cat foo");next} 1' Input_file
No need to get the complete file into a variable, we could simply print it whichever line of Input_file you want to print it.
Solution2:
line=15; sed -e "${line}r foo" target
Or (in script mode)
cat script.ksh
line=15;
sed -e "${line}r foo" target
Where you could change the number of line where you want to insert the lines from another file.
The i command in sed inserts the lines of text that end with a newline, up until a line that doesn't end with a backslash. The a and c commands are similar. Classic sed doesn't like the first line to appear on the same line as the i command; GNU sed isn't as fussy.
If you were writing the command manually, you'd need to write:
15i\
echo "apples\
oranges\
bananas\
carrots" > someotherfile
At issue now is "how do you want to create this given the file foo contains the list of names?". Sometimes, using sed to generate the sed script is useful. However, it can also be intricate if you need to get backslashes at the ends of lines which are subject to an i (or a or c) command, and it is simpler to circumvent the problem.
{
echo "15i\\"
sed -e '1s/^/echo "/' -e 's/$/\\/' -e '$s/\\$/" > someotherfile/' foo
} | sed -f /dev/stdin target
GNU sed can read its script from standard input using -f -; BSD (macOS) sed doesn't like that, but you can use -f /dev/stdin instead (which also works with GNU sed), at least on systems where there is a /dev/stdin.
Interesting issue.
As already mentioned the whole story for sed to be able to insert multiline text in another file is that this new multiline text must have actually literral \n for line breaks.
So we can use sed to convert real new line chars to literal \n:
$ a=$(tr '\n' '\\' <file3 |sed 's#[\]$##' |sed "s#[\]#\0n#g")
#Alternative: a=$(sed "s#[\]#\0n#g" <(sed 's#[\]$##' <(tr '\n' '\\' <file3)))
$ echo "$a"
apples\noranges\nbananas\ncarrots
How this translation works:
* First we replace all new lines with a single backslash using tr
* Then we remove the backslash from the end of the string
* Then we replace all other backslashes with backaslash and n char.
Since now variable $a contains literal \n between lines, sed will translate them back to actuall new lines:
$ cat file4
Line1
line2
line3
$ sed "2i $a" file4
Line1
apples
oranges
bananas
carrots
line2
line3
Result:
Mutliline replacement can be done with two commands:
$ a=$(tr '\n' '\\' <file3 |sed 's#[\]$##' |sed "s#[\]#\0n#g")
$ sed "2i $a" file4
sed 2i means insert a text before line2. 2a can be used in order to insert something after line2.
Remark:
According to this post which seems to be a duplicate, translation of new lines to literal \n seems that can be done with just :
a=$(echo ${a} | tr '\n' "\\n")
But this method never worked in my system.
Remark2:
The sed operation sed "2i $a" = insert variable $a before line 2 , can be also expressed as sed "1 s/.*/\0\n$a/" = replace all chars of first line with the same chars \0 plus a new line \n plus the contents of variable $a => insert $a after line1 = insert $a before line2.
I'm try to print with bash two lines of a text file in the same line and with only one command.
for example, if you have the next file and you want the lines 1 and 3
Cat
Bye
Bash
Dog
Hello
then you need a command that returns the following
$ cmd
Cat Bash
Is that possible? I try with
$ sed -n '1,3{p;n;}' [FILENAME]
but it prints the two lines of text in two different lines, like:
Cat
Bash
thanks
$ sed -n '1 h; 3{x;G;s/\n/ /;p}' fname
Cat Bash
Explanation
-n
This tells sed not to print anything unless we explicitly ask it to.
1 h
When we reach line 1, this tells sed to save it in the 'hold' space.
3 {x;G;s/\n/ /;p;}
When we reach line 3, we do the following:
x exchanges the pattern space and hold space. When this is done, the pattern space has line 1 and the hold space has line 5.
G appends the hold space to the pattern space. When this is done, the pattern space has lines 1 and 3.
s/\n/ / replaces the newline character that separates line 1 and line 3 with a space. When this is done, the data from lines 1 and 3 are on the same line.
p tells sed to print the result.
On Mac OSX, try:
sed -n -e '1 h' -e '3{x;G;s/\n/ /;p;}' fname
This might work for you (GNU sed):
sed '1H;3H;$!d;x;s/\n//;y/\n/ /' file
In general append required lines to the hold space (HS) and delete all lines except the last. On the last line swap to the HS, delete the first newline and translate all other newlines to spaces.
N.B. The lines to be collected can be stated in any order as long as they come before the address for the last line.
If the order of the collection is known the method can be shortened to:
sed '1h;2H;$!d;x;y/\n/ /' file
This should work:
echo `sed -n '3,5{p;n;}' [FILENAME]`
Just an extra 'echo' with command substitution `` to join the lines.
I need to replace the whole line with sed if it matches a pattern.
For example if the line is 'one two six three four' and if 'six' is there, then the whole line should be replaced with 'fault'.
You can do it with either of these:
sed 's/.*six.*/fault/' file # check all lines
sed '/six/s/.*/fault/' file # matched lines -> then remove
It gets the full line containing six and replaces it with fault.
Example:
$ cat file
six
asdf
one two six
one isix
boo
$ sed 's/.*six.*/fault/' file
fault
asdf
fault
fault
boo
It is based on this solution to Replace whole line containing a string using Sed
More generally, you can use an expression sed '/match/s/.*/replacement/' file. This will perform the sed 's/match/replacement/' expression in those lines containing match. In your case this would be:
sed '/six/s/.*/fault/' file
What if we have 'one two six eight eleven three four' and we want to
include 'eight' and 'eleven' as our "bad" words?
In this case we can use the -e for multiple conditions:
sed -e 's/.*six.*/fault/' -e 's/.*eight.*/fault/' file
and so on.
Or also:
sed '/eight/s/.*/XXXXX/; /eleven/s/.*/XXXX/' file
Above answers worked fine for me, just mentioning an alternate way
Match single pattern and replace with a new one:
sed -i '/six/c fault' file
Match multiple pattern and replace with a new one(concatenating commands):
sed -i -e '/one/c fault' -e '/six/c fault' file
To replace whole line containing a specified string with the content of that line
Text file:
Row: 0 last_time_contacted=0, display_name=Mozart, _id=100, phonebook_bucket_alt=2
Row: 1 last_time_contacted=0, display_name=Bach, _id=101, phonebook_bucket_alt=2
Single string:
$ sed 's/.* display_name=\([[:alpha:]]\+\).*/\1/'
output:
100
101
Multiple strings delimited by white-space:
$ sed 's/.* display_name=\([[:alpha:]]\+\).* _id=\([[:digit:]]\+\).*/\1 \2/'
output:
Mozart 100
Bach 101
Adjust regex to meet your needs
[:alpha] and [:digit:]
are Character Classes and Bracket Expressions
This might work for you (GNU sed):
sed -e '/six/{c\fault' -e ';d}' file
or:
sed '/six/{c\fault'$'\n'';d}' file