I'm lost trying to do following substitution with sed:
# edit: to capture the full complexity of my problem,
I added the fact that filenames are contained in variables afterwards.
Solutions might therefore directly use the filenames.
given a variable I='insert.txt':
'insert.txt':
Text I wanna skip.
This is text to insert containing
spaces and new lines
given a variable M='toModify.txt':
'toModify.txt':
Insert the new text: here.
I would like to replace the 'here' from $M with the content
of $I:
Insert the new text: This is text to insert containing
spaces and new lines.
I tried:
sed -e "s/here/$(tail -n2 $I | sed -e 's/ /\\ /g' | tr '\n' '\\n')/" $M
with error:
sed unterminated `s' command
The problem is that I don't get the spaces and new lines without terminating the s command.
Any solution?
You can't replace one character with two with tr. Escaping the individual spaces is pointless anyway. The reason for the immediate error is that you end up escaping the final slash, too:
linux$ tail -n2 "$I" | sed -e 's/ /\\ /g' | tr '\n' '\\n'
This\ is\ text\ to\ insert\ containing\spaces\ and\ new\ lines\/
Escaping the spaces is pointless anyway. I guess you want something like this:
linux$ sed '1,2d;$!s/$/\\/' "$I"
This is text to insert containing\
spaces and new lines
We delete lines 1 and two; then add a backslash before every newline except the last.
linux$ sed -e "s/here/$(sed '1,2d;$!s/$/\\/' "$I")/" "$M"
Insert the new text: This is text to insert containing
spaces and new lines.
This is one detail of sed which isn't entirely portable. But the above works for me on Linux and MacOS. (Note you might need to set +H to disable csh-style history expansion aka -bash: !s/$/\\/': event not found errors).
You may use this awk:
awk 'BEGIN{prs=RS; RS=""; getline s < "insert.txt"; RS=prs}
{gsub(/here/, s)} 1' toModify.txt
Insert the new text: This is text to insert containing
spaces and new lines.
Using Perl one-liner
> cat insert.txt
This is text to insert containing
spaces and new lines
> cat toModify.txt
Insert the new text: here
> export I=insert.txt
> export M=toModify.txt
> perl -ne 'BEGIN{$x=qx(cat $ENV{M});$x=~s/here/qx(cat $ENV{I})/e; print $x;exit }'
Insert the new text: This is text to insert containing
spaces and new lines
>
Related
I have an text file, with line inside...
line: <version="AAA" email="...ANY..." file="BBB">
new desired line in text file to be: <version="AAA" email="NEW_TEXT" file="BBB">
I want to replace the ...ANY... expression with variable (replace entire line)
I have this script text-file script in #!/bin/bash, but I have problem when expanding double quotes in variables.
LINE_NUMBER="$(grep -nr 'email' *.txt | awk '{print $1}' | sed 's/[^0-9]*//g')"
VAR1="$(grep 'email' *.txt | cut -d '"' -f1-3)"
VAR2="$(grep 'email' *.txt | cut -d '"' -f5-)"
VAR3='NEW_TEXT'
NEW_LINE=$VAR1'"'$VAR3'"'$VAR2
new desired line in text file to be... <version="AAA" email="NEW_TEXT" file="BBB">
awk -i inplace 'NR=='"$LINE_NUMBER"'{sub(".*",'"'$NEW_LINE'"')},1' *.txt
but I get this new line:
<version="" email="NEW_TEXT" file="">
what do I do wrong?
How can I prevent expand duouble quotes inside variable?
please better write me an working example, I had tried other topics, forums, posts....but I have no luck.
You cas use sed :
VAR3='NEW_TEXT'
sed -i "s/email=\"[^\"]*\"/email=\"$VAR3\"/" myfile.xml
Suggesting:
var3="text space % special < chars"
Note var3 may not contain & which is special replacement meaning in sed
sed -E 's|email="[^"]*"|email="'"${var3}"'"|' input.1.txt
Explanation
[^"]* : Match longest string not having " till next ".
I have a sed expression in a function that I pass parameters to.
insert_after_new_line() {
if ! grep -q "${2}" "${3}"; then
sed -i "/${1}/{N;N;a ${2}}" "${3}"
fi
}
insert_after_new_line search insert textfile.txt
I am trying to have a blank line inserted below the search string and the insert string inserted after.
so
text
text
search
text
becomes
text
text
search
insert
text
but I keep getting the error
sed: -e expression #1, char 0: unmatched `{'
I tested this. works in command line
sed -i '/search/a \\ninsert' file
sed really delimiters commands by a newline. There is a ; but it does not work for all commands, mostly these which take file name as an argument. ; does not work for r R or for example a. Sed will read everything after a command, so sed interprets is like a ${2}} as a single command, in result it does not find enclosing }, cause it's been eaten by a command. You need a newline:
sed -i "/${1}/{N;N;a ${2}
}" "${3}"
or
sed -i "/${1}/{N;N;a ${2}"$'\n'"}" "${3}"
this should work:
sed -i '/search/{G;ainsert
}' file
You can replace the text by shell variable, but replace the single quotes by double quotes too.
I am trying to get a substring between &DEST= and the next & or a line break.
For example :
MYREQUESTISTO8764GETTHIS&DEST=SFO&ORIG=6546
In this I need to extract "SFO"
MYREQUESTISTO8764GETTHIS&DEST=SANFRANSISCO&ORIG=6546
In this I need to extract "SANFRANSISCO"
MYREQUESTISTO8764GETTHISWITH&DEST=SANJOSE
In this I need to extract "SANJOSE"
I am reading a file line by line, and I need to update the text after &DEST= and put it back in the file. The modification of the text is to mask the dest value with X character.
So, SFO should be replaced with XXX.
SANJOSE should be replaced with XXXXXXX.
Output :
MYREQUESTISTO8764GETTHIS&DEST=XXX&ORIG=6546
MYREQUESTISTO8764GETTHIS&DEST=XXXXXXXXXXXX&ORIG=6546
MYREQUESTISTO8764GETTHISWITH&DEST=XXXXXXX
Please let me know how to achieve this in script (Preferably shell or bash script).
Thanks.
$ cat file
MYREQUESTISTO8764GETTHIS&DEST=SFO&ORIG=6546
MYREQUESTISTO8764GETTHIS&DEST=PORTORICA
MYREQUESTISTO8764GETTHIS&DEST=SANFRANSISCO&ORIG=6546
MYREQUESTISTO8764GETTHISWITH&DEST=SANJOSE
$ sed -E 's/^.*&DEST=([^&]*)[&]*.*$/\1/' file
SFO
PORTORICA
SANFRANSISCO
SANJOSE
should do it
Replacing airports with an equal number of Xs
Let's consider this test file:
$ cat file
MYREQUESTISTO8764GETTHIS&DEST=SFO&ORIG=6546
MYREQUESTISTO8764GETTHIS&DEST=SANFRANSISCO&ORIG=6546
MYREQUESTISTO8764GETTHISWITH&DEST=SANJOSE
To replace the strings after &DEST= with an equal length of X and using GNU sed:
$ sed -E ':a; s/(&DEST=X*)[^X&]/\1X/; ta' file
MYREQUESTISTO8764GETTHIS&DEST=XXX&ORIG=6546
MYREQUESTISTO8764GETTHIS&DEST=XXXXXXXXXXXX&ORIG=6546
MYREQUESTISTO8764GETTHISWITH&DEST=XXXXXXX
To replace the file in-place:
sed -i -E ':a; s/(&DEST=X*)[^X&]/\1X/; ta' file
The above was tested with GNU sed. For BSD (OSX) sed, try:
sed -Ee :a -e 's/(&DEST=X*)[^X&]/\1X/' -e ta file
Or, to change in-place with BSD(OSX) sed, try:
sed -i '' -Ee :a -e 's/(&DEST=X*)[^X&]/\1X/' -e ta file
If there is some reason why it is important to use the shell to read the file line-by-line:
while IFS= read -r line
do
echo "$line" | sed -Ee :a -e 's/(&DEST=X*)[^X&]/\1X/' -e ta
done <file
How it works
Let's consider this code:
search_str="&DEST="
newfile=chart.txt
sed -E ':a; s/('"$search_str"'X*)[^X&]/\1X/; ta' "$newfile"
-E
This tells sed to use Extended Regular Expressions (ERE). This has the advantage of requiring fewer backslashes to escape things.
:a
This creates a label a.
s/('"$search_str"'X*)[^X&]/\1X/
This looks for $search_str followed by any number of X followed by any character that is not X or &. Because of the parens, everything except that last character is saved into group 1. This string is replaced by group 1, denoted \1 and an X.
ta
In sed, t is a test command. If the substitution was made (meaning that some character needed to be replaced by X), then the test evaluates to true and, in that case, ta tells sed to jump to label a.
This test-and-jump causes the substitution to be repeated as many times as necessary.
Replacing multiple tags with one sed command
$ name='DEST|ORIG'; sed -E ':a; s/(&('"$name"')=X*)[^X&]/\1X/; ta' file
MYREQUESTISTO8764GETTHIS&DEST=XXX&ORIG=XXXX
MYREQUESTISTO8764GETTHIS&DEST=XXXXXXXXXXXX&ORIG=XXXX
MYREQUESTISTO8764GETTHISWITH&DEST=XXXXXXX
Answer for original question
Using shell
$ s='MYREQUESTISTO8764GETTHIS&DEST=SFO&ORIG=6546'
$ s=${s#*&DEST=}
$ echo ${s%%&*}
SFO
How it works:
${s#*&DEST=} is prefix removal. This removes all text up to and including the first occurrence of &DEST=.
${s%%&*} is suffix removal_. It removes all text from the first & to the end of the string.
Using awk
$ echo 'MYREQUESTISTO8764GETTHIS&DEST=SFO&ORIG=6546' | awk -F'[=\n]' '$1=="DEST"{print $2}' RS='&'
SFO
How it works:
-F'[=\n]'
This tells awk to treat either an equal sign or a newline as the field separator
$1=="DEST"{print $2}
If the first field is DEST, then print the second field.
RS='&'
This sets the record separator to &.
With GNU bash:
while IFS= read -r line; do
[[ $line =~ (.*&DEST=)(.*)((&.*|$)) ]] && echo "${BASH_REMATCH[1]}fooooo${BASH_REMATCH[3]}"
done < file
Output:
MYREQUESTISTO8764GETTHIS&DEST=fooooo&ORIG=6546
MYREQUESTISTO8764GETTHIS&DEST=fooooo&ORIG=6546
MYREQUESTISTO8764GETTHISWITH&DEST=fooooo
Replace the characters between &DEST and & (or EOL) with x's:
awk -F'&DEST=' '{
printf("%s&DEST=", $1);
xlen=index($2,"&");
if ( xlen == 0) xlen=length($2)+1;
for (i=0;i<xlen;i++) printf("%s", "X");
endstr=substr($2,xlen);
printf("%s\n", endstr);
}' file
I am trying to insert a few lines of text before a specific line, but keep getting sed errors when I try to add a new line character. My command looks like:
sed -r -i '/Line to insert after/ i Line one to insert \\
second new line to insert \\
third new line to insert' /etc/directory/somefile.txt
The error that is reported is:
sed: -e expression #1, char 77: unterminated `s' command
I've tried, using \n, \\ (as in the example), no character at all, just moving the second line to the next line. I've also tried something like:
sed -r -i -e '/Line to insert after/ i Line one to insert'
-e 'second new line to insert'
-e 'third new line to insert' /etc/directory/somefile.txt
EDIT!: Apologies, I wanted the text inserted BEFORE the existing, not after!
This should work:
sed -i '/Line to insert after/ i Line one to insert \
second new line to insert \
third new line to insert' file
For anything other than simple substitutions on individual lines, use awk instead of sed for simplicity, clarity, robustness, etc., etc.
To insert before a line:
awk '
/Line to insert before/ {
print "Line one to insert"
print "second new line to insert"
print "third new line to insert"
}
{ print }
' /etc/directory/somefile.txt
To insert after a line:
awk '
{ print }
/Line to insert after/ {
print "Line one to insert"
print "second new line to insert"
print "third new line to insert"
}
' /etc/directory/somefile.txt
On MacOs I needed a few more things.
Double backslash after the i
Empty quotes after the -i to specify no backup file
Leading backslashes to add leading whitespace
Trailing double backslashes to add newlines
This code searches for the first instance of </plugins in pom.xml and inserts another XML object immediately preceding it, separated by a newline character.
sed -i '' "/\<\/plugins/ i \\
\ <plugin>\\
\ <groupId>org.apache.maven.plugins</groupId>\\
\ <artifactId>maven-source-plugin</artifactId>\\
\ <executions>\\
\ <execution>\\
\ <id>attach-sources</id>\\
\ <goals>\\
\ <goal>jar</goal>\\
\ </goals>\\
\ </execution>\\
\ </executions>\\
\ </plugin>\\
" pom.xml
This ll works from the first line.. For eg: If you want to insert from 3rd line of a file, replace "1i" to "3i".
sed -i '1i line1'\\n'line2'\\n'line3' 1.txt
cat 1.txt
line1
line2
line3
Hai
When the lines to be inserted are the result of some command "mycmd" (like cat results.txt or printf "%s\n" line{1..3}), you can do
sed -i 's/Line to insert after/r' <(cmd) file
or
sed -i 's/Line to insert after/echo "&";cmd/e' file
The last command can be simple modified when you want to insert before some match.
sed -i '/Line to insert after/ i\
Line one to insert\
second new line to insert\
third new line to insert' /etc/directory/somefile.txt
This might work for you (GNU sed & Bash):
sed -i $'/Line to insert after/a\line1\\nline2\\nline3' file
To be POSIX compliant and run in OS X, I used the following (single quoted line and empty line are for demonstration purposes):
sed -i "" "/[pattern]/i\\
line 1\\
line 2\\
\'line 3 with single quotes\`
\\
" <filename>
This can be easily done with Perl also
$ cat MeanwhileInHell.txt
Iran|XXXXXX|Iranian
Iraq|YYYYYY|Iraquian
Saudi|ZZZZZ|Saudi is a Rich Country
USA|AAAAAA|USA is United States of America.
India|IIII|India got freedom from British.
Scot|SSSSS|Canada Mexio.
$ perl -pe 'BEGIN {$x="Line one to insert\nLine 2\nLine3\n"} $_=$x.$_ if /USA/ ' MeanwhileInHell.txt
Iran|XXXXXX|Iranian
Iraq|YYYYYY|Iraquian
Saudi|ZZZZZ|Saudi is a Rich Country
Line one to insert
Line 2
Line3
USA|AAAAAA|USA is United States of America.
India|IIII|India got freedom from British.
Scot|SSSSS|Canada Mexio.
$
It took me a while to figure out how to do this, so posting in case anyone else is looking for the same.
For adding a newline after a pattern, you can also say:
sed '/pattern/{G;}' filename
Quoting GNU sed manual:
G
Append a newline to the contents of the pattern space, and then append the contents of the hold space to that of the pattern space.
EDIT:
Incidentally, this happens to be covered in sed one liners:
# insert a blank line below every line which matches "regex"
sed '/regex/G'
This sed command:
sed -i '' '/pid = run/ a\
\
' file.txt
Finds the line with: pid = run
file.txt before
; Note: the default prefix is /usr/local/var
; Default Value: none
;pid = run/php-fpm.pid
; Error log file
and adds a linebreak after that line inside file.txt
file.txt after
; Note: the default prefix is /usr/local/var
; Default Value: none
;pid = run/php-fpm.pid
; Error log file
Or if you want to add text and a linebreak:
sed -i '/pid = run/ a\
new line of text\
' file.txt
file.txt after
; Note: the default prefix is /usr/local/var
; Default Value: none
;pid = run/php-fpm.pid
new line of text
; Error log file
A simple substitution works well:
sed 's/pattern.*$/&\n/'
Example :
$ printf "Hi\nBye\n" | sed 's/H.*$/&\nJohn/'
Hi
John
Bye
To be standard compliant, replace \n by backslash newline :
$ printf "Hi\nBye\n" | sed 's/H.*$/&\
> John/'
Hi
John
Bye
sed '/pattern/a\\r' file name
It will add a return after the pattern while g will replace the pattern with a blank line.
If a new line (blank) has to be added at end of the file use this:
sed '$a\\r' file name
Another possibility, e.g. if You don't have an empty hold register, could be:
sed '/pattern/{p;s/.*//}' file
Explanation:
/pattern/{...} = apply sequence of commands, if line with pattern found,
p = print the current line,
; = separator between commands,
s/.*// = replace anything with nothing in the pattern register,
then automatically print the empty pattern register as additional line)
The easiest option -->
sed 'i\
' filename