I'm trying to replace a single line in a file with a multiline string stored in a variable.
I am able to get the correct result when printing to screen, but not if I want to do an in-place replacement.
The file has form:
*some code*
*some code*
string_to_replace
*some code*
I want the resulting file to be:
*some code*
*some code*
line number 1
line number 2
line number 3
*some code*
The code I tried was:
new_string="line number 1\nline number 2\nline number 3"
# Correct output on screen
sed -e s/"string_to_replace"/"${new_string}"/g $file
# Single-line output in file: "line number 1line number 2line number 3"
sed -i s/"string_to_replace"/"${new_string}"/g $file
When trying to combine -i and -e options, the result is the same as when only using -i.
I'm using GNU sed version 4.1.5 on CentOS (connected to it through ssh from Mac).
Although you have specifically asked for sed, you can accomplish this using awk by storing your multiline variable in a file using the following
awk '/string_to_replace/{system("cat file_with_multiple_lines");next}1' file_to_replace_in > output_file
In sed you can double quote the command string and let the shell do the expansion for you, like this:
new_string="line number 1\nline number 2\nline number 3"
sed -i "s/string_to_replace/$new_string/" file
Inlining a multi-line string into a sed script requires you to escape any literal newlines (and also any literal & characters, which otherwise interpolates the string you are replacing, as well as of course any literal backslashes, and whichever character you are using as the replacement delimiter). What exactly will work also depends slightly on the precise sed dialect. Ultimately, this may be one of those cases where using something else than sed is more robust and portable. But try e.g.
sed -e 's/[&%\\]/\\&/g' \
-e '$!s/$/\\/' \
-e '1s/^/s%string_to_replace%/' \
-e '$s/$/%g/' <<<$replacement |
# pass to second sed instance
sed -f - "$file"
The <<<"here string" syntax is Bash-specific; you can replace it with printf '%s\n' "$replacement" | sed.
Not all sed versions allow you to pass in a script on standard input with -f -. Maybe try replacing the lone dash with /dev/stdin or /dev/fd/0; if that doesn't work, either, you'll have to save the generated script to a temporary file. (Bash lets you use a command substitution sed -f <(sed ...) "$file" which can be quite convenient, and saves you from having to remove the temporary file when you are done.)
Demo: https://ideone.com/uMqqcx
Related
This question already has answers here:
Sed Insert Multiple Lines
(3 answers)
Closed 1 year ago.
I have a multiline variable that I captured from STDOUT.
I want to insert an echo command using this multiline variable to line 15 in another script (target).
#!/bin/bash
TEST=`cat foo`
echo "$TEST"
sed -i "15i echo \"$TEST\" > someotherfile" target
Contents of foo :
apples
oranges
bananas
carrots
I thought the sed command read in line feeds, which I confirmed my foo has:
user#test$ cat foo | tr -cd '\n' | wc -c
4
When I run my test.sh script, I see what's in $TEST, but am getting an error for the sed command:
user#test$ ./test.sh
apples
oranges
bananas
carrots
sed: -e expression #1, char 18: unknown command: `o'
What am I doing wrong?
Thanks in advance.
GNU sed is assumed, as implied by the syntax used in the question.
#!/bin/bash
# Read contents of file 'foo' into shell variable $test.
test=$(<foo)
# \-escape the newlines in $test for use in Sed.
testEscapedForSed=${test//$'\n'/\\$'\n'}
sed -i "15i echo \"$testEscapedForSed\" > someotherfile" target
Your problem was that passing multi-line strings to sed functions such as i (insert) requires the newlines embedded in those strings to be \-escaped, so that sed knows where the string ends and additional commands, if any, start.
A (nonstandard) parameter expansion is used to replace all newlines in $test with themselves prefixed by \, using ANSI C-quoted string $'\n' to generate actual newline chars.
Also note:
I've renamed TEST to test, because all-uppercase shell-variable names should be avoided.
I've used modern command-substitution syntax $(..) in lieu of legacy syntax `...`.
$(<foo) is a slightly more efficient - although nonstandard - way of reading the content of a file at once.
Try:
Solution1:
awk 'NR==15{print;system("cat foo");next} 1' Input_file
No need to get the complete file into a variable, we could simply print it whichever line of Input_file you want to print it.
Solution2:
line=15; sed -e "${line}r foo" target
Or (in script mode)
cat script.ksh
line=15;
sed -e "${line}r foo" target
Where you could change the number of line where you want to insert the lines from another file.
The i command in sed inserts the lines of text that end with a newline, up until a line that doesn't end with a backslash. The a and c commands are similar. Classic sed doesn't like the first line to appear on the same line as the i command; GNU sed isn't as fussy.
If you were writing the command manually, you'd need to write:
15i\
echo "apples\
oranges\
bananas\
carrots" > someotherfile
At issue now is "how do you want to create this given the file foo contains the list of names?". Sometimes, using sed to generate the sed script is useful. However, it can also be intricate if you need to get backslashes at the ends of lines which are subject to an i (or a or c) command, and it is simpler to circumvent the problem.
{
echo "15i\\"
sed -e '1s/^/echo "/' -e 's/$/\\/' -e '$s/\\$/" > someotherfile/' foo
} | sed -f /dev/stdin target
GNU sed can read its script from standard input using -f -; BSD (macOS) sed doesn't like that, but you can use -f /dev/stdin instead (which also works with GNU sed), at least on systems where there is a /dev/stdin.
Interesting issue.
As already mentioned the whole story for sed to be able to insert multiline text in another file is that this new multiline text must have actually literral \n for line breaks.
So we can use sed to convert real new line chars to literal \n:
$ a=$(tr '\n' '\\' <file3 |sed 's#[\]$##' |sed "s#[\]#\0n#g")
#Alternative: a=$(sed "s#[\]#\0n#g" <(sed 's#[\]$##' <(tr '\n' '\\' <file3)))
$ echo "$a"
apples\noranges\nbananas\ncarrots
How this translation works:
* First we replace all new lines with a single backslash using tr
* Then we remove the backslash from the end of the string
* Then we replace all other backslashes with backaslash and n char.
Since now variable $a contains literal \n between lines, sed will translate them back to actuall new lines:
$ cat file4
Line1
line2
line3
$ sed "2i $a" file4
Line1
apples
oranges
bananas
carrots
line2
line3
Result:
Mutliline replacement can be done with two commands:
$ a=$(tr '\n' '\\' <file3 |sed 's#[\]$##' |sed "s#[\]#\0n#g")
$ sed "2i $a" file4
sed 2i means insert a text before line2. 2a can be used in order to insert something after line2.
Remark:
According to this post which seems to be a duplicate, translation of new lines to literal \n seems that can be done with just :
a=$(echo ${a} | tr '\n' "\\n")
But this method never worked in my system.
Remark2:
The sed operation sed "2i $a" = insert variable $a before line 2 , can be also expressed as sed "1 s/.*/\0\n$a/" = replace all chars of first line with the same chars \0 plus a new line \n plus the contents of variable $a => insert $a after line1 = insert $a before line2.
I have a file pattern.txt which is composed of one very long line of complicated code (~8200 chars).
This code can be found in multiple files inside multiple directories.
I can easily identify a list of these files using
grep -rli 'uniquepartofthecode' *
My concern is how do I replace it with the exact text from within the file ?
I tried to do:
var=$(cat pattern.txt)
sed -i "s/$var//g" targetfile.txt
but I got the following error :
sed: -e expression #1, char 96: unknown option to `s'
sed is interpreting my $var content as a regular expression, I would like it to just match the exact text.
The pattern.txt content could be more or less any combination of characters so I'm afraid I cannot escape every characters efficiently.
Is there a solution using sed ? Or should I use another tool for that ?
EDIT:
I tried using this solution to make a proper regex pattern from my text file.
Is it possible to escape regex metacharacters reliably with sed
the overall process is:
var=$(cat pattern.txt)
searchEscaped=$(sed 's/[^^]/[&]/g; s/\^/\\^/g' <<<"$var")
sed -n "s/$searchEscaped/foo/p" <<<"$var" # if ok, echoes 'foo'
This last command displays "foo". $searchEscaped seems to be properly escaped.
Though, this is not returning anything (it should display foo + the rest of the file without the matched part):
sed -n "s/$searchEscaped/foo/p" targetfile.txt
I think that the best solution is to not use regular expressions at all and resort to string replacement.
One way to do this is using perl:
$ echo "$string_to_replace"
some other stuff abc$^%!# some more
$ echo "$search"
abc$^%!#
$ perl -spe '$len = length $search;
while (($pos = index($_, $search, $n)) > -1) {
substr($_, $pos, $len) = "replacement";
$n = $pos + $len;
}' <<<"$string_to_replace" -- -search="$search"
some other stuff replacement some more
The -p switch tells perl to loop through each line of the variable $string_to_replace (which could easily be replaced by a file). -s allows options to be passed to the script - in this case, I've passed a shell variable containing the search string.
For each line of the file, the while loop runs through all of the matches of the search string. substr is used on the left hand of the assignment to replace a substring of $_, which refers to the current line being processed.
I am having trouble using sed to substitute values and write to a new file. It writes to a new file, but fails to change any values. Here is my code:
cd/mydirectory
echo "Enter file name:"
read file_input
file1= "$file_input"
file1= "$file1.b"
file2= "$file_input"
file2= "${file2}Ins.b"
sed "/\!cats!/s/\!cats!.*/cats!300!/g $file1>$file2
I simply want to substitute whatever text was after cats with the value 300. Whenever I run this script it doesn't overwrite the previous value with 300. Any suggestions?
Try changing
sed "/\!cats!/s/\!cats!.*/cats!300!/g $file1>$file2
to
sed "s/cats.*/cats300/g" $file1 > $file2
To replace text, you often have to use sed like sed "s/foo/bar/g" file_in > file_out, to change all occurrences of foo with bar in file_in, redirecting the output to file_out.
Edit
I noticed that you are redirecting the output to the same file - you can't do that. You have 2 options:
Redirect the results to another file, with a different filename. e.g.:
sed "s/cats.*/cats300/g" $file1 > $file2.tmp
Note the .tmp after $file2
Use the -i flag (if using GNU sed):
sed -i "s/cats.*/cats300/g" $file1
The i stands for inline replacement.
I think this modified version of your script should work:
echo "Enter file name:"
read file_input
file1="$file_input" # No space after '='
file1="$file1.b" # No space after '='
file2="$file_input" # No space after '='
file2="${file2}Ins.b" # No space after '='
sed 's/!cats!.*/!cats!300!/g' "$file1" > "$file2"
Note the single quotes around sed expression: with them, there's no need to escape the !s in your expression. Note also the double quotes around "$file1" and "$file2": if one of those variables contain spaces, this will prevent your command from breaking.
Some further remarks:
As pointed by jim, you may want to use the GNU sed -i option.
Your regex will currently replace everything after !cats! in matching lines. If they were several occurences of !cats! on your line, only one will remain. If instead you just want to replace the value between two ! delimiters, you may consider use following sed command instead:
sed 's/!cats![^!]*/!cats!300/g'
I have redirected some string into one parameter for ex: ab=jyoti,priya, pranit
I have one file : Name.txt which contains -
jyoti
prathmesh
John
Kelvin
pranit
I want to delete the records from the Name.txt file which are contain in ab parameter.
Please suggest if this can be done ?
If ab is a shell variable, you can easily turn it into an extended regular expression, and use it with grep -E:
grep -E -x -v "${ab//,/|}" Name.txt
The string substitution ${ab//,/|} returns the value of $ab with every , substituted with a | which turns it into an extended regular expression, suitable for passing as an argument to grep -E.
The -v option says to remove matching lines.
The -x option specifies that the match needs to cover the whole input line, so that a short substring will not cause an entire longer line to be removed. Without it, ab=prat would cause pratmesh to be removed.
If you really require a sed solution, the transformation should be fairly trivial. grep -E -v -x 'aaa|bbb|ccc' is equivalent to sed '/^\(aaa\|bbb\|ccc)$/d' (with some dialects disliking the backslashes, and others requiring them).
To do an in-place edit (modify Name.txt without a temporary file), try this:
sed -i "/^\(${ab//,/\|}\)\$/d" Name.txt
This is not entirely robust against strings containing whitespace or other shell metacharacters, but if you just need
Try with
sed -e 's/\bjyoti\b//g;s/\bpriya\b//g' < Name.txt
(using \b assuming you need word boundaries)
this will do it:
for param in `echo $ab | sed -e 's/[ ]+//g' -e 's/,/ /g'` ; do res=`sed -e "s/$param//g" < name.txt`; echo $res > name.txt; done
echo $res
I have a large number of words in a text file to replace.
This script is working up until the sed command where I get:
sed: 1: "*.js": invalid command code *
PS... Bash isn't one of my strong points - this doesn't need to be pretty or efficient
cd '/Users/xxxxxx/Sites/xxxxxx'
echo `pwd`;
for line in `cat myFile.txt`
do
export IFS=":"
i=0
list=()
for word in $line; do
list[$i]=$word
i=$[i+1]
done
echo ${list[0]}
echo ${list[1]}
sed -i "s/{$list[0]}/{$list[1]}/g" *.js
done
You're running BSD sed (under OS X), therefore the -i flag requires an argument specifying what you want the suffix to be.
Also, no files match the glob *.js.
This looks like a simple typo:
sed -i "s/{$list[0]}/{$list[1]}/g" *.js
Should be:
sed -i "s/${list[0]}/${list[1]}/g" *.js
(just like the echo lines above)
So myFile.txt contains a list of from:to substitutions, and you are looping over each of those. Why don't you create a sed script from this file instead?
cd '/Users/xxxxxx/Sites/xxxxxx'
sed -e 's/^/s:/' -e 's/$/:/' myFile.txt |
# Output from first sed script is a sed script!
# It contains substitutions like this:
# s:from:to:
# s:other:substitute:
sed -f - -i~ *.js
Your sed might not like the -f - which means sed should read its script from standard input. If that is the case, perhaps you can create a temporary script like this instead;
sed -e 's/^/s:/' -e 's/$/:/' myFile.txt >script.sed
sed -f script.sed -i~ *.js
Another approach, if you don't feel very confident with sed and think you are going to forget in a week what the meaning of that voodoo symbols is, could be using IFS in a more efficient way:
IFS=":"
cat myFile.txt | while read PATTERN REPLACEMENT # You feed the while loop with stdout lines and read fields separated by ":"
do
sed -i "s/${PATTERN}/${REPLACEMENT}/g"
done
The only pitfall I can see (it may be more) is that if whether PATTERN or REPLACEMENT contain a slash (/) they are going to destroy your sed expression.
You can change the sed separator with a non-printable character and you should be safe.
Anyway, if you know whats on your myFile.txt you can just use any.