I have a huge text file. I want to replace all strings that start with Affx- and then some numbers (like Affx-74537382 or Affx-4374575) with rs (and the same numbers like: rs74537382 or rs4374575. Is this possible with sed -i 's/Affx-/rs/ ?
Since the file is so huge I am not sure how to verify that the command is working correctly.
You can use
sed -E 's/^Affx(-[0-9]+)/rs\1/' file > tmp && mv tmp file
Details:
-E - POSIX ERE syntax enabled
^ - start of string
Affx - a literal text
(-[0-9]+) - Group 1 (\1 refers to the value in this group): - and one or more digits.
See the online demo:
#!/bin/bash
s='Blah-1233455
Affx-74537382
Some line here
Affx-4374575
End of text 123456778.'
sed -E 's/^Affx(-[0-9]+)/rs\1/' <<< "$s"
Output:
Blah-1233455
rs-74537382
Some line here
rs-4374575
End of text 123456778.
I have an TARGET.md file, I'm looking for a string and I want to replace it with the content of other md file, I have tried many combinations but it seems like the newline in the files are the ones sed is not liking, I just need to do this using pure bash(it doesn't have to be sed) because this is how the whole script is running:
This works:
local search="##### Header"
local replace="##### Header\\
\\
Line 1\\
Line 2\\
Line 3\\
Line 4"
sed -i '' -e "s/${search}/${replace}/" TARGET.md
But this won't:
file1.md content:
##### Header
Line 1
Line 2
Line 3
Line 4
Script:
local search="##### Header"
local replace=$(curl "path/to/file/in/other/place/file1.md")
sed -i '' -e "s/${search}/${replace}/" TARGET.md
NOTE: I don't have the file1.md in the same place, I'm doing a curl to get the raw content from it, this is why the replace is in a variable.
I'm assuming the concept is possible but my sed syntax is wrong knowing sed can handle newlines out of the box, but not sure what is the proper way to do this.
I've been searching for some days now, any help, tip or guide is appreciated!
You are using the wrong tool. sed is a line editor at heart. While you can repeatedly append to pattern space in some instances, awk with getline provides a more flexible solution. For example with your file1.md:
##### Header
Line 1
Line 2
Line 3
Line 4
and your TARGET.md as:
##### Unreleased
my dog
has fleas
The to replace "##### Unreleased" with the content of file1.md, you can do:
awk -v replace="file1.md" -v search="##### Unreleased" '
$0 == search {while (getline line < replace ) { print line }; next }
{ print }
' TARGET.md
Above you have your replace and search as with sed, but instead of using the line-editor, you use awk to locate the line containing search and the read all lines from replace using getline`. The second rule just prints all other lines as is.
Example Use/Output
In the directory containing each file, you can simply select-copy the above and middle-mouse paste into the terminal to test:
$ awk -v replace="file1.md" -v search="##### Unreleased" '
> $0 == search {while (getline line < replace ) { print line }; next }
> { print }
> ' TARGET.md
##### Header
Line 1
Line 2
Line 3
Line 4
my dog
has fleas
Look things over and let me know if you have further questions.
Taking TARGET.md file from David's answer:
cat TARGET.md
##### Unreleased
my dog
has fleas
You can run sed with r command like this:
search="##### Unreleased"
sed -e "/$search/{r file1.md" -e ';d;}' TARGET.md
##### Header
Line 1
Line 2
Line 3
Line 4
my dog
has fleas
I have a file with below records
user1,fuser1,luser1,user1#test.com,data,user1
user2,fuser2,luser2,user2#test.com,data,user2
user3,fuser3,luser3,user3#test.com,data,user3
I wanted to perform some text replacements from
user1,fuser1,luser1,user1#test.com,data,user1
to
New_user1,New_fuser1,New_luser1,New_user1#test.com,data,New_user1
so I wrote below sed script.
sed -i -e 's/user/New_user/g; s/fuser/New_fuser/g; s/luser/New_luser/g' file
This works perfect. Now I have a requirement that I want to replace in specific line range.
start=2
end=3
sed -i -e ''${start},${end}'s/user/New_user/g; s/fuser/New_fuser/g; s/luser/New_luser/g' file
but this command is replacing pattern in all lines. example output is,
user1,New_fuser1,New_luser1,user1#test.com,data,New_user1
user2,New_fuser2,New_luser2,user2#test.com,data,New_user2
user3,New_fuser3,New_luser3,user3#test.com,data,New_user3
Looks like range is getting applied only to first expression and remaining expressions are getting applied on whole file. How to apply this range to all expressions?
You can use awk variables to use for this functionality, controlling the row and column numbers used for replacing
awk -vFS="," -vOFS="," -v columnStart=2 -v columnEnd=3 -v rowStart=1 -v rowEnd=2 \
'NR>=rowStart&&NR<=rowEnd{for(i=columnStart; i<=columnEnd; i++) \
$i="New_"$i; print }' file
where the awk variables columnStart, columnEnd, rowStart and rowStart determine which columns and rows to replace with , as the de-limiter adopted.
For your input file:-
$ cat input-file
user1,fuser1,luser1,user1#test.com,data,user1
user2,fuser2,luser2,user2#test.com,data,user2
user3,fuser3,luser3,user3#test.com,data,user3
Assuming I want to do replacement in lines 2 and 3 from columns 3-4, I can set-up my awk as
awk -vFS="," -vOFS="," -v columnStart=3 -v columnEnd=4 -v rowStart=2 -v rowEnd=3 \
'NR>=rowStart&&NR<=rowEnd{for(i=columnStart; i<=columnEnd; i++) \
$i="New_"$i; print }' file
user2,fuser2,New_luser2,New_user2#test.com,data,user2
user3,fuser3,New_luser3,New_user3#test.com,data,user3
To apply on the say the last column, set the columnStart and columnEnd to the same value e.g. say on column 6 and on last line only.
awk -vFS="," -vOFS="," -v columnStart=6 -v columnEnd=6 -v rowStart=3 -v rowEnd=3 \
'NR>=rowStart&&NR<=rowEnd{for(i=columnStart; i<=columnEnd; i++) \
$i="New_"$i; print }' file
user3,fuser3,luser3,user3#test.com,data,New_user3
When using GNU Sed (present on Ubuntu, probably Debian, and probably others).
There is a feature which makes this easy:
https://www.gnu.org/software/sed/manual/sed.html#Common-Commands
A group of commands may be enclosed between { and } characters. This
is particularly useful when you want a group of commands to be
triggered by a single address (or address-range) match.
Example: perform substitution then print the second input line:
$ seq 3 | sed -n '2{s/2/X/ ; p}'
X
Given the original question, this should do the trick:
sed -i -e '2,3 {s/user/New_user/g; s/fuser/New_fuser/g; s/luser/New_luser/g}' file
The following works for me:
START=2
NUM=1
sed -i -e "$START,+${NUM} s/user/New_user/g; $START,+${NUM} s/fuser/New_fuser/g; $START,+${NUM} s/luser/New_luser/g" file
As you can see, there are several changes:
The line range has to be present at each expression
The range should be represented (in this case) as the start line number and number of lines (the number of affected lines is NUM+1)
You put extra apostrophe symbols.
Using a single s command:
start=1
end=2
sed -e "$start,$end s/\([fl]*\)user/New_\1user/g" file
[fl]*user will match user with optional f or l first letter
output:
New_user1,New_fuser1,New_luser1,New_user1#test.com,data,New_user1
New_user2,New_fuser2,New_luser2,New_user2#test.com,data,New_user2
user3,fuser3,luser3,user3#test.com,data,user3
I am trying to insert a few lines of text before a specific line, but keep getting sed errors when I try to add a new line character. My command looks like:
sed -r -i '/Line to insert after/ i Line one to insert \\
second new line to insert \\
third new line to insert' /etc/directory/somefile.txt
The error that is reported is:
sed: -e expression #1, char 77: unterminated `s' command
I've tried, using \n, \\ (as in the example), no character at all, just moving the second line to the next line. I've also tried something like:
sed -r -i -e '/Line to insert after/ i Line one to insert'
-e 'second new line to insert'
-e 'third new line to insert' /etc/directory/somefile.txt
EDIT!: Apologies, I wanted the text inserted BEFORE the existing, not after!
This should work:
sed -i '/Line to insert after/ i Line one to insert \
second new line to insert \
third new line to insert' file
For anything other than simple substitutions on individual lines, use awk instead of sed for simplicity, clarity, robustness, etc., etc.
To insert before a line:
awk '
/Line to insert before/ {
print "Line one to insert"
print "second new line to insert"
print "third new line to insert"
}
{ print }
' /etc/directory/somefile.txt
To insert after a line:
awk '
{ print }
/Line to insert after/ {
print "Line one to insert"
print "second new line to insert"
print "third new line to insert"
}
' /etc/directory/somefile.txt
On MacOs I needed a few more things.
Double backslash after the i
Empty quotes after the -i to specify no backup file
Leading backslashes to add leading whitespace
Trailing double backslashes to add newlines
This code searches for the first instance of </plugins in pom.xml and inserts another XML object immediately preceding it, separated by a newline character.
sed -i '' "/\<\/plugins/ i \\
\ <plugin>\\
\ <groupId>org.apache.maven.plugins</groupId>\\
\ <artifactId>maven-source-plugin</artifactId>\\
\ <executions>\\
\ <execution>\\
\ <id>attach-sources</id>\\
\ <goals>\\
\ <goal>jar</goal>\\
\ </goals>\\
\ </execution>\\
\ </executions>\\
\ </plugin>\\
" pom.xml
This ll works from the first line.. For eg: If you want to insert from 3rd line of a file, replace "1i" to "3i".
sed -i '1i line1'\\n'line2'\\n'line3' 1.txt
cat 1.txt
line1
line2
line3
Hai
When the lines to be inserted are the result of some command "mycmd" (like cat results.txt or printf "%s\n" line{1..3}), you can do
sed -i 's/Line to insert after/r' <(cmd) file
or
sed -i 's/Line to insert after/echo "&";cmd/e' file
The last command can be simple modified when you want to insert before some match.
sed -i '/Line to insert after/ i\
Line one to insert\
second new line to insert\
third new line to insert' /etc/directory/somefile.txt
This might work for you (GNU sed & Bash):
sed -i $'/Line to insert after/a\line1\\nline2\\nline3' file
To be POSIX compliant and run in OS X, I used the following (single quoted line and empty line are for demonstration purposes):
sed -i "" "/[pattern]/i\\
line 1\\
line 2\\
\'line 3 with single quotes\`
\\
" <filename>
This can be easily done with Perl also
$ cat MeanwhileInHell.txt
Iran|XXXXXX|Iranian
Iraq|YYYYYY|Iraquian
Saudi|ZZZZZ|Saudi is a Rich Country
USA|AAAAAA|USA is United States of America.
India|IIII|India got freedom from British.
Scot|SSSSS|Canada Mexio.
$ perl -pe 'BEGIN {$x="Line one to insert\nLine 2\nLine3\n"} $_=$x.$_ if /USA/ ' MeanwhileInHell.txt
Iran|XXXXXX|Iranian
Iraq|YYYYYY|Iraquian
Saudi|ZZZZZ|Saudi is a Rich Country
Line one to insert
Line 2
Line3
USA|AAAAAA|USA is United States of America.
India|IIII|India got freedom from British.
Scot|SSSSS|Canada Mexio.
$
I'm trying to use sed to change a variable in the site.js file on my server.
Here is the line: var url = "page.php"; I'm looking to just substitute page.php for whatever.php.
I thought this would be pretty simple and I figured this would work with no issues:
sed -i "s/\url = \".*\"/\url = \"page2.php\"/" /home/site.js
It works okay except instead of getting: var url = "page2.php"; I get: var R1 = "page2.php";
Why is the url value being changed to R1 when I use sed here?
You don't need \ before url.
sed -i -r 's#url\s*=\s*"[^"]+"#url = "page2.php"#' /home/site.js
Extra escaping of " can be eliminated by enclosing sed expression with ' instead of "
It's better to use different separator than / (here #) when the strings themselves may contain /
Try doing this :
sed -i -r 's#(var\s+url\s*=\s*")[^"]+"#\1whatever.php"#' file.js
/ is not mandatory as delimiter, I've picked up # there.
Here's another example: Took me while to figure that you change the / for delimiter and not the / in the directory path.
Use # instead of / for sed delimiter if you have dir path names.
First I tried this:
[root#ip-172-35-24-37 ec2-user]# egrep -q "^(\s*\S+\s+)/dev/shm(\s+\S+\s+\S+)(\s+\S+\s+\S+)(\s*#.*)?\s*$" /etc/fstab && sed -ri "s/^(\s*\S+\s+)/dev/shm(\s+\S+\s+\S+)(\s+\S+\s+\S+)(\s*#.*)?\s*$/\1/dev/shm\2nodev\3\4/" /etc/fstab
And got this error:
sed: -e expression #1, char 20: unknown option to `s'
So then I used # for the sed delimiter instead of /:
[root#ip-172-35-24-37 ec2-user]# egrep -q "^(\s*\S+\s+)/dev/shm(\s+\S+\s+\S+)(\s+\S+\s+\S+)(\s*#.*)?\s*$" /etc/fstab && sed -ri "s#^(\s*\S+\s+)/dev/shm(\s+\S+\s+\S+)(\s+\S+\s+\S+)(\s*#.*)?\s*$#\1/dev/shm\2nodev\3\4##" /etc/fstab
[root#ip-172-35-24-37 ec2-user]#
And it worked.
You can use something else besides # for a delimiter like ! or ? or %. Just don't use / if you have dir paths.