I am trying to replace all the patterns
s#_coded_block[#] with s#_coded_block_# in myfile. I looked online on how to replace patterns with groupings and my command is:
sed -i -E 's/s\([0-9]*\)_coded_block\[\([0-9]*\)\]/s\1_coded_block_\2/g' myfile
However, I am getting
invalid reference \2 on `s'
command's RHS when I execute this command.
With the -E option, you don't need backslashes before the capturing parentheses:
sed -i -E 's/s([0-9]*)_coded_block\[([0-9]*)\]/s\1_coded_block_\2/g' myfile
You might want one-or-more digits, in which case you use + instead of *. If you decide to drop the -E, your original code should work, though if you want at least one digit, you need to write \{1,\}:
sed -i 's/s\([0-9]\{1,\}\)_coded_block\[\([0-9]\{1,\}\)\]/s\1_coded_block_\2/g' myfile
The -i notation shown only works reliably with GNU sed. BSD (macOS or Mac OS X) sed would treat the -E in the first command line as the suffix (in the second, you'd get a complaint about m not being a valid sed command because the script would be treated as the suffix and the m of myfile would be an erroneous sed command. You'd use -i '' to back up (overwrite) a file with no suffix. If you want portable code, use -i.bak which creates a backup file with both variants — the .bak must be attached to the -i for GNU sed.
Related
I'm using sed to replace values in other bash scripts, such as:
somedata="$(<somefile.sh)"
somedata=`sed 's/ ==/==/g' <<< $somedata` # [space]== becomes ==
somedata=`sed 's/== /==/g' <<< $somedata` # ==[space] becomes ==
The same for ||, &&, !=, etc. I think steps should be reduced with the right regex match. The operator does not need surrounding spaces, but may have a space before and after, only before, or only after. Is there a way to handle all of these with one sed command?
There are many other conditions not mentioned also. The script takes more time to execute than desired.
The goal is to reduce the overall execution time so I am hoping to reduce the number of commands used with clever regex to match multiple conditions.
I'm also considering tr, awk or perl - whichever is fastest?
With GNU sed, you can use the | (or) operator:
$ sed -r 's/ *(&&|\|\|) */\1/g' <<< "foo && bar || baz"
foo&&bar||baz
*(&&|\|\|) *: search for zero or more space followed by any of the | separated strings followed by zero or more space
the matching strings are captured and output using backreference
Edit:
As pointed out in comments, you can use the -E flag with GNU sed in place of -r. Your command will be more portable:
sed -E 's/ *(\&\&|\|\|) */\1/g'
As GNU sed also supports \| alternation operator with Basic Regular Expressions, you can use it for better readability:
sed 's/ *\(&&\|||\) */\1/g'
You can chain multiple sed substitutions with the -e flag:
$ echo -n "test data here" | sed -e 's/test/TEST/' \
-e 's/data/HERE/' \
-e 's/here/DATA/'
$ TEST HERE DATA
you can use a sedfile (-f option) alongside with the -i option (replace in-place, no need to store in env. variable):
sed -i -f mysedfile somefile.sh
mysedfile may contain expressions, 1 per line
s/ *&& */\&\&/g
s/ *== */==/g
(or use the -e option to use several expression, but if you have a lot of them, it wil become quickly unreadable)
BTW: -i option creates a temporary file within the processed file directory, so in the end, if operation succeeds, the original file is deleted and the temporary file is renamed into the original file name
When the end of the file is reached, the temporary file is renamed
to the output file's original name. The extension, if supplied,
is used to modify the name of the old file before renaming the
temporary file, thereby making a backup copy(2))
so there's no I/O overhead with that option. No need at all to store in a variable.
I am trying to delete a line from a text file that has a matching ID number.
Student id variable: $sid, for example 12345678;
$FILE = student_record
I first tried:
sed -i '/$sid/d' student_record.txt
Which gave me file not found. Next:
sed -i '/$sid/d' $FILE
And I get: sed: 1: "student_record": unterminated substitute in regular expression
sed -i '/12345678/d' $FILE
Same error as above
sed -i '/$sid/ d' student_record.txt
yields:
sed 1: "student_record.txt": bad flag in substitute command: 'x'
If I try without -i,
sed '/$sid/ d' $FILE
It just prints the whole file and doesn't delete any lines.
Advice would be great.
If the file is called student_record as you say for $FILE, you may be making a mistake using student_record.txt which would explain while you get file not found.
For many of the others, if you use single quotes it will not expand variables, so you'll literally be looking for the string "$sid". If you use double quotes it will expand, so try
sed -i "/$sid/d" "$FILE"
assuming you have GNU sed. If you're on something that does not have GNU, you may not have -i or it may require an argument.
I have a plain text document, which I want to compile inside LaTeX. However, sometimes it has the characters, "#", "$", "%", "&", and "_". To compile properly in LaTeX, I must first replace these characters with "#", "\$", "\%", "\&", and "_". I have used this line in sed:
sed -i 's/\#/\\\#/g' ./file.txt
sed -i 's/\$/\\\$/g' ./file.txt
sed -i 's/\%/\\\%/g' ./file.txt
sed -i 's/\&/\\\&/g' ./file.txt
sed -i 's/\_/\\\_/g' ./file.txt
Is this correct?
Unfortunately, the file is too large to open in any GUI software, so checking if my sed line is correct with a text editor is difficult. I tried searching with grep, but the search does not work as expected (e.g. below, I searched for any lines containing "$"):
grep "\$" file.txt
What is the best way to put "\" in front of these characters?
How can I use grep to successfully check the lines with the replacements?
You can do the replacement with a single call to sed:
sed -i -E 's/([#$%&_\])/\\&/g' file.txt
The & in the replacement text fills in for whichever single character is enclosed in parentheses. Note that since \ is the LaTeX escape character, you'll have to escape it as well in the original file.
sed -i 's/\#/\\\#/g' ./file.txt
sed -i 's/\$/\\\$/g' ./file.txt
sed -i 's/\%/\\\%/g' ./file.txt
sed -i 's/\&/\\\&/g' ./file.txt
sed -i 's/\_/\\\_/g' ./file.txt
You don't need the \ on the first (search) string on most of them, just $ (it's a special character, meaning the end of a line; the rest aren't special). And in the replacement, you only need two \\, not three. Also, you could do it all in one with several -e statements:
sed -i.bak -e 's/#/\\#/g' \
-e 's/\$/\\$/g' \
-e 's/%/\\%/g' \
-e 's/&/\\&/g' \
-e 's/_/\\_/g' file.txt
You don't need to double-escape anything (except the \\) because these are single-quoted. In your grep, bash is interpreting the escape on the $ because it's a special character (specifically, a sigil for variables), so grep is getting and searching for just the $, which is a special character meaning the end of a line. You need to either single-quote it to prevent bash from interpreting the \ ('\$', or add another pair of \\: "\\\$". Presumably, that's where you're getting the\` from, but you don't need it in the sed as it's written.
I think your problem is that bash itself is handling those escapes.
What you have looks right to me. But warning: it will also doubly escape e.g. a \# that is already escaped. If that's not what you want, you might want to modify your patterns to check that there isn't a preceding \ already.
$ is used for bash command substitution syntax. I guess grep "\\$" file.txt should do what you expect.
I do not respond for sed, the other answers are good enougth ;-)
You can use less as viewer to check your huge file (or more, but less is more comfortable than more).
For searching, you can use fgrep: it ignores regular expression => fgrep '\$' will really search for text \$. fgrep is the same as invoking grep -F.
EDIT:
fgrep '\$' and fgrep "\$" are different. In the second case, bash interprets the string and will replace it by a single character: $ (i.e. fgrep will search for $ only).
Suppose my file a.conf is as following
Include /1
Include /2
Include /3
I want to replace "Include /2" with a new line, I write the code in .sh file :
line="Include /2"
rep=""
sed -e "s/${line}/${rep}/g" /root/new_scripts/a.conf
But after running the sh file, It give me the following error
sed: -e expression #1, char 14: unknown option to `s'
If you are using a newer version of sed you can use -i to read from and write to the same file. Using -i you can specify a file extension so a backup will be made, incase something went wrong. Also you don't need to use the -e flag unless you are using multiple commands
sed -i.bak "s/${line}/${rep}/g" /root/new_scripts/a.conf
I have just noticed that as the variables you are using are quoted strings you may want to use single quotes around your sed expression. Also your string contains a forward slash, to avoid any errors you can use a different delimiter in your sed command (the delimiter doesn't need to be a slash):
sed -i.bak 's|${line}|${rep}|g' /root/new_scripts/a.conf
You have to write the changes to a new file and then, move the new file over the old one. Like this:
line="Include 2"
rep=""
sed -e "s/${line}/${rep}/g" /root/new_scripts/a.conf > /root/new_scripts/a.conf-new
mv /root/new_scripts/a.conf-new /root/new_scripts/a.conf
The redirection (> /root/new_scripts/a.conf) wipes the contents of the file before sed can see it.
You need to pass the -i option to sed to edit the file in-place:
sed -i "s/${line}/${rep}/g" /root/new_scripts/a.conf
You can also ask sed to create a backup of the original file:
sed -i.bak "s/${line}/${rep}/g" /root/new_scripts/a.conf
So, if you have to replace a substring in a file, you can use sed command like this, say we have a file as file.txt, so replacing a substring in it can be done like this
searchString="abc";
replaceString="def";
sed -i '' "s|$searchString|$replaceString|g" file.txt
This will all the occurrences of "abc" with "def" in file.txt. Also, this keeps a check for any / character present in the variables used, and with no backup file made.
I have a large number of words in a text file to replace.
This script is working up until the sed command where I get:
sed: 1: "*.js": invalid command code *
PS... Bash isn't one of my strong points - this doesn't need to be pretty or efficient
cd '/Users/xxxxxx/Sites/xxxxxx'
echo `pwd`;
for line in `cat myFile.txt`
do
export IFS=":"
i=0
list=()
for word in $line; do
list[$i]=$word
i=$[i+1]
done
echo ${list[0]}
echo ${list[1]}
sed -i "s/{$list[0]}/{$list[1]}/g" *.js
done
You're running BSD sed (under OS X), therefore the -i flag requires an argument specifying what you want the suffix to be.
Also, no files match the glob *.js.
This looks like a simple typo:
sed -i "s/{$list[0]}/{$list[1]}/g" *.js
Should be:
sed -i "s/${list[0]}/${list[1]}/g" *.js
(just like the echo lines above)
So myFile.txt contains a list of from:to substitutions, and you are looping over each of those. Why don't you create a sed script from this file instead?
cd '/Users/xxxxxx/Sites/xxxxxx'
sed -e 's/^/s:/' -e 's/$/:/' myFile.txt |
# Output from first sed script is a sed script!
# It contains substitutions like this:
# s:from:to:
# s:other:substitute:
sed -f - -i~ *.js
Your sed might not like the -f - which means sed should read its script from standard input. If that is the case, perhaps you can create a temporary script like this instead;
sed -e 's/^/s:/' -e 's/$/:/' myFile.txt >script.sed
sed -f script.sed -i~ *.js
Another approach, if you don't feel very confident with sed and think you are going to forget in a week what the meaning of that voodoo symbols is, could be using IFS in a more efficient way:
IFS=":"
cat myFile.txt | while read PATTERN REPLACEMENT # You feed the while loop with stdout lines and read fields separated by ":"
do
sed -i "s/${PATTERN}/${REPLACEMENT}/g"
done
The only pitfall I can see (it may be more) is that if whether PATTERN or REPLACEMENT contain a slash (/) they are going to destroy your sed expression.
You can change the sed separator with a non-printable character and you should be safe.
Anyway, if you know whats on your myFile.txt you can just use any.