I would like to run a find and replace on an HTML file through the command line.
My command looks something like this:
sed -e s/STRING_TO_REPLACE/STRING_TO_REPLACE_IT/g index.html > index.html
When I run this and look at the file afterward, it is empty. It deleted the contents of my file.
When I run this after restoring the file again:
sed -e s/STRING_TO_REPLACE/STRING_TO_REPLACE_IT/g index.html
The stdout is the contents of the file, and the find and replace has been executed.
Why is this happening?
When the shell sees > index.html in the command line it opens the file index.html for writing, wiping off all its previous contents.
To fix this you need to pass the -i option to sed to make the changes inline and create a backup of the original file before it does the changes in-place:
sed -i.bak s/STRING_TO_REPLACE/STRING_TO_REPLACE_IT/g index.html
Without the .bak the command will fail on some platforms, such as Mac OSX.
An alternative, useful, pattern is:
sed -e 'script script' index.html > index.html.tmp && mv index.html.tmp index.html
That has much the same effect, without using the -i option, and additionally means that, if the sed script fails for some reason, the input file isn't clobbered. Further, if the edit is successful, there's no backup file left lying around. This sort of idiom can be useful in Makefiles.
Quite a lot of seds have the -i option, but not all of them; the posix sed is one which doesn't. If you're aiming for portability, therefore, it's best avoided.
sed -i 's/STRING_TO_REPLACE/STRING_TO_REPLACE_IT/g' index.html
This does a global in-place substitution on the file index.html. Quoting the string prevents problems with whitespace in the query and replacement.
use sed's -i option, e.g.
sed -i bak -e s/STRING_TO_REPLACE/REPLACE_WITH/g index.html
To change multiple files (and saving a backup of each as *.bak):
perl -p -i -e "s/\|/x/g" *
will take all files in directory and replace | with x
this is called a “Perl pie” (easy as a pie)
You should try using the option -i for in-place editing.
Warning: this is a dangerous method! It abuses the i/o buffers in linux and with specific options of buffering it manages to work on small files. It is an interesting curiosity. But don't use it for a real situation!
Besides the -i option of sed
you can use the tee utility.
From man:
tee - read from standard input and write to standard output and files
So, the solution would be:
sed s/STRING_TO_REPLACE/STRING_TO_REPLACE_IT/g index.html | tee | tee index.html
-- here the tee is repeated to make sure that the pipeline is buffered. Then all commands in the pipeline are blocked until they get some input to work on. Each command in the pipeline starts when the upstream commands have written 1 buffer of bytes (the size is defined somewhere) to the input of the command. So the last command tee index.html, which opens the file for writing and therefore empties it, runs after the upstream pipeline has finished and the output is in the buffer within the pipeline.
Most likely the following won't work:
sed s/STRING_TO_REPLACE/STRING_TO_REPLACE_IT/g index.html | tee index.html
-- it will run both commands of the pipeline at the same time without any blocking. (Without blocking the pipeline should pass the bytes line by line instead of buffer by buffer. Same as when you run cat | sed s/bar/GGG/. Without blocking it's more interactive and usually pipelines of just 2 commands run without buffering and blocking. Longer pipelines are buffered.) The tee index.html will open the file for writing and it will be emptied. However, if you turn the buffering always on, the second version will work too.
sed -i.bak "s#https.*\.com#$pub_url#g" MyHTMLFile.html
If you have a link to be added, try this. Search for the URL as above (starting with https and ending with.com here) and replace it with a URL string. I have used a variable $pub_url here. s here means search and g means global replacement.
It works !
The problem with the command
sed 'code' file > file
is that file is truncated by the shell before sed actually gets to process it. As a result, you get an empty file.
The sed way to do this is to use -i to edit in place, as other answers suggested. However, this is not always what you want. -i will create a temporary file that will then be used to replace the original file. This is problematic if your original file was a link (the link will be replaced by a regular file). If you need to preserve links, you can use a temporary variable to store the output of sed before writing it back to the file, like this:
tmp=$(sed 'code' file); echo -n "$tmp" > file
Better yet, use printf instead of echo since echo is likely to process \\ as \ in some shells (e.g. dash):
tmp=$(sed 'code' file); printf "%s" "$tmp" > file
And the ed answer:
printf "%s\n" '1,$s/STRING_TO_REPLACE/STRING_TO_REPLACE_IT/g' w q | ed index.html
To reiterate what codaddict answered, the shell handles the redirection first, wiping out the "input.html" file, and then the shell invokes the "sed" command passing it a now empty file.
I was searching for the option where I can define the line range and found the answer. For example I want to change host1 to host2 from line 36-57.
sed '36,57 s/host1/host2/g' myfile.txt > myfile1.txt
You can use gi option as well to ignore the character case.
sed '30,40 s/version/story/gi' myfile.txt > myfile1.txt
With all due respect to the above correct answers, it's always a good idea to "dry run" scripts like that, so that you don't corrupt your file and have to start again from scratch.
Just get your script to spill the output to the command line instead of writing it to the file, for example, like that:
sed -e s/STRING_TO_REPLACE/STRING_TO_REPLACE_IT/g index.html
OR
less index.html | sed -e s/STRING_TO_REPLACE/STRING_TO_REPLACE_IT/g
This way you can see and check the output of the command without getting your file truncated.
I want to find specific text in a directory, and then delete the lines from the files that include the specific text.
Now I have two questions:
How can I achieve the task?
What is wrong with What I have tried? I have tried the methods below, but failed. the details are following:
grep -rnw "./" -e "webdesign"
This searches the current directory with pattern "webdesign", and I get the result:
.//pages/index.html:1:{% load webdesign %}
.//pages/pricing.html:1:{% load webdesign %}
.//prototypes.py:16: 'django.contrib.webdesign',
Then I use sed to remove the lines from those files, which doesn't work, only get blank file ( I mean it deletes all my file content):
sed -i "/webdesign/d" ./pages/index.html
or
sed "/webdesign/d" ./pages/index.html > ./pages/index.html
My software environment is: OS X Yosemite, Mac Terminal, Bash
A loop in bash will do the trick provided that there are no filenames with spaces (in which case other solutions are possible, but this is the simplest)
for i in `grep -lrnw "yourdirectory/" -e "webdesign"`
do
sed "/webdesign/d" $i > $i.tmp
# safety to avoid destroying the file if problem arises (disk full?)
if [ $? = 0 ] ; then
mv -f $i.tmp $i
fi
done
note that you should not locate this script in the current directory because it contains webdesign and it will be modified as well :)
Thanks to choroba, I know that -i option doesn't work like wished. But it has another meaning or it would be rejected by the opt parser. It has something to do with suffixes, well, it doesn't matter now, but it's difficult to see the problem at first.
Without -i you cannot work on a file in-place. And redirecting output to the input just destroys the input file (!). That's why your solution did not work.
You can install GNU sed that supports the -i option, then
sed -i '/webdesign/d' files
should work. Note that it's safer to use -i~ to create a backup.
You cannot write to the same file you're reading from, that's why
sed /webdesign/d file > file
doesn't work (it overwrites the file before you can read anything from it). Create a temporary file
sed /webdesign/d file > file.tmp
mv file.tmp file
sed -e "\$aDNS3=" <filedirectory>
I'm trying to add in the line "DNS3=" in to the end of the file, so after executing the command in a bash script, it opens the file and include the DNS3= into the last line but did not save the changes. Is there anyway to save the changes and also not show the changes made?
You can use -i option. This will edit the file in-place.
From man sed:
-i[SUFFIX], --in-place[=SUFFIX]
edit files in place (makes backup if SUFFIX supplied)
Using sed -e "\$aDNS3=" -i filename would set DNS3= into the last line of the file.
echo "DNS=" >> file
would be a lot simpler, and faster because it does not need to read the file.
I would like to add the text foo to the start of all files in a certain directory. I tried the following command:
sed '1i foo' *
But that only added the text to the first file. How can I append to all files in a single sed command? (I know that it can be done with a for loop, I'm specifically asking for a single sed command)
Use option --separate (or short: -s) to consider files as separate rather than as a single continuous long stream.
If you want to alter the files, just add the -i flag:
sed -i '1i foo' *
It'll add foo at the beginning of every file.
Note: without -i, the files weren't actually modified. foo was added at the beginning of the whole content to the standard output only, thus appearing only once.
How about:
sed -i '1 s/^/foo/g' *
#!/bin/sed -f
s/","/|/g; # global change of "," to bar
# do some more stuff
#s/|/","/g; # global change of bar back to ","
#---end of script---
The above script removes the 2nd field from a CSV, and clears out quotes and such. I didn't include most of the script because it's not pertinent to the question.
The script is saved in the file fix.sh.
I can run it on a file like this:
$ ./fix.sh <myfile.txt >outputfile.txt
And it works great.
But I want it to replace in file. This doesn't work:
$ ./fix.sh <myfile.txt >myfile.txt
It results in an empty myfile.txt.
This doesn't work either:
$ ./fix.sh myfile.txt
I tried finding some documentation on sed bash scripts but didn't find anything to help me.
I'm sure the answer is simple, I just can't find it. Thanks for your help.
EDIT: I should have mentioned that this is running on a CentOS 6 machine.
Full script is below. Its overall result is to remove field#2 and strip quotes.
#!/bin/sed -nf
# adapted from http://www.linuxtopia.org/online_books/linux_tool_guides/the_sed_faq/sedfaq4_005.html
s/","/|/g; # global change of "," to bar
s/^"//;
s/"$//;
s/^\([^|]*\)|[^|]*|/\1|/; # delete 2nd field contents
s/||/|/; # change || to |
s/ //g; # remove spaces
s/|/,/g;
#s/|/","/g; # global change of bar back to ","
#---end of script---
If your sed supports -i option then you can run your script like this:
./fix.sh -i myfile.txt
-i option of sed does the in-file substitutions.
If your version of sed does not support the -i option then you can do the following which is pretty much the same thing that -i does behind the scene:
./fix.sh myfile.txt > temp && mv temp myfile.txt
Why redirecting to the same file doesn't work?
The reason is that the redirection opens the file for writing and ends up clearing any existing contents. sed then tries to read this empty file, and does nothing. The file is then closed and there by you get an empty file.