I am trying to use sed -i -- for find and replace few strings in my file. What it there is error while doing the sed command, do the original file get corrupted ? or does it roll back to original file? . I know if i give sed -i"SUFFIX" --
it will create a backup file before changes but does it delete the backup file on successful execution of it ? or do we need to do that manually?.
I am looking for something in sed to make changes in place but on error or issue it needs to roll back all the changes until happened until then and give me back the original file as-is.
I can do it like below but looking for any optimized solution:
sed 's/abc/def/g' file1 > tmp_file
cp tmp_file file1
You've got a mis-understanding of how -i with the extension works in sed. Its sole purpose is to create a backup of the file as-is in-case when needed to revert back. Your requirement calls for this perfectly!
The backup file is never generated in-case your original command fails out of syntax errors when called. See the following
$ echo 'foo' > file
$ ls
file
$ cat file
foo
$ sed -i.bak 's/foo/bar/s' file
sed: -e expression #1, char 11: unknown option to `s'
$ cat file
foo
$ ls
file
As you can see, even in the forced command failure case, the backup file is not created.
$ sed -i.bak 's/foo/bar/g' file
$ cat file
bar
$ ls
file file.bak
$ cat file.bak
foo
But wait a moment, you've replaced incorrectly with bar, but you wanted to replace with foobar, now revert the file back
$ mv file.bak file
$ ls
file
$ cat file
foo
Related
I want to append bash variable (which has html tags) at the beginning of a file.
INSERTTO=<h2>title</h2>
<li>sdfdsf</li>
Below is the command I am using -
sed -i '1i'$INSERTTO file.html
But i am getting error -
sed: -e expression #1, char 177: unknown command: `<'
Do i need to encode the html tags in INSERTTO variable ?
First of all, use quotes:
INSERTTO='<h2>title</h2>
<li>sdfdsf</li>'
Then try this:
sed "1 i $INSERTTO" file.html
Start with some text in your file, e.g.
$ cat file
some text
Then you need your variable to contain an explicit '\n' character where the line break is, e.g. INSERTTO='<h2>title</h2>\n<li>sdfdsf</li>'. Then you can use the sed expression to place both lines as the beginning lines in the file, e.g.
$ INSERTTO='<h2>title</h2>\n<li>sdfdsf</li>'; sed "1i $INSERTTO" file
<h2>title</h2>
<li>sdfdsf</li>
some text
Now at present, what will be done has only been written to the terminal stdout. To modify the file in place, you will need to add the -i option for sed (or -i.bak to save a backup of the original file with the .bak extension. (however you prefer to do it)
I offer you an ed(1) solution.
INSERTTO=<h2>title</h2>
<li>sdfdsf</li>
printf '%s\n' 1i "$INSERTTO" . w | ed -s file.html
Assuming the file is not empty that will work. otherwise use 0a as the address and command instead of 1i
... Or use the cat(1) and mv(1)
INSERTTO=<h2>title</h2>
<li>sdfdsf</li>
Add the stdin flag and use a herestring works in bash but not in POSIX shells.
cat - file.html <<< "$INSERTTO"
You should see the output to stdout, redirect it to another file and move that file to the original file, something like this.
cat - file.html <<< "$INSERTTO" > tempfile && mv tempfile file.html
However if the html file is a symlink then it is now broken... a work around would be to use another cat.
cat - file.html <<< "$INSERTTO" > tempfile && cat tempfile > file.html && rm tempfile.
tempfile is just an example you should take a loot at How to create a tempfile in a secure manner
I succeeded in using a file descriptor with sed and giving the result on the standard output. Giving a file "file.txt" containing :
$ cat file.txt
foo
Foo
I open a file descriptor to file.txt, open a sub-shell, and give this file descriptor to sed :
$ (sed "/Foo/c\\bar" <&9 ) 9< file.txt
foo
bar
The result is correct.
Now, if I want to use the -i option of sed to change in place, I have troubles. I open the file descriptor in read and write mode, then give it to sed as input file :
$ (sed -i "/Foo/c\\bar" <&9 ) 9<> file.txt
sed: no input file
I do not understand why an input file is missing. Maybe sed needs a filename, and not a file descriptor when using the -i option ?
I tried a workaround which, of course, does not work as expected :
$ (sed "/Foo/c\\bar" <&9 >&9 ) 9<> file.txt
$ cat file.txt
foo
Foo
foo
bar
while I expected :
$ cat file.txt
foo
bar
Thanks in advance for your help !
Dunatotatos
You cannot edit anything "in place" with sed, and this is a great example of why -i is misnamed. gnu sed implements -i by creating a new file, writing the output to it, and then renaming the file. If you don't give sed the original filename, it doesn't know what to rename it.
sed -i expects a filename. You can't pass /dev/stdin (or similar), as sed will attempt to create a temporary file inside /dev.
You can't even save the output of sed into a temporary file and then write the output in the file descriptor again, as you can't rewind a file descriptor in Bash.
What you can do is figure out the original file name from the file descriptor. You can do this by using the link /proc/self/fd/9, like this:
sed -i "/Foo/c\\bar" "$(readlink /proc/self/fd/9)"
However, note that the original file may have been deleted or renamed, in which case this solution won't work. Also, this solution expects /proc to be available, which might not always be the case. /dev/fd/9 may be a good replacement.
Another thing to be aware of is that sed -i works by replacing the the file with a new one: after running sed -i, your fd 9 won't refer the newly created file. To workaround this problem:
name="$(readlink /proc/self/fd/9)"
cp "$name" "$name.tmp"
sed "/Foo/c\\bar" "$name.tmp" > "$name"
This way, your fd 9 will still refer the same file before and after running sed. You might want to use mktemp to create the temporary file, and atexit to ensure that it gets deleted.
I have hundreds of text files in one directory. For all files, I want to delete all the lines that begin with HETATM. I would need a csh or bash code.
I would think you would use grep, but I'm not sure.
Use sed like this:
sed -i -e '/^HETATM/d' *.txt
to process all files in place.
-i means "in place".
-e means to execute the command that follows.
/^HETATM/ means "find lines starting with HETATM", and the following d means "delete".
Make a backup first!
If you really want to do it with grep, you could do this:
#!/bin/bash
for f in *.txt
do
grep -v "^HETATM" "%f" > $$.tmp && mv $$.tmp "$f"
done
It makes a temporary file of the output from grep (in file $$.tmp) and only overwrites your original file if the command executes successfully.
Using the -v option of grep to get all the lines that do not match:
grep -v '^HETATM' input.txt > output.txt
I recently wrote a script that does a sed command, to replace all the occurrences of "string1" with "string2" in a file named "test.txt".
It looks like this:
sed -i 's/string1/string2/g' test.txt
The catch is, "string1" does not necessarily exist in test.txt.
I notice after executing a bunch of these sed commands, I get a number of empty files, left behind in the directory, with names that look like this:
"sed4l4DpD"
Does anyone know why this might be, and how I can correct it?
-i is the suffix given to the new/output file. Also, you need -e for the command.
Here's how you use it:
sed -i '2' -e 's/string1/string2/g' test.txt
This will create a file called test.txt2 that is the backup of test.txt
To replace the file (instead of creating a new copy - called an "in-place" substitution), change the -i value to '' (ie blank):
sed -i '' -e 's/string1/string2/g' test.txt
EDIT II
Here's actual command line output from a Mac (Snow Leopard) that show that my modified answer (removed space from between the -i and the suffix) is correct.
NOTE: On a linux server, there must be no space between it -i and the suffix.
> echo "this is a test" > test.txt
> cat test.txt
this is a test
> sed -i '2' -e 's/a/a good/' test.txt
> ls test*
test.txt test.txt2
> cat test.txt
this is a good test
> cat test.txt2
this is a test
> sed -i '' -e 's/a/a really/' test.txt
> ls test*
test.txt test.txt2
> cat test.txt
this is a really good test
I wasn't able to reproduce this with a quick test (using GNU sed 4.2.1) -- but strace did show sed creating a file called sedJd9Cuy and then renaming it to tmp (the file named on the command line).
It looks like something is going wrong after sed creates the temporary file and before it's able to rename it.
My best guess is that you've run out of room in the filesystem; you're able to create a new empty file, but unable to write to it.
What does df . say?
EDIT:
I still don't know what's causing the problem, but it shouldn't be too difficult to work around it.
Rather than
sed -i 's/string1/string2/g' test.txt
try something like this:
sed 's/string1/string2/g' test.txt > test.txt.$$ && mv -f test.txt.$$ test.txt
Something is going wrong with the way sed creates and then renames a text file to replace your original file. The above command uses sed as a simple input-output filter and creates and renames the temporary file separately.
So after much testing last night, it turns out that sed was creating these files when trying to operate on an empty string. The way i was getting the array of "$string1" arguments was through a grep command, which seems to be malformed. What I wanted from the grep was all lines containing something of the type "Text here '.'".
For example the string, "Text here 'ABC.DEF'" in a file, should have been caught by grep, then the ABC.DEF portion of the string, would be substituted by ABC_DEF. Unfortunately the grep I was using would catch lines of the type "Text here ''" (that is, nothing between the ''). When later on, the script attempted to perform a sed replacement using this empty string, the random file was created (probably because sed died).
Thanks for all your help in understanding how sed works.
Its better if you do it in this way:
cat large_file | sed 's/string1/string2/g' > file_filtred
I'm trying to extract lines from certain files that do not begin with # (commented out). How would I run through a file, ignore everything with a # in front of it, but copy each line that does not start with a # into a different file.
Thanks
Simpler: grep -v '^[[:space:]]*#' input.txt > output.txt
This assumes that you're using Unix/Linux shell and the available Unix toolkit of commands AND that you want to keep a copy of the original file.
cp file file.orig
mv file file.fix
sed '/^[ ]*#/d' file.fix > file
rm file.fix
Or if you've got a nice shiny new GNU sed that all be summarized as
cp file file.orig
sed -i '/^[ ]*#/d' file
In both cases, the regexp in the sed command is meant to be [spaceCharTabChar]
So you saying, delete any line that begins with an (optional space or tab chars) #, but print everything else.
I hope this helps.
grep -v ^\# file > newfile
grep -v ^\# file | grep -v ^$ > newfile
Not fancy regex, but I provide this method to Jr. Admins as it helps with understanding of pipes and redirection.