I succeeded in using a file descriptor with sed and giving the result on the standard output. Giving a file "file.txt" containing :
$ cat file.txt
foo
Foo
I open a file descriptor to file.txt, open a sub-shell, and give this file descriptor to sed :
$ (sed "/Foo/c\\bar" <&9 ) 9< file.txt
foo
bar
The result is correct.
Now, if I want to use the -i option of sed to change in place, I have troubles. I open the file descriptor in read and write mode, then give it to sed as input file :
$ (sed -i "/Foo/c\\bar" <&9 ) 9<> file.txt
sed: no input file
I do not understand why an input file is missing. Maybe sed needs a filename, and not a file descriptor when using the -i option ?
I tried a workaround which, of course, does not work as expected :
$ (sed "/Foo/c\\bar" <&9 >&9 ) 9<> file.txt
$ cat file.txt
foo
Foo
foo
bar
while I expected :
$ cat file.txt
foo
bar
Thanks in advance for your help !
Dunatotatos
You cannot edit anything "in place" with sed, and this is a great example of why -i is misnamed. gnu sed implements -i by creating a new file, writing the output to it, and then renaming the file. If you don't give sed the original filename, it doesn't know what to rename it.
sed -i expects a filename. You can't pass /dev/stdin (or similar), as sed will attempt to create a temporary file inside /dev.
You can't even save the output of sed into a temporary file and then write the output in the file descriptor again, as you can't rewind a file descriptor in Bash.
What you can do is figure out the original file name from the file descriptor. You can do this by using the link /proc/self/fd/9, like this:
sed -i "/Foo/c\\bar" "$(readlink /proc/self/fd/9)"
However, note that the original file may have been deleted or renamed, in which case this solution won't work. Also, this solution expects /proc to be available, which might not always be the case. /dev/fd/9 may be a good replacement.
Another thing to be aware of is that sed -i works by replacing the the file with a new one: after running sed -i, your fd 9 won't refer the newly created file. To workaround this problem:
name="$(readlink /proc/self/fd/9)"
cp "$name" "$name.tmp"
sed "/Foo/c\\bar" "$name.tmp" > "$name"
This way, your fd 9 will still refer the same file before and after running sed. You might want to use mktemp to create the temporary file, and atexit to ensure that it gets deleted.
Related
I want to append bash variable (which has html tags) at the beginning of a file.
INSERTTO=<h2>title</h2>
<li>sdfdsf</li>
Below is the command I am using -
sed -i '1i'$INSERTTO file.html
But i am getting error -
sed: -e expression #1, char 177: unknown command: `<'
Do i need to encode the html tags in INSERTTO variable ?
First of all, use quotes:
INSERTTO='<h2>title</h2>
<li>sdfdsf</li>'
Then try this:
sed "1 i $INSERTTO" file.html
Start with some text in your file, e.g.
$ cat file
some text
Then you need your variable to contain an explicit '\n' character where the line break is, e.g. INSERTTO='<h2>title</h2>\n<li>sdfdsf</li>'. Then you can use the sed expression to place both lines as the beginning lines in the file, e.g.
$ INSERTTO='<h2>title</h2>\n<li>sdfdsf</li>'; sed "1i $INSERTTO" file
<h2>title</h2>
<li>sdfdsf</li>
some text
Now at present, what will be done has only been written to the terminal stdout. To modify the file in place, you will need to add the -i option for sed (or -i.bak to save a backup of the original file with the .bak extension. (however you prefer to do it)
I offer you an ed(1) solution.
INSERTTO=<h2>title</h2>
<li>sdfdsf</li>
printf '%s\n' 1i "$INSERTTO" . w | ed -s file.html
Assuming the file is not empty that will work. otherwise use 0a as the address and command instead of 1i
... Or use the cat(1) and mv(1)
INSERTTO=<h2>title</h2>
<li>sdfdsf</li>
Add the stdin flag and use a herestring works in bash but not in POSIX shells.
cat - file.html <<< "$INSERTTO"
You should see the output to stdout, redirect it to another file and move that file to the original file, something like this.
cat - file.html <<< "$INSERTTO" > tempfile && mv tempfile file.html
However if the html file is a symlink then it is now broken... a work around would be to use another cat.
cat - file.html <<< "$INSERTTO" > tempfile && cat tempfile > file.html && rm tempfile.
tempfile is just an example you should take a loot at How to create a tempfile in a secure manner
I want to modify file content with shell script like replace the line 3 in file with a new string, and then save to the original file. Can anyone give advice to achieve that?
It's a bit unusual to do that with vim from a shell script, but since you asked:
vim -es '+3s/.*/a new string' '+wq' file
Usually, you would chose another tool like (sed -i is in-place edit):
sed -i '3s/.*/a new string/' file
Or with awk
gawk -i inplace 'NR==3{$0="a new string"}1' file
In a Unix-like system, you can use sed to replace the content of specific line. For example, below command will replace the 3rd line with "HelloWorld" in text.txt file.
sed -i '3c HelloWorld' text.txt
In case you only want to change part of 3rd line content, you can use :
sed -i '3s/aaa/bbb/' text.txt
this will only replace string "aaa" into "bbb" in 3rd line.
ed would be more appropriate than sed or vim, as it is designed to edit files in-place programmatically. (sed is the stream version of ed; -i is a non-standard extension.)
printf '3s/.*/new stuff/\nw\n' | ed my_file.txt
I am trying to use sed -i -- for find and replace few strings in my file. What it there is error while doing the sed command, do the original file get corrupted ? or does it roll back to original file? . I know if i give sed -i"SUFFIX" --
it will create a backup file before changes but does it delete the backup file on successful execution of it ? or do we need to do that manually?.
I am looking for something in sed to make changes in place but on error or issue it needs to roll back all the changes until happened until then and give me back the original file as-is.
I can do it like below but looking for any optimized solution:
sed 's/abc/def/g' file1 > tmp_file
cp tmp_file file1
You've got a mis-understanding of how -i with the extension works in sed. Its sole purpose is to create a backup of the file as-is in-case when needed to revert back. Your requirement calls for this perfectly!
The backup file is never generated in-case your original command fails out of syntax errors when called. See the following
$ echo 'foo' > file
$ ls
file
$ cat file
foo
$ sed -i.bak 's/foo/bar/s' file
sed: -e expression #1, char 11: unknown option to `s'
$ cat file
foo
$ ls
file
As you can see, even in the forced command failure case, the backup file is not created.
$ sed -i.bak 's/foo/bar/g' file
$ cat file
bar
$ ls
file file.bak
$ cat file.bak
foo
But wait a moment, you've replaced incorrectly with bar, but you wanted to replace with foobar, now revert the file back
$ mv file.bak file
$ ls
file
$ cat file
foo
I have a small problem. In my original.txt file I got some scheme with one variable zmuser to change.
I'm using sed command to change my variable 'zmuser' to other string, which is also a variable.
sed -i -e s/zmuser/${user}/g original.txt
It changes the zmuser for my $user variable in original.txt.
How can I make changes from the file original.txt but save it in another .txt file?
This should do it:
sed -e "s/zmuser/${user}/g" original.txt > another.txt
How it works:
If you omit the -i flag from your sed command, the output will be printed on stdout. The > another.txt construct redirects that output to another.txt file.
you can also do
sed -n -e "s/zmuser/${user}/g" -e "w another.txt" original.txt
I recently wrote a script that does a sed command, to replace all the occurrences of "string1" with "string2" in a file named "test.txt".
It looks like this:
sed -i 's/string1/string2/g' test.txt
The catch is, "string1" does not necessarily exist in test.txt.
I notice after executing a bunch of these sed commands, I get a number of empty files, left behind in the directory, with names that look like this:
"sed4l4DpD"
Does anyone know why this might be, and how I can correct it?
-i is the suffix given to the new/output file. Also, you need -e for the command.
Here's how you use it:
sed -i '2' -e 's/string1/string2/g' test.txt
This will create a file called test.txt2 that is the backup of test.txt
To replace the file (instead of creating a new copy - called an "in-place" substitution), change the -i value to '' (ie blank):
sed -i '' -e 's/string1/string2/g' test.txt
EDIT II
Here's actual command line output from a Mac (Snow Leopard) that show that my modified answer (removed space from between the -i and the suffix) is correct.
NOTE: On a linux server, there must be no space between it -i and the suffix.
> echo "this is a test" > test.txt
> cat test.txt
this is a test
> sed -i '2' -e 's/a/a good/' test.txt
> ls test*
test.txt test.txt2
> cat test.txt
this is a good test
> cat test.txt2
this is a test
> sed -i '' -e 's/a/a really/' test.txt
> ls test*
test.txt test.txt2
> cat test.txt
this is a really good test
I wasn't able to reproduce this with a quick test (using GNU sed 4.2.1) -- but strace did show sed creating a file called sedJd9Cuy and then renaming it to tmp (the file named on the command line).
It looks like something is going wrong after sed creates the temporary file and before it's able to rename it.
My best guess is that you've run out of room in the filesystem; you're able to create a new empty file, but unable to write to it.
What does df . say?
EDIT:
I still don't know what's causing the problem, but it shouldn't be too difficult to work around it.
Rather than
sed -i 's/string1/string2/g' test.txt
try something like this:
sed 's/string1/string2/g' test.txt > test.txt.$$ && mv -f test.txt.$$ test.txt
Something is going wrong with the way sed creates and then renames a text file to replace your original file. The above command uses sed as a simple input-output filter and creates and renames the temporary file separately.
So after much testing last night, it turns out that sed was creating these files when trying to operate on an empty string. The way i was getting the array of "$string1" arguments was through a grep command, which seems to be malformed. What I wanted from the grep was all lines containing something of the type "Text here '.'".
For example the string, "Text here 'ABC.DEF'" in a file, should have been caught by grep, then the ABC.DEF portion of the string, would be substituted by ABC_DEF. Unfortunately the grep I was using would catch lines of the type "Text here ''" (that is, nothing between the ''). When later on, the script attempted to perform a sed replacement using this empty string, the random file was created (probably because sed died).
Thanks for all your help in understanding how sed works.
Its better if you do it in this way:
cat large_file | sed 's/string1/string2/g' > file_filtred