This question already has answers here:
How can I use a file in a command and redirect output to the same file without truncating it?
(14 answers)
Closed 3 years ago.
If I try to redirect output for a command into a file that is open for reading within the command, I get an empty file.
For example, suppose I have a file named tmp.txt:
ABC
123
Now if I do this:
$ grep --color=auto A tmp.txt > out.txt
$ cat out.txt
ABC
But if I do this:
$ grep --color=auto A tmp.txt > tmp.txt
$ cat out.txt
$
I get no output.
I'd like to be able to redirect to a file that I am reading within the same command.
Okay, so I have my answer and would like to share it with you all.
You simply have to use a pipe with tee.
$ grep --color=auto A tmp.txt | tee tmp.txt
ABC
$cat tmp.txt
ABC
Perhaps someone who understands pipes well can explain why.
Related
This question already has answers here:
Why does reading and writing to the same file in a pipeline produce unreliable results?
(2 answers)
Closed 3 years ago.
echo "hello" | tee test.txt
cat test.txt
sudo sed -e "s|abc|def|g" test.txt | tee test.txt
cat test.txt
Output:
The output of 2nd command and last command are different, where as the command is same.
Question:
The following line in above script gives an output, but why it is not redirected to output file?
sudo sed -e "s|abc|def|g" test.txt
sudo sed -e "s|abc|def|g" test.txt | tee test.txt
Reading from and writing to test.txt in the same command line is error-prone. sed is trying to read from the file at the same time that tee wants to truncate it and write to it.
You can use sed -i to modify a file in place. There's no need for tee. (There's also no need for sudo. You made the file, no reason to ask for root access to read it.)
sed -e "s|abc|def|g" -i test.txt
You shouldn't use the same file for both input and output.
tee test.txt is emptying the output file when it starts up. If this happens before sed reads the file, sed will see an empty file. Since you're running sed through sudo, it's likely to take longer to start up, so this is very likely.
I succeeded in using a file descriptor with sed and giving the result on the standard output. Giving a file "file.txt" containing :
$ cat file.txt
foo
Foo
I open a file descriptor to file.txt, open a sub-shell, and give this file descriptor to sed :
$ (sed "/Foo/c\\bar" <&9 ) 9< file.txt
foo
bar
The result is correct.
Now, if I want to use the -i option of sed to change in place, I have troubles. I open the file descriptor in read and write mode, then give it to sed as input file :
$ (sed -i "/Foo/c\\bar" <&9 ) 9<> file.txt
sed: no input file
I do not understand why an input file is missing. Maybe sed needs a filename, and not a file descriptor when using the -i option ?
I tried a workaround which, of course, does not work as expected :
$ (sed "/Foo/c\\bar" <&9 >&9 ) 9<> file.txt
$ cat file.txt
foo
Foo
foo
bar
while I expected :
$ cat file.txt
foo
bar
Thanks in advance for your help !
Dunatotatos
You cannot edit anything "in place" with sed, and this is a great example of why -i is misnamed. gnu sed implements -i by creating a new file, writing the output to it, and then renaming the file. If you don't give sed the original filename, it doesn't know what to rename it.
sed -i expects a filename. You can't pass /dev/stdin (or similar), as sed will attempt to create a temporary file inside /dev.
You can't even save the output of sed into a temporary file and then write the output in the file descriptor again, as you can't rewind a file descriptor in Bash.
What you can do is figure out the original file name from the file descriptor. You can do this by using the link /proc/self/fd/9, like this:
sed -i "/Foo/c\\bar" "$(readlink /proc/self/fd/9)"
However, note that the original file may have been deleted or renamed, in which case this solution won't work. Also, this solution expects /proc to be available, which might not always be the case. /dev/fd/9 may be a good replacement.
Another thing to be aware of is that sed -i works by replacing the the file with a new one: after running sed -i, your fd 9 won't refer the newly created file. To workaround this problem:
name="$(readlink /proc/self/fd/9)"
cp "$name" "$name.tmp"
sed "/Foo/c\\bar" "$name.tmp" > "$name"
This way, your fd 9 will still refer the same file before and after running sed. You might want to use mktemp to create the temporary file, and atexit to ensure that it gets deleted.
This question already has answers here:
Input and output redirection to the same file [duplicate]
(4 answers)
Closed 7 years ago.
I am facing this strange behaviour in a simple pipe:
me$ echo "AAA" > tmp.txt
me$ cat tmp.txt | sed 's/A/B/g' > tmp.txt
me$ cat tmp.txt
The result is an empty file and not the desired "BBB" inside the tmp.txt
It works though if I chose a different file for output. some ideas? thx in advance!
You can write this:
sed 's/A/B/g' tmp.txt > tmp2.txt
mv tmp2.txt tmp.txt
The first line writes the contents of the file, with the relevant string replacement, to a new file. The second line moves the new file to the location of the old file, overwriting the old file.
Why not prefer the 'in-place' edit to a cat or rename ?
sed 's/A/B/g' -i tmp.txt
To change a file in place, use sed -i:
$ echo "AAA" > tmp.txt
$ sed -i 's/A/B/g' tmp.txt
$ cat tmp.txt
BBB
The above uses GNU sed syntax. If you are using Mac OSX (BSD), use:
sed -i '' 's/A/B/g' tmp.txt
Discussion
From the question, consider this line of code:
cat tmp.txt | sed 's/A/B/g' > tmp.txt
cat tmp.txt attempts to read from tmp.txt. When the shell sees > tmp.txt, however, it truncates tmp.txt to an empty file in preparation for input. The result of something like this is not reliable.
sed -i by contrast was explicitly designed to handle this situation. It completely avoids the conflict.
If you like, sed -i can create a back-up of the original file. With GNU sed, use:
sed -i.bak 's/A/B/g' tmp.txt
With BSD (Mac OSX) sed, add a space:
sed -i .bak 's/A/B/g' tmp.txt
This question already has answers here:
Remove duplicate entries in a Bash script [duplicate]
(4 answers)
Closed 8 years ago.
Hi I am writing a script in bash which read the contents of files that have the word "contact"(in the current directory) in them and sorts all the data in those files in alphabetical order
and writes them to a file called "out.txt". I was wondering if there was any way in which I could get rid of duplicate content. Any help would be appreciated
The code I have written so far.
#!/bin/bash
cat $(ls | grep contact) > out.txt
sort out.txt -o out.txt
sort has option -u (long option: --unique) to output only unique lines:
sort -u out.txt -o out.txt
EDIT: (Thanks to tripleee)
Your script, at present, contains problems of parsing ls output,
This is a better substitute for what you are trying to do:
sort -u *contact* >out.txt
Use this using the uniq command (easier to remember than flags)
#!/bin/bash
cat $(ls | grep contact) | sort | uniq > out.txt
or the -u flag for sort like this
#!/bin/bash
cat $(ls | grep contact) | sort -u > out.txt
uniq may do what you need. It copies lines from input to output, omitting a line if it was the line it just output.
Take a look at the "uniq" command, and pipe it through there after sorting.
This question already has answers here:
How can I use a file in a command and redirect output to the same file without truncating it?
(14 answers)
Redirect output from sed 's/c/d/' myFile to myFile
(10 answers)
Using the same file for stdin and stdout with redirection
(3 answers)
How to redirect and replace the input file with the output (don't erase myfile when doing "cat myfile > myfile")
(3 answers)
Why piping to the same file doesn't work on some platforms?
(5 answers)
Closed 5 years ago.
There's a way to cat a file, filter it by something and then output a new file with the same name? I'm doing that and I'm getting an empty file, but if I create it with different file name is working. I don't want to create a new file.
Example:
File="My_test_file.txt"
cat ${File} | grep -v "test" > ${File}
in that way is not working, I have to create another file to make it work, as follow:
File="My_test_file.txt"
cat ${File} | grep -v "test" > ${File}.tmp
any idea?
There's a package called moreutils that contains the tool sponge for this exact purpose:
grep -v test foo.txt | sponge foo.txt
If installing tools is not an option, you can implement a naive version that first reads all data into memory, and then finally writes it out:
#!/bin/sh
sponge() (
var="$(cat; printf x)"
printf '%s' "${var%x}" > "$1"
)
grep -v test foo.txt | sponge foo.txt
What you are attempting to do with cat and grep -v can be easily done using sed -i '/pattern/d' and that allows to save changes inline as well:
sed -i.bak '/test/d' "$file"