This question already has answers here:
Why does reading and writing to the same file in a pipeline produce unreliable results?
(2 answers)
Closed 3 years ago.
echo "hello" | tee test.txt
cat test.txt
sudo sed -e "s|abc|def|g" test.txt | tee test.txt
cat test.txt
Output:
The output of 2nd command and last command are different, where as the command is same.
Question:
The following line in above script gives an output, but why it is not redirected to output file?
sudo sed -e "s|abc|def|g" test.txt
sudo sed -e "s|abc|def|g" test.txt | tee test.txt
Reading from and writing to test.txt in the same command line is error-prone. sed is trying to read from the file at the same time that tee wants to truncate it and write to it.
You can use sed -i to modify a file in place. There's no need for tee. (There's also no need for sudo. You made the file, no reason to ask for root access to read it.)
sed -e "s|abc|def|g" -i test.txt
You shouldn't use the same file for both input and output.
tee test.txt is emptying the output file when it starts up. If this happens before sed reads the file, sed will see an empty file. Since you're running sed through sudo, it's likely to take longer to start up, so this is very likely.
Related
This question already has answers here:
How can I use a file in a command and redirect output to the same file without truncating it?
(14 answers)
Closed 3 years ago.
If I try to redirect output for a command into a file that is open for reading within the command, I get an empty file.
For example, suppose I have a file named tmp.txt:
ABC
123
Now if I do this:
$ grep --color=auto A tmp.txt > out.txt
$ cat out.txt
ABC
But if I do this:
$ grep --color=auto A tmp.txt > tmp.txt
$ cat out.txt
$
I get no output.
I'd like to be able to redirect to a file that I am reading within the same command.
Okay, so I have my answer and would like to share it with you all.
You simply have to use a pipe with tee.
$ grep --color=auto A tmp.txt | tee tmp.txt
ABC
$cat tmp.txt
ABC
Perhaps someone who understands pipes well can explain why.
This question already has answers here:
sed edit file in place
(15 answers)
Closed 5 years ago.
Let's say we want to replace the letter "a" with its Morse code equivalent from a text.txt file to a hidden .out.txt file using sed, with the contents of the original file being "this is a test".
#!/bin/bash
touch .out.txt
cat text.txt > .out.txt
sed 's/a/.-/g' .out.txt
The output, as expected, is "this is .- test".
However, let's say we want to do multiple instances of sed, without constantly printing the result of each instance of sed using the -n parameter. In this case:
#!/bin/bash
touch .out.txt
cat text.txt > .out.txt
sed -n 's/a/.-/g' .out.txt
sed -n 's/t/-/g' .out.txt
cat .out.txt
However, in this case, the output of cat is the same as the contents of text.txt, "this is a test".
Is there a possible substitute for -n, or in general a way to prevent sed from printing anything with our wanted result?
Check your seds sed --help for the syntax of -ioption.
This is necessary, because the syntax is different depending on which sed you are using, i.e. I cannot simply tell you.
Then use it accordingly to have the effects of sed take place in-place on the file.
I'm using the sed 'w' command to get the labels from a TeX document using:
/\\label{[a-zA-Z0-9]*}/w labels.list
This script is part of a pipeline in which, later on, awk reads the file that sed has just written. e.g
cat bob | sed -f sedScript | awk -f awkScript labels.list -
Sometimes the pipeline produces the correct output, sometimes it doesn't (for exactly the same input file 'bob'). It's random.
I can only conclude that sometimes awk tries to read the file before sed has closed it properly. Is there anyway I can force sed to close the file at the end of the script, or any other suggestions as to what the problem may be?
All stages in a pipeline run in parallel. This is an extremely important and defining feature of pipes, and there is nothing you can or should attempt to do in order to prevent or circumvent that.
Instead, you should rewrite your script so that all data dependencies are executed and finished in the order you need them to be. In the general case, you'd do
cat bob | sed -f sedScript > tempfile
cat tempfile | awk -f awkScript labels.list -
or equivalently in your case:
grep '\\label{[a-zA-Z0-9]*}' bob > labels.list
awk -f awkScript labels.list bob
I am writing a KornShell (ksh) script that is logging to a file. I am redirecting the output of one of my commands (scp) to the same file, but I would like to add a tab at the start of those lines in the log file if possible.
Is this possible to do?
EDIT: Also I should mention that the text I am redirecting is coming from stderr. My line currently looks like this:
scp -q ${wks}:${file_location} ${save_directory} >> ${script_log} 2>&1
Note: the below doesn't work for ksh (see this question for possible solutions).
You probably can do something like
my_command | sed 's/^/\t/' >> my.log
The idea is to process the output of the command with a stream editor like sed in some manner. In this case, a tab will be added at the beginning of every line. Consider:
$ echo -e 'Test\nfoobar' | sed 's/^/\t/'
Test
foobar
I haven't tested this in ksh, but a quick web search suggests that it should work.
Also note that some commands can write to both stdout and stderr, don't forget to handle it.
Edit: in response to the comment and the edit in the question, the adjusted command can look like
scp -q ${wks}:${file_location} ${save_directory} 2>&1 | \
sed 's/^/\t/' >> ${script_log}
or, if you want to get rid of stdout completely,
scp -q ${wks}:${file_location} ${save_directory} 2>&1 >/dev/null | \
sed 's/^/\t/' >> ${script_log}
The technique is described in this answer.
This question already has answers here:
How can I use a file in a command and redirect output to the same file without truncating it?
(14 answers)
Redirect output from sed 's/c/d/' myFile to myFile
(10 answers)
Using the same file for stdin and stdout with redirection
(3 answers)
How to redirect and replace the input file with the output (don't erase myfile when doing "cat myfile > myfile")
(3 answers)
Why piping to the same file doesn't work on some platforms?
(5 answers)
Closed 5 years ago.
There's a way to cat a file, filter it by something and then output a new file with the same name? I'm doing that and I'm getting an empty file, but if I create it with different file name is working. I don't want to create a new file.
Example:
File="My_test_file.txt"
cat ${File} | grep -v "test" > ${File}
in that way is not working, I have to create another file to make it work, as follow:
File="My_test_file.txt"
cat ${File} | grep -v "test" > ${File}.tmp
any idea?
There's a package called moreutils that contains the tool sponge for this exact purpose:
grep -v test foo.txt | sponge foo.txt
If installing tools is not an option, you can implement a naive version that first reads all data into memory, and then finally writes it out:
#!/bin/sh
sponge() (
var="$(cat; printf x)"
printf '%s' "${var%x}" > "$1"
)
grep -v test foo.txt | sponge foo.txt
What you are attempting to do with cat and grep -v can be easily done using sed -i '/pattern/d' and that allows to save changes inline as well:
sed -i.bak '/test/d' "$file"