Please explain the output of this shell command:
ls >file1 >file2
Why does the output go to file2 instead of file1?
bash only allows one redirection per file descriptor. If multiple redirections are provided, like in your example, they are processed from left to right, with the last one being the only one that takes effect. (Notice, though, that each file will still be created, or truncated if already in existence; the others just won't be used by the process.)
Some shells (like zsh) have an option to allow multiple redirections. In bash, you can simulate this with a series of calls to tee:
ls | tee file1 file2 > /dev/null
Each call to tee writes its input to the named file(s) and its standard output.
If the shell finds multiple redirections of any output, it will redirect it to the last file given, in your case file2, since redirections are evaluated from left to right.
While it works, you should not do something like that!
You first redirected the STDOUT stream to go to file1 but then immediately redirected it to go to file2. So, the output goes to file2.
Every process initially has three file descriptors - 0 for STDIN, 1 for STDOUT and 2 for STDERR. >file1 means that open the file1 and assign its descriptor the id 1. Note that the process which is about to execute doesn't care about what is the end point where its STDOUT goes. It just writes to whatever is described by file descriptor 1.
For a more technical description of how this works, see this answer.
The redirect operator is short for a stdout redirection(1>). Since the command is evaluated left to right, the last stdout redirection is used for the running of ls.
ls 1> file1 1>file2
is equivalent to
ls >file1 >file2
If you're trying to redirection stderr, use
ls > file1 2> file2
0 = stdin
1 = stdout
2 = stderr
try this, you'll notice that file2 will receive the stderr message.
ls ------ > file1 2> file2
then these, in both cases output will be in stdout and will go to file1.
ls >file1 2>file2
ls 1>file1 2>file2
Because first redirection gets overridden by the second. Note though, that an empty file1 is still created when file1 was opened for output.
Related
I can't differentiate between these two lines of code as the output for each command is same
cat volcanoes.txt
cat < volcanoes.txt
< reads a file (specified on the Right-Hand Side) and pipes it into the STDIN of the command on the Left-Hand Side.
cat takes input and outputs it to STDOUT.
If you provide an argument to cat, it takes input from there. Otherwise, it takes it from STDIN.
It isn't usually useful to use < in conjunction with cat.
cat volcanoes.txt passes volcanoes.txt as an argument, which cat will attempt to locate on disk and open.
cat < volcanoes.txt runs cat with no arguments, and the interpreter opens volcanoes.txt as cat's stdin.
For a clearer example, try testing with multiple files:
echo 1 > a
echo 2 > b
now you can see the difference by comparing
grep . a b
vs
cat a b | grep .
In the first one, a & b are passed as arguments, so grep opens them itself and knows the source of each line of data, so it tells you which file each line came from.
$: grep . a b
a:1
b:2
Done the second way, cat reads both files and puts the content on grep's stdin as a single anonymized stream, much the same as you did with a single file when you said cat < volcanoes.txt. This way, grep only knows data is coming on stdin, and it can't give you the additional info.
$: cat a b | grep .
1
2
For cat, it's functionally the same because of what cat is and does, but it's still mechanically different, and for some programs the difference could be crippling, or at least relevant.
I am looking for a bash one-liner that duplicates stdin to stdout without interleaving. The only solution I have found so far is to use tee, but that does produced interleaved output. What do I mean by this:
If e.g. a file f reads
a
b
I would like to execute
cat f | HERE_BE_COMMAND
to obtain
a
b
a
b
If I use tee - as the command, the output typically looks something like
a
a
b
b
Any suggestions for a clean solution?
Clarification
The cat f command is just an example of where the input can come from. In reality, it is a command that can (should) only be executed once. I also want to refrain from using temporary files, as the processed data is sort of sensitive and temporary files are always error-prone when the executed command gets interrupted. Furthermore, I am not interested in a solution that involves additional scripts (as stated above, it should be a one-liner) or preparatory commands that need to be executed prior to the actual duplication command.
Solution 1:
<command_which_produces_output> | { a="$(</dev/stdin)"; echo "$a"; echo "$a"; }
In this way, you're saving the content from the standard input in a (choose a better name please), and then echo'ing twice.
Notice $(</dev/stdin) is a similar but more efficient way to do $(cat /dev/stdin).
Solution 2:
Use tee in the following way:
<command_which_produces_output> | tee >(echo "$(</dev/stdin)")
Here, you're firstly writing to the standard output (that's what tee does), and also writing to a FIFO file created by process substitution:
>(echo "$(</dev/stdin)")
See for example the file it creates in my system:
$ echo >(echo "$(</dev/stdin)")
/dev/fd/63
Now, the echo "$(</dev/stdin)" part is just the way I found to firstly read the entire file before printing it. It echo'es the content read from the process substitution's standard input, but once all the input is read (not like cat that prints line by line).
Store the second input in a temp file.
cat f | tee /tmp/showlater
cat /tmp/showlater
rm /tmp/showlater
Update:
As shown in the comments (#j.a.) the solution above will need to be adjusted into the OP's real needs. Calling will be easier in a function and what do you want to do with errors in your initial commands and in the tee/cat/rm ?
I recommend tee /dev/stdout.
cat f | tee /dev/stdout
One possible solution I found is the following awk command:
awk '{d[NR] = $0} END {for (i=1;i<=NR;i++) print d[i]; for (i=1;i<=NR;i++) print d[i]}'
However, I feel there must be a more "canonical" way of doing this using.
a simple bash script ?
But this will store all the stdin, why not store the output to a file a read the file both if you need ?
full=""
while read line
do
echo "$line"
full="$full$line\n"
done
printf $full
The best way would be to store the output in a file and show it later on. Using tee has the advantage of showing the output as it comes:
if tmpfile=$(mktemp); then
commands | tee "$tmpfile"
cat "$tmpfile"
rm "$tmpfile"
else
echo "Error creating temporary file" >&2
exit 1
fi
If the amount of output is limited, you can do this:
output=$(commands); echo "$output$output"
Why echo a > file1 > file2 creates both files but only write to file2? (file1 is empty.)
Because I/O redirections are processed from left to right. The sequence of actions is:
Open file1 for writing (creating it if it doesn't exist).
Redirect stdout to file1.
Open file2 for writing (creating it if it doesn't exist).
Redirect stdout to file2.
Run echo a.
I've simplified a shell script down to two commands:
Terminal A (Redirect STDIN to a named pipe):
tee -a >>pipe
Terminal B (Read from the pipe used above):
tail -f pipe
The results I don't understand:
Result 1: Start tee, start tail: any input into the first terminal will be buffered and only show up in the 2nd after the tee command is stopped (ctrl-c).
Result 2: Start tee, start tail, stop tee, start tee again: Now only each line is buffered (the result I want). Results show up in terminal 2 at the end of each line of input into terminal 1.
Result 3 (for what it's worth): Start tail first, then tee: same result as #1.
I also wrote a similar script using exec and cat commands and it exhibits the same behavior.
I'm not an expert on this, but the behavior seems straightforward.
Suppose you apply tail to an ordinary text file; it will print the last 10 lines and quit. If you use tail -f, it will print the last 10 lines, then monitor the file; from then on it will print each new line that is appended to the file. This is the line buffering you're looking for.
Now apply tail -f to a named pipe. Whatever you put in the other end is like the initial contents of the file, and tail waits patiently for the end so that it can print the "last" 10 lines. When that process ends, it sends an "end of file" symbol (I don't know what that is, only that it exists) through the pipe, and tail prints-- and starts monitoring. If you then start one or more new processes that write to the pipe, tail takes the new lines as, well, new, and prints them out.
If you want to buffer and print all lines, you could start-and-stop tee to prime the pump, or just use
tail -n +1 -f pipe
Suppose I have three files file1 file2 file3 having some content
Now when I do this on shell prompt cat file1 > file2 >file3
Content of file1 is copied to file3 and file2 becomes empty
Similarly when I do cat > file1 > file2 > file3
It ask for input and this input is stored in file3 and both file1 and file2 are empty
and also for cat > file1 > file2 < file3 contents of file3 is copied to file2 and file1 is empty.
Can someone please explain to me what is happening I am new to UNIX. Also any website where I can learn more about these redirection operators.
Thanks
Consider how the shell processes each part of the command as it parses it:
cat file1 > file2 >file3
cat file1: prepare a new process with the cat program image with argument file1. ( given 1 or more arguments, cat will read from each argument as a file and write to its output file descriptor)
> file2: change the new process' output file descriptor to write to file2 instead of the current output sink (initially the console for an interactive shell) - create `file2 if necessary.
> file3: change the new process' output file descriptor to write to file3 instead of the current output sink (was file2) - create file3 if necessary
End of command: Spawn the new process
So in the end, file2 is created, but unused. file3 gets the data.
cat > file1 > file2 > file3
cat: prepare a new process with the cat program/image with no arguments. (given no arguments, cat will read from its input file descriptor and write to its output file descriptor)
> file1: change the new process' output file descriptor to write to file1 instead of the current output sink (initially the console for an interactive shell) - create file1 if necessary.
> file2: change the new process' output file descriptor to write to file2 instead of the current output sink (was file1) - create file2 if necessary.
> file3: change the new process' output file descriptor to write to file3 instead of the current output sink - (was file2) create file3 if necessary
End of command: Spawn the new process
So in the end, file1 and file2 are created, but unused. file3 gets the data. cat waits for input on its input device (the console device as default for an interactive shell). Any input that cat receives will go to its output device (which ends up being file3 by the time the shell finished processing the command and invoked cat).
cat > file1 > file2 < file3
cat: prepare a new process with the cat program/image with no arguments. (given no arguments, cat will read from its input file descriptor and write to its output file descriptor)
> file1: change the new process' output file descriptor to write to file1 instead of the current output sink (initially the console for an interactive shell) - create file1 if necessary.
> file2: change the new process' output file descriptor to write to file2 instead of the current output sink (was file1) - create file2 if necessary.
< file3: change the new process' input file descriptor to read from file3 instead of the current input source (initially the console for an interactive shell)
End of command: Spawn the new process
So in the end, file1 is created, but unused. file2 gets the data. cat waits for input on its input device (which as set to file3 by the time the shell finished processing the command and invoked cat). Any input that cat receives will go to its output device (which ends up being file2 by the time the shell finished processing the command and invoked cat).
--
Note that in the first example, cat is the one who processes/opens file1. The shell simply passed the word file1 to the program as an argument. However, the shell opened/created file2 and file3. cat knew nothing about file3 and has no idea where the stuff it was writing to its standard output was going.
In the other 2 examples, the shell opened all the files. cat knew nothing about any files. cat had no idea where its standard input was coming from and where its standard output was going to.
Per #Sorpigal comment - the BASH manual has some good descriptions of what the different redirection operators do. Much of it is the same across different Unix shells to varying degrees, but consult your specific shell manual/manpage to confirm. Thanks #Sorpigal.
http://gnu.org/software/bash/manual/html_node/Redirections.html
You can redirect the standard input < standard output 1> or > error output 2> or both outputs &> but you can only redirect 1:1, you can't redirect one output into two different files.
What you are looking for is the tee utility.
If you don't want to lose original content, you should use redirect and append >> or << operators instead. You can read more here.