Can someone explain to me what this line would do in a shell script?
exec 3<&0 </dev/null
I tried googling, but couldn't hone in on the details. I believe 3 is a new file descriptor, 0 is STDIN? and am not sure what the last /dev/null does, or the purpose of exec or the "<" signs.
exec without a command argument changes the I/O redirection for the rest of the script.
3<&0 duplicates the current stdin descriptor to file desscriptor 3.
</dev/null redirects stdin to /dev/null, which is a special device that contains nothing (reading it returns EOF immediately, writing to it discards the data).
The purpose of all this is to redirect standard input to the null device, but save it on FD 3 so that it can be reverted later. So somewhere later in the script you should see:
exec <&3 3<&-
This duplicates FD 3 back to stdin, and then closes FD 3.
Redirection syntax is described in the Redirections section of the Bash Manual.
Related
ftp -v -n <<! > /tmp/ftp$$ 2>&1
open $TARGET_HOST
user $TARGET_USER $TARGET_PWORD
binary
cd $TARGET_PUT_DIR
put $RESULTS_OUT_DIR/$FILE $FILE
bye
!
I inderstand that <<! is a "here-document" and is passing the commands to ftp until it reaches the delimiter "!" but I can't seem to wrap my head around this redirection:
> /tmp/ftp$$ 2>&1
Could someone please explain what is happening here?
First, the heredoc could be listed last without affecting what happens. Heredocs are traditionally written last but the <<NAME but can actually be written anywhere within the command. The order of << relative to the two > redirections doesn't matter since the former changes stdin and the latter change stdout and stderr.
It'd be clearer if it were written:
ftp -v -n > /tmp/ftp$$ 2>&1 <<!
...
!
Second, to explain the output redirections:
> /tmp/ftp$$ redirects stdout to a file named /tmp/ftp1234, where 1234 is the PID of the current shell process. It's an ad hoc way of making a temporary file with a relatively unique name. If the shell script were run several times in parallel each copy would write to a different temp file.
2>&1 redirects stderr (fd 2) to stdout (fd 1). In other words, it sends error messages to the same file /tmp/ftp$$.
My intent was to have all the output of my bash script displayed on the console and logged to a file.
Here is my script that works as expected.
#!/bin/bash
LOG_FILE="test_log.log"
touch $LOG_FILE
# output to console and to logfile
exec > >(tee $LOG_FILE) 2>&1
echo "Starting command ls"
ls -al
echo "End of script"
However I do not understand why it works that way.
I expected to have exec >>(tee $LOG_FILE) 2>&1 work but it fails although exec >>$LOG_FILE 2>&1 indeed works.
I could not find the reason for the construction exec > >(command ) in the bash manual nor in advanced bash scripting. Can you explain the logic behind it ?
The >(tee $LOG_FILE) is an example of Process substitution, you might wish to search for that. Advanced Shell Scriptng and Bash manual
Using the syntax, <(program) for capturing output and >(program) for feeding input, we can pass data just one record at a time. It is more powerful than command substitution (backticks, or $( )) because it substitutes for a filename, not text. Therefore anywhere a file is normally specified we can substitute a program's standard output or input (although process substitution on input is not all that common).
This is particularly useful where a program does not use standard streams for what you want.
Note that in your example you are missing a space, exec >>(tee $LOG_FILE) 2>&1 is wrong (you will get a syntax error). Rather,
exec > >(tee $LOG_FILE) 2>&1
is correct, that space is critical.
So, the exec > part changes file descriptor 1 (the default), also known as stdout or standard output, to refer to "whatever comes next", in this case it is the process substitution, although normally it would be a filename.
2>&1 redirects file descriptor 2 (stderr or standard error) to refer to the same place as file descriptor 1 (stdout or standard out). Important: if you omit the & you end-up with a file called 1 rather than successful redirection.
Once you have called the exec line above, then you have changed the current process's standard output, so output from the commands which follow go to that tee process instead of to regular stdout.
When writing a bash script. Sometimes you are running a command which opens up another program such as npm, composer.. etc. But at the same time you need to use read in order to prompt the user.
Inevitable you hit this kind of error:
read: read error: 0: Resource temporarily unavailable
After doing some research there seems to be a solution by piping the STDIN of those programs which manipulate the STDIN of your bash script to /dev/null.
Something like:
npm install </dev/null
Other research has shown it has something to do with the fact that the STDIN is being set to some sort of blocking/noblocking status and it isn't being reset after the program finishes.
The question is there some sort of fool proof, elegant way of reading user prompted input without being affected by those programs that manipulate the STDIN and not having to hunt which programs need to have their STDIN redirected to /dev/null. You may even need to use the STDIN of those programs!
Usually it is important to know what input the invoked program expects and from where, so it is not a problem to redirect stdin from /dev/null for those that shouldn't be getting any.
Still, it is possible to do it for the shell itself and all invoked programs. Simply move stdin to another file descriptor and open /dev/null in its place. Like this:
exec 3<&0 0</dev/null
The above duplicates stdin file descriptor (0) under file descriptor 3 and then opens /dev/null to replace it.
After this any invoked command attempting to read stdin will be reading from /dev/null. Programs that should read original stdin should have redirection from file descriptor 3. Like this:
read -r var 0<&3
The < redirection operator assumes destination file descriptor 0, if it is omitted, so the above two commands could be written as such:
exec 3<&0 </dev/null
read -r var <&3
When this happens, run bash from within your bash shell, then exit it (thus returning to the original bash shell). I found a mention of this trick in https://github.com/fish-shell/fish-shell/issues/176 and it worked for me, seems like bash restores the STDIN state. Example:
bash> do something that exhibits the STDIN problem
bash> bash
bash> exit
bash> repeat something: STDIN problem fixed
I had a similar issue, but the command I was running did need a real STDIN, /dev/null wasn't good enough. Instead, I was able to do:
TTY=$(/usr/bin/tty)
cmd-using-stdin < $TTY
read -r var
or combined with spbnick's answer:
TTY=$(/usr/bin/tty)
exec 3<&0 < $TTY
cmd-using-stdin
read -r var 0<&3`
which leaves a clean STDIN in 3 for you to read and 0 becomes a fresh stream from the terminal for the command.
I had the same problem. I solved by reading directly from tty like this, redirecting stdin:
read -p "Play both [y]? " -n 1 -r </dev/tty
instead of simply:
read -p "Play both [y]? " -n 1 -r
In my case, the use of exec 3<&0 ... didn't work.
Clearly (resource temporarily unavailable is EAGAIN) this is caused by programs that exits but leaves STDIN in nonblocking mode.
Here is another solution (easiest to script?):
perl -MFcntl -e 'fcntl STDIN, F_SETFL, fcntl(STDIN, F_GETFL, 0) & ~O_NONBLOCK'
The answers here which suggest using redirection are good. Fortunately, Bash's read should soon no longer need such fixes. The author of Readline, Chet Ramey, has already written a patch: http://gnu-bash.2382.n7.nabble.com/read-may-fail-due-to-nonblocking-stdin-td18519.html
However, this problem is more general than just the read command in Bash. Many programs presume stdin is blocking (e.g., mimeopen) and some programs leave stdin non-blocking after they exit (e.g., cec-client). Bash has no builtin way to turn off non-blocking input, so, in those situations, you can use Python from the command line:
$ python3 -c $'import os\nos.set_blocking(0, True)'
You can also have Python print the previous state so that it may be changed only temporarily:
$ o=$(python3 -c $'import os\nprint(os.get_blocking(0))\nos.set_blocking(0, True)')
$ somecommandthatreadsstdin
$ python3 -c $'import os\nos.set_blocking(0, '$o')'
I often have trouble figuring out certain language constructs because they won't register when googling or duckduckgoing them. With a bit of experimenting, it's often simple to figure it out, but I don't get this one.
I often see stuff like 2>&1 or 3>&- in bash scripts. I know this is some kind of redirection. 1 is stdout and 2 is stderror. 3 is probably custom. But what is the minus?
Also, I have a script whose output I want to log, but also want to see on screen. I use exec > >(tee $LOGFILE); exec 2>&1 for that. It works. But sometimes when I bashtrap this script, I cannot type at the prompt anymore. Output is hidden after Ctrl+C. Can I use a custom channel and the minus sign to fix this, or is it unrelated?
2>&1 means that stderr is redirected to stdout
3>&- means that file descriptor 3, opened for writing(same as stdout), is closed.
You can see more examples of redirection here
As for questions number 3, I think this is a good link.
The 3>&- close the file descriptor number 3 (it probably has been opened before with 3>filename).
The 2>&1 redirect the output of file descriptor 2 (stderr) to the same destination as file descriptor 1 (stdout). This dies call dup2() syscall.
For more information about redirecting file descriptor please consult the bash manpages (`man bash). They are dense but great.
For your script, I would do it like that:
#!/bin/bash
if [[ -z $recursive_call ]]; then
recursive_call=1
export recursive_call
"$0" "$#" | tee filename
exit
fi
# rest of the script goes there
It lose the exit code from the script though. There is a way in bash to get it I guess but I can't remember it now.
I'm following this example from Advaced Bash Guide,IO redirection, that shows ' Redirecting only stderr to a pipe'. I understood how it works except when it close fd 3.
why it need close fd 3 in each command when the last command closes is globally ?
exec 3>&1
ls -l 2>&1 >&3 3>&- | grep bad 3>&-
exec 3>&-
In the shell, initially you have 1:terminal, 2:terminal. Then 1 is duplicated so that 3:terminal.
When ls is executed, the file descriptors are inherited, but the pipe replaces the first, so that you have 1:pipe, 2:terminal, 3:terminal; then the redirects make it 1:terminal, 2:pipe, (3:closed). Meanwhile, grep has 0 connected to the pipe, and inherits 1:terminal, 2:terminal, 3:terminal, but the redirect turns it into 1:terminal, 2:terminal, (3:closed).
Finally, back in the shell, 3 is closed, returning to the initial state of 1:terminal, 2:terminal.
The thing to understand is that file descriptors are inherited when a process is forked, but become independent from then on, so each process's descriptor 3 must be closed separately. In this case, there would probably be no harm in leaving it open for ls and grep, but for tidyness it's closed anyway.