I have a quite unusual problem in one of my bash scripts. I want to do something (in fact I want to create/remve LVs, this is MWE) like this:
#! /bin/bash
# ...
exec {flock}>/tmp/lock
# Do something with fd ${flock} e.g.
flock -n ${flock} || exit 1
# ...
lvs ${flock}>&-
# ...
The problem is the ${flock}>&-. Why do I want this? The LVM tools complain with a warning about any opened file descriptors except for stdin, stdout and stderr. So when I drop this small redirecting part, the script works but writes out a warning message.
Thus I wanted to redirect the fd $flock only for the LVM command to be closed. I do not want to close the file but only redirect for this single command invocation.
In my case $flock is set to 10 (first free fd greater or equal to 10, see man bash). However I do not get the corresponding fd remapped as sketched above. Instead the 10 is considered a parameter of the (lvs) command and the stdout should be redirected. Of course this is not what I intend.
If I hardcode 10>&- this works but is very bad style. For now I switched to completely hardcode the fd in the whole file. Nevertheless I would like to know how it would be done correctly.
Don't use the dollar
Your example code has:
lvs ${flock}>&-
The correct syntax is:
lvs {flock}>&-
From the redirection section of the Bash manual:
Each redirection that may be preceded by a file descriptor number may instead be preceded by a word of the form {varname}. In this case, for each redirection operator except >&- and <&-, the shell will allocate a file descriptor greater than 10 and assign it to {varname}. If >&- or <&- is preceded by {varname}, the value of varname defines the file descriptor to close. If {varname} is supplied, the redirection persists beyond the scope of the command, allowing the shell programmer to manage the file descriptor himself.
(emphasis mine)
Related
We know that:
The pipe operator | is used to take the standard output of left side command as the standard input for the right side process.
The stdout redirection operator > is used to redirect the stdout to a file
And the question is, why cannot ls -la | > file redirect the output of ls -la to file? (I tried, and the file is empty)
Is it because that the stdout redirection operator > is not a process?
Is it because that the stdout redirection operator > is not a process?
In short, yes.
In a bit more detail, stdout, stderr and stdin are special file descriptors (FDs), but these remarks work for every FD: each FD refers to exactly one resource. It can be a file, a directory, a pipe, a device (such as terminal, a hard drive etc) and more. One FD, one resource. It is not possible for stdout to output to both a pipe and a file at the same time. What tee does is takes stdin (typically from a pipe, but not necessarily), opens a new FD associated with the filename provided as its argument, and writes whatever it gets from stdin to both stdout and the new FD. This copying of content from one to two FDs is not available from bash directly.
EDIT: I tried answering the question as originally posted. As it stands now, DevSolar's comment is actually more on point: why does > file, without a command, make an empty file in bash?
The answer is in Shell Command Language specification, under 2.9.1 Simple commands. In the first step, the redirection is detected. In the second step, no fields remain, so there is no command to be executed. In step 3, redirections are performed in a subshell; however, since there is no command, standard input is simply discarded, and the empty standard output of no-command is used to (try to) create a new file.
From the bash man pages, under the section "Redirection":
Redirecting Standard Output and Standard Error
This construct allows both the standard output (file descriptor 1) and the standard error output (file descriptor 2) to be redirected to the file whose name is the expansion of word.
There are two formats for redirecting standard output and standard error:
&>word
and
>&word
Of the two forms, the first is preferred.
That made me wonder, why is the first preferred? I see that &>> works, but >>& does not, so the preference makes sense. So why does >>& not work? Is it ambiguous?
Here's what I'm running
$ bash --version
GNU bash, version 4.2.46(1)-release (x86_64-redhat-linux-gnu)
&>dest is unambiguous, whereas >&dest looks like fdup() syntax ...and can be construed as fdup syntax when you have a numeric filename.
Bash, since 4.1 or later, allows the destination of a fdup operation to be parameterized -- thus, not just 2>&1, but also 2>&$stderr_fd. Putting the & after the > puts us into the namespace of syntax used for fdup() operations; keeping it beforehand is unambiguous.
Example: I want to redirect the stdout and stderr of the command cmd into my file. If my file's name is 0, >& causes issues.
cmd >& 0 becomes cmd 1>&0. That is, redirect stdout to fd#0.
cmd &> 0 becomes cmd > 0 2>&1. That is, redirect stdout to the file named 0, then redirect stderr to fd#1.
[n]>&word is POSIX-standardized; &>word is not found anywhere in the POSIX standard.
The only forms for duplicating existing file descriptors defined by the POSIX standard are [n]<&word and [n]>&word. In the grammar, these are given the token names GREATAND and LESSAND.
There are no POSIX-defined tokens for &> or &< – they are merely syntactic sugar for the commonly used operation of "I don't want to see anything on my screen", or "send stdout and stderr to a file".
This is useful, because cmd 2>&1 > myFile - surprisingly to those new to bash - doesn't work as intended for the "clean screen" goal, whereas cmd > myFile 2>&1, does.
So.. why does &>> work when >>& does not? Because whoever wrote &>> didn't feel the need to create an ambiguity, and consciously chose to not allow >>&.
fdup() synonyms with >> would not add value.
To explain a bit more about why [n]>&word synonyms are pointless -- keep in mind that the difference between >bar and >>bar is the presence of the O_APPEND flag and absence of O_TRUNC in the flags argument to open(). However, when you're providing a file descriptor number -- and thus performing an fdup() from an old FD number to a new one -- the file is already open; the flags thus cannot be changed. Even the direction -- > vs < -- is purely informational to the reader.
I've just seen the following in a script and I'm not sure what it means:
.............
started=$STATUSDIR/.$EVENT_ID-started
errs=$STATUSDIR/.$EVENT_ID-errors
# started is used to capture the time we started, so
# that it can be used as the latest-result marker for
# the next run...
>"$started"
>"$errs"
# store STDERR on FD 3, then point STDERR to $errs
exec 3>&2 2>"$errs"
..............
Specifically, the ">" at the beginning of the lines. The script actually fails with "No such file or directory". The vars are all supplied via subsidiary scripts and there doesn't seem to be any logic to create the directories it's complaining about.
It's not the easiest thing to Google for, so I thought I'd ask it here so that future bash hackers might find the answers you lovely people are able to provide.
This is a redirection. It's the same syntax used for echo hello >file (or its less-conventional but equally-correct equivalent >file echo hello), just without the echo hello. :)
When attached to an empty command, the effect of a redirection is identical to what it would be with a command that ran and immediately exited with no output: It creates (if nonexistent) or truncates (if existent) the output file, closes that file, and lets the script move on to the next command.
How to explain the output of cat /etc/passwd | cat </etc/issue?
In this case, the second cat receives contents from /etc/passwd as $STDIN and again /etc/issue is redirected. Why there is only /etc/issue left?
What's more, cat </etc/passwd </etc/issue only outputs the contents in /etc/issue. Is /etc/passwd overwritten?
I am not looking for a solution how to cat two files, but confused with how pipeline works.
Piping and redirection are processed from left to right.
So first the input of cat is redirected to the pipe. Then it is redirected to /etc/issue. Then the program is run, using the last redirection, which is the file.
When you do cat <file1 <file2, stdin is first redirected to file1, then it is redirected to file2. Then the program is run, and it gets its input from the last redirection.
It's like variable assignments. If you do:
stdin=passwd
stdin=issue
The value of stdin at the end is the last one assigned.
This is explained in the bash documentation, in the first paragraph of the section on Redirection:
Before a command is executed, its input and output may be redirected using a special notation interpreted by the shell. Redirection may also be used to open and close files for the current shell execution environment. The following redirection operators may precede or appear anywhere within a simple command or may follow a command. Redirections are processed in the order they appear, from left to right.
(emphasis mine). I assume it's also in the POSIX shell specification, I haven't bothered to look it up. This is how Unix shells have always behaved.
The pipe is created first: the standard output of cat /etc/passwd is sent to write side of the pipe, and the standard input of cat </etc/issue is set to the read side of the pipe. Then the command on each half of the pipe is processed. There's no other I/O redirection on the LHS, but on the RHS, the standard input is redirected so it comes from /etc/issue. That means there's nothing actually reading the read end of the pipe, so the LHS cat is terminated with a SIGPIPE (probably; alternatively, it writes data to the pipe but no process ever reads it). The LHS cat never knows about the pipe input — it only has the the file input for its standard input.
I am a Bash beginner but I am trying to learn this tool to have a job in computers one of these days.
I am trying to teach myself about file descriptors now. Let me share some of my experiments:
#!/bin/bash
# Some dummy multi-line content
read -d '' colours <<- 'EOF'
red
green
blue
EOF
# File descriptor 3 produces colours
exec 3< <(echo "$colours")
# File descriptor 4 filters colours
exec 4> >(grep --color=never green)
# File descriptor 5 is an unlimited supply of violet
exec 5< <(yes violet)
echo Reading colours from file descriptor 3...
cat <&3
echo ... done.
echo Reading colours from file descriptor 3 again...
cat <&3
echo ... done.
echo Filtering colours through file descriptor 4...
echo "$colours" >&4
echo ... done. # Race condition?
echo Dipping into some violet...
head <&5
echo ... done.
echo Dipping into some more violet...
head <&5
echo ... done.
Some questions spring to mind as I see the output coming from the above:
fd3 seems to get "depleted" after "consumption", is it also automatically closed after first use?
how is fd3 different from a named pipe? (something I have looked at already)
when exactly does the command yes start executing? upon fd declaration? later?
does yes stop (CTRL-Z or other) and restart when more violet is needed?
how can I get the PID of yes?
can I get a list of "active" fds?
very interesting race condition on filtering through fd4, can it be avoided?
will yes only stop when I exec 5>&-?
does it matter whether I close with >&- or <&-?
I'll stop here, for now.
Thanks!
PS: partial (numbered) answers are fine.. I'll put together the different bits and pieces myself.. (although a comprehensive answer from a single person would be impressive!)
fd3 seems to get "depleted" after "consumption", is it also automatically closed after first use?
No, it is not closed. This is due to the way exec works. In the mode in which you have used exec (without arguments), its function is to arrange the shell's own file descriptors as requested by the I/O redirections specified to itself, and then leave them that way until the script terminated or they are changed again later.
Later, cat receives a copy of this file descriptor 3 on its standard input (file descriptor 0). cat's standard input is implicitly closed when cat exits (or perhaps, though unlikely, cat closes it before it exists, but that doesn't matter). The original copy of this file, which is the shell's file descriptor 3, remains. Although the actual file has reached EOF and nothing further will be read from it.
how is fd3 different from a named pipe? (something I have looked at already)
The shell's <(some command) syntax (which is not standard bourne shell syntax and I believe is only available in zsh and bash, by the way) might actually be implemented using named pipes. It probably isn't under Linux because there's a better way (using /dev/fd), but it probably is on other operating systems.
So in that sense, this syntax may or may not be a helper for setting up named pipes.
when exactly does the command yes start executing? upon fd declaration? later?
As soon as the <(yes violet) construct is evaluated (which happens when the exec 5< <(yes violet) is evaluated).
does yes stop (CTRL-Z or other) and restart when more violet is needed?
No, it does not stop. However, it will block soon enough when it starts producing more output than anything reading the other end of the pipe is consuming. In other words, the pipe buffer will become full.
how can I get the PID of yes?
Good question! $! appears to contain it immediately after yes is executed. However there seems to be an intermediate subshell and you actually get the pid of that subshell. Try <(exec yes violet) to avoid the intermediate process.
can I get a list of "active" fds?
Not from the shell. But if you're using an operating system like Linux that has /proc, you can just consult /proc/self/fd.
very interesting race condition on filtering through fd4, can it be avoided?
To avoid it, you presumably want to wait for the grep process to complete before proceeding through the script. If you obtain the process ID of that process (as above), I think you should be able to wait for it.
will yes only stop when I exec 5>&-?
Yes. What will happen then is that yes will continue to try to produce output forever but when the other end of the file descriptor is closed it will either get a write error (EPIPE), or a signal (SIGPIPE) which is fatal by default.
does it matter whether I close with >&- or <&-?
No. Both syntaxes are available for consistency's sake.