I read a part of manual of bash. The item is "COMMAND EXECUTION ENVIRONMENT". The part says,
Builtin commands that are invoked as part of a pipeline are also executed in a subshell environment. Changes made to the subshell environment cannot affect the shell's execution environment.
I suppose it means value changed in pipeline is local because each command in pipeline runs in its own sub-shell. Like following,
value='1'
echo "Before pipe, ${value}"
value='2' | echo "${value}" | value='3' | echo "In another pipe, ${value}"
echo "After pipe, ${value}"
Before pipe, 1
In another pipe, 1
After pipe, 1
I read "SHELL BUILTIN COMMANDS" in bash. But I could not find "=" as builtin command. What does "builtin commands" means here? And are there "non-builtin commands" which can affect the change globally even in pipe-line?
And if you don't mind please let me know when the new sub-shell runs except for:
(...)
pipeline |
I think that the manual is basically saying that built-in commands, such as echo, printf, read, etc. don't get any special treatment and still run within their own sub-shell, even though in principal it would be possible for the shell to determine that all of the commands in the pipeline could be run natively in the same shell.
If you ask to pipe one command into another, then sub-shells are created, no matter what is on either side of the pipe.
For example:
echo string | read foo
uses the two built-ins, echo and read but the variable $foo ceases to exist after the pipeline finishes.
Related
I have this command sequence that I'm having trouble understanding:
[me#mine ~]$ (echo 'test'; cat) | bash
echo $?
1
echo 'this is the new shell'
this is the new shell
exit
[me#mine ~]$
As far as I can understand, here is what happens:
A pipe is created.
stdout of echo 'test' is sent to the pipe.
bash receives 'test' on stdin.
echo $? returns 1, which is what happens when you run test without args.
cat runs.
It is copying stdin to stdout.
stdout is sent to the pipe.
bash will execute whatever you type in, but stderr won't get printed to the screen (we used |, not |&).
I have three questions:
It looks like, even though we run two commands, we use the same pipe and bash process for both commands. Is that the case?
Where do the prompts go?
When something like cat uses stdin, does it take exclusive ownership of stdin as long as the shell runs, or can other things use it?
I suspect I'm missing some detail with ttys, but I'm not sure. Any help or details or man excerpt appreciated!
So...
Yes, there's a single pipe sending commands to a single instance of bash. Note:
$ echo 'date "+%T hello $$"; sleep 1; date "+%T world $$"' | bash
22:18:52 hello 72628
22:18:53 world 72628
There are no prompts. From the man page:
An interactive shell is one started without non-option arguments (unless -s is specified) and without the -c option whose standard input and error are both connected to terminals. PS1 is set and $- includes i if bash is interactive.
So a pipe is not an interactive shell, and therefore has no prompt.
Stdin and stdout can only connect to one thing at a time. cat will take stdin from the process that ran it (for example, your interactive shell) and send its stdout through the pipe to bash. If you need multiple things to be able to submit to the stdin of that cat, consider using a named pipe.
Does that cover it?
I'm looking at https://stackoverflow.com/a/10225050/1737158
And in same Q there is an answer with timeout command but it's not in all OSes, so I want to avoid it.
What I try to do is:
demo="$(top)" &
TASK_PID=$!
sleep 3
echo "TASK_PID: $TASK_PID"
echo "demo: $demo"
And I expect to have nothing in $demo variable while top command never ends.
Now I get an empty result. Which is "acceptable" but when i re-use the same thing with the command which should return value, I still get an empty result, which is not ok. E.g.:
demo="$(uptime)" &
TASK_PID=$!
sleep 3
echo "TASK_PID: $TASK_PID"
echo "demo: $demo"
This should return uptime result but it doesn't. I also tried to kill the process by TASK_PID but I always get. If a command fails, I expect to have stderr captures somehow. It can be in different variable but it has to be captured and not leaked out.
What happens when you execute var=$(cmd) &
Let's start by noting that the simple command in bash has the form:
[variable assignments] [command] [redirections]
for example
$ demo=$(echo 313) declare -p demo
declare -x demo="313"
According to the manual:
[..] the text after the = in each variable assignment undergoes tilde expansion, parameter expansion, command substitution, arithmetic expansion, and quote removal before being assigned to the variable.
Also, after the [command] above is expanded, the first word is taken to be the name of the command, but:
If no command name results, the variable assignments affect the current shell environment. Otherwise, the variables are added to the environment of the executed command and do not affect the current shell environment.
So, as expected, when demo=$(cmd) is run, the result of $(..) command substitution is assigned to the demo variable in the current shell.
Another point to note is related to the background operator &. It operates on the so called lists, which are sequences of one or more pipelines. Also:
If a command is terminated by the control operator &, the shell executes the command asynchronously in a subshell. This is known as executing the command in the background.
Finally, when you say:
$ demo=$(top) &
# ^^^^^^^^^^^ simple command, consisting ONLY of variable assignment
that simple command is executed in a subshell (call it s1), inside which $(top) is executed in another subshell (call it s2), the result of this command substitution is assigned to variable demo inside the shell s1. Since no commands are given, after variable assignment, s1 terminates, but the parent shell never receives the variables set in child (s1).
Communicating with a background process
If you're looking for a reliable way to communicate with the process run asynchronously, you might consider coprocesses in bash, or named pipes (FIFO) in other POSIX environments.
Coprocess setup is simpler, since coproc will setup pipes for you, but note you might not reliably read them if process is terminated before writing any output.
#!/bin/bash
coproc top -b -n3
cat <&${COPROC[0]}
FIFO setup would look something like this:
#!/bin/bash
# fifo setup/clean-up
tmp=$(mktemp -td)
mkfifo "$tmp/out"
trap 'rm -rf "$tmp"' EXIT
# bg job, terminates after 3s
top -b >"$tmp/out" -n3 &
# read the output
cat "$tmp/out"
but note, if a FIFO is opened in blocking mode, the writer won't be able to write to it until someone opens it for reading (and starts reading).
Killing after timeout
How you'll kill the background process depends on what setup you've used, but for a simple coproc case above:
#!/bin/bash
coproc top -b
sleep 3
kill -INT "$COPROC_PID"
cat <&${COPROC[0]}
Say I start with the following statement, which echo-s a string into the ether:
$ echo "foo" 1>/dev/null
I then submit the following pipeline:
$ echo "foo" | cat -e - 1>/dev/null
I then leave the process out:
$ echo "foo" | 1>/dev/null
Why is this not returning an error message? The documentation on bash and piping doesn't seem to make direct mention of may be the cause. Is there an EOF sent before the first read from echo (or whatever the process is, which is running upstream of the pipe)?
A shell simple command is not required to have a command name. For a command without a command-name:
variable assignments apply to the current execution environment. The following will set two variables to argument values:
arg1=$1 arg3=$3
redirections occur in a subshell, but the subshell doesn't do anything other than initialize the redirect. The following will truncate or create the indicated file (if you have appropriate permissions):
>/file/to/empty
However, a command must have at least one word. A completely empty command is a syntax error (which is why it is occasionally necessary to use :).
Answer summarized from Posix XCU§2.9.1
When I'm looking at bash script code, I sometimes see | and sometimes see ||, but I don't know which is preferable.
I'm trying to do something like ..
set -e;
ret=0 && { which ansible || ret=$?; }
if [[ ${ret} -ne 0 ]]; then
# install ansible here
fi
Please advise which OR operator is preferred in this scenario.
| isn't an OR operator at all. You could use ||, though:
which ansible || {
true # put your code to install ansible here
}
This is equivalent to an if:
if ! which ansible; then
true # put your code to install ansible here
fi
By the way -- consider making a habit of using type (a shell builtin) rather than which (an external command). type is both faster and has a better understanding of shell behavior: If you have an ansible command that's provided by, say, a shell function invoking the real command, which won't know that it's there, but type will correctly detect it as available.
There is a big difference between using a single pipe (pipe output from one command to be used as input for the next command) and a process control OR (double pipe).
cat /etc/issue | less
This runs the cat command on the /etc/issue file, and instead of immediately sending the output to stdout it is piped to be the input for the less command. Yes, this isn't a great example, since you could instead simply do less /etc/issue - but at least you can see how it works
touch /etc/testing || echo Did not work
For this one, the touch command is run, or attempted to run. If it has a non-zero exit status, then the double pipe OR kicks in, and tries to execute the echo command. If the touch command worked, then whatever the other choice is (our echo command in this case) is never attempted...
This shell script behaves as expected.
trap 'echo exit' EXIT
foo()
{
exit
}
echo begin
foo
echo end
Here is the output.
$ sh foo.sh
begin
exit
This shows that the script exits while executing foo.
Now see the following script.
trap 'echo exit' EXIT
foo()
{
exit
}
echo begin
foo | cat
echo end
The only difference here is that the output of foo is being piped into `cat. Now the output looks like the following.
begin
end
exit
This shows that the script does not exit while executing foo because end is printed.
I believe this happens because in bash a pipeline causes a subshell to be opened, so foo | cat is equivalent to (foo) | cat.
Is this behaviour guaranteed in any POSIX shell? I could not find anything in the POSIX standard at http://pubs.opengroup.org/onlinepubs/9699919799/ that implies that a pipeline must lead to a subshell. Can someone confirm if this behaviour can be relied upon?
In 2.12 Shell Execution Environment you find this quote:
A subshell environment shall be created as a duplicate of the shell environment, except that signal traps that are not being ignored shall be set to the default action. Changes made to the subshell environment shall not affect the shell environment. Command substitution, commands that are grouped with parentheses, and asynchronous lists shall be executed in a subshell environment. Additionally, each command of a multi-command pipeline is in a subshell environment; as an extension, however, any or all commands in a pipeline may be executed in the current environment. All other commands shall be executed in the current shell environment.
Where the key sentence for this question is
Additionally, each command of a multi-command pipeline is in a subshell environment; as an extension, however, any or all commands in a pipeline may be executed in the current environment
So without the extension (which bash uses for things like lastpipe and, I thought, for the first element in a pipeline as well but apparently not or at least not always) it looks like you can assume there will be a subshell for each part of the pipeline but the exception means you can't quite count on that.