Bash can use wait to wait for the processes it started directly. However, if the process forks a child, and then execs bash (that is, parent turns into bash), the newly exec'd Bash process cannot wait for the "inherited" child. Here is the minimal reproduction:
#/bin/bash
sleep inf &
pid=$!
exec bash -c "wait $pid;"'echo This shell is $$; sleep inf'
which gives this output:
$ bash test.sh
bash: wait: pid 53984 is not a child of this shell
This shell is 53983
The pstree, however, shows that the child pid is indeed the child of the shell:
$ pstree -p 53983
bash(53983)─┬─sleep(53984)
└─sleep(53985)
It seems that Bash tracks the spawned processes internally, and consults this list rather than calling waitpid(2) directly (zsh has the same problem, but ksh works as expected).
Is there any way to workaround this behavior, and have Bash add the "inherited" child to its internal structures?
I could not reproduce it, but I wrote a script that shows that this behaviour is consistent in at least 6 well maintained shells (including the mentioned ksh).
As you can see in the report, all shells won't list the first sleep job in the replaced shell, only the new one created after the exec call.
When invoking exec the new shell does not inherit the job list managed by the replaced one. It seems an intended behaviour but I could not find it anywhere in the POSIX specification.
Related
I am trying to study the about shells and I was exploring about how variables are shared among child and parent shells. I am able to open a child shell in the same tab as the parent shell. However, at this time, I am not being able to make changes in the variables in the parent shell since the child shell is running. So I needed to open the child shell in a new tab instead, so that I can work on both the child and parent shells simultaneously. Is this even possible?
You can stop the child shell then tell it to run in the background with "bg". The echo $$ command returns the shells pid, and the kill -19 sends the stop signal to the pid.
$ echo $$
19863
$ kill -19 19863
[1]+ Stopped sh
$ bg
[1]+ sh &
$
After you've made your changes to the parent shell environment variables you can "fg" to foreground the child shell.
You can also look into "screen" and "xterm &" these are 2 other ways to launch a sub shell and easily interact with the parent shell.
I am using a coprocess inside my main parent process to spawn commands to a shell that otherwise cannot be solved (the shell that I open in the coprocess is not maintained by me and executes the "newgrp" and "exec" commands that stop me from sending commands to that shell simply from my script... So I need the coprocess to be able to execute commands in that shell from a script). So far I have been using one thread, the parent process to push commands to the coprocess but now I am in need of spawning commands from several child processes, too, because of an optimization step. The bash doc says, file descriptors are not inherited by child processes, and this is in fact true, when I opened a subshell I got the following error message from bash:
[...]/automated_integration/clif_ai_common.sh: line 396: ${!clifAi_sendCmdToCoproc_varName}: Bad file descriptor
The code that makes this message appear is as follows:
if [[ ${PARAM_NO_MOVING_VERIF_TB_TAGS} != true ]]; then
(
clifAi_log ${CLIFAI_LOGLEVEL_INFO} "" "clifAi_sanityRegression_callbackRunning" "Populating moving VERIF and TB tags in the background..."
clifAi_popVerifTags "${clifAi_sanityRegression_callbackRunning_coproc}" "${clifAi_sanityRegression_callbackRunning_wslogfile}" "${PARAM_OPTLEVEL}" "${CONST_EXCLUDE_FILTER}" "${CONST_DIR_TO_OPT}" ${clifAi_sanityRegression_callbackRunning_excludeList}
clifAi_popTbTags "${clifAi_sanityRegression_callbackRunning_coproc}" "${clifAi_sanityRegression_callbackRunning_wslogfile}"
rm -rf ${VAR_VERIFTBTAG_SEMAPHORE_FILE}
) &
fi
Bash reports the same error if I move this piece of code into a function and call it with & without the ( ), so no subshell. This is also understandable; it will still spawn a child process, regardless of running it in a subshell or not.
My question is, how can I write to the coprocess owned by the parent process from child processes, too? What is the best practice?
Many thanks in advance,
Geza Balazs
I have the following script which monitors the /tmp directory indefinitely and if there are any operations with files in this directory, then file name is read by while loop and first a character in file name is replaced with b character and this modified file name is logged to test.log file:
#!/bin/bash
trap ':' HUP
trap 'kill $(jobs -p)' EXIT
/usr/local/bin/inotifywait -q -m /tmp --format %f |
while IFS= read -r filename; do
echo "$filename" | sed 's/a/b/' > test.log
done
This is simplified version of the actual script. I also have a Sys-V type init script for the script above and as I would like to stay LSB compliant, my init script has force-reload(Causes the configuration to be reloaded if the service supports this. Otherwise, the service is restarted.) option which sends the HUP signal to script. Now before executing the force-reload, which executes killproc -HUP test.sh, the output of pstree is following:
# pstree -Ap 4424
test.sh(4424)-+-inotifywait(4425)
`-test.sh(4426)
#
After executing the strace killproc -HUP test.sh the child shell is terminated:
# pstree -Ap 4424
test.sh(4424)---inotifywait(4425)
#
According to strace, killproc sent SIGHUP to processes 4424 and 4426, but only the latter was terminated.
What is the point of this child-shell with PID 4426 in my example, i.e why is it created in the first place? In addition, is there a way to ignore HUP signal?
Pipeline commands are run in a subshell
The first part of your question is explained by the mechanism through which a shell (in this case Bash) runs commands in a pipeline.
A pipe is a FIFO (first in, first out) one-way inter-process communication (IPC) channel: it allows bytes to be written at one end (the write-only end) and read from the other (read-only end) without needing to read from or write to a physical filesystem.
A pipeline allows two different commands to communicate with each other through an anonymous or unnamed (i.e., has no entry in the filesystem) pipe.
When a simple command is executed by a shell, the command is run in a child process of the shell. If no job control is used, control of the terminal is regained by the shell when the child process terminates.
When two commands are run in a pipeline, both commands in the pipeline are executed as two separate child processes which run concurrently.
In Unix systems, pipes are created using the pipe(2) system call, which creates a new pipe and returns a pair of file descriptors with one referring to the read end and the other to the write end of the pipe.
With Bash on a GNU/Linux system, the clone(2) system call is used to create the sub-processes. This allows the child process to share the table of file descriptors with its parent process so that both child sub-processes inherit the file descriptor of the anonymous pipe so that one can read to it and the other can write to it.
In your case, the inotifywait command gets a PID of 4425 and writes to the write-only end of the pipe by connecting its stdout to the file descriptor of the write end.
At the same time, the right hand side of the pipe command gets the PID, 4426 and its stdin file descriptor is set to that of the read-only end of the pipe. Since the subshell for the right hand side of the pipe isn’t an external command, the name to represent the child process is the same as that of its parent, test.sh.
For more info, see man 7 pipe and the following links:
Anonymous pipe, Wikipedia article
Unix Pipeline, Wikipedia article
Signal handling
It took me ages (a couple of hours of research, in fact) to figure out why the trap for the SIGHUP signal wasn’t being ignored.
All my research indicated that child process created by a clone(2) system call should also be able to share the table of signal handlers of the parent process.
The Bash man page also states that
Command substitution, commands grouped with parentheses, and asynchronous commands are invoked in a subshell environment that is a duplicate of the shell environment, except that traps caught by the shell are reset to the values that the shell inherited from its parent at invocation.
It later states that
Signals ignored upon entry to the shell cannot be trapped or reset. Trapped signals that are not being ignored are reset to their original values in a subshell or subshell environment when one is created.
This indicates that subshells do not inherit signal handlers that are not ignored. As I understood it, your trap ':' HUP line meant that the SIGHUP signal was (effectively) being ignored (since the : builtin does nothing except return success) – and should in turn be ignored by the pipeline’s subshell.
However, I eventually came across the description of the trap builtin in the Bash man page which defines what Bash means by ignore:
If arg is the null string the signal specified by each sigspec is ignored by the shell and by the commands it invokes.
Simply changing the trap command to trap '' HUP ensures that the SIGHUP signal is ignored, for the script itself – and any subshells.
For example in the redis official image:
https://github.com/docker-library/redis/blob/master/2.8/docker-entrypoint.sh
#!/bin/bash
set -e
if [ "$1" = 'redis-server' ]; then
chown -R redis .
exec gosu redis "$#"
fi
exec "$#"
Why not just run the commands as usual without exec preceding them?
As #Peter Lyons says, using exec will replace the parent process, rather than have two processes running.
This is important in Docker for signals to be proxied correctly. For example, if Redis was started without exec, it will not receive a SIGTERM upon docker stop and will not get a chance to shutdown cleanly. In some cases, this can lead to data loss or zombie processes.
If you do start child processes (i.e. don't use exec), the parent process becomes responsible for handling and forwarding signals as appropriate. This is one of the reasons it's best to use supervisord or similar when running multiple processes in a container, as it will forward signals appropriately.
Without exec, the parent shell process survives and waits for the child to exit. With exec, the child process replaces the parent process entirely so when there's nothing for the parent to do after forking the child, I would consider exec slightly more precise/correct/efficient. In the grand scheme of things, I think it's probably safe to classify it as a minor optimization.
without exec
parent shell starts
parent shell forks child
child runs
child exits
parent shell exits
with exec
parent shell starts
parent shell forks child, replaces itself with child
child program runs taking over the shell's process
child exits
Think of it as an optimization like tail recursion.
If running another program is the final act of the shell script, there's not much of a need to have the shell run the program in a new process and wait for it. Using exec, the shell process replaces itself with the program.
In either case, the exit value of the shell script will be identical1. Whatever program originally called the shell script will see an exit value that is equal to the exit value of the exec`ed program (or 127 if the program cannot be found).
1 modulo corner cases such as a program doing something different depending on the name of its parent.
I am reading The TTY demystified. In the "Jobs and sessions" section there is an example of a user using an xterm:
$ cat
hello
hello
^Z
[1]+ Stopped cat
$ ls | sort
And there is a table listing the processes involved: xterm, bash (child of the xterm), and the three last processes (cat, ls and sort) all have the same PPID (parent process ID) -- they are all children of the same bash process.
Now, I know that pipelines in bash are executed in subshells. I have always thought that this subshell thing meant that there was an extra bash process for each subshell. My question is: shouldn't there be another two bash processes, both children of the first bash, and then ls would be a child of the first bash, and sort would be a child of the second bash? Is the table in the article simplified, or is my understanding of subshells wrong?
Programs are executed in child processes, but these are not subshells. The shell forks a child, redirects standard input/output/error as necessary, and then immediately calls execv() to execute the program.
For a very brief period the child process is still running bash, but we don't consider this a subshell because it's not doing any shell command processing -- that was all done in the original shell, and the child is just starting up the external program (as if via an explicit exec for commands like ls).
In the case of a pipeline, if any of the commands are shell built-ins, they run in a subshell. So if you do:
ls | read var
it will create two child processes. One child will run ls, the other will be a subshell executing read var.
Invoking an executable, whether directly or via a pipe, does not spawn a subshell. Only explicitly invoking it within a subshell (via (...), $(...), and so on) does so.