I'm trying to execute a shell script with systemd.
If I run my script from the bash everything works fine. But if I run the same script via systemd it never finishes. The command where it seems to hang is:
random="$(LC_ALL=C tr -cd '[:alnum:]' < /dev/urandom | fold -w128 | head -n1)"
If I'm replacing this line with random="1234" it also runs with systemd. I guess the 'hanging' command is the tr - its process never finishes.
And this is the systemd unit file I'm using:
[Unit]
Description=my script
[Service]
Type=forking
Restart=on-failure
ExecStart=/mypath/script.sh start
ExecStop=/bin/kill $MAINPID
ExecStopPost=/bin/rm -f /mypath/RUNNING_PID
[Install]
WantedBy=multi-user.target
Edit: Made the explanation clearer and added new information.
Short answer: systemd is filtering SIGPIPEs, so
set IgnoreSIGPIPE=false under [Service] in the .service file. From the systemd.exec manual:
IgnoreSIGPIPE=
Takes a boolean argument. If true, causes
SIGPIPE to be ignored in the executed process.
Defaults to true because SIGPIPE generally is
useful only in shell pipelines.
Long explanation:
random="$(LC_ALL=C tr -cd '[:alnum:]' < /dev/urandom | fold -w128 |
head -n1)"
When the head command exits after the first newline received from fold, its open file descriptors are closed.
Thus when the fold command later tries to write to the pipe, it will receive a SIGPIPE signal (as described in pipe(7)).
The default action for this signal is a termination of the process.
That mechanism repeats towards the start of the pipeline, so the fold command would be terminated next, and eventually the tr command.
However, when the pipeline is run under systemd, systemd sets the default action for SIGPIPE to SIG_IGN, which makes the processes in the pipeline ignore the signal.
When fold (and the other components of the pipeline) inherits that signal handler, it won't be notified (killed) via SIGPIPE when writing to the now closed output pipe (Instead each write to the pipe will return an EPIPE error).
As fold command does not check the return value of those write calls (at least not in coreutils-8.26), this makes fold continue reading from the pipe (stdin) and to write to the closed pipe (stdout), even though an error is reported each time.
So fold keeps its input pipe from tr open, and tr will happily write to the output pipe feeding fold forever.
Related
I am looking for an alternative to something like ssh user#node1 uptime && ssh user#node2 uptime, where both of the SSH-commands are run simultaneosly. As they are both blocking until the command returns, && and ; between them don't work.
My goal is to run infinite while loops on both nodes via SSH. So the first one would never return, and the second one would never be run. I would then like to save the output after terminating the loops with Ctrl+C to a log-file and read that one via Python.
Is there an easy solution to this?
Thanks in advance!
Capturing SSH output
On the one hand, you need to capture the ssh output/error and store it into a file so that you can process it afterwards with Python. To this purpose you can:
1- Store output and error directly into a file
ssh user#node cmd 2>&1 > session.log
2- Show output/error in the console while storing it into a file (I would recommend this one)
ssh user#node cmd 2>&1 | tee session.log
Check this for further information about the tee command.
Running commands in parallel
On the other hand, you want to run both commands in parallel and block the current bash process. You can achieve this by:
1- Blocking the current bash process until their childs are done.
cmd1 & ; cmd2 & ; wait
Check this for further information about the wait command.
2- Spawning the child processes and freeing the current bash process. Notice that the processes will be kept alive although the main process ends.
nohup cmd & ; nohup cmd &
The whole thing
I would recommend combining both approaches using tee (so you can still see the ssh outputs on your terminal) and blocking the current process until everything is done (so that when you kill the main process all the processes are killed too).
ssh user#node1 uptime 2>&1 | tee session1.log & ; ssh user#node2 uptime 2>&1 | tee session2.log & ; wait
I have the following script which monitors the /tmp directory indefinitely and if there are any operations with files in this directory, then file name is read by while loop and first a character in file name is replaced with b character and this modified file name is logged to test.log file:
#!/bin/bash
trap ':' HUP
trap 'kill $(jobs -p)' EXIT
/usr/local/bin/inotifywait -q -m /tmp --format %f |
while IFS= read -r filename; do
echo "$filename" | sed 's/a/b/' > test.log
done
This is simplified version of the actual script. I also have a Sys-V type init script for the script above and as I would like to stay LSB compliant, my init script has force-reload(Causes the configuration to be reloaded if the service supports this. Otherwise, the service is restarted.) option which sends the HUP signal to script. Now before executing the force-reload, which executes killproc -HUP test.sh, the output of pstree is following:
# pstree -Ap 4424
test.sh(4424)-+-inotifywait(4425)
`-test.sh(4426)
#
After executing the strace killproc -HUP test.sh the child shell is terminated:
# pstree -Ap 4424
test.sh(4424)---inotifywait(4425)
#
According to strace, killproc sent SIGHUP to processes 4424 and 4426, but only the latter was terminated.
What is the point of this child-shell with PID 4426 in my example, i.e why is it created in the first place? In addition, is there a way to ignore HUP signal?
Pipeline commands are run in a subshell
The first part of your question is explained by the mechanism through which a shell (in this case Bash) runs commands in a pipeline.
A pipe is a FIFO (first in, first out) one-way inter-process communication (IPC) channel: it allows bytes to be written at one end (the write-only end) and read from the other (read-only end) without needing to read from or write to a physical filesystem.
A pipeline allows two different commands to communicate with each other through an anonymous or unnamed (i.e., has no entry in the filesystem) pipe.
When a simple command is executed by a shell, the command is run in a child process of the shell. If no job control is used, control of the terminal is regained by the shell when the child process terminates.
When two commands are run in a pipeline, both commands in the pipeline are executed as two separate child processes which run concurrently.
In Unix systems, pipes are created using the pipe(2) system call, which creates a new pipe and returns a pair of file descriptors with one referring to the read end and the other to the write end of the pipe.
With Bash on a GNU/Linux system, the clone(2) system call is used to create the sub-processes. This allows the child process to share the table of file descriptors with its parent process so that both child sub-processes inherit the file descriptor of the anonymous pipe so that one can read to it and the other can write to it.
In your case, the inotifywait command gets a PID of 4425 and writes to the write-only end of the pipe by connecting its stdout to the file descriptor of the write end.
At the same time, the right hand side of the pipe command gets the PID, 4426 and its stdin file descriptor is set to that of the read-only end of the pipe. Since the subshell for the right hand side of the pipe isn’t an external command, the name to represent the child process is the same as that of its parent, test.sh.
For more info, see man 7 pipe and the following links:
Anonymous pipe, Wikipedia article
Unix Pipeline, Wikipedia article
Signal handling
It took me ages (a couple of hours of research, in fact) to figure out why the trap for the SIGHUP signal wasn’t being ignored.
All my research indicated that child process created by a clone(2) system call should also be able to share the table of signal handlers of the parent process.
The Bash man page also states that
Command substitution, commands grouped with parentheses, and asynchronous commands are invoked in a subshell environment that is a duplicate of the shell environment, except that traps caught by the shell are reset to the values that the shell inherited from its parent at invocation.
It later states that
Signals ignored upon entry to the shell cannot be trapped or reset. Trapped signals that are not being ignored are reset to their original values in a subshell or subshell environment when one is created.
This indicates that subshells do not inherit signal handlers that are not ignored. As I understood it, your trap ':' HUP line meant that the SIGHUP signal was (effectively) being ignored (since the : builtin does nothing except return success) – and should in turn be ignored by the pipeline’s subshell.
However, I eventually came across the description of the trap builtin in the Bash man page which defines what Bash means by ignore:
If arg is the null string the signal specified by each sigspec is ignored by the shell and by the commands it invokes.
Simply changing the trap command to trap '' HUP ensures that the SIGHUP signal is ignored, for the script itself – and any subshells.
I have a bash script with a loop that calls a hard calculation routine every iteration. I use the results from every calculation as input to the next. I need make bash stop the script reading until every calculation is finished.
for i in $(cat calculation-list.txt)
do
./calculation
(other commands)
done
I know the sleep program, and i used to use it, but now the time of the calculations varies greatly.
Thanks for any help you can give.
P.s>
The "./calculation" is another program, and a subprocess is opened. Then the script passes instantly to next step, but I get an error in the calculation because the last is not finished yet.
If your calculation daemon will work with a precreated empty logfile, then the inotify-tools package might serve:
touch $logfile
inotifywait -qqe close $logfile & ipid=$!
./calculation
wait $ipid
(edit: stripped a stray semicolon)
if it closes the file just once.
If it's doing an open/write/close loop, perhaps you can mod the daemon process to wrap some other filesystem event around the execution? `
#!/bin/sh
# Uglier, but handles logfile being closed multiple times before exit:
# Have the ./calculation start this shell script, perhaps by substituting
# this for the program it's starting
trap 'echo >closed-on-calculation-exit' 0 1 2 3 15
./real-calculation-daemon-program
Well, guys, I've solved my problem with a different approach. When the calculation is finished a logfile is created. I wrote then a simple until loop with a sleep command. Although this is very ugly, it works for me and it's enough.
for i in $(cat calculation-list.txt)
do
(calculations routine)
until [[ -f $logfile ]]; do
sleep 60
done
(other commands)
done
Easy. Get the process ID (PID) via some awk magic and then use wait too wait for that PID to end. Here are the details on wait from the advanced Bash scripting guide:
Suspend script execution until all jobs running in background have
terminated, or until the job number or process ID specified as an
option terminates. Returns the exit status of waited-for command.
You may use the wait command to prevent a script from exiting before a
background job finishes executing (this would create a dreaded orphan
process).
And using it within your code should work like this:
for i in $(cat calculation-list.txt)
do
./calculation >/dev/null 2>&1 & CALCULATION_PID=(`jobs -l | awk '{print $2}'`);
wait ${CALCULATION_PID}
(other commands)
done
I'm using bash, and as I understand it, exec followed by a command is supposed to replace the shell without creating a new process. For example,
exec echo hello
has the appearance of printing "hello" and then immediately exiting, because after echo is done, the shell process isn't there to return to anymore.
If I put this as part of a pipeline - for instance,
exec echo hello | sed 's/hell/heck/'
or
echo hello | exec sed 's/hell/heck/'
my expectation is that, similarly, the shell would terminate as a result of its process being replaced away. This is not what happens in reality, though - both the commands above print "hecko" and return to the shell normally, just as if the word "exec" wasn't there. Why is this?
There is sentence in bash manual:
Each command in a pipeline is executed as a separate process (i.e., in
a subshell).
So in both examples two processes are spawned by the pipeline first and 'exec' is executed inside one of spawned process - without impact on shell executing the pipeline.
I have a script that creates a FIFO and launches a program that writes output to the FIFO. I then read and parse the output until the program exits.
MYFIFO=/tmp/myfifo.$$
mkfifo "$MYFIFO"
MYFD=3
eval "exec $MYFD<> $MYFIFO"
external_program >&"$MYFD" 2>&"$MYFD" &
EXT_PID=$!
while kill -0 "$EXT_PID" ; do
read -t 1 LINE <&"$MYFD"
# Do stuff with $LINE
done
This works fine reading input while the program is still running, but it looks like the timeout to read is ignored, and read call hangs after the external program exits.
I've used read with a timeout successfully in other scripts, and a simple test script that leaves out the external program times out correctly. What am I doing wrong here?
EDIT: It looks like read -t functions as expected when I run my script from the command line, but when I run it as part of an xcodebuild build process, the timeout does not function. What is different about these two environments?
I don't think -t will work with redirection.
From the man page here:
-t timeout
Cause read to time out and return failure if a complete line
of input is not read within timeout seconds. This option has no
effect if read is not reading input from the terminal or a pipe.