I sometimes launch long running tasks on my server and want the server to do something after those tasks finish (usually shut down). If there was only one task, I could simply type the next command into the window running the task, then bash will run it after the current one finishes. But what if there was multiple processes that I want to wait on?
In my workflow, the different tasks are running in different panes on tmux, so I cannot directly use wait since the processes I want to wait for are not child processes in one particular pane.
I have included a possible approach as an answer below.
This question's answer offers a related solution:
tail --pid=$pid -f /dev/null
However, that particular answer can only handle waiting for one process, but we can extend it using wait for multiple processes, then run our own command on completion:
tail --pid=$pid1 -f /dev/null &
tail --pid=$pid2 -f /dev/null &
tail --pid=$pid3 -f /dev/null &
tail --pid=$pid4 -f /dev/null &
wait; <your-command-here>
Related
I am looking for an alternative to something like ssh user#node1 uptime && ssh user#node2 uptime, where both of the SSH-commands are run simultaneosly. As they are both blocking until the command returns, && and ; between them don't work.
My goal is to run infinite while loops on both nodes via SSH. So the first one would never return, and the second one would never be run. I would then like to save the output after terminating the loops with Ctrl+C to a log-file and read that one via Python.
Is there an easy solution to this?
Thanks in advance!
Capturing SSH output
On the one hand, you need to capture the ssh output/error and store it into a file so that you can process it afterwards with Python. To this purpose you can:
1- Store output and error directly into a file
ssh user#node cmd 2>&1 > session.log
2- Show output/error in the console while storing it into a file (I would recommend this one)
ssh user#node cmd 2>&1 | tee session.log
Check this for further information about the tee command.
Running commands in parallel
On the other hand, you want to run both commands in parallel and block the current bash process. You can achieve this by:
1- Blocking the current bash process until their childs are done.
cmd1 & ; cmd2 & ; wait
Check this for further information about the wait command.
2- Spawning the child processes and freeing the current bash process. Notice that the processes will be kept alive although the main process ends.
nohup cmd & ; nohup cmd &
The whole thing
I would recommend combining both approaches using tee (so you can still see the ssh outputs on your terminal) and blocking the current process until everything is done (so that when you kill the main process all the processes are killed too).
ssh user#node1 uptime 2>&1 | tee session1.log & ; ssh user#node2 uptime 2>&1 | tee session2.log & ; wait
I have written a bash script to carry out some tests on my system. The tests run in the background and in parallel. The tests can take a long time and sometimes I may wish to abort the tests part way through.
If I Control+C then it aborts the parent script, but leaves the various children running. I wish to make it so that I can hit Control+C or otherwise to quit and then kill all child processes running in the background. I have a bit of code that does the job if I'm running running the background jobs directly from the terminal, but it doesn't work in my script.
I have a minimal working example.
I have tried using trap in combination with pgrep -P $$.
#!/bin/bash
trap 'kill -n 2 $(pgrep -P $$)' 2
sleep 10 &
wait
I was hoping that on hitting control+c (SIGINT) would kill everything that the script started but it actually says:
./breakTest.sh: line 1: kill: (3220) - No such process
This number changes, but doesn't seem to apply to any running processes, so I don't know where it is coming from.
I guess if the contents of the trap command get evaluated where the trap command occurs then it might explain the outcome. The 3220 pid might be for pgrep itself.
I'd appreciate some insight here
Thanks
I have found a solution using pkill. This example also deals with many child processes.
#!/bin/bash
trap 'pkill -P $$' SIGINT SIGTERM
for i in {1..10}; do
sleep 10 &
done
wait
This appears to kill all the child processes elegantly. Though I don't properly understand what the issue was with my original code, apart from sending the correct signal.
in bash whenever you you use & after a command it places that command as a background job ( this background jobs are called job_spec ) which is incremented by one until you exit that terminal session. You can use the jobs command to get the list of the background jobs running. To work with this jobs you have to use the % with the job id. The jobs command also accept other options such as jobs -p to see the proces sids of all jobs , jobs -p %JOB_SPEC to see the process of id of that particular job.
#!/usr/bin/env bash
trap 'kill -9 %1' 2
sleep 10 &
wait
or
#!/usr/bin/env bash
trap 'kill -9 $(jobs -p %1)' 2
sleep 10 &
wait
I implemented something like this few years back, you can take a look at it async bash
You can try something like the following:
pkill -TERM -P <your_parent_id_here>
I have a script which starts many processes in background and uses nohup to make sure these processes keeps on running -
nohup "./$__service_script1.pl" $__service_args < /dev/null > /var/log/$__service_name.log 2>&1 &
The problem is its important for me to make sure that processes starts in the order of their invocation. Is their a way to wait until the process has definitely started before attempting to start another process?
I tried wait, but it waits till the process is finished, I just want to make sure that process has started. Probably the simplest solution would be to put sleep for a few seconds in between processes, is there a better solution ?
Thanks
It depends on what you mean by "definitely started". If you mean that fork(2) has completed and the new process exists, then each process is started by the time nohup returns. A new process has been created.
The problem you are running into is that there is no guarantee how long the nohup'ed process gets to run before the shell returns. When the process you start is "definitely started" depends on what the process does for initialization. If you do not have source of the applications or are not able to modify them for some other reason, you will be limited to looking at their output. Many daemons will output a log message at various stages of their initialization. You can modify your script to
Look for a log file, and create an empty one if it does not exist
Open the log file for reading (at the end to avoid false messages from previous invocations), watching for the log message that indicates the process has started,
Start your process with nohup,
Wait for your log file watcher
In bash, it would look something like this might work (this code is completely untested):
log=<path to log file>
msg=<message service prints when it is ready>
svc=<path to service>
# Create log file if it does not exist
if [ ! -f "$log" ] ; then
echo > "$log"
fi
# watch for message to appear on a single line in the log file
tail -0 -f "$log" | egrep "$msg" | head -1 &
ready_pid=$!
# Start the service
nohup "$svc" < /dev/null >> "$log" 2>&1 &
# Wait for the message
wait $ready_pid
You want to start watching the log file before forking the service, because otherwise, the message might go by in the log before the script starting the service can attach to the log file.
I would like to run n processes (in my case simulations) simultaneously, using bash.
Right now this is what I'm running:
for file in $ini/SAN*.ini;
do
echo "Running $file...";
temp=$(basename $file .ini)
mosrun -G opp_run -r 0 -u Cmdenv -n ..:../../src -l ../../src/inet SAN.ini > $outputs/$temp.out;
done
Problem is, the loop only progresses to the next iteration after the simulation is done. Any suggestions? Thanks!
You should be able to run your command in the background by adding a & after it.
Should make them run in parallell, although in the background.
(Small side note: the processes will continue to run even if you abort the script, so you might want to add a trap to kill the processes if you hit for eg. ctrl-c when script is running. Look at bash manual.)
My current script looks like this:
cd ~/.wine/drive_c/
echo "test123" > foo$$.txt
wine start "C:\foo$$.txt"
wineserver -w
echo "Wine is done!"
which works fine when only one program is running in wine at a time. However if I run this a second time, before the first program is done, both scripts will wait for each others programs to exit.
This does not work:
cd ~/.wine/drive_c/
echo "test123" > foo$$.txt
$(wine start "C:\foo$$.txt") &
wait ${!}
echo "Wine is done!"
as it will exit before you close the text editor.
I need to use the start command, because I want a file to be run with its default editor/viewer.
To wait for the process started by wine to exit, you can pipe its output to another program.
On my system, the following achieves the desired effect:
wine "program.exe" | cat
echo "program.exe has finished"
If you want to wait asynchronously:
wine "program.exe" | cat & pid=$!
# ...
wait $pid
echo "program.exe has finished"
wineserver has a --wait flag which can be used to do exactly that.
However if you run multiple programs at once, it will wait for all of them to finish.
wine <program> waits until the program exits. wine start program does not.
A summary:
wine <program> starts the program and waits until it is finished. I recommend using this method.
wine start <program> starts the program and immediately exits without waiting. The program will keep running in the background.
wine start \wait <program> starts the program and waits until it is finished. This is the same behavior as wine <program>.
wineserver --wait waits until all programs and all services in Wine are finished. This command does not launch any program itself but waits for existing programs and services.
Services like services.exe, plugplay.exe, and winedevice.exe keep on running a few seconds after the last program finishes, and wineserver --wait also waits until these services exit.
Some of these services hold state and write their state (and the registry) to disk when they exit. So if you want to backup or remove your wine prefix, make sure to wait until these services have exited.
What happens is that wine just asks wineserver to start the program and exits, and I have found no good mechanism to get notifications from wineserver about the processes that it spawns.
My suggestion would be to wait for the completion of the process started by wineserver using one of the methods in How to wait for exit of non-children processes, but you need to know its PID. Possible ideas: run wineserver -f -d | grep init_thread( and get PIDs from there, but I can see no easy way to find out which is which, to avoid race conditions, and to ignore noise; or try to find your process in the output of ps, but it's ugly, and definitely not robust.
If nothing better surfaces, you might want to suggest the addition of such a feature to the Wine devs (probably as a flag to wine).