Bash script to run a detached loop that sequentially starts backgound processes - bash

I am trying to run a series of tests on a remote Linux server to which I am connecting via ssh.
I don't want to have to stay logged in the ssh session during the runs -> nohup(?)
I don't want to have to keep checking if one run is done -> for loop(?)
Because of licensing issues, I can only run a single testing process at a time -> sequential
I want to keep working while the test set is being processed -> background
Here's what I tried:
#!/usr/bin/env bash
# Assembling a list of commands to be executed sequentially
TESTRUNS="";
for i in `ls ../testSet/*`;
do
MSG="running test problem ${i##*/}";
RUN="mySequentialCommand $i > results/${i##*/} 2> /dev/null;";
TESTRUNS=$TESTRUNS"echo $MSG; $RUN";
done
#run commands with nohup to be able to log out of ssh session
nohup eval $TESTRUNS &
But it looks like nohup doesn't fare too well with eval.
Any thoughts?

nohup is needed if you want your scripts to run even after the shell is closed. so yes.
and the & is not necessary in RUN since you execute the command with &.
Now your script builds the command in the for loop, but doesn't execute it. It means you'll have only the last file running. If you want to run all of the files, you need to execute the nohup command as part of your loop. BUT - you can't run the commands with & because this will run commands in the background and return to the script, which will execute the next item in the loop. Eventually this would run all files in parallel.
Move the nohup eval $TESTRUNS inside the for loop, but again, you can't run it with &. What you need to do is run the script itself with nohup, and the script will loop through all files one at a time, in the background, even after the shell is closed.

You could take a look at screen, an alternative for nohup with additional features. I will replace your testscript with while [ 1 ]; do printf "."; sleep 5; done for testing the screen solution.
The commands screen -ls are optional, just showing what is going on.
prompt> screen -ls
No Sockets found in /var/run/uscreens/S-notroot.
prompt> screen
prompt> screen -ls
prompt> while [ 1 ]; do printf "."; sleep 5; done
# You don't get a prompt. Use "CTRL-a d" to detach from your current screen
prompt> screen -ls
# do some work
# connect to screen with batch running
prompt> screen -r
# Press ^C to terminate the batch (script printing dots)
prompt> screen -ls
prompt> exit
prompt> screen -ls
Google for screenrc to see how you can customize the interface.
You can change your script into something like
#!/usr/bin/env bash
# Assembling a list of commands to be executed sequentially
for i in ../testSet/*; do
do
echo "Running test problem ${i##*/}"
mySequentialCommand $i > results/${i##*/} 2> /dev/null
done
Above script can be started with nohup scriptname & when you do not use screen or simple scriptname inside the screen.

Related

Restart a crashed screen

I have a simple bash script, which uses the inotifywait command and based on the modification that is done, it run a rsync command. This is done in an infinite loop.
The script is run in a "screen" session, but unfortunately it crashes from time to time (no error as to why).
I've been searching for a way to "monitor" that specific screen/script and restart it when it crashes, but I'm struggling to find a solution to that.
The script is run "screen -AmdS script ./script.sh".
Script example:
#!/usr/bin/bash
while inotifywait --exclude "(.log)" -r -e modify,create,delete /home/backups/
do
rsync -avz -e --update --rsh='ssh -pxxxxx' /home/backups/* user#target:/location/ --delete --force
done
So my question basically is, is there a way to monitor the 'screen' session and if it stops to start a new one or is there another way to keep this script running constantly and restart it (possibly not utilizing screen).
You can rerun your failing script in a loop until it succeeds. You can do this with
screen -AmdS script bash -c 'until ./script.sh; do echo "Crashed with exit code $?. Restaring."; sleep 1; done'
Every time your script fails, this will print that your script crashed and what exit code it had, pause for 1 second, and then rerun your script. As soon as your script succeeds (i.e., ./script.sh terminates with a non-zero exit code), then the loop will terminate.
Note that if your script never succeeds, this is an infinite loop.
Edit: attempt to be clearer
You could just issue
until ./script.sh ; do echo crashed ; sleep 1 ; done
in your terminal. That will restart the script whenever it exits with non-zero result.

How to run "script -a xxx.txt" properly in a shell script?

I have a shell script and I want the session text to be saved automatically every time the script runs, so I included the command "script -a output.txt" at the beginning of my script. However, the script stops running after this line of code, which only displays a "bash-3.2$" on the screen and won't go on. Any ideas?
Thanks in advance!
The problem is script starts a separate sub-shell than the one that is running the actual script, to club them together. Use the -c flag in script
-c, --command command
Run the command rather than an interactive shell. This makes
it easy for a script to capture the output of a program that
behaves differently when its stdout is not a tty.
Just do,
script -c 'bash yourScript.sh' -a output.txt

How can I tell if a script was run in the background and with nohup?

Ive got a script that takes a quite a long time to run, as it has to handle many thousands of files. I want to make this script as fool proof as possible. To this end, I want to check if the user ran the script using nohup and '&'. E.x.
me#myHost:/home/me/bin $ nohup doAlotOfStuff.sh &. I want to make 100% sure the script was run with nohup and '&', because its a very painful recovery process if the script dies in the middle for whatever reason.
How can I check those two key paramaters inside the script itself? and if they are missing, how can I stop the script before it gets any farther, and complain to the user that they ran the script wrong? Better yet, is there way I can force the script to run in nohup &?
Edit: the server enviornment is AIX 7.1
The ps utility can get the process state. The process state code will contain the character + when running in foreground. Absence of + means code is running in background.
However, it will be hard to tell whether the background script was invoked using nohup. It's also almost impossible to rely on the presence of nohup.out as output can be redirected by user elsewhere at will.
There are 2 ways to accomplish what you want to do. Either bail out and warn the user or automatically restart the script in background.
#!/bin/bash
local mypid=$$
if [[ $(ps -o stat= -p $mypid) =~ "+" ]]; then
echo Running in foreground.
exec nohup $0 "$#" &
exit
fi
# the rest of the script
...
In this code, if the process has a state code +, it will print a warning then restart the process in background. If the process was started in the background, it will just proceed to the rest of the code.
If you prefer to bailout and just warn the user, you can remove the exec line. Note that the exit is not needed after exec. I left it there just in case you choose to remove the exec line.
One good way to find if a script is logging to nohup, is to first check that the nohup.out exists, and then to echo to it and ensure that you can read it there. For example:
echo "complextag"
if ( $(cat nohup.out | grep "complextag" ) != "complextag" );then
# various commands complaining to the user, then exiting
fi
This works because if the script's stdout is going to nohup.out, where they should be going (or whatever out file you specified), then when you echo that phrase, it should be appended to the file nohup.out. If it doesn't appear there, then the script was nut run using nohup and you can scold them, perhaps by using a wall command on a temporary broadcast file. (if you want me to elaborate on that I can).
As for being run in the background, if it's not running you should know by checking nohup.

Running processes simultaneously, Bash

I would like to run n processes (in my case simulations) simultaneously, using bash.
Right now this is what I'm running:
for file in $ini/SAN*.ini;
do
echo "Running $file...";
temp=$(basename $file .ini)
mosrun -G opp_run -r 0 -u Cmdenv -n ..:../../src -l ../../src/inet SAN.ini > $outputs/$temp.out;
done
Problem is, the loop only progresses to the next iteration after the simulation is done. Any suggestions? Thanks!
You should be able to run your command in the background by adding a & after it.
Should make them run in parallell, although in the background.
(Small side note: the processes will continue to run even if you abort the script, so you might want to add a trap to kill the processes if you hit for eg. ctrl-c when script is running. Look at bash manual.)

Why is this command in the bash script not running in the background?

I have a bash script that includes the following lines:
.
.
if [ -z "$(pgrep mplayer)" ]; then
/usr/bin/mplayer -slave -input file=/home/administrator/files/mplayer-control.pipe http:/www.musicserveraddress.com/ &
fi
.
.
Other things to execute
.
exit
what happens is that mplayer connects to the streamingserver and start playing the stream. However, the script never moves on. I added an ampersand to move this process to the background so that the script should continue to run and then exit itself (keeping the audio stream playing).
How should I do to achieve that?
Thanks in advance/J
Edit: It runs as planned when I run the script from the command line, but it is intended to be run as a cron job (and the pgrep is intended to start mplayer only if it has crashed since last cron job). When run as a cron job, nothing happens...
Try with a nohup at the beginning of the command :
nohup /usr/bin/mplayer -slave -input file=/home/administrator/files/mplayer-control.pipe http:/www.musicserveraddress.com/ &
The command you've used works perfectly fine for me. The only thing that can confuse people is that you don't get the bash prompt when the script finishes, but it's actually there, try pressing Enter.
As you want the player to be controlled via a pipe and be in the background, I'd recommend to redirect standard streams from the console as well and send all the output to a logfile:
(
exec </dev/null
exec >/dev/null
exec 2>/dev/null
umask 0
cd /
exec setsid mplayer -slave -idle -input /home/user/control.pipe http://server.com > /var/log/mplayer.log 2>&1
) &
You may want to use setsid as well. That worked for me.

Resources