How to retry a command in Bash? - bash

Question

I have a command that should take less than 1 minute to execute, but for some reason has an extremely long built-in timeout mechanism. I want some bash that does the following:
success = False
try(my_command)
while(!(success))
wait 1 min
if my command not finished
retry(my_command)
else
success = True
end while
How can I do this in Bash?

Answer 1

Look at the GNU timeout command. This kills the process if it has not completed in a given time; you'd simply wrap a loop around this to wait for the timeout to complete successfully, with delays between retries as appropriate, etc.
while timeout -k 70 60 -- my_command; [ $? = 124 ]
do sleep 2 # Pause before retry
done
If you must do it in pure bash (which is not really feasible - bash uses lots of other commands), then you are in for a world of pain and frustration with signal handlers and all sorts of issues.
Please expand on your answer a little. -k 70 is --kill-after= 70 seconds, 124 exit on timeout; what is the 60?
The linked documentation does explain the command; I don't really plan to repeat it all here. The synopsis is timeout [options] duration command [arg]...; one of the options is -k duration. The -k duration says "if the command does not die after the SIGTERM signal is sent at 60 seconds, send a SIGKILL signal at 70 seconds" (and the command should die then). There are a number of documented exit statuses; 124 indicates that the command timed out; 137 that it died after being sent the SIGKILL signal, and so on. You can't tell if the command itself exits with one of the documented statuses.

Answer 2

I found a script from:
http://fahdshariff.blogspot.com/2014/02/retrying-commands-in-shell-scripts.html
#!/bin/bash
# Retries a command on failure.
# $1 - the max number of attempts
# $2... - the command to run
retry() {
local -r -i max_attempts="$1"; shift
local -i attempt_num=1
until "$#"
do
if ((attempt_num==max_attempts))
then
echo "Attempt $attempt_num failed and there are no more attempts left!"
return 1
else
echo "Attempt $attempt_num failed! Trying again in $attempt_num seconds..."
sleep $((attempt_num++))
fi
done
}
# example usage:
retry 5 ls -ltr foo

Answer 3

I liked #Jonathan's answer, but tried to make it more straight forward for future use:
until timeout 1 sleep 2
do
echo "Happening after 1s of sleep"
done

Answer 4

Adapting #Shin's answer to use kill -0 rather than jobs so that this should work even with classic Bourne shell, and allow for other background jobs. You may have to experiment with kill and wait depending on how my_command responds to those.
while true ; do
my_command &
sleep 60
if kill -0 $! 2>/dev/null; then
# Job took too long
kill $!
else
echo "Job is done"
# Reap exit status
wait $!
break
fi
done

Answer 5

You can run a command and retain control with the & background operator. Run your command in the background, sleep for as long as you wish in the foreground, and then, if the background job hasn't terminated, kill it and start over.

Answer 6

while true ; do
my_command &
sleep 60
if [[ $(jobs -r) == "" ]] ; then
echo "Job is done"
break
fi
# Job took too long
kill -9 $!
done

Answer 7

# Retries a given command given number of times and outputs to given variable
# $1 : Command to be passed : handles both simple, piped commands
# $2 : Final output of the command(if successfull)
# $3 : Number of retrial attempts[Default 5]
function retry_function() {
echo "Command to be executed : $1"
echo "Final output variable : $2"
echo "Total trials [Default:5] : $3"
counter=${3:-5}
local _my_output_=$2 #make sure passed variable is not same as this
i=1
while [ $i -le $counter ]; do
local my_result=$(eval "$1")
# this tests if output variable is populated and accordingly retries,
# Not possible to provide error status/logs(STDIN,STDERR)-owing to subshell execution of command
# if error logs are needed, execute the same code, outside function in same shell
if test -z "$my_result"
then
echo "Trial[$i/$counter]: Execution failed"
else
echo "Trial[$i/$counter]: Successfull execution"
eval $_my_output_="'$my_result'"
break
fi
let i+=1
done
}
retry_function "ping -c 4 google.com | grep \"min/avg/max\" | awk -F\"/\" '{print \$5}'" avg_rtt_time
echo $avg_rtt_time
- To pass in a lengthy command, pass a method echoing the content. Take care of method expansion accordingly in a subshell at appropriate place.
- Wait time can be added too - just before the increment!
- For a complex command, youll have to take care of stringifying it(Good luck)