Retry a Bash command with timeout - bash

Question

How to retry a bash command until its status is ok or until a timeout is reached?
My best shot (I'm looking for something simpler):
NEXT_WAIT_TIME=0
COMMAND_STATUS=1
until [ $COMMAND_STATUS -eq 0 || $NEXT_WAIT_TIME -eq 4 ]; do
command
COMMAND_STATUS=$?
sleep $NEXT_WAIT_TIME
let NEXT_WAIT_TIME=NEXT_WAIT_TIME+1
done

Answer 1

You can simplify things a bit by putting command right in the test and doing increments a bit differently. Otherwise the script looks fine:
NEXT_WAIT_TIME=0
until [ $NEXT_WAIT_TIME -eq 5 ] || command; do
sleep $(( NEXT_WAIT_TIME++ ))
done
[ $NEXT_WAIT_TIME -lt 5 ]

Answer 2

One line and shortest, and maybe the best approach:
timeout 12h bash -c 'until ssh root#mynewvm; do sleep 10; done'
Credited by http://jeromebelleman.gitlab.io/posts/devops/until/

Answer 3

retry fuction is from:
http://fahdshariff.blogspot.com/2014/02/retrying-commands-in-shell-scripts.html
#!/bin/bash
# Retries a command on failure.
# $1 - the max number of attempts
# $2... - the command to run
retry() {
local -r -i max_attempts="$1"; shift
local -r cmd="$#"
local -i attempt_num=1
until $cmd
do
if (( attempt_num == max_attempts ))
then
echo "Attempt $attempt_num failed and there are no more attempts left!"
return 1
else
echo "Attempt $attempt_num failed! Trying again in $attempt_num seconds..."
sleep $(( attempt_num++ ))
fi
done
}
# example usage:
retry 5 ls -ltr foo
if you want to retry an function in your script, you should do like this:
# example usage:
foo()
{
#whatever you want do.
}
declare -fxr foo
retry 3 timeout 60 bash -ce 'foo'

Answer 4

Put together some tools.
retry: https://github.com/kadwanev/retry
timeout: http://manpages.courier-mta.org/htmlman1/timeout.1.html
Then see the magic
retry timeout 3 ping google.com
PING google.com (173.194.123.97): 56 data bytes
64 bytes from 173.194.123.97: icmp_seq=0 ttl=55 time=13.982 ms
64 bytes from 173.194.123.97: icmp_seq=1 ttl=55 time=44.857 ms
64 bytes from 173.194.123.97: icmp_seq=2 ttl=55 time=64.187 ms
Before retry #1: sleeping 0.3 seconds
PING google.com (173.194.123.103): 56 data bytes
64 bytes from 173.194.123.103: icmp_seq=0 ttl=55 time=56.549 ms
64 bytes from 173.194.123.103: icmp_seq=1 ttl=55 time=60.220 ms
64 bytes from 173.194.123.103: icmp_seq=2 ttl=55 time=8.872 ms
Before retry #2: sleeping 0.6 seconds
PING google.com (173.194.123.103): 56 data bytes
64 bytes from 173.194.123.103: icmp_seq=0 ttl=55 time=25.819 ms
64 bytes from 173.194.123.103: icmp_seq=1 ttl=55 time=16.382 ms
64 bytes from 173.194.123.103: icmp_seq=2 ttl=55 time=3.224 ms
Before retry #3: sleeping 1.2 seconds
PING google.com (173.194.123.103): 56 data bytes
64 bytes from 173.194.123.103: icmp_seq=0 ttl=55 time=58.438 ms
64 bytes from 173.194.123.103: icmp_seq=1 ttl=55 time=94.828 ms
64 bytes from 173.194.123.103: icmp_seq=2 ttl=55 time=61.075 ms
Before retry #4: sleeping 2.4 seconds
PING google.com (173.194.123.103): 56 data bytes
64 bytes from 173.194.123.103: icmp_seq=0 ttl=55 time=43.361 ms
64 bytes from 173.194.123.103: icmp_seq=1 ttl=55 time=32.171 ms
...
Check exit status for ultimate pass/fail.

Answer 5

For anyone wanting to actually wait until some time passed, taking into account the time of your command might be significant:
TIMEOUT_SEC=180
start_time="$(date -u +%s)"
while [ condition_or_just_true ]; do
current_time="$(date -u +%s)"
elapsed_seconds=$(($current_time-$start_time))
if [ $elapsed_seconds -gt $TIMEOUT_SEC ]; then
echo "timeout of $TIMEOUT_SEC sec"
exit 1
fi
echo "another attempt (elapsed $elapsed_seconds sec)"
some_command_and_maybe_sleep
done

Answer 6

I made some tweaks to this answer which let you switch on whether the timeout was reached, or whether the command succeed. Also, in this version there is a retry every second:
ELAPSED=0
started=$(mktemp)
echo "False" > $started
until the_command_here && echo "True" > $started || [ $ELAPSED -eq 30 ]
do
sleep 1
(( ELAPSED++ ))
done
if [[ $(cat $started) == "True" ]]
then
echo "the command completed after $ELAPSED seconds"
else
echo "timed out after $ELAPSED seconds"
exit 111
fi