This question already has answers here:
Execute a shell function with timeout
(11 answers)
Elegant solution to implement timeout for bash commands and functions
(3 answers)
Closed last year.
#!/usr/bin/env bash
exec 3>&1
fun_1(){
urlcount=$(wc -l < list.txt)
loopcount=0
for url in $(cat list.txt);
do
((loopcount++))
echo -e "\nProcessing URL #${loopcount} (of ${urlcount}) [ ${url} ]\n"
#the below curl command is the problem which i need to time it to maximum 5 minuts the continue the loop (because sometime it could take massive time to complete)
curl -s "http://localhost:5555/?url=$url"
# check for api status percentage
until [[ $(curl -s "http://localhost:5555/view/status" | jq -r '.status') == "100" ]]
do
echo -e "\n[-] Waiting for command $url\n"
sleep 5 || break
done
curl -s "http://localhost:5555/results" | jq -r '.results[]' >> results.txt
done
}
for domain in "$#"
do
fun_1 $domain 2>&1 >&3 | tee -a $WORKING_DIR/error_log.txt
done
This script has multiple functions like fun_1 which is execute one after another.
The problem is some functions which have a loop function using for loop or while loop could be running for very long time,
which is exhausting my server (VPS) and of course waste of time.
THE QUESTION is can I time this function to run for a certain time like one or two hour as maximum?
You could check how many seconds the script is running since start time, using the $SECONDS variable, and see if it's greater than some defined deadline.
I have replicated your execution flow while swapping something for dummy functionality.
#!/usr/bin/env bash
exec 3>&1
fun_1() {
domain="$1"
deadline="$2"
for i in {1..100}; do
echo "iteration $i"
# try until deadline is exceeded or curl succeeds
curl_success=0
until [ "$SECONDS" -gt "$deadline" ] || [ "$curl_success" -eq 1 ]; do
echo "retrying..."
sleep 5
done
# if deadline is exceeded, break out of the loop
if [ "$SECONDS" -gt "$deadline" ]; then
echo "Deadline exeeded"
break
fi
# curl results if deadline not exeeded
echo "curling results..."
done
}
deadline=$((SECONDS + 10))
for domain in "$#"; do
# if deadline is exceeded, break out of the loop
if [ "$SECONDS" -gt "$deadline" ]; then
echo "Deadline exeeded"
break
fi
fun_1 "$domain" "$deadline" 2>&1 >&3 | tee -a ./error_log.txt
done
You could also try to do some tricks with timeout. For example, if you can't modify your function to build a timeout into it.
Related
I am monitoring a log file and if PATTERN didn't appear in it within THRESHOLD seconds, the script should print "error", otherwise, it should print "clear". The script is working fine, but only if the log is rolling.
I've tried reading 'timeout' but didn't work.
log_file=/tmp/app.log
threshold=120
tail -Fn0 ${log_file} | \
while read line ; do
echo "${line}" | awk '/PATTERN/ { system("touch pattern.tmp") }'
code to calculate how long ago pattern.tmp touched and same is assigned to DIFF
if [ ${diff} -gt ${threshold} ]; then
echo "Error"
else
echo "Clear"
done
It is working as expected only when there is 'any' line printed in the app.log.
If the application got hung for any reason and the log stopped rolling, there won't be any output by the script.
Is there a way to detect the 'no output' of tail and do some command at that time?
It looks like the problem you're having is that the timing calculations inside your while loop never get a chance to run when read is blocking on input. In that case, you can pipe the tail output into a while true loop, inside of which you can do if read -t $timeout:
log_file=/tmp/app.log
threshold=120
timeout=10
tail -Fn0 "$log_file" | while true; do
if read -t $timeout line; then
echo "${line}" | awk '/PATTERN/ { system("touch pattern.tmp") }'
fi
# code to calculate how long ago pattern.tmp touched and same is assigned to diff
if [ ${diff} -gt ${threshold} ]; then
echo "Error"
else
echo "Clear"
fi
done
As Ed Morton pointed out, all caps variable names are not a good idea in bash scripts, so I used lowercase variable names.
How about something simple like:
sleep "$threshold"
grep -q 'PATTERN' "$log_file" && { echo "Clear"; exit; }
echo "Error"
If that's not all you need then edit your question to clarify your requirements. Don't use all upper case for non exported shell variable names btw - google it.
To build further on your idea, it might be beneficial to run the awk part in the background and a continuous loop to do the checking.
#!/usr/bin/env bash
log_file="log.txt"
# threshold in seconds
threshold=10
# run the following process in the background
stdbuf -oL tail -f0n "$log_file" \
| awk '/PATTERN/{system("touch "pattern.tmp") }' &
while true; do
match=$(find . -type f -iname "pattern.tmp" -newermt "-${threshold} seconds")
if [[ -z "${match}" ]]; then
echo "Error"
else
echo "Clear"
fi
done
This looks to me like a watchdog timer. I've implemented something like this by forcing a background process to update my log, so I don't have to worry about read -t. Here's a working example:
#!/usr/bin/env bash
threshold=10
grain=2
errorstate=0
while sleep "$grain"; do
date '+[%F %T] watchdog timer' >> log
done &
trap "kill -HUP $!" 0 HUP INT QUIT TRAP ABRT TERM
printf -v lastseen '%(%s)T'
tail -F log | while read line; do
printf -v now '%(%s)T'
if (( now - lastseen > threshold )); then
echo "ERROR"
errorstate=1
else
if (( errorstate )); then
echo "Recovered, yay"
errorstate=0
fi
fi
if [[ $line =~ .*PATTERN.* ]]; then
lastseen=$now
fi
done
Run this in one window, wait $threshold seconds for it to trigger, then in another window echo PATTERN >> log to see the recovery.
While this can be made as granular as you like (I've set it to 2 seconds in the example), it does pollute your log file.
Oh, and note that printf '%(%s)T' format requires bash version 4 or above.
I'm writing a script to monitor my sip trunk and attempt to fix it. If it fails to fix the issue 6 times, then reboot the server. The script is called by cron via #reboot. I first had nested While Loops but that didn't work correctly so I switched to a never ending While Loop with two nested If Loops to perform the functions of the script.
I was wondering if somebody could take a quick look and see if the way I am attacking it makes sense and is logical approach.
Thank You,
Script as it stands:
#!/bin/bash
pwd="/srv/scripts"
count=0
echo "Script Started on $(date -u) Failure.Count=$count" >> "$pwd/failures.count"
start=start
while [ $start = "start" ]; do
sleep 420
var="$(asterisk -rx "pjsip show registrations" | grep -o Registered)"
if [ "$var" != "Registered" ]; then
amportal restart
count=$(( $count + 1 ))
echo "Trunk Failure on $(date -u) Failure.Count=$count" >> "$pwd/failures.count"
fi
if [ "$count" -gt 5 ]; then
echo "Server Reboot due to Failure.Count=$count on $(date -u)" >> "$pwd/reboot.notification"
reboot
fi
done
There is no need to use a variable in the while loop, or to capture the grep output into a variable.
#!/bin/bash
pwd="/srv/scripts"
count=0
echo "Script Started on $(date -u) Failure.Count=$count" >> "$pwd/failures.count"
# No need for a variable here
while true; do
# Fix indentation
sleep 420
# Again, no need for a variable; use grep -q
if ! asterisk -rx "pjsip show registrations" | grep -q Registered
then
amportal restart
count=$(( $count + 1 ))
echo "Trunk Failure on $(date -u) Failure.Count=$count" >> "$pwd/failures.count"
fi
if [ "$count" -gt 5 ]; then
echo "Server Reboot due to Failure.Count=$count on $(date -u)" >> "$pwd/reboot.notification"
reboot
fi
done
I would perhaps also collect all the log notices in a single log file, and use a more traditional log format with a time stamp and the script's name bofore each message.
Should the counter reset to zero if you see a success? Having the server reboot because you disconnected the network cable at the wrong time seems like something you'd want to avoid.
How to check whether a long time task running properly? (How to launch a function after given time while a command is running)?
I'm writing a bash script to download some files regularly. I'd like to informed while a successful download is started.
But I couldn't make it right.
#!/bin/bash
URL="http://testurl"
FILENAME="/tmp/test"
function is_downloading() {
sleep 11
echo -e "$DOWNLOADING" # 0 wanted here with a failed download but always get empty
if [[ $DOWNLOADING -eq 1 ]]; then
echo "Send Message"
# send_msg
fi
}
while [[ 0 ]]; do
is_downloading &
DOWNLOADING=1
curl --connect-timeout 10 --speed-time 10 --speed-limit 1 --location -o "$FILENAME" "$URL"
DOWNLOADING=0
echo -e "$DOWNLOADING"
sleep 3600
done
is_downloading is running in another process, the best it could see is a copy of our variables at the time it started. Variables are not shared, bash does not support multi-threading (yet).
So you need to arrange some form of Inter-Process Communication (IPC). There are many methods available, I favour a named pipe (also known as a FIFO). Something like this:
function is_downloading() {
thepipe="$1"
while :
do
read -r DOWNLOADING < "$thepipe"
echo "$DOWNLOADING"
if [[ $DOWNLOADING -eq 1 ]]; then
echo "Send Message"
# send_msg
fi
done
}
pipename="/tmp/$0$$"
mkfifo "$pipename"
is_downloading "$pipename" &
trap 'kill %1;rm "$pipename"' INT TERM EXIT
while :
do
DOWNLOADING=1
echo "$DOWNLOADING" > "$pipename"
curl --connect-timeout 10 --speed-time 10 --speed-limit 1 --location -o "$FILENAME" "$URL"
DOWNLOADING=0
echo "$DOWNLOADING" > "$pipename"
sleep 3600
done
Modifications: taken the function call out of the loop. Tidy-up code put into a trap statement.
Given the following example:
#!/bin/bash
# var timeout = 5min.
while ( ! grep "Start" /var/log/azure/Microsoft.Azure.Extensions.DockerExtension/{ver}/extension.log); do
sleep 5
done
echo "hello world"
how would one change the script such it looped while not finding the line "start" in the extension.log file with a timeout option.
Additional it should also take into account that the {ver} is not static, and is a semver version "2.3.4" ect, it should take the highest version folder that exist.
Tested code:
#! /bin/bash
timeout=300 #seconds
now=$(date +%s)
let deadline=now+timeout
log=/tmp/log.txt
while ( [ ! -f "$log" ] || (! grep "Start" $log) )
do
if [ $(date +%s) -gt "$deadline" ]
then
echo "Timeout"
break
fi
echo "Sleeping"
sleep 3
done
echo "Done waiting"
Just change the log variable and it should work.
You're fairly close except BASH syntax issues as condition is not evaluated in (...).
This should work:
logfile='/var/log/azure/Microsoft.Azure.Extensions.DockerExtension/{ver}/extension.log'
while ! grep -q "Start" "$logfile"; do
sleep 5
done
echo "hello world"
I am trying to use a bash script to do several jobs in parallel. The jobs are memory intensive so I need to control the number that launch at a time. What I have is below, and it broadly works, but sometimes the delay loop is unaware of a job that has just been started, so several extra jobs get launched, causing the system to run out of memory.
Adding a sleep before the while statement in the delay loop reduces this problem, but does not completely eliminate it. Anyone know of a way to cure this. I'm running on Solaris if that's relevant.
#!/bin/bash
delay(){
while [ 8 -le $(ps -ef |grep myjob |wc -l) ]
do
sleep 1
done
}
./myjob -params1 &
delay
./myjob -params2 &
delay
./myjob -params3 &
delay
./myjob -params4 &
delay
.
.
.
GNU parallel utility http://www.gnu.org/software/parallel/ might be the right tool as it can be said more easy to use than xargs
First, I'll give you a stripped down example of something I do in a couple of my linux scripts. This should work on solaris, but I don't have any systems currently to test on. I modified a couple of things that used /proc, so if anything doesn't work let me know.
#!/bin/bash
# set the max # of threads
max_threads=4
# set the max system load
max_load=4
print_jobs(){
# flush finished jobs messages
jobs > /dev/null
for x in $(jobs -p) ; do
# print all jobs
echo "$x"
done
}
job_count(){
cnt=$(print_jobs $1)
if [ -n "$cnt" ]; then
wc -l <<< "$cnt"
else
echo 0
fi
}
cur_load(){
# get the 1 minute load average integer
uptime |sed 's/.*load average[s]*:[[:space:]]*\([^.]*\)\..*/\1/g'
}
main_function(){
# get current job count and load
jcnow=$(job_count)
loadnow=$(cur_load)
# first, enter a loop waiting for load/threads to be below thresholds
while [ $loadnow -ge $max_load ] || [ $jcnow -ge $max_threads ]; do
if ! [ $firstout ]; then
echo "entering sleep loop. load: $loadnow, threads: $jcnow"
st=$(date +%s)
local firstout=true
else
now=$(date +%s)
# if it's been 5 minutes, echo again:
if [ $(($now - $st)) -ge 300 ]; then
echo "still sleeping. load: $loadnow, threads: $jcnow"
st=$(date +%s)
fi
fi
sleep 5s
# refresh these variables for loop
loadnow=$(cur_load)
jcnow=$(job_count)
unset firstout
done
( ./myjob $# ) &
}
# do some actual work
for jobparams in "params1" "params2" "params3" "params4" "params5" "params6" "params7" ; do
main_function $jobparams
done
wait
A couple of caveats:
you should trap signals so you can kill child processes. I do not know how to do this in solaris, but this works for on linux: trap 'echo "exiting" ; rm -f $lockfile ; kill 0 ; exit' INT TERM EXIT
if load climbs while jobs are already running there's no facility to throttle down
If you're not concerned about load at all, this can be a bit simpler:
#!/bin/bash
# set the max # of threads
max_threads=4
print_jobs(){
# flush finished jobs messages
jobs > /dev/null
for x in $(jobs -p) ; do
# print all jobs
echo "$x"
done
}
job_count(){
cnt=$(print_jobs $1)
if [ -n "$cnt" ]; then
wc -l <<< "$cnt"
else
echo 0
fi
}
main_function(){
# get current job count
jcnow=$(job_count)
# first, enter a loop waiting for threads to be below thresholds
while [ $jcnow -ge $max_threads ]; do
if ! [ $firstout ]; then
echo "entering sleep loop. threads: $jcnow"
st=$(date +%s)
local firstout=true
else
now=$(date +%s)
# if it's been 5 minutes, echo again:
if [ $(($now - $st)) -ge 300 ]; then
echo "still sleeping. threads: $jcnow"
st=$(date +%s)
fi
fi
sleep 5s
# refresh these variables for loop
jcnow=$(job_count)
unset firstout
done
( ./myjob $# ) &
}
# do some actual work
for jobparams in "params1" "params2" "params3" "params4" "params5" "params6" "params7" ; do
main_function $jobparams
done
wait
Use xargs to do this. Pass it -n 1 to indicate one parameter per job and use the --max-jobs parameter to specify the number of concurrent processes.
Formulate your script in terms of a makefile and let make -j N sort it out through Parallel Execution.