I would like to use timeout command, but It supports only to timeout a single command.
My goal is to do something like this - waiting untill a list of ports are up:
timeout 60 for port in $ports
do
while ! nc -zv localhost $port; do
sleep 1
done
done
if [[ $? -ne 0 ]]; then
echo not all ports up on time
fi
I want the for loop to stop if 60 seconds have passed, and check if it was success or not.
I understand that I can achieve this by using something like:
timeout 60 bash -c "..."
But this will be very unreadable. I thought maybe a bash function would work as a command but it didn't work..
Any ideas?
After some tests, I successfully found a way to implement this using bash functions, although it looks a bit weird, it is still more readable. Here is an example:
function my_loop() {
for port in $ports; do
while ! nc -zv localhost $port; do
sleep 1
done
done
}
export -f my_loop
timeout 60 bash -c "my_loop"
if [[ $? -ne 0 ]]; then
echo not all ports up on time
fi
Related
When writing shell scripts, I repeatedly have the need to wait for a given condition to become true, e.g. a remote URL becoming available (checked with curl) or a file that should exist, etc.
Ideally, I'd like to have a function or script await such that I can write, e.g.,
await [[ some condition ]]
and it would check the condition every second until it becomes true or a timeout occurs. Ideally I can set the polling interval and the timeout.
Is there a tool for this out there?
You can use an until loop:
until some condition
do
sleep 5
done
e.g.
until nc -z localhost 22
do
echo "SSHd is not up yet. Waiting..."
sleep 5
done
If you want to add a timeout, you'll have to add that separately with a counter or using the internal SECONDS variable:
SECONDS=0
until nc -z localhost 22
do
if (( SECONDS > 60 ))
then
echo "Giving up..."
exit 1
fi
echo "SSHd is not up yet. Waiting..."
sleep 5
done
I'd use the timeout utility around a while loop that uses sleep.
E.g.:
timeout 1 bash -c 'while :; do echo check; sleep 0.1; done '
I have a bash program that checks that a daemon in a given port is working:
nc -z localhost $port > /dev/null
if [ "$?" != "0" ]
then
echo The server on port $port is not working
exit
fi
This program works perfectly in CentOS 6. However, it seems that CentOS 7 has changed the underlying implementation for nc command (CentOS 6 seems to use Netcat and CentOS 7 uses another thing called Ncat) and now the -z switch doesn't work:
$ nc -z localhost 8080
nc: invalid option -- 'z'
Looking to the man nc page in CentOS 7 I don't see any clear alternative to -z. Any suggestion on how I should fix my bash program to make it work in CentOS 7?
And a trimmed down version for what you really want:
#!/bin/bash
# Start command: nohup ./check_server.sh 2>&1 &
check_server(){ # Start shell function
checkHTTPcode=$(curl -sLf -m 2 -w "%{http_code}\n" "http://10.10.10.10:8080/" -o /dev/null)
if [ $checkHTTPcode -ne 200 ]
then
# Check failed. Do something here and take any corrective measure here if nedded like restarting server
# /sbin/service httpd restart >> /var/log/check_server.log
echo "$(date) Check Failed " >> /var/log/check_server.log
else
# Everything's OK. Lets move on.
echo "$(date) Check OK " >> /var/log/check_server.log
fi
}
while true # infinite check
do
# Call function every 30 seconds
check_server
sleep 30
done
according this post, I used:
nc -w1 localhost $port < /dev/null
( this work on debian too, but it's slower than nc -z )
Creating a Kubernetes LoadBalancer returns immediatly (ex: kubectl create -f ... or kubectl expose svc NAME --name=load-balancer --port=80 --type=LoadBalancer).
I know a manual way to wait in shell:
external_ip=""
while [ -z $external_ip ]; do
sleep 10
external_ip=$(kubectl get svc load-balancer --template="{{range .status.loadBalancer.ingress}}{{.ip}}{{end}}")
done
This is however not ideal:
Requires at least 5 lines Bash script.
Infinite wait even in case of error (else requires a timeout which increases a lot line count).
Probably not efficient; could use --wait or --wait-once but using those the command never returns.
Is there a better way to wait until a service external IP (aka LoadBalancer Ingress IP) is set or failed to set?
Just to add to the answers here, the best option right now is to use a bash script. For convenience, I've put it into a single line that includes exporting an environmental variable.
Command to wait and find Kubernetes service endpoint
bash -c 'external_ip=""; while [ -z $external_ip ]; do echo "Waiting for end point..."; external_ip=$(kubectl get svc NAME_OF_YOUR_SERVICE --template="{{range .status.loadBalancer.ingress}}{{.ip}}{{end}}"); [ -z "$external_ip" ] && sleep 10; done; echo "End point ready-" && echo $external_ip; export endpoint=$external_ip'
I've also modified your script so it only executes a wait if the ip isn't available. The last bit will export an environment variable called "endpoint"
Bash Script to Check a Given Service
Save this as check-endpoint.sh and then you can execute $sh check-endpoint.sh SERVICE_NAME
#!/bin/bash
# Pass the name of a service to check ie: sh check-endpoint.sh staging-voting-app-vote
# Will run forever...
external_ip=""
while [ -z $external_ip ]; do
echo "Waiting for end point..."
external_ip=$(kubectl get svc $1 --template="{{range .status.loadBalancer.ingress}}{{.ip}}{{end}}")
[ -z "$external_ip" ] && sleep 10
done
echo 'End point ready:' && echo $external_ip
Using this in a Codefresh Step
I'm using this for a Codefresh pipeline and it passes a variable $endpoint when it's done.
GrabEndPoint:
title: Waiting for endpoint to be ready
image: codefresh/plugin-helm:2.8.0
commands:
- bash -c 'external_ip=""; while [ -z $external_ip ]; do echo "Waiting for end point..."; external_ip=$(kubectl get svc staging-voting-app-vote --template="{{range .status.loadBalancer.ingress}}{{.ip}}{{end}}"); [ -z "$external_ip" ] && sleep 10; done; echo "End point ready-" && echo $external_ip; cf_export endpoint=$external_ip'
This is little bit tricky by working solution:
kubectl get service -w load-balancer -o 'go-template={{with .status.loadBalancer.ingress}}{{range .}}{{.ip}}{{"\n"}}{{end}}{{.err}}{{end}}' 2>/dev/null | head -n1
We had a similar problem on AWS EKS and wanted to have a one-liner for that to use in our CI pipelines. kubectl wait would be ideal, but will not be able to wait on arbitrary jsonpath until v1.23 (see this PR).
Until then we can simply "watch" the output of a command until a particular string is observed and then exit using the until loop:
until kubectl get service/<service-name> --output=jsonpath='{.status.loadBalancer}' | grep "ingress"; do : ; done
To avoid an infinite loop you could enhance it using timeout (brew install coreutils on a Mac):
timeout 10s bash -c 'until kubectl get service/<service-name> --output=jsonpath='{.status.loadBalancer}' | grep "ingress"; do : ; done'
Getting the ip after that is easy using:
kubectl get service/<service-name> --output=jsonpath='{.status.loadBalancer.ingress[0].ip}'
or when using a service like AWS EKS you most likely have hostname populated instead of ip:
kubectl get service/<service-name> --output=jsonpath='{.status.loadBalancer.ingress[0].hostname}'
Maybe this is not the solution that you're looking for but at least it has less lines of code:
until [ -n "$(kubectl get svc load-balancer -o jsonpath='{.status.loadBalancer.ingress[0].ip}')" ]; do
sleep 10
done
There's not really a "failed to set" condition because we will retry it forever. A failure might have been a transient error in the cloud provider or a quota issue that gets resolved over the course of hours or days, or any number of things. The only failure comes from "how long are you willing to wait?" - which only you can know.
We don't have a general "wait for expression" command because it ends up being arbitrarily complex and you're better off just coding that in a real language. Ergo the bash loop above. We could do better about having a 'watch' command, but it's still a timeout in the end.
Really just a clean-up of #Dan Garfield's working example; My OCD wouldn't let this slide. In this case:
on GCP
requesting an internal lb
with an annotation in a service definition
apiVersion: v1
kind: Service
metadata:
name: yo
annotations:
cloud.google.com/load-balancer-type: "Internal"
# external-dns.alpha.kubernetes.io/hostname: vault.stage.domain.tld.
...
NOTE: I've only been able to get external-dns to associate names to public IP addresses.
This has been scripted to accept a few arguments, now it's a library; example:
myServiceLB=$1
while true; do
successCond="$(kubectl get svc "$myServiceLB" \
--template="{{range .status.loadBalancer.ingress}}{{.ip}}{{end}}")"
if [[ -z "$successCond" ]]; then
echo "Waiting for endpoint readiness..."
sleep 10
else
sleep 2
export lbIngAdd="$successCond"
pMsg """
The Internal LoadBalancer is up!
"""
break
fi
done
Later, $lbIngAdd can be used to set records. Seems like -o jsonpath="{.status.loadBalancer.ingress[*].ip}" would work as well; whatever works.
Thanks for getting us started Dan :-)
Here's a generic bash function to watch with timeout, for any regexp in the output of a given command:
function watch_for() {
CMD="$1" # Command to watch. Variables should be escaped \$
REGEX="$2" # Pattern to search
ATTEMPTS=${3:-10} # Timeout. Default is 10 attempts (interval of second)
COUNT=0;
echo -e "# Watching for /$REGEX/ during $ATTEMPTS seconds, on the output of command:\n# $CMD"
until eval "$CMD" | grep -m 1 "$REGEX" || [[ $COUNT -eq $ATTEMPTS ]]; do
echo -e "$(( COUNT++ ))... \c"
sleep 1
done
if [[ $COUNT -eq $ATTEMPTS ]]; then
echo "# Limit of $ATTEMPTS attempts has exceeded."
return 1
fi
return 0
}
And here's how I used it to wait until a worker node gets an external IP (which took more than a minute):
$ watch_for "kubectl get nodes -l node-role.kubernetes.io/worker -o wide | awk '{print \$7}'" \
"[0-9]" 100
0... 1... 2... 3... .... 63... 64... 3.22.37.41
I've been trying to learn the syntax of logic statements in bash, how to do if/else, pipes and stuff. I'm trying to build a bash script, but I fail miserably after 3 hours of not getting how this stuff works.
Now I need this little script, I'll try to explain it using a generalized code, or call it whatever you want. Here you go:
while variable THRESHOLD = 10
{
if netstat -anltp contains a line with port 25565
then set variable THRESHOLD to 0 and variable PROCNUM to the process number,
else add 1 to variable THRESHOLD
sleep 5 seconds
}
kill the process No. PROCNUM
restart the script
Basically, what it does is, that once the socket closes, after a few tries, it kills the process which was listening on that port.
I'm pretty sure it's possible, but I can't figure out how to do it properly. Mostly because I don't understand pipes and am not really familiar with grep. Thank you for your help, in advance.
Don't want be offensive, but if you can write a "generalized" program all you need is learn th syntax of the while, if for bash and read the man pages of the grep and kill and so on...
And the pipes are the same as in your garden. Having two things: tap and pond. You can fill your pond with many ways (e.g. with rain). Also, you can open your tap getting water. But if you want fill the pond with the water from a tap, need a pipe. That's all. Syntax:
tap | pond
the output from a tap
connect with a pipe
to the (input) of the pond
e.g.
netstat | grep
the output from a netstat
connect with a pipe
to the input of the grep
that's all magic... :)
About the syntax: You tagged your question as bash.
So googling for a bash while syntax will show to you, this Beginners Bash guide
http://tldp.org/LDP/Bash-Beginners-Guide/html/sect_09_02.html
to, and you can read about the if in the same website.
Simply can't believe than after 3 hours you cannot understand basic while and if syntax to write your program with a bash syntax - especially, when you able write an "generalized" program...
is is not to hard (with modifying the 1st example in the above page) to write:
THRESHOLD="0"
while [ $THRESHOLD -lt 10 ]
do
#do the IF here
THRESHOLD=$[$THRESHOLD+1]
done
and so on...
#!/bin/bash
# write a little function
function do_error {
echo "$#" 1>&2
exit 1
}
# make the user pass in the path to the executable
if [ "$1" == "" ]; then
do_error "USAGE: `basename $0` <path to your executable>"
fi
if [ ! -e $1 ]; then
do_error "Unable to find executable at $1"
fi
if [ ! -x $1 ]; then
do_error "$1 is not an executable"
fi
PROC="$1"
PROCNAME=`basename $PROC`
# forever
while [ 1 ]; do
# check whether the process is up
proc=`ps -ef | grep $PROCNAME 2>/dev/null`
# if it is not up, start it in the background (unless it's a daemon)
if [ "$proc" == "" ]; then
$PROC &
fi
# reinitialize the threshold
threshold=0
# as long as we haven't tried 10 time, continue trying
while [ threshold -lt 10 ]; do
# run netstat, look for port 25565, and see if the connection is established.
# it would be better to checks to make sure
# that the process we expect is the one that established the connection
output=`netstat -anp | grep 25565 | grep ESTABLISHED 2>/dev/null`
# if netstat found something, then our process was able to establish the connection
if [ "$output" != "" ]; then
threshold = 0
else
# increment the threshold
threshold=$((threshold + 1))
fi
# i would sleep for one second
sleep 1
done
kill -9 $PROCNUM
done
My connection is flaky, however I have a backup one. I made some bash script to check for connectivity and change connection if the present one is dead. Please help me improve them.
The scripts almost works, except for not waiting long enough to receive an IP (it cycles to next step in the until loop too quick). Here goes:
#!/bin/bash
# Invoke this script with paths to your connection specific scripts, for example
# ./gotnet.sh ./connection.sh ./connection2.sh
until [ -z "$1" ] # Try different connections until we are online...
do
if eval "ping -c 1 google.com"
then
echo "we are online!" && break
else
$1 # Runs (next) connection-script.
echo
fi
shift
done
echo # Extra line feed.
exit 0
And here is an example of the slave scripts:
#!/bin/bash
ifconfig wlan0 down
ifconfig wlan0 up
iwconfig wlan0 key 1234567890
iwconfig wlan0 essid example
sleep 1
dhclient -1 -nw wlan0
sleep 3
exit 0
Here's one way to do it:
#!/bin/bash
while true; do
if ! [ "`ping -c 1 google.com; echo $?`" ]; then #if ping exits nonzero...
./connection_script1.sh #run the first script
sleep 10 #give it a few seconds to complete
fi
if ! [ "`ping -c 1 google.com; echo $?`" ]; then #if ping *still* exits nonzero...
./connection_script2.sh #run the second script
sleep 10 #give it a few seconds to complete
fi
sleep 300 #check again in five minutes
done
Adjust the sleep times and ping count to your preference. This script never exits so you would most likely want to run it with the following command:
./connection_daemon.sh 2>&1 > /dev/null & disown
Have you tried omitting the -nw option from the dhclient command?
Also, remove the eval and quotes from your if they aren't necessary. Do it like this:
if ping -c 1 google.com > /dev/null 2>&1
Trying using ConnectTimeout ${timeout} somewhere.