How to wait until Kubernetes assigned an external IP to a LoadBalancer service? - bash

Question

Creating a Kubernetes LoadBalancer returns immediatly (ex: kubectl create -f ... or kubectl expose svc NAME --name=load-balancer --port=80 --type=LoadBalancer).
I know a manual way to wait in shell:
external_ip=""
while [ -z $external_ip ]; do
sleep 10
external_ip=$(kubectl get svc load-balancer --template="{{range .status.loadBalancer.ingress}}{{.ip}}{{end}}")
done
This is however not ideal:
Requires at least 5 lines Bash script.
Infinite wait even in case of error (else requires a timeout which increases a lot line count).
Probably not efficient; could use --wait or --wait-once but using those the command never returns.
Is there a better way to wait until a service external IP (aka LoadBalancer Ingress IP) is set or failed to set?

Answer 1

Just to add to the answers here, the best option right now is to use a bash script. For convenience, I've put it into a single line that includes exporting an environmental variable.
Command to wait and find Kubernetes service endpoint
bash -c 'external_ip=""; while [ -z $external_ip ]; do echo "Waiting for end point..."; external_ip=$(kubectl get svc NAME_OF_YOUR_SERVICE --template="{{range .status.loadBalancer.ingress}}{{.ip}}{{end}}"); [ -z "$external_ip" ] && sleep 10; done; echo "End point ready-" && echo $external_ip; export endpoint=$external_ip'
I've also modified your script so it only executes a wait if the ip isn't available. The last bit will export an environment variable called "endpoint"
Bash Script to Check a Given Service
Save this as check-endpoint.sh and then you can execute $sh check-endpoint.sh SERVICE_NAME
#!/bin/bash
# Pass the name of a service to check ie: sh check-endpoint.sh staging-voting-app-vote
# Will run forever...
external_ip=""
while [ -z $external_ip ]; do
echo "Waiting for end point..."
external_ip=$(kubectl get svc $1 --template="{{range .status.loadBalancer.ingress}}{{.ip}}{{end}}")
[ -z "$external_ip" ] && sleep 10
done
echo 'End point ready:' && echo $external_ip
Using this in a Codefresh Step
I'm using this for a Codefresh pipeline and it passes a variable $endpoint when it's done.
GrabEndPoint:
title: Waiting for endpoint to be ready
image: codefresh/plugin-helm:2.8.0
commands:
- bash -c 'external_ip=""; while [ -z $external_ip ]; do echo "Waiting for end point..."; external_ip=$(kubectl get svc staging-voting-app-vote --template="{{range .status.loadBalancer.ingress}}{{.ip}}{{end}}"); [ -z "$external_ip" ] && sleep 10; done; echo "End point ready-" && echo $external_ip; cf_export endpoint=$external_ip'

Answer 2

This is little bit tricky by working solution:
kubectl get service -w load-balancer -o 'go-template={{with .status.loadBalancer.ingress}}{{range .}}{{.ip}}{{"\n"}}{{end}}{{.err}}{{end}}' 2>/dev/null | head -n1

Answer 3

We had a similar problem on AWS EKS and wanted to have a one-liner for that to use in our CI pipelines. kubectl wait would be ideal, but will not be able to wait on arbitrary jsonpath until v1.23 (see this PR).
Until then we can simply "watch" the output of a command until a particular string is observed and then exit using the until loop:
until kubectl get service/<service-name> --output=jsonpath='{.status.loadBalancer}' | grep "ingress"; do : ; done
To avoid an infinite loop you could enhance it using timeout (brew install coreutils on a Mac):
timeout 10s bash -c 'until kubectl get service/<service-name> --output=jsonpath='{.status.loadBalancer}' | grep "ingress"; do : ; done'
Getting the ip after that is easy using:
kubectl get service/<service-name> --output=jsonpath='{.status.loadBalancer.ingress[0].ip}'
or when using a service like AWS EKS you most likely have hostname populated instead of ip:
kubectl get service/<service-name> --output=jsonpath='{.status.loadBalancer.ingress[0].hostname}'

Answer 4

Maybe this is not the solution that you're looking for but at least it has less lines of code:
until [ -n "$(kubectl get svc load-balancer -o jsonpath='{.status.loadBalancer.ingress[0].ip}')" ]; do
sleep 10
done

Answer 5

There's not really a "failed to set" condition because we will retry it forever. A failure might have been a transient error in the cloud provider or a quota issue that gets resolved over the course of hours or days, or any number of things. The only failure comes from "how long are you willing to wait?" - which only you can know.
We don't have a general "wait for expression" command because it ends up being arbitrarily complex and you're better off just coding that in a real language. Ergo the bash loop above. We could do better about having a 'watch' command, but it's still a timeout in the end.

Answer 6

Really just a clean-up of #Dan Garfield's working example; My OCD wouldn't let this slide. In this case:
on GCP
requesting an internal lb
with an annotation in a service definition
apiVersion: v1
kind: Service
metadata:
name: yo
annotations:
cloud.google.com/load-balancer-type: "Internal"
# external-dns.alpha.kubernetes.io/hostname: vault.stage.domain.tld.
...
NOTE: I've only been able to get external-dns to associate names to public IP addresses.
This has been scripted to accept a few arguments, now it's a library; example:
myServiceLB=$1
while true; do
successCond="$(kubectl get svc "$myServiceLB" \
--template="{{range .status.loadBalancer.ingress}}{{.ip}}{{end}}")"
if [[ -z "$successCond" ]]; then
echo "Waiting for endpoint readiness..."
sleep 10
else
sleep 2
export lbIngAdd="$successCond"
pMsg """
The Internal LoadBalancer is up!
"""
break
fi
done
Later, $lbIngAdd can be used to set records. Seems like -o jsonpath="{.status.loadBalancer.ingress[*].ip}" would work as well; whatever works.
Thanks for getting us started Dan :-)

Answer 7

Here's a generic bash function to watch with timeout, for any regexp in the output of a given command:
function watch_for() {
CMD="$1" # Command to watch. Variables should be escaped \$
REGEX="$2" # Pattern to search
ATTEMPTS=${3:-10} # Timeout. Default is 10 attempts (interval of second)
COUNT=0;
echo -e "# Watching for /$REGEX/ during $ATTEMPTS seconds, on the output of command:\n# $CMD"
until eval "$CMD" | grep -m 1 "$REGEX" || [[ $COUNT -eq $ATTEMPTS ]]; do
echo -e "$(( COUNT++ ))... \c"
sleep 1
done
if [[ $COUNT -eq $ATTEMPTS ]]; then
echo "# Limit of $ATTEMPTS attempts has exceeded."
return 1
fi
return 0
}
And here's how I used it to wait until a worker node gets an external IP (which took more than a minute):
$ watch_for "kubectl get nodes -l node-role.kubernetes.io/worker -o wide | awk '{print \$7}'" \
"[0-9]" 100
0... 1... 2... 3... .... 63... 64... 3.22.37.41