I'm migrating a bunch of openvz containers and can only do one at a time for reasons. This is very time consuming and if I'm not watching the destination node constantly I won't know if the migrations fail or are done.
So I'm trying to write a little shell script to do two things. First, make sure the container ID that is being migrated shows up in the list of containers. If it does not, exit the script and send me an email. Second, as long as the first condition is true, watch for the status of the container to change to running and once that is true send me an email.
I have the second part of this working using until, I'm not sure that is the best way to go about it though and I need the first part, making sure the container exists to work as well. Obviously both these tests need to run every loop in case the migration fails. I just can't wrap my head around how to do this.
Here is what I have so far:
#!/bin/bash
read -p "Container ID: " -e CID
until vzlist -a | grep $CID | grep running
do
sleep 600
done
echo "Migration of container $CID complete" | mail -s "Migration complete" red#cted.com
If I'm understanding how to interpret vxlist -a correctly, something like this should work:
#!/bin/bash
emailTarget="red#cted.com"
read -p "Container ID: " -e CID
while true; do # This loops until something `break`s it out of the loop
# Capture the container status, so we can run multiple checks with
# only one run of `vzlist`.
containerStatus=$(vzlist -a | grep "$CID")
if [[ -z "$containerStatus" ]]; then
# If the the result was the empty string, our container is not
# in the list, so apparently it's failed.
echo "Migration of container $CID failed" | mail -s "Migration failed" "$emailTarget"
break
elif [[ "$containerStatus" = *"running"* ]]; then
# It's in the list *and* has "running" status -- migration succeeded!
echo "Migration of container $CID complete" | mail -s "Migration complete" "$emailTarget"
break
fi
# If neither of those conditions was met, it's still trying;
# wait 10 minutes and check again.
sleep 600
done
Related
This is part of bigger code that checking the OS version and choosing the correct condition by it.
( after checking OS version, it will go to this if condition: )
if [ "$AK" = "$OS6" ]
then
if [ "$(ls -la /etc/init.d/discagent 2>/dev/null | wc -l)" == 1 ]
then
/etc/init.d/discagent restart 2>&1 > /dev/null
/etc/init.d/discagent status |& grep -qe 'running'
if [ $? -eq 0 ] ; then
echo " Done "
else
echo " Error "
fi
fi
fi
If im hashing service discagent restart the pipeline is passing.
But if im not hashing it then it hang and not given any errors, And on the output file it is showing only the second server (out of few) that its hang on, And not moving to the next server.
what could be the issue?
p. S
while running it direct on the server it is working.
Run this script manually on the servers, that this script will be running on.
You can use xtrace -x which would show each statement before being executed, when you use with -v with -x so this would be -xv and the statement would be outputted, and then the line after the output of the statement with the variables substituted before the code is substituted.
using -u would show you the exact error when this occurs.
Another command is the trap command to debug your bash script.
More information on the following webpage,
https://linuxconfig.org/how-to-debug-bash-scripts
Initially I was having trouble piping a curl and grep command because I was getting the same exit code if the server was down as I was for if the pattern was not found and that wasn't good. Here are some examples of solutions I was given for that:
#!/bin/bash
if curl http://192.168.1.2/api/query | grep -q mypattern; then
echo "Found pattern"
elif [ "${PIPESTATUS[0]}" -eq 0 ]; then
echo "Server up"
else
echo "Server down"
fi
Here is the other one:
if while ! curl http://192.168.1.2:8080/api/query; do sleep 1; done |
grep -q mypattern; then
echo "success"
fi
Now I am stuck again and trying to adapt them. Right now it will retrieve the contents of the queue. If it can't reach the server, it keeps trying until is does reach it. It will then process the queue to find the pattern. But I need to confirm the server is reachable AND process the queue all in one action. If EITHER the server is unreachable or the pattern is found, keep trying until the server responds AND the pattern is not found in the queue. After that, then I will do whatever. I am having trouble trying to come up with an efficient solution.
You can pipe the standard output of the entire while loop (which includes the output of the curl command) to another loop that runs until it finds your pattern in its input:
while ! curl http://...; do
# Write to standard error so that it isn't sent to grep
echo "Server down, waiting one second" >&2
sleep 1
done |
until grep -q mypattern; do
echo "Waiting for pattern to match"
done
Creating a Kubernetes LoadBalancer returns immediatly (ex: kubectl create -f ... or kubectl expose svc NAME --name=load-balancer --port=80 --type=LoadBalancer).
I know a manual way to wait in shell:
external_ip=""
while [ -z $external_ip ]; do
sleep 10
external_ip=$(kubectl get svc load-balancer --template="{{range .status.loadBalancer.ingress}}{{.ip}}{{end}}")
done
This is however not ideal:
Requires at least 5 lines Bash script.
Infinite wait even in case of error (else requires a timeout which increases a lot line count).
Probably not efficient; could use --wait or --wait-once but using those the command never returns.
Is there a better way to wait until a service external IP (aka LoadBalancer Ingress IP) is set or failed to set?
Just to add to the answers here, the best option right now is to use a bash script. For convenience, I've put it into a single line that includes exporting an environmental variable.
Command to wait and find Kubernetes service endpoint
bash -c 'external_ip=""; while [ -z $external_ip ]; do echo "Waiting for end point..."; external_ip=$(kubectl get svc NAME_OF_YOUR_SERVICE --template="{{range .status.loadBalancer.ingress}}{{.ip}}{{end}}"); [ -z "$external_ip" ] && sleep 10; done; echo "End point ready-" && echo $external_ip; export endpoint=$external_ip'
I've also modified your script so it only executes a wait if the ip isn't available. The last bit will export an environment variable called "endpoint"
Bash Script to Check a Given Service
Save this as check-endpoint.sh and then you can execute $sh check-endpoint.sh SERVICE_NAME
#!/bin/bash
# Pass the name of a service to check ie: sh check-endpoint.sh staging-voting-app-vote
# Will run forever...
external_ip=""
while [ -z $external_ip ]; do
echo "Waiting for end point..."
external_ip=$(kubectl get svc $1 --template="{{range .status.loadBalancer.ingress}}{{.ip}}{{end}}")
[ -z "$external_ip" ] && sleep 10
done
echo 'End point ready:' && echo $external_ip
Using this in a Codefresh Step
I'm using this for a Codefresh pipeline and it passes a variable $endpoint when it's done.
GrabEndPoint:
title: Waiting for endpoint to be ready
image: codefresh/plugin-helm:2.8.0
commands:
- bash -c 'external_ip=""; while [ -z $external_ip ]; do echo "Waiting for end point..."; external_ip=$(kubectl get svc staging-voting-app-vote --template="{{range .status.loadBalancer.ingress}}{{.ip}}{{end}}"); [ -z "$external_ip" ] && sleep 10; done; echo "End point ready-" && echo $external_ip; cf_export endpoint=$external_ip'
This is little bit tricky by working solution:
kubectl get service -w load-balancer -o 'go-template={{with .status.loadBalancer.ingress}}{{range .}}{{.ip}}{{"\n"}}{{end}}{{.err}}{{end}}' 2>/dev/null | head -n1
We had a similar problem on AWS EKS and wanted to have a one-liner for that to use in our CI pipelines. kubectl wait would be ideal, but will not be able to wait on arbitrary jsonpath until v1.23 (see this PR).
Until then we can simply "watch" the output of a command until a particular string is observed and then exit using the until loop:
until kubectl get service/<service-name> --output=jsonpath='{.status.loadBalancer}' | grep "ingress"; do : ; done
To avoid an infinite loop you could enhance it using timeout (brew install coreutils on a Mac):
timeout 10s bash -c 'until kubectl get service/<service-name> --output=jsonpath='{.status.loadBalancer}' | grep "ingress"; do : ; done'
Getting the ip after that is easy using:
kubectl get service/<service-name> --output=jsonpath='{.status.loadBalancer.ingress[0].ip}'
or when using a service like AWS EKS you most likely have hostname populated instead of ip:
kubectl get service/<service-name> --output=jsonpath='{.status.loadBalancer.ingress[0].hostname}'
Maybe this is not the solution that you're looking for but at least it has less lines of code:
until [ -n "$(kubectl get svc load-balancer -o jsonpath='{.status.loadBalancer.ingress[0].ip}')" ]; do
sleep 10
done
There's not really a "failed to set" condition because we will retry it forever. A failure might have been a transient error in the cloud provider or a quota issue that gets resolved over the course of hours or days, or any number of things. The only failure comes from "how long are you willing to wait?" - which only you can know.
We don't have a general "wait for expression" command because it ends up being arbitrarily complex and you're better off just coding that in a real language. Ergo the bash loop above. We could do better about having a 'watch' command, but it's still a timeout in the end.
Really just a clean-up of #Dan Garfield's working example; My OCD wouldn't let this slide. In this case:
on GCP
requesting an internal lb
with an annotation in a service definition
apiVersion: v1
kind: Service
metadata:
name: yo
annotations:
cloud.google.com/load-balancer-type: "Internal"
# external-dns.alpha.kubernetes.io/hostname: vault.stage.domain.tld.
...
NOTE: I've only been able to get external-dns to associate names to public IP addresses.
This has been scripted to accept a few arguments, now it's a library; example:
myServiceLB=$1
while true; do
successCond="$(kubectl get svc "$myServiceLB" \
--template="{{range .status.loadBalancer.ingress}}{{.ip}}{{end}}")"
if [[ -z "$successCond" ]]; then
echo "Waiting for endpoint readiness..."
sleep 10
else
sleep 2
export lbIngAdd="$successCond"
pMsg """
The Internal LoadBalancer is up!
"""
break
fi
done
Later, $lbIngAdd can be used to set records. Seems like -o jsonpath="{.status.loadBalancer.ingress[*].ip}" would work as well; whatever works.
Thanks for getting us started Dan :-)
Here's a generic bash function to watch with timeout, for any regexp in the output of a given command:
function watch_for() {
CMD="$1" # Command to watch. Variables should be escaped \$
REGEX="$2" # Pattern to search
ATTEMPTS=${3:-10} # Timeout. Default is 10 attempts (interval of second)
COUNT=0;
echo -e "# Watching for /$REGEX/ during $ATTEMPTS seconds, on the output of command:\n# $CMD"
until eval "$CMD" | grep -m 1 "$REGEX" || [[ $COUNT -eq $ATTEMPTS ]]; do
echo -e "$(( COUNT++ ))... \c"
sleep 1
done
if [[ $COUNT -eq $ATTEMPTS ]]; then
echo "# Limit of $ATTEMPTS attempts has exceeded."
return 1
fi
return 0
}
And here's how I used it to wait until a worker node gets an external IP (which took more than a minute):
$ watch_for "kubectl get nodes -l node-role.kubernetes.io/worker -o wide | awk '{print \$7}'" \
"[0-9]" 100
0... 1... 2... 3... .... 63... 64... 3.22.37.41
I am using the code found in this docker issue to basically start a container run a process within 20 seconds and if the process completes / does not complete / fails to execute / times out the container is killed regardless.
The code I am using currently is this:
#!/bin/bash
set -e
to=$1
shift
cont=$(docker run -d "$#")
code=$(timeout "$to" docker wait "$cont" || true)
docker kill $cont &> /dev/null
echo -n 'status: '
if [ -z "$code" ]; then
echo timeout
else
echo exited: $code
fi
echo output:
# pipe to sed simply for pretty nice indentation
docker logs $cont | sed 's/^/\t/'
docker rm $cont &> /dev/null
Which is almost perfect however if you run an infinite process (for example this python infinite loop):
while True:
print "inifinte loop"
The whole system jams up and the app crashes, after reading around a bit I think it has something to do with the STDOUT Buffer but I have absolutely no idea what that means?
The problem you have is with a process that is writing massive amounts of data to stdout.
These messages get logged into a file which grows infinitely.
Have a look at (depending on your system's location for log files):
sudo find /var/lib/docker/containers/ -name '*.log' -ls
You can remove old log files if they are of no interest.
One possibility is to start your docker run -d daemon
under a ulimit restriction on the max size a file can be.
Add to the start of your script, for example:
ulimit -f 20000 -c 0
This limits file sizes to 20000*1024 bytes, and disables core file dumps, which you expect
to get from infinite loops where writes are forced to fail.
Please add & at the end of
cont=$(docker run -d "$#")&
It will run the process in background.
I don't know dockers but if it still fail to stop you may also add just after this line the following :
mypid=$!
sleep 20 && kill $mypid
Regards
I am writing a bash script to manage deployments to a GF server for several environments. What I would like to know is how can I get the result of a GF command and then determine whether to continue or exit.
For example
Say I want to redeploy, I have this script
$GF_ASADMIN --port $GF_PORT redeploy --name $EAR_FILE_NAME --keepstate=true $EAR_FILE | tee -a $LOG
The variables are already defined. So GF will start to redeploy and either suceed or fail. I want to check if it does and act accordingly. I have this right after it.
RC=$?
if [[ $RC -eq 0 ]];
then echoInfo "Application Successfully redeployed!" | tee -a $LOG;
else
echoError "Failed to redeploy application!"
exit 1
fi;
However, it doesnt really seem to work .
The problem is the pipe
$GF_ASADMIN ... | tee -a $LOG
$? reflects the return code of tee.
Your are looking for PIPESTATUS. See man bash:
PIPESTATUS
An array variable (see Arrays below) containing a list of exit
status values from the processes in the most-recently-executed
foreground pipeline (which may contain only a single command).
See also this example to clarify the PIPESTATUS
false | true
echo ${PIPESTATUS[#]}
Output is: 1 0
The corrected code is:
RC=${PIPESTATUS[0]}
Or try using a code block redirect, for example:
{
if "$GF_ASADMIN" --port $GF_PORT redeploy --name "$EAR_FILE_NAME" --keepstate=true "$EAR_FILE"
then
echo Info "Application Successfully redeployed!"
else
echo Error "Failed to redeploy application!" >&2
exit 1
fi
} | tee -a "$LOG"