how to check if mongodb is up and ready to accept connections from bash script? - bash

I have a bash shell script which does a bunch of stuff before trying to mongorestore.
I want to make sure that not only MongoDB is up, but it is also ready to accept connections before i try restore.
Right now, what I am seeing is, mongo process is up but it take 45+ seconds to do the initial setup (setting up journal files etc) before it is ready to accept connections. Ideally I want to keep testing the connection in a loop and only when I am able to connect, I want to run mongorestore.
Can someone show me how to do this in Bash or point me in the right direction?

To test the connection in a loop like you suggest,
until nc -z localhost 27017
do
sleep 1
done

A solution using MongoDB Tools. Useful in a docker container or something similiar where you do not want to install nc.
until mongo --eval "print(\"waited for connection\")"
do
sleep 60
done
Based on that other guy's answer.

I recently had the same problem. I decided to configure mongod to log all it's output to a logfile and then wait in a loop checking the logfile until we see some output that suggests mongod is ready.
This is an example logfile output line we need to wait for:
Tue Dec 3 14:25:28.217 [initandlisten] waiting for connections on port 27017
This is the bash script I came up with:
#!/bin/bash
# Initialize a mongo data folder and logfile
mkdir -p /data/db
touch /var/log/mongodb.log
# Start mongodb with logging
# --logpath Without this mongod will output all log information to the standard output.
# --logappend Ensure mongod appends new entries to the end of the logfile. We create it first so that the below tail always finds something
/usr/bin/mongod --quiet --logpath /var/log/mongodb.log --logappend &
# Wait until mongo logs that it's ready (or timeout after 60s)
COUNTER=0
grep -q 'waiting for connections on port' /var/log/mongodb.log
while [[ $? -ne 0 && $COUNTER -lt 60 ]] ; do
sleep 2
let COUNTER+=2
echo "Waiting for mongo to initialize... ($COUNTER seconds so far)"
grep -q 'waiting for connections on port' /var/log/mongodb.log
done
# Now we know mongo is ready and can continue with other commands
...
Notice the script will not wait forever, it will timeout after 60s - you may or may not want that depending on your use case.

I needed Mongo running in Docker to initialize before creating a user. I combined the answers of Tom and Björn. This is the script I am using:
#!/bin/bash
# Wait until Mongo is ready to accept connections, exit if this does not happen within 30 seconds
COUNTER=0
until mongo --host ${MONGO_HOST} --eval "printjson(db.serverStatus())"
do
sleep 1
COUNTER=$((COUNTER+1))
if [[ ${COUNTER} -eq 30 ]]; then
echo "MongoDB did not initialize within 30 seconds, exiting"
exit 2
fi
echo "Waiting for MongoDB to initialize... ${COUNTER}/30"
done
# Connect to the MongoDB and execute the create users script
mongo ${FWRD_API_DB} /root/create-user.js --host ${MONGO_HOST} -u ${MONGO_ROOT_USER} -p ${MONGO_ROOT_PASSWORD} --authenticationDatabase admin

While Tom's answer will work quite well in most situations, it can fail if you are running your script with the -e flag. I mean you run your script with set -e at the very top. Why? Because Tom's answer relies on the exit status of the previous command, in this case grep -q which will fail if it does not find the required string, and therefore the entire script will fail. The -e flag documentation gives a clue on how to avoid this problem:
The shell does not exit if the command that fails is part of the
command list immediately following a while or until keyword, part of
the test in an if statement, part of any command executed in a && or
|| list except the command following the final && or ||, any command
in a pipeline but the last, or if the command’s return status is being
inverted with !.
So, one solution is to make the grep command part of the while condition. However, since he is running mongodb with the --logappend option the search string could appear as a result of a previous run. I combined that other guy answer with Tom's answer and it works really well:
# Wait until mongo logs that it's ready (or timeout after 60s)
COUNTER=0
while !(nc -z localhost 27017) && [[ $COUNTER -lt 60 ]] ; do
sleep 2
let COUNTER+=2
echo "Waiting for mongo to initialize... ($COUNTER seconds so far)"
done
I find that using tomcat is the best solution because it actually tests if there is something listening.

Possible Docker solution:
Given the docker_id for me it works reading the docker logs like:
until [ $(docker logs --tail all $docker_id | grep "waiting for connections on port" | wc -l) -gt 0 ]; do
printf '.'
sleep 1
done
then continue with any mongo-dependent task.

This approach is using in bitnami mongodb helm chart, but requires installed mongodb shell client:
mongosh --host {your_mongo_host} --port {your_mongo_port} --eval "db.adminCommand('ping')"

Related

How to wait until Kubernetes assigned an external IP to a LoadBalancer service?

Creating a Kubernetes LoadBalancer returns immediatly (ex: kubectl create -f ... or kubectl expose svc NAME --name=load-balancer --port=80 --type=LoadBalancer).
I know a manual way to wait in shell:
external_ip=""
while [ -z $external_ip ]; do
sleep 10
external_ip=$(kubectl get svc load-balancer --template="{{range .status.loadBalancer.ingress}}{{.ip}}{{end}}")
done
This is however not ideal:
Requires at least 5 lines Bash script.
Infinite wait even in case of error (else requires a timeout which increases a lot line count).
Probably not efficient; could use --wait or --wait-once but using those the command never returns.
Is there a better way to wait until a service external IP (aka LoadBalancer Ingress IP) is set or failed to set?
Just to add to the answers here, the best option right now is to use a bash script. For convenience, I've put it into a single line that includes exporting an environmental variable.
Command to wait and find Kubernetes service endpoint
bash -c 'external_ip=""; while [ -z $external_ip ]; do echo "Waiting for end point..."; external_ip=$(kubectl get svc NAME_OF_YOUR_SERVICE --template="{{range .status.loadBalancer.ingress}}{{.ip}}{{end}}"); [ -z "$external_ip" ] && sleep 10; done; echo "End point ready-" && echo $external_ip; export endpoint=$external_ip'
I've also modified your script so it only executes a wait if the ip isn't available. The last bit will export an environment variable called "endpoint"
Bash Script to Check a Given Service
Save this as check-endpoint.sh and then you can execute $sh check-endpoint.sh SERVICE_NAME
#!/bin/bash
# Pass the name of a service to check ie: sh check-endpoint.sh staging-voting-app-vote
# Will run forever...
external_ip=""
while [ -z $external_ip ]; do
echo "Waiting for end point..."
external_ip=$(kubectl get svc $1 --template="{{range .status.loadBalancer.ingress}}{{.ip}}{{end}}")
[ -z "$external_ip" ] && sleep 10
done
echo 'End point ready:' && echo $external_ip
Using this in a Codefresh Step
I'm using this for a Codefresh pipeline and it passes a variable $endpoint when it's done.
GrabEndPoint:
title: Waiting for endpoint to be ready
image: codefresh/plugin-helm:2.8.0
commands:
- bash -c 'external_ip=""; while [ -z $external_ip ]; do echo "Waiting for end point..."; external_ip=$(kubectl get svc staging-voting-app-vote --template="{{range .status.loadBalancer.ingress}}{{.ip}}{{end}}"); [ -z "$external_ip" ] && sleep 10; done; echo "End point ready-" && echo $external_ip; cf_export endpoint=$external_ip'
This is little bit tricky by working solution:
kubectl get service -w load-balancer -o 'go-template={{with .status.loadBalancer.ingress}}{{range .}}{{.ip}}{{"\n"}}{{end}}{{.err}}{{end}}' 2>/dev/null | head -n1
We had a similar problem on AWS EKS and wanted to have a one-liner for that to use in our CI pipelines. kubectl wait would be ideal, but will not be able to wait on arbitrary jsonpath until v1.23 (see this PR).
Until then we can simply "watch" the output of a command until a particular string is observed and then exit using the until loop:
until kubectl get service/<service-name> --output=jsonpath='{.status.loadBalancer}' | grep "ingress"; do : ; done
To avoid an infinite loop you could enhance it using timeout (brew install coreutils on a Mac):
timeout 10s bash -c 'until kubectl get service/<service-name> --output=jsonpath='{.status.loadBalancer}' | grep "ingress"; do : ; done'
Getting the ip after that is easy using:
kubectl get service/<service-name> --output=jsonpath='{.status.loadBalancer.ingress[0].ip}'
or when using a service like AWS EKS you most likely have hostname populated instead of ip:
kubectl get service/<service-name> --output=jsonpath='{.status.loadBalancer.ingress[0].hostname}'
Maybe this is not the solution that you're looking for but at least it has less lines of code:
until [ -n "$(kubectl get svc load-balancer -o jsonpath='{.status.loadBalancer.ingress[0].ip}')" ]; do
sleep 10
done
There's not really a "failed to set" condition because we will retry it forever. A failure might have been a transient error in the cloud provider or a quota issue that gets resolved over the course of hours or days, or any number of things. The only failure comes from "how long are you willing to wait?" - which only you can know.
We don't have a general "wait for expression" command because it ends up being arbitrarily complex and you're better off just coding that in a real language. Ergo the bash loop above. We could do better about having a 'watch' command, but it's still a timeout in the end.
Really just a clean-up of #Dan Garfield's working example; My OCD wouldn't let this slide. In this case:
on GCP
requesting an internal lb
with an annotation in a service definition
apiVersion: v1
kind: Service
metadata:
name: yo
annotations:
cloud.google.com/load-balancer-type: "Internal"
# external-dns.alpha.kubernetes.io/hostname: vault.stage.domain.tld.
...
NOTE: I've only been able to get external-dns to associate names to public IP addresses.
This has been scripted to accept a few arguments, now it's a library; example:
myServiceLB=$1
while true; do
successCond="$(kubectl get svc "$myServiceLB" \
--template="{{range .status.loadBalancer.ingress}}{{.ip}}{{end}}")"
if [[ -z "$successCond" ]]; then
echo "Waiting for endpoint readiness..."
sleep 10
else
sleep 2
export lbIngAdd="$successCond"
pMsg """
The Internal LoadBalancer is up!
"""
break
fi
done
Later, $lbIngAdd can be used to set records. Seems like -o jsonpath="{.status.loadBalancer.ingress[*].ip}" would work as well; whatever works.
Thanks for getting us started Dan :-)
Here's a generic bash function to watch with timeout, for any regexp in the output of a given command:
function watch_for() {
CMD="$1" # Command to watch. Variables should be escaped \$
REGEX="$2" # Pattern to search
ATTEMPTS=${3:-10} # Timeout. Default is 10 attempts (interval of second)
COUNT=0;
echo -e "# Watching for /$REGEX/ during $ATTEMPTS seconds, on the output of command:\n# $CMD"
until eval "$CMD" | grep -m 1 "$REGEX" || [[ $COUNT -eq $ATTEMPTS ]]; do
echo -e "$(( COUNT++ ))... \c"
sleep 1
done
if [[ $COUNT -eq $ATTEMPTS ]]; then
echo "# Limit of $ATTEMPTS attempts has exceeded."
return 1
fi
return 0
}
And here's how I used it to wait until a worker node gets an external IP (which took more than a minute):
$ watch_for "kubectl get nodes -l node-role.kubernetes.io/worker -o wide | awk '{print \$7}'" \
"[0-9]" 100
0... 1... 2... 3... .... 63... 64... 3.22.37.41

Docker kill an infinite process in a container after X amount of time

I am using the code found in this docker issue to basically start a container run a process within 20 seconds and if the process completes / does not complete / fails to execute / times out the container is killed regardless.
The code I am using currently is this:
#!/bin/bash
set -e
to=$1
shift
cont=$(docker run -d "$#")
code=$(timeout "$to" docker wait "$cont" || true)
docker kill $cont &> /dev/null
echo -n 'status: '
if [ -z "$code" ]; then
echo timeout
else
echo exited: $code
fi
echo output:
# pipe to sed simply for pretty nice indentation
docker logs $cont | sed 's/^/\t/'
docker rm $cont &> /dev/null
Which is almost perfect however if you run an infinite process (for example this python infinite loop):
while True:
print "inifinte loop"
The whole system jams up and the app crashes, after reading around a bit I think it has something to do with the STDOUT Buffer but I have absolutely no idea what that means?
The problem you have is with a process that is writing massive amounts of data to stdout.
These messages get logged into a file which grows infinitely.
Have a look at (depending on your system's location for log files):
sudo find /var/lib/docker/containers/ -name '*.log' -ls
You can remove old log files if they are of no interest.
One possibility is to start your docker run -d daemon
under a ulimit restriction on the max size a file can be.
Add to the start of your script, for example:
ulimit -f 20000 -c 0
This limits file sizes to 20000*1024 bytes, and disables core file dumps, which you expect
to get from infinite loops where writes are forced to fail.
Please add & at the end of
cont=$(docker run -d "$#")&
It will run the process in background.
I don't know dockers but if it still fail to stop you may also add just after this line the following :
mypid=$!
sleep 20 && kill $mypid
Regards

Terminal Application to Keep Web Server Process Alive

Is there an app that can, given a command and options, execute for the lifetime of the process and ping a given URL indefinitely on a specific interval?
If not, could this be done on the terminal as a bash script? I'm almost positive it's doable through terminal, but am not fluent enough to whip it up within a few minutes.
Found this post that has a portion of the solution, minus the ping bits. ping runs on linux, indefinitely; until it's actively killed. How would I kill it from bash after say, two pings?
General Script
As others have suggested, use this in pseudo code:
execute command and save PID
while PID is active, ping and sleep
exit
This results in following script:
#!/bin/bash
# execute command, use '&' at the end to run in background
<command here> &
# store pid
pid=$!
while ps | awk '{ print $1 }' | grep $pid; do
ping <address here>
sleep <timeout here in seconds>
done
Note that the stuff inside <> should be replaces with actual stuff. Be it a command or an ip address.
Break from Loop
To answer your second question, that depends in the loop. In the loop above, simply track the loop count using a variable. To do that, add a ((count++)) inside the loop. And do this: [[ $count -eq 2 ]] && break. Now the loop will break when we're pinging for a second time.
Something like this:
...
while ...; do
...
((count++))
[[ $count -eq 2 ]] && break
done
ping twice
To ping only a few times, use the -c option:
ping -c <count here> <address here>
Example:
ping -c 2 www.google.com
Use man ping for more information.
Better practice
As hek2mgl noted in a comment below, the current solution may not suffice to solve the problem. While answering the question, the core problem will still persist. To aid to that problem, a cron job is suggested in which a simple wget or curl http request is sent periodically. This results in a fairly easy script containing but one line:
#!/bin/bash
curl <address here> > /dev/null 2>&1
This script can be added as a cron job. Leave a comment if you desire more information how to set such a scheduled job. Special thanks to hek2mgl for analyzing the problem and suggesting a sound solution.
Say you want to start a download with wget and while it is running, ping the url:
wget http://example.com/large_file.tgz & #put in background
pid=$!
while kill -s 0 $pid #test if process is running
do
ping -c 1 127.0.0.1 #ping your adress once
sleep 5 #and sleep for 5 seconds
done
A nice little generic utility for this is Daemonize. Its relevant options:
Usage: daemonize [OPTIONS] path [arg] ...
-c <dir> # Set daemon's working directory to <dir>.
-E var=value # Pass environment setting to daemon. May appear multiple times.
-p <pidfile> # Save PID to <pidfile>.
-u <user> # Run daemon as user <user>. Requires invocation as root.
-l <lockfile> # Single-instance checking using lockfile <lockfile>.
Here's an example of starting/killing in use: flickd
To get more sophisticated, you could turn your ping script into a systemd service, now standard on many recent Linuxes.

How to wait for an open port with netcat?

I'm trying to do a custom dockerfile with jenkins on it. I would to wait until port 8080 is open instead of doing an ugly 'sleep 60' with netcat but I'm not very confident with bash scripts and netcat.
Here is an example of what i'm trying to do:
#!/bin/bash
opened=0
while [ "$opened" == "0" ]; do
echo "Waiting jenkins to launch on 8080..."
nc -vz localhost 8080
done
echo "Jenkins launched"
You can't set netcat to wait until some port is open, so you have to add part for waiting before next check is made. Try this:
#!/bin/bash
echo "Waiting jenkins to launch on 8080..."
while ! nc -z localhost 8080; do
sleep 0.1 # wait for 1/10 of the second before check again
done
echo "Jenkins launched"
I suggest the following one liners:
## netcat version:
timeout 22 sh -c 'until nc -z $0 $1; do sleep 1; done' stackoverflow.com 443
## pure bash version:
timeout 22 bash -c 'until printf "" 2>>/dev/null >>/dev/tcp/$0/$1; do sleep 1; done' stackoverflow.com 443
Both commands exit as soon as connection is established, trying every second for up to 22 seconds.
Note that thanks to timeout command exit code is 0 when port is accessible otherwise 124 (if no connection established within given time).
As suggested here, you could also do the following if you don't have nc installed but just bash and coreutils:
#!/bin/bash
echo "Waiting jenkins to launch on 8080..."
while ! timeout 1 bash -c "echo > /dev/tcp/localhost/8080"; do
sleep 1
done
echo "Jenkins launched"
I have found this a common enough problem to write a utility to wait for a port to open, with an optional timeout:
# without timeout
wait-port localhost:8080
# timeout after a minute
wait-port -t 60000 localhost:8080
It's open source and available at github.com/dwmkerr/wait-port. Hopefully others will find it useful!
To expand on user987339's answer, here's how to easily wait for a port in your terminal:
waitport function
Add this function to your ~/.bashrc setup file:
waitport() {
while ! nc -z localhost $1 ; do sleep 1 ; done
}
Log out then back in to load ~/.bashrc. Then, run this command to verify that port 3000 has a server listening to it:
$ waitport 3000
Connection to localhost port 3000 [tcp/hbci] succeeded!
This has been validated on macOS. It might not work on Fedora/CentOS, as they lack the -z option for netcat.
To add onto the excellent answers above, if this is something used very often it may be worthwhile to use tooling for that purpose. I wrote and use uup all the time for this use case.
In your example, the command to run would be:
uup localhost:8080 -r
providing an output like:
Here is a for-loop example that has a timeout, so it tries e.g. for 10 times, with exponential backoff (2,4,8,16 seconds etc), but finally gives up. Netcat has also 1 second timeout.
for EXPONENTIAL_BACKOFF in {1..10}; do
nc -w 1 -z db.local 3306 && break;
DELAY=$((2**$EXPONENTIAL_BACKOFF))
echo "db not yet available, sleeping for $DELAY seconds"
sleep $DELAY
done
The output is:
db not yet available, sleeping for 2 seconds
db not yet available, sleeping for 4 seconds
db not yet available, sleeping for 8 seconds
db not yet available, sleeping for 16 seconds
I use this script to check the port before running tests on CI.
#!/bin/bash
for _ in `seq 1 20`; do
echo -n .
if nc -z localhost $1; then
exit 0
fi
sleep 0.5
done
exit 1
$ bin/wait-port 3306
Here is my one-line Bash solution (with netcat) that waits for 10 sec for a TCP connection, and give you feedback whether succeeded or not and while is waiting, and return an exit 0 code if the port is open, otherwise 1:
bash -c 'echo -n "Waiting port 8080 .."; for _ in `seq 1 40`; do echo -n .; sleep 0.25; nc -z localhost 8080 && echo " Open." && exit ; done; echo " Timeout!" >&2; exit 1'
You can replace the hardcoded port 8080 by $1 and remove the bash -c if the snippet is saved in a script file wait-port than then is called within a console with: wait-port 8080.
This is a recording of 3 terminals, two waiting until a port is opened and the other terminals open one of the port, so while one of the wait succeed, the other timed-out:
Although the line has many instructions not one, it may be useful if you need to execute the wait "remotely" in a host where you cannot store the script first, e.g. in a Docker container.
I used this to wait for a couple of ports to be open, without netcat:
while (! (: </dev/tcp/localhost/27017) &> /dev/null || ! (: </dev/tcp/localhost/9200) &> /dev/null); do
sleep 2;
done
Change localhost and the ports as needed.
For those people who are having trouble with nc: invalid option -- 'z'
I was trying to set this up in a docker image. Surprisingly, there was no option of -z in nc in that image.
Image was - Linux elasticsearch 4.4.0-101-generic #124~14.04.1-Ubuntu SMP Fri Nov 10 19:05:36 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
I used the following loop to wait until the port was opened.
#!/bin/bash
echo "Waiting elastic search to launch on 9200..."
open=0;
while [ $open -eq 0 ]
do
check_port=`nc -v -w 1 -i 1 127.0.0.1 9200 &> /dev/stdout`
echo $check_port
if [[ "$check_port" == *"Connected to"* ]]
then
break
fi
sleep 1
done
echo "Elastic Search launched"
Following is the one-liner of the above script:
open=0;while [ $open -eq 0 ]; do check_port=`nc -v -w 1 -i 1 127.0.0.1 9200 &> /dev/stdout`; echo $check_port; if [[ "$check_port" == *"Connected to"* ]]; then break; fi; sleep 1; done
I have written a utility to wait for a port to open, it can also check MySQL, PostgreSQL, Redis and etc availability.
# Checking TCP port
wait4x tcp localhost:8080
# Checking TCP port with specific timeout (5 Minutes)
wait4x tcp localhost:8080 -t 5m
It's open source and available at https://github.com/atkrad/wait4x. Hopefully others will find it useful!
If you are looking for a docker-run-one-liner, check out my Docker image based on the netcat loop:
$ docker run --rm igops/wait-for-port HOST PORT
E.g.,
$ docker run --rm igops/wait-for-port 172.17.0.1 80
will query 172.17.0.1:80 each 0.1s and exit when the connection is established:
Waiting for 172.17.0.1:80...
OK
Waiting for another container to response:
$ docker network create my-bridge
$ docker run --rm -d --net my-bridge --net-alias my-mongo mongo
$ docker run --rm --net my-bridge igops/wait-for-port my-mongo 27017
$ echo "Mongo is up at this point, do some useful stuff here"
Waiting for some service on the docker host:
$ docker run --rm --add-host="docker-host:host-gateway" igops/wait-for-port docker-host 22
$ echo "SSH server is running"
Waiting for another container which published some port:
$ docker run --rm -d -p 27017:27107 mongo
$ docker run --rm --add-host="docker-host:host-gateway" igops/wait-for-port docker-host 27017
$ echo "Mongo is up"
Read more

Why are my local commands failing to run after remote ssh commands in the same script?

I've done my homework, but I think I may be mixing apples and oranges here. My script is designed to run a remote inline series of commands, exit, and then run some additional LOCAL commands. It has to be done remote first, as these services are for a fail-over agent. The problem is that after the remote ssh line disconnects, the entire script just stops. I'm not sure why the disconnect is halting the entire script. Perhaps the exit line is to blame?
#!/bin/bash
#
### Run remote svc restarts and then Local restarts
#
exec ssh -t REMOTEHOST 'stop svc1; restart svc2; start svc3; exit'
(SCRIPT FAILS HERE)
## Run local shell (This works independently, but not in the entire script)
rst=`pgrep -n failoversvc`
echo "Stopping 1st service at `date | awk '{print $2,$3,$4}'`" && service 1 stop >> SYNCLOG.txt
sleep 2
echo "Restarting 2nd service at `date | awk '{print $2,$3,$4}'`" && service 2 restart >> SYNCLOG.txt
if rst="";then
echo "Starting 3rd service at `date | awk '{print $2,$3,$4}'`" && service 3 start >> SYNCLOG.txt
else
echo "3rd Service PID not found! Check for functionality"
fi
I took a look at but THIS I wasn't able to get the results I was looking for.
exec is a very brutal command: it completely replaces the current process (in this case, your shell that's running the script) with the command you specify. Unless exec fails, nothing after that line in your script will ever run. This is by design, that's what exec is for.
If you want your script to continue after the ssh, simply remove exec.

Resources