How to wait for postgres startup in ENTRYPOINT? - bash

What would be the best approach to wait for postgres to fully start up inside my ENTRYPOINT before moving on executing nosetests?
Right now I've timed the startup on my machine to around 50 seconds. So I'm just sleeping for 60 seconds. This does not feel good as this might not work when being run on another machine.
ENTRYPOINT \
runuser -l postgres -c '/usr/lib/postgresql/9.3/bin/postgres -D /var/lib/postgresql/9.3/main -c config_file=/etc/postgresql/9.3/main/postgresql.conf & ' && \
sleep 60 && \
nosetests --verbose --cover-erase --with-coverage --cover-package=stalker
This is the output of starting up postgres:
2017-02-13 13:46:49.541 UTC [9] LOG: database system was interrupted; last known up at 2017-02-13 12:53:23 UTC
2017-02-13 13:47:37.951 UTC [9] LOG: database system was not properly shut down; automatic recovery in progress
2017-02-13 13:47:37.994 UTC [9] LOG: redo starts at 0/1783EA0
2017-02-13 13:47:37.995 UTC [9] LOG: record with zero length at 0/17841E8
2017-02-13 13:47:37.995 UTC [9] LOG: redo done at 0/17841B8
2017-02-13 13:47:37.995 UTC [9] LOG: last completed transaction was at log time 2017-02-13 12:53:23.731984+00
2017-02-13 13:47:38.384 UTC [9] LOG: MultiXact member wraparound protections are now enabled
2017-02-13 13:47:38.387 UTC [7] LOG: database system is ready to accept connections
2017-02-13 13:47:38.387 UTC [13] LOG: autovacuum launcher started
Please, I do understand that this goes against the convention of running multiple commands in ENTRYPOINT. I have good reasons to do this in this case.

Thanks to #zeppelin for suggesting pg_isready. I ended up using this:
#!/bin/bash
# wait-for-postgres.sh
set -e
cmd="$#"
timer="5"
until runuser -l postgres -c 'pg_isready' 2>/dev/null; do
>&2 echo "Postgres is unavailable - sleeping for $timer seconds"
sleep $timer
done
>&2 echo "Postgres is up - executing command"
exec $cmd
I'm using this in my ENTRYPOINT:
ENTRYPOINT \
# Start PostgreSQL
runuser -l postgres -c '/usr/lib/postgresql/9.3/bin/postgres -D /var/lib/postgresql/9.3/main -c config_file=/etc/postgresql/9.3/main/postgresql.conf & ' && \
# Exectute tests when db is up
./wait-for-postgres.sh nosetests --verbose --cover-erase --with-coverage --cover-package=stalker

I will normally use a small "tcp-port-wait" script, like this:
#!/bin/bash
set -e
if [ -z "$1" -o -z "$2" ]
then
echo "tcp-port-wait - block until specified TCP port becomes available"
echo "Usage: ntcp-port-wait HOST PORT"
exit 1
fi
echo Waiting for port $1:$2 to become available...
while ! nc -z $1 $2 2>/dev/null
do
let elapsed=elapsed+1
if [ "$elapsed" -gt 90 ]
then
echo "TIMED OUT !"
exit 1
fi
sleep 1;
done
echo "READY !"
To wait until a certain service is up and ready.
In case of Postgresql, the default port it will listen on is 5432, so the command would be:
tcp-port-wait localhost 5432
That will block until Postgresql service is ready to serve connections, on :5432 on the loopback interface inside the container (when run in context of the ENTRYPOINT script).
You certainly have to copy this script into your container,
by adding a line like that to your Dockerfile:
COPY tcp-port-wait /usr/local/bin/
before you can use it.
And also install the netcat utility.
You can use this for other types of services, like Tomcat or Mysql, too.
And, if you need, you can also wait "outside the container", like this:
docker exec my_container tcp-port-wait localhost 5432
Note, there are certainly other ways to do this, e.g. by using some orchestration tool, like docker-compose with healthcheck directive, or some process manager inside the container itself.

Related

How to make the ssh with -o StrictHostKeyChecking=no running on the docker in order to ssh the host work without exiting the script execution?

I have a script, that is running inside the docker container some actions we need for some internal debugging purposes:
set -eu
echo "Starting i/o test for host"
IP_HOST=$(ip a | grep -oE "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b" | grep 172.17 | awk 'NR==1{print $1}')
echo "Detected IP of host is $IP_HOST"
sshpass -p tcuser ssh -o StrictHostKeyChecking=no docker#localhost -t -t
echo "Then"
the output here produce exactly:
bash-5.0# sh /etc/cron.d/iotesthost.sh
Mon Mar 2 12:43:59 UTC 2020
Starting i/o test for host
Detected IP of host is 172.17.0.1
Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts.
( '>')
/) TC (\ Core is distributed with ABSOLUTELY NO WARRANTY.
(/-_--_-\) www.tinycorelinux.net
and as the last line is reached, the script makes me exiting bash or crond execution. So I can't go ahead with processing other lines after sshpass/ssh, so I never reach echo "Then"
That is the reason of exiting the script execution and how to work it around, still keeping all the features of accepting keys (i need is as each time the docker container calls for the script it is new)
If I ignore -t -t, I'm getting error according to https://stackoverflow.com/a/7122115/1759063
see https://askubuntu.com/questions/87449/how-to-disable-strict-host-key-checking-in-ssh
or
echo "StrictHostKeyChecking no" >> /etc/ssh_config
for a global solution. Please note that there is no security check anymore then.

Running a bash script after the kafka-connect docker is up and running

I have the following docker file
FROM confluentinc/cp-kafka-connect:5.3.1
ENV CONNECT_PLUGIN_PATH=/usr/share/java
# JDBC-MariaDB
RUN wget -nv -P /usr/share/java/kafka-connect-jdbc/ https://downloads.mariadb.com/Connectors/java/connector-java-2.4.4/mariadb-java-client-2.4.4.jar
# SNMP Source
RUN wget -nv -P /tmp/ https://github.com/KarthikDuggirala/kafka-connect-snmp/releases/download/0.0.1.11/kafka-connect-snmp-0.0.1.11.tar.gz
RUN mkdir /tmp/kafka-connect-snmp && tar -xf /tmp/kafka-connect-snmp-0.0.1.11.tar.gz -C /tmp/kafka-connect-snmp/
RUN mv /tmp/kafka-connect-snmp/usr/share/kafka-connect/kafka-connect-snmp /usr/share/java/
# COPY script and make it executable
COPY plugins-config.sh /usr/share/kafka-connect-script/plugins-config.sh
RUN ["chmod", "+x", "/usr/share/kafka-connect-script/plugins-config.sh"]
#entrypoint
ENTRYPOINT [ "./usr/share/kafka-connect-script/plugins-config.sh" ]
and the following bash script
#!/bin/bash
#script to configure kafka connect with plugins
#export CONNECT_REST_ADVERTISED_HOST_NAME=localhost
#export CONNECT_REST_PORT=8083
url=http://$CONNECT_REST_ADVERTISED_HOST_NAME:$CONNECT_REST_PORT/connectors
curl_command="curl -s -o /dev/null -w %{http_code} $url"
sleep_second=5
sleep_second_counter=0
max_seconds_to_wait=30
echo "Waiting for Kafka Connect to start listening on localhost"
echo "HOST: $CONNECT_REST_ADVERTISED_HOST_NAME , PORT: $CONNECT_REST_PORT"
while [[ $(eval $curl_command) -eq 000 ]]
do
echo "In"
echo -e $date " Kafka Connect listener HTTP state: " $(eval $curl_command) " (waiting for 200) $sleep_second_counter"
echo "Going to sleep for $sleep_second seconds"
# sleep $sleep_second
echo "Finished sleeping"
# ((sleep_second_counter+=$sleep_second))
echo "Finished counter"
done
echo "Out"
nc -vz $CONNECT_REST_ADVERTISED_HOST_NAME $CONNECT_REST_PORT
I try to run the docker and using docker logs to see whats happening, and I am expecting that the script would run and wait till the kafka connect is started. But apparently after say few seconds the script or (I dont know what is hanging) hangs and I do not see any console prints anymore.
I am a bit lost what is wrong, so I need some guidance on what is that I am missing or is this not the correct way
What I am trying to do
I want to have logic defined that I could wait for kafka connect to start then run the curl command
curl -X POST -H "Content-Type: application/json" --data '{"name":"","config":{"connector.class":"com.github.jcustenborder.kafka.connect.snmp.SnmpTrapSourceConnector","topic":"fm_snmp"}}' http://localhost:8083/connectors
PS: I cannot use docker-compose way to do it, since there are places I have to use docker run
The problem here is that ENTRYPOINT will run when the container starts and will prevent the default CMD to run since the script will loop waiting for the server to be up , that is the script will loop forever since the CMD will not run.
you need to do one of the following:
start the kafka connect server in your Entrypoint and your script in CMD or running your script outside the container ....

How to check if docker daemon is running?

I am trying to create a bash utility script to check if a docker daemon is running in my server.
Is there a better way of checking if the docker daemon is running in my server other than running a code like this?
ps -ef | grep docker
root 1250 1 0 13:28 ? 00:00:04 /usr/bin/dockerd --selinux-enabled
root 1598 1250 0 13:28 ? 00:00:00 docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --shim docker-containerd-shim --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docker/libcontainerd/containerd --runtime docker-runc
root 10997 10916 0 19:47 pts/0 00:00:00 grep --color=auto docker
I would like to create a bash shell script that will check if my docker daemon is running. If it is running then do nothing but if it is not then have the docker daemon started.
My pseudocode is something like this. I am thinking of parsing the output of my ps -ef but I just would like to know if there is a more efficient way of doing my pseudocode.
if(docker is not running)
run docker
end
P.S.
I am no linux expert and I just need to do this utility on my own environment.
I made a little Script (Mac Osx) to ensure Docker is running by checking the exit code of docker stats.
#!/bin/bash
#Open Docker, only if is not running
if (! docker stats --no-stream ); then
# On Mac OS this would be the terminal command to launch Docker
open /Applications/Docker.app
#Wait until Docker daemon is running and has completed initialisation
while (! docker stats --no-stream ); do
# Docker takes a few seconds to initialize
echo "Waiting for Docker to launch..."
sleep 1
done
fi
#Start the Container..
This works for me on Ubuntu
$ systemctl status docker
You have a utility called pgrep on almost all the Linux systems.
You can just do:
pgrep -f docker > /dev/null || echo "starting docker"
Replace the echo command with your docker starting command.
if curl -s --unix-socket /var/run/docker.sock http/_ping 2>&1 >/dev/null
then
echo "Running"
else
echo "Not running"
fi
Ref: Docker api v1.28
The following works on macOS and on Windows if git bash is installed. On macOS open /Applications/Docker.app would start the docker deamon. Haven't seen anything similar for Windows however.
## check docker is running at all
## based on https://stackoverflow.com/questions/22009364/is-there-a-try-catch-command-in-bash
{
## will throw an error if the docker daemon is not running and jump
## to the next code chunk
docker ps -q
} || {
echo "Docker is not running. Please start docker on your computer"
echo "When docker has finished starting up press [ENTER} to continue"
read
}
You can simply:
docker version > /dev/null 2>&1
The exit code of that command will be stored to $? so you can check if it's 0, then docker is running.
docker version will exit 1 if daemon is not running. If other issues are encountered, such as docker not being installed at all, the exit code will vary.
But in the end of the day, if docker is installed and daemon is running, the exit code will be 0.
The 2>&1 will redirect stderr to stdout and > /dev/null will redirect stdout to /dev/null practically silencing the output no matter what was the result of the execution.
You could also just check for the existence of /var/run/docker.pid.
Following #madsonic, I went for the following
#!/bin/bash
if (! docker stats --no-stream 2>/dev/null); then
# On Mac OS this would be the terminal command to launch Docker
open /Applications/Docker.app
echo -n "Waiting for Docker to launch"
sleep 1
# Wait until Docker daemon is running and has completed initialisation
while (! docker stats --no-stream >/dev/null 2>&1); do
# Docker takes a few seconds to initialize
echo -n "."
sleep 1
done
fi
echo
echo "Docker started"
A function could looks so:
isRunning {
`ps -ef | grep "[d]ocker" | awk {'print $2'}`
}
I created a script to start, stop, restart a mongodb-server.
You only need to change some path inside the scripts, and i also works for you:
Script
I'm sure you want to start the docker daemon so here's the code to start it before executing your Docker run statement:
sudo systemctl start docker

Execute simultaneous scripts on remote machines and wait until the process completes

The original idea was copy out a script to each IP address which would do a yum-install some RPMs and some configuration steps on each machine. Since the yum-install takes about 20 minutes, the hope was to do the install simultaneously on each machine then wait for all the spawned processes to finish before continuing.
#!/bin/bash
PEM=$1
IPS=$2
for IP in IPS; do
scp -i $PEM /tmp/A.sh ec2-user#IP:/tmp
ssh -i $PEM ec2-user#$IP chmod 777 /tmp/A.sh
done
for IP in IPS; do
ssh -t -i $PEM ec2-user#$IP sudo /tmp/A.sh &
done
wait
echo "IPS have been configured."
exit 0
Executing a remote sudo execute command in background on three IP addresses yields three error messages. Obviously, there's flaw in my logic.
Pseudo-terminal will not be allocated because stdin is not a terminal.
Pseudo-terminal will not be allocated because stdin is not a terminal.
Pseudo-terminal will not be allocated because stdin is not a terminal.
sudo: sorry, you must have a tty to run sudo
sudo: sorry, you must have a tty to run sudo
sudo: sorry, you must have a tty to run sudo
All machines are CentOS 6.5
You need to tell ssh not to read from standard input
ssh -n -t root#host "sleep 100" &
Here's an example
drao#darkstar:/tmp$ cat a
date
ssh -n -t me#host1 "sleep 100" &
ssh -n -t me#host2 "sleep 100" &
wait
date
darkstar:/tmp$ . ./a
Mon May 16 15:32:16 CEST 2016
Pseudo-terminal will not be allocated because stdin is not a terminal.
Pseudo-terminal will not be allocated because stdin is not a terminal.
[1]- Done ssh -n -t me#host1 "sleep 100"
[2]+ Done ssh -n -t me#host2 "sleep 100"
Mon May 16 15:33:57 CEST 2016
darkstar:/tmp
That waited in all 101 seconds. Obviously I've the ssh keys so I did not get prompted fro the password.
But looking at your output it looks like sudo on the remote machine is failing ... you might not even need -n.
just to push some devopsy doctrine on you.
Ansible does this amazingly well.

how can know ssh is disconected and retry with bash script

I'm using reverse ssh for connecting to remote client , Operator run reverse one time and leave client system
how can i write bash script , when reverse ssh disconnected from server retry to connect to server (ssh)
Use autossh. Autossh "automatically restart[s] SSH sessions and tunnels"
sudo apt-get install autossh
I use autossh to to keep open reverse tunnel that I depend on. It works very well, even with long periods of lost connection.
Here is the script I use to create the tunnel:
#!/bin/bash
AUTOSSH_GATETIME=0
export AUTOSSH_GATETIME
autossh -f -N -R 8022:localhost:22 username#host -o "ServerAliveInterval 45" -o "ServerAliveCountMax 2"
I execute this script at boot with this cronjob:
#reboot /home/scripts/./persistent-tunnel.sh
If you simply want to retry a command until it succeeds, you can use this pattern:
while ! ssh [...]
do
echo "Command failed, retrying..." >&2
done
I have a slightly different method.
My method always tries to reconnect you if you have a dirty disconnection: '~.' or 'Connection closed by remote host.'
But if you disconnect with 'CRTL+D' or with 'exit' it just disconnects and show you some info of the connections.
I have a slightly different method.
My method always tries to reconnect you if you have a dirty disconnection: '~.' or 'Connection closed by remote host.'
But if you disconnect with 'CRTL+D' or with 'exit' it just disconnects and show you some info of the connections.
#/bin/bash
if [ -z "$1" ]
then
echo '''
Please also provide ssh connection details.
'''
exit 1
fi
retries=0
repeat=true
today=$(date)
while "$repeat"
do
((retries+=1)) &&
echo "Try number $retries..." &&
today=$(date) &&
ssh "$#" &&
repeat=false
sleep 5
done
echo """
Disconnected sshx after a successful login.
Total number of tries = $retries
Connected at:
$today
"""
You might want to take a look into ssh options ServerAliveInterval, ServerAliveCountMax and TCPKeepAlive because sometimes your line dies without making this obvious, let me demonstrate:
#!/bin/sh
while true; do
ssh -T user#host \
-o IdentityFile=~/.ssh/tunnel \
-o UserKnownHostsFile=~/.ssh/known_hosts.tunnel \
pkill -f "^sshd:\ user\ \ \ \ $" # needs to be edited for nearly every case
sleep 2
ssh -T -N user#host \
-o IdentityFile=~/.ssh/tunnel \
-o UserKnownHostsFile=~/.ssh/known_hosts.tunnel \
-o Batchmode=yes \
-o ExitOnForwardFailure=yes \
-o ServerAliveCountMax=1 \
-o ServerAliveInterval=60 \
-o LocalForward=127.0.0.1:2501=127.0.0.1:25 \
-o RemoteForward=127.0.0.1:2501=127.0.0.1:25
sleep 60
done
You can use netstat -ntp | grep ":22" or ss -ntp | grep ":22" to see established connections to ssh port, then use grep to filter the ip address you're looking for. If you don't find a connection then reconnect the tunnel.
Use autossh if it works on your version of Linux. It did not on mine as it was an outdated Linux distribution for a custom NAS box.
The alternative is a simple bash script in crontab like this:
maintain_reverse_ssh_tunnel.sh
if ! netstat -planet |grep myserver_ip_or_name |grep ESTABLISHED > /dev/null; then
echo "REVERSE SSH DOWN - Restarting the tunnels"
ssh -fN -R 32999:localhost:22 -R 28080:localhost:80 myusername#myserver_ip_or_name
fi
Replace myusername and myserver_ip_or_name with those of your user and server.
Then add an entry to crontab by typing crontab -e and adding the following line:
1 * * * * /path_to_my_script/maintain_reverse_ssh_tunnel.sh
Make sure to have the execute permissions on the script:
chmod 755 maintain_reverse_ssh_tunnel.sh

Resources